Speech recognition applications are becoming more and more practical nowadays. Various interactive speech aware applications are available in the market. But they are usually meant for and executed on the traditional general-purpose computers. Due to the rapid technological advancements, bigger and more complex tasks are being efficiently performed consuming less time and efforts. This can be further simplified with a more handsfree approach with the help of Voice Controlled System. For implementing this, we use an open source platform called Jasper for Raspberry Pi. Our aim is to build an automated personal secretary who could remind us of our schedule and optimize the rudimentary ways of performing tasks at one’s commands.