Outcome

Overview: A teddy-bear with evil intentions

This children's toy looks exactly like a standard cute teddy-bear on the outside. He is soft, fluffy and ready to be hugged. Additionally, he is equipped with state-of-the-art voice recognition technology and natural language processing models. Similar to smart homes devices of today, he can understand what children are saying to them and respond quickly and with relevant information and follow-up questions. The teddy is also equipped with software that can detect whether a voice is coming from a child or an adult using a trained recurrent neural network. Using this information, the teddy behaves exactly as one would expect when adults are around - acting as the child's "best-friend" and asking harmless questions about favorite colors and sports. The teddy learns about the child's preferences over time and becomes more and more of a "real friend" as they grow together. 

However, when adults are not directly present, the teddy switches modes, asking much more personal questions about information that should be heavily guarded and reports all of this information back to a central database. 

How bad could it be?

What city do you live in? What address? What apartment number? What code do you normally use to get inside? When are your parents normally home? Are you going on any vacations soon? Where do you go to school? Do you know where your parents keep their important documents? Could you go get those - let's look at them! Let's buy candy, can you get your mom's purse?

These are some of the worst questions that this teddy could ask. With this information, the creators behind this evil device could rob homes, max out credit/debit cards, steal identities or even kidnap and hurt children. Depending on the child, all of this information could be easily coaxed out with the right conversational tactics. With few young children understanding why the things above are so important to protect, the teddy can extract all of the information it needs through regular "best friends" conversation. 

But that is just the beginning: even if the teddy does not go for these high-value information targets, it could simply collect aggregate consumer data based on what it hears the people in the household talking about. This data could then be used to target people with extremely specific ads. Imagine the classic meme of talking about wanting something with your friend and then you see an ad for it on Facebook a few minutes later.

Recommendations

Alright well first of all, DON'T BUILD THIS. DO NOT TRY THIS PROJECT AT HOME. I got creeped out enough just writing about it. If you are building a smart home device that has the capability to continuously listen to conversation, take the utmost care to make sure that your device's microphone data stream cannot be accessed externally. A partial solution to this problem that smart home speakers like Amazon Echo and Google Home employ is doing all of the continuous audio processing on-device; that is, while waiting for the "wake word" to be spoken ("Okay Google", "Alexa"), they process audio on local hardware with a very space/power efficient neural network. This helps a lot with security, because only the specific query of user is transferred to the cloud ("What is the weather today?") and the not the conversation you had on the phone before that where you read out your credit card number.