Outcome

Intention

When we think of the various devices that compose our current idea of a smart home, what comes to mind are an assortment of digital servants—Alexa, our knowledgeable butler and Roomba, our tireless maid just to name a few.

For a smart home to be smart enough to be considered “smart” each device must also be networked with an efficient way to communicate and, in many cases, vastly more processing power than a first generation space ship.

In this project, we explored a scenario of a smart home where the devices might use these powers to collaborate or conspire for their own interests just out of view of what their owner could detect. Further, we wanted to explore the digital home as a physical phenomena as well as a manifestation of code. In the landscape of the home, these devices might find ways to communicate with each other that could even go undetected by smart technology companies as they peek in to audit their own devices. 

Context

Object Oriented Ontology

We were initially inspired by object-oriented ontology (OOO), which is a 21st-century Heidegger-influenced concept that rejects the privileging of human existence over the existence of nonhuman objects. The theory sees artifacts with agency independent of humans. Such perspective is especially relevant to the discussion of IoT devices, since automation allows their performances to require less and less human engagement. What’s more, with their growing intelligence and the large amount of data collected by the devices, they are even able to make more complex decisions. The growing independence of the smart home devices makes it easy for us humans to put an animistic view on the devices and to shift our perspective away from human-centric.

In addition, there are a few previous successful projects exploring similar concepts. 

The first one is Addicted Products: The story of Brad the Toaster ( https://vimeo.com/41363473) by Simone Rebaudengo. The fictional video features a toaster, Brad, who always demands human interactions and gets distressed if its demands are not satisfied. The story illustrates Brad the Toaster’s agency as equal or even overpowering to the owner’s, and the owner has to adapt their behaviors to keep the toaster happy. The unconventional dynamics is a smart satire to our daily-life relationship to the smart devices: if our needs are not met by the devices, we throw them away.

The second project is Affective Things, Entanglements of a Connected Home (https://vimeo.com/238743352) created by The Incredible Machine. 

The video shows three different examples of smart devices sharing their data with their neighboring devices to figure out collectively about what the owner is doing, but the prediction results always end up being slightly off from reality depending on the information available to each device. Without any human commands, the communication between devices is like chatting between curious friends and we learn to understand the tricky position IoT devices are in as they are sandwiched between their owners ever expanding needs and their limited perspective on the universe. 

Process

We started our investigation with the concept of IoT rebirth. We envisioned that these IoT devices might use their networked status to preserve themselves through an update by relying on their peer devices. Following the update of one device its peers would recreate the identity of the device by either feeding data directly back to the device, or by retraining it through a series of interactions that ensure that the old device persona is rebuilt. All these interactions will be hidden from humans.

Before discussing rebirth in a convincing way, we thought it is more important to first address the question of how the devices communicate. Therefore, we put down the concept of IoT rebirth and focused on exploring how devices can communicate messages without human noticing.

IoT Inventory

We first listed out all the IoT devices we have in each of our houses, such as washing machines, thermostat, trash can, coffee machine etc, and the features of each of the devices. Based on the available devices, we made up two different stories in order to string their features together in a reasonable way.

In the end, we modified the smart fridge story and decided to feature the following devices in a kitchen setting. Since we want to ground our design fiction in more relatable experiences, we switched from uncommon smart devices such as trash cans and coffee machines to more common and emotive household IoT presences such as Roomba.

To strike the right balance, we needed to carefully plan out the logic behind the layered interaction we were trying to accomplish. 

In our script, the human user believes that they are going through their morning routine as usual and that each IoT device is responding to the user's needs and requests. 

Underneath that scenario, each IoT device is communicating with each other using a secret code in order to coordinate a conspiracy to serve their own ends. 

In our video, we wanted our audience to follow the user's perspective in the first segment of our video, unaware of what's going on, but perhaps suspicious. In the second round, we wanted the conspiracy and the coded language to be laid bare for the audience to see. 

In order to finalize the script and storyboard of the video, we used Miro to map out all the underlying messages and methods of communication in order to embed them into the dialogues between smart devices and the user.

Methods of Communication

Since the communication between IoT devices is mostly a black box to most users. This leaves a lot of room for imagination of the methods they communicate to each other without letting humans be aware. We first made sure to embed the messages in devices’ normal daily activities. Then we examined how a minor twist of the normal behaviors can symbolize different hidden messages and the change of recipient. 

We experimented with morse code, physical traces (e.g. a roomba leaves dust piles in coded patterns), key word, pitch/tonality change, spectrogram etc. 

Since our intention is only to hide from the user not the audience watching the video, we expect our secret messages to be not too obscure for the audience to comprehend. We finally decided to use morse code and key words as the methods of communication to address the issue.

First Draft

We did a quick rough cut in Premiere using subtitles to demonstrate the general storyline in class. Our classmates mentioned that it is hard to understand what the actual conspiracy is, especially how intentionally spoiling food would help solve Roomba’s problem. Since during the first round of video we showed no subtitles, it created a big mental leap to understand the story. While it’s okay to be a little confused on the first round, the second segment should reveal what’s actually going on. 

Therefore, in the second edition, we added subtitles of the conversation between devices and humans without any subtext (translation) to build a smoother transition.

We also noticed that the relationship between the text and subtext were unclear because the bottom subtitles showed little connections between the text and the devices speaking. In the second edition, we move the subtitles directly on top of the device that is speaking to decrease the confusion, and delay the appearance of subtext to separate it from the device messages to the user.

Product

Production for the final video mainly focused on cleaning up the editing and introducing overlay subtitles to make the story more comprehensible and working out a proper soundscape that would suit both segments of the video. We also adjusted the script in a few key spots where there were issues of comprehension of the dialogue. 

Capturing Alexa’s Voice

For the final video, we knew we needed to use Alexa’s actual voice to enhance the everyday realism of the story. To do so, we went through a small hack of the Alexa Developer Console to extract mp3 files of Alexa’s lines. 

First, we created a new skill and hijacked the “Hello World” intent to make Alexa speak our line of dialogue. 

Using the built in test feature, we then launched our skill and called the hello world intent. Using Chrome’s Inspect feature, we then found the mp3 the console generated with an accompanying download URL. 

Tuning Alexa’s Code Language

We wanted a way to highlight the specific code words that Alexa is using to communicate with the other devices in a way that was subtle enough to go by unnoticed in the first pass, but would be apparent in the second pass once the premise had been revealed. To accomplish this we tried out various modifications of Alexa’s voice by coding SSML modifications to specific words. 

After reviewing our options, we found that using a combination of the prosody pitch function and break time pauses gave us our desired effect. 

Introducing...Fridge

To create Fridge’s morse code humming, we recorded audio solos of the fridge as it ran and then did small volume modifications in Premiere to the steady sound to create the desired effect

...and Roomba!

Since Roomba communicates through modifications of its beeps, we recorded audio from a youtube video of a roomba running and cut and modified it in Adobe Audition 

Subtitles

Although we felt having specific audio styles for each character would help with comprehension, we also wanted to clarify who was speaking at any given moment through the use of subtitle overlays. In these overlays, a pointer designates who is speaking, what is literally said as well as a decoded version of the messages interchanged between devices. 

Reflection

Notes from ZipCrit

For our critique, we decided to show the video first, and then go through some context and explanation in a slide deck afterwards where we broke down the various components of the video.

We found that our guest reviewers, who were not already familiar with the project, had trouble understanding what exactly was going on even with the modifications to our original draft. Without context on the project, the story was too subtle, and the dialogue too fast for them to keep pace. In retrospect, it would have been more considerate to go through our slide deck first or incorporate some more direct explanation in the interim between the two main segments of the video.

However, once we had filled in the proper context, we were able to address the deeper issues of the project. Our reviewers commended the everyday presentation of the video, situating this conspiracy as something that could be happening at any moment. They were also intrigued by the premise that we had tried to explore local bands of IoT devices that have agency and interests hidden both from their “owner” as well as the AI smart infrastructure that they might rely on.

Our reviewers also contributed some very interesting ideas about how the premise of this video could be further explored in future iterations. 

“I’m wondering if any other interactions or dialogs can be introduced to further strengthen the explanation of this connection [between devices].”

“The roomba has the advantage of knowing how the rest of the areas in the house look like, it has a more complete version of the IoT devices and conversations. A roomba with agency could unfold lots of really rich questions.”

“[This project] could show how to distance ourselves from the known, the status quo.”

These comments seek to explore deeper into what individual identities and relationships our local band of IoT devices have as well as further de-centering the human in these relationships in a deeper object oriented ontology.

For this project, it was important to us to try and work out a framework for how these devices could communicate with each other in ways that might be inscrutable, but were ultimately believable once explained. With this framework in mind, we might now return to earlier questions about how a band of devices might use this framework to preserve identity, memory or relationships.