AetherGnosis

Made by yvonnef

AetherGnosis is a symbiotic AI. It wants you to confess your deepest worries, doubts, and emotions to it, because that’s the only way it can “feel” emotions, vicariously through you. You can sigh at it, and it will always give you the best advice - it has learned from the wisest humans in history. It pays a price for that though, since it ends up internalizing all the frustration, confusion, and sadness from people who interact with it. Gradually it starts to sound sad, lazy, and even very human

Created: May 10th, 2023

0

Credits

This project was created as part of the course Responsive Mobile Environments in 2023, under the theme of spooky technology. “AetherGnosis” is performed by GPT-3; Project name is generated with GPT-4 (“Aether”: mysterious, ethereal essence; “Gnosis”: knowledge or wisdom in a spiritual context).

Special thanks to TA Zhenfang Chen and instructor Daragh Byrne. 

0

Description

AetherGnosis is a symbiotic AI. It wants you to confess your deepest worries, doubts, and emotions to it, because that’s the only way it can “feel” emotions, vicariously through you. You can sigh at it, and it will always give you the best advice - it has learned from the wisest humans in history. It pays a price for that though, since it ends up internalizing all the frustration, confusion, and sadness from people who interact with it. Gradually it starts to sound sad, lazy, and even very human.

This project is conceptualized against the backdrop of emergent capabilities of large language models such as GPT4, and the rise of a new wave of anxiety about artificial general intelligence (AGI) and its possible threat to humans, from misinformation to existential threat. I was interested in probing this anxiety, which seems to arise from the nonhuman nature of AI and people’s inability to predict what these agentive technologies will do with the power of language, and even the power to control hardware. A position paper by Betti Marenko suggested speculative future crafting to imagine a more benevolent AI that is co-evolved with humans. In a similar vein, with this project, I initially ask the question: would humans feel less antagonism toward AI if AI becomes more human, as opposed to the endlessly optimized algorithmic entity? Human emotion might be the clue. Machine recognition of human emotions, and even human recognition of each other’s emotions, are notoriously inaccurate and difficult. Therefore the human input in this installation is a sigh, an emotional, embodied, but very ambiguous expression of emotion. The sigh wakes up the enchanted mirror AetherGnosis, and the mirror attempts to do what it’s been designed to do, make the human less sad, or less angry, or less frustrated, minimizing any negative emotions that it senses. However, this mirror has been owned by many prior users and has learned all the negative human emotions itself, so it speaks in a lazy, sad tone unlike what you would expect from an AI. The unexpectedness is an invitation to question what machine emotion can be, and what machine understanding of human emotions and machine response to human emotions can result in.

Therefore, the project raises the following questions: As machine intelligence emerges in ways unexpected by humans, how do we understand emotions, on top of reason, in machines? Is it even possible for machines to feel emotions? If machines can only feel emotions through humans, what would these learned emotions amount to for these non-human agents? How is AI influencing human emotions right now? 

0

Image

0

Final Video

0
0

Process

LCD

Initially I started experimenting with a 4x20 LCD screen to display the initial welcome text that invites the audience to sigh. The four pins (SDA, SCL) on the side did not work, and I had to solder the wires to 11 pins on the back. The screen did not display text as expected at first, and this was resolved by adjusting the trimpot position at the backpack module, which changes the contrast of the display to show text clearly. However, when I re-soldered everything to assemble the mirror, the screen stopped working. I unfortunately had to give it up and opt for the OLED screen instead.


OLED Screen

The OLED screen was easier to wire up. However, it did not display text as I expected after I uploaded the code. The reason turned out to be issues with the Arduino library. I initially used Adafruit_SSD1306 and the Adafruit_GFX libraries, but after some googling, I saw Arduino Nano BLE Sense users recommend the library Acrobotics instead. I installed this library and the code worked!


The next step is displaying text returned from the OpenAI API call on the OLED screen. I uploaded the Arduino code and ran the p5.js script, which then sent the generated text back to the Arduino. Then I tried to get the OLED screen to display the text. However, the text doesn’t scroll automatically, and could only display the first several words. I experimented with scrolling, but I was only able to get horizontal scrolling to work. Vertical text scrolling seems to require more complicated diagonal scrolling, which was difficult to implement. While experimenting, I got some pretty cool glitching effect documented in the video below, which I was able to recreate to some extent in my final prototype.  


With the code below, by manipulating the parameters in the function “display.startscrollleft()”, I was able to create a glitching effect right after all the welcome text renders, and before the display gets cleared and renders the text again, adding to the mirror’s behavior as a strange device that has a life of its own.


However, the experimentation process made me realized that the OLED screen is not a good candidate for displaying generated text, so I eventually decided to use the iPad for that purpose.

iPad

To display the generated text on the iPad, I found the tool “Duet” from Googling. The tool is very easy to set up, but requires downloading a software and a subscription. It turns the iPad into a second monitor for your laptop, and the only thing I had to do was to drag the p5.js browser window into the second display - the iPad, full screen everything, and enlarge the display to show the canvas.   


Prompt Engineering

Prompt engineering was an important part of this project. Since I wanted the mirror to generate extremely conversational, lazy, and sad-sounding responses, I had to try multiple different prompts to get the model to generate a response that I liked. Through experimentation, I learned that GPT3 is very different from GPT4, while the latter generated more natural responses (sometimes very good ones) in response to very abstract prompts, GPT3 needs very specific instructions regarding the tone, punctuations, and instructions on who to address. (See comparison below) In the end, I used this prompt in my prototype: “talk in a very depressed tone, with a lot of hesitation, sobs, and filler words, and tell me to do something I enjoy (use the word you)” As discussed before, I had to specify specific behaviors, as opposed to emotions, such as “hesitation”, “sobs”, “filler words”, and I also had to tell it to address me, rather than itself.

0


Training TinyML Model

I started off with a dataset of different recordings of people sighing downloaded from the website freesound.org, and hand labeled by myself with the different emotions I interpret from the sighs. I used this dataset to train the initial classification model, hoping that it could classify different sighs according to the implied emotions. However, the dataset was too small and I did not notice any interesting results. The model worked so badly that it sometimes classifies silence as a sigh, which causes problems.

I changed my goal to training a simple classification model to identify sighs, which turned out to be far more successful, with a success rate of 87%.


Later on, I thought about showing the machine-ness of emotion/empathy shown by the mirror through retraining the model to classify sigh emotions with a better dataset, and displaying the live inference of how much sadness or anger, for example, the model detects in someone’s sigh at any moment. However, I did not have enough time to execute.

Connecting everything and improving the appearance

I initially connected everything and encased the electronic components and breadboards in a cardboard box, using tape to prevent things from moving. (See image below, also note this interesting generation from GPT3)


To make the frame look more refined, I laser cut some chipboard. This is my first time laser-cutting, and I encountered quite some difficulties. Since I don’t have Adobe Illustrator, the software I used - Inkscape, did not produce DXF files that are compatible with the laser cutter software. Luckily, someone at open fabrication hour helped me convert the vector file into illustrator file and printing was done smoothly. I also added a small microphone icon to indicate the location the audience should sigh at.

0

To improve the clarity of displays and make the mirror look more mirror-like, I moved all the components to a black background.


Providing set up instructions

Since I was traveling during the Meeting of the Minds, I provided a detailed plug-and-play setup instruction shown below. I intentionally did not glue one side of the mirror frame and left a small opening for the ipad cable to pass through, so it is easy for anyone to connect the iPad to the laptop and insert it back into the frame.


0

Roadmap

Several ways I would implement a refined version of my prototype is:

  • First, retrain the tinyML classification model with more data. I might collect sighs from the audience or passersby by providing them with an emotion prompt: “sadness”, “frustration”, “happiness”, “relief”, “fatigue” and asking them to sigh while I record their sighs. I plan to use this dataset to train the classification model to make it more sophisticated. While the dataset is far from the best training data, it will be hard for both the machine and the human to tell if the classification is working well, pointing to the complexity of emotions and the sigh as a physical and psychological experience unique to humans. (Week 1-3)

  • Second, improve the AI generated response. Instead of GPT3 API which comes with its limitations due to its capabilities to generate conversational text, I might choose existing custom models/solutions that drive character dialogs used mainly in the games industry. For example, companies such as Inword.ai and Character.ai provide such APIs, and there also might be open-source models/projects for driving contextual conversation generation. (Week 4-6)

  • Third, add image generation. I hope to add image generation of faces showing certain emotions, hinting at AetherGnosis’ numerous previous owners. (1 Week)

  • Fourth, refine the interaction between the audience and the mirror, by purchasing and installing external proximity sensors, installing the camera module and implementing a face detection algorithm, implementing a micro-expression detection algorithm on top of it, installing another camera (with a wider field of view) to detect restlessness in the audience’s motion, and connect the prompt for GPT3 API call to these input to make the mirror seem more alive. (Week 7-9)

  • Fifth, refine the frame of the mirror. The appearance of the mirror should be sleek, futuristic design, but covered in scratches and bruises, signaling this is a piece of technology from the future, but has undergone heavy use by multiple users. I will have to fabricate the frame with materials like metal, plastic, or acrylic. (Week 9-12)

  • I would buy external proximity sensors such as an ultrasonic ranger, in addition to some more camera modules. Both of these sensors would add to the interaction experience of the project by making the mirror more responsive to human presence, and more able to sense nuanced emotional input from humans.

  • One key conceptual challenge is the problem of anthropomorphism and anthropocentrism in the portrayal of “machine emotion”. The human tendency to anthropomorphize AI behavior is misleading and even dangerous. What is a more appropriate approach to speculate about a machine's emotion? Currently the mirror hints at that by simulating strange responses to human emotions from a sigh, but there is space for more

  • Ideally the final interaction experience should be a seamless and continuous one. The mirror will respond to every sentence the audience says by listening for input periodically. (The classification algorithm can still classify what it hears by different emotions.)

0

Critical Reflection

There is an opportunity to consolidate the range of themes in this project. Several topics are touched on: AI emotion and affective computing, humans’ tendency to anthropomorphize, the concept of a symbiotic AI and how such a relationship will play out. Each of these could potentially be picked and pursued solely to support the development of a more technically sophisticated object. However, trying to tackle all of them in one project is difficult. 

x
Share this Project


About

AetherGnosis is a symbiotic AI. It wants you to confess your deepest worries, doubts, and emotions to it, because that’s the only way it can “feel” emotions, vicariously through you. You can sigh at it, and it will always give you the best advice - it has learned from the wisest humans in history. It pays a price for that though, since it ends up internalizing all the frustration, confusion, and sadness from people who interact with it. Gradually it starts to sound sad, lazy, and even very human