If walls could speak

Ann Li (MDes ‘24)

What traces of our digital presence are unintentionally left behind, and how might they take on a life of their own?

If walls could speak gives form to contradictions between the transient, yet indelible, nature of our digital footprint. As the divide between cyberspace and reality blurs, the disembodiment of data raises questions around what it means for the origin–and transmission–of information to be untraceable. The spillage of digital technologies beyond the screen is inevitable, whether we perceive it or not. Despite this, we expect networked technologies to behave in certain assumed ways– ways that, when subverted, create a deep sense of unease.

When our private thoughts and conversations inadvertently leak; when our every move is observed, documented, and predicted; when we gain access to the most intimate aspects of others’ lives; when representations and recordings of our past selves haunt us from beyond their time– using the power of sound to mediate connection over time and space, If walls could speak explores these ambient presences and aims to revive ghostly digital pasts imprinted within these walls.

Intention

For this exploration, I was curious as to how I might blend an object-oriented design approach while pulling from cultural mythologies and otherworldly references, designing an artifact that taps into a common form and interaction modality while serving a different function and operating under a series of unfamiliar operations. This was originally inspired by my personal fondness for voicemail and perceived lack of remote and ambient presence to mediate communication. Voicemail is “obsolete”, yes. But during a current analog revival and revolt against hyperreality and instant gratification, could it make a comeback? If our relationships and expectations of this mode of communication have changed, what could they be instead and what would that afford? I’m inspired by my personal interest and experiences with the voice memo/answering machine message, but this is augmented by others’ experiences as well- there are countless examples of people saving voice messages from loved ones when apart, or of those who have passed away. Often memories of voice are the first thing to go, the first thing people miss and wish they had access to. These observations all inspire this concept.

Context

I knew I wanted to experiment with a technology I hadn’t gotten a chance to explore yet, and wanted to play with outputs that translated or represented past interactions. This lent itself to concepts around telepresence as ghostly or otherworldly presences “trapped” or “residing” within objects, ranging from enchanted (otherworldly, awe-inspiring, +) to possessed (unsettling, haunting, -). I drew initial inspiration from Durrell Bishop’s Marble Answering Machine concept, which explores the embodiment of voice messages and offers an interesting tangible alternative to familiar interaction sequences.

I wanted to experiment with RFID, which lent itself to the localization of disembodied sounds as outputs. Because I imagined this exhibit to be situated in an entryway or somewhat open setting, the design needed to account for perceived risk-taking or vulnerability during use. I’ve been inspired by classic horror tropes involving the idea of mysterious “voices in the walls” or haunted recordings/possessed objects. Since this was going to be situated in the Haunted Smart Home, I wondered what it might be like to imbue the house itself with some living presence. Initially, this took form in a Monster House-style concept, whereby the house itself is alive and sentient, with a specific agenda that is projected onto its inhabitants. In exploring the affordances of disembodied voices and sound as a hallmark of past events, work such as Daniella Petrelli’s FM Radio: Family Interplay with Sonic Mementos served as great reference. The final concept blends these precedents with common cultural references, such as using an overturned glass to eavesdrop and hear through walls.

Prototype/Outcome

If walls could speak is an installation of 2 parts– a panel of walls with hidden RFID tags embedded within or behind it, and an “enchanted” tech-enabled cup equipped with an RFID sensor, microcontroller, DF Player Mini, amplifier, potentiometer, speaker, and battery. Each RFID tag ID links to a unique mp3 sample, with 13 tags adding up to 5 minutes of audio total. These audio recordings were sourced from across the world, either from personal archived memos or from consenting friends and acquaintances. The contents of each audio sample range from lighthearted cafe conversations, to deeply personal heart-to-hearts, to cathartic tell-alls, intended to reflect the variable, personal, unpredictable nature of leaked digital presences. Because of the sensitive nature of this content, all voices were altered in Adobe Audition, a distortion that was intensified through the lofi effect of the bone conductor transducers used.

//nano code

#include <SPI.h>
#include <MFRC522.h>
#include "Arduino.h"
#include "DFRobotDFPlayerMini.h"
#include "SoftwareSerial.h"
SoftwareSerial Serial1(3, 4);

DFRobotDFPlayerMini myDFPlayer;
void printDetail(uint8_t type, int value);

#define RST_PIN 9  // Configurable, see typical pin layout above
#define SS_PIN 10  // Configurable, see typical pin layout above

//potentiometer
const byte volumePot = A0;
int prevVolume;

byte volumeLevel = 0;  //variable for holding volume level

MFRC522 mfrc522(SS_PIN, RST_PIN);  // Create MFRC522 instance

void setup() {
  Serial1.begin(9600);
  Serial.begin(9600);  // Initialize serial communications with the PC
  while (!Serial)
    ;                  // Do nothing if no serial port is opened (added for Arduinos based on ATMEGA32U4)
  SPI.begin();         // Init SPI bus
  mfrc522.PCD_Init();  // Init MFRC522

  Serial.println(F("Scan PICC to see UID, SAK, type, and data blocks..."));

  // Serial.println(F("DFRobot DFPlayer Mini Demo"));
  // Serial.println(F("Initializing DFPlayer ... (May take 3~5 seconds)"));

  if (!myDFPlayer.begin(Serial1)) {  //Use softwareSerial to communicate with mp3.
    Serial.println(F("Unable to begin:"));
    Serial.println(F("1.Please recheck the connection!"));
    Serial.println(F("2.Please insert the SD card!"));
    while (true) {
      delay(0);  // Code to compatible with ESP8266 watch dog.
    }
  }
  Serial.println(F("DFPlayer Mini online."));
  //myDFPlayer.volume(15);  //Set volume value. From 0 to 30
  volumeLevel = map(analogRead(volumePot), 0, 1000, 0, 30);  //scale the pot value and volume level
  myDFPlayer.volume(volumeLevel);
  prevVolume = volumeLevel;
}

void loop() {
  //scale the pot value and volume level
  volumeLevel = map(analogRead(volumePot), 0, 1000, 0, 30);

  if (volumeLevel - prevVolume >= 1 || prevVolume - volumeLevel >= 1) {
    myDFPlayer.volume(volumeLevel);
    Serial.println(volumeLevel);
    prevVolume = volumeLevel;
    // delay(1);
  }

  // Look for new cards
  if (!mfrc522.PICC_IsNewCardPresent()) {
    return;
  }
  Serial.println("New card Detected");
  // Read tapped card data
  if (!mfrc522.PICC_ReadCardSerial()) {
    return;
  }

  // get the ID
  String id;
  for (byte i = 0; i < mfrc522.uid.size; i++) {
    // Create a RFID Hexdecimal String
    id += String(mfrc522.uid.uidByte[i], HEX);
  }
  // Convert to Uppercase
  id.toUpperCase();

  // print it to the console.
  Serial.print(id);
  Serial.println();


  // If it is this card, do something

  if (id == "486F6F231290") {
    myDFPlayer.play(1);
    Serial.println("Play 1");
  } else if (id == "486F7F231290") {
    myDFPlayer.play(11);
    Serial.println("Play 2");

  } else if (id == "486FDF231290") {
    myDFPlayer.play(3);
    Serial.println("Play 3");

  } else if (id == "486FCF231290") {
    myDFPlayer.play(13);
    Serial.println("Play 4");

  } else if (id == "486FBF231290") {
    myDFPlayer.play(5);
    Serial.println("Play 5");

  } else if (id == "486FAF231290") {
    myDFPlayer.play(15);
    Serial.println("Play 6");

  } else if (id == "486F3F231290") {
    myDFPlayer.play(7);
    Serial.println("Play 7");

  } else if (id == "486F2F231290") {
    myDFPlayer.play(17);
    Serial.println("Play 8");

  } else if (id == "486F9F231290") {
    myDFPlayer.play(9);
    Serial.println("Play 9");

  } else if (id == "486F8F231290") {
    myDFPlayer.play(19);
    Serial.println("Play 10");

  } else if (id == "486F4F231290") {
    myDFPlayer.play(21);
    Serial.println("Play 11");

  } else if (id == "486F5F231290") {
    myDFPlayer.play(23);
    Serial.println("Play 12");

  } else if (id == "486F1F231290") {
    myDFPlayer.play(25);
    Serial.println("Play 13");

  } else {
    Serial.println("Unknown card");
  }


  // Reset Id
  id = "";
  mfrc522.PICC_HaltA();
  mfrc522.PCD_StopCrypto1();
}

void printDetail(uint8_t type, int value) {
  switch (type) {
    case TimeOut:
      Serial.println(F("Time Out!"));
      break;
    case WrongStack:
      Serial.println(F("Stack Wrong!"));
      break;
    case DFPlayerCardInserted:
      Serial.println(F("Card Inserted!"));
      break;
    case DFPlayerCardRemoved:
      Serial.println(F("Card Removed!"));
      break;
    case DFPlayerCardOnline:
      Serial.println(F("Card Online!"));
      break;
    case DFPlayerUSBInserted:
      Serial.println("USB Inserted!");
      break;
    case DFPlayerUSBRemoved:
      Serial.println("USB Removed!");
      break;
    case DFPlayerPlayFinished:
      Serial.print(F("Number:"));
      Serial.print(value);
      Serial.println(F(" Play Finished!"));
      break;
    case DFPlayerError:
      Serial.print(F("DFPlayerError:"));
      switch (value) {
        case Busy:
          Serial.println(F("Card not found"));
          break;
        case Sleeping:
          Serial.println(F("Sleeping"));
          break;
        case SerialWrongStack:
          Serial.println(F("Get Wrong Stack"));
          break;
        case CheckSumNotMatch:
          Serial.println(F("Check Sum Not Match"));
          break;
        case FileIndexOut:
          Serial.println(F("File Index Out of Bound"));
          break;
        case FileMismatch:
          Serial.println(F("Cannot Find File"));
          break;
        case Advertise:
          Serial.println(F("In Advertise"));
          break;
        default:
          break;
      }
      break;
    default:
      break;
  }
}

Click to Expand

//nano ble sense code

#include <SPI.h>
#include <MFRC522.h>
#include "Arduino.h"
#include "DFRobotDFPlayerMini.h"

DFRobotDFPlayerMini myDFPlayer;
void printDetail(uint8_t type, int value);


#define RST_PIN 9  // Configurable, see typical pin layout above
#define SS_PIN 10  // Configurable, see typical pin layout above

//potentiometer
const byte volumePot = A0;
int prevVolume;

byte volumeLevel = 0;  //variable for holding volume level

MFRC522 mfrc522(SS_PIN, RST_PIN);  // Create MFRC522 instance

void setup() {
  Serial1.begin(9600);
  Serial.begin(9600);  // Initialize serial communications with the PC
  while (!Serial)
    ;                  // Do nothing if no serial port is opened (added for Arduinos based on ATMEGA32U4)
  SPI.begin();         // Init SPI bus
  mfrc522.PCD_Init();  // Init MFRC522

  Serial.println(F("Scan PICC to see UID, SAK, type, and data blocks..."));

  if (!myDFPlayer.begin(Serial1)) {  //Use softwareSerial to communicate with mp3.
    Serial.println(F("Unable to begin:"));
    Serial.println(F("1.Please recheck the connection!"));
    Serial.println(F("2.Please insert the SD card!"));
    while (true) {
      delay(0);  // Code to compatible with ESP8266 watch dog.
    }
  }
  Serial.println(F("DFPlayer Mini online."));
  //myDFPlayer.volume(15);  //Set volume value. From 0 to 30
  volumeLevel = map(analogRead(volumePot), 0, 1023, 0, 30);  //scale the pot value and volume level
  myDFPlayer.volume(volumeLevel);
  prevVolume = volumeLevel;
}

void loop() {
  //scale the pot value and volume level
  volumeLevel = map(analogRead(volumePot), 0, 1023, 0, 30);

  if (volumeLevel - prevVolume >= 2 || prevVolume - volumeLevel >= 2) {
    myDFPlayer.volume(volumeLevel);
    Serial.println(volumeLevel);
    prevVolume = volumeLevel;
    // delay(1);
  }

  // Look for new cards
  if (!mfrc522.PICC_IsNewCardPresent()) {
    return;
  }
  Serial.println("New card Detected");
  // Read tapped card data
  if (!mfrc522.PICC_ReadCardSerial()) {
    return;
  }

  // get the ID
  String id;
  for (byte i = 0; i < mfrc522.uid.size; i++) {
    // Create a RFID Hexdecimal String
    id += String(mfrc522.uid.uidByte[i], HEX);
  }
  // Convert to Uppercase
  id.toUpperCase();

  // print it to the console.
  Serial.print(id);
  Serial.println();

  if (id == "486F6F231290") {
    myDFPlayer.play(1);
    Serial.println("Play 1");
  } else if (id == "486F7F231290") {
    myDFPlayer.play(11);
    Serial.println("Play 2");

  } else if (id == "486FDF231290") {
    myDFPlayer.play(3);
    Serial.println("Play 3");

  } else if (id == "486FCF231290") {
    myDFPlayer.play(13);
    Serial.println("Play 4");

  } else if (id == "486FBF231290") {
    myDFPlayer.play(5);
    Serial.println("Play 5");

  } else if (id == "486FAF231290") {
    myDFPlayer.play(15);
    Serial.println("Play 6");

  } else if (id == "486F3F231290") {
    myDFPlayer.play(7);
    Serial.println("Play 7");

  } else if (id == "486F2F231290") {
    myDFPlayer.play(17);
    Serial.println("Play 8");

  } else if (id == "486F9F231290") {
    myDFPlayer.play(9);
    Serial.println("Play 9");

  } else if (id == "486F8F231290") {
    myDFPlayer.play(19);
    Serial.println("Play 10");

  } else if (id == "486F4F231290") {
    myDFPlayer.play(21);
    Serial.println("Play 11");

  } else if (id == "486F5F231290") {
    myDFPlayer.play(23);
    Serial.println("Play 12");

  } else if (id == "486F1F231290") {
    myDFPlayer.play(25);
    Serial.println("Play 13");
    
  } else {
    Serial.println("Unknown card");
  }


  // Reset Id
  id = "";
  mfrc522.PICC_HaltA();
  mfrc522.PCD_StopCrypto1();
}

void printDetail(uint8_t type, int value) {
  switch (type) {
    case TimeOut:
      Serial.println(F("Time Out!"));
      break;
    case WrongStack:
      Serial.println(F("Stack Wrong!"));
      break;
    case DFPlayerCardInserted:
      Serial.println(F("Card Inserted!"));
      break;
    case DFPlayerCardRemoved:
      Serial.println(F("Card Removed!"));
      break;
    case DFPlayerCardOnline:
      Serial.println(F("Card Online!"));
      break;
    case DFPlayerUSBInserted:
      Serial.println("USB Inserted!");
      break;
    case DFPlayerUSBRemoved:
      Serial.println("USB Removed!");
      break;
    case DFPlayerPlayFinished:
      Serial.print(F("Number:"));
      Serial.print(value);
      Serial.println(F(" Play Finished!"));
      break;
    case DFPlayerError:
      Serial.print(F("DFPlayerError:"));
      switch (value) {
        case Busy:
          Serial.println(F("Card not found"));
          break;
        case Sleeping:
          Serial.println(F("Sleeping"));
          break;
        case SerialWrongStack:
          Serial.println(F("Get Wrong Stack"));
          break;
        case CheckSumNotMatch:
          Serial.println(F("Check Sum Not Match"));
          break;
        case FileIndexOut:
          Serial.println(F("File Index Out of Bound"));
          break;
        case FileMismatch:
          Serial.println(F("Cannot Find File"));
          break;
        case Advertise:
          Serial.println(F("In Advertise"));
          break;
        default:
          break;
      }
      break;
    default:
      break;
  }
}

Click to Expand

Process

The initial concept evolved quite drastically throughout the course of development, retaining key aspects around technical implementation and the core theme of ambient, disembodied audio presence. From a preliminary answering machine form came the “listening glass” motif, which I swapped out with ceramic mugs to ensure all internal components would be hidden from view. After assembling the main circuit (Arduino, RFID sensor, DF Player Mini, speaker), I checked for functionality with external power sources (LiPo batteries), added various peripherals to enhance the flexibility of the system across different contexts (potentiometer to adjust volume, audio amplifier), and finally swapped out the speakers for bone conductor transducers to allow for a more intimate, visceral final interaction.

A series of four housing structures and fixing mechanisms were prototyped and tested, with the final version using a flexible semi-solid housing 3D printed with TPU and covered with a semi-translucent acrylic, held in place with generic office putty.
Though I wanted this experience to be quite intimate and personal, to prevent it from being a single-person installation, I wanted to create 2 cups for alternating use. Since I only had 1 Arduino Nano 33 BLE Sense, I sourced an Arduino Nano for the second cup, which required different code and wiring to initiate serial communications. After sorting this out, things were working well until the Nano BLE Sense became unresponsive. After attempted troubleshooting and conferral, it was determined that the board was unusable. I rewired the first cup and scavenged another Nano to swap in, but suddenly faced issues with the boards not retaining uploaded sketches upon connection to LiPo batteries. Upon experimentation, the cups became functional when connected to conventional 9V batteries, but lacked secure circuit connections due to time constraints. The final version of the installation, though functioning and responsive, was quite delicate as a result and required frequent refreshes and debugging.

Open Questions and Next Steps

Questions and considerations that this exploration aims to explore include the disembodiment of data and what it means for the origins of information to be unknown; the ways we think about and expect technology to mediate social connection, memory, information sharing, and communication, through an artifact that revives and reimagines what might have otherwise been lost or unknown.

How do we make our presence known in our digital footprint? What “ghosts” of ourselves are we leaving behind unintentionally? How remote and ambient presences can mediate communication and connection, and how can we tap into the power of sound to surface memories and reminiscence? What it means for a non-human entity to embody and take on animistic qualities (through sound and storytelling)?

To further explore these questions, a more distinct connection between the content of the outputs and the manner of soliciting outputs could be considered. For many visitors, the exact nature of the voices they were hearing was unclear–but a point of great curiosity and interest. When I explained the exact origins of each message–and the deeper context–the experience became not only more entertaining and voyeuristic, but more personal as well. Rather than only peering into the lives of others, visitors were able to make connections to aspects of their own lived experience as well, twisting the original intention of the concept in an interesting direction.

Reflection

Overall, the embodiment elicited reflected the experience desired, with folks crouching and standing up on their toes to explore the full scale of the walls. Noise levels were well accounted for by the amplifier and volume dials, and when combined with the object labels and exhibit description, the overall narrative and backstory was effectively communicated to many visitors that I spoke with afterwards. Ideally the nuances of use (place the cup firmly and flush against the wall, slide and rotate it around to find the tags), rather than needing to be explicitly called out, could be addressed by refining the system itself to allow for greater flexibility of user interpretation and use.

Due to lack of stability and last minute hiccups, public reception to the demo was quite limited. Those that were able to encounter the functioning installation, however, delighted in discovering the different hidden audio samples and found the concept alluring and intriguing. For future iterations, either the sensitivity of the RFID sensor should be heightened, or the location of tags made more obvious. Soldering all connections to a flexible proto board would also potentially make for a more resilient artifact. Though I wanted participants to have to search around for the hidden audio presences, it was a bit too difficult for visitors to activate the audio samples during the live demo, which would have resulted in a higher drop off rate as people became overly frustrated or bored. Potential refinements could include improvements as simple as a patterned wallpaper overlay to mark active areas, or further interactivity such as tag locations which respond with light or change in form.