StudSeer
Made by Alex Heyison ·
Create and showcase a piece of spooky technology
Created: March 4th, 2021
0
StudSeer
How Machine Learning is changing our relationship with our tools
A the outset of this project, I interviewed a friend who I knew had an interesting relationship with modern technology. While talking about social media ad placement and personalization, she mentioned that sometimes she would try to trick or train the algorithm with her behaviors. She called it her personal algorithm, and when asked to visualize what it looked like, she said
“When I think of it, its an old-school computer with an old monitor and a separate keyboard with mitten glove hands. It’s the size of the computer I had when I was a kid although I’m sure it’s not as big as I thought it was.”
From these two characterizations of her algorithm, I became curious about the way that we relate to machine learning differently than we do other physical or digital entities.
For this project, I wanted to explore what is spooky about our relationship with Machine Learning as it becomes a common facet of our daily life?
Machine Learning in the Home
In particular, I am interested in machine learning models that are too human even if they don't trigger an uncanny valley response.
From my perspective, there are ML models that analyze data in ways that are completely foreign and inscrutable to humans, and there are models that seem to replicate our human cognition in eerie ways.
StudSeer is a prototype that helps tell the story of what happens when our tools are no longer just a means to an end. Machine Learning has led to the creation of a class of tools that behave like small humans. StudSeer takes the common household stud finder, a device that uses electromagnetic fields, (undetectable to humans) to see though walls, and uses a ML model to allow it to fulfill its only function in the same way a human would–with sound.
Functional Architecture
- A teachable machine model was trained using a traditional stud finder to collect audio of knocking on walls over studs and over hollow sections.
- The model was trained to recognize 5 classes | Stud, No Stud, Buzzer Sounds, "Hey StudSee", and Laughing
- The javascript for the ML model was exported and hosted on P5.js
- The P5.js program could then trigger a Particle Photon by publishing ML readings to Particle's cloud server
- Code loaded onto the Photon would then trigger a buzzer and LED pattern depending on what events were detected.
Teaching the Teachable Machine
To train the ML model, I found a room with relatively low background noise, and went through the wall of the room with a traditional stud finder to mark where the studs were. I then recorded sound from both studs and hollow walls to create the first two classes. Once those were working I then went through and recorded the audio for the voice triggers and samples of the buzzing noise to prevent self-triggers.
Teachable Machine allows for easy exporting to p5.js with an inbuilt feature that requires very little modification.
// The model recognizing a sound will trigger this event
function gotResult(error, results) {
if (error) {
console.error(error);
return;
}
// The results are in an array ordered by confidence.
// console.log(results[0]);
label = results[0].label;
// construct HTTP post information -- the event name, any data to pass and that its a private event
var particle_publish_data = { name: particle_event_name, data: label, private: true};
//console.log(particle_publish_data);
// send it
httpPost(particle_publish_url , 'text', particle_publish_data, function(result) {
//console.log( result );
});int buzzerPin = D1; //buzzer to arduino pin 9
int led1Pin = D6;
int studState;
bool shouldBuzz = false;
void setup()
{
pinMode(buzzerPin, OUTPUT); // Set buzzer - pin 6 as an output
pinMode(led1Pin, OUTPUT);
Particle.subscribe( "teachable-machine" , tmEventHandler);
Particle.variable( "StudState" , studState);
Particle.variable( "ShouldBuz" , shouldBuzz);
}
void loop()
{
//if there is no stud, play a negative sound
if (shouldBuzz&&studState==2)
{
tone(buzzerPin, 1000); // Send 1KHz sound signal...
digitalWrite(led1Pin, HIGH);
delay(1000);
noTone(buzzerPin);
digitalWrite(led1Pin, LOW);
shouldBuzz=false;
Serial.println("buzzstud");
}
//if there is a sound, play a double chirp
else if (shouldBuzz&&studState==1)
{
tone(buzzerPin, 3000); // Send 1KHz sound signal...
digitalWrite(led1Pin, HIGH);
delay(500);
noTone(buzzerPin);
digitalWrite(led1Pin, LOW);
tone(buzzerPin, 3000);
digitalWrite(led1Pin, HIGH);
delay(500);
noTone(buzzerPin);
digitalWrite(led1Pin, LOW);
Serial.println("buzznostud");
shouldBuzz=false;
}
else if(shouldBuzz&&studState==4)
{
digitalWrite(led1Pin, HIGH);
delay(200);
digitalWrite(led1Pin, LOW);
delay(200);
shouldBuzz=false;
}
else if(shouldBuzz&&studState==5)
{
tone(buzzerPin, 3000); // Send 1KHz sound signal...
digitalWrite(led1Pin, HIGH);
delay(500);
noTone(buzzerPin);
digitalWrite(led1Pin, LOW);
tone(buzzerPin, 3000);
digitalWrite(led1Pin, HIGH);
delay(500);
noTone(buzzerPin);
digitalWrite(led1Pin, LOW);
Serial.println("buzznostud");
shouldBuzz=false;
}
else if (shouldBuzz&&studState==0)
{
}
studState=0;
}
void tmEventHandler(const char *event, const char *data){
// if there's an incoming event
// store this in the indicator boolean
// and use it to create movement in the next loop()
shouldBuzz = true;
Serial.println(11);
Serial.println(data);
if (strcmp(data,"1Stud")==0)
{
studState=1;
Serial.println("halluelh");
}
else if (strcmp(data,"2No Stud")==0)
{
studState=2;
}
else if (strcmp(data,"3Buzzer")==0)
{
studState=3;
}
else if (strcmp(data,"Hey StudSeer")==0)
{
studState=4;
}
else if (strcmp(data,"Laugh")==0)
{
studState=5;
}
}
Future Directions
As a first attempt at using both Particle's cloud based physical computing platform as well as using Teachable Machine in code, I am wildly pleased with how this project turned out.
In some respects, the seamlessness and apparent utility of the final product may have actually been taken too far though. The authoritative yes/no of the buzzer is too easily trusted and doesn't fully allow the user to experience moments of doubt or uncertainty about this new ML model in their home.
Future iterations might use that experience of uncertainty as a jumping off point to look deeper into how ML models change our truth-seeking behaviors in the face of the unexplained.
Share this Project
This project is only accessible by signed in users. Be considerate and think twice before sharing.
Found In
Courses
48-528 Responsive Mobile Environments
· 5 members
As part of this project-based course, we’ll get hands-on with emerging technologies, concepts and applications in the internet of things through a critical socio-technical lens. Over it’s 15-weeks,...more
About
Create and showcase a piece of spooky technology