StudSeer

Made by Alex Heyison ·

Create and showcase a piece of spooky technology

Created: March 4th, 2021

0

StudSeer

How Machine Learning is changing our relationship with our tools

A the outset of this project, I interviewed a friend who I knew had an interesting relationship with modern technology. While talking about social media ad placement and personalization, she mentioned that sometimes she would try to trick or train the algorithm with her behaviors. She called it her personal algorithm, and when asked to visualize what it looked like, she said

“When I think of it, its an old-school computer with an old monitor and a separate keyboard with mitten glove hands. It’s the size of the computer I had when I was a kid although I’m sure it’s not as big as I thought it was.”

From these two characterizations of her algorithm, I became curious about the way that we relate to machine learning differently than we do other physical or digital entities. 

For this project, I wanted to explore what is spooky about our relationship with Machine Learning as it becomes a common facet of our daily life?

0
0

Machine Learning in the Home

In particular, I am interested in machine learning models that are too human even if they don't trigger an uncanny valley response. 

From my perspective, there are ML models that analyze data in ways that are completely foreign and inscrutable to humans, and there are models that seem to replicate our human cognition in eerie ways. 

0

StudSeer is a prototype that helps tell the story of what happens when our tools are no longer just a means to an end. Machine Learning has led to the creation of a class of tools that behave like small humans. StudSeer takes the common household stud finder, a device that uses electromagnetic fields, (undetectable to humans) to see though walls, and uses a ML model to allow it to fulfill its only function in the same way a human would–with sound. 

0

Functional Architecture

  1. A teachable machine model was trained using a traditional stud finder to collect audio of knocking on walls over studs and over hollow sections.
  2. The model was trained to recognize 5 classes | Stud, No Stud, Buzzer Sounds, "Hey StudSee", and Laughing
  3. The javascript for the ML model was exported and hosted on P5.js
  4. The P5.js program could then trigger a Particle Photon by publishing ML readings to Particle's cloud server
  5. Code loaded onto the Photon would then trigger a buzzer and LED pattern depending on what events were detected. 
0
Early Prototype using a Particle Photon, sound sensor and LED to work out basic code and networking
20210302 125617.thumb
0

Training the Algorithm

The first prototype of StudSeer mostly focused on working out the basics of physical prototyping–light, sound and triggering them.  After working through the basic elements of the code, the focus then shifted towards the Machine Learning and Networking components. 

0

Teaching the Teachable Machine

To train the ML model, I found a room with relatively low background noise, and went through the wall of the room with a traditional stud finder to mark where the studs were. I then recorded sound from both studs and hollow walls to create the first two classes. Once those were working I then went through and recorded the audio for the voice triggers and samples of the buzzing noise to prevent self-triggers. 

Teachable Machine allows for easy exporting to p5.js with an inbuilt feature that requires very little modification. 

0

A small bit of added code allows p5.js running on a laptop to publish the readouts of our ML model as events in the Particle Cloud IDE. 

0
A short snippet of code added to the TM export to allow it to trigger a Particle cloud event
// The model recognizing a sound will trigger this event
function gotResult(error, results) {
  if (error) {
    console.error(error);
    return;
  }
  // The results are in an array ordered by confidence.
  // console.log(results[0]);
  label = results[0].label;
    // construct HTTP post information -- the event name, any data to pass and that its a private event
  var particle_publish_data = { name: particle_event_name, data: label, private: true};
  //console.log(particle_publish_data);
  // send it
  httpPost(particle_publish_url , 'text', particle_publish_data,  function(result) {
    //console.log( result );
  });
Click to Expand
0

Through the Cloud

The code below is loaded onto the Particle Photon which allows it to receive the published data from p5.js and trigger outputs through simple conditional statements in the loop function. 

0
int buzzerPin = D1; //buzzer to arduino pin 9
int led1Pin   = D6;
int studState;
bool shouldBuzz = false;


void setup()
{
 
  pinMode(buzzerPin, OUTPUT); // Set buzzer - pin 6 as an output
pinMode(led1Pin, OUTPUT);

Particle.subscribe( "teachable-machine" , tmEventHandler);
  Particle.variable( "StudState" , studState);
  Particle.variable( "ShouldBuz" , shouldBuzz);
    
}

void loop()
{
    
    //if there is no stud, play a negative sound
   if (shouldBuzz&&studState==2)
    {
        tone(buzzerPin, 1000); // Send 1KHz sound signal...
        digitalWrite(led1Pin, HIGH);
        delay(1000);
        noTone(buzzerPin);
        digitalWrite(led1Pin, LOW);
        shouldBuzz=false;
        Serial.println("buzzstud");
    }
    //if there is a sound, play a double chirp
   else if (shouldBuzz&&studState==1)
    {
        tone(buzzerPin, 3000); // Send 1KHz sound signal...
        digitalWrite(led1Pin, HIGH);
        delay(500);
        noTone(buzzerPin);
        digitalWrite(led1Pin, LOW);
        tone(buzzerPin, 3000);
        digitalWrite(led1Pin, HIGH);
        delay(500);
        noTone(buzzerPin);
        digitalWrite(led1Pin, LOW);
         Serial.println("buzznostud");
        shouldBuzz=false;
    }
    else if(shouldBuzz&&studState==4)
    {
        digitalWrite(led1Pin, HIGH);
        delay(200);
        digitalWrite(led1Pin, LOW);
        delay(200);
        shouldBuzz=false;
    }
    else if(shouldBuzz&&studState==5)
    {
       tone(buzzerPin, 3000); // Send 1KHz sound signal...
        digitalWrite(led1Pin, HIGH);
        delay(500);
        noTone(buzzerPin);
        digitalWrite(led1Pin, LOW);
        tone(buzzerPin, 3000);
        digitalWrite(led1Pin, HIGH);
        delay(500);
        noTone(buzzerPin);
        digitalWrite(led1Pin, LOW);
         Serial.println("buzznostud");
        shouldBuzz=false;
    }
    
    else if (shouldBuzz&&studState==0)
    {
         
    }
    
   studState=0;
  

}

void tmEventHandler(const char *event, const char *data){
    // if there's an incoming event 
    // store this in the indicator boolean
    // and use it to create movement in the next loop()
    shouldBuzz = true;
    Serial.println(11);
    Serial.println(data);
    if (strcmp(data,"1Stud")==0)
    {
        studState=1;
        Serial.println("halluelh");
    }
    else if (strcmp(data,"2No Stud")==0)
    {
        studState=2;
    }
    else if (strcmp(data,"3Buzzer")==0)
    {
        studState=3;
    }
    else if (strcmp(data,"Hey StudSeer")==0)
    {
        studState=4;
    }
    else if (strcmp(data,"Laugh")==0)
    {
        studState=5;
    }
}
Click to Expand
0

Future Directions

As a first attempt at using both Particle's cloud based physical computing platform as well as using Teachable Machine in code, I am wildly pleased with how this project turned out. 

In some respects, the seamlessness and apparent utility of the final product may have actually been taken too far though. The authoritative yes/no of the buzzer is too easily trusted and doesn't fully allow the user to experience moments of doubt or uncertainty about this new ML model in their home. 

Future iterations might use that experience of uncertainty as a jumping off point to look deeper into how ML models change our truth-seeking behaviors in the face of the unexplained. 

x
Share this Project

This project is only accessible by signed in users. Be considerate and think twice before sharing.


Courses

48-528 Responsive Mobile Environments

· 3 members

As part of this project-based course, we’ll get hands-on with emerging technologies, concepts and applications in the internet of things through a critical socio-technical lens. Over it’s 15-weeks,...more


About

Create and showcase a piece of spooky technology