The moody frame
Made by Ponn Thanasathit, Ben Oppenheim, Nick Possi-Moses and KRITTIKA BHATNAGAR
Created: November 30th, 2022
0
Intention
The Moody Frame is all about connecting to friends and family through the experience of music. Combining subtle light output of the Ambient Orb and the smart, but not intelligent pictures displayed in a smart frame, The Moody Frame turns pictures of friends and family into an interactive IoT device that subtly shares your "music mood" using light to outupt what kinds of music you are listening to. Additionally, if you are listening to a song you really like and want to share with friends, you can press a button on The Moody Frame which will subtly broadcast that to your friends with in a cascade of colored light, and automatically add it to a shared playlist you have set up.
We started off this project knowing that we wanted to create a device that revolved around music. We’re all music lovers and enjoy listening to and sharing music with each other. Music is a powerful tool for expressing emotions and can connect people across languages and cultures.
We also knew that we wanted to create a device that not only connects people but also allows people to communicate with each other.
We started with an idea that we had come up with during one of our in-class co-creation sessions which was a digital frame that cycles through photos of friends and family who we have not communicated with for a certain amount of time. What we liked about this idea was the photo frame aspect which is a throwback to our childhood when special people and memories in our lives were commemorated by being framed. Since we wanted to create a device that connected people, we decided to use the photo frame form where a person can put a Polaroid of the person they want to be connected to in the device.
We knew we wanted to create a device that connects people through music, but felt that creating an audio IoT device would feel intrusive and demanding of our attention, which goes against the principles of glanceability. Therefore, we decided to take music as the input and lights as the output.
Historical Case
We were inspired by two devices for this project - The Ambient Orb and the Good Night Lamp. We really enjoyed the glanceable nature of the Ambient Orb and how it was able to communicate information in a simple and non-intrusive manner. The Ambient Orb was our inspiration for how we wanted to use light in our device - simple and elegant.
Additionally, we liked how the Good Night Lamps used light as a visual cue to communicate across far distances and allowed for social connectedness through the network of lamps. We wanted to replicate the concept of connecting photo frames together and using light as a communication tool.
Approach
Initially, we started with just a single device, which would retrieve two pieces of information:
1) Currently playing song from Spotify
2) Currently playing song's genre
Based on the genre of the currently playing song, the frame would display a specific color (we had created a genre-color mapping in the code).
We realized that there were shortcomings to just using one device, so we decided to create another device that would communicate with the first device. This second device would show what genre of music the first device's owner was listening to. In this way, it could 'communicate' the mood of one person to the other. Obviously, there are limitations to using genre as a way to communicate mood, as different genres can have very different feelings associated with them. We decided to use genre, though, because it can generally communicate overarching mood (think classical music = relaxed, pop = high energy, rock = high energy, cool jazz = relaxed, etc.)
An alternative would have been to use other pieces of meta data associated with a song that is captured by Spotify, such as danceability (which would indicate how fast-paced and rhythmic a song is).
Next, we wanted to leverage the goodnight lamp to communicate songs that make you think of the other person. To do this, we added a button to the frame that, when pushed, would pull the currently playing track and add it to a shared playlist so that the other person with the other frame could receive notification that a song had been added to their shared playlist.
Process
Below, we included a process for how the frame works.
With respect to a reflection on the process itself and the many iterations, refinements, and challenges:
Initially, we wanted to pull the 50 most recently played songs and display the ratio of those genres on the neopixel (i.e. if there are 50% pop songs, 50% rock songs, 50% of the neopixels would show as one color and 50% another). However, we realized that there were limitations to the API and how many webhooks could be used, so we had to be very deliberate with the webhooks we used for our device.
Knowing that the particle can only use 4 webhooks in a program, we decided to leverage the 'currently playing' webhook to retrieve the track's artist (and their primary genre).
We struggled initially with the Sptofiy APIs, too. For example, with the add song to playlist webhook, we were unsure initially how to pass the value of one API (get currently playing track) to the other. After some research, we were able to figure out how to execute this task.
Additionally, as we were finalizing the completion of our frames, we ran into an unexpected issue. For some reason, our program was no longer running. We hadn't changed the code, so we began to look at the wiring to see if perhaps there was loose wire. We also reviewed the code to ensure nothing had changed. After nearly an hour of pouring through the code and checking the breadboard and the wires and all the connections, we realized that the only explanation was that the particle had been fried. When we checked the code on another device (and the same wire setup), we realized that this was indeed the case.
// SENDING DEVICE CODE
// This #include statement was automatically added by the Particle IDE.
#include <neopixel.h>
// IMPORTANT: Set pixel COUNT, PIN and TYPE
#define PIXEL_PIN D4
#define PIXEL_COUNT 22
#define PIXEL_TYPE WS2812
// Define variables
int numGenres = 126;
String genreToFind = "";
String artistID = "";
String genreSimplified = "";
String trackID = "";
String spotify = "spotify:track:";
String track_success_cd = "";
int color[3] = {0};
int r = 0;
int g = 0;
int b = 0;
int buttonState = 0;
int GlowState = 0;
unsigned long lastCheckTime = 0;
// unsigned long now = 0;
String genres [] = { "acoustic","afrobeat","alt-rock","alternative","ambient", "anime", "black-metal", "bluegrass", "blues", "bossanova", "brazil", "breakbeat", "british", "cantopop", "chicago-house", "children", "chill", "classical", "club", "comedy", "country", "dance", "dancehall", "death-metal", "deep-house", "detroit-techno", "disco", "disney", "drum-and-bass", "dub", "dubstep", "edm", "electro", "electronic", "emo", "folk", "forro", "french", "funk", "garage", "german", "gospel", "goth", "grindcore", "groove", "grunge", "guitar", "happy", "hard-rock", "hardcore", "hardstyle", "heavy-metal", "hip-hop", "holidays", "honky-tonk", "house", "idm", "indian", "indie", "indie-pop", "industrial", "iranian", "j-dance", "j-idol", "j-pop", "j-rock", "jazz", "k-pop", "kids", "latin", "latino", "malay", "mandopop", "metal", "metal-misc", "metalcore", "minimal-techno", "movies", "mpb", "new-age", "new-release", "opera", "pagode", "party", "philippines-opm", "piano", "pop", "pop-film", "post-dubstep", "power-pop", "progressive-house", "psych-rock", "punk", "punk-rock", "r-n-b", "rainy-day", "reggae", "reggaeton", "road-trip", "rock", "rock-n-roll", "rockabilly", "romance", "sad", "salsa", "samba", "sertanejo", "show-tunes", "singer-songwriter", "ska", "sleep", "songwriter", "soul", "soundtracks", "spanish", "study", "summer", "swedish", "synth-pop", "tango", "techno", "trance", "trip-hop", "turkish", "work-out", "world-music"};
int colors [][3] = {{255,255,0}, {0,255,0}, {255,0,0}, {128,128,128}, {255,0,255}, {255,0,255}, {128,0,128}, {0,0,255}, {0,0,255}, {128,128,0}, {0,255,0}, {0,255,255}, {128,0,0}, {0,128,128}, {255,0,255}, {255,255,255}, {255,0,255}, {0,0,128}, {255,0,0}, {255,255,0}, {128,128,0}, {255,0,0}, {255,255,0}, {128,0,0}, {255,0,255}, {255,0,0}, {0,128,128}, {0,255,255}, {255,0,0}, {0,255,0}, {0,255,0}, {0,0,255}, {255,255,0}, {255,255,0}, {255,0,255}, {128,128,128}, {255,255,0}, {0,255,255}, {128,0,128}, {128,0,0}, {255,0,0}, {0,0,255}, {255,0,255}, {128,128,0}, {0,255,255}, {128,0,0}, {128,128,0}, {255,255,0}, {128,0,0}, {128,0,0}, {128,0,0}, {128,0,0}, {0,255,0}, {0,255,0}, {128,128,0}, {255,0,255}, {255,255,255}, {255,255,255}, {128,0,128}, {128,0,128}, {255,0,0}, {0,255,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {0,0,128}, {0,0,255}, {255,255,255}, {0,255,0}, {0,255,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {192,192,192}, {255,0,255}, {0,255,255}, {128,128,0}, {255,0,255}, {128,0,128}, {128,0,128}, {255,0,0}, {128,0,128}, {0,255,255}, {0,255,255}, {0,255,255}, {0,255,255}, {0,0,255}, {255,0,0}, {255,0,0}, {255,0,0}, {0,0,255}, {0,0,128}, {0,255,0}, {0,255,0}, {128,128,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,255}, {0,0,128}, {255,0,0}, {0,255,0}, {255,0,0}, {255,0,0}, {0,128,128}, {0,255,0},
{0,128,128}, {255,255,255}, {255,255,0}, {255,255,255}, {255,0,0}, {0,128,128}, {255,255,0}, {0,0,255}, {0,255,255}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {255,0,0}, {0,255,255}
};
// Define Neopixel
Adafruit_NeoPixel strip = Adafruit_NeoPixel(PIXEL_COUNT, PIXEL_PIN, PIXEL_TYPE);
// Define Switch
int switchPin = D3;
int buttonPin = D2;
// int buttonState = LOW;
int switchState = LOW;
//Initiate millis fade calculator & fadeOut
unsigned long lastFade = 0;
int fadeOut = 255;
void setup() {
//set up Neopixel
pinMode( switchPin , INPUT_PULLUP); // sets pin as input
pinMode( buttonPin , INPUT_PULLUP); // sets pin as input
strip.begin();
strip.show(); // Initialize all pixels to 'off'
//set up Particle variables
Particle.variable("Genre",genreToFind);
Particle.variable("R",r);
Particle.variable("G",g);
Particle.variable("B",b);
Particle.variable("Artist",artistID);
Particle.variable("Root Genre",genreSimplified);
Particle.variable("Switch",buttonState);
Particle.variable("TrackID", trackID); // establish variable to hold value of Track ID received from get_currently_playing_track webhook
Particle.variable("TrackAddedToPlaylist", track_success_cd);
// Particle.variable("TrackAddedToPlaylist", track_success_cd);
Particle.variable("ButtonState", digitalRead( buttonPin ));
Serial.print("Setup Complete");
//Subscribe functions
// Particle.subscribe("hook-response/get_recent_track", get_artistID);
Particle.subscribe("hook-response/get_currently_playing_artist", get_artistID);
Particle.subscribe("hook-response/get_genre", get_genreName);
Particle.subscribe("hook-response/add_song_to_playlist", add_to_playlist);
Particle.subscribe("hook-response/get_currently_playing_track", get_track_id);
for(int i = 0; i < strip.numPixels(); i++){
strip.setPixelColor(i, 255,255,255); // set a color
// delay(100);
strip.show();
}
delay(30000);
}
void loop() {
unsigned long now = millis();
int switchState = digitalRead( switchPin );
int buttonState = digitalRead( buttonPin );
if( switchState == LOW){
if (GlowState == 0) {
startupGlow();
GlowState = 1;
}
if (buttonState == LOW) { //Feature 2 send song to playlist
getCurrentlyPlaying();
delay(3000);
if(trackID!="") {
// delay(3000);
addSongToPlaylist();
successPulse(r, g, b);
Particle.publish("AddSongSuccess");
}
}
else { //Feature 1 grab song and show light
if ((now - lastCheckTime) >= 10000){
getGenre2();
lightShow();
lastCheckTime = now;
}
}
}
else {
pixelOff();
// GlowState = 0;
}
}
void get_artistID(const char *event, const char *data) {
artistID = data;
}
void get_genreName(const char *event, const char *data) {
genreToFind = data;
}
void get_track_id(const char *event, const char *data) {
trackID = spotify + data;
}
void add_to_playlist(const char *event, const char *data) {
track_success_cd = data;
}
void lightShow() {
// find the ID of a genre
int indexOfGenre = -1;
// loop over each item
for( int i = 0; i < numGenres; i++ ){
String currentGenre = genres[ i ];
if( genreToFind.indexOf(currentGenre) != -1){
indexOfGenre = i;
r = colors[i][0];
g = colors[i][1];
b = colors[i][2];
for(int y = 0; y < strip.numPixels(); y++){
strip.setPixelColor(y, r,g,b); // set a color
// delay(100);
strip.show();
// strip.show();
}
genreSimplified = currentGenre;
}
}
}
// void getGenre() {
// //initiate webhook get artistID
// Particle.publish("get_recent_track");
// //initiate webhook get genre
// delay(3000);
// Particle.publish("get_genre",artistID);
// }
void getGenre2() {
//initiate webhook get artistID
Particle.publish("get_currently_playing_artist");
//initiate webhook get genre
if(artistID!="") {
// delay(1500);
Particle.publish("get_genre",artistID);
}
}
void lightResponseAddSong() {
for(int a = 0; a < strip.numPixels(); a++){
strip.setPixelColor(a, 255,0,0); // set a color
strip.show();
}
delay(3000);
}
void addSongToPlaylist() {
Particle.publish("add_song_to_playlist", trackID);
}
void getCurrentlyPlaying() {
Particle.publish("get_currently_playing_track");
}
void pixelOff() {
for(int x = 0; x < strip.numPixels(); x++){
strip.setPixelColor(x,0,0,0); // set a color
strip.show();
}
}
void startupLED(int r, int g, int b, int speed) {
for (int i = 0; i < strip.numPixels(); i++) {
strip.setPixelColor(i, r, g, b);
strip.show();
delay(speed);
}
}
void startupGlow() {
startupLED(0, 150, 150, 50); //Initiate Startup Sequence
startupLED(0, 50, 255, 50);
lastFade = 0;
fadeOut = 255;
for (int fadeOut = 255; fadeOut > 0; fadeOut--) {
for (int i = 0; i < strip.numPixels(); i++) {
//Pulse to green
strip.setPixelColor(i, 255 - fadeOut, max(50, 255 - fadeOut), 255);
strip.show();
}
delay(1);
}
}
void successPulse(int r, int g, int b) {
lastFade = 0;
fadeOut = 255;
for (int fadeOut = 255; fadeOut > 0; fadeOut--) {
for (int i = 0; i < strip.numPixels(); i++) {
//Pulse to green
strip.setPixelColor(i, max(0, r - (255 - fadeOut)), min(255, g + (255 - fadeOut)), max(0, b - (255 - fadeOut)));
strip.show();
}
}
}// RECEIVING DEVICE CODE
// This #include statement was automatically added by the Particle IDE.
#include <neopixel.h>
// We will be using D2 to control our LED
int ledPin = D2;
// IMPORTANT: Set pixel COUNT, PIN and TYPE
#define PIXEL_PIN D2
#define PIXEL_COUNT 22
#define PIXEL_TYPE WS2812
Adafruit_NeoPixel strip = Adafruit_NeoPixel(PIXEL_COUNT, PIXEL_PIN, PIXEL_TYPE);
//Set RGB colors
int r = 0;
int g = 0;
int b = 0;
//Define variable to trigger startup fade
bool startupTrigger = false;
//Set loop speed
int speed = 50;
//Store blink state
bool doBlink = false;
//Initiate blink success state
bool doBlinksuccess = false;
//Initiate millis fade calculator & fadeOut
unsigned long lastFade = 0;
int fadeOut = 255;
int fadeIn = 0;
void setup()
{
//Initiate neopixels to off
strip.begin();
strip.show();
//Initiate addedToPlaylist fallback function with call "addedToPlaylist"
Particle.subscribe( "addedToPlaylist.Success", addedToPlaylist );
startupGlow();
}
//Turns all LED's to specified color (r, g, b) with adjustable ms delay
void startupLED(int r, int g, int b, int speed) {
for (int i = 0; i < strip.numPixels(); i++) {
strip.setPixelColor(i, r, g, b);
strip.show();
delay(speed);
}
}
void startupGlow() {
startupLED(0, 150, 150, 50);
startupLED(0, 50, 255, 50);
lastFade = 0;
fadeOut = 255;
for (int fadeOut = 255; fadeOut > 0; fadeOut--) {
for (int i = 0; i < strip.numPixels(); i++) {
//Pulse to green
strip.setPixelColor(i, 255 - fadeOut, max(50, 255 - fadeOut), 255);
strip.show();
}
delay(1);
}
}
//using rgb, input light pulse when change of genre or startup song
void successPulse(int r, int g, int b) {
lastFade = 0;
fadeOut = 255;
for (int fadeOut = 255; fadeOut > 0; fadeOut--) {
for (int i = 0; i < strip.numPixels(); i++) {
//Pulse to green
strip.setPixelColor(i, max(0, r - (255 - fadeOut)), min(255, g + (255 - fadeOut)), max(0, b - (255 - fadeOut)));
strip.show();
}
}
r = 255;
g = 50;
b = 140;
for (int fadeIn = 0; fadeIn < 255; fadeIn++) {
for (int i = 0; i < strip.numPixels(); i++) {
//Pulse to original color
strip.setPixelColor(i, max(r, fadeIn), min(g, 255 - fadeIn), max(b,fadeIn));
strip.show();
}
delay(10);
}
}
//Turns all LED's to specified color (r, g, b)
void setLED(int r, int g, int b) {
for (int i = 0; i < strip.numPixels(); i++) {
strip.setPixelColor(i, r, g, b);
strip.show();
}
}
//Turns all LED's to white
void setwhite( ){
for (int i = 0; i < strip.numPixels(); i++) {
strip.setPixelColor(i, 255, 255, 255);
strip.show();
}
}
//Turns NeoPixelOff
void neoPixelOff( ){
for (int i = 0; i < strip.numPixels(); i++) {
strip.setPixelColor(i, 0, 0, 0);
strip.show();
}
}
//Blink LEDs to input color
void blinkLED( int times, int r, int g, int b){
for( int i = 0; i < times; i++) {
for( int i = 0; i < strip.numPixels(); i++ ){
strip.setPixelColor(i, r, g, b); // set blue
strip.show();
}
delay(500);
for( int i = 0; i < strip.numPixels(); i++ ){
strip.setPixelColor(i, 0, 0, 0); // turn off
strip.show();
}
delay(500);
}
}
//define listen code to run Blink event
void addedToPlaylist( const char *event, const char *data) {
doBlink = true;
}
void loop()
{
setLED(255, 50, 140);
//Blink LED's green 6 times
if( doBlink == true ) {
//Run Success pulse
successPulse(r, g, b);
//LEDs return to previous values
// setLED(255, 50, 140);
// delay(200);
successPulse(r, g, b);
// setLED(255, 50, 140);
// delay(200);
doBlink = false;
}
}Resulting Prototype
Deliver a functional mock up of the device(s) prototyped using Particle. Document the outcome itself (code, circuit diagrams, photos, design files, 3d models, video demonstrations, etc. as required) and provide a short narrative. This should include: a workflow diagram and a bill of materials (sensors, input devices, actuators, and other components). This documentation should be sufficiently rich to allow anyone to repeat / recreate it.
Initial Prototype:
Building on a simple picture frame, an initial 3d model was constructed in Rhino to determine what the look and feel of the device could be. Initially, we took the dimensions of a Polaroid and a reasonably sized frame and attempted to allocate the remaining space for the electronics and Neopixels. The construction is simple: in the model, pink represents translucent acrylic to allow the light to shine through, grey is black acrylic, and orange is 3mm plywood. Unfortunately, after laser cutting and building this first version we realized we needed more space for the device and image, and the lights were too close to the outer frame and did not provide the right diffuse glow effect.
Revised (Final) Prototype:
After testing the first prototype we went back to the drawing board to make some adjustments. First, we decided to enlarge the frame to accommodate the entire Polaroid and not just the image portion. Next, after testing the Neopixels and translucent acrylic, we determined that the LED's needed to sit 10mm from the edge of the acrylic to give the correct light effect. Using this as a guide, we revised the frame, and added some internal tabs for the plywood component to allow for a tight press fit that would keep the device closed, but still operable. Last, we developed two types for the devices, one wall mounted, and one desk mounted. The difference between the two devices rests in how they meet the horizontal plane. In the desk mounted version, the device is raised off of the horizontal surface to allow an even cast of light on all sides.
Conceptual Design:
If we had more time we would consider the following enhancements:Increase the interactive potential with Spotify: Make a more seamless connection with Spotify that allows you to log in once and not have to refresh the oauth token every hour. With this approach we could allow users to set up their own color patterns related to genres that they could then share with their friends.
Consider adding more dynamic light display. Instead using a static color to display mood/genre, the device could be improved by incorporating other elements such as danceability or BPM, which could be used to flash the lights at a certain rate.
Add additional feedback loops to allow friends to communicate more effectively. This could be ways to add additional interactivity through other points of interaction with the device, or other light patterns that may showcase or signal other moods. One specific option is to create a "together" mode that allows connected individuals to team up for a visual listening session through the device where lights move to the beat of a song that a friend is currently playing.
Iterate on the form: Is a picture frame the right form factor? We would experiment with alternatives to see what is more effective. It would be great to include a smart element to the picture display to allow users to swipe through different friends and see what they are listening to visually.
Reflection and Critique:
As predicted in our premortum, our biggest challenge was indeed configuring the Spotify API. The circuitry was relatively simple; however, we did have some difficulty in soldering the neopixels and ensuring that they would fit within the specifications of the frame.
While we believe that the value this product exists, there are certainly shortfalls to our approach - and several ways in which it could be improved. Using genre to communicate mood may not be the most effective approach, as genres do not necessarily reflect mood. Some genres can better be generalized in reflecting a mood, such as EDM, but other genres, such as rock, pop, and blues, are more nuanced and therefore less indicative of mood. Instead, we would have wanted to explore using other elements associated with a Spotify song, such as BPM or danceability, to communicate one's mood. Higher BPM would likely indicate a more elevated mood whereas lower BPM would indicate a more relaxed mood. The same is true for danceability. Nonetheless, we think the frame in its current state, and in the context of the time we had to work on it, poses a powerful way to communicate with your friends. It sits passively in the background and informs you of what your friends are listening to, their 'mood,' and also allows you to communicate to your friend(s) that you're thinking of them via the addition of a song to a shared playlist.
Another avenue we wish we had more time to explore is privacy. One question that comes to our minds if what if you don't want your friends, or others, to see what you're feeling or listening to? Is there some way to receive incoming messages but prevent outgoing messages? This is a feature we would want to explore in future iterations; though, as for the immediate version, one could argue that privacy can simply be addressed by turning the frame off.
One thing we really enjoy about this communication approach was that it could fail gracefully. If the IoT mechanisms were to fail, the frame would still be fully functional - that is, it would still function for what it was ultimately intended for - to hold and display a picture.
The value of looking back at overlooked, forgotten, or otherwise older works has been infinitely valuable in our approach. We were able to apply different features that worked well and learn about what didn't work well (and making changes accordingly). We were able to synthesize these features into an ordinarily functional device into one that was enchanted.
Acknowledgements & References:
Provide a list of academic papers, articles, videos, etc that you make reference to in your documentation.
Rose, David. Enchanted Objects: Innovation, Design, and the Future of Technology, Scribner, New York, NY, 2015, pp. 173–178
Spotify APIs: https://developer.spotify.com/documentation/web-api/reference/#/
Ambient Orb: http://www.ambientdevices.com/about/consumer-devices
DioT Lab Website: https://diotlabs.daraghbyrne.me/