Outcome

Concept

(Video 1)

Our initial idea is to make a playground that allows users to be as creative as they want, so we provide the tools and make it aesthetically pleasing, and they create the cool stuff using our environment. We were really inspired by the 21 Swings project by Daily Tous Les Jours so we thought it'd be a cool idea to somehow use a Leap Motion controller, sounds, and building blocks (much like legos) to allow users to create music. http://www.thisiscolossal.com/2012/09/musical-swings-on-the-streets-of-montreal/

(Video 2)

We also came across this 'musical playground' of sorts which is similar to the swing project but it's individual and more powerful, in that the tones put together form songs that repeat. http://www.madewithmonsterlove.com/portfolio/cadence-synesthesis/

(Video 3)

Another interesting project we've seen was a game where you explore and as you change settings or get near different objects, the sounds change. It can be played here: niion.co.uk/synthphony/ .

(Video 4)
Yet another audio game that changes the sound/score/audio as you move around the world.

http://www.visitproteus.com/

Another idea would be to do something similar to tonematrix, where a line sweeps a square grid from left to right and plays the notes associated with the blocks you highlighted when the line sweeps that column. However we'd use leap motion and a cool interactive space in both the physical world and the unity scene. For example, have a box hiding the leap motion controller and get the user to put their hand in the box and explore.

http://tonematrix.audiotool.com/

To provide a rudimentary image of what this concept might look like, below is a sketch of the setup we had in mind. It consists of multiple people using the leap motion devices which can detect many different gestures. You can also see a screen that displays current information of the interactions, in particular, how harmonizing the audience is.

Determining what to display to the user and how to present it will also be a challenge. One idea could be that the users could try to form a simple shape, and once that shape is regular, the participants would know that they have achieved harmony. A perfect example, which came up when google searching ideas for "harmony", is shown below.

The circle might not be the best fit, but the square would be a good start. Perhaps a triangle if we do three participants.

In any case, our concept would be very interactive. That is the facet we are trying to focus on. We would like to have a sort of cooperative experience where participants can play, discover, and have fun with their friends or total strangers. Once we have a solid idea nailed down, we will be able to explore how we can break rules or conventions. We're currently contemplating somehow tricking the audience, but we have yet to determine in what way that would integrate with the presentation well. We would rather it not be an afterthought.

Curatorial Statement

Tone Farm (Dominic Liu, Justice Frimpong, Xavi Villalta)

This project allows users to create music by picking up cubes on the ground in a virtual world and throwing/placing them on the back wall using the Leap Motion Device. The pitches vary depending on the height of the placement. The catch is that the tones being used aren't what people expect.

Product 

We used Unity and the Leap Motion Device to create this musical composition device, and we used Audacity to trim and pitch shift the sounds we downloaded. 

Here is an early mock up drawing showing what the project looks like from both the front view, and the side view. The final product has since come a long way.

Intention 

We made this project because we wanted to make something that allowed people to be as free and creative as possible. We wanted to give them a base to work with so they could explore what they felt was right for them. We originally intended this to be a serious piece of work and use the pentatonic scale for our tones, but for the demo presentation, we used cow noises because we thought it was funny and unique. After our presentation, both in-person and online comments revealed that the humor and bizarreness of the animal noise really entertained people. So we felt that we could both allow people a playground to express themselves with music as well as providing them with humor along the way by sticking with the animal noises.

Context 

1. We were really inspired by the 21 Swings project. We really enjoyed how people could come together with a familiar physical object (toy) and use it to express themselves musically. We especially liked the collaboration aspect of it. However, the more we thought about what we could do that was similar to the 21 Swings, we realized that we had to choose between collaboration, and expression, and we chose expression. We felt that a musically expressive playground would be fun to create and play with so we took that idea from the 21 Swings.

2. From that, we looked at other auditory playgrounds. We found this one which loops the sounds associated to these orbs that you can connect with lines. We liked the idea of looping the sounds because it allows someone to have the feeling of accomplishment after playing in the playground when the hear their end product, and this is a feature that the 21 Swings lacked.

3. We also played around with the idea like this and this. These two games are completely auditory games with no real objective other than to change/create the musical score and sounds. You do this by moving and interacting through the environment. While we didn't end up doing something exactly like this, we were inspired by how the environment played a part in the game and we thought we could do something similar to this using a 3D space (both virtual and physical) with the Leap Motion Device (as opposed to the Kinect because it's nice and small and simple to use).

4. The reading that resonated the most with us is Dixon's Performing Interactivity because towards the end of the reading, Dixon mentions that play could be named an essential element to media performance and artwork. We saw this in action when, for the previous project, our group members created a game (the Madlib constructor) for the audience to play as part of our performance. In the second to last page, Dixon states "Interactive works encourage a playful, childlike fascination for the pleasure of cause and effect, where a simple hand movement [...] causes a domino effect [...]". We realized that almost every other source we looked to for motivation had this 'childlike fascination' that Dixon mentions and this led to our last source of inspiration.

5. During class when we were discussing ideas, Xavi remembered this game called tonematrix. It's just a 16x16 grid of squares. There's an invisible line that sweeps across the grid from left to right and as the line touches a lit up square, it plays the associated tone, as if the program was reading sheet music. It was really fun to play with because we could visualize shapes and hear what they sounded like if we created them on the grid. This is where we got our primary inspiration from as it incorporated all of the elements we wanted to include. We realized using the Leap Motion that we could add another interactive element that could be fun as long as it was easy enough to use. Then we'd just recreate the scene in Unity and use the act of throwing a cube as out point and click.

Process 

We went through several iterations of games before we arrived at our final outcome. At first we thought it would be cool to do something similar to the swings project where only collaboration would reveal a more beautiful and full outcome, but we quickly came to realize that we were simply copying an existing project. We weren't quite able to put our own unique spin on it and keep it as a cohesive project.

However, we wanted to focus on using a different form of interaction that none of us had any real experience with: the Leap Motion controller. This was the core of our project. It provides an almost tangible way of interacting with digital media and allows for the user to connect with the music in a completely different way. This form of interactivity allows us to bring the concept of sound and music to the audience in a simple package.

About the sounds being outputted, originally we were going to go with plain old sine or triangular waves. However we quickly came to realize that these sounds were too boring, they weren't unique enough to tie into the project well. Then, the day before the initial presentation, we had the idea to use farm animal noises instead of regular tones. At first, we weren't sure how well this aspect would be received, but after the initial presentation, we knew that we had made the right decision with this iteration.

Collaboration

We all came up with ideas and refined them together but once we settled on our final idea, Dominic and Justice handled the scripting in Unity (which involved the Leap Motion) and Xavi found the sounds and the textures we used in the Unity scene. He also created mock-ups and story boards on how the playground should look like. Then we all contributed to the documentation.

We convened several times during and outside of class to work on the project and to continue refining our ideas and execution. We also kept in constant contact via a group message so that we could talk about what we were working on, what still needed to be done, and any new ideas that we came up with along the way.

Reflection

We think our project turned out pretty well as it allowed people to have fun as well as create cool sounds that went together. We learned though that giving instructions to people is a lot harder than it seems. We had drawn out instructions for people to follow but as it turned out, they weren't as clear as we though they were. While we were able to use our project well, others weren't as good. We should've expected this though because we were the ones to create the project. We found that others had a hard time being as precise with it as we were so we stepped in to give some pointers that we picked up along the way. We also found that people would put their virtual hand through the barn and this would cause issues with the cubes stickiness to the barn. Next time, we'd have better instructions and maybe have a demo that people could follow. But all in all, we got mainly positive feedback from those that played with it and that's what we set out to accomplish.

Attribution

Minecraft animal textures

Animal sounds

Background