Proposal
Set up tablets as "drawing stations", when people draw, sound is produced. Sound is based on color. Use the pentatonic scale because it harmonizes well. Project the images.
Outcome
Curatorial Statement
The first instances of digital painting date back to the 1960s, when the Sketchpad was built. Digital painting and painting software and hardware have since evolved into the programs artists use today, such as Photoshop, GIMP, and CorelPainter. Drawing itself is a practice used to communicate and build new worlds, especially by children, who can often be seen today imagining new worlds with just a surface and a drawing tool.
Sound, while not as prominent a sense as sight, is often used as a type of cue across media, such as films, video games, concerts, and permeates life as an unconscious signal to the listener. While humans rely on sight for most things, sound can set the atmosphere -a chilling silence to unnerve or a cheery tone to lighten it.
Synesthesia is a condition where hearing and seeing are the same thing. It is a sort of melding point where sound equates to sight, where hearing a symphony is the same as seeing it.
Ensemble utilizes both sound and sight in order to explore the point where hearing and sight meet. It is inspired by 21 Balançoires (21 Swings), where play, and musical cooperation are explored through musical swings. Like the musical swings, Ensemble draws upon the simple forms of play by children. Here, Ensemble's canvas acts as the expanse of asphalt where children can draw and collaborate on where it emphasizes exploration and discovery through not only sight but sound, and experience what it is like to see and hear at once, like those with synesthesia. It doesn't dictate to the audience how to interact with it, and allows the audience to break the boundaries and rules of traditional drawing by allowing them to draw based on sounds rather than the visual outcome.
Product
We created an installation that plays music based on the color and object that is drawn. Each color represents a note on the pentatonic scale, and the object represents a percussion instrument. The circle plays a cymbal sound, the square a drum, and the triangle a triangle.
To draw one of the shapes, simply click on the object button and then click anywhere on the screen to stamp the object. To create a line that produces the pentatonic note, hold and/or drag the cursor around. If the color currently being drawn goes over a color on the canvas, a chord is produced.
We created this using Processing 3.0 and used Twitch to stream it on 4 input computers. Originally, this was to be the Macs checked out from IDeATe, but we did have permission to install the necessary components (Processing and OBS), so we used our own machines. One computer uses MultiTwitch, a site that shows multiple streams side by side, to show all 4 canvases at once, and this is projected onto a wall. Each input computer has a different method of input: mouse, touchscreen, drawing tablet, and trackpad.
Intention
Our intention was to make an interactive and collaborative project that was simple. We wanted something that everyone would immediately have had experience with and not need much explanation. Similar to how everyone has played with swings, everyone has experience scribbling. Another goal we had was to connect the visual and the audio. To link certain colors to certain sounds and allow for chords to be made gives freedom to the audience to choose either to focus on drawing what they want, to make music with color, or to just mess around. In this way, our project doesn't have an explicit rule for the audience on how to interact with it.
One of the rules we broke was that of expectation. When people typically draw they rely on their sight to make beauty and collaborate. Here, we wanted to break that convention by emphasizing the sound, and exploring and creating through sound, rather than by sight.
Context
21 Balançoires (21 Swings) - In Montreal, this installation is a playful exploration of cooperation by using musical swings to generate sound. We wanted to build something just as playful and collaborative, as well as fun and exiting to try out and explore.
Interactive installations
Chalk drawings on sidewalks- Chalk drawings often are used by people to send messages, but by children it is a means to imagine and create games and worlds. We wanted to make our Ensemble a portal for people to explore and imagine with sounds and color, and create beauty with sound.
Synesthesia - is where some people have two or more senses mixed together, where any combination of two senses are felt together. We wanted to emulate this experience by playing color and sound together.
Kandinsky's Point and Line to Plane. -In this essay, Kandinsky discusses how the point is both at once sound and its absence, and multiple points have sounds which interact. Lines are color, motion, and also sound, while the plane adds lyrics and temperature. Here in our canvass, we take Kandinsky's elements and express them literally. Points are a cacophony of percussion, while lines form notes and the whole canvass a song
Illusory Contours - Drawing over existing colors plays the color that you've now drawn over, showing you the shape that used to be there that now is up to the imagination.
Process
We discussed many many different possibilities for the final project, but most of them focused on already existing playground games, which was somewhat limiting. One of our more fleshed out ideas involved playing tag with phones using Android Beam. But eventually we branched out into other activities rather than just games, considering what kind of messages we wanted to convey rather than focusing on the end product and what kinds of rules we could break. We discussed our expectations when playing games, such as difficulty ramping up.
After a few meetings, we decided to do something almost completely unrelated to what we had previously discussed. We decided to make a drawing program that plays sound when drawing. We found a programming language that would facilitate creating this program: Processing. One issue we had, however, was implementing sound. Michelle tried tactu5, Marie tried Minim, and Rachel and Lucy tried SoundFile. tactu5 required extra software we couldn't port properly to our program, and SoundFile didn't play correctly. Marie was able to get the basic sound system laid out with Minim, so we used that. After we got our basic program created, we added some extra things, like the percussion instruments.
Collaboration
We did most of the creation and implementation physically together. We all contributed ideas and concepts about "playground" when we were in class, and decided upon our final idea. Everyone had a hand in programming. Michelle was the only one who used Processing before, but we all struggled through getting it to actually work. Michelle created the color buttons, which Lucy later touched up. Marie was working on getting the sound to actually play using Minim. Marie and Rachel was also trying to look into getting the sound to loop, but it didn't work. Marie added the buttons for the percussion instruments, and Rachel and Michelle worked on getting the sounds. Lucy implemented sound looping and the functionality for the drawing tool where they would play the sound of the color below it at the same time as its own sound (to make 2 note chords). Rachel debugged the code to get the percussion sounds working properly and differentiate between mouse clicks (shapes and percussion sounds) and mouse presses (lines and notes). Using Github let us control the versions of code easily, and we created a task sheet on Google Docs to keep track of the tasks and who was doing what and what was done. Michelle and Marie did the final presentation.
Reflection
We thought that our project was a pretty neat concept. It fulfilled our expectations and functioned the way we expected it to.
We learned how to use Processing and GitHub while working on our project.
If we were to do things differently, we would make the canvas larger, add the ability to draw with different shapes, have the colors affect the stamps, change it so that clicking on the buttons doesn't make a sound, and possibly add a clear-all button.
If we had the time, we would like to add multitouch capabilities and perhaps the ability for multiple people to draw on one canvas. We would also have liked to make each station have a different color palette associated with a different octave of the pentatonic scale so that players are more inclined to work together to produce beautiful music.
One of the things we noticed during the presentations was that a lot of people preferred to explore the sounds and colors by themselves, rather than work collaboratively, though what we would have liked to see more collaboration. Another feature we would like to add would be that sounds created on one canvass (if there are multiple) would also be played on other canvasses, so players are encouraged to collaborate to create beautiful music. Perhaps a table which would record the sounds you made and play them back once the picture is finished.
Attribution
Sources and materials used in the documentation or composition.
Kaniza's Triangle and Illusory Contours
Wassily Kandinsky. Point and Line to Plane.
You can upload files of up to 20MB using this form.