Visual Translation

Paul Klee would practice the violin in preparation to painting. His desire for expressive power can hardly be better found than in the wide-ranging, voluminous sound the instrument produces. What he called “warm up” I treat as direct influence, willingly or unwillingly finding its way into the visual outcome, or residue. Resultantly, the visual rhythm in Klee’s painting matches and conveys musical composition structures, but beyond the connection Klee’s mind claims, the relationship between the two remains unclear. In analyzing the association between musical piece and visual elements in painting, I seek to construct a translation mechanism between audible and visual form predicated on Klee’s artistic practice.

I decided to iterate the “warm up” assignment from our first class module where I created a piece of digital homage to a 20th century artist/artwork, in my case Paul Klee’s Insula Dulcamara. With this digital recreation as with my final project, an important consideration was to ask what “realm” the work I produced was in. Viewing reality as one of many and art as a window, Klee developed a “transcendentalist approach to the material world” as musician and painter.

It seems undeniable that there exists a link between music and visual form. This link for Klee constituted a duality rooted in Nature and the Cosmos. He believed, “Dualism should not be treated as a complimentary whole… Truth requires the consideration of all elements, the work of art — a fusion of them all.” To find and understand these elements, Klee stipulated the essential condition that the artist communicate with nature: “The artist is human; himself nature; part of nature within natural space.” This mindset led Klee to draw inspiration from both painting and music. “Senses of form and tone are of human primordial heritage. Klee fused both of these creative impulses into a new entity.”

The “new entity” where creative impulses fused had been theoretically described by philosopher and social critic Theodore Adorno as the “point of convergence” between music and painting. Klee “visually revealed” this contact between the two art forms in his work with a focus on rhythm as the important link between music and painting. He saw rhythm as “capable of illustrating temporal movements in both” and in his early paintings attempted to "solve structural affinities between the two." Patrice Pavis noted that “Klee’s painting “plays simultaneously ‘on both panels’, visual and auditive, according to the two rhythms peculiar to each system.” Klee had linked painting and music through the course of his life, but what was the causal relationship between the two in his practice?

To create a visual translation mechanism, it was important for me to understand how music influenced Klee’s painting. One view is that music is the main component that influences visual outcome, such that the visual is more like residue from the experienced audible. In this system, creation concentrates around the suscitation of sound combination, and the visual stemming from it falls to a secondary standing. “Sound is not an arbitrary signifier" denoting another substance.  Jim Drobnik alleges that “representation and interpretation are issues in which sound shares with pictures, yet sound reconfigures these very issues by inflecting representation with affect, and interpretation with embodiment,” challenging the conventions of visual models. “Signifying through its own materiality,” sound is substantive in itself and the visual merely produces a "transfiguration of an already perceived form.". The music-visual relationship can also be understood in a way that the visual is constructed to match and convey the audio, amplifying it through a different medium. When Klee became a teacher at the Bauhaus, he also became an “interpreter of signs.” Klee envisioned an art which translated the musical achievements of Bach and Mozart into visual terms. Ultimately for my project, I operate on the assumption that Klee visualizes sounds with the purpose that the "auditive cements the visual.”

Klee endeavored to convert music into painting and my visual translation mechanism strives to reimagine this translation process. Many of Klee’s lecture notes to Bauhaus students contain musical notations and explanations of the mathematical approach to musical patterns. Before beginning any construction of my own, I looked at some of Klee’s music inspired paintings, focusing on the re-representation of rhythm, as this was a parallel already brought to my attention from research. The works that came up were Fugue in Red, In the Style of Bach, Polyphony, and Cooling in the Garden of the Torrid Zone.

Fugue in Red, based on Mozart’s Fugue in G minor, is a 1921 watercolor focusing on polyphony and counterpoint. "Klee perceived a clear visual connection to the structural articulations found in music. He connects structural articulations, exposing figurative forms or abstract derivations."  Georg Predota offers a great analysis:

“Overlapping shapes float over a two-dimensional surface, with the temporal aspect graphically represented by a gradual shift in color. Moving from the dark background to maximum transparency, the visualized counterpoint combines in a cosmic harmony that reaches towards a new sense of spirituality. Although essentially structural in approach, this painting embodies Klee’s believe in 'harmony, autonomy, and universality in humankind.' As a musician and a painter, Klee essentially created a harmonious arrangement that echoes a universal order.”

In the Style of Bach is "conceived as a musical score with its implied linearity, and with plants, symbols and signs used as fermatas and musical pauses."

In Polyphony, 1932, "Klee uses primarily color to express his musical ideas. Background color blocks simulate the deep bass chords of a musical composition, but then on the painting’s surface, Klee superimposes tiny dots in different colors in luminous fashion — acting here too like the counterpoint of Bach’s polyphonic musical structures." This piece, in conjunction with other information about Klee, brought me to focus on color in relation to relative pitch in my final project. In 1914 Klee traveled to Tunisia, a trip which changed his perception of color, leading him towards abstraction and making the statement “Color and I are one. I am a painter”. Furthermore, in addition to constructing abstraction, Klee’s usage of color contrasted academic art since the Renaissance based on principles of absolute beauty and conventional color canons.

Klee’s painting Cooling in the Garden of the Torrid Zone, which was influenced by Bach’s Fugue in E minor, is "the very example of a painting which shows the rhythmic, musical structure of elements divided by horizontal lines similar to staves in a musical score with signs and characters simulating the notes of a musical bar. Just as in the fugal structure, there is a balance between regular elements and irregularities, creating tension and, in turn, repetitive structures perceived as rhythmic, which are clearly musical." The book Paul Klee -The Nature of Creation says of Klee’s use of rhythm: “Repetitive divisions alone do not constitute a rhythm, and the regular repetition of an individual mark simply renders it indistinguishable … in order to perceive something as rhythmic, a balance has to be set up between a regular element and irregularities …. only through this tension can a repetitive structure be perceived as rhythmic.”

We see that Klee has translated audible input into visual output in these examples, but what are the parameters along which he does so? The mind is subject to thinking in patterns which makes it plausible that a certain type of audio input consistently yields the same or similar visual output. To understand this transferral process more precisely, I focused specifically on Bach’s Fugue in E minor and the corresponding painting Cooling in the Garden of the Torrid Zone. Klee had spoken of his art as “andacht zum kleinen,” meaning devotion to small things, so I respectively broke down the two works into their constituent structures and elements, and then established the cause and effect relationship between the music and the visual components.

I started with the music. First, I listened to the fugue, played on both organ and piano, in different tempos. I listened a few times, once sitting and looking out, once with my eyes closed, twice while looking at solid color colored papers, and the last three times while following the sheet music. In between these listens I drew while listening to the fugue, using a single gel ink color and solid background. I was anticipating to relate structures in the fugue with visual structures, but the fast paced tempo made it impossible for me to do so in real time while the piece was playing. I noticed that instead, I focused on correlating my stroke intensity (strength) and stroke speed with progression peaks. Listening to the piece and drawing at the same time also blocked any intentions to illustrate.

After experiencing real time visualization of the fugue, I moved on to analyzing the music, which I could only do using the score. This raised the important philosophical question for me of whether I can treat the score as interchangeable with the musical piece itself. My view is that the score is a line of code instructing the realization of the fugue such that it is not the musical piece itself, but is a valid substitute for analyzing structural musical elements. The same elements could be theoretically gathered from listening, the “true” channel of music perception, but it is in fact more plausible to do so by analyzing the score. Additionally, from the perspective of the player, hearing music comes with looking at the notations laid out on the page.

Thus in my analysis of the fugue, I started by looking at the score and exploring different modes of score representation. “Since at least the 14th-century composers have questioned the visual boundaries of the musical score. From the heart-shaped chanson of Baude Cordier in the Chantilly Codex to the Hörpartitur created in 1970 by Rainer Wehinger for György Ligeti’s Artikulation. Graphic uses of notation have expanded the available palette for composers beyond the limitations of the 5-line staff.”

First I looked at pages of non-conventional notating composers, like Sylvano Bussotti, Luigi Russolo of the Futurist movement, and John Cage. Although the notations were adapted to meet technical needs, like accomodating for non standard orchestral instruments, I used these excerpts of novel music notation to explore formal possibilities.

Then I looked at graphic scores on YouTube that played along with midis. 

Bar Representation

Circular Representation

Chromadepth 3D

These graphic scores all operate based on a guiding principle that follows a consistent pattern for the same music sequences. After understanding these patterns, I improvised with making a few of my own graphic scores, in part to understand how score construction affects perception of the music from the player’s perspective and how the deviation from standard conception of what music looks like affects the music produced. I chose Paganini’s Cantabile for this exercise.

The following graphic score is governed by a basic rule: a pineapple equals a half note, apple equals quarter note, lime equals eighth note, and lemon equals sixteenth note. The spatial positions are based on the staff, but the lines are removed. The sheet is purged in single color. The arrangement of the floating fruit implies a progression continuously to the right. This visualization follows a straightforward rule and only two elements were changed (note value and the staff), but already the result seems so distant from what we’d typically associate with music, as if the new score leans more toward visual art. The structure of this first score was unchanged, however, so with the second version I sought to implement my own structure.

For the second graphic score I made I again used Paganini’s Cantabile. The seven notes that make up the entire range of musical sound are on top. The line of “code” is moving from the top left corner, resembling a violin fingerboard. I had to tackle addressing note value (length) and pitch. For simplicity my score does not differentiate between octave changes for the same pitch. A unit rectangle equals an eighth and longer notes are composed of attached unit rectangles. As the note is sounded, the value indicator connects to the pitch, while the next two consecutive notes are hinted in more transparent color. This offers some foresight into the direction of the piece, but the constraint (since only a limited portion of the music can be displayed at a time) forces a focus on the music at hand. Already I was feeling like the score I created was addressing a different type of music - certainly not the classical kind I’ve been listening for the project. Additionally, designing how my score works allowed me to understand the task the artist (Klee in my case) faces when instituting an order in his paintings.

Having understood the characteristics and nature of musical scores better now, I began my analysis of the fugue in the sheet music. Fugue in itself tells of certain characteristics of the piece. The definition of fugue is a contrapuntal composition founded on a concise theme called the “subject.” All the voices are of separate melodic importance. Its form is an application of the “ternary” - three principal divisions recognizable usually according to the keys employed. “There is not, however, any actual feeling of thinness; the extensive use which is made of arpeggio serving to enrich the harmonic effect.” I annotated certain structures in the score: ascending and descending arpeggios, arpeggio peaks, exposition, sequence parallels, subject and countersubject, fugue and counterfugue, key changes, register reversal, and returns to the theme. After this analysis, Bach’s fugue was broken down into a disjointed collection of standard elements and partially lost the awareness of human ingenuity behind it.

Having a sense of the fugue’s rhythm and movement, I moved on to breaking down the visual elements in the Cooling in a Garden of the Torrid Zone. The painting is divided by horizontal lines that “are not unlike staves in a musical score. Onto the lines are hung different, repeated signs and characters that are like the notes of a musical bar.” In order to better understand the fundamental units of a visual work and what structures Klee paid attention to when painting I found a primary source, namely Klee’s pedagogical sketchbook. In it, Klee goes chapter by chapter to explain the components of visual construction. I arranged his notes by sections and summarized them in my own words, with my own illustrations that explain and provide quick reference to the visual units I will be dealing with when analyzing the building blocks of his painting.

Because rhythm and repetition are so essential to my project, before moving on to the translation step of my work, I manipulated the first line in the Cooling in a Garden of the Torrid Zone to demonstrate Klee’s own “principle of repetition.” Using an actual work of his and by investing the time to preserve almost all stylistic elements of the original in my versions of the line, I was able to discover the importance of intent in art making. Although visually my lines highly resembled Klee’s and the structure followed that proposed by Klee himself, (1 unit repetition, 2 unit repetition, 1+2 unit repetition seen below) because the arrangement I produced was of my own design, they felt void in comparison to the work from which they stem.

Having decomposed the two pieces into fundamental structural elements, my analysis aimed to directly connect “music code” to a specific visual element in an “A to B” translation method. However, I found it not worthwhile to connect specific music structures with specific visual outcomes because I felt like that took away from the imaginative aspect of the project and that the amount of variation possible would require I either make more general what I treat as component structure or make so many different options that the pairing might equally well be done randomly. Building off of Klee’s notes on direction of movement and the saturation of arpeggios throughout the fugue I chose to connect the motion of objects depicted with the relative pitches in the music. My space of visual translation was divided into top and bottom half, corresponding to the first and second voices of the fugue. This would allow the translation to flow according to the music and reveal visually as much simultaneously as the fugue does. Furthermore, seeing both sections at once would allow the viewer to follow the flow of the subject from top to bottom voice. I visually translated the first four measures of the fugue, with the eighth note being the core unit of time measurement. This means that each eighth note corresponds to one frame in the translation sequence. In the animation of the sequence, I made each frame equal to 0.3 seconds, which is a breath slower than the tempo in most recordings of the fugue. A big consideration was whether to have the animation include the appropriately slowed music as well or whether, following more in Klee’s habit of listening first and then separately painting, have the visual translation I’m creating be presented separately. I opted with the latter.

My visual translation mechanism, set up for public display at the showcase, defers any notion of stage setting. This choice too was in tune with Klee’s mindset: “Klee was not interested in the theatrical productions at the Bauhaus.” The device on which my animation could be viewed was placed on a music stand, to indicate and fit its original context. Additionally, Klee would have approached the stand just as viewers had at the showcase to warm up before painting. Similarly, viewing the animation I made is a warm up for appreciating and understanding the transfer of musical reception into visual conception.

Finally, certain aspects of my project are left open for even further development. I want to develop a more general instruction oriented mechanism that is able to recognize sound patterns based on relative pitch to generate fast moving objects on screen in real time. Technology like Shazam or Siri are able to listen to a given song and identify it based on a short excerpt but this recognition process only works for select produced songs. A violin concerto recorded by the New York Philharmonic for example would be correctly identified, whereas the same concerto played by someone on the violin at home would not. My next iteration of the visual translation mechanism would contain all Bach fugues, for example, categorized by unique 5-10 opening notes. A tuner would register the live notes played on the instrument and then match the heard note combination with one in the database. This would inform my mechanism which visualization patterns to use for the rest of the piece and the pace would adapt to the speed of the instrument - measured by the time elapsed per bar.

Using the same approach Klee employed, playing (hearing) the musical piece first and then creating his visual artwork (the painting), here is Bach's Fugue in E minor. 

Having heard the music, below is the animation of the visual translation of the first four measures, my final project.

Bibliography & Sources 

Because footnotes are not available in the Ideate Gallery format of presentation, all sources quoted and otherwise used in the above documentation are compiled in the below general bibliography.

Wolfman, Ursula Rehn. "Paul Klee — Painting and Music." Interlude. N.p., 13 Nov. 2016. Web. 10 Dec. 2016.

Ovadija, Mladen. "Introduction." Introduction. Dramaturgy of Sound in the Avant-garde and Postdramatic Theatre. Montreal & Kingston: McGill-Queen's UP, 2013. Print.

Wolfman, Ursula Rehn. “leit. N.p., 13 Nov. 2016. Web. 10 Dec. 2016.

Predota, Georg. "Paul Klee: Fugue in Red." Interlude. N.p., 15 Nov. 2014. Web. 10 Dec. 2016.

Prout, Ebenezer, and Louis B. Prout. Analysis of J. S. Bach's Forty-eight Fugues: Das Wohltemperierte Klavier. London: Edwin Ashdown, 1910. Print.

Klee’s Diaries, p. 380, letter of July 17, 1917 to his wife, Lily

Smalin. "Chopin, Fantasie-Impromptu, Opus 66." YouTube. YouTube, 03 Feb. 2013. Web. 18 Dec. 2016.

Smalin. "Bach, Toccata and Fugue in D Minor, 10-year Anniversary Edition." YouTube. YouTube, 02 Dec. 2015. Web. 18 Dec. 2016.

Smalin. "Pachelbel, Canon in D (Chromadepth 3D)." YouTube. YouTube, 11 Mar. 2016. Web. 18 Dec. 2016.

Saggin, Valerio. "Intonarumori." Theremin Vox - Intonarumori. Theremin Vox, 21 Feb. 2004. Web. 18 Dec. 2016.

Gritton, Frankie Harriet Needham. "Post Inspiration." Sound and Image. N.p., 12 Apr. 2014. Web. 18 Dec. 2016.

"Inventing Abstraction at The Museum of Modern Art | Image: Luigi Russolo and His Assistant, Ugo..." Inventing Abstraction at The Museum of Modern Art. N.p., 25 Feb. 2013. Web. 18 Dec. 2016.

McCabe, Denton. "Graphic Music Notation." New Mexico Museum of Art - Sylvano Bussotti's Graphic Music Notation - 4/1/2015. New Mexico Museum of Art, 1 Apr. 2015. Web. 18 Dec. 2016.

Bruhn, Siglind. "WTC I/10 in E Minor – Fugue." Wtc-I-10.htm. N.p., n.d. Web. 18 Dec. 2016.

Smith, Timothy. "Fugue in E Minor from the Well-Tempered Clavier Book 1 by JS Bach." Fugue in E Minor from the Well-Tempered Clavier Book 1 by JS Bach. Oregon Bach Festival, 2002. Web. 18 Dec. 2016.

Saunder, Matthew. "Matthew Saunders’ Blog." Matthew Saunders Blog. N.p., 20 Nov. 2009. Web. 18 Dec. 2016.

Prout, Ebenezer, and Louis B. Prout. Analysis of J. S. Bach's Forty-eight Fugues: Das Wohltemperierte Klavier. London: Edwin Ashdown, 1910. Print.

Klee, Paul, and Sibyl Moholy-Nagy. Pedagogical Sketchbook. London: Faber and Faber, 1953. Print.