Video
Sonification

Sound Design
The term Sound Design came into vogue in the 1970s to describe a new role in the creation of the sound film, analogous to a “director” or “cinematographer” of the soundtrack. In the early ‘70s Dolby noise reduction, which had already established itself widely in music production and distribution during the ‘60s, expanded its application to the area of film sound. The specific properties of Dolby- increased dynamic range, improved spatialization, better frequency response, and reduction of the noise floor, combined with Dolby’s strategy of providing relatively affordable licensing to theater owners so that it’s noise reduction technology could be widely adapted, provided for the first time a universal standard in cinematic sound reproduction, allowing the sound mix heard in the theaters to be closer to that heard on the mix stage than at any time previously.(i)

The aesthetic possibilities opened up by this technological change were exploited by filmmakers- particularly those based in San Francisco’s “Hollywood North.” For instance, the two artist-technicians usually credited with being the first “sound designers” (the first to receive this designation), Ben Burtt and Walter Murch, were each given unprecedented periods of time to explore sound as a dimension of film. Burtt was given more than a year to build a sound effects library for George Lucas’s Star Wars film soundtrack, famously (in widely circulated images) “wandering the desert” with field recording gear, tapping on phone wires and recording sounds that would eventually support such futuristic technologies as X-Wing Fighters and light sabers. Murch spent a year mixing and re-mixing Francis Ford Coppola’s Apocalypse Now, trying out multiple edits and approaches to what is likewise (as in the case of Star Wars) regarded as a paradigm shift in film mixing.

For purposes of this discussion, we can situate sound design as a practice where sound has a role- often described in the relevant literature (such as Nicholas Cook’s(ii)) as “subordinate” to the film’s imagery. In other words, the work of the soundtrack is to reinforce, through its specific effects (heighted emotion, spatial depth, “ontologically” representing sound sources, clarity of speech, rhythmic pacing and the like) the narratological and often “realist” motivations of the image track (and reciprocally, reserving “weird sound”- often simply taken from the realm of avant garde music- for dream sequences, aliens, monsters and the like(iii)). Such an approach is typical of more narrative, mainstream or commercial projects, such as my sound design and mix of the Isaac’s Storm documentary for the History Channel: (http://www.youtube.com/watch?v=VVUZEfwe-SA)

In contrast, experimental approaches to sound design tend to assert a relative autonomy for the soundtrack in relation to the moving image. In the clips below (from the “art world” side of my sound design practice), I have explicitly aimed to subvert the traditional subordination of sound to image by creating soundscapes that instantiate a play of “going along with” and at the same time “resisting” the kind of affective fusion usually aimed for in designing and editing sound for film. But at the same time there is an “associational intent” at work, in that there is an attempt to create poetic or connotational relationships between sound and image, as illustrated in the clips below.

The Siamese Connection [clip]:
In this clip, sound samples reminiscent of discordant and metallic percussive instruments found in some southeastern Asian music traditions are used as raw material for subsequent processing and synthesis. For more information on this film, visit: http://www.imdb.com/title/tt1354565/plotsummary

Stepping on the Light [clip]:
In this clip, I have extrapolated two sonic dimensions of the video imagery for sonic elaboration- noise (produced by the low res, pocket video technology) and the soundscape of video games (an aspect of the avatar-like navigation through downtown Chicago treated as a virtual space with Cartesian trappings). For more information on this work, visit: http://spatula.ca/sotl

Visual Music
In Visual Music of the moving image variety (e.g. film and video, rather than music as the “referent” or metaphor for abstractions in painting, such as those of Kandinsky or Klee), the image track is often in a fantasia mode vis-à-vis the soundtrack. While in the process of sound design the script, footage and first edits usually precede the production of sound (though there are exceptions, as we saw in the example of Ben Burtt’s collection of sounds for Star Wars ), with visual music there is typically a pre-existent work of music, which the moving image takes as an inspiration or motivation for free form and abstract play. Visual Music as a form can range from formal abstraction (e.g. a play on geometric shapes, light or color), Clip example: Norman McLaren- Dots (1940), Clip example: Mary Ellen Bute, Synchrony No. 2, to specific references to technologies of mediation (e.g. bright flashes of over-exposed film, video footage modulated by rhythms in a techno beat, or even the generative imagery produced by iTunes Visualizer), Clip example: “I Gotta Feeling” (Black Eyed Peas)- fan video, Clip example: “Powercord VS Philter Phreak” to highly stylized and very abstract characterizations of personae with degrees of recognizable action and even plot (for instance, the struggle of an orange triangle to escape from thick lines and grids of black, which metaphorically morph into the bars of a prison or cage). Clip example: Escape: Synchrony No. 4 (1938) by Mary Ellen Bute and Ted Nemeth

Photocollage and montage editing have also been a feature of Visual Music, and highly stylized music video production (especially for electronica genres) can often blur the usual stylistic boundaries between what one might typically refer to as a “music video” versus a work of Visual Music. One Dot Zero collection. Also, the usual order of music-existing-first has at times be reversed for work with a Visual Music sensibility, as with Ballet Mecanique, a film intentionally designed by Fernand Léger to be a kind of musical vessel for George Antheil’s score (the film and the score were finally combined into a sound film in the 1990s). Ballet Mecanique (1924) Fernand Leger/ George Antheil

In its “classic mode” visual music is linked to the tradition of seeking an experience of synaesthesia as a spiritually heightened fusion of the senses. Its antecedents are in Wagner’s Gesamstkunstwerk (indeed, Richard Wagner’s “Evening Star” is the music of Mary Ellen Bute’s Synchrony No. 2 above) or total artwork (a fusion of the all the arts and consequently, of the senses), but also has roots in the Symbolist and Spiritualist movements of the 19th century- all of this, of course, has its origins in Schopenhauer’s philosophy of music, in which music was framed as the highest art due to its ability to directly represent the Will (keeping in mind that for Schopenhauer, the Will as a category included magnetism, love, rage, and electricity- in other words, forces in general, whether internal to one’s self and unconscious, or external in the workings of the world).

“As an outgrowth of the Romantic and Symbolist movements, music was elevated to a status of supremacy over all the other forms of creative expression. The other arts, notably poetry and painting, were said to aspire to the “condition of music” [the phrase is Walter Pater’s- MF]. Artists came to believe that painting should be analogous to music.”

“Proponents of musical analogy based their aesthetic theories on an abstraction of the idea of music, rather than on a clear understanding of musicology. For them music represented a non-narrative, non-discursive mode of expression. They reasoned that music, in its direct appeal to emotions and senses, transcended language. Just as music was a universal form of expression, so should the visual arts attain universality by evoking sensual pleasure or an emotional response in the viewer.”

“Advocates of musical analogy and color music also depended upon the related notion of synaesthesia; that is, they believed in the subjective interaction of all sensory perceptions. This common acceptance of synaesthesia resulted from two divergent philosophical positions. According to the more romantically inclined artists and writers, the interchangeability of the senses was evidence of mystical correspondence to a higher reality. On the other hand, some artists joined forces with scientific researchers to study synaesthesia as a phenomenon of human perception.” (i)

Video sonification is thus linked to a tradition going back to the 19th century interest in cross-modal perceptions. If we understand as a component of Modernism the search for a purity of praxis (e.g. each medium should only develop those properties which are uniquely its own), the synaesthetic tradition can in some way be read as an a counter-modernist inclination, a vestige of romantic impulse in the development of 20th century mediation.

Experimental Music
In her paper “Experimental Music Semiotics,” Morag Josephine Grant elaborates an intriguing Peircean semiotic approach to understanding the distinction between experimental music and other forms of avant garde, classical or “new” music. In her analysis, experimental music has a heightened interest in the indexical relationship to sound, whereas other forms may be better described as having a stronger affinity for the symbolic (in Peirce’s sense of a symbol requiring a subsequent connection to an interpretant).

“Definitive for the icon in similarity with the object referred to, definitive for the index is contiguity with the object referred to, definitive for the symbol is its dependence on a standard rule of interpretation.”(i)

It is in this notion of “a standard rule of interpretation” that one can find a slew of correspondences to non-experimental (including avant garde) music practices. For instance, an interval can be “read” as also referring to a moment in the score, a “minor third,” and all the rules of harmony with its allowances and strictures of what one is to do with a minor third (or not do with it). In the recapitulation of a theme, the work itself can be understood as the interpretant which contextualizes its significance. One can expand this to other fields of music as well. Jazz, for instance, can be understood as a metaphor or symbol for communication (call and response, dialogue). To draw her distinction sharply, she offers the striking example of the sound of a telephone:

“So why don’t telephones ring in music? They ring all over the place in literature. They appear in pictures more often than they do in music. They are the hinge of countless film plots.”(ii)

“The case of experimental music is immediately different because it can deal with the telephone as a telephone.”(iii)

Grant cites Winfried Nöth, noting that “the index makes no assertions regarding its object, but merely shows us the object or draws our attention to it,” the index is “of fact, of reality, and of experience in time and space.”(iv)

Experimental music does not draw a distinction between the realms of “music” and that of “sound and noise.” It forces our attention to the causal dimensions of sonic experience, the productions of sonorous bodies, rather than to the systemic embeddedness of a sound in a formal logic or system (such as score, or the rules of harmony). Indeed, to further bolster her argument that non-experimental (but still scored) music has more of a symbolic character, we need only note the aspects of rhetoric that accompany such compositions: themes, argument, development, recapitulation, verse, refrain, and the like.

Grant does not assert, however, that non-experimental music is only symbolic, or that experimental music is only indexical. Indeed, her essay devotes much time to exploring devil’s advocate, borderline, and seemingly contradictory examples to her schema. She notes that “signification generally involves complex hybrids of these categories [icon, index, symbol].” But as a general description of what may make some music “experimental” and others not, it does have a subtle cogency. For instance, John Cage’s silent piece 4’33” can be understood in relation to Peirce’s example of the indexical weather vane signifying even the absence of wind. “Even if there is no wind at a particular moment, the weather vane still fulfills its purpose, confirming that there is no wind... It is specifically created to draw our attention to something by contiguous relationship with it. Even if there is never wind again, a weather vane will not stop being a weather vane..."(v) The silence evoked by Cage in this work is analogous to the absence of wind- it is still significant (hence a work of experimental music) even though it does not sound.

To take this indexical affinity of experimental music toward visualized sound, we can note two works, one analog and one digital, which display an interesting oscillation between this indexical character and an iconic one that comes into play as a result of the processes of visualization. Norman McLaren’s experiments in writing/scratching directly onto the soundtrack side of a film strip is an analog example of a specifically indexical approach to sonic visualizations. This clip on YouTube, “Pen Point Percussion” illustrates his method (since this is a long clip, you can skip ahead to 2’38” for the illustration of this technique): Pen Point Percussion

Iannis Xenakis’ work Mycenae Alpha was realized with his UPIC computer system, which allowed one to draw sounds onto a tablet and from this visual graph synthesize a score, with the X-axis standing in for time (scalable, from seconds to hours), while the Y-axis represented pitch. This work can be viewed online at: Mycenae Alpha

What is interesting to note about visualized sound in a Peircean semiotic context is that the sound which results from such processes (including Video Sonification) has its “origin” in an indexical (directly causal) relationship. The nature of the index-causality is different in these cases : pen or scratches on film emulsion in McLaren’s work, digital drawing tablet in the Xenakis example, or a mapping via Fourier transforms in the case of Video Sonification. However, once we as listener-viewers are experiencing the work, the synaesthetic play of visual-sonic percepts takes on an iconic dimension as well, as we start to see that the sounds high up in the visual frame also sound high-pitched, or thick bands may produce clusters, while thin bands produce purer tones, or as we notice the way in which rhythmic syncretism in sound and image reinforce each other. At times one feels that one is really “seeing the sound” but simultaneously sounds and images have their own autonomy – in fact, in an iconic sense one is only seeing certain aspects of the sound- both the visual and sonic imagination have their own resonances which can’t be entirely merged.(vi)