Quartet Research and Performance Project 2004 – 2007
Report on the Quartet Project (February 2007)
Dr Caterina Albano
Curator and Research Fellow Artakt @ Central Saint Martins Central Saint Martins College of Art and Design Quartet: Musical Moves, A real-time exploration that plays across the senses of the human body (Feb. 07)
Evaluation of the historical, art/science components In the historical setting of the Great Hall at St Bartholomew Hospital, Quartet brings together a violinist whose movements and auditory perception as she plays are translated through sensors in digital data and sound that inform the movements of a virtual female figure; and a dancer who controls a robot that replicates her movements through the sensor transmission of locomotion data. Both the use of robotics, virtual components and the human body are well-established elements in art practice and live performance. Most notably, the work of the Australian artist Stelarc focuses on the relationship between the artist’s body and technology incorporating in his works medical imagining, prosthetics, robotics, virtual systems and internet interaction creating meta-bodies that investigate the conditioning and external prompting of bodily functioning. The integration of virtual environments in dance performances is also a growing area, as testified by Metapolis -Project 972 (2000), a collaboration between Charleroi Dance Company, the choreographer Frédéric Flaman, the architect Zaha Hadid and digital technologists to explore the interrelation of body, movement, real and virtual spaces. Quartet locates itself at the interface of different art practices in the exploration of the relations between real and artificial bodies, and live experience of interactivity for the audience. The artist Margie Medlin consistently explores in her work the relation between dance and imaging, and her work Miss World using a virtual dancer was the origin of the current collaborative project as a development of its interactive components in relation to real life performance.
Historically, already in the Renaissance, the study of human anatomy led to the conception of life-like models and automata. Notably, Leonardo da Vinci sketched a robot based on the structure of an armour to be set in motion by internal mechanisms (1495 c.), Fabricius ab Aquapendente conceived a mechanical body (exoskeleton) also based on a medieval armour as illustrated in his Opera Chirurgica (1620) and Ambroise Paré similarly developed artificial limbs, precursors of modern prosthesis (Les Oevres, 1575). These early modern attempts to translate the anatomical and physiological knowledge of the time in mechanical simulacra of the human body is telling of an ongoing interest in exploring the physical boundaries between the real and the artificial, the human and the machine. Three-dimensional anatomical models added to the construction of life-like figures. Indeed contemporary virtual figures, as in the case of Quartet’s female virtual dancer, bear strong aesthetic and conceptual similarities to 18th century wax models. In comparable ways, virtual design relies on a transposition of real human proportions on an ideal grid of bodily relations to draw the silhouette of the virtual figure, often based on real models, that confirm bodily cannons and gender stereotypes. The external surface of the virtual figure is smooth and shiny, as that of earlier models. The layering of internal muscular and skeletal structures, as suggested by the virtual dancer in the section “Framebreak Solo virtual dance”, is also visually reminiscent of early modern and modern anatomical representations, rather than of more recent X-rays and digital imaging. It is worth underlying that the field of body imaging is one of ongoing interface between art, design and science, and draws attention to the visual connotations that inform choices in medical images as well as in art and design. The visual choices made for Quartet’s virtual dancer implicitly alerts us to broader issues related to the making of images in art as well as in medical science through new technologies, of modes of reflecting and departing from a visual tradition and cultural standards of representations of the human body. Early anatomical images and models, however – unlike the contemporary virtual counterparts – were fixed in a pose that only hinted to motion and sensory capacities.
Not unlike today’s digital technologies, the addiction in the 18th century of movement to artificial figures did not happen in the field of medicine as such, rather in that of technology in the creation of mechanic automata for entertainment. These were also based on the careful study of the human body with movable mechanisms that corresponded to its physiological structure. Worth mentioning in relation to Quartet because of the musical content are Jacques de Vaucanson music players (1730s), including a flute player whose sound was obtained by varying the amount of air blown into the flute controlled through a moveable metal tongue, a mandolin player that taps the foot while playing, and a piano player that moves the head and simulates the breath; and Pierre and Henri Louis Jacquet-Droz organ player (1780s), a female figure that simulates breath, gaze direction, and movement of the hands.
Although 18th century, anthropomorphised and clothed mechanic figures of minute proportions are far removed from contemporary digital figures and metallic robotic structures – as the spine-like one in Quartet – analogies can be drawn on the attempts of imitating bodily movements through mechanic actions by reading the body in terms of physics; mechanic systems in the 18th century related to network systems in the 21st. This coincided, in the 18th century, with a growing interest in the nervous system and the brain in relation to sensation, with a focus on the interaction between the nerves and the senses in terms of perception and emotional responses that echoes contemporary analogies in the neuro-physiological understanding of the brain’s sensory capacity as a communicating network system. The replication of biomechanism of 18th century automata resonates with Quartet’s central tenet of exploring bodily movement through sensory perception in real time producing an interactive sensory and emotional landscape for the audience. Perhaps, even beyond of the expectations or aims of the artists involved, Quartet reflects current theories on the constant integration of sensory data, neuronal transmission, and movements, also as involuntary imitation, external prompting and conditioning, thus creating what may be described as a synesthetic experience for the audience.
Using the structural relationship of the quartet, Quartet develops in three movements that variously explore the interaction of the player and dancers (including the robot) through solos, duets, trios and the final coming together of the whole quartet. Central to the piece are the modes of interaction between the members of the quartet, and the audience. The well-choreographed duets between the dancer and the robot unfold through an almost eerie closeness as the robot’s structure acquires visual plasticity according to the dancer. A camera on the robot also captures the visual field of the dancer projecting it on a screen. This creates a subtle reciprocity of growing intensity during the performance conflating movement and points of views, the subject and objects of sensory perception, that ultimately converge in the visual and acoustic experience of the audience. The sound also resulting from the transmitted data of the violinist’s movements as she plays, convey the sensory qualities of the physical processes of making and hearing sound. Improvisation adds to the interactive unfolding of the performance. On the whole, Quartet creates a visual and acoustic environment, which not only enhances the senses but most interesting renders us aware of the sensory feeling that pervades any motion. Sensory and digital technologies have the potential of expanding normal sensory experiences enabling us to feel what is like to see, what is like to hear, what is like to move. The synaesthetic context developed by Quartet enabled the exploration of such experience almost suggesting a poetic of technology, and vision of the inherent process of sensing. This is a complex and rich field for original collaborations in the arts, technology and bioscience, with potential for broader developments for Quartet itself.
Report on the Quartet Project (February 2007)
Simon Emmerson
Professor of Music, Technology and Innovation Faculty of Humanities De Montfort University
Background remarks
The term ‘interaction’ is one which is much used but little understood. A musician routinely ‘acts’ with other musicians in the course of performance. They ‘read’ the actions and reactions of fellow musicians and adjust their responses accordingly. This is extended to ‘reacting’ to audiences, venue, acoustic of the space. In more improvised musical genres this becomes more substantive in that reactions become more unpredictable; new material may be created which has a direct influence (unmediated by any written ‘score’) and the musician may be expected to respond directly. This crosses an unclear threshold to true interactivity. The borderline between ‘mutual influence’ and ‘interactivity’ is clearly not fixed.
Musicians know and deeply value this feeling of mutual influence. The problem is that an audience may not. With the advent of technological applications to sound and music production the source of the sound – the performative gesture – may (and usually does) become dislocated from the sound as heard (usually from a loudspeaker). In the first 20 years of live computer music in which ideas of interaction have played a substantial part, much work has been done which has played scant attention to this dislocation: the performers may believe they have created human computer interaction but this is seldom perceived by the audience.
It is evident both from declared intentions that this project seeks as a core objective to address this issue: how to (re)establish perceptible relationships between the performance elements. Furthermore the aim is to do this at the material level – that is not just performers influencing ‘interpretation’ but that musical material is actually determined through the analysis of sound and body gesture in live performance.
There is a second element of core concern in this project which is easy to take for granted. While possible before the advent of technological mediation, the technology allows and encourages the interaction of different art forms to a much greater extent. The addition of a dance component which proactively influences musical material can be substantially enhanced using technological mediation. A dance movement and a violinist’s movement may be tracked by technology to produce a similar kind of ‘control information’ – although the tracking mechanisms may, for sure, be specific to the type of body movement involved. In this project the technology seemed to be unobtrusive and did not inhibit in any way the performers’ actions – in earlier technologies this has not always been the case.
The identity of the work
Comparing the performance I attended (15th February) with the video (made on the 17th February) the identity of the work was clearly defined: while divided into three ‘movements’, each was itself divided into a number of sections – effectively small ‘scenes’. A series of solos, duos and ensembles presented nearly every possible combination of musician, dancer, robot and animation. Without describing each in detail we can assess their general characteristics; the whole was coherent and it is a paradoxical tribute to the structuring of the show that the division of the whole according to the programme note was not always clear in the result, there were no obvious hitches or hiatuses and the whole moved steadily without disjunction. While the individual scenes were clearly differentiated with respect to the forces at work the overall narrative continuity of the show was maintained. Although using the term narrative there was no ‘story’ line inherent in the sequence. There were moments of humour, emotional intensity (especially between human and non-human elements) and lyrical expressionism (sometimes bordering on the melancholic). The use of the technology was – in this sense – quite remote from the emotional disengagement of ‘hard line modernism’. The electronic vistas created were sometimes stark and hard edged but always spoke of a feeling or state of being as much as a ‘structure or form’.
Sound world – sources
The musical character of each section was defined by the type of performance gesture which in general also closely defined the sound type. These followed the classic series of ‘progressive remoteness’ away from what we could directly perceive to be the source of the sound:
Violin or voice produced (clear origin); Probably violin or voice produced (surmised) – processed electronically; Probably not instrumental or vocal (or at least so highly processed we would never know).
Stevie Wishart balanced these sound types with flair and sensitivity; she retained a strongly melodic approach in many sections; from modal to ‘late expressionist’. It is clear that she often deliberately triggers ‘subversive’ material which counters this initial melody and sets up quite stark contrasts within the musical flow. The vocal material contrasted between the two performances I had access to which gave an interesting insight into the degree of definition of material types she had set herself in the plans for the work.
Gesture world – controls
Generally the relationship of gesture to result was commendably clear. The relationships covered all possible combinations of physical gesture, sound quality (analysed), real dance, animated dance, robot gesture movement.
Focusing on two which involved music and the gestures of its production:
Physical gesture (Wishart) to sound: this was the clearest to perceive as a member of the audience and clearly reengaged the pre-technological physical world of physical gesture as sound cause; Sound gesture (that is the envelope of the sound quality) to live dance, animated dance, robot: this is a more intimate relationship, movement is not dictated by the sound but the dancers (real or animated) ‘respond’. This response was inevitably more subtle with respect to the human. The ‘expressivity’ of the animated dancer was somewhat limited to limb movement and overall body shape. Future developments of this software will need to embody some very advanced concepts of muscle tension, posture and the like to capture this advanced body expression. (For example, that of the athlete ‘on the blocks’, stationary yet we sense the stored energy ready to spring and the attendant emotional tension.)
Questions raised and issues to be developed
The development of such a rich system of audio visual dance interactivity has made a very real contribution to this field both artistically and technically (the two cannot be separated in this context). The refining of questions for research (in the broadest sense to include creation and performance) is a key function of research itself:
Complexity and Polyphony
This project focused on the interaction of soloists – single ‘voices’. Technology can produce polyphony from these with ease but risks losing the ‘connection’ of gesture to sound which is key to the approach of this project. Nonetheless could systems involving more human performers be developed?
Causality and time delay
This project commendably developed a sensible relationship with what is known in sound for animation as ‘Mickey mousing’ – the crude coordination of ‘cause and (sound) effect’. Perhaps influenced by this background there has been much live electronic music which has positively tried to obscure cause/effect relations. This can lead to much audience alienation – the listener has no idea what action (if any) of a performer produces a particular result. This performance in re-engaging clear cause/effect relationships suggests there is more to be done to explore and exploit cause/effect delays (longer term memory) and how these are perceived.
Causality, interactivity, predictability
Especially in improvised musics there are issues of predictability versus surprise. In this show truly surprising sounds and music were actually quite rare (the occasional ‘subversive’ sound referred to above). This is ‘not surprising’ if a model of interactivity takes a close adherence to literal cause/effect – a present result is bound to be based on something which has recently occurred. Throwing the unexpected into interactive performance will be an interesting investigation.
Conclusion
The project is a major achievement and a substantial success. While the questions asked have been around for a decade this is one of the first really focused attempts to bring these worlds together. In engaging the world of movement and dance, primal gesture/movement types have been used to ensure that the audience sees and hears cause/effect relations and hence interactivity more clearly. Any shortcomings (for example in the expressivity of the real-time animation) were clearly those of a technique at the earliest stage of its development. The technology enabled an extraordinary range of music performer-sound interactions. While the sound world was not in itself extended from that previously available, the audience could easily apprehend a substantial increase in performer control over that environment.
Quartet – Evaluation in the context of New Media and Electronic Arts
Susanne Ackers, Executive Director of Hartware MedienKunstVerein, Dortmund, Germany.
Quartet has been in development since 2003 and is based on an earlier work by Margie Medlin “Miss World” (2001-2003), which can be discussed in the context of virtual and augmented reality. While the notion of virtual reality in the 1990s included the understanding of creating cyberspace as a parallel world to physical space (as in Miss World’s Virtual Dancer and her placement on one of the three single screens), a new approach to this notion is developed in Quartet; the augmented space of the stage in front of an audience. The term augmented reality, as virtual reality, was coined in the late 1980s/early 1990s and is based on the understanding of the physical world, which is augmented with digital data and technology. Our daily life today is more and more based on mobile technology that provides us with information transmitted through the air in a digital form that is not perceivable with the human sensory organs. This development is changing our perception of the world. Through Quartet Margie Medlin took an artistic approach in creating complex new tools that can be used by humans/artists living in this augmented reality. The resulting 60-minute performance presented to the audience in real time a highly aesthetical rendering of Margie Medlin’s interpretation of our world as an augmented space.
While “Miss World“ already described the two different spaces of virtual and augmented and their links as a three projection installation, Quartet enfolds a complex web of different connections in real-time between the place of the performance, the human actors on the stage (dancer and musician), the robot on the stage and the world of the virtual dancer. Thus bringing the virtual dancer into communication with the outside, the physical world, respecting the existence of this figure as a given fact, as an invention from the 1990s, and as an individual and equal partner during the performance. In a similar way, the robot on the stage is presented as a character in its own right.
The virtual and the real are represented in Quartet through different elements. The physical stage opens up through two screens to the virtual dancers’ world, which at some point during the performance is flooded with the moving image material of the physical stage and the theatre hall with the audience captured by the robot camera of the robot. In this way a visual bridge is set up, inviting the audience from the physical space into the virtual dancers’ cyberspace.
Next to the locative aspect and the mapping of the two spaces, through a number of complex relations, the two spaces are connected through the movements of the virtual dancer and its visible analogies in the movement to the musician, the dancer and the robot on stage.
The physical space is limited by natural laws, as the real dancer, the musician and the robot show on stage during the performance – each player with its own temporary spatial limits. The cyberspace, and in it the virtual dancer, underlies other laws, those of software and hardware limits. This is not an exciting insight. The new aspect that is covered and put on stage by Quartet lies in the visualization of a common language that the cyber-creature, the virtual dancer, the robot with its camera eye, the dancer and the musician establish and learn over the process of the performance. To my knowledge no other performance in this area has reached such a precise research and presentation of interactions between technology and human.
The title “Quartet“ points towards the use of technology in various combinations, as does the line up of the different sections on the programme. The four instruments: the dancer, the robot, the musician and the virtual dancer play in different combinations together. The duet between the dancer and the robot is visually easily perceivable and rewarding for the audience as it is clear most of the time that the robot follows the dancer in its movements. An interesting connection between spatial analogies is evident when the camera eye of the robot is projected onto the screen, seemingly having a conscious will to recognize the dancer. On the other hand, the wish of the dancer to understand its own existence is perceptible through its play with the machine.
The duet between the musician and the virtual dancer is more complex and perceivable only partly on a visual level. The audiovisual connection becomes visible only during certain sections of the performance. This is partly due to the fact that there is a double layer of data creation in the dancer; audio and spatial. Quartet can also be seen in the context of the discussion of immersion. Again, while immersion usually describes a human’s immersion into technology, here the focus lies on immersion in several ways, not only in terms of the audiences’ immersion into the performance of the interaction, but also through the theme of immersion evident in the interaction itself. On the one hand through the immersion of the virtual dancer into the physical stage space through the fact that “they” dance together, and on the other hand, the musician and real dancer are immersed into the virtual dancer’s space through the fact that she copies movements that are given to her remotely. In relation to the augmented space, Quartet immerses the virtual dancer and the programmed robot into the physical space of the audience. Technically speaking, in Movement 3: Personal Space Duet dance with robot camera the dancer controls the movements of the robot, metaphorically, the dancer is respecting the robots restrictive moving possibilities and like this integrates the robot into human physical space.
Although Translation Duet dance and violin improvisation is an interaction between the musician and the dancer without the use of technology, the language that is used, both body and musical, relate to the other sections of the performance. As a summary of the whole set up of the performance, broken down in three movements, it can be said that a language between the four players is established over time. Starting by introducing the dancer and the robot with its camera eye and continuing with a first meeting between the musician and the dancer with the virtual dancer in Movement 1: Phasing, in Movement 2: Breath Song different player present themselves in their spaces and in their languages with their own characteristics. The virtual dancer is shown in a sort of “awakening” when it first appears; a memory towards the human dream of creating a mechanical human is touched in the audience – a avatar with its own history, its own character, maybe even with its own form of consciousness.
The development of a common body language based on visual figures and musical themes is the result of the artistic competence of Margie Medlin, who’s skills as a highly knowledgeable visual artist, programmer and manager of a team of specialists, are evident in the staging of this complex performance of virtual and augmented space onto a stage in real time. The tools developed – the figure of the virtual dancer, the robot with its camera, the electronically enhanced violin – serve as part of the artistic work, as they were developed to enable the final real time performance. Thus, Margie Medlin is not only to be seen as a visual artist, but also as a stage director and producer, bringing together specialists in the field of robot construction and programming, music, dance, creation of a virtual body and its movements and an overall programming of all different inputs in real time. The artistic achievement of this performance lies in the complex staging of technologically based live interaction between live and virtual dancers providing an interpretation of the augmented space.
In terms of classical genres of art, Quartet combines dance, music and visual art. Dance as a spatial art, measures movement in space by creating three-dimensional data through motion tracking technology.
Music, as one of the arts liberals, is based on relations between pitches. In Quartet, new possibilities of creating useful data have been developed; loudness and spatial movements connected to playing the violin.
Visual art, through the invention of central perspective in the Renaissance, is also based on numbers: there are two levels of visualization: programming the virtual dancer and the camera imagery. Quartet combines all of these genres and is based on the real time event of the digital quadriga which in itself is based on numbers – on the material of the augmented space. The mapping of cyberspace and real space frees the Virtual Dancer out of the cyberspace of the 1990s and offers her an entrance into our physical world of augmented reality in the 21st
century – an artistic masterpiece perceivable for a public that is interested in understanding the world we are living in within 60 minutes of highly artistic employment of technology in a context based on the history of art.
Unsited 