Eurographics Word Template for CAe 2011 Computational Aesthetics in Graphics, Visualization, and Imaging (2012) D. Cunningham and D. House (Editors) Integrarte: digital art using body interaction B. P. Castro1,2, L. Velho1 and D. Kosminsky2 1 Instituto Nacional de Matemática Pura e Aplicada, Brazil 2 Escola de Belas Artes, UFRJ, Brazil Abstract This paper emphasizes the participation of the whole body in the interaction process in art, which becomes essen- tial in the development of new insights and artistic expressions. We approach the postmodernism and the post-me - dium condition, having in mind the remediation process between digital and analog media to rethink the possibilit - ies of interactive art through new technologies. We focus in the body as a living medium, to introduce the INTEG- RARTE project, an installation about body experience through movement visualizations and sounds. Categories and Subject Descriptors (according to ACM CCS): I.3.6 [Computer Graphics]: Methodology and Tech - niques - Interaction Techniques 1. Introduction This paper presents the INTEGRARTE project, a theoretical and practical research that reflects on the possibility of interactivity as an artistic experience. The relationship between art and technology is explored throughout the article as a result of an ongoing process of hybridization in art history, which can be detected even in the early twentieth century. After giving a brief historical background, we describe a stage of development of the INTEGRARTE installation, which seeks to bring the public to an artistic body experience through audio-visual stimuli. A inclusion of the whole body within the interaction process becomes essential to establish a denser connection between the public and the artwork during the aesthetic experience. This project proposes a form of integration between body, art and media. 2. Art History Background Since the end of the nineteenth century until the 1980s, the art scenario was marked by several vanguards, such as Cubism, Futurism, Suprematism, Surrealism. Each one sought a cutting edge artistic ideal, usually making detailed statements in written manifestos. The art production of this period, in general, is called modern art. In that period, there was a constant attempt to develop a new language that would take over the previous one. It was a period of artistic effervescence and abundant production all over the world. Questioning the art nature was inevitable. One of these artist groups, the 'Dadaists', were responsible for initiating a process of contesting art essence. Marcel Duchamp was essential in discontinuing with the traditional values of art. He inserted everyday objects such as a urinal or a bottle rack in the art gallery. This act brought up a number of issues to the art legitimacy system. This raised the major question: what could be con- sidered art? If art was really ubiquitous to our lives, we wouldn't need a restricted institution to legitimize neither the artwork nor the artist. This last point is critical to understanding the develop- ment of what is often called postmodernism. "A period of quite perfect freedom. Today there is no longer any pale of history. Everything is permitted." [Art06]. This is some- thing completely opposite to modernist posture. The mod- ernist approach is commonly exemplified with Clement Greenberg's thoughts. Greenberg was an art critic that sup- ported Abstract Expressionism. He consistently defended the 'purity' of Jackson Pollock's works and his 'ac- tion-painting'. Pollock's paintings explored the superficial- ity and flatness of the picture, which Greenberg considered the essence of the media, the specificity of painting. One of the greatest difference between modernism and postmodernism is their posture towards past production. While modernists try to deny the previous art production and are segmented into several distinct groups, postmod- © The Eurographics Association 2011. Integrarte: digital art using body interaction B. Castro, L. Velho and D. Kosminsky ernists care less about these rules, and include the possibil- ity of absorbing the past into new creative process. Also, the distinction between different media become less relev- ant as the figurative art loses its dominance. 2.1 The post-medium condition and digital art In 1960 Frank Stella developed the 'Black Paintings' series, in which he explored the physical materiality of paint and therefore showed the painting as a three- dimensional object like any other. For Rosalind Krauss [Ros00] this work extinguishes the difference between painting and sculpture and is one of the first manifestations of what the author will call post-medium condition. This condition is directly related to post-modernism, and the development of conceptual art, in which arts are liberated from a specific media categorization. From the 1960s, three-dimensional painting is explored by various artists. They turn into sculptures, objects, installations. The distinction between specific forms of art not only loses its relevance, but becomes a limitation to artistic creation. "If the hybridization process is a guiding element of the twentieth century aesthetics, either by the mix of media separated before, either by public participation, in the digital art this process becomes even more complex" [Pri05] Definitely, the digital technology development makes this division even more delicate. The process of hybridization is inherent to computers. This provides integration of different forms of expression as still images, film, sound and text. All of these expressions are assembled in a unique digital media made possible by one single language: the binary code. However, it is interesting to note that even after the post- medium condition that made the specificity of each medium more undefined, there is a process of 'remediation' of analog media by digital media. According to McLuhan, "in the name of “progress”, our official culture is striving to force the new media to do the work of the old” [MF11]. This can be seen at a basic level of computer interfaces that are based on office objects: folder, the magnifying glass, the file, the recycle bin. Also seen at image editing software, in which the tools are the paintbrush, pencil, rubber, etc., and even in their accessories such as pen tablets in which the instrument used for image creation and manipulation has the exact same shape as the analog media. “The digital medium wants to erase itself, so that the viewer stands in the same relationship to the content as she would if she were confronting the original medium.” [BG00] When the digital media tries to 'erase itself', it isn't main- taining only the same relationship with the user through the process. There is also a strong continuity in the form of production and creation result. The use of new media for new forms of expression is limited because we maintain the same creation habits. Since we can try to break with the hand gesture habit in- herited from drawing and painting, why not to try other gestures? Other body parts may also be associated with visual arts creation. We can develop expressions that integ- rate the whole body, its image, its sound and its volume with external images and sounds. 3 New media artistic approach Aware of this remediation process, we developed interactive experiments that were distinguishable from the way we use tools like a paintbrush or a pencil. These experiments use different parts of the body and the relationships between them to create real time drawings from a complex vision of the human movement. We approach the concept of 'body as living media', developed by Hans Belting. For the author, our body is a vital agent in connection with the images because it is responsible for perception, projection and transformation of these, i.e., the individual human subjectivity is inherent in the process of image mediation. "Bodies perform images (of themselves or even against themselves) as much as they perceive external images. In this double sense, they are a living medium that transcends the capacity of their prosthetic media."[Han05] Actually, when we use sensors to help the computer recognize our body movement, our body is nothing but another big mass of data interpreted by the machine. We may say that the computer sees us just like pictures, sounds and texts but organized in a different way. In our relationship with the computer the sense of sight is usually emphasized and we often hear discussions about visual media, but these boundaries aren't that clear anymore. All media are mixed and that perception is never through one sense only. We can recall the study of the "sense ratios" of McLuhan, in which the author studies the different proportions of the the presence of senses in each media. “Media, by altering the environment, evoke in us unique ratios of sense perceptions. The extension of any one sense alters the way we think and act – the way we perceive the world. When these ratios change, men change.” [MF11] Given this multi-sensorial vision of the possibilities of hybridization focusing on digital technology, sound was added to our proposition so that the public could also re- ceive a sonorous answer to their movement. Therefore, we propose an interactive audible and visual installation that suggests an alternative body awareness, a perception ac- quired through these stimulus. The installation was de- veloped using existing technologies in a creative artistic way, looking to extend horizons in their possibilities. 3.1 The INTEGRARTE Framework For executing the installation we set up a framework composed of a body sensing and data mapping system, and the artistic output. The framework, as seen in Figure 1, consists of a sensor, a computer, a projector and an audio © The Eurographics Association 2011. Computational Aesthetics in Graphics, Visualization, and Imaging (2012) D. Cunningham and D. House (Editors) system. With this framework we can track the human movement in real time, get audio input, process those, display images and amplify the audio. We have chosen to use Microsoft Kinect because it is an affordable technology that met our prerequisites. The Microsoft Kinect is a 3d camera developed for the video game Xbox 360. The sensor includes RGB camera and an Infrared camera. From the images combined, it is possible to distinguish the human movement in the area, so we can track the skeleton in real time. To access the algorithm that accomplishes this, we used the Simple OpenNI wrapper for Processing. This allows us to use the body data to map into other expressions such as sounds and drawings. Processing has its own programming language (Java based) and is designed for artists and designers. This software uses OpenGL to render the images generated from the body data. In addition, we also used Ableton Live to work on the sound design. The use of the Kinect allows the integration of the entire body in visual arts creation. It also enables a new way to approach the public in the experience with interactive in - stallations. The Kinect reads the movement of the body without the use of infrared markers. Thus, the Kinect dif- fers from previous optical body reading systems because it eliminates the need to attach objects (infrared markers) to the public's body. The interaction is liberated from a previ - ous public's preparation for the experience, the system can recognize any body that is in the field-of-view of the cam- era. This Kinect body recognition is also distinguished from previous image processing system that could only capture the body outline. This contribution comes from the algorithm that allows skeleton tracking from some key joints. Figure 1: The INTEGRARTE Framework As we could see, Kinect provides some possibilities that profoundly change the way of interacting with interactive content in installations. We chose to use Kinect in INTEG- RARTE because of these capabilities that enable the body perception by a intuitive and spontaneous reading of the movements, contributing to the public aesthetic experience in an undeniable way. 3.2 INTEGRARTE Experiments We developed four kinds of interactions; each one con- trolled by s different body part or by relationships between body parts. The framework allows us to read the movement from specific points of the body. The skeleton tracking uses head, neck, shoulders, elbows, hands, hips, knees and feet as joints. 3.2.1 Wave. In the experiment 'WAVE', a strip of blue light was created from the arms of the interactor. The wave was composed of five lines with different levels of opacity. The weight of the lines was also updated according to the proximity of the camera. The sound developed for this ex- periment was repeated like a mantra, chanting “WAVE, WAVE, WAVE …” The repetition of the word 'WAVE' (in English) gradually merged with the word 'VOEI' (which means 'I flew' in Portuguese), alternating between the visu- al experience of a blue wave, and the body experience of moving your arms like a bird. The volume and the echo of this mantra was influenced by the height of the arms. To complete this experience, a wind whistle could be heard in accordance with the approach of the public. Figure 2: Wave © The Eurographics Association 2011. Integrarte: digital art using body interaction B. Castro, L. Velho and D. Kosminsky 3.2.2 Eco. In this sketch the visual content was formed as a smooth abstraction of the interactor's upper body sil- houette. An organic shape was created from the head, hands and torso. It was interesting that the body was easier to recognize when the arms were open. At many times the twisted body formed a non-convex shape. In this experi- ment the flow of the movement was beautifully visualized from semi-transparent layers of different colors. The sound of this experiment was developed from the definition of several small fields: for each one of these areas we recor- ded a tone of the vowel 'O'. These recordings integrated a flexible texture of sounds that kept playing according to the interactor's body. The sound interaction of this sketch was not very intuitive. There were too many layers of sound, making it difficult to distinguish the relationship of each body part to each field, but despite of this complexity the final sonority was very rich. Figure 3: Eco 3.2.3 Circle Field. This sketch generated circles at different scales according to the distance between the two hands. We defined three sound patterns that we adapted to the circle sizes. The frequency of the sound pattern de- creased as the circle was bigger. This caused the impres- sion of a tighter field, saturated with sound, contrasting with a wider one, in which the sound dissipated, a simple and intuitive interaction. Figure 4: Circle Field 3.2.4 Graf. This sketch used only one hand to create series of squares. The level of transparency and size were mapped to the z axis and the color was determined by the horizontal position. In this sketch the integration of the whole body was limited to sound interaction. The accelera- tion of the movement determined the pitch. The faster the movements, the higher the pitch. The slower the move- ments, the slower the pitch. Figure 5: Graf 3.3 Evaluation and Final Thoughts After developing the first experiments, we had the op - portunity to expose them at the International Festival of Di- gital Culture which gathered 6,000 people over three days at the Museum of Modern Art in Rio de Janeiro. INTEG- RARTE was one of the sixteen projects selected to be ex- posed for one day in the category Visuality. The recognition process of the method of interaction led the public to explore the body through sonorous and visual stimuli. The installation alternated between interaction styles, surprising the interactors that should try out various movements to discover which part of the body was in con- trol. Also, in each style, sound and visual interactions re- acted to different actions, requiring them to explore the whole body to understand the complete process. The advantage of exhibiting four experiments that included different forms of interaction was to compare the public's reaction to different levels of user-friendliness. In addressing the participants informally, the most suc- cessful interaction was 'Circle Field'. This sketch used a variation of a simple body ability - to delimit a space with your hands. This type of movement is easily controlled and does not require a complex level of body coordination. The noticeable association between visual and audible interac - tion also helped the public to understand this experiment. So even though this sketch already brings a new element in terms of interaction, it remains simple and intuitive. 'Graf' was the least mentioned sketch in the informal ap - proach of the public. Perhaps because it has the most obvi- ous and common interaction, making this sketch the less © The Eurographics Association 2011. Computational Aesthetics in Graphics, Visualization, and Imaging (2012) D. Cunningham and D. House (Editors) memorable. The sketches 'WAVE' and 'Eco' demanded a greater willingness for interaction and some body coordina- tion and although aesthetically attractive they were not very explored by the public. This may be due to the Festival's spatial arrangement. The installation took place next to an- other Festival activity. Many people were observing who was interacting and this may have inhibited the parti- cipants, especially those who do not usually explore their body movements, or are naturally shy. Analyzing the feedback from the public about the differ- ent sketches, we can notice that there is public interest in a more engaging experience. This can be seen through the lack of interest in the sketch that used only one hand, the body part commonly associated with visual creations. Nev- ertheless, we realize that there is an average limit of open- ness of the participants to explore the experience proposed. This will be checked in the next opportunity to exhibit the installation,: it will be done in a way that the participant is less exposed, so they can be more spontaneous. We em- phasize that these experiments were limited to the use of the upper body and the research foresees experiments that use legs as well. Moreover, we can make a more specific comment on this Festival. The Festival had a predominant presence of artists from different profiles. It was clear that the background of each person influenced his/her form of interaction. The musician took over the interaction with sound, while visual artists and designers were more interested in the drawings they were creating with their bodies. Despite the simultaneity of sound and visual interaction, visual interaction usually overpowers sound interaction, but the music plays a vital role on creating the experience within the immersive environment. These experiments are the beginning of a research on interactive installations and public engagement. From these considerations and results we manage to integrate different forms of human expressions through digital media. Using the evaluation made during this festival, our next step is to provide a meeting between the public’s and the artist’s bodies through virtual instances of the artist’s body. These instances, a kind of digital presence of the artist's body, will be made in the form of images, data and voice. Further information about this will be presented in the future. Interactive installations can bring together different art forms and provide the participant with a complete artistic experience. The body awareness and the visual and auditive experiences are enhanced through time and space interactively. This project is a practical experiment on some theoretical reflections about the inclusion of art in the digital universe, which can have numerous unfoldings. We believe that the possibilities of art are limitless, and that digital technology only enhances this infinite. Our biggest challenge is to en- courage people to explore the possibilities of new media and, in doing so, find new ways to create and perceive art and the world around us. References: [Art06] ARTHUR D.: Após o Fim da Arte. Odysseus, 2006. [Ros00] ROSALIND K.: A Voyage on the North Sea: Art in the Age of the Post-Medium Condi- tion. Thames & Hudson, 2000. [Pri05] PRISCILA A.: @rte e mídia – perspectivas da estética digital. Editora Senac São Paulo, 2005. [MF11] MCLUHAN M., FIORE Q..: The medium is the massage: an inventory of effects. Gingko Press, 2011. (First Published at 1967) [BG00] BOLTER, D., GRUSIN, R.: Immediacy, Hyper- mediacy, and Remediation In: Remediation. Understanding New Media.: MIT Press, 2000. [Han05] HANS. B.: Image, medium, body: A new ap- proach to iconology. In: Critical Inquiry 31. The University of Chicago Press: 2005 © The Eurographics Association 2011.