Panel: Where is the body in Code?

Sofian Audry, Aesthetics of Adaptive Behaviors in Agent-based Art

Since the post-war era, a range of artists have used embodied, artificial agents in media installations. Their work runs in parallel with scientific research in domains associated with Computer Science, such as Cybernetics, Artificial Intelligence (AI) and Artificial Life (AL). Two important concepts are central to these scientific approaches: emergence — the mechanism whereby higher-order forms or processes emanate from the complex interactions of lower-order units — and adaptation — the real-time adjustment of a machine to achieve a better performance in its environment. Recent advances in AI are largely attributable to major breakthroughs in the fields of Artificial Neural Networks and Machine Learning, both of which feed upon these two core ideas. But while notions of adaptation and learning are an extremely important part of that research, artists and media theorists working with agent-based systems have widely ignored them, often in favor of emergence and self-organization. Inasmuch as emergence offers a rich ground for art-making, adaptation offers an equally important, yet complementary dimension of it. To re-position adaptive systems within the theoretical and practical field of agent-based artworks, I examine (1) the historical context surrounding adaptive systems; (2) its relationship with emergence and self-organization; and (3) the aesthetic potential of Machine Learning algorithms by examining their intrinsic characteristics. Building upon that research, I propose an aesthetic framework for adaptive systems based on the morphological aspects of agent behaviors as they evolve through time, supported by examples from my own art practice.

Fabio Paolizzo, Enabling Embodied Analogies in Interactive Music Systems

When human beings make or listen to music, they regularly draw on analogies to other forms of knowledge and embody how they feel through their own body. Recent advances in interactive music systems brought a dramatic increase in human-centered and reflexive retrieval and indexing methods based on subjective concepts such as emotion, preference and aesthetics. However, listeners are able to change their semantic relations with the sonic world through functional adaptation at the level of sensing, acting and coordinating between action and perception, in biological, psychological, and cultural terms that involve motor, kinesthetic, haptic and visual, besides the purely auditory components. Still, current interactive music systems have no “embeddedness” in a world beyond what they immediately do. Implementing some understanding of how we feel and attribute meaning, when interacting with music technology, requires next-generation systems of interactive music and information retrieval to operate in terms that can represent human cognition as multimodal and embodied. Investigating how we translate between these domains, “making analogies”, is the aim of the Musical-Moods project: a mood-indexed database of scores, lyrics, musical excerpts, vector-based 3D animations, and dance video recordings.

The research is aimed at cross-modal machine learning and uses multidisciplinary tools and methods drawn from a broad range of disciplines, including music, musicology, dance, motion capture, human-computer interaction, computational linguistics and audio signal processing. Current activities involve: (1) adapting wisdom-of-the-crowd approaches to embodiment in music and dance performance to create a dataset of music and music lyrics that covers a variety of emotions, and (2) applying audio/language-informed machine learning techniques to that dataset to identify automatically the emotional content of the music and the lyrics.

John Seberger, Affordances and Mediation: The Disappearing Subject in Computerized Empiricism

James Gibson initially described affordances as relational possibilities mutually arising from the visual, biological, temporal, and spatial characteristics of animals and environments. For him, affordances bridged the subject/object dichotomy and were not exclusively resident in either category. Contemporary technologists, on the other hand, frequently define affordances as properties of apparatuses. In this paper, I discuss the impacts of this discursive mutation of ‘affordances’ in terms of the emergent and increasingly pervasive epistemic culture of computerization—one fundamentally grounded in the mediation of post-Enlightenment empiricism.

Beginning in the early 19th century, as agents of empiricism shifted their gaze to naturally imperceptible phenomena, direct observation was gradually replaced by mediated observation. In such mediated observation, the apparatuses increasingly definitive of empirical knowledge production took on characteristics of both subject and object, becoming chimeric. As empiricists’ imperceptible phenomena expanded to include topics of massive scale—the social, the informatic, the ecological—the affordances arising between subject and object came to be categorized as properties of the apparatus. As if by transference, the dyad of observer and observed became the lens; as if by amnesia, empirical knowledge production became ‘always already’ mediated.

Through an analysis of various apparatuses and related literature, I identify a set of possible ramifications of this discursive shift in ‘affordances.’ I argue that contemporary technologists’ tendency to locate affordances exclusively in apparatuses is a symptom of an emergent epistemic culture that risks the future invalidation of knowledge produced via the dynamic, unmediated relationships between subjects and objects.

Chris Salter, Technologies of Sense: Quantification, Embodiment and Making up People in the Age of the Data Driven Self

In “Making Up People,” philosopher of science Ian Hacking famously argued that the human sciences through counting, correlation and quantification “create new kinds of people that in a certain sense did not exist before.” Yet, unlike Hacking’s 19th century statistical models, our systems of calculation are based on algorithms, self-organized networks of smart sensor nodes and machine learning within the context of what Andrejevic (2014) has called “the sensor society” – “in which the interactive devices and applications that populate the digital information environment come to double as sensors” and […] “emerging practices of data use complicate and reconfigure received categories of privacy, surveillance, and even sense-making.” SImultaneously, our increased habitation within these new sensory environments produces a strange double bind where we are torn between different forms of subject-hood – becoming “new objects” of sensory inquiry (to update Hacking) – and different understandings of what human experience is and can be. One form relies on measures gathered by sensors that track biophysical data to provide a portrait of the self as “quantified,” and “conductable” (Foucault), while the other exploits new experiences of sensory-based, bodily-driven affects produced by encounters with multi-modal, real time haptic, visual, acoustic, olfactory and other technologies that organize what Hayek called “the sensory order” in previously unprecedented ways. From worn haptic devices to the tracking of sense modalities on bodies, such “technologies of the senses” are exploited to “make up” new people with heightened sensory awareness who, simultaneously, increasingly rely on numbers as forms of sensory truth and self knowledge. Using examples from recent art practice that critically examine this double bind, this talk explores the construction of a new form of “sensory reason” which only serves to heighten the age old tensions between mind and body, external calculation and felt, “phenomenal” data (Varela).