Name: Curatorial A(i)gents
Date: 2021
A metaLAB series of interactive projects exploring machine learning in, around, and about the Harvard Art Museums

Curatorial A(i)gents presents a series of machine-learning-based experiments with museum collections and data developed by members and affiliates of metaLAB (at) Harvard, a creative research group working in the networked arts and humanities.

Long before computers came to pervade every aspect of modern life, museums were collecting, organizing, and storing data. The art museum is a kind of vast machine for making all kinds of objects interoperable, from bronze-age figurines to Renaissance paintings to contemporary performance-art works. Like our digital machines, museums engender wonderful experiences—and they’re also engines of bias, power, and invisibility. The term “machine learning” represents a family of systems that use algorithms to find patterns in data inferentially, without explicit instructions. Artists and media makers are experimenting with machine-learning tools to create new kinds of artworks. But roles for machine learning in the art museum are still rare in practice. Presented in the Lightbox Gallery, metaLAB’s projects explore emerging possibilities for machine-learning systems while exploring vital issues at the intersection of technology and culture. Variously playful, analytic, and critical, metaLAB’s experiments use the museums’ own data to expressive ends. The names and dates of works and their makers; curatorial descriptions and histories of exhibitions; colors and dimensions; images of objects themselves—encountering such data as these, algorithms chart invisible relations, forge new connections, and breed monsters.

Some projects are playful: in Sympoietic Systems, Philipp Schmitt uses machine vision to connect paintings and prints with current weather conditions outside the museum; Dan Newman, Keith Hartwig and Kevin Brewster’s Aixquisite Corpse identifies body-like images in the collections, inviting visitors to construct their own strange hybrid beings; Dario Rodighiero restages Alan Turing’s “imitation game,” with computing pioneer Ada Lovelace as foil, in Surprise Machines. Other projects in the series are analytic: through algorithms and visualization, Lins Derry’s Second Look reveals vital assumptions about gender and sentiment at play in art history; in Watching Machines Loving Grace, designer Kim Albrecht critically reflects upon commercial facial recognition systems; and in A Flitting Atlas of the Human Gaze, metaLAB founder Jeffrey Schnapp, working with Dietmar Offenhuber, Todd Linkner, and Kevin Brewster, uses machine vision to chart the ocular politics—who is looking at whom—across the museum’s coin, print, photo, and painting collections. And some projects take a critical perspective on the ethical and technical problems of intelligent machines: riffing on the shopping systems found in streaming media and online markets, in This Recommendation System is Broken, Giulia Taurino plays with our museum-going expectations as consumers and viewers; in Ocean Amplitude, Francisco Alarcón (in collaboration with MIT’s Stefan Helmreich) infuses sublime experience with questions about the environmental costs of computation.

The gallery program for Curatorial A(i)gents is scheduled to run nine weeks beginning February 1, 2021. In the meantime, project teams continue to explore and refine their work, their learning deepening about relations between algorithms and curatorial enterprise. Emerging from that process, the program’s accompanying publication will be not a catalogue but a prologue, released before the show reaches the gallery. Reflecting emergent conversation rather than settled knowledge, this prologue will combine participants’ projects and reflections with critical writing from a range of experts and practitioners. Unfolding from pamphlet to poster, Chelsea Qiu’s ingenious design for the prologue publication features a lively “combinatory” structure that emphasizes dialogue among cross-cutting perspectives.

More than our schedule and publication strategy has changed, however. The combination of pandemic, economic and institutional distress, and cries for justice and equity in an election year, have challenged us to think about the precarity and resilience in our community. In a sense, the university is a collection—of courses, buildings, and research efforts; like the museum or the archive, the collective choices comprising it build a reality, even as they obscure alternative possibilities. As an added project, the Curatorial A(i)gents team is putting its tools in mapping, design, and analysis to work tracking the impact of emergent crises on the university as collection and community. Called Hindsight 2021, the project will visualize effects of pandemic and unrest on the curriculum and wellbeing. Conducted with a growing consortium of student researchers, the project will culminate in a suite of interactive visualizations and conversations, to be staged in the Harvard Art Museums in the Spring of 2021.

A collective headquartered at Harvard’s Berkman Klein Center for Internet and Society, metaLAB explores the digital arts and humanities through research, experimentation, tool building, teaching, through publications in print and online, and via exhibition, performance, and social practice. Here at the Harvard Art Museums, as with partners across the university, and in the world at large, metaLAB’s work infuses traditional modes of academic inquiry with an enterprising spirit of hacking, making, and creative research.


Second Look: Gender and Sentiment on Show Not unlike animal forms of intelligence, artificial intelligence relies upon pattern recognition. But its understanding of patterns is dependent upon rigid and rigorous categories predefined by human programmers and upon far more limited data inputs than those provided by the sensory apparatus of a sentient being. Second Look calls attention to the circularity of how an artificial intelligence “sees” and “knows” by asking it to infer gender and sentiment in paintings from the Harvard Art Museums. The result is a curatorial exercise that, as it exposes aspects of how gender has been interpreted by painters across time within the Harvard collections, showcases the powers and limitations of AI as an analytical tool.

Conventional AI tools and services like the ones on exhibit, there exist only eight emotions and two genders; conflicted emotions and interstitial gender identities exist only as degrees of uncertainty along the path to a definitive single identification. By making a show of how an AI “sees” and “knows,” Second Look seeks not only to probe how an AI’s predetermined codes filter and interpret the perception of emotion and binary gender identity but also how similar assumptions propagate cultural norms within contemporary society. (Lins Derry) Link coming soon

Ocean Amplification The shape of the world’s oceans is changing. According to a 2019 report in the journal Science, wind speed increases in the Earth’s southernmost oceans induced by ocean warming have led over the last thirty or so years to an .25% surge in the wave height of the largest 10 percent of waves. As such, the wave emerges as a key symbol of ocean transformation: of the material effects of climate change, alongside intensified storms, sea-level rise, and increasing temperatures. The present project explores visualizations and simulations of rising waves, read as avatars of a hybrid human-inhuman political ecology.

The installation is built on a GAN, a machine-learning algorithm that generates digital images of waves. These waves become increasingly realistic as the program is trained on archival images of waves. As the imagined AI network “learns,” it consumes ever more electricity. Employing the magnitude of its energy usage — the carbon footprint of the required calculations — as an input to grow the height of its virtual waves, the work is structured around a feedforward loop linking the electro-anthropogenic generation of climate-changed wave power to the added computational loads created by the need for scientific assessments of climate change itself. (Francisco Alarcon with Stefan Helmreich)

Aixquisite Corpse Images of the objects that museums collect are increasingly analyzed by algorithms. What is the relationship between algorithmic image analysis and time-honored traditions of human study, analysis, and curation? How might humans interact with algorithms to produce new modes of creative and curatorial expression? 

Aixquisite Corpse invites museum patrons to explore these questions by playing a variant of the surrealist game of Exquisite Corpse. According the game’s rules, players take turns drawing individual sections of a body and, as the drawing is passed along, it is folded so as to conceal previous sections. So players have no prior knowledge of the image their drawing connects to… much like an image recognition algorithm sifting through a training set.

In the case of Aixquisite Corpse, images from the Harvard Art Museum’s database have been analyzed, yielding automated descriptive metadata, as well as indications of the presence of human figures. The latter subset of images has been further parsed to extract heads, torsos, and legs, and their associated metadata, so as to then allow visitors to recompose the figures using this very metadata as a guide. The result is a machine for generating fantastic pairings of heads, torsos, and legs that expose the conjunctions and disjunctions between computer vision and human ways of making sense of the world. (Kevin Brewster, Keith Hartwig, Daniel Newman)

Surprise Machines Can machines think? In 1950 Alan Turing famously answered this question in the affirmative by means of a so-called “imitation game” in the course of which an examiner is asked to distinguish between humans and machines while communicating via a typewriter. He argued that, once the responses appear indistinguishable, the machine can be correctly be understood as engaged in thought. In an article entitled “Computing Machines and Intelligence,” Turing approached the subject of artificial intelligence from multiple perspectives, one of them inspired by the English mathematician Ada Lovelace. Here Lovelace figures as Turing’s foil, arguing that machines are incapable of thought because they are incapable of “tak[ing] us by surprise.” Turing counters by stating that machines are a frequent source of surprise, behaving unpredictably and, thereby, generating surprises.

Surprise Machines reprises Turing’s experiment some seven decades later by means of an AI-based curatorial experiment that relies upon “black box” algorithms whose behaviors, once set in motion, cannot be predicted by their programmers. It sets out to visualize the entire universe of Harvard Art Museum’s collections with the aim of opening up unexpected vistas on the more than 250,000 objects that make it up. (Dario Rodighiero)

Sympoietic System Expectations of artificial intelligence are typically drawn from expectations of ourselves as autonomous, thinking agents. However, humans are social as well as cognitive beings; they make worlds by interacting with one another, with objects, and with systems. Dempster and Haraway have called this phenomenon “sympoiesis,” or “making-together.” Here, the observed weather becomes the curator of an exhibition.

The project was developed for the Harvard Art Museums’ Lightbox Gallery, which is located between a glass roof above and a sweeping courtyard view below. Using weather data and a camera feed of the sky, a custom software continuously selects works from the museum’s collections by linking color between meteorological observations and image metadata. Artworks are presented on a digital display, and the color of the sky outside tints the naturally-lighted gallery. A contact microphone attached to the glass roof creates a soundscape, subtly meandering between indoors and out.

The casting of weather in the role of curator is political as well as art historical. The work alludes to a welter of world-making systems—climate change, computation, shipping and logistics—while also nodding to Cloud Music (1974). The latter work, produced by the trio of Robert Watts, David Behrman and Bob Diamond, generated a live music score by scanning the color of passing clouds. Both the present piece and its predecessor invoke chance operations and computational thinking, gesturing towards the complexity generated by the collision between man-made and natural systems. (Philipp Schmitt)

The Loving Grace of Machine Eyes is observing the otherwise unwanted parts of facial recognition. The project visualizes the media negativity of algorithmic visual sense-making within the Harvard Art Museums collection.

Computer vision is reductive by design. It proceeds by splicing rectangles out of images to determine age, gender, or facial expression, among others but removes the contextual framework to perform the task. For a human observer, the image of a smiling 24-year-old belly dancer and a smiling 24-year-old old soldier may appear sharply distinct. The algorithmic interpretation highlights the sameness of the two while removing its context. Through the advances of machine learning and computational capacities, machine vision moved from scientific endeavors into a broad spectrum of domains. Consumer goods such as Apple’s Face ID, aging apps, or Instagram filters shaped popular culture over the last decade. Similar algorithms are currently reshaping mass surveillance and the tracking of unwanted individuals. 

In the 21st century, it is not an all-encompassing god who tenderly watches over us but the »loving grace« of watching machines as Richard Brautigan imagines. The word computer has its origins in the Latin »putare« or »prune« its broader meaning is to reduce or remove something to get rid of unwanted parts. This project observes the unwanted parts of our watching machines. Link (Kim Albrecht)

This Recommendation System is Broken Imagine that you are gazing at Claude Monet’s The Gare Saint-Lazare: Arrival of a Train. A typical recommendation algorithm will suggest that you look at other Impressionist paintings: why not another work by Claude Monet, perhaps Red Boats, Argenteuil? It relies upon conventional, longstanding, consecrated associations: between artists, art movements, periods, schools, and subject matters. The present project turns this familiar matchmaking practice on its head, seeking to transport the viewer from center to periphery, from the gallery wall to deep in the storage closet, from the highly valued to the undervalued and rarely seen.

This recommendation system is, in a sense, broken, which is to say inaccurate. It does not respond to personal choices; it doesn’t collect data; nor does it attempt to interpolate preferences. Instead, it surfaces otherwise invisible artifacts in the collection, filtering for objects either rarely accessed or supported by incomplete records. The project employs randomization techniques to disrupt norms, to de-center the viewer’s perspective, to generate alternative instances of visibility and reimagine peripheries and borders.

“Brokenness” is a no less relative concept that wholeness. But here it is featured rather than concealed. The term becomes a window into the instrumental ethics of algorithms, one that observes how information filtering systems shape culture by reinscribing what is always already preferred. Brokenness provides an antidote by which further clicks reveal ever less familiar artists, defined objects, and identifiable regions. (Giulia Taurino)

A Flitting Atlas of the Human Gaze When the Harvard Art Museum collection looks back at us, which direction does it look? Up, down, left, or right? How deeply or shallowly does it cast its gaze? Do most images peer straight into the visitor’s eyes? What is the orientation of the subject’s head, frontal or rotated? Do particular media or cultural traditions correlate with preferences regarding the directionality of the human gaze? The installation is built upon the AI-based extraction and analysis, fine-tuned via human supervision, of pairs of eyes from the Harvard Art Museum painting, print, sculpture, and coin collections. It allows the visitor, equipped with an input device, to explore the collections from the standpoint of the depicted subject’s gaze direction. A red dot appears where the input device is pointed towards the wall of monitors, establishing a focal point, a point of convergence around which arrays of images are summoned up nine at a time. Opposite the monitors, highlighted zones within the overall cartography of gazes are presented via the gallery’s projection system. For centuries visitors have navigated collections on the basis of culture, chronology, genre, and medium; to those conventional forms of exploration, A Flitting Atlas of the Human Gaze adds a new mode based on the distribution of looks across media and time. (Kevin Brewster, Todd Linkner, Dietmar Offenhuber, Jeffrey Schnapp)


Harvard Art Museums