Name: Curatorial A(i)gents
Date: 2021
A metaLAB series of interactive projects exploring machine learning in, around, and about the Harvard Art Museums

Curatorial A(i)gents presents a series of machine-learning-based experiments with museum collections and data developed by members and affiliates of metaLAB (at) Harvard.

Long before computers came to pervade every aspect of modern life, museums were collecting, organizing, and storing data. The art museum is a kind of vast machine for making all kinds of objects interoperable, from bronze-age figurines to Renaissance paintings to contemporary performance-art works. Like our digital machines, museums engender wonderful experiences—and they’re also engines of bias, power, and invisibility.

The term “machine learning” represents a family of systems that use algorithms to find patterns in data inferentially, without explicit instructions. Artists and media makers are experimenting with machine-learning tools to create new kinds of artworks. But roles for machine learning in the art museum are still rare in practice. Presented in the Lightbox Gallery, metaLAB’s projects explore emerging possibilities for machine-learning systems while exploring vital issues at the intersection of technology and culture. Variously playful, analytic, and critical, metaLAB’s experiments use the museums’ own data to expressive ends. The names and dates of works and their makers; curatorial descriptions and histories of exhibitions; colors and dimensions; images of objects themselves—encountering such data as these, algorithms chart invisible relations, forge new connections, and breed monsters.

Some projects are playful: Philipp Schmitt uses machine vision to connect paintings and prints with current weather conditions visible through the Lightbox Gallery’s glass ceiling; Dan Newman, Keith Hartwig and Kevin Brewster’s AI Exquisite Corpse identifies body-like images in the collections, inviting visitors to construct their own strange hybrid beings; Sarah Newman and Ken Goldberg are exploring ways of relating artificial intelligence and robotics to care, cultivation, and images of gardens. Other projects in the series are analytic: through algorithms and visualization, Lins Derry’s visualization work reveals vital assumptions about gender and sentiment at play in art history; in Watching Machines Loving Grace, designer Kim Albrecht uncovers the biases in commercial machine-learning systems; and metaLAB founder Jeffrey Schnapp, working with Dario Rodighiero, Dietmar Offenhuber and Satvik Shulka, maps the vastness of the museums’ collections through an alien, artificially-intelligent eye. And some projects take a critical perspective on the ethical and technical problems of intelligent machines: riffing on the shopping systems found in streaming media and online markets, in This Recommendation System is Broken, Giulia Taurino plays with our museum-going expectations as consumers and viewers; in Ocean Amplitude, Francisco Alarcón (in collaboration with MIT’s Stefan Helmreich) infuses sublime experience with questions about the environmental costs of computation.

Curatorial A(i)gents will open at the Harvard Art Museums’ Lightbox Gallery at the beginning of February 2021 and run through mid-April 2021. Over the course of the summer and fall of 2020, project teams will continue to refine their work and metaLAB will be working on a print-based publication, combining project documentation with critical writings from a range of experts and practitioners in the fields of art, AI, and data science.

A collective headquartered at Harvard’s Berkman Klein Center for Internet and Society, metaLAB explores the digital arts and humanities through research, experimentation, tool building, teaching, through publications in print and online, and via exhibition, performance, and social practice. Here at the Harvard Art Museums, as with partners across the university, and in the world at large, metaLAB’s work infuses traditional modes of academic inquiry with an enterprising spirit of hacking, making, and creative research.


Sympoietic System Our expectations of artificial intelligence are drawn from our expectations of ourselves as autonomous, thinking agents. But we’re social beings, making worlds by interacting with one another, with objects, and with systems. Dempster and Haraway have called this “sympoiesis,” or “making-together.” In Philipp Schmitt’s “Sympoietic System,” the weather becomes curator for an art exhibition, alluding to a wealth of world-making networks — weather and climate, computation, and aesthetics — that are intertwined and influencing each other. (Philipp Schmitt)

Watching Machines Loving Grace Machine Learning algorithms rely on training data—vast sets of tagged and identified images or other data objects—to develop models of relations. Museum collections present themselves as ideal training data. And yet with their assumptions of value, hierarchies of objects and forms of expertise, these collections are biased in highly peculiar ways. Algorithms, too, have their biases; they confer with collections in a dance of strange and estranging presumptions. Kim Albrecht’s project visualizes machine-learning tags identifying features and sentiments so we can navigate and explore these choreographies of bias in collections and computational systems. (Kim Albrecht)

AI Exquisite Corpse The most famous of the Surrealist “games” for fostering collective creativity, Exquisite Corpse proceeds by challenging players to make partial contributions to collaborative drawings or poems, proceeding from fragmentary evidence. AI Exquisite Corpse asks us to finish images in partnership with an algorithm, which might have radically different presumptions about what an image might be. (Kevin Brewster, Keith Hartwig, Daniel Newman)

Ocean Amplification According to a 2019 paper in Science, ocean-warming induced wind speed increases in the Earth’s southernmost oceans have led to amplifications there in wave height of some 30 centimeters since 1985, for the largest 10 percent of waves — significant because the Southern Ocean is source for many swell patterns worldwide. Material effects of climate change — intensified storms, sea-level rise — here manifest through a key symbol of ocean transformation: the wave. “Ocean Amplification” is a montage that explores visualizations and simulations of rising waves, read as avatars of hybrid human-inhuman political ecology. (Francisco Alarcon with Stefan Helmreich)

Lexical Cartography Recent artificial intelligence algorithms allows us to manage a large number of objects at the same time, like the HAM collection. The idea is to create a meta-organization of items belonging to the collection by using their description. The descriptions filled up by curators are then employed to place 250,000 items on the screen, pointing out what was exhibited and what is still unseen. The spectator will be invited to explore a galaxy of accessible and inaccessible collection objects using an ipad as a controller. (Dario Rodigiero)

This Recommendation System is Broken “If you like this, you might also like…” algorithmic recommendation systems have become ubiquitous in online platforms. And yet A.I.-driven processes of content personalization often bear biases and limitations. This Recommendation System is Broken upends our expectations of automated choice, surfacing suggestions from the vast trove of undervalued, hidden, unseen artworks in the museum collection. Giulia Taurino challenges us to rethink museum collections in terms of visible and invisible objects, by inviting us to explore what we might call “brokenness” in recommendation systems and to reconsider our understanding of marginalized art history. (Giulia Taurino)

Second Look Second Look is a data visualization system that utilizes commercial AI services to analyze the gender and sentiment of faces in paintings found within the digital data repositories of the Harvard Art Museums. By observing how AI infers gender and sentiment in paintings in relationship to metadata about the paintings’ color and chronology, we are able to attain new views of the collection. In particular, we are able to observe how gender is represented by artists through time, appropriated by museums, and understood by AI. (Lins Derry) Link coming soon

A Flitting Atlas of the Human Gaze When the Harvard Art Museum collection looks back at us, which direction does it look? Up, down, left, or right? How deeply or shallowly does it cast its gaze? Do most images peer straight into the visitor’s eyes? What is the orientation of the subject’s head, frontal or rotated? Do particular media or cultural traditions correlate with preferences regarding the directionality of the human gaze? The installation is built upon the AI-based extraction and analysis, fine-tuned via human supervision, of pairs of eyes from the Harvard Art Museum painting, print, sculpture, and coin collections. It allows the visitor, equipped with an input device, to explore the collections from the standpoint of the depicted subject’s gaze direction. A red dot appears where the input device is pointed towards the wall of monitors, establishing a focal point, a point of convergence around which arrays of images are summoned up nine at a time. Opposite the monitors, highlighted zones within the overall cartography of gazes are presented via the gallery’s projection system. For centuries visitors have navigated collections on the basis of culture, chronology, genre, and medium; to those conventional forms of exploration, A Flitting Atlas of the Human Gaze adds a new mode based on the distribution of looks across media and time. (Kevin Brewster, Dietmar Offenhuber, Jeffrey Schnapp)

Robots in Gardens Cultivation, growth, and care—how do these things come together in gardens, museums, and technology? Artists Newman and Ken Goldberg explore horticultural imagery, automation, and algorithms in a playful, networked garden of robots, plants, and pictures. (Sarah Newman with Ken Goldberg)


Harvard Art Museums