2DCLIP

… what can we know about the para-visual concepts that CLIP operates on? 1

Background

2DCLIP emerges out of Leonardo Impett and Fabian Offert’s research paper, “There is a Digital Art History”, a “close-reading of CLIP”, OpenAI’s 2021 Contrastive Language-Image Pretraining model.

They argue that the advent of multimodal embedding networks ushers in a new era of digital art history, by “facilitat

a completely new form of image retrieval … based on para-visual concepts of arbitrary complexity” (192–3). Moreover, the discipline of digital art history appears to be uniquely well-positioned to address the entanglement of model and dataset, or ‘way of seeing’ and object: “in the age of foundation models, we can never quite isolate the neural network from the object of study” (202).

The CLIP model lies at the heart of generative models such as OpenAI’s Dall-E and the first Stable Diffusion. Yet prior to its impact through such generative models, it functions as a model embedding visual culture.

By understanding CLIP as a model of visual culture, it is possible to develop, on the one hand, a powerful new research technique for the digital humanities and, on the other, a double-sided critique wherein model and dataset emerge as inextricable from one another.

Use

2DCLIP points the way toward techniques commensurable to this new imbrication: one can use it either to sort a dataset along two semantic dimensions—or one can use it to ‘turn the model on itself’, investigating the ways of seeing that structure both the model and the dataset. For instance, in their paper Impett and Offert take up the fraught art-historical distinction between the naked and the nude (as discussed by Kenneth Clark, John Berger, and Linda Nead, among many others) in order to analyse how the model and collection determine these two entangled concepts.

2DCLIP includes four precomputed datasets: those of the Fitzwilliam Museum, the Victoria & Albert Museum Wallpaper collection, LAION, and ImageNet. One can also apply it to folders of images from one’s own computer.

Technical

2DCLIP runs entirely in the browser—that is, locally—consuming minimal compute. It uses threejs, d3, and OpenAI’s CLIP.