Research

A multimodal pipeline for captioning and visual question answering in the art domain. ArtSeek extends ColQwen2 for multimodal retrieval, introduces WikiFragments — a dataset of multimodal fragments mined from Wikipedia — and a multitask classification framework, unlocking agentic retrieval-augmented generation for multimodal LLMs through in-context learning.

We ask what multimodal LLMs are actually looking at when they describe a painting. Using Token Activation Maps, we produce a heatmap for every generated token that isolates the visual evidence specific to that token, then analyse five categories across periods and genres: common objects, style descriptors, metadata, iconographic tokens and affective expressions. Grounding varies sharply by token type — models attribute artists far more reliably than they predict titles, where hallucination concentrates.

Image-conditioned music systems are trained on natural photographs, which leaves them unable to capture the semantic, stylistic and cultural content of artworks. We introduce ArtSound, a dataset of 105,884 artwork–music pairs with dual-modality captions, and ArtToMus, a framework that maps digitized artworks directly to music by projecting visual embeddings into the conditioning space of a latent diffusion model — no image-to-text step in between.

A neural architecture for few-shot semantic segmentation that supports multi-class segmentation from points, boxes or masks as prompts, relaxing several of the constraints that normally govern how support sets have to be built.

We inpaint several regions of a painting at once, each driven by its own text prompt, and train a multimodal LLM to propose those prompts automatically — so the model suggests what could plausibly fill each gap rather than requiring a human to describe every mask.

An investigation into which visual features vision transformers actually rely on when classifying artworks, and how to surface them in a form an art historian can interrogate.

Topic modelling over artwork collections that combines a fine-tuned CLIP with knowledge-driven prompts, so the discovered topics line up with art-historical vocabulary instead of arbitrary clusters.

An overview of the lab’s work on domain-aware generative models spanning images, music and text.

We extend the AudioLDM2 architecture to generate music directly from artworks, trained on a dataset of image–music pairings collected with ImageBind.

A fully containerized architecture for building LLM chatbots, with conversational forms that measurably improve how reliably they collect structured data from a user.

Caption generation for digitized artworks from a noisy dataset of LLM-written descriptions. We introduce CLIPScore weighting, which weighs each caption by its estimated quality during training and recovers much of the performance lost to the noise.

A short position piece on where computer vision and art history usefully meet.