# Find Similar Artworks

Six independent signals, one comparison page

Given any artwork in the Rijksmuseum collection, `find_similar` discovers related works through six orthogonal similarity signals — visual appearance, catalogue description, Iconclass subject codes, creator lineage, depicted persons, and depicted places — then renders an interactive comparison page with IIIF thumbnails. Each signal captures a different dimension of “similar”, and artworks that appear across three or more signals are pooled as the strongest matches.

6
similarity signals
833K
artworks searchable
511K
description embeddings
3+
signals to pool

## The problem: “similar” means different things

When a researcher says “find artworks similar to The Night Watch”, they could mean any of several things: works that look similar, works that depict the same people, works from the same artistic circle, or works that share the same subject matter. No single similarity metric captures all of these.

The Night Watch
Rembrandt van Rijn, 1642 · SK-C-5

The iconic civic guard painting yields very different results depending on which dimension of similarity you ask about:

Visual other large group scenes with dramatic lighting
Iconclass artworks tagged with civic guard, weapons, drums
Description works described with the same compositional vocabulary
Person other portraits of Frans Banninck Cocq, Willem van Ruytenburch, etc.

Each signal is a different lens on the collection. Combining them reveals connections that no single signal would surface alone.

## Six similarity signals

Each signal uses a different data source and scoring algorithm. They run concurrently and produce independent result lists, which are then rendered side-by-side.

Signal | Data source | Scoring | Coverage
--- | --- | --- | ---
Visual | Rijksmuseum image similarity API | Rank order (external model) | ~87%
Description | 511K Dutch catalogue descriptions → e5-small 384d int8 embeddings | Cosine similarity (1 − distance) | ~61%
Iconclass | 658K artworks with Iconclass notation codes | depth × IDF per shared notation | ~79%
Lineage | 208K artworks with attribution qualifiers (after, workshop of, circle of …) | qualifier strength × creator IDF | ~25%
Person | 217K artworks with depicted person metadata | IDF per shared person | ~26%
Place | 116K artworks with depicted place metadata (broad regions excluded) | IDF per shared place | ~14%

Coverage across the collection (833K artworks)
Visual
~724K
external API
Iconclass
~658K
vocab DB
Description
~511K
embeddings DB
Person
~217K
vocab DB
Lineage
~208K
vocab DB
Place
~116K
vocab DB

Not every artwork produces results for every signal. An artwork with no depicted persons will have an empty Person row; a primary attribution (no qualifier) will have an empty Lineage row. The comparison page only shows signal rows that produced results.

## How each signal works

Visual Image-based similarity

Calls the Rijksmuseum’s own visual search API, which uses an internal image embedding model. The object number is first resolved to an internal node ID, then passed to the `/api/v1/collection/visualsearch` endpoint. Best-effort: API failures are silently skipped. Results are rank-ordered (no numeric similarity score).

The Night Watch
query artwork
→
Copy of The Night Watch
attr. Gerrit Lundens
rank #1
Description Catalogue description similarity

Each of 511K artworks has a Dutch catalogue description embedded with `intfloat/multilingual-e5-small` (384 dimensions, int8 quantized). At query time, the embedding of the query artwork is looked up and used for KNN search via `sqlite-vec` (vec0 virtual table). Score is cosine similarity (1 − distance). Artworks with similar compositional vocabulary (“links een X, rechts een Y”) cluster together.

The Milkmaid
query artwork
→
works with similar
Dutch descriptions of
interior domestic scenes
Kitchen scenes, still lifes
similarity: 0.82–0.91
Iconclass Shared subject codes

Finds artworks sharing the same Iconclass notation codes. Deeper codes (more specific subjects) that appear on fewer artworks contribute more — the score is depth × IDF (inverse document frequency). Generic top-level notations like “historical persons” (`61B2`, 90K artworks) are noise-filtered. Single shallow matches (depth < 5) are pruned to avoid overly broad hits.

Night Watch shares `45(+26)` "citizen militia", `45C1` "weapons", `48C7341` "drum"
→ deep, rare codes like `45(+26)` score highest
→ matches: other civic guard paintings, militia group portraits
Lineage Creator attribution chains

Finds artworks that share visual-style lineage through attribution qualifiers. Works “after” the same artist, from the same “workshop”, “attributed to” the same hand, or in the same “circle” are connected. Score is weighted by qualifier strength × creator IDF: rarer creators contribute more.

Qualifier | Weight | Meaning
--- | --- | ---
after / copyist of | 3.0× | direct visual derivation
workshop of | 2.0× | produced in the master’s studio
attributed to | 1.5× | probably by this artist
circle of / follower of | 1.0× | stylistic sphere of influence

Laughing Young Man
circle of Rembrandt
→
other works in Rembrandt’s
circle, workshop, and
follower attributions
Circle / workshop / after
qualifier: circle of (1.0×)
Depicted Person Shared historical figures

Finds artworks that depict the same named individuals. Scoring uses IDF: people who appear in fewer artworks contribute more. A rarer figure like Jan Pietersen Bronchorst scores higher than a frequently depicted person. Results link to Wikidata where available.

Night Watch depicts 14 named guardsmen
→ Frans Banninck Cocq appears in 40 artworks → lower IDF
→ Jan Pietersen Bronchorst appears in 2 → higher IDF
→ matches: other portraits, prints, and reproductions of these individuals

Known limitation: Due to a harvest-level classification issue (#145), 74 organisations and institutions (e.g. the VOC, Binnenhof) are currently misclassified as depicted persons in the vocabulary database. These entities may appear in Person signal results. A fix is tracked but requires a database update.

Place Shared depicted locations

Finds artworks depicting the same specific locations — streets, buildings, monuments, waterways. Broad administrative regions (countries, provinces, and cities with >20 child places in the hierarchy) are excluded to avoid matching everything in Amsterdam or the Netherlands. Scoring uses place IDF.

An artwork depicting "Oude Kerk, Amsterdam"
→ "Amsterdam" excluded (too broad, >20 children)
→ "Oude Kerk" retained (specific landmark)
→ matches: other paintings, prints, and photographs of the Oude Kerk

## Pooling: the strongest connections

Artworks that appear in three or more signal modes are pooled into a combined row at the top of the comparison page. These are the works most deeply connected to the query artwork — similar in multiple independent dimensions simultaneously.

### Why pooling matters

Single-signal noise

Any individual signal can produce false positives: the description signal may match works with similar Dutch phrasing but unrelated subjects; Iconclass may match on a generic shared code; visual similarity may surface works with similar colour palettes but no thematic connection.

Multi-signal convergence

When three or more signals independently identify the same artwork, the connection is unlikely to be noise. A work that is visually similar, shares subject codes, and depicts the same people is almost certainly a meaningful match — a copy, a companion piece, or a closely related work.

The pooled row shows which signals matched for each artwork via coloured mode badges (V Desc IC Lin Per Pl), sorted by the number of matching signals.

## Execution pipeline

When `find_similar` is called, all six signals run concurrently. The visual signal is best-effort (external API); the other five are local database queries. Results are assembled into an HTML page and served via a time-limited URL.

Step 1

### Resolve query

Look up artwork metadata, types, descriptions, and IIIF image ID

→
Step 2

### 6 signals

Run all signals concurrently: 5 local DB queries + 1 external API call

→
Step 3

### Pool

Identify artworks appearing in 3+ signals, sort by count

→
Step 4

### Render HTML

Generate self-contained page with IIIF thumbnails, signal rows, metadata

→
Step 5

### Serve

HTTP: `/similar/:uuid` (30-min TTL). Stdio: temp file

## Anatomy of a comparison page

The HTML page is a self-contained document with no external dependencies. It displays the query artwork with its full metadata, followed by horizontal scroll rows for each signal that produced results.

⌖

### Query artwork header

IIIF thumbnail (300px), title, creator, date, type badge. Below: description (Dutch), Iconclass codes (linked to iconclass.org), lineage qualifiers (linked to Getty AAT), depicted persons and places (linked to Wikidata).

≡

### Signal rows

One horizontal scroll strip per signal, in fixed order: Visual → Lineage → Iconclass → Description → Person → Place. Each row has a coloured header, methodology note, and result cards. Empty rows are hidden.

⊕

### Result cards

200px-wide cards with IIIF thumbnails (3:4 aspect ratio), rank + score, title (2-line clamp), creator. Detail line varies by signal: Iconclass codes, qualifier + creator, description snippet, or shared person/place names.

¤

### Pooled row

Artworks in 3+ signals, sorted by count descending. Each card shows coloured mode badges indicating which signals matched. This row appears first when present, highlighting the strongest connections.

## Example results

Three artworks from different periods illustrating how different signal combinations reveal different kinds of connections.

The Night Watch
Rembrandt van Rijn, 1642 · SK-C-5
Visual 8 results Iconclass 8 Description 8 Person 8 Pooled 1
No Lineage (primary attribution) or Place (Amsterdam too broad) results. The pooled artwork is the Lundens copy — visually identical, shares all depicted persons.
Laughing Young Man
circle of Rembrandt van Rijn, c. 1629–1630 · SK-A-3934
Visual 6 results Lineage 6 Iconclass 6 Description 6
The “circle of” qualifier activates the Lineage signal — finding other works from Rembrandt’s workshop, pupils, and followers. No depicted persons (generic tronie).
Self-portrait
Vincent van Gogh, 1887 · SK-A-3262
Visual 6 results Iconclass 6 Description 6 Person 6
The Person signal finds other depictions of Vincent van Gogh across the collection — other self-portraits, photographs, and posthumous works. Iconclass matches on “portrait, self-portrait of painter” (`48C513`).

## Scoring and ranking

Each signal uses a scoring formula appropriate to its data. All vocabulary-based signals use IDF (inverse document frequency) as a core building block: terms that appear on fewer artworks get higher weights, preventing generic matches from dominating results.

Signal | Formula | Precision | Notes
--- | --- | --- | ---
Visual | (rank order) | — | No numeric score; ordered by the external API
Description | 1 − cosine_dist | 3 dp | Embedding similarity, normalised to [0, 1]
Iconclass | depth × ln(N/df) | 1 dp | Deeper + rarer notations score higher
Lineage | strength × ln(N/df) | 2 dp | “After” (3×) beats “circle of” (1×)
Person | ln(N/df) | 2 dp | Rare people score higher
Place | ln(N/df) | 2 dp | Broad regions (>20 children) excluded

## Infrastructure and serving

The comparison page is generated server-side as self-contained HTML. Two serving modes support both Claude Desktop (stdio, file-based) and claude.ai (HTTP, URL-based).

### HTTP mode

The page is stored in a module-scope `similarPages` Map, keyed by a random UUID. The route `GET /similar/:uuid` serves the HTML. A 30-minute TTL sweeper runs every 60 seconds, deleting pages not accessed within the window.

https://….up.railway.app/similar/06818252-591e-4508-843a-cf2bdb1ebf8b

### Stdio mode

The page is written to `os.tmpdir()` as a local HTML file. Same 30-minute TTL sweep applies. The LLM returns the file path for the user to open in a browser.

/tmp/rijksmuseum-similar-a1b2c3d4.html

### Feature gate

The tool is enabled by default. Set `ENABLE_FIND_SIMILAR=false` to disable it (e.g. in environments without the vocabulary database). When disabled, the tool is not registered and does not appear in the tool list.

### Data requirements

Requires the vocabulary database (1.1 GB) for all DB-based signals, plus the embeddings database (997 MB) for the Description signal. Visual signal needs internet access. Total: ~2.2 GB across three SQLite files, all memory-mapped.