Demo

Benchmark for Cultural Preservation in the Age of AI

An open benchmark that measures how well AI models preserve cultural heritage: reading old documents, describing objects, linking them to the right records, and not making things up.

Readme
↓  Open

Museums, archives and libraries are starting to use AI to transcribe, describe and catalogue their collections. But nobody can say which models actually do this well, and where they quietly damage the record: a wrong date, an invented provenance, a lost dialect word, a mislabelled object.

This project builds a small, open benchmark from Swiss open cultural data (for example Wikimedia Commons, Wikidata and GLAM collections) and uses it to test AI models, including Apertus, on the tasks that matter for preservation:

  • Reading: transcribing handwritten and printed historical documents (OCR/HTR), including old scripts and Swiss German.
  • Describing: writing catalogue descriptions and metadata for objects and images, judged against the institution's own records.
  • Linking: matching people, places and objects to the correct Wikidata entries.
  • Translating: carrying historical texts between German, French, Italian and English without losing meaning.
  • Faithfulness: catching hallucinations, invented facts and false provenance, and checking that models keep licences and private persons' data intact.

Each task has a datasheet (sources, licences, known gaps), a scoring script and a simple leaderboard, so any institution can rerun it on a new model.

Goal for the weekend: 3 to 5 tasks with test data, a first comparison of several models, and a leaderboard others can extend.

Skills welcome: archivists and curators (to say what "correct" means), data wranglers (OpenRefine, Wikidata), Python, machine learning, and anyone curious how good AI really is at preservation.

Team: Codezilla

OpenGLAM-Bench

A living, open benchmark for AI in cultural heritage collections, built around what GLAM institutions actually need: truthful description, source-cited claims, respect for rights and licences, and the ability to say "insufficient evidence" instead of inventing heritage.

Created at GLAMhack 2026 (Swiss Open Cultural Data Hackathon, 12โ€“14 Nov 2026, Winterthur) run by the Opendata.ch association. See ETHICS.md at the repository root for the ethical code adapted from the Apertus Charter.

Tech stack: OpenCode, Apertus 1.5 70B (initial spec), Inkling 2 Small (scaffold), NeoVim (human-in-the-loop), Python, Manjaro Linux

Overview

Five tasks, built from real collection work rather than toy prompts:

Task Input Output Core metrics
reading Historical document (text proxy until images are curated) Transcription CER/WER accuracy
describing Object record fragment Catalogue fields (KEY: VALUE / JSON) Field-level precision/recall/F1
linking Mention + candidate Wikidata IDs QID or "no answer" Hits@1, MRR, abstention correctness
translating Historical text (de/fr/it/de-ch) Translation preserving meaning Token-F1, key-term retention
faithfulness Record + claim "supported" / "unsupported" + error type Accuracy, hallucination-detection F1

Design principles: evidence-grounded outputs, abstention is rewarded over guessing, localized multilingual data over translated English, multi-metric reporting (never one score), contamination awareness, community governance.

Repository layout

openglam_bench/
โ”œโ”€โ”€ openglam_bench/          # the benchmark package (stdlib only)
โ”‚   โ”œโ”€โ”€ scoring/             # reading, describing, linking, translating, faithfulness
โ”‚   โ”œโ”€โ”€ models/              # dummy adapter + OpenAI-compatible adapter
โ”‚   โ”œโ”€โ”€ runner.py            # load test data โ†’ prompt model โ†’ score โ†’ JSON
โ”‚   โ”œโ”€โ”€ leaderboard.py       # multi-metric ranking (mean rank, no blended score)
โ”‚   โ”œโ”€โ”€ textnorm.py          # CER/WER/token-F1/Levenshtein from scratch
โ”‚   โ””โ”€โ”€ dotenv.py            # minimal .env loader (shell env wins)
โ”œโ”€โ”€ tasks/<task>/            # datasheet.md + test_data/test.jsonl per task
โ”œโ”€โ”€ data/ckan/               # CKAN snapshot + recommendations (metadata only)
โ”œโ”€โ”€ scripts/
โ”‚   โ”œโ”€โ”€ run_benchmark.py     # evaluate a model
โ”‚   โ”œโ”€โ”€ make_leaderboard.py  # build CSV + HTML leaderboard
โ”‚   โ””โ”€โ”€ fetch_ckan_datasets.py  # refresh OpenGLAM dataset list + recommendations
โ”œโ”€โ”€ tests/                   # smoke/unit tests (pytest)
โ””โ”€โ”€ results/                 # per-model result JSON (gitignored)

Requirements

  • Python โ‰ฅ 3.10. The benchmark itself has zero third-party dependencies (HTTP via urllib; metrics implemented from scratch).
  • uv (optional, for tests) โ€” or any environment with pytest.

Quickstart (offline, no API key)

cd openglam_bench
python3 scripts/run_benchmark.py --model dummy --output results/dummy.json
python3 scripts/make_leaderboard.py --results results --out leaderboard
uv run --with pytest python -m pytest tests/ -q   # or: python -m pytest tests/

Running against an LLM API

1. Configure .env

cp .env.template .env
# edit .env

.env is gitignored. Existing shell variables win over .env values. Never commit .env or paste keys into issues/logs.

Variable Meaning Example
OPENAI_BASE_URL OpenAI-compatible /v1 endpoint http://localhost:8000/v1 (self-hosted), https://api.openai.com/v1, https://openrouter.ai/api/v1
OPENAI_MODEL Model id exactly as the endpoint expects swiss-ai/Apertus-8B-Instruct-2509 (local vLLM), gpt-4.1-mini
OPENAI_API_KEY Key for that endpoint leave empty for unauthenticated local servers
OPENAI_API_KEY_ENV (optional) read the key from a different variable instead MY_PROVIDER_KEY

Works with any OpenAI-compatible /v1/chat/completions server: self-hosted open-weight models (including Apertus served via vLLM/sglang at localhost:8000/v1), OpenRouter, OpenAI, or gateway proxies.

2. Run the benchmark

python3 scripts/run_benchmark.py --model openai-compatible --output results/my-model.json
# subset of tasks / quick smoke:
python3 scripts/run_benchmark.py --model openai-compatible --tasks reading,linking --limit 10 --output results/smoke.json

--base-url, --name, --api-key-env override the .env values; --tasks takes a comma-separated subset; --limit N runs the first N items per task. Each output JSON contains per-item outputs + metrics and a per-task summary.

3. Update the leaderboard

python3 scripts/make_leaderboard.py --results results --out leaderboard
# then regenerate the website's embedded pages:
python3 ../site/build.py

Multi-metric by design: CER/WER accuracy, field F1, Hits@1/MRR, token-F1/key-term retention, hallucination F1, plus mean_rank. No single blended score.

Test data and licences

  • tasks/*/test_data/test.jsonl ships synthetic CC0 placeholders (clearly marked synthetic://). They exist to pin the harness, not to measure models.
  • Real records will be added per institution after rights review โ€” see ETHICS.md (rights first: no dataset enters the benchmark until its licence is verified) and scripts/fetch_ckan_datasets.py, which fetches the OpenGLAM dataset list from https://opendata.swiss (metadata only, never payloads) and ranks candidates per task.

Contributing

  • Tasks, ground truth, scoring, and leaderboard changes all require domain-expert + community review (GLAMhack / Opendata.ch). Technical and ethical changes are reviewed separately.
  • Before opening anything, read ETHICS.md โ€” abstention scoring, evidence-citation, and contamination rules are load-bearing, not decoration.
  • Site development: python3 site/build.py regenerates the embedded leaderboard/dataset pages.
Preview of external content.
All attendees, sponsors, partners, volunteers, and staff are required to abide by the Hack Code of Conduct. The organisers are committed to enforcing it throughout the event and expect everyone to contribute to creating a safe, respectful, and welcoming environment for all.

The contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License. The application that powers this site is available under the MIT license.