SHELF, the Synthetic Harness for Evaluating LLM Fitness, is a Python system that turns a labelled taxonomy, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. The first release targets bibliographic work: classifying, clustering, retrieving, and pairing documents by subject, genre, audience, register, and geography using Library of Congress vocabularies.

General embedding benchmarks do not cover this work. Across all 1,451 task definitions in MTEB 2.20.4, there is no Library of Congress, Dewey, MARC, or cataloguing task. SHELF fills that gap with documents whose bibliographic facets are set independently, so a benchmark can ask whether a model reads what a document is about or how it is written.

Corpus

Metric	Value
Total documents	62,899
Train / validation / test	37,795 / 12,600 / 12,504
Generation models	25
Factorial subset	18,345 documents from 15 models
LCC subject classes	21
LCGFT genre and form terms	133
LCSH topics / geographic	112 / 44
LCDGT audience groups	25
Writing registers	8

Every document records its writing model, prompt style, temperature, and nucleus-sampling probability, so the source of each document is a measured axis rather than a hidden one. The factorial subset covers all 60 model-prompt pairings with generator balance. All released documents were generated in or after November 2025, after the training cutoff of every embedding checkpoint evaluated in the paper.

Tasks

Classification by LCC subject, LCGFT form, topic, audience, and register
Retrieval by subject and genre form
Clustering by taxonomy
Pair classification for same-subject, same-form, same-audience, same-register, and topic-overlap pairs
Instruction retrieval

Findings

The tasks separate easy labels from hard ones. Subject classification reaches 0.8887 macro-F1, while classification across 133 genre forms reaches only 0.2605. Subject retrieval reaches 0.7104, but genre-form retrieval reaches 0.1173. Register and audience pairs remain near chance. Sparse methods such as TF-IDF and BM25 compete with dense encoders on classification and trail them on retrieval. In the timing experiment, TF-IDF has a near-best subject score and the highest measured processing rate.

Calibration against Project Gutenberg passages and LCSHBench catalogue records shows that model rankings transfer more reliably than absolute scores. The two human-catalogued corpora transfer to each other worse than either does with SHELF, so corpus mismatch is not unique to synthetic-to-natural transfer. SHELF supports model comparison and diagnosis, not estimates of production catalogue accuracy.

Availability

Dataset: mjbommar/SHELF on Hugging Face, CC BY 4.0
Code and stored per-task results: github.com/mjbommar/shelf-benchmark
Paper: SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking, 2026 preprint

from datasets import load_dataset

dataset = load_dataset("mjbommar/SHELF", "all")
lcc_pairs = load_dataset("mjbommar/SHELF", name="same_lcc_pairs")