SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

Bommarito, M. J., II (2026). SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking. arXiv preprint. 16 pages, 15 tables. Libraries and archives manage large collections with limited staff and computing budgets, yet general embedding benchmarks do not systematically test bibliographic tasks. SHELF, the Synthetic Harness for Evaluating LLM Fitness, is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies and supports classification, clustering, retrieval, pair classification, and instruction retrieval. We evaluate methods ranging from TF, TF-IDF, and BM25 to popular encoders, plus a subject-only comparison with small zero-shot decoders. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605; several pair and clustering tasks also remain near chance. Sparse methods remain competitive on classification, and TF-IDF is the fastest measured arm in the subject timing experiment. Unlike natural-record benchmarks such as LCSHBench, SHELF varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Calibration against LCSHBench and Project Gutenberg shows that model rankings transfer more reliably than absolute scores. Code and data are released under permissive licenses on GitHub and Hugging Face..