Share lm-evaluation-harness results
The run finishes after four hours on the GPU box and prints a table. You have
numbers for ten tasks, with stderr, and the JSON is at --output_path.
Someone asks how the fine-tune did. You paste the table into Slack, where it reflows into a column of misaligned pipes.
Benchmark numbers travel badly
A score without its conditions is not a claim, it is a rumour. hellaswag: 0.7421 means nothing until you also know:
- Which model, at which revision — not "our 7B", the actual checkpoint.
- Harness version. Task implementations change; scores move with them.
- Task version,
n-shot, and whether you used the chat template. - dtype and batch size — both can move a score by more than the improvement you are reporting.
- stderr. A 0.4-point gain with a 1.1-point standard error is not a gain.
The stdout table carries some of this. Slack carries none of it, and the JSON carries all of it in a form nobody will open.
Publish the results JSON as a page
The JSON has everything you need to render an honest page:
import json
results = json.load(open("results.json"))
rows = results["results"] # per-task metrics with stderr
config = results["config"] # model args, batch size, dtype
versions = results["versions"] # task versions
# render an HTML table plus a config block, then publish
Lead with the comparison — base model against fine-tune, per task, with stderr
and the delta. Put the full config in a <details> block underneath. The
reader sees the claim; the sceptic can expand the conditions without asking you
for them.
Include failures people can read
Run with --log_samples and put ten of them on the page: five the model got
right, five it got wrong, with the prompt and the completion.
This is the part that changes minds. A table says the model scores 74% on a reasoning benchmark. Ten failure samples show whether the misses are genuine reasoning errors or an answer-format mismatch the harness is scoring as wrong — and those two situations call for completely different next steps.
Comments are the review
Benchmark results attract specific, well-founded objections: contamination, the wrong chat template, a task version that changed. Those objections belong next to the number they are about — see commenting on HTML — not in a thread that scrolls away before the retrain.
Automate it in the eval job
lm_eval --model hf --model_args pretrained=$MODEL \
--tasks hellaswag,arc_challenge,gsm8k \
--output_path out/ --log_samples
python scripts/publish_eval.py out/results.json
Update the same report id per model line, and the link in the model card always shows the current numbers. If you want the schedule owned for you, see routines.
Worth knowing
- Never report a delta without stderr. It is the single most common way benchmark reporting misleads, usually unintentionally.
- Pin the harness commit in the page. "Latest" is not reproducible.
- 5 MB per report body — plenty for tables plus sampled generations.
Try it
Free — unlimited reports, commenters and revisions.