# How to Share lm-evaluation-harness Results (2026)

Canonical: https://commareports.com/agents/share-lm-eval-harness-report
Published: 2026-09-14

> lm-eval prints a markdown table and writes JSON to a path. Publish the table, the config and the sample-level output as a page people can read and argue with.

# Share lm-evaluation-harness results

The run finishes after four hours on the GPU box and prints a table. You have
numbers for ten tasks, with stderr, and the JSON is at `--output_path`.

Someone asks how the fine-tune did. You paste the table into Slack, where it
reflows into a column of misaligned pipes.

## Benchmark numbers travel badly

A score without its conditions is not a claim, it is a rumour. `hellaswag:
0.7421` means nothing until you also know:

- **Which model**, at which revision — not "our 7B", the actual checkpoint.
- **Harness version.** Task implementations change; scores move with them.
- **Task version**, `n-shot`, and whether you used the chat template.
- **dtype and batch size** — both can move a score by more than the
  improvement you are reporting.
- **stderr.** A 0.4-point gain with a 1.1-point standard error is not a gain.

The stdout table carries some of this. Slack carries none of it, and the JSON
carries all of it in a form nobody will open.

## Publish the results JSON as a page

The JSON has everything you need to render an honest page:

```python
import json

results = json.load(open("results.json"))
rows = results["results"]          # per-task metrics with stderr
config = results["config"]         # model args, batch size, dtype
versions = results["versions"]     # task versions
# render an HTML table plus a config block, then publish
```

Lead with the comparison — base model against fine-tune, per task, with stderr
and the delta. Put the full config in a `<details>` block underneath. The
reader sees the claim; the sceptic can expand the conditions without asking you
for them.

## Include failures people can read

Run with `--log_samples` and put ten of them on the page: five the model got
right, five it got wrong, with the prompt and the completion.

This is the part that changes minds. A table says the model scores 74% on a
reasoning benchmark. Ten failure samples show whether the misses are
genuine reasoning errors or an answer-format mismatch the harness is scoring as
wrong — and those two situations call for completely different next steps.

## Comments are the review

Benchmark results attract specific, well-founded objections: contamination,
the wrong chat template, a task version that changed. Those objections belong
next to the number they are about — see
[commenting on HTML](/comment-on-html) — not in a thread that scrolls away
before the retrain.

## Automate it in the eval job

```bash
lm_eval --model hf --model_args pretrained=$MODEL \
        --tasks hellaswag,arc_challenge,gsm8k \
        --output_path out/ --log_samples
python scripts/publish_eval.py out/results.json
```

Update the same report id per model line, and the link in the model card always
shows the current numbers. If you want the schedule owned for you, see
[routines](/docs/routines).

## Worth knowing

- **Never report a delta without stderr.** It is the single most common way
  benchmark reporting misleads, usually unintentionally.
- **Pin the harness commit** in the page. "Latest" is not reproducible.
- **5 MB per report body** — plenty for tables plus sampled generations.

## Try it

Free — unlimited reports, commenters and revisions.

**[Read the API reference →](/docs/api)**

### Related

- [Share an LLM eval report](/agents/share-llm-eval-report) ·
  [Share a promptfoo report](/share-promptfoo-report)
- [Share a DeepEval report](/agents/share-deepeval-report) ·
  [Share a Giskard scan](/agents/share-giskard-report)
- [The API](/docs/api) · [Commenting on HTML](/comment-on-html)