How do I share an LLM eval report?

Export the run as HTML — promptfoo's `export`, DeepEval's report output, or your own table rendered from the results JSON — and POST it to https://commareports.com/api/v1/reports with a Bearer comma_sk_ token. You get a URL instead of a local server, and the eval becomes something a teammate can open and argue with.

Why not just share the aggregate score?

Because the aggregate is the least informative number in the run. 'Pass rate 84%' produces agreement without understanding; the 16% is where the disagreement lives, and it is per-row. A shared report puts the failing cases in front of people who can say whether the grader was wrong.

Can a reviewer disagree with a specific grade?

That is the point. Comment threads anchor to the rendered content, so 'the judge marked this a refusal but it's a correct safety response' attaches to that row and stays attached when you rerun — which is how a rubric actually gets fixed.

How do I compare evals across model versions or prompt revisions?

PATCH the same report id on every run. One stable URL per eval suite, a revision per run, and a diff between any two — so 'did the new system prompt help' is answered by the rows that changed rather than by two numbers.

Can an agent publish its own eval run?

Yes. The same REST surface is available over MCP, so an agent that just finished a run can publish the report with one tool call and hand back the link — no human in the loop to move the artifact.

Can I run the eval on a schedule?

Yes. A routine can run the eval on a cron and PATCH the same report, so the eval refreshes itself and the URL in your team channel is always current.

Share an LLM eval report

Eval tooling has converged on a good shape: a table of test cases, one column per model or prompt variant, a grade per cell, and a rollup at the top. What none of it has solved is that the table lives behind a local command — promptfoo view, a notebook cell, a Streamlit process on port 8501 — so the only thing that escapes into a team conversation is the rollup.

And the rollup is the least useful part of an eval. 84% pass is a number everyone nods at. The 16% is where the actual disagreement lives, and it is per-row: this grader was too strict, that test case is wrong, this refusal was correct behaviour scored as a failure.

Publish the run

npx promptfoo eval
npx promptfoo export latest --output eval.html

import os, requests

r = requests.post(
    "https://commareports.com/api/v1/reports",
    headers={"Authorization": f"Bearer {os.environ['COMMA_API_TOKEN']}"},
    json={
        "title": "Support agent eval — system prompt v7",
        "description": "84% pass. 12 regressions vs v6, all in the refund flow.",
        "html": open("eval.html").read(),
    },
)
print(r.json()["url"])

Same shape for anything else that emits HTML — DeepEval, a Ragas notebook converted with nbconvert, or a table you render yourself from the results JSON, which is often the better report because you control what a reviewer sees first.

Write a real description. "84% pass, 12 regressions, all in the refund flow" is the sentence that makes someone open the link.

The review is the deliverable

An eval only improves a system if somebody disagrees with it out loud. Anchored comment threads put that disagreement on the row it belongs to:

"The judge scored this a hallucination — it's quoting the doc correctly."
"This test case has been wrong since we wrote it. Remove it."
"Correct refusal, graded as a failure. Rubric bug, not a model regression."

Each thread stays attached across reruns, which is what turns a rubric from folklore into something maintained. See commenting on HTML and giving an agent feedback.

One URL per suite, a revision per run

curl -fsS -X PATCH "https://commareports.com/api/v1/reports/$REPORT_ID" \
  -H "Authorization: Bearer $COMMA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile html eval.html '{html: $html}')"

Model versions change under you, prompts change weekly, and the only honest way to read an eval is against the last one. PATCHing a single report id makes the comparison structural: the link in your team channel is always current, and every previous run is a diff away.

Two ways to make that automatic:

From CI, on every prompt change — see publishing from CI.
On a schedule, with a routine that reruns the eval and republishes to the same report.

Let the agent publish it

The REST surface is also exposed over MCP, so an agent that just ran an eval can publish the report itself and return the link — one tool call, scoped token, no human moving files between machines. That matters more as evals get run by the thing being evaluated. See where should my agent post.

Limits

HTML body: 5 MB. A large eval matrix with full model outputs will exceed it — publish the summary table plus the failing cases, and attach the raw results JSON as an asset (25 MB per file).
Scripts run, sandboxed: allow-scripts, no allow-same-origin.
60 requests/minute per token.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Publish an eval →