How do I get an HTML report out of DeepEval?

DeepEval runs through pytest, so `pytest --html=deepeval.html --self-contained-html` works on a DeepEval test file the same way it does on any other suite.

What does a DeepEval failure look like?

A metric below its threshold, with the metric's own reason string explaining why — which is the part worth publishing, because it is written in English rather than as a number.

Should LLM evals gate a deploy?

Many teams gate on a subset — the regression cases that must never break — and treat the rest as reporting. Keep the gate as its own unmuted step and publish the full run regardless.

How do I compare runs?

PATCH the same report id each run. A prompt or model change is then a diff between two revisions rather than two exported screenshots.

Whoever owns the behaviour — usually a product owner or domain expert rather than the engineer who wrote the metric. That is exactly the audience a link serves and a notebook does not.

Share a DeepEval report

DeepEval makes LLM evaluation look like unit testing, which is a genuinely good idea: assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7)]) either passes or it doesn't, and it runs in the same pipeline as everything else.

The catch is that a failure here is not like a failing unit test. It is a judgment call rendered as a number, and the person qualified to review it is usually not the person who can read pytest output.

pytest-html on a DeepEval suite

pip install deepeval pytest-html

deepeval test run test_llm.py || true
# or, for the HTML artifact:
pytest test_llm.py --html=deepeval.html --self-contained-html || true

--self-contained-html inlines the CSS and JS so the result is one file. The metric reason strings — DeepEval's explanation of why a score landed where it did — come through in the captured output, which is the part that makes the report readable to a non-engineer.

Publish it

Drag deepeval.html into the app, or from CI:

curl -fsS -X PATCH "https://commareports.com/api/v1/reports/$REPORT_ID" \
  -H "Authorization: Bearer $COMMA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile html deepeval.html \
        --arg title "LLM eval — $GITHUB_REF_NAME" '{title: $title, html: $html}')"

Keep the hard regression cases as a separate unmuted gate step; publish the full run with if: always() either way.

Thresholds are opinions

threshold=0.7 is a choice somebody made on a Tuesday. Whether it is right depends on the product, and the argument about it should happen where the failing cases are visible:

Anchored threads on the failing case — "this one is fine, the metric is penalising the disclaimer we're required to include." See commenting on HTML.
Revisions, so raising a threshold shows exactly which cases it newly catches.
Access per report — test cases often contain real prompts and customer phrasing; keep them private or team-scoped. See the sharing model.

Limits

Entry HTML: 5 MB. Assets: 25 MB per file, 250 MB and 500 files total. Long generations in the captured output are what grows the file — truncate or sample for the shareable view.
60 requests/minute per token.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Publish an eval report →