How do I get an HTML report out of DeepEval?

DeepEval test cases run under pytest, so `deepeval test run test_llm.py` accepts the usual pytest reporting plugins — `pytest --html=deepeval.html --self-contained-html test_llm.py` produces one self-contained file. Publish that file to Comma.

Why not use Confident AI's dashboard?

Confident AI is DeepEval's hosted platform and it is good at trend lines. It is also an account someone has to have. A Comma link opens for anyone you share it with, including a PM with no login, and it carries the comment thread.

Can I publish results for a failing metric?

Yes — use `--html` together with a step that runs regardless of exit status. A run where `AnswerRelevancyMetric` fell under threshold is the run people need to read.

Does the report keep the per-case reasoning?

If it is in the pytest output it is in the HTML. DeepEval writes the metric score and its reason into the assertion message, so the failing case carries the explanation the reviewer needs.

How do I track a metric over time?

PATCH one report id per suite. Each publish appends a revision, so the URL shows the latest run and keeps the history without you building a dashboard.

Share a DeepEval report

DeepEval's best design decision is that an LLM eval is a test. It runs under pytest, it fails a build, and it produces the same artifact every other pytest suite produces.

Which means the sharing problem is the same one too: the HTML report is a file on the runner, and the runner is gone in an hour.

Produce one self-contained file

pytest --html=deepeval.html --self-contained-html test_llm.py

--self-contained-html matters. Without it the report references a CSS and JS directory next to the file, and a report you send as one file renders as unstyled text on the other end — the usual cause of broken CSS in a shared report.

Then publish it from the same job:

curl -fsS -X PATCH "https://commareports.com/api/v1/reports/$REPORT_ID" \
  -H "Authorization: Bearer $COMMA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile html deepeval.html \
        --arg title "LLM suite — $GIT_SHA" '{title: $title, html: $html}')"

Guard the step with if: always() on GitHub Actions, when: always on GitLab. See publishing from CI for the per-provider form.

What the review layer adds

A DeepEval failure says AnswerRelevancyMetric score 0.58 < 0.7. Whether that is a model regression or a bad threshold is a judgment call, and judgment calls need a thread:

Anchored threads — "this threshold is too strict for this intent" sits on the failing case. See commenting on HTML.
Revisions — one report id per suite, one revision per run.
Access per report — private, team-only, or domain-gated, since test cases usually encode real prompts. See the sharing model.

Close the loop with the agent

The agent that edits the prompt can read the comments back through the MCP server and re-run the suite — let an agent respond to comments covers the loop, and a routine keeps it running without a human kicking it off.

Limits

Entry HTML: 5 MB. Self-contained pytest reports inline their assets; a suite with long model outputs per case can approach the cap. Split by metric or truncate the captured output.
Assets: 25 MB per file, 250 MB and 500 files per report.
60 requests/minute per token.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Publish a DeepEval report →