How do I get HTML out of Ragas?

`result.to_pandas()` returns a DataFrame with one row per sample and a column per metric. `df.to_html('ragas.html')` renders it; styling it with `df.style` first makes the low scores findable.

What should the report contain?

The question, the retrieved contexts, the generated answer and the per-metric scores — faithfulness, answer relevancy, context precision and recall. The aggregate alone tells you nothing actionable.

Why does a domain expert need to see it?

Because the metrics are proxies. Whether a faithfulness score of 0.6 is a real problem depends on the answer's content, and the person who can judge that is rarely the person running the eval.

Can I compare two runs?

PATCH the same report id after each eval and the revisions are your comparison — a prompt change or a retriever swap shows up as a diff between two published runs.

Can experts comment on one bad answer?

Yes. Threads anchor to the rendered content, so 'this context was retrieved from the deprecated policy doc' sits on that row.

Share a Ragas evaluation

Ragas gives a RAG pipeline something it badly needs: numbers. Faithfulness, answer relevancy, context precision — computed per sample, aggregated at the end.

The aggregate is what ends up in the standup. The per-sample rows are where the actual finding lives, and they end up in a notebook nobody else opens.

Render the per-sample table

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])

df = result.to_pandas()
styled = (
    df.style
      .background_gradient(subset=["faithfulness", "answer_relevancy"], cmap="RdYlGn")
      .format(precision=3)
)
styled.to_html("ragas.html")

The gradient is not decoration — it is what makes the twelve bad rows findable in a table of four hundred. See also sharing a pandas DataFrame as HTML.

Publish it

Drag ragas.html into the app, or from the eval job:

curl -fsS -X PATCH "https://commareports.com/api/v1/reports/$REPORT_ID" \
  -H "Authorization: Bearer $COMMA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile html ragas.html \
        --arg title "RAG eval — $(date +%F)" '{title: $title, html: $html}')"

The metric is a proxy; the expert is the judge

A faithfulness score of 0.62 is not a verdict. It might be a hallucination, or it might be the metric penalising a correct answer that paraphrased the source. Deciding which requires someone who knows the domain — the support lead, the clinician, the compliance officer — and that person is not going to run your notebook.

Anchored threads on the row: "this is retrieving the 2019 policy doc; it should be excluded from the index." Recorded next to the sample. See commenting on HTML.
Revisions, so a prompt change is evaluated as a diff between two published runs rather than two screenshots.
A routine to re-run the eval on a schedule as the corpus drifts.
Access per report — eval sets usually contain real user questions; keep them private or team-scoped. See the sharing model.

Limits

Entry HTML: 5 MB. Assets: 25 MB per file, 250 MB and 500 files total. A table with full retrieved contexts gets large fast — truncate the context column or publish a sample for the shareable view.
60 requests/minute per token.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Publish an eval report →