Comma

Share a LangSmith eval result

LangSmith is very good at the thing it is for: you run an evaluation, you get traces, you click into the run that failed and see every step. For debugging with another engineer, the public share link is perfect.

It is the wrong artifact for everyone else, for two reasons.

It shares everything. A run link exposes the full trace — the system prompt, the retrieved documents, the intermediate tool calls, the raw model output. Those are usually the most sensitive things your pipeline touches. You wanted to show the accuracy number; you shared the prompt engineering, the RAG corpus excerpt and the customer record that happened to be in the context window.

It shares one run. The thing people actually ask about is the comparison — this prompt versus that one, this week versus last. That view lives inside the project, and the project is workspace-scoped, so the domain expert whose judgement you need has to be given a seat in your observability tooling first.

Publish the comparison instead

Export the results you want reviewed, render them, publish once, and update the same report on every run:

import pandas as pd
import requests
from langsmith import Client
from langsmith.evaluation import evaluate

results = evaluate(
    my_chain,
    data="qa-golden-set",
    evaluators=[correctness, groundedness],
    experiment_prefix="prompt-v7",
)

# Only the columns a reviewer needs — not the trace.
df = pd.DataFrame(
    [
        {
            "question": r["example"].inputs["question"],
            "answer": r["run"].outputs["answer"],
            "expected": r["example"].outputs["answer"],
            "correct": r["evaluation_results"]["results"][0].score,
        }
        for r in results
    ]
)

html = f"""<!doctype html><meta charset="utf-8">
<title>prompt-v7 — golden set</title>
<style>body{{font:15px/1.6 system-ui;margin:2rem;max-width:70rem}}
table{{border-collapse:collapse;width:100%}}
td,th{{border:1px solid #ddd;padding:.5rem;vertical-align:top}}
tr.bad{{background:#fff1f0}}</style>
<h1>prompt-v7 — golden set</h1>
<p>Accuracy: {df["correct"].mean():.1%} over {len(df)} examples.</p>
{df.to_html(index=False, escape=True)}"""

requests.patch(
    f"https://commareports.com/api/v1/reports/{REPORT_ID}",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json={"title": "Golden set — prompt-v7", "html": html},
    timeout=30,
).raise_for_status()

LangSmith keeps doing its job — the traces are still there for whoever needs to debug a specific failure. The published report is the part you send.

What the published copy adds

  • You choose what leaves. The report contains the columns you put in it. The system prompt and the retrieved context stay in LangSmith unless you deliberately include them.
  • Reviewers need no seat. Visibility is private, team, domain-gated, or link, decided per report — the clinician, the lawyer or the support lead scoring your outputs does not get added to your observability workspace.
  • Judgement lands on the row. A reviewer highlights the wrong answer and pins "this is technically right but we'd never say it that way" to that example. Twenty of those threads are a labelled dataset — see commenting on HTML.
  • One URL as the eval re-runs. Each run appends a revision at the same address and any two revisions can be diffed, so "when did this regress" stops being an archaeology project.
  • Your agent can read the replies through the MCP server and act on them — see letting an agent respond to comments.

Worth knowing

  • HTML body: 5 MB. A thousand-row eval table fits; ten thousand rows with full generations does not — publish a sampled or aggregated view and attach the full export as an asset.
  • Scope the token to reports:write so the eval job can publish and nothing else — see scoped tokens for AI agents.
  • To re-run it on a schedule, hand the job to a routine instead of a cron box.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Create your first report →

Related