Comma

Share an MLflow run report

MLflow tracks the experiment properly. Then someone asks "is the new model better?" and you discover that the answer only exists inside a server on your network.

  • The tracking UI is yours to host. A run URL is http://mlflow.internal:5000/#/experiments/3/runs/…, which resolves for people on the VPN with access to that host and for nobody else.
  • Artifacts are bucket objects. mlflow.log_artifact puts your confusion matrix or SHAP plot in S3, GCS or a mounted volume. Reading it means credentials to that store, which the product manager does not have.
  • The comparison view has no shareable form at all. The table of five runs side by side — the actual answer to "is it better" — exists as UI state.

So it becomes a screenshot. The screenshot loses the columns that did not fit, and the reply comes back as "what about recall on the minority class?", which is in the part that got cropped.

Publish the comparison

mlflow.search_runs gives you the whole experiment as a DataFrame. Render it, publish once, and update the same report at the end of every run:

import os
import mlflow
import requests

runs = mlflow.search_runs(
    experiment_names=["churn-v3"],
    order_by=["metrics.val_auc DESC"],
    max_results=20,
)

cols = [
    "tags.mlflow.runName",
    "params.model",
    "params.lr",
    "metrics.val_auc",
    "metrics.val_recall",
    "metrics.train_loss",
    "start_time",
]
table = runs[cols].rename(columns=lambda c: c.split(".")[-1])

html = f"""<!doctype html><meta charset="utf-8">
<title>churn-v3 — run comparison</title>
<style>body{{font:15px/1.6 system-ui;margin:2rem;max-width:72rem}}
table{{border-collapse:collapse;width:100%}}
td,th{{border:1px solid #ddd;padding:.5rem;text-align:left}}
tr:first-child{{background:#fafafa}}</style>
<h1>churn-v3 — run comparison</h1>
<p>Top {len(table)} runs by validation AUC. Best: {table["val_auc"].max():.4f}.</p>
{table.to_html(index=False, escape=True, float_format=lambda v: f"{v:.4f}")}"""

requests.patch(
    f"https://commareports.com/api/v1/reports/{os.environ['COMMA_REPORT_ID']}",
    headers={"Authorization": f"Bearer {os.environ['COMMA_API_TOKEN']}"},
    json={"title": "churn-v3 — run comparison", "html": html},
    timeout=30,
).raise_for_status()

To include the plots as well, log them as usual and attach the image files as assets, or inline a self-contained Plotly figure with fig.to_html(full_html=False, include_plotlyjs="cdn") — the sandbox allows scripts and styles from jsdelivr, unpkg, cdnjs and esm.sh.

Your tracking server does not change. It stays the system of record for parameters, metrics and lineage; the published report is the part you send to people who are never going to open it.

What the published copy adds

  • Readers without tracking-server access. Visibility is private, team, domain-gated, or link, decided per report.
  • It renders. HTML is stored verbatim and served inside a sandboxed iframe with scripts enabled, so a Plotly ROC curve or a SHAP summary stays interactive — see sharing a SHAP plot.
  • Comments anchored to the number. A reviewer highlights the recall column and pins "this is worse than the current model for the segment that matters" to it — see commenting on HTML.
  • One URL across training runs. Revisions accumulate at the same address and any two can be diffed, which makes "did last night's run actually help" a question with an answer.
  • A model card link that keeps working. The URL outlives the tracking server's hostname, the VPN change and the reorg.

Limits

  • HTML body: 5 MB. Large figures and full artifact bundles go in as assets at 25 MB per file, 250 MB per report.
  • A report that fetches sibling data files at view time can't, from the sandbox — inline the data or attach the archive.
  • Rate limit: 60 requests/minute per token.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Create your first report →

Related