Share an Evidently drift report
Model monitoring has a distribution problem shaped exactly like the
notebook problem, because it usually is a notebook. Evidently produces a
genuinely good self-contained HTML report — drift per feature,
distribution plots, test results — and then it lives in the ML engineer's
reports/ directory, where the people who could explain the drift never
see it.
And drift is almost always explained by someone outside the model. The upstream vendor changed units. Marketing ran a campaign in a new region. A form field became optional. None of that is visible in the data; all of it is one sentence from someone who isn't running the notebook.
Publish it
import os, requests
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref, current_data=cur)
report.save_html("drift.html")
requests.patch(
f"https://commareports.com/api/v1/reports/{os.environ['COMMA_REPORT_ID']}",
headers={"Authorization": f"Bearer {os.environ['COMMA_API_TOKEN']}"},
json={"title": "Churn model — drift", "html": open("drift.html").read()},
timeout=30,
)
One report id per model, PATCHed on every run: one URL, a revision per
run, and diffs between any two. Use a
scoped token with reports:write.