Comma

Share an HTML report from Dagster

Dagster is unusually good at knowing what happened: the asset materialized, the check passed, here is the metadata. It is a data platform, though, and the report your pipeline produced still has to reach a person who does not use it.

  • Materialization metadata is a summary, not a document. Markdown metadata on an asset is genuinely useful in the UI, and it is not where a 40-row quality table with charts belongs.
  • The webserver is behind a login. Usually an internal one. The analyst, the finance partner and the customer waiting on the number are outside it.
  • The IO manager writes to storage, not to an address. A bucket path is something your code can read. Turning it into something a person can read means handing out bucket credentials.
  • The asset is fresh, and nobody can see it. Which is the failure mode worth fixing, because freshness is the thing Dagster gets right.

Publish from the asset

Publishing is one call at the end of the asset function, and the URL is worth attaching as metadata so the link is discoverable from the UI too:

import json, os, urllib.request
import dagster as dg

REPORT_ID = "rep_…"  # created once; PATCHed from every materialization

@dg.asset
def revenue_quality_report(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    html = build_report_html()
    body = json.dumps({"title": "Revenue data quality", "html": html}).encode()
    req = urllib.request.Request(
        f"https://commareports.com/api/v1/reports/{REPORT_ID}",
        data=body,
        method="PATCH",
        headers={
            "Authorization": f"Bearer {os.environ['COMMA_API_TOKEN']}",
            "Content-Type": "application/json",
        },
    )
    urllib.request.urlopen(req).read()
    return dg.MaterializeResult(
        metadata={"report": dg.MetadataValue.url(f"https://commareports.com/p/{REPORT_ID}")}
    )

Two things make it work in practice:

  • PATCH a saved id, don't POST. One report per asset, a revision per materialization. That is what lets someone compare this run to last Tuesday's instead of hunting through a folder of dated links.
  • Publish on failure too. Wire a run-failure or asset-check-failure sensor that publishes the digest. The interesting report is the one from the run that went wrong.

Scope the token to reports:write and keep it in a resource backed by your secrets manager — see API tokens.

What the reader gets

  • A URL and nothing to install. No Dagster login, no bucket credentials, no seat.
  • The document, rendered. Great Expectations output, a Sweetviz profile, a Plotly figure — see sharing Great Expectations data docs and sharing an EDA report.
  • A comment on the failing check, anchored to that row — see commenting on HTML.
  • A revision history that answers "when did this metric drift", which is the question a data platform gets asked most.

Worth knowing

  • Partitioned assets. One report per partition is a lot of links; usually the right shape is one report per asset, PATCHed by a summary step, with the partition breakdown inside it.
  • Rate limit: 60 requests/minute per token. A fan-out over hundreds of partitions should publish once, from downstream.
  • HTML body: 5 MB. Publish the aggregate; attach the extract.
  • Multi-file output — dbt docs, a profiling directory — needs the folder treatment, see sharing an HTML folder and sharing dbt docs.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Create your first report →

Related