Comma

Share an HTML report from Airflow

The DAG builds the report. That part works. The trouble starts one line later, when the task writes report.html and the question becomes who can read it.

  • Worker filesystems are ephemeral. With the Celery or Kubernetes executor the file is on a container that may be gone before you go looking. Even with a local executor, "SSH to the scheduler box" is not a distribution strategy.
  • The bucket moves the problem. S3 or GCS is the standard answer, and it turns "who can read this report" into "who has credentials to this bucket" — which the analyst waiting on the daily refresh does not, and should not need.
  • The Airflow UI won't render it. It shows logs and task state, both behind an Airflow login. It is not a place to publish a document to a business reader.
  • Emailing the file gets you an attachment that opens as a blank page from someone's downloads folder — see why an HTML file won't open.

Publish from the task

One task at the end of the DAG, unconditional, PATCHing a saved report id:

import json, os, urllib.request
from airflow.models import Variable

def publish_report(**_):
    html = open("/tmp/report.html", encoding="utf-8").read()
    report_id = Variable.get("comma_report_id")
    body = json.dumps({"title": "Daily revenue refresh", "html": html}).encode()
    req = urllib.request.Request(
        f"https://commareports.com/api/v1/reports/{report_id}",
        data=body,
        method="PATCH",
        headers={
            "Authorization": f"Bearer {os.environ['COMMA_API_TOKEN']}",
            "Content-Type": "application/json",
        },
    )
    urllib.request.urlopen(req).read()

Wire it as the last task with trigger_rule="all_done" so it runs when an upstream task fails too — that is the run people most want to look at. An on_failure_callback publishing a short failure digest works for the same reason.

Three details make this hold up in practice:

  • PATCH a saved id, don't POST. POSTing creates a new report every night, which is a spray of orphan links nobody bookmarks. One id gives the DAG a permanent URL and a revision per run.
  • Keep the id in an Airflow Variable, so it survives redeploys and is visible to whoever is on support.
  • Scope the token to reports:write and store it in your secrets backend — see API tokens.

What the reader gets

  • A URL, and nothing to install. No Airflow login, no bucket credentials.
  • Faithful rendering of the Plotly figure or the pandas table the DAG generated — see sharing a pandas DataFrame and sharing a Plotly chart.
  • A comment on the row that's wrong, anchored there, rather than a Slack message you have to reconstruct — see commenting on HTML.
  • Yesterday, still readable. The revision history is the audit trail for "when did this number change".

Worth knowing

  • Multi-file output — a directory with index.html and assets beside it — needs the folder treatment, see sharing an HTML folder.
  • HTML body: 5 MB. A dataframe dumped whole will exceed that; publish the aggregate and attach the raw extract.
  • Rate limit: 60 requests/minute per token. A DAG that fans out over 200 partitions should publish once from a summary task, not once per partition.
  • Backfills. A backfill PATCHing the same id will write one revision per backfilled day, in whatever order the scheduler runs them. If that matters, publish backfills to a separate report.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Create your first report →

Related