# Daily Claude Eval Refresh, on Autopilot | Comma Routines

Canonical: https://commareports.com/features/routines/daily-claude-eval-refresh

> Run your Claude eval harness every morning. The results post as a new revision on a shared report; reviewers see the diff against yesterday and pin anchored comments. No babysitting.

# Daily Claude eval refresh, on autopilot

If your team runs Claude (or any model) on a recurring task, you have an
eval harness. The hard part isn't the eval — it's making sure someone
actually runs it every day, sees the result, and notices when it regresses.

This is what that loop looks like as a routine.

## The setup

- **Skill.** Your eval harness, packaged as a Claude skill. It pulls today's
  test set, runs the prompts through the model, scores them, and emits an
  HTML report with the scoreboard, the regressions, and the diffs.
- **Report.** A single Comma report named something like _Claude eval —
  production prompts_. Every routine run posts the new HTML as the next
  revision.
- **Cadence.** Daily, hourly, or whatever matches your release cycle — any
  cadence, on any plan.
- **Cost ceiling.** Per-run cap tuned to the size of your eval set.

## The shape, day by day

**Monday 09:00 UTC.** Routine fires. The eval runs against today's prompts.
HTML revision lands on the eval report. Pings the eng-quality channel.

**Monday 09:15.** Lead opens the link. Notices that a regression appeared on
the "summarize support ticket" task. Selects the offending row in the
results table and pins a comment: _"This one started after the prompt change
Friday — investigate."_

**Tuesday 09:00 UTC.** Routine fires again. New revision lands. The comment
from Monday is still there, anchored to the same row. The diff against
yesterday is one click. The investigation continues without anyone re-finding
context.

**Friday.** Two regressions are fixed, one is a false positive. Comments
get resolved. The historical revisions stay; the audit trail is intact.

This is the loop that's hard to keep running on a laptop. It's not hard on a
routine.

## Why a Comma report and not a dashboard

The eval result is HTML the agent already produces. Sending it to a dashboard
tool means rebuilding it in that tool's schema. Sending it to Slack means
losing the per-row anchoring. Sending it to email means losing the
conversation. The Comma report keeps the original HTML and adds the comment
layer outside the iframe — the eval keeps its shape and the discussion
attaches in the right places.

## What it costs

You pay only for the AI compute each run uses, billed one of two ways:

- **Prepaid credit.** Use Comma's keys and draw down a prepaid balance (at
  cost plus 20%). A daily eval that runs in a few hundred tokens costs cents;
  hosted runs pause at a $0 balance rather than overflowing.
- **Bring your own Bedrock keys.** Bill the spend straight to your own AWS
  account — free on Comma, on any plan.

If the eval set grows, the per-run cap protects the budget. Runaway prompts
pause rather than overflow.

## Setup in three steps

1. **Wrap the harness as a skill.** A short Markdown skill that says "run
   the eval against the production prompt set, emit HTML."
2. **Create the eval report.** Either upload your current eval HTML or
   start with a placeholder.
3. **Wire the routine.** From the report, **Add routine →** pick the skill,
   pick daily cadence, set the cap.

That's it. The next morning, the routine has already run.

## Variations

- **Daily regression sweep.** Tighter eval set, runs at 06:00 UTC. Pings
  eng-quality only when scores drop below a threshold.
- **PR-triggered eval.** Combine a routine (daily baseline) with an MCP-driven
  one-off run an agent fires when a PR lands. Same report, different
  revisions.
- **Per-model eval.** One routine per model, all posting to the same report
  family — easier to compare drift across versions.

## Try it

**[Create your first routine →](https://commareports.com/)**

### Related

- [Routines — full feature reference](/features/routines)
- [Run a Claude skill on a schedule](/features/routines/run-claude-skill-on-schedule)
- [Hosted cron for AI agents](/features/routines/cron-for-ai-agents)