Flaky test
A flaky test is one that both passes and fails against the same code, with no change to the code under test. The failure genuinely happened; what is unreliable is the answer, not the run.
Flakiness is expensive in a specific way: it trains a team to re-run red builds without reading them, and a suite that is always re-run is a suite that no longer gates anything.
The six usual causes
| Cause | Signature |
|---|---|
| Timing / implicit waits | Fails on a slow runner, passes locally |
| Shared state | Fails only when run after a specific other test |
| Order dependence | Fails under a randomised or sharded order |
| Real network or clock | Fails at midnight, at month end, or when an API is slow |
| Concurrency in the code | Fails rarely, everywhere, with no pattern |
| Runner resource contention | Whole suites go red together on a busy agent |
The last one is worth checking before any of the others. A suite that goes flaky the week CI parallelism was raised is not six new races.
Retries hide, they do not fix
--retries 2 turns a red build green while leaving the race in place.
Worse, most setups then discard the failing attempt's artifacts and keep
only the passing one — deleting the exact evidence needed to diagnose it.
If retries are on, keep the first attempt's trace. Playwright, Cypress, pytest-rerunfailures and most modern runners record per-attempt output; the reporter setting for retaining it matters more than the retry count.
The evidence is in the HTML report
A flaky failure is by definition the one you cannot reproduce on demand. That makes the failing run's own artifacts the only evidence there is:
- The trace — every action, network call and DOM snapshot, replayable step by step (Playwright reports).
- The screenshot at the moment of failure, which frequently shows a spinner still running.
- The network log, which shows the request that hadn't resolved.
- Per-test timings across runs, which identify runner contention.
None of that survives JUnit XML. All of it lives in the runner's HTML report — which in most pipelines is uploaded as a build artifact and expires under retention before the flakiness is understood.
What publishing changes
Publish the report per run to a stable URL and the diagnosis becomes possible: a permanent link to the run where it failed, opened by whoever picks it up on Monday, with the trace intact. Threads attached to a specific assertion persist across revisions, so "we decided this one was runner contention in March" is still readable next to the test in September (comment on HTML).
Try it
Comma is free — unlimited reports, unlimited commenters, unlimited revision history.