# What Is a Flaky Test? — Definition, Causes and Why the Report Is the Evidence

Canonical: https://commareports.com/glossary/flaky-test
Published: 2026-09-11

> A flaky test passes and fails on the same code. The six causes worth checking, how retries hide them, and why the run artifacts — traces, screenshots, timings — are the only real evidence.

# Flaky test

**A flaky test is one that both passes and fails against the same code,
with no change to the code under test.** The failure genuinely happened;
what is unreliable is the answer, not the run.

Flakiness is expensive in a specific way: it trains a team to re-run red
builds without reading them, and a suite that is always re-run is a suite
that no longer gates anything.

## The six usual causes

| Cause                      | Signature                                               |
| -------------------------- | ------------------------------------------------------- |
| Timing / implicit waits    | Fails on a slow runner, passes locally                  |
| Shared state               | Fails only when run after a specific other test         |
| Order dependence           | Fails under a randomised or sharded order               |
| Real network or clock      | Fails at midnight, at month end, or when an API is slow |
| Concurrency in the code    | Fails rarely, everywhere, with no pattern               |
| Runner resource contention | Whole suites go red together on a busy agent            |

The last one is worth checking before any of the others. A suite that
goes flaky the week CI parallelism was raised is not six new races.

## Retries hide, they do not fix

`--retries 2` turns a red build green while leaving the race in place.
Worse, most setups then discard the failing attempt's artifacts and keep
only the passing one — deleting the exact evidence needed to diagnose it.

If retries are on, keep the first attempt's trace. Playwright, Cypress,
pytest-rerunfailures and most modern runners record per-attempt output;
the reporter setting for retaining it matters more than the retry count.

## The evidence is in the HTML report

A flaky failure is by definition the one you cannot reproduce on demand.
That makes the failing run's own artifacts the only evidence there is:

- The **trace** — every action, network call and DOM snapshot, replayable
  step by step ([Playwright reports](/share-playwright-report)).
- The **screenshot** at the moment of failure, which frequently shows a
  spinner still running.
- The **network log**, which shows the request that hadn't resolved.
- **Per-test timings** across runs, which identify runner contention.

None of that survives [JUnit XML](/glossary/junit-xml). All of it lives in
the runner's [HTML report](/glossary/html-report) — which in most
pipelines is uploaded as a [build artifact](/glossary/build-artifact) and
expires under [retention](/glossary/artifact-retention) before the
flakiness is understood.

## What publishing changes

Publish the report per run to a stable URL and the diagnosis becomes
possible: a permanent link to the run where it failed, opened by whoever
picks it up on Monday, with the trace intact. Threads attached to a
specific assertion persist across revisions, so "we decided this one was
runner contention in March" is still readable next to the test in
September ([comment on HTML](/comment-on-html)).

## Try it

Comma is free — unlimited reports, unlimited commenters, unlimited
revision history.

**[Publish a test report →](https://commareports.com/)**

### Related

- [Share a Playwright report](/share-playwright-report) · [Cypress](/share-cypress-report) · [pytest](/share-pytest-report)
- [JUnit XML](/glossary/junit-xml) · [Mutation testing](/glossary/mutation-testing)
- [QA engineers](/for/qa-engineers) · [Glossary](/glossary)