What is test quarantine?

Test quarantine is moving a known-flaky test out of the set of tests that can fail the build, while it keeps running and reporting. The build stays trustworthy, and the test keeps producing the evidence needed to fix it.

How is quarantining a test different from skipping it?

A skipped test does not run, so it produces no evidence and nobody notices when it would have started passing — or failing for a new reason. A quarantined test still runs on every build in a non-blocking job; only its result stops gating the merge.

How do you quarantine a test?

Tag it, exclude the tag from the gating job, and run the tag in a separate job that is allowed to fail. In Playwright that is a `@quarantine` tag with `--grep-invert` in the gate; in pytest a registered marker with `-m "not quarantine"`; in Cypress a tag filtered by a grep plugin.

How long should a test stay in quarantine?

Briefly, with an owner and a date. Teams that quarantine without an expiry accumulate a second, silent suite that nobody reads. A common rule is: every quarantined test has a ticket, an owner and a review date, and the quarantine set has a size cap.

Test quarantine

Test quarantine is moving a known-flaky test out of the build gate while it keeps running and reporting. The merge stops depending on it; the test does not stop producing evidence.

It exists because the two obvious responses to a flaky test are both bad. Leave it in the gate and the team learns to re-run red builds without reading them. Delete or skip it and the coverage — and the evidence needed to fix it — disappears.

Quarantine vs skip vs retry

	Still runs	Still reported	Can fail the build
Normal test	Yes	Yes	Yes
Quarantined	Yes	Yes	No
Skipped (`skip`, `fixme`)	No	As skipped	No
Retried until green	Yes	Usually hidden	Only if every try fails

Retries are the trap. They keep the test in the gate but throw away the signal, and most setups keep only the passing attempt's artifacts.

How to set it up

The pattern is the same everywhere: tag, exclude from the gate, run the tag in a job that is allowed to fail.

Playwright — tags are built in:

test("checkout applies coupon", { tag: "@quarantine" }, async ({ page }) => {
  /* … */
});

npx playwright test --grep-invert @quarantine   # gating job
npx playwright test --grep @quarantine          # non-blocking job

pytest — register a marker so typos fail loudly:

# pytest.ini
[pytest]
markers = quarantine: known flaky, does not gate merges

pytest -m "not quarantine"   # gating job
pytest -m quarantine         # non-blocking job

Cypress — tag the test and filter with a grep plugin such as @cypress/grep, using an inverted tag in the gating run.

In CI, the non-blocking job is continue-on-error: true on GitHub Actions or allow_failure: true on GitLab — it still shows red, it just does not block.

The rules that keep it honest

Quarantine without rules becomes a second suite that nobody reads. The ones that work:

An owner and a ticket for every quarantined test, at the moment it is quarantined.
A review date. Past it, the test is fixed, deleted, or explicitly re-quarantined by someone who looked.
A size cap. When the quarantine set is full, nothing new goes in until something comes out.
Exit on evidence. A test leaves quarantine after a run of consecutive passes, not after someone thinks it is fixed.

The evidence is the point

Quarantine only works if somebody reads the quarantined job's output. That output — the trace, the failure screenshot, the timings across runs — lives in the runner's HTML report, and in most pipelines that report is a build artifact that expires before the review date comes round.

Publish the quarantine job's report to the same link on every run and each build becomes a revision: the owner opens one URL and sees the latest failure, with earlier runs still in the history. Notes like "fails only on the 4-core runner" go on the failing test itself, where the next person to pick it up will see them (comment on HTML).

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Publish a test report →