Flaky Test Management

Pathrule1 Rule • 2 Memories • 2 Skills

A flaky test has at least two outcomes for the same intended inputs because time, order, shared state, concurrency, network, environment, randomness, or the product itself is nondeterministic. This pattern constrains retry interpretation, records quarantine and evidence ownership, and supplies separate flake-triage and repair-verification procedures. It complements the Vitest and Playwright pattern by focusing on organizational and diagnostic handling after tests exist, independent of the specific test runner.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

triage-flaky-test

verify-flake-repair

.github/

workflows/

Retries collect evidence but never erase failure

docs/

testing/

Quarantine is a visible temporary risk decision

Flake classes guide the first experiment

Rules

Retries collect evidence but never erase failure/.github/workflowshighstrictRecord first-attempt failure, retry count, final outcome, seed, order, worker, shard, timing, and artifacts as a flaky result.

1	Retries can keep a pipeline moving while hiding reliability loss. A pass on retry proves nondeterminism or changed conditions, not correctness of the first execution.
2
3	- Mark the test and job as flaky or unstable when any attempt fails, even if a later attempt passes.
4	- Preserve logs, screenshots, traces, video, network evidence, seed, test order, worker, shard, platform, and timing from the first failure.
5	- Use a small bounded retry count only where the team accepts the release tradeoff; do not apply unlimited or suite-wide retries silently.
6	- Track first-attempt pass rate and flake frequency by stable test identity so chronic failures cannot disappear in aggregate green status.
7
8	See /docs/testing for the adjacent decision or procedure that completes this constraint.

Memories

Quarantine is a visible temporary risk decision/docs/testingMove a test out of the blocking lane only with an owner, reason, evidence link, scope, and expiry or repair target.

1	A quarantined test no longer protects the main release path. Without explicit ownership and visibility, quarantine becomes deletion with extra syntax.
2
3	- Keep the test running in a non-blocking lane when possible so frequency and artifacts continue to accumulate.
4	- Record whether the suspected risk is test-only or product behavior and what coverage is temporarily missing.
5	- Assign an owner and review or expiry date and surface overdue quarantine in the same engineering workflow as other release risks.
6	- Restore blocking status only after the causal repair passes confidence verification under the triggering conditions.
7
8	See /tests for the rule or workflow that puts this decision into practice.

Flake classes guide the first experiment/docs/testingClassify timing, order, state, concurrency, network, resource, randomness, environment, and product races before changing assertions.

1	Different flake classes require different evidence. Increasing a timeout cannot repair state leakage, and clearing shared storage cannot repair an event-order race.
2
3	- Timing flakes wait for elapsed time instead of observable state; replace sleeps with bounded condition or event synchronization.
4	- Order and state flakes depend on prior tests, shared accounts, caches, files, ports, databases, or globals; randomize order and isolate ownership.
5	- Concurrency and product races require controlled interleaving and event evidence; do not weaken assertions until the product invariant is understood.
6	- Environment and resource flakes correlate with runner, shard, CPU, memory, clock, browser, network, or dependency saturation and need those dimensions captured.
7
8	See /.github/workflows for the rule or workflow that puts this decision into practice.

Skills

triage-flaky-test/rootCapture the failure, estimate frequency, classify nondeterminism, choose quarantine, and assign one falsifiable experiment.

1	---
2	name: triage-flaky-test
3	description: Triage a test that passes and fails across repeated or CI executions.
4	---
5
6	# Triage Flaky Test
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	- [ ] Preserve the first failing artifacts and identify stable test, revision, environment, shard, worker, seed, order, timing, and dependency state.
11	- [ ] Repeat under the same conditions enough to estimate frequency, then vary one dimension such as order, worker count, load, clock, or network.
12	- [ ] Classify the dominant flake mechanism and decide whether the intermittent behavior is in the product, test, fixture, environment, or dependency.
13	- [ ] Quarantine only when release risk requires it and attach owner, reason, missing coverage, evidence, and review deadline.
14	- [ ] State one hypothesis with a predicted result and implement the smallest diagnostic probe before changing waits, retries, or assertions.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

verify-flake-repair/rootRun the triggering conditions at sufficient repetition and concurrency, then remove retries and quarantine with preserved evidence.

1	---
2	name: verify-flake-repair
3	description: Verify that a flaky-test repair removed the causal nondeterminism rather than lowering observation probability.
4	---
5
6	# Verify Flake Repair
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Reproduce the old failure on the prior code with the captured seed, order, state, timing, load, and environment or document why exact replay is impossible.
11	2. Apply the targeted synchronization, isolation, ownership, or product fix without weakening the asserted behavior.
12	3. Run focused repetitions and parallel or shuffled suites at a volume justified by the prior flake frequency and release risk.
13	4. Run the ordinary full CI matrix without retry assistance and monitor first-attempt results after merge.
14	5. Remove quarantine and temporary diagnostics, retain a stable regression signal, and document the cause and confidence evidence.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often add retries until CI turns green, increase timeouts without identifying the missing condition, or delete an intermittent test that is exposing a real product race.

Built for Engineering teams maintaining CI suites whose intermittent failures slow or undermine releases.

Keeps your assistant from:

Counting a retry pass as a clean test result
Leaving quarantined tests without owner or repair deadline
Masking a product race as test-only instability
Declaring a low-frequency flake repaired after one passing run

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25