# Pathrule Pattern: Flaky Test Management (1.0.0) # ::pathrule:package:flaky-test-management ### [RULE] Retries collect evidence but never erase failure (path: /.github/workflows) Retries can keep a pipeline moving while hiding reliability loss. A pass on retry proves nondeterminism or changed conditions, not correctness of the first execution. - Mark the test and job as flaky or unstable when any attempt fails, even if a later attempt passes. - Preserve logs, screenshots, traces, video, network evidence, seed, test order, worker, shard, platform, and timing from the first failure. - Use a small bounded retry count only where the team accepts the release tradeoff; do not apply unlimited or suite-wide retries silently. - Track first-attempt pass rate and flake frequency by stable test identity so chronic failures cannot disappear in aggregate green status. See /docs/testing for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] Quarantine is a visible temporary risk decision (path: /docs/testing) A quarantined test no longer protects the main release path. Without explicit ownership and visibility, quarantine becomes deletion with extra syntax. - Keep the test running in a non-blocking lane when possible so frequency and artifacts continue to accumulate. - Record whether the suspected risk is test-only or product behavior and what coverage is temporarily missing. - Assign an owner and review or expiry date and surface overdue quarantine in the same engineering workflow as other release risks. - Restore blocking status only after the causal repair passes confidence verification under the triggering conditions. See /tests for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Flake classes guide the first experiment (path: /docs/testing) Different flake classes require different evidence. Increasing a timeout cannot repair state leakage, and clearing shared storage cannot repair an event-order race. - Timing flakes wait for elapsed time instead of observable state; replace sleeps with bounded condition or event synchronization. - Order and state flakes depend on prior tests, shared accounts, caches, files, ports, databases, or globals; randomize order and isolate ownership. - Concurrency and product races require controlled interleaving and event evidence; do not weaken assertions until the product invariant is understood. - Environment and resource flakes correlate with runner, shard, CPU, memory, clock, browser, network, or dependency saturation and need those dimensions captured. See /.github/workflows for the rule or workflow that puts this decision into practice. --- ### [SKILL] triage-flaky-test (path: /) --- name: triage-flaky-test description: Triage a test that passes and fails across repeated or CI executions. --- # Triage Flaky Test Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. - [ ] Preserve the first failing artifacts and identify stable test, revision, environment, shard, worker, seed, order, timing, and dependency state. - [ ] Repeat under the same conditions enough to estimate frequency, then vary one dimension such as order, worker count, load, clock, or network. - [ ] Classify the dominant flake mechanism and decide whether the intermittent behavior is in the product, test, fixture, environment, or dependency. - [ ] Quarantine only when release risk requires it and attach owner, reason, missing coverage, evidence, and review deadline. - [ ] State one hypothesis with a predicted result and implement the smallest diagnostic probe before changing waits, retries, or assertions. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks. --- ### [SKILL] verify-flake-repair (path: /) --- name: verify-flake-repair description: Verify that a flaky-test repair removed the causal nondeterminism rather than lowering observation probability. --- # Verify Flake Repair Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Reproduce the old failure on the prior code with the captured seed, order, state, timing, load, and environment or document why exact replay is impossible. 2. Apply the targeted synchronization, isolation, ownership, or product fix without weakening the asserted behavior. 3. Run focused repetitions and parallel or shuffled suites at a volume justified by the prior flake frequency and release risk. 4. Run the ordinary full CI matrix without retry assistance and monitor first-attempt results after merge. 5. Remove quarantine and temporary diagnostics, retain a stable regression signal, and document the cause and confidence evidence. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.