Needs investigation

test_at_least_one_replica_is_up

test_R12_depth2_two_constraints.py · failed in 20% of captured runs

Forcing read_secondary before set_secondary failed 60% of the time — necessary but not sufficient. This race needs at least two scheduling constraints; the minimal sufficient set was not found within budget, so no patch was generated.

Where the two runs diverge

The same test, twice, on a shared time axis. The outlined pair is the ordering that differs.

The same test twice on a shared time axis. Each operation is anchored at the moment it started. In the failing run read_secondary#0 starts before set_secondary#0, and set_secondary, set_primary never ran at all.
Passing run5 operations in 0.35 ms
set_secondary
read_primary
read_secondary
assert
set_primary
Failing run3 operations in 0.14 ms
read_primary
read_secondary
assert
set_secondary — never ran
set_primary — never ran
0 ms0.18 ms0.35 ms

Outlined: read_secondary#0 starts before set_secondary#0 in the failing run, and after it when the test passes.

set_secondary, set_primary never started in the failing run. The run flushed its spans normally, so that absence is evidence: the operation had not happened by the time the assertion read the state.

Evidence

Suspicion comes from comparing runs. The decision comes from forcing the ordering and seeing what happens.

OrderingSuspiciousnessWhen forcedVerdict
read_secondary#0 → set_secondary#0the assertion depends on this0.71fails 60%reproduces it sometimes — necessary, but not on its own
read_primary#0 → set_primary#0the assertion depends on this0.53fails 40%reproduces it sometimes — necessary, but not on its own

No single ordering reproduces this failure on its own, so it needs at least two scheduling constraints at once. ChronoTrace reports that rather than patching the nearest symptom.

Policy gate

Every check the proposed patch had to pass before it was allowed to run.

No patch was proposed, so there was nothing for the policy gate to review.

Verification

What was established, and at which strength. A weaker check is never presented as proof.

Verification did not run: no patch reached it.

← All incidents