How do you handle flaky tests, broken CI pipelines, and false positive alarms?

In Simple Words: Flaky tests are toxic to team morale and engineering velocity because they teach engineers to ignore test failures ("just re-run it until it passes"). Treat flaky tests as bugs: quarantine immediately, fix, and verify.

The Flaky Test Protocol:

1. Immediate Quarantine: As soon as a test fails non-deterministically without code changes, move it to a @flaky quarantine suite or skip it with a mandatory tracking ticket.
2. Do Not Block Main: Main branch CI must always remain green and trustworthy.
3. Root Cause Analysis: Common causes include shared mutable state in databases, hardcoded time zones, async race conditions (setTimeout instead of awaiting events), or missing cleanup in afterEach.
4. Automated Stress Testing: Run suspected tests 100 times in a loop (for i in {1..100}) before un-quarantining.

Practical Script & Summary Template

# Automated bash loop to detect flaky tests locally:
for i in {1..50}; do
  echo "Executing test iteration $i..."
  npx jest test/integration/checkout.test.ts --runInBand || { echo "Flaky test failed on iteration $i!"; exit 1; }
done
echo "Test passed 50/50 consecutive runs successfully!"