Behavioral & Leadership (50 Qs)•Part 4: Production Incident Management & Post-Mortems (37 - 44)
Walk me through your end-to-end incident response protocol during an active P0/P1 production outage.
In Simple Words: Follow a militarized, calm protocol: Acknowledge → Appoint Roles → Mitigate Customer Impact → Restore → Post-Mortem.
Step-by-Step Incident Lifecycle:
- Triage & Page: Alert triggers in PagerDuty; primary on-call engineer acknowledges within 5 minutes.
- Appoint Incident Commander (IC): The IC leads the incident bridge (Zoom/Slack), assigns investigation streams, and approves rollbacks.
- Appoint Scribe / Communications Lead: Posts internal status updates every 15 minutes to
#incident-announcementsand updates customer-facing status page (e.g. status.company.com). - Prioritize Mitigation over Root Cause: Never spend 2 hours debugging live production under an outage; rollback immediately or enable feature-flag kill switches to restore traffic.
- Close Incident & Schedule Post-Mortem: Once metrics return to baseline, close incident bridge and schedule blameless post-mortem within 48 hours.
Practical Script & Summary Template
markdown
### Live Incident Slack Update Format:
- **Status**: [INVESTIGATING / MITIGATED / RESOLVED]
- **Current Impact**: Checkout API returning 500 errors for ~15% of North America users.
- **Actions Taken**: Rolled back PR #941 to commit `a8f9c21`; database CPU dropped back to 35%.
- **Next Steps**: Monitoring error rates for 15 minutes before officially closing the incident.
- **Next Update**: 14:30 UTC