InterviewHubPRO
Section Overview
Q37

Question 37 of 50

Part 4: Production Incident Management & Post-Mortems (37 - 44)

74%
Behavioral & Leadership (50 Qs)•Part 4: Production Incident Management & Post-Mortems (37 - 44)

Walk me through your end-to-end incident response protocol during an active P0/P1 production outage.

In Simple Words: Follow a militarized, calm protocol: Acknowledge → Appoint Roles → Mitigate Customer Impact → Restore → Post-Mortem.

Step-by-Step Incident Lifecycle:

  1. Triage & Page: Alert triggers in PagerDuty; primary on-call engineer acknowledges within 5 minutes.
  2. Appoint Incident Commander (IC): The IC leads the incident bridge (Zoom/Slack), assigns investigation streams, and approves rollbacks.
  3. Appoint Scribe / Communications Lead: Posts internal status updates every 15 minutes to #incident-announcements and updates customer-facing status page (e.g. status.company.com).
  4. Prioritize Mitigation over Root Cause: Never spend 2 hours debugging live production under an outage; rollback immediately or enable feature-flag kill switches to restore traffic.
  5. Close Incident & Schedule Post-Mortem: Once metrics return to baseline, close incident bridge and schedule blameless post-mortem within 48 hours.

Practical Script & Summary Template

markdown
### Live Incident Slack Update Format:
- **Status**: [INVESTIGATING / MITIGATED / RESOLVED]
- **Current Impact**: Checkout API returning 500 errors for ~15% of North America users.
- **Actions Taken**: Rolled back PR #941 to commit `a8f9c21`; database CPU dropped back to 35%.
- **Next Steps**: Monitoring error rates for 15 minutes before officially closing the incident.
- **Next Update**: 14:30 UTC