How do you structure on-call rotations to prevent engineer burnout and alert fatigue?

In Simple Words: Every page should be actionable, urgent, and customer-impacting. If an alert fires at 3 AM and the engineer just hits "acknowledge" and goes back to sleep, it should be a Jira ticket, not a pager alarm.

Best Practices:

1. Primary + Secondary Rotation: Weekly shifts where the primary handles alerts and small operational tasks, while the secondary acts as backup.
2. On-Call Compensation & Time Off: Offer on-call stipends and compensatory time off after nights with severe escalations.
3. Alert Hygiene & Pager Tuning: Review all pages weekly during team standups; any alert that fired without requiring immediate action must be tuned or deleted.
4. Comprehensive Runbooks: Every alert in PagerDuty must link directly to an up-to-date Markdown runbook with explicit remediation steps.

Practical Script & Summary Template

# Runbook: HighDatabaseConnectionPoolSaturation

## Alert Description
Fires when active connections to primary PostgreSQL exceed 85% of `max_connections` for >3 minutes.

## Severity
P1 - Critical (Risk of service-wide 504 gateway timeouts)

## Step-by-Step Remediation:
1. Open Datadog Database Dashboard: `https://app.datadoghq.com/db/postgres-prod`
2. Identify top connection-consuming services:
```sql
   SELECT client_addr, usename, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC;
```
3. If PgBouncer pool is saturated, restart PgBouncer pods in Kubernetes:
```bash
   kubectl rollout restart deployment/pgbouncer -n production
```
4. If problem persists, escalate to Database Administrator (DBA) on-call.