How do you establish engineering observability, Service Level Objectives (SLOs), and error budgets?

In Simple Words: Observability is about understanding system state from external outputs. Use the Four Golden Signals (Latency, Traffic, Errors, Saturation) and define user-centric SLOs.

The Core Concepts:

• SLI (Service Level Indicator): What you measure (e.g. % of successful HTTP requests returning in <300ms).
• SLO (Service Level Objective): Target agreed upon with business (e.g. 99.9% of requests return <300ms over a rolling 30-day window).
• Error Budget: The allowable unreliability (0.1% = ~43 minutes of downtime per month). If the error budget is exhausted, freeze feature work to focus on reliability.

Practical Script & Summary Template

// OpenTelemetry middleware recording the Four Golden Signals in Node.js:
import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('checkout-service');
const httpRequestDuration = meter.createHistogram('http_request_duration_ms', {
  description: 'Measures HTTP request latency in milliseconds'
});
const httpErrorsCounter = meter.createCounter('http_errors_total', {
  description: 'Counts total HTTP 5xx responses'
});