DevOps & Kubernetes (50 Qs)•Part 5: Infrastructure as Code, SRE Observability & Production Troubleshooting (43 - 50)
The Three Pillars of Observability: Metrics, Structured Logs, and Distributed Tracing (OpenTelemetry).
Architecture of Modern Observability
Observability is the ability to infer the internal state of a complex distributed system based on its external outputs.
sql
+-----------------------+ +-----------------------+ +-----------------------+
| METRICS | | LOGS | | TRACES |
| (Prometheus / StatsD) | | (FluentBit / Loki) | | (OpenTelemetry/Jaeger)|
| | | | | |
| Aggregated numeric | | Structured JSON event | | End-to-end request |
| time-series counters, | | records with context: | | spans showing latency |
| gauges, & histograms. | | timestamp, msg, trace | | across microservices. |
| Tells you: "SOMETHING | | Tells you: "WHY IT IS | | Tells you: "WHERE THE |
| IS WRONG & WHERE" | | HAPPENING" | | LATENCY/ERROR OCCURS" |
+-----------------------+ +-----------------------+ +-----------------------+
Distributed Trace Context Propagation (W3C TraceContext)
When an HTTP request enters an API Gateway and traverses downstream microservices, OpenTelemetry automatically injects and extracts the traceparent header:
http
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
| | | |
Version Trace ID (Unique per user call) Parent Span ID Sampled Flag
This unifies logs, metrics, and trace graphs across dozens of disparate services under a single correlation ID.