Q47: Prometheus Architecture & PromQL Fundamentals: Pull-based scraping, TSDB storage, and essential PromQL queries. - DevOps & Kubernetes (50 Qs)

Prometheus Core Mechanics

Prometheus uses a pull-based scraping model over HTTP, collecting metrics from targets registered via Kubernetes Service Discovery.

scss

[ K8s API Server Service Discovery ]
                 |
                 v
+------------------------------------+         +------------------------+
|       Prometheus Server            | ------> |      Alertmanager      |
|  - Retrieval (Pulls /metrics HTTP) | (Alert) | - Deduplication        |
|  - Storage (Local TSDB Chunking)   |         | - Routing to Slack/PD  |
|  - PromQL Engine                   |         +------------------------+
+------------------------------------+
                 ^
                 | PromQL Queries
       [ Grafana Dashboards ]

Production PromQL Query Cheat Sheet

HTTP Request Rate per second (QPS) over the last 5 minutes:

promql

sum(rate(http_requests_total{status=~"2.."}[5m])) by (service)

HTTP Error Rate Percentage (4xx / 5xx):

promql

(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100

95th Percentile Latency (p95) across API endpoints:

promql

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint))

Pods Near Memory Limits (Identify OOM Risks):

promql

(container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) > 0.85