Q12: The etcd Distributed Datastore: Raft consensus, quorum calculation, snapshot backups, and write latency. - DevOps & Kubernetes (50 Qs)

Architecture & Mechanics

etcd is a strongly consistent (CP under CAP theorem), distributed key-value store using the Raft consensus algorithm. Every Kubernetes object (Pods, Deployments, Secrets, CRDs) is stored in etcd under keys prefixed with /registry/.

lua

       +-------------------------------------------------------------+
       |                  etcd 3-Node Quorum Cluster                 |
       |                                                             |
       |       +------------+                +------------+          |
       |       |  etcd 01   |<--- Heartbeat->|  etcd 02   |          |
       |       |  (LEADER)  |                | (FOLLOWER) |          |
       |       +------------+                +------------+          |
       |             ^                             ^                 |
       |             |                             |                 |
       |             +------------+   +------------+                 |
       |                          v   v                              |
       |                      +------------+                         |
       |                      |  etcd 03   |                         |
       |                      | (FOLLOWER) |                         |
       |                      +------------+                         |
       +-------------------------------------------------------------+

1. Quorum Calculation & Node Sizing

To tolerate node failures while avoiding split-brain scenarios, etcd requires a strict majority quorum: $$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$$

3 nodes: Tolerates 1 node failure ($3/2 + 1 = 2$ quorum required).
5 nodes: Tolerates 2 node failures ($5/2 + 1 = 3$ quorum required).
Always use odd numbers of nodes (3 or 5). Adding a 4th node increases network sync overhead without increasing fault tolerance (both 3 and 4 nodes can only tolerate 1 failure).

2. Why Disk Write Latency is Critical

etcd synchronously writes its Write-Ahead Log (WAL) to disk via fdatasync before acknowledging writes to the Raft cluster. If disk write latency exceeds 10–20ms (e.g. slow magnetic storage or throttled cloud IOPS), leader heartbeats time out, triggering frequent election storms and rendering the Kubernetes API unresponsive.

3. Disaster Recovery: Snapshot Backup & Restore

bash

# 1. Take consistent point-in-time snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/backup/etcd-snapshot.db

# 2. Verify snapshot integrity
ETCDCTL_API=3 etcdctl --write-out=table snapshot status /opt/backup/etcd-snapshot.db

# 3. Restore snapshot to new data directory
ETCDCTL_API=3 etcdctl snapshot restore /opt/backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restored