DaemonSets & Job/CronJob Controllers: Concurrency policies, node maintenance, and backoff limits.

DaemonSets, Jobs & CronJobs

While Deployments and StatefulSets run continuous workloads, DaemonSets guarantee node coverage, and Jobs handle finite run-to-completion tasks.

1. DaemonSet Architecture:
   [ Master ]
       |
       +---> [ Node 1 ] -> Runs exactly 1 replica (FluentBit / CNI / NodeExporter)
       +---> [ Node 2 ] -> Runs exactly 1 replica
       +---> [ Node 3 ] -> Runs exactly 1 replica

2. CronJob Lifecycle:
   [ CronJob Schedule: "0 * * * *" ]
                   |
                   v
   [ Spawns Job ] ---> [ Creates Pod ] ---> Runs to completion (Exit Code 0)

1. DaemonSets for Node-Level Agents

• Automatically schedules one Pod per node (using Node Affinity and Tolerations for control plane taints).
• When a new worker node joins the cluster, the DaemonSet controller automatically spins up a pod on it.
• Use Cases: Log collection (FluentBit, Promtail), host monitoring (Node Exporter, Datadog), storage daemons (Ceph), and network plugins (Cilium, Calico).

2. Jobs & CronJobs Concurrency Policies

When scheduling recurring CronJobs, if a previous run takes longer than the schedule interval, the concurrencyPolicy controls execution:

• Allow (Default): Spawns new Job concurrently. Risk: Runaway jobs exhausting database connections.
• Forbid (Recommended for batch processing): Skips the new run if the previous one is still executing.
• Replace: Cancels the currently running Job and starts a new one.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-reconciliation
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 3 # Retry failed pod at most 3 times before failing Job
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: job
              image: registry.example.com/batch:v1