DaemonSets¶
What You'll Learn¶
- Why some workloads need "exactly one pod per node," not a fixed replica count
- How DaemonSet update strategies differ from a Deployment's
- How to target a DaemonSet at a subset of nodes with node selectors and tolerations
Why This Matters¶
A Deployment answers "how many replicas do I want." A DaemonSet answers a different question: "this pod needs to exist on every node (or every node of a certain kind), automatically, including nodes that don't exist yet." That's the right shape for infrastructure-level agents, and the wrong shape for application replicas — mixing the two up either under-provisions your log pipeline or wastes a node running an app pod nobody scheduled.
Mental Model¶
A DaemonSet doesn't have a
replicasfield. Its "replica count" is however many nodes match its scheduling rules — the DaemonSet controller adds a pod when a matching node joins, and removes it when the node leaves or stops matching.
Typical uses:
| Category | Examples |
|---|---|
| Log/metrics collection | Fluent Bit, Filebeat, Datadog Agent, node-exporter |
| Networking (CNI) | Calico, Cilium, Flannel — the per-node network agent itself |
| Storage | CSI node plugins (csi-node-driver) |
| Security | Falco, node-level admission/scanning agents |
How It Works¶
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-log-collector
namespace: observability
spec:
selector:
matchLabels:
app: node-log-collector
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
template:
metadata:
labels:
app: node-log-collector
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
nodeSelector:
kubernetes.io/os: linux
containers:
- name: fluent-bit
image: fluent/fluent-bit:4.0.3
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
volumeMounts:
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
Update strategies¶
| Strategy | Behavior |
|---|---|
RollingUpdate (default) |
Replaces pods node-by-node, respecting maxUnavailable (percentage or count) |
OnDelete |
New pod spec is staged but a node's pod is only replaced when you manually delete it |
RollingUpdate is almost always right for agents you want to upgrade automatically; OnDelete is useful when you need to control the exact rollout pace by hand (e.g. upgrading a CNI plugin one node at a time with manual verification between each).
Targeting a subset of nodes¶
DaemonSets run on all nodes by default, including control-plane nodes if their taints are tolerated. To restrict to a subset, combine:
nodeSelector/ node affinity — restrict which nodes can run the pod (e.g.disktype: ssd)tolerations— allow the pod to run on tainted nodes (e.g. control-plane nodes taintedNoSchedule) it would otherwise skip
nodeSelector:
workload-type: gpu
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
This pattern — a GPU-monitoring DaemonSet that only runs on GPU nodes — is common: the agent only makes sense on nodes with the hardware, and the taint keeps ordinary application pods off those (usually more expensive) nodes in the first place.
Affinity and taints in depth
Node/pod affinity, taints, and tolerations are covered fully in Scheduling, Affinity, and Taints — this page only covers the DaemonSet-specific usage.
What Most Engineers Miss¶
-
A DaemonSet's API load is multiplied by your node count. An agent that lists every Pod in the cluster from every node makes 500 expensive LIST calls on a 500-node cluster, every time it resyncs. Make each agent watch only its own node, through an informer:
-
So are its requests. A 200m CPU, 256Mi request is small per node and 100 CPUs plus 125 GiB across 500 nodes, reserved before any application Pod schedules. Size node agents carefully.
- One node at a time is slow on big clusters. The default
maxUnavailable: 1with readiness gates means a rollout across 1,000 nodes takes 1,000 sequential steps. Use a percentage for agents that can tolerate it, ormaxSurge(stable since v1.25) to start the new Pod before stopping the old one. - Cordons don't stop DaemonSets. DaemonSet Pods tolerate
unschedulable,not-ready, andunreachableautomatically, andkubectl drainrefuses to run until you pass--ignore-daemonsets. - Critical agents need a priority. Give CNI, CSI, and log agents
priorityClassName: system-node-critical, so they aren't the Pods evicted or preempted when a node runs short.
Real incident: OpenAI, December 2024
A new telemetry service made every node in each cluster run resource-intensive Kubernetes API operations whose cost grew with cluster size. The API servers in most of OpenAI's large clusters were overwhelmed, DNS-based service discovery failed once caches expired, and ChatGPT, the API, and Sora were down for hours, while the fix itself needed the API servers that were down. What to change in your cluster.
Common Mistakes¶
- Setting resource requests too high on a DaemonSet — since it runs on every node, an oversized request multiplies across the whole cluster and eats capacity every other workload could have used.
- Forgetting control-plane taints — a DaemonSet meant to run everywhere silently skips control-plane nodes unless it tolerates their taint.
- Using a Deployment with a high replica count and anti-affinity rules to approximate "one per node" — it's fragile and doesn't auto-adjust as nodes join or leave, unlike a DaemonSet.
- Not setting
maxUnavailabledeliberately — the default rolling update pace may be too aggressive for a CNI plugin, where taking down too many nodes' network agents at once can partition the cluster.
Check Yourself¶
Draining a node
kubectl drain node-7 stops with an error about DaemonSet-managed Pods. What's the usual fix?
- Delete the DaemonSet first
- Rerun with
--ignore-daemonsets: DaemonSet Pods aren't evicted, and they'd come straight back anyway - Cordon the node and wait
Drain evicts everything else and leaves DaemonSet Pods running, which is what you want for log and network agents during maintenance.
Cheap on one node
A monitoring agent requests 200m CPU. The cluster has 400 nodes. How much CPU does the DaemonSet reserve?
- 200m
- 80 CPUs: 200m on each of 400 nodes
- Nothing, because DaemonSet Pods don't count against node capacity
DaemonSet requests are reserved on every node they run on. On large clusters, a small per-node request is a big line on the bill.
Interview Questions¶
- Why doesn't a DaemonSet have a
replicasfield, and how does its pod count actually get determined? - How would you run a DaemonSet only on GPU nodes without also scheduling ordinary pods there?
- What's the difference between
RollingUpdateandOnDeleteupdate strategies for a DaemonSet, and when would you pickOnDelete?
See Interview Prep for full answers.
Next¶
Continue to Scheduling, Affinity, and Taints to see the general mechanism DaemonSets, Deployments, and StatefulSets all rely on to land pods on the right nodes.