Eric Campbell 6 min read Infrastructure

Kubernetes DaemonSets: Node-Level Services

DaemonSets are Kubernetes workload controllers that guarantee a pod runs on all (or some) nodes in your cluster. When you add a node, the DaemonSet automatically schedules its pod there. When you...

Key Insights

  • DaemonSets ensure exactly one pod runs on each node (or selected nodes), making them ideal for node-level infrastructure services like log collectors, monitoring agents, and network plugins
  • Unlike Deployments, DaemonSets bypass the scheduler and create pods directly on nodes, requiring careful resource management and security considerations since they often need privileged access
  • Proper use of node selectors, taints/tolerations, and rolling update strategies prevents cluster-wide outages when deploying or updating critical node-level services

Introduction to DaemonSets

DaemonSets are Kubernetes workload controllers that guarantee a pod runs on all (or some) nodes in your cluster. When you add a node, the DaemonSet automatically schedules its pod there. When you remove a node, those pods are garbage collected. This is fundamentally different from Deployments, which distribute a specified number of replicas across the cluster based on scheduler decisions, or StatefulSets, which maintain persistent identities for pods.

The key distinction: DaemonSets are node-centric, not replica-centric. You don’t specify “I want 3 replicas”—you specify “I want this on every node that matches my criteria.”

Here’s a basic DaemonSet for a logging agent:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
      - name: log-agent
        image: fluent/fluent-bit:2.1
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

This manifest creates a pod on every node that reads logs from /var/log. Notice the hostPath volume—a common pattern in DaemonSets since they typically interact with node-level resources.

Common Use Cases

DaemonSets shine in scenarios requiring node-level presence. The most common use cases include:

Log Collection: Every application writes logs. DaemonSets ensure log collectors run on every node to aggregate and forward logs to centralized systems.

Monitoring Agents: Tools like Prometheus Node Exporter, Datadog agents, or custom metric collectors need to run on each node to gather system and application metrics.

Network Plugins: CNI plugins like Calico, Weave, or Cilium deploy as DaemonSets to provide networking capabilities to pods on each node.

Storage Daemons: Distributed storage systems like Ceph or GlusterFS use DaemonSets to run storage daemons on nodes designated for storage.

Here’s a production-grade Fluentd DaemonSet for cluster-wide log aggregation:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

This configuration tolerates control plane nodes (so it runs there too) and mounts both /var/log and Docker container paths to capture all logs.

DaemonSet Configuration and Scheduling

While DaemonSets bypass the normal scheduler, you still control pod placement through node selectors, node affinity, and taints/tolerations.

Node Selectors are the simplest approach—specify labels that nodes must have:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: gpu-monitor
  template:
    metadata:
      labels:
        app: gpu-monitor
    spec:
      nodeSelector:
        accelerator: nvidia-gpu
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: gpu-exporter
        image: nvidia/dcgm-exporter:latest
        securityContext:
          privileged: true
        volumeMounts:
        - name: nvidia
          mountPath: /usr/local/nvidia
          readOnly: true
      volumes:
      - name: nvidia
        hostPath:
          path: /usr/local/nvidia

This DaemonSet runs only on nodes labeled accelerator=nvidia-gpu and tolerates the GPU taint, ensuring it doesn’t interfere with regular workloads while monitoring GPU nodes.

Taints and tolerations are critical for DaemonSets. Nodes often have taints to prevent regular pods from scheduling. Your DaemonSet needs appropriate tolerations to run on those nodes.

Update Strategies

DaemonSets support two update strategies: RollingUpdate (default) and OnDelete.

RollingUpdate automatically updates DaemonSet pods when you change the pod template. This is safer for most use cases but requires careful configuration to avoid taking down critical services across all nodes simultaneously.

OnDelete requires manual pod deletion before updates apply. Use this when you need precise control over update timing or when updates are risky.

Here’s a DaemonSet with a conservative rolling update strategy:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      containers:
      - name: agent
        image: company/node-agent:v2.3.1
        resources:
          limits:
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi

maxUnavailable: 1 ensures only one pod updates at a time. For critical services, this prevents cluster-wide failures. For less critical services, you might increase this number or use a percentage like maxUnavailable: 25%.

Resource Management and Monitoring

DaemonSets consume resources on every node. Multiply your resource requests by your node count to understand the total cluster impact. A seemingly modest 100Mi memory request becomes 10GB across 100 nodes.

Always set resource requests and limits. Without them, a misbehaving DaemonSet can starve node-level resources and impact all pods on affected nodes.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: metrics-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: metrics-collector
  template:
    metadata:
      labels:
        app: metrics-collector
    spec:
      containers:
      - name: collector
        image: prom/node-exporter:v1.6.1
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
        ports:
        - containerPort: 9100
          name: metrics
        resources:
          limits:
            cpu: 200m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi
        livenessProbe:
          httpGet:
            path: /
            port: metrics
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: metrics
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

The liveness and readiness probes ensure Kubernetes can detect and restart unhealthy pods. This is crucial for DaemonSets since they provide infrastructure services that other workloads depend on.

Advanced Patterns

DaemonSets often require elevated privileges and access to node resources. This introduces security considerations you must address carefully.

hostPath Volumes: Grant access to node filesystems. Use sparingly and always with read-only mounts when possible.

hostNetwork Mode: Allows pods to use the node’s network namespace. Required for some network plugins and monitoring tools but bypasses network policies.

Privileged Containers: Grant nearly all capabilities of the host. Necessary for some system-level operations but should be your last resort.

Here’s a node monitoring DaemonSet demonstrating these patterns:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: system-monitor
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: system-monitor
  template:
    metadata:
      labels:
        app: system-monitor
    spec:
      hostNetwork: true
      hostPID: true
      hostIPC: true
      containers:
      - name: monitor
        image: company/system-monitor:v1.2.0
        securityContext:
          privileged: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          limits:
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
        volumeMounts:
        - name: root
          mountPath: /host
          readOnly: true
        - name: run
          mountPath: /var/run
      volumes:
      - name: root
        hostPath:
          path: /
      - name: run
        hostPath:
          path: /var/run

This configuration is powerful but dangerous. Only use these settings when absolutely necessary, and always:

  • Document why each privilege is required
  • Run as non-root user when possible
  • Use Pod Security Standards to limit which namespaces can run privileged DaemonSets
  • Regularly audit DaemonSet permissions

Troubleshooting Common Issues

When DaemonSets misbehave, start with these diagnostic commands:

# Check DaemonSet status
kubectl get daemonset -n kube-system

# See which nodes have pods
kubectl get pods -n kube-system -l app=fluentd -o wide

# Check for scheduling issues
kubectl describe daemonset fluentd -n kube-system

# View events for a specific pod
kubectl describe pod fluentd-xyz -n kube-system

# Check if nodes have taints preventing scheduling
kubectl get nodes -o json | jq '.items[].spec.taints'

# See DaemonSet pod distribution
kubectl get pods -n kube-system -l app=fluentd \
  --field-selector=status.phase=Running \
  -o custom-columns=NODE:.spec.nodeName,NAME:.metadata.name

Common issues include:

Pods not scheduling on all nodes: Check node selectors, taints, and resource availability. Use kubectl describe node to see resource pressure.

Update stuck: Verify maxUnavailable settings aren’t too conservative. Check if pods are failing readiness probes.

Resource exhaustion: DaemonSets can overwhelm nodes if resource limits are too high. Monitor node resource usage and adjust accordingly.

Permission errors: Ensure ServiceAccounts have necessary RBAC permissions, especially for DaemonSets that interact with the Kubernetes API.

DaemonSets are powerful tools for managing node-level infrastructure. Use them thoughtfully, secure them carefully, and monitor them closely. When configured correctly, they provide reliable, cluster-wide services that form the foundation of your Kubernetes infrastructure.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.