systemd: Service Management in Linux
systemd has become the de facto init system and service manager across major Linux distributions. Whether you're running Ubuntu, Fedora, Debian, or RHEL, you're almost certainly using systemd to...
Key Insights
- systemd unit files provide declarative service configuration with built-in dependency management, automatic restarts, and resource limits that surpass traditional init scripts
- The Type directive fundamentally changes how systemd manages your service lifecycle—choosing between simple, forking, notify, and oneshot determines supervision behavior
- Security hardening through systemd directives like PrivateTmp, ProtectSystem, and NoNewPrivileges requires zero application changes while significantly reducing attack surface
Introduction to systemd
systemd has become the de facto init system and service manager across major Linux distributions. Whether you’re running Ubuntu, Fedora, Debian, or RHEL, you’re almost certainly using systemd to manage system initialization and service supervision.
Unlike traditional SysV init scripts that relied on shell scripts and sequential execution, systemd introduces parallel service startup, on-demand activation, and sophisticated dependency resolution. It manages the entire system state—from initial boot through service lifecycle management to system shutdown.
For application developers and system administrators, systemd offers powerful primitives for service supervision, automatic restarts, resource constraints, and security sandboxing. Understanding systemd isn’t optional anymore—it’s fundamental infrastructure knowledge.
Understanding systemd Unit Files
systemd uses unit files to define services, mount points, devices, and other system resources. Service units (.service files) live in /etc/systemd/system/ for custom services or /lib/systemd/system/ for package-managed services.
A unit file consists of three primary sections:
[Unit] - Metadata and dependencies
[Service] - Service-specific configuration
[Install] - Installation information for enabling/disabling
Here’s a minimal service unit:
[Unit]
Description=My Application Service
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/myapp
Restart=on-failure
[Install]
WantedBy=multi-user.target
Let’s examine a real-world example for a Node.js application:
[Unit]
Description=Node.js API Server
Documentation=https://docs.example.com/api
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
[Service]
Type=simple
User=nodeapp
Group=nodeapp
WorkingDirectory=/opt/api-server
Environment="NODE_ENV=production"
EnvironmentFile=/etc/api-server/env
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=api-server
# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/api-server/uploads
[Install]
WantedBy=multi-user.target
This configuration demonstrates dependency management (waits for PostgreSQL), runs as a non-root user, implements security restrictions, and ensures automatic restarts on failure.
Essential systemctl Commands
The systemctl command is your primary interface to systemd. Master these operations:
# Start a service immediately
systemctl start myapp.service
# Stop a running service
systemctl stop myapp.service
# Restart (stop then start)
systemctl restart myapp.service
# Reload configuration without restarting (if supported)
systemctl reload myapp.service
# Enable service to start at boot
systemctl enable myapp.service
# Enable and start in one command
systemctl enable --now myapp.service
# Disable from starting at boot
systemctl disable myapp.service
# Check service status
systemctl status myapp.service
# Reload systemd configuration after editing unit files
systemctl daemon-reload
The status command provides rich information:
$ systemctl status nginx.service
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-01-15 10:23:45 UTC; 2 days ago
Docs: man:nginx(8)
Process: 1234 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 1235 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 1236 (nginx)
Tasks: 5 (limit: 4915)
Memory: 12.3M
CGroup: /system.slice/nginx.service
├─1236 nginx: master process /usr/sbin/nginx
└─1237 nginx: worker process
Always run systemctl daemon-reload after modifying unit files. systemd caches configurations, and changes won’t take effect until you reload.
Service Types and Execution Models
The Type directive fundamentally affects how systemd supervises your service. Choose incorrectly, and you’ll encounter unexpected behavior.
Type=simple - The default. systemd considers the service started immediately after ExecStart launches. Use for foreground processes:
[Service]
Type=simple
ExecStart=/usr/bin/myapp --foreground
Type=forking - For traditional daemons that fork to background. systemd expects the parent process to exit:
[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/bin/myapp --daemon
Type=notify - The service sends a notification when ready via sd_notify(). Best for applications with complex initialization:
[Service]
Type=notify
ExecStart=/usr/bin/myapp
NotifyAccess=main
Type=oneshot - For tasks that run and exit. Useful for initialization scripts:
[Service]
Type=oneshot
ExecStart=/usr/bin/setup-script.sh
RemainAfterExit=yes
Configure restart behavior based on your reliability requirements:
[Service]
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60
This configuration retries up to 5 times within 60 seconds, with 5-second delays between attempts.
Dependency management ensures correct startup order:
[Unit]
After=network.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service
Requires creates a hard dependency—if PostgreSQL fails, your service fails. Wants is softer—your service starts even if Redis fails. After controls ordering without creating dependencies.
Advanced Configuration and Best Practices
systemd provides resource management without requiring cgroups manipulation:
[Service]
# Limit memory to 512MB
MemoryMax=512M
MemoryHigh=400M
# Limit CPU to 50% of one core
CPUQuota=50%
# Limit number of processes
TasksMax=100
# I/O weight (100-10000, default 100)
IOWeight=500
Security hardening through systemd directives significantly reduces attack surface:
[Service]
# Run as specific user
User=appuser
Group=appuser
# Prevent privilege escalation
NoNewPrivileges=true
# Isolate /tmp
PrivateTmp=true
# Make most of filesystem read-only
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp
# Hide /home
ProtectHome=true
# Prevent access to kernel tunables
ProtectKernelTunables=true
# Restrict system calls
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
# Capabilities (if needed)
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
Use environment files for sensitive configuration:
[Service]
EnvironmentFile=/etc/myapp/secrets.env
EnvironmentFile=/etc/myapp/config.env
Environment="LOG_LEVEL=info"
The environment file format is simple:
# /etc/myapp/secrets.env
DATABASE_URL=postgresql://user:pass@localhost/db
API_KEY=secret-key-here
Monitoring and Troubleshooting
journalctl provides powerful log querying integrated with systemd:
# Follow logs for a specific service
journalctl -u myapp.service -f
# Show logs since boot
journalctl -u myapp.service -b
# Last 100 lines
journalctl -u myapp.service -n 100
# Logs from the last hour
journalctl -u myapp.service --since "1 hour ago"
# Filter by priority (emerg, alert, crit, err, warning, notice, info, debug)
journalctl -u myapp.service -p err
# Show logs in reverse order (newest first)
journalctl -u myapp.service -r
# Export to JSON for processing
journalctl -u myapp.service -o json
When a service fails, check its status first:
systemctl status myapp.service
If the status doesn’t reveal the issue, examine recent logs:
journalctl -u myapp.service -n 50 --no-pager
Analyze boot performance with systemd-analyze:
# Overall boot time
systemd-analyze
# Services sorted by initialization time
systemd-analyze blame
# Critical path to target
systemd-analyze critical-chain
# Visualize boot process (requires graphviz)
systemd-analyze plot > boot.svg
For debugging service startup issues, increase logging verbosity:
# Temporary debug mode
systemctl service-log-level myapp.service debug
# Or edit the unit file
[Service]
Environment="DEBUG=*"
LogLevelMax=debug
Conclusion
systemd represents a fundamental shift in Linux service management. Its declarative configuration model, integrated dependency management, and built-in security features make it superior to legacy init systems for production workloads.
Start by converting your shell-script-based services to systemd units. You’ll gain automatic restarts, proper dependency ordering, resource limits, and security sandboxing with minimal effort. The investment in understanding systemd pays dividends in system reliability and operational simplicity.
Focus on choosing the correct Type for your service, implementing security directives appropriate to your threat model, and leveraging journalctl for centralized logging. These three practices alone will improve your service reliability and security posture significantly.