SYSTEMD

ESSENTIAL SYSTEMCTL COMMANDS

ESSENTIAL JOURNALCTL COMMANDS

SERVICE STATE MANAGEMENT

start - Activate/start one or more units:

systemctl start nginx.service
systemctl start nginx apache2  # Start multiple

stop - Deactivate/stop one or more units:

systemctl stop nginx.service

Note: Stopping does not prevent it from starting at next boot.

restart - Stop and then start a service:

systemctl restart nginx.service

Use when configuration changes require full restart.

reload - Reload service configuration without stopping:

systemctl reload nginx.service

Sends signal to service to reload config. Not all services support this.

reload-or-restart - Reload if supported, otherwise restart:

systemctl reload-or-restart nginx.service

SERVICE BOOT BEHAVIOR

enable - Configure service to start at boot:

systemctl enable nginx.service

Creates symlinks in appropriate targets. Service will start on next boot but is not started immediately.

disable - Prevent service from starting at boot:

systemctl disable nginx.service

Removes symlinks but does not stop the service if currently running.

enable --now - Enable and start immediately:

systemctl enable --now nginx.service

disable --now - Disable and stop immediately:

systemctl disable --now nginx.service

MASKING AND UNMASKING

mask - Completely prevent service from starting:

systemctl mask nginx.service

Creates symlink to /dev/null. Service cannot be started manually or automatically, even as a dependency. This is stronger than disable.

unmask - Remove mask to allow service to be started:

systemctl unmask nginx.service

Key Differences:

• stop: Service not running but can be started manually or automatically
• disable: Service won't start at boot but can be started manually or by dependencies
• mask: Service cannot be started at all (manual or automatic) until unmasked

LIST AND INSPECT COMMANDS

list-units - Show active units:

systemctl list-units                    # All active units
systemctl list-units --type=service     # Only services
systemctl list-units --state=failed     # Only failed units
systemctl list-units --all              # Include inactive

list-unit-files - Show all installed unit files:

systemctl list-unit-files
systemctl list-unit-files --type=service
systemctl list-unit-files --state=enabled

list-dependencies - Show dependency tree:

systemctl list-dependencies nginx.service
systemctl list-dependencies --reverse nginx.service
systemctl list-dependencies --all nginx.service

cat - View unit file and drop-in snippets:

systemctl cat nginx.service

show - Display all unit properties:

systemctl show nginx.service
systemctl show nginx.service -p MainPID

UNIT FILE STRUCTURE

Unit files are plain text INI-style files located in:

• /etc/systemd/system/ - Local, administrator-created (highest priority)
• /run/systemd/system/ - Runtime units
• /usr/lib/systemd/system/ - Distribution-provided (lowest priority)

A unit file consists of sections:

• [Unit] - Generic metadata and dependencies
• Type-specific section ([Service], [Socket], [Timer], etc.)
• [Install] - Installation information for enable/disable

[UNIT] SECTION

Common directives in the [Unit] section:

[Unit]
Description=Human-readable description of the unit
Documentation=https://example.com/docs
After=network.target
Before=nginx.service
Requires=postgresql.service
Wants=redis.service
BindsTo=mount-point.mount
Conflicts=apache2.service
ConditionPathExists=/etc/myapp/config
OnFailure=recovery.service

[SERVICE] SECTION

The Type= directive defines how systemd determines when the service has started:

COMMON SERVICE DIRECTIVES

[Service]
# Execution
ExecStart=/usr/bin/myapp --config /etc/myapp/config.conf
ExecStartPre=/usr/local/bin/pre-start.sh
ExecStartPost=/usr/local/bin/post-start.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/graceful-stop.sh

# Restart Behavior
Restart=on-failure
RestartSec=5s
StartLimitBurst=5
StartLimitIntervalSec=60s

# Environment
WorkingDirectory=/var/lib/myapp
User=myappuser
Group=myappgroup
Environment="VAR1=value1" "VAR2=value2"
EnvironmentFile=/etc/myapp/environment

# Security
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=yes

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

EXAMPLE COMPLETE UNIT FILE

[Unit]
Description=My Application Service
Documentation=https://myapp.example.com/docs
After=network.target postgresql.service
Wants=redis.service
ConditionPathExists=/etc/myapp/config.conf

[Service]
Type=notify
User=myappuser
Group=myappgroup
WorkingDirectory=/var/lib/myapp
Environment="NODE_ENV=production"
EnvironmentFile=/etc/myapp/environment

ExecStartPre=/usr/local/bin/check-config.sh
ExecStart=/usr/bin/myapp --config /etc/myapp/config.conf
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/graceful-stop.sh

Restart=on-failure
RestartSec=10s
StartLimitBurst=3
StartLimitIntervalSec=120s

TimeoutStartSec=60
TimeoutStopSec=30

StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

PrivateTmp=yes
ProtectSystem=full
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

EXECUTION COMMANDS

ExecStart - Main command to start the service:

# Simple command
ExecStart=/usr/bin/myapp

# With arguments
ExecStart=/usr/bin/myapp --port 8080 --config /etc/myapp.conf

# Prefix modifiers:
# - = errors (non-zero exit) are ignored
# + = run with full privileges (even if User= is set)
# ! = elevate privileges
ExecStart=-/usr/bin/may-fail
ExecStart=+/usr/bin/needs-root

ExecStartPre - Commands to run before ExecStart:

ExecStartPre=/usr/local/bin/validate-config.sh
ExecStartPre=/usr/bin/mkdir -p /var/run/myapp

ExecStartPost - Commands to run after ExecStart:

ExecStartPost=/usr/local/bin/register-service.sh

ExecReload - Command to reload configuration:

ExecReload=/bin/kill -HUP $MAINPID

ExecStop - Command to stop the service:

ExecStop=/usr/local/bin/graceful-stop.sh

ExecStopPost - Commands to run after service stops:

ExecStopPost=/usr/bin/rm -rf /var/run/myapp

RESTART BEHAVIOR

[Service]
Restart=on-failure
RestartSec=5s               # Wait 5 seconds before restart
StartLimitBurst=5           # Try max 5 times
StartLimitIntervalSec=60s   # Within 60 second window

PROCESS MANAGEMENT

KillMode - How to kill processes:

KillMode=control-group  # Kill all processes in cgroup (default)
KillMode=process        # Kill only main process
KillMode=mixed          # SIGTERM to main, SIGKILL to others
KillMode=none           # Don't kill processes

Timeouts:

TimeoutStartSec=90s     # Wait max 90 seconds for startup
TimeoutStopSec=30s      # Wait 30s for stop, then SIGKILL
TimeoutSec=60s          # Sets both start and stop timeout

Signals:

KillSignal=SIGTERM      # Default signal when stopping
KillSignal=SIGINT

ORDERING DEPENDENCIES

After= - Start this unit after the specified units:

After=network.target postgresql.service

Does NOT cause the units to be started automatically. Only controls order IF both are starting.

Before= - Start this unit before the specified units:

Before=nginx.service

Does NOT cause the units to be started automatically. Only controls order IF both are starting.

REQUIREMENT DEPENDENCIES

DEPENDENCY EXAMPLES

Web application with database:

[Unit]
Description=Web Application
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service

Service with mount point:

[Unit]
Description=Data Processor
BindsTo=mnt-data.mount
After=mnt-data.mount
Wants=network-online.target

Viewing dependencies:

systemctl list-dependencies myapp.service
systemctl list-dependencies --reverse myapp.service
systemctl show myapp.service -p Requires -p Wants -p After

BASIC USAGE

# View all logs
journalctl

# Follow logs in real-time
journalctl -f

# Show recent entries
journalctl -n 50

# Reverse order (newest first)
journalctl -r

# No pager
journalctl --no-pager

FILTERING BY UNIT

# Specific service
journalctl -u nginx.service
journalctl -u nginx               # .service can be omitted

# Follow service logs
journalctl -u nginx.service -f

# Multiple units
journalctl -u nginx.service -u postgresql.service

# Kernel messages
journalctl -k
journalctl --dmesg

FILTERING BY TIME

# Relative time
journalctl --since "1 hour ago"
journalctl --since "30 minutes ago"
journalctl --since today
journalctl --since yesterday

# Absolute time
journalctl --since "2025-01-01"
journalctl --since "2025-01-01 14:30:00"

# Time range
journalctl --since "09:00" --until "17:00"
journalctl --since "2025-01-15" --until "2025-01-20"

FILTERING BY PRIORITY

Priority levels: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7)

# Error priority and above
journalctl -p err

# Warning priority and above
journalctl -p warning

# Numeric priority
journalctl -p 3

# Priority range
journalctl -p warning..err

FILTERING BY BOOT

# Current boot
journalctl -b
journalctl -b 0

# Previous boots
journalctl -b -1               # Previous boot
journalctl -b -2               # Boot before that

# List all boots
journalctl --list-boots

OUTPUT FORMATS

# Short format (default)
journalctl -o short

# Verbose (all fields)
journalctl -o verbose

# JSON
journalctl -o json
journalctl -o json-pretty

# Cat (just message, no metadata)
journalctl -o cat

# With millisecond precision
journalctl -o short-precise

JOURNAL MAINTENANCE

# Disk usage
journalctl --disk-usage

# Vacuum by size
sudo journalctl --vacuum-size=500M

# Vacuum by time
sudo journalctl --vacuum-time=7d

# Verify journal integrity
journalctl --verify

# Rotate journal
sudo journalctl --rotate

TIMER STRUCTURE

A timer requires TWO files:

• .timer file - defines when to run
• .service file - defines what to run

Basic timer file:

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily Backup Timer
Requires=backup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Corresponding service file:

# /etc/systemd/system/backup.service
[Unit]
Description=Daily Backup Job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

ONCALENDAR - CALENDAR TIME

Format: DayOfWeek Year-Month-Day Hour:Minute:Second

# Daily at midnight
OnCalendar=daily
OnCalendar=*-*-* 00:00:00

# Daily at 2:30 AM
OnCalendar=*-*-* 02:30:00

# Every Monday at 9 AM
OnCalendar=Mon *-*-* 09:00:00

# Weekdays at 10 AM
OnCalendar=Mon..Fri *-*-* 10:00:00

# First day of month
OnCalendar=monthly
OnCalendar=*-*-01 00:00:00

# Every hour
OnCalendar=hourly
OnCalendar=*-*-* *:00:00

# Every 15 minutes
OnCalendar=*:0/15

# Weekends at noon
OnCalendar=Sat,Sun *-*-* 12:00:00

Test calendar expressions:

systemd-analyze calendar "Mon..Fri *-*-* 10:00:00"

MONOTONIC TIMERS

Run relative to events:

# 15 minutes after boot
OnBootSec=15min

# 1 hour after last activation
OnUnitActiveSec=1h

# 5 minutes after service becomes inactive
OnUnitInactiveSec=5min

# 2 minutes after timer itself is activated
OnActiveSec=2min

Time units: us, ms, s, sec, min, h, hr, d, day, w, week, month, y, year

TIMER OPTIONS

[Timer]
# Catch up on missed runs
Persistent=true

# Randomization window (spreads load)
AccuracySec=1h

# Add random delay
RandomizedDelaySec=30min

# Activate specific service
Unit=different-name.service

# Keep timer active after trigger
RemainAfterElapse=yes

MANAGING TIMERS

# Enable and start timer
sudo systemctl enable --now backup.timer

# Check timer status
systemctl status backup.timer

# List all timers
systemctl list-timers
systemctl list-timers --all

# View logs
journalctl -u backup.timer
journalctl -u backup.service

# Manually trigger service
sudo systemctl start backup.service

MAJOR SYSTEM TARGETS

TARGET MANAGEMENT

# View current target
systemctl get-default

# Set default target
sudo systemctl set-default multi-user.target
sudo systemctl set-default graphical.target

# Switch to target immediately
sudo systemctl isolate multi-user.target

# List available targets
systemctl list-units --type=target
systemctl list-units --type=target --all

BOOT ANALYSIS

systemd-analyze provides tools to analyze boot performance:

# Boot time summary
systemd-analyze

# Per-unit timing
systemd-analyze blame

# Critical chain (bottleneck analysis)
systemd-analyze critical-chain
systemd-analyze critical-chain nginx.service

# Graphical boot chart
systemd-analyze plot > boot.svg

# Verify unit files
systemd-analyze verify /etc/systemd/system/myapp.service

# Security analysis
systemd-analyze security myapp.service

# Calendar expression testing
systemd-analyze calendar "Mon..Fri *-*-* 10:00:00"

BOOT DEBUGGING

# Boot messages
journalctl -b                    # Current boot
journalctl -b -1                 # Previous boot
journalctl -b -p err             # Errors from current boot

# Failed units
systemctl --failed
systemctl list-units --failed

# Boot in rescue mode (add to GRUB kernel line)
systemd.unit=rescue.target
systemd.unit=emergency.target

# Boot with debug output
systemd.log_level=debug
systemd.log_target=console

CPU CONTROL

[Service]
# Limit CPU usage
CPUQuota=20%              # Max 20% of one core
CPUQuota=200%             # Max 2 full cores

# CPU priority/share
CPUWeight=500             # Range: 1-10000, default: 100

# Enable CPU accounting
CPUAccounting=yes

MEMORY CONTROL

[Service]
# Hard memory limit
MemoryMax=1G              # Kill at 1 GB

# Soft memory limit
MemoryHigh=800M           # Throttle at 800 MB

# Swap limit
MemorySwapMax=500M        # Max swap usage
MemorySwapMax=0           # Disable swap

# Enable memory accounting
MemoryAccounting=yes

I/O CONTROL

[Service]
# I/O priority
IOWeight=500              # Range: 1-10000, default: 100

# Limit read bandwidth
IOReadBandwidthMax=/dev/sda 10M

# Limit write bandwidth
IOWriteBandwidthMax=/dev/sda 5M

# Limit IOPS
IOReadIOPSMax=/dev/sda 1000
IOWriteIOPSMax=/dev/sda 500

TASK CONTROL

[Service]
# Limit number of tasks (threads/processes)
TasksMax=100              # Max 100 tasks
TasksMax=infinity         # No limit

MONITORING RESOURCES

# Real-time cgroup monitoring
systemd-cgtop

# View service resources
systemctl status myapp.service
systemctl show myapp.service -p MemoryCurrent
systemctl show myapp.service -p CPUUsageNSec
systemctl show myapp.service -p TasksCurrent

COMPLETE RESOURCE EXAMPLE

[Unit]
Description=Resource-Limited Web Application
After=network.target

[Service]
Type=notify
User=webapp
ExecStart=/usr/bin/webapp

# CPU limits
CPUAccounting=yes
CPUQuota=100%             # Max 1 core
CPUWeight=200             # Higher priority

# Memory limits
MemoryAccounting=yes
MemoryHigh=1.5G           # Throttle
MemoryMax=2G              # Hard limit
MemorySwapMax=500M        # Limited swap

# I/O limits
IOWeight=200
IOWriteBandwidthMax=/dev/sda 50M

# Task limits
TasksMax=200

Restart=on-failure

[Install]
WantedBy=multi-user.target

SOCKET UNIT STRUCTURE

Socket activation requires TWO files:

• .socket file - defines socket to listen on
• .service file - service to activate when connection arrives

Basic socket file:

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080

[Install]
WantedBy=sockets.target

Corresponding service:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application

[Service]
ExecStart=/usr/bin/myapp

SOCKET TYPES

# TCP socket
ListenStream=80
ListenStream=8080
ListenStream=192.168.1.1:8080
ListenStream=[::]:9000           # IPv6

# UDP socket
ListenDatagram=514

# Unix socket
ListenStream=/run/myapp.sock

# Multiple sockets
ListenStream=80
ListenStream=443

SOCKET OPTIONS

[Socket]
# Unix socket permissions
SocketMode=0660
SocketUser=www-data
SocketGroup=www-data

# Connection backlog
Backlog=128

# Limit connections
MaxConnections=64
MaxConnectionsPerSource=10

# TCP keepalive
KeepAlive=yes
KeepAliveTimeSec=30

# Buffer sizes
ReceiveBuffer=256K
SendBuffer=256K

MANAGING SOCKETS

# Enable and start socket
sudo systemctl enable --now myapp.socket

# Check socket status
systemctl status myapp.socket

# List all sockets
systemctl list-sockets
systemctl list-sockets --all

# Test socket
nc localhost 8080
nc -U /run/myapp.sock

DEBUGGING FAILED SERVICES

# Check service status
systemctl status myapp.service

# View full logs
journalctl -u myapp.service -n 100
journalctl -u myapp.service --since "1 hour ago"

# Check exit code and reason
systemctl show myapp.service -p ExecMainStatus
systemctl show myapp.service -p Result

# Validate unit file
systemd-analyze verify myapp.service

# Emulate service execution
sudo runuser -u  -g  -c "cd  && "

Common failure reasons:

• Wrong User/Group (permission denied)
• Missing WorkingDirectory
• Wrong ExecStart path (file not found)
• Missing environment variables
• Service exits immediately (needs Type=forking or Type=oneshot)
• Timeout (increase TimeoutStartSec)
• Dependency failure (check Required/Wanted units)

SYSTEMD-RUN FOR TRANSIENT UNITS

systemd-run creates temporary units that exist only while running:

# Run command as transient service
systemd-run /usr/bin/mycommand

# Run with resource limits
systemd-run --property=MemoryMax=1G \
            --property=CPUQuota=50% \
            /usr/bin/memory-hungry-app

# Run in foreground (scope unit)
systemd-run --scope /usr/bin/mycommand

# Run as specific user
systemd-run --property=User=webapp /usr/bin/webapp

# Run on timer
systemd-run --on-calendar=daily /usr/local/bin/backup.sh

# Wait for completion
systemd-run --wait /usr/bin/long-task

USER SERVICES

systemd can manage services for individual users (not just system-wide):

User unit file locations:

• ~/.config/systemd/user/ - User's own units
• /etc/systemd/user/ - System-wide user units

# Manage user services (as user, NOT with sudo)
systemctl --user start myapp.service
systemctl --user enable myapp.service
systemctl --user status myapp.service

# User service logs
journalctl --user -u myapp.service

# Enable lingering (services run without login)
sudo loginctl enable-linger username

OVERRIDE FILES (systemctl edit)

Modify units without editing original files using drop-in snippets:

# Edit service with drop-in
sudo systemctl edit myapp.service
# Opens editor with empty override file
# Saves to /etc/systemd/system/myapp.service.d/override.conf

# Edit full unit file
sudo systemctl edit --full myapp.service

# View effective configuration
systemctl cat myapp.service

# Remove overrides
sudo systemctl revert myapp.service

Example overrides:

# Increase memory limit
[Service]
MemoryMax=2G

# Add environment variable
[Service]
Environment="DEBUG=true"

# Override ExecStart (must clear first)
[Service]
ExecStart=
ExecStart=/usr/bin/myapp --new-flag

QUICK CHECKS

# Status checks
systemctl is-active myapp.service
systemctl is-enabled myapp.service
systemctl is-failed myapp.service

# List all failed units
systemctl --failed

# Reset failed state
sudo systemctl reset-failed myapp.service
sudo systemctl reset-failed  # Reset all

# Reload unit files
sudo systemctl daemon-reload

# Set property at runtime
sudo systemctl set-property myapp.service MemoryMax=2G

# Template units
sudo systemctl start myapp@instance1.service
sudo systemctl start myapp@instance2.service

SECURITY HARDENING

[Service]
# Private namespaces
PrivateTmp=yes          # Private /tmp
PrivateDevices=yes      # Private /dev
ProtectSystem=strict    # Read-only /usr, /boot
ProtectHome=yes         # Inaccessible /home
ProtectKernelTunables=yes
ProtectControlGroups=yes

# Capabilities instead of root
User=myapp
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Prevent manual control
RefuseManualStop=yes
RefuseManualStart=yes

# Watchdog for health monitoring
Type=notify
WatchdogSec=30s