ConFoo 2020: systemd for developers and devops

systemd
for developers and DevOps
ConFoo 2020 / Montreal 2020-02-26
Christian Heimes
Principal Software Engineer
[email protected] / [email protected]
@ChristianHeimes

View Slide

systemd, ConFoo 2020, @ChristianHeimes, CC BY-SA 4.0
Is systemd evil?
Does using systemd
make me a bad developer?
Is Lennart Poettering
the Antichrist?

View Slide

Who am I?
●
from Hamburg/Germany
●
Python core developer / Python security team
●
Principal Software Engineer at Red Hat
Identity Management and Platform Security

View Slide

●
Why systemd?
●
How to
●
create and customize service configuration
●
manage services
●
improve security & server stability
●
run applications
●
best practices
Takeaways
https://speakerdeck.com/tiran/

View Slide

●
POSIX 101 / old SyV init
●
what is systemd?
●
systemd services and units
●
resource management
●
security & sandboxing
●
tips & tricks
●
~ 5 minutes Q&A
Agenda

View Slide

how expensive is a hitman?
we can pool some bitcoins
cut his hands, so he will not be able to write any new line of code
Pali: there's speech-to-text now
i believe that any solution has to be more... radical
i'm afraid redhat would pull a clone out of their fridge if
anything ever happened to poettering
Is Lennart Poettering the Antichrist?

View Slide


View Slide

Abuse
FUD
(fear, uncertainty, and doubt)

View Slide

“The Tragedy of systemd”
Benno Rice
Linux.conf.au 2019
BSDCan 2018 Keynote

View Slide

Change

View Slide

It violates UNIX philosophy!

View Slide

Unix is dead! Linux has won!

View Slide

Windows Subsystem for Linux (WSL-2)

View Slide

POSIX & Linux 101

View Slide

●
permissions
●
capabilities (CAP_SYS_ADMIN)
●
seccomp filter
Kernel & user space
hardware
kernel
userspace
syscall
man: syscalls(2)
man: capabilities(7)

View Slide

●
fork(), clone()
●
exec()
●
exit(), signal (SIGTERM)
●
parent must call waitpid()
●
zombies…
●
PID 1 becomes parent of orphans
process man: execve(2)

View Slide

●
handle to a Kernel resource
●
file, socket, pipe, epoll, inotify, …
●
inherited from parent process
●
shared over AF_UNIX socket
●
usually one permission check on access
file descriptor man: open(2)

View Slide

●
control groups
●
resource limiting (memory, tasks, ...)
●
priorization (CPU, IO, ...)
●
resource accounting (CPU, memory, network)
●
namespace isolation
●
mount, pid, network, user, UTS, IPC, cgroups
●
hierarchical
cgroups & namespaces man: cgroups(7)

View Slide

In the past…
/etc/init.d

View Slide

●
complicated shell scripts (sh, not bash!)
●
serialized start
●
inflexible ordering, no dependencies
●
limited respawn with inittab
●
no service management
●
singleton processes are hard (stale lock files)
Old init

View Slide

0) [ident] fork(), exec()
1) [p] close fds
2) [p] reset signal handlers
3) [p] sigprocmask()
4) [p] sanitize env vars
5) [p] create pipe
6) [p] fork() to create background task
7) [c] setsid()
8) [c] fork() to detach terminal
9) [c] exit() to re-parent [d] to PID 1
Unix daemons
10) [d] stdin, stdout, stderr /dev/null
→
11) [d] reset umask
12) [d] chdir("/")
13) [d] write PID file
14) [d] drop privileges
15) [d] notify parent [p] via pipe
16) [d] close pipe
17) [p] exit()
18) [ident] … continue
man: daemon(7)

View Slide

By default a service running as root is able to
●
overwrite system files, /usr, Kernel, …
●
read your email, bitcoins, SSH private keys
●
change time, hostname, firewall settings
●
connect to any host or local service
●
consume all memory or CPU resources
●
write /dev/sda
(SELinux confined services or AppArmor prevent some problems)
Security problem

View Slide

What is systemd?

View Slide

●
Start less
●
Start more in parallel
●
React on hardware and software events
●
Keep track of processes
●
Control and secure process environments (sandboxing)
●
API
●
(declarative) configuration
"Rethinking PID 1"
https://0pointer.de/blog/projects/systemd.html

View Slide

●
an init system for Linux
●
a system and service manager
●
a collection of services and tools
●
> 50 binaries
●
an active and mature Open Source project
●
10 years old
●
> 1,000 contributors
●
Arch, Debian, Gentoo, Red Hat, SuSE, Ubuntu, ...
What is systemd?

View Slide

●
service
●
socket
●
target
●
path
●
timer
●
slice
●
scope
units
●
device
●
mount
●
automount
●
swap
man: systemd.unit(5)

View Slide

●
systemd: PID 1, user instances
●
dbus: RPC
●
systemd-journald: logging
●
systemd-udevd: hardware events, firmware
●
systemd-logind: login manager
●
systemd-localed: locale and key mappings
●
systemd-machined: VM and container registration manager
●
systemd-timedated: time, timezone
●
…
components

View Slide


View Slide

●
policy for shutdown/sleep
●
shutdown/sleep inhibition
●
multi-seat support
●
user services
●
per-user instances: dbus, pulseaudio, systemd
●
runtime directories: /run/user/1000
●
user sessions
●
Gnome, KDE, terminal, …
●
lingering user services
$ loginctl
SESSION UID USER SEAT TTY
3 1000 heimes seat0
4 1000 heimes seat0 tty2
$ loginctl
3 1000 heimes seat0
4 1000 heimes seat0 tty2
systemd-logind man: systemd-logind(8)

View Slide

simple
service unit

View Slide

simple service
# simple.service
[Unit]
Description=Simple service
[Service]
ExecStart=sleep 20
[Install]
WantedBy=multi-user.target
# simple.service
[Unit]
[Service]
ExecStart=sleep 20
[Install]
# cp simple.service /etc/systemd/system/
# systemctl daemon-reload
# cp simple.service /etc/systemd/system/
# systemctl enable simple.service
Created symlink /etc/systemd/system/multi-user.target.wants/simple.service
→ /etc/systemd/system/simple.service.
# systemctl start simple.service
Created symlink /etc/systemd/system/multi-user.target.wants/simple.service
→ /etc/systemd/system/simple.service.
man: systemd.service(5)

View Slide

simple service (2)
# systemctl status simple.service
● simple.service - Simple service
Loaded: loaded (/etc/systemd/system/simple.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-01-08 11:11:27 CET; 13s ago
Main PID: 866159 (sleep)
Tasks: 1 (limit: 19012)
Memory: 248.0K
CGroup: /system.slice/simple.service
└─866159 /usr/bin/sleep 20
Active: active (running) since Wed 2020-01-08 11:11:27 CET; 13s ago
Main PID: 866159 (sleep)
Tasks: 1 (limit: 19012)
Memory: 248.0K
CGroup: /system.slice/simple.service
└─866159 /usr/bin/sleep 20
Active: inactive (dead) since Wed 2020-01-08 11:11:47 CET; 6s ago
Process: 866159 ExecStart=/usr/bin/sleep 20 (code=exited, status=0/SUCCESS)
Main PID: 866159 (code=exited, status=0/SUCCESS)
Active: inactive (dead) since Wed 2020-01-08 11:11:47 CET; 6s ago
Process: 866159 ExecStart=/usr/bin/sleep 20 (code=exited, status=0/SUCCESS)
Main PID: 866159 (code=exited, status=0/SUCCESS)
# journalctl -u simple.service
-- Logs begin at Wed 2019-08-28 09:54:44 CEST, end at Wed 2020-01-08 11:15:11 CET. --
Jan 08 11:11:27 seneca systemd[1]: Started Simple service.
Jan 08 11:11:47 seneca systemd[1]: simple.service: Succeeded.
# journalctl -u simple.service
-- Logs begin at Wed 2019-08-28 09:54:44 CEST, end at Wed 2020-01-08 11:15:11 CET. --
Jan 08 11:11:27 seneca systemd[1]: Started Simple service.
Jan 08 11:11:47 seneca systemd[1]: simple.service: Succeeded.

View Slide

systemctl
reload configuration
reload configuration
unit start/stop
# systemctl stop simple.service
# systemctl restart simple.service
# systemctl reload simple.service
# systemctl kill simple.service
unit start/stop
# systemctl stop simple.service
# systemctl reload simple.service
# systemctl kill simple.service
block unit (no implicit start)
# systemctl mask simple.service
# systemctl unmask simple.service
block unit (no implicit start)
# systemctl mask simple.service
# systemctl unmask simple.service
man: systemctl(1)
unit explicit auto-start
# systemctl disable simple.service
unit explicit auto-start
# systemctl disable simple.service

View Slide

systemctl
unit status
# systemctl show simple.service
# systemctl is-active simple.service
# systemctl is-enabled simple.service
unit status
# systemctl show simple.service
# systemctl is-active simple.service
# systemctl is-enabled simple.service
man: systemctl(1)
# systemctl enable --now simple.service
# systemctl enable --now simple.service

View Slide

unit search paths
/etc/systemd/system.control/
/run/systemd/system.control/
/run/systemd/transient/
/run/systemd/generator.early/
/etc/systemd/system/
/etc/systemd/systemd.attached/
/run/systemd/system/
/run/systemd/systemd.attached/
/run/systemd/generator/
...
/usr/lib/systemd/system/
/run/systemd/generator.late/
/etc/systemd/system.control/
/run/systemd/system.control/
/run/systemd/transient/
/run/systemd/generator.early/
/etc/systemd/systemd.attached/
/run/systemd/systemd.attached/
/run/systemd/generator/
...
/run/systemd/generator.late/
●
admin customization:
●
non-persistent customization:
●
package maintainer:
man: systemd.unit(5)

View Slide

customize service
# mkdir /etc/systemd/system/simple.service.d
# edit /etc/systemd/system/simple.service.d/custom.conf
# mkdir /etc/systemd/system/simple.service.d
# edit /etc/systemd/system/simple.service.d/custom.conf
[Unit]
Wants=network-online.target remote-fs.target
After=network-online.target remote-fs.target
[Service]
Restart=always
[Unit]
[Service]
Restart=always
# systemctl edit simple.service
# systemctl edit simple.service

View Slide

customize service (2)
Loaded: loaded (/etc/systemd/system/simple.service; enabled; vendor preset:
disabled)
Drop-In: /etc/systemd/system/simple.service.d
└─custom.conf
Loaded: loaded (/etc/systemd/system/simple.service; enabled; vendor preset:
disabled)
Drop-In: /etc/systemd/system/simple.service.d
└─custom.conf
# systemctl show -p Restart simple.service
Restart=always
# systemctl show -p Restart simple.service
Restart=always

View Slide

customize service (3)
# /etc/systemd/system/simple.service
[Unit]
[Service]
ExecStart=sleep 20
[Install]
# /etc/systemd/system/simple.service.d/custom.conf
[Unit]
[Service]
Restart=always
# /etc/systemd/system/simple.service
[Unit]
[Service]
ExecStart=sleep 20
[Install]
# /etc/systemd/system/simple.service.d/custom.conf
[Unit]
[Service]
Restart=always
# systemctl cat simple.service

View Slide

target unit

View Slide

●
grouping of units
●
synchronization points
●
replacement for SysV runlevels (rc3 → multi-user.target)
●
multiple targets can be active
Target unit (.target)
[Unit]
Description=Emergency Mode
Documentation=man:systemd.special(7)
Requires=emergency.service
After=emergency.service
AllowIsolate=yes
[Unit]
Description=Emergency Mode
Documentation=man:systemd.special(7)
Requires=emergency.service
After=emergency.service
AllowIsolate=yes
man: systemd.target(5)

View Slide

Default units
# systemctl list-units --type=target
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
bluetooth.target loaded active active Bluetooth
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network-pre.target loaded active active Network (Pre)
network.target loaded active active Network
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target loaded active active Paths
remote-fs.target loaded active active Remote File Systems
slices.target loaded active active Slices
sockets.target loaded active active Sockets
sound.target loaded active active Sound Card
sshd-keygen.target loaded active active sshd-keygen.target
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
# systemctl list-units --type=target
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
bluetooth.target loaded active active Bluetooth
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network-pre.target loaded active active Network (Pre)
network.target loaded active active Network
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target loaded active active Paths
remote-fs.target loaded active active Remote File Systems
slices.target loaded active active Slices
sockets.target loaded active active Sockets
sound.target loaded active active Sound Card
sshd-keygen.target loaded active active sshd-keygen.target
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
man: bootup(7)

View Slide

service units (2)

View Slide

SysV daemon
[Service]
Type=forking
ExecStart=/usr/bin/myservice
PIDFile=/run/myservice.pid
ExecReload=/bin/kill -HUP $MAINPID
Environment=MYSERVICE_LOGGING=verbose
EnvironmentFile=/etc/sysconfig/myservice
EnvironmentFile=-/etc/sysconfig/myservice-override
[Service]
Type=forking
ExecStart=/usr/bin/myservice
PIDFile=/run/myservice.pid
ExecReload=/bin/kill -HUP $MAINPID
Environment=MYSERVICE_LOGGING=verbose
EnvironmentFile=/etc/sysconfig/myservice
EnvironmentFile=-/etc/sysconfig/myservice-override
man: systemd.exec(5)

View Slide

[Service]
Type=oneshot
ExecStart=/usr/bin/custom-firewall.sh start
ExecStop=/usr/bin/custom-firewall.sh stop
ExecReload=/usr/bin/custom-firewall.sh reload
RemainAfterExit=yes
[Service]
Type=oneshot
ExecStart=/usr/bin/custom-firewall.sh start
ExecStop=/usr/bin/custom-firewall.sh stop
ExecReload=/usr/bin/custom-firewall.sh reload
RemainAfterExit=yes
Oneshot service
short-duration process
[Unit]
Description=One-time temporary TLS key generation for httpd.service
ConditionPathExists=|!/etc/pki/tls/certs/localhost.crt
ConditionPathExists=|!/etc/pki/tls/private/localhost.key
[Service]
Type=oneshot
ExecStart=/usr/libexec/httpd-ssl-gencerts
[Unit]
Description=One-time temporary TLS key generation for httpd.service
ConditionPathExists=|!/etc/pki/tls/certs/localhost.crt
ConditionPathExists=|!/etc/pki/tls/private/localhost.key
[Service]
Type=oneshot
ExecStart=/usr/libexec/httpd-ssl-gencerts

View Slide

[Service]
Type=notify
ExecStart=/usr/sbin/httpd
[Service]
Type=notify
ExecStart=/usr/sbin/httpd
Status notification man: sd_notify(3)
/* server is ready to handle requests */
sd_notify(0, "READY=1");
/* server is ready to handle requests */
sd_notify(0, "READY=1");
# systemctl status httpd | grep Status
Status: "Running, listening on: port 80, port 443"
Status: "Total requests: 10000; Idle/Busy workers 100/0;Requests/sec: 0.141; Bytes
served/sec: 831 B/sec"
Status: "Running, listening on: port 80, port 443"
Status: "Total requests: 10000; Idle/Busy workers 100/0;Requests/sec: 0.141; Bytes
served/sec: 831 B/sec"
from systemd.daemon import notify
notify("READY=1")
from systemd.daemon import notify
notify("READY=1")

View Slide

●
Type=simple
●
foreground service
●
Typing=forking
●
Unix daemon mode (PIDFile=/path/to/file)
●
Type=oneshot
●
short-running script
●
Type=notify
●
sd_notify()
●
Type=dbus
●
BusName=org.freedesktop.example
Service Type
process monitoring & readiness

View Slide

●
ExecStart=/path/to/binary
●
exactly one
●
oneshot: 0...n
●
ExecStartPre, ExecStartPost
●
ExecStopPost
●
ExecCondition
●
ExecReload
●
ExecStop
Service Exec
Execute command

View Slide

●
Restart
●
no, always, on-success, on-failure, on-abnormal, on-abort, watchdog
●
RestartSec
●
WatchdogSec
Service Restart / Watchdog
Restart crashed processes
def watchdog_task():
while True:
if check_app():
notify("WATCHDOG=1") # ok
else:
notify("WATCHDOG=trigger") # trigger failure
wait = int(os.environ("WATCHDOG_USEC")) / 1_000_000
sleep(wait / 2)
Thread(target=watchdog_task).start()
def watchdog_task():
while True:
if check_app():
notify("WATCHDOG=1") # ok
else:
notify("WATCHDOG=trigger") # trigger failure
wait = int(os.environ("WATCHDOG_USEC")) / 1_000_000
sleep(wait / 2)
Thread(target=watchdog_task).start()
man: sd_watchdog_enabled(3)

View Slide

activation units

View Slide

# borgmatic.timer
[Unit]
Description=Run borgmatic backup
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
# borgmatic.timer
[Unit]
Description=Run borgmatic backup
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
timer unit (.timer)
cron & at
man: systemd.timer(5)
# borgmatic.service
[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true
[Service]
Type=oneshot
ExecStart=/usr/bin/borgmatic
# borgmatic.service
[Unit]
Description=borgmatic backup
[Service]
Type=oneshot
ExecStart=/usr/bin/borgmatic
# systemctl enable --now borgmatic.timer
# systemctl enable --now borgmatic.timer

View Slide

●
Monotonic timers
●
OnActiveSec, OnBootSec, OnStartupSec
●
OnUnitActive, OnUnitDeactive
●
Wallclocker timer
●
OnCalendar=Mon..Fri, 08:00
●
Randomization
●
AccuracySec=1h
●
RandomizedDelaySec
●
… more
●
OnClockChange, OnTimezoneChange
●
WakeSystem
Timer options man: systemd.timer(5)
man: systemd.time(7)
$ systemd-analyze calendar "Mon..Fri 08:00"
Original form: Mon..Fri 08:00
Normalized form: Mon..Fri *-*-* 08:00:00
Next elapse: Mon 2020-01-17 08:00:00 CET
(in UTC): Mon 2020-01-17 07:00:00 UTC
From now: 2 days left
$ systemd-analyze calendar "Mon..Fri 08:00"
Original form: Mon..Fri 08:00
Normalized form: Mon..Fri *-*-* 08:00:00
Next elapse: Mon 2020-01-17 08:00:00 CET
(in UTC): Mon 2020-01-17 07:00:00 UTC
From now: 2 days left

View Slide

●
PathExists
●
PathExistsGlob
●
PathChanged
●
PathModified
●
DirectoryNotEmpty
Path activation unit (.path) man: systemd.path(5)
# myservice-import.path
[Unit]
Description=My Service data importer
[Path]
DirectoryNotEmpty=/var/lib/myservice/import
[Install]
# myservice-import.path
[Unit]
Description=My Service data importer
[Path]
DirectoryNotEmpty=/var/lib/myservice/import
[Install]

View Slide

# myserver.socket
[Unit]
Description=My Server socket
[Socket]
ListenStream=9000
BindIPv6Only=both
[Install]
WantedBy=sockets.target
# myserver.socket
[Unit]
Description=My Server socket
[Socket]
ListenStream=9000
BindIPv6Only=both
[Install]
WantedBy=sockets.target
Socket activation unit (.socket) man: systemd.socket(5)
# myserver.service
[Unit]
Description=My Server application
[Service]
ExecStart=/usr/bin/myserver
# myserver.service
[Unit]
Description=My Server application
[Service]
ExecStart=/usr/bin/myserver
# systemctl enable --now myserver.socket
# systemctl enable --now myserver.socket

View Slide

from socket import socket
from systemd.daemon import listen_fds
fds = listen_fds()
srv_sock = socket(fileno=fds[0])
while True:
client_conn, addr = srv_sock.accept()
...
from socket import socket
from systemd.daemon import listen_fds
fds = listen_fds()
srv_sock = socket(fileno=fds[0])
while True:
client_conn, addr = srv_sock.accept()
...
socket activation man: sd_listen_fds(3)
$ systemd-socket-activate -l 9000 \
python3 -c \
"from systemd.daemon import listen_fds; from socket import socket;
print(socket(fileno=listen_fds()[0]))"
...
proto=6, laddr=('::', 9000, 0, 0)>
$ systemd-socket-activate -l 9000 \
python3 -c \
"from systemd.daemon import listen_fds; from socket import socket;
print(socket(fileno=listen_fds()[0]))"
...
proto=6, laddr=('::', 9000, 0, 0)>

View Slide

●
Socket families and types
●
TCP, UDP, AF_UNIX, FIFO, netlink, special, …
●
sockets survive:
●
crash
●
service restart
●
package upgrade
●
port < 1024 for unprivileged processes
Socket activation features

View Slide

security
sandboxing

View Slide

By default a service running as root is able to
●
overwrite system files, /usr, Kernel, …
●
read your email, bitcoins, SSH private keys
●
change time, hostname, firewall settings
●
connect to any host or local service
●
consume all memory or CPU resources
●
write /dev/sda
(SELinux confined services or AppArmor prevent some problems)
"root" services are a security problem

View Slide

●
User=myuser (effective uid)
●
Group (effective gid)
●
DynamicUser
●
dynamically allocated, unique UID/GID
User, groups, directories
seteuid / user namespace

View Slide

●
PrivateTmp=yes
●
Private namespace /tmp and /var/tmp
●
auto-cleanup
●
JoinsNamespaceOf=parent.service
●
ProtectSystem={yes, strict, full}
●
/usr and /boot read-only, strict: also /etc, full: everything except /dev and /tmp
●
ProtectHome={yes, read-only}
●
/home, /root, /run/user appear empty
●
PrivateDevices=yes
●
Minimal /dev (zero, null, urandom, …), drop CAP_MKNOD
File system protection
mount namespace

View Slide

●
ReadWriteDirectories, ReadOnlyDirectories, InaccessibleDirectories
●
RuntimeDirectory, StateDirectory, CacheDirectory, LogsDirectory
●
automatically created and chowned
●
StateDirectory=myservice →/var/lib/myservice
●
STATE_DIRECTORY env var
●
systemd-tmpfiles service
Directories
mount namespace

View Slide

●
PrivateNetwork
●
only private loopback network device
●
JoinsNamespaceOf=parent.service
●
PrivateHostname
●
[email protected] @module
●
CapabilityBoundingSet=~CAP_SYS_ADMIN
●
NoNewPrivileges
●
suid / sgid binaries
●
MemoryDenyWriteExecute=yes
●
...
Many more security flags
network ns, UTS ns, seccomp, ...

View Slide

●
ulimit
●
ulimit -n 1024 LimitNOFILE=1024
→
●
nice
●
OOMScoreAdjust
●
Out-Of-Memory killer score
●
CPU scheduler and affinity
●
CPUAffinity=0-3
●
CPU NUMA policy
●
IO scheduler
●
IOSchedulingClass=idle
Resource limits, scheduling
ulimit, rlimit, priority, ...

View Slide

Security analyzer
NAME DESCRIPTION EXPOSURE
✗ PrivateNetwork= Service has access to the host's network 0.6
✗ User=/DynamicUser= Service runs as root user 0.5
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP) Service may change UID/GID identities/capabilities 0.4
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN Service has administrator privileges 0.4
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE Service has ptrace() debugging abilities 0.4
✓ RestrictAddressFamilies=~AF_(INET|INET6) Service cannot allocate Internet sockets
✓ RestrictNamespaces=~CLONE_NEWUSER Service cannot create user namespaces
✓ RestrictAddressFamilies=~… Service cannot allocate exotic sockets
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP) Service may change file ownership/access mode/capabilities unrestricted 0.3
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER) Service may override UNIX file/IPC permission checks 0.3
✓ CapabilityBoundingSet=~CAP_NET_ADMIN Service has no network configuration privileges
✓ CapabilityBoundingSet=~CAP_RAWIO Service has no raw I/O access
✓ CapabilityBoundingSet=~CAP_SYS_MODULE Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TIME Service processes cannot change the system clock
✗ DeviceAllow= Service has no device ACL 0.3
✓ IPAddressDeny= Service blocks all IP address ranges
✓ KeyringMode= Service doesn't share key material with other services
✗ NoNewPrivileges= Service processes may acquire new privileges 0.3
✓ NotifyAccess= Service child processes cannot alter service state
✗ PrivateDevices= Service potentially has access to hardware devices 0.3
✗ PrivateMounts= Service may install system mounts 0.3
PrivateTmp= Service runs in special boot phase, option does not apply
✗ PrivateUsers= Service has access to other users 0.3
✗ ProtectControlGroups= Service may modify the control group file system 0.3
ProtectHome= Service runs in special boot phase, option does not apply
✗ ProtectKernelModules= Service may load or read kernel modules 0.3
✗ ProtectKernelTunables= Service may alter kernel tunables 0.3
ProtectSystem= Service runs in special boot phase, option does not apply
✓ RestrictAddressFamilies=~AF_PACKET Service cannot allocate packet sockets
NAME DESCRIPTION EXPOSURE
✗ PrivateNetwork= Service has access to the host's network 0.6
✗ User=/DynamicUser= Service runs as root user 0.5
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP) Service may change UID/GID identities/capabilities 0.4
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN Service has administrator privileges 0.4
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE Service has ptrace() debugging abilities 0.4
✓ RestrictAddressFamilies=~AF_(INET|INET6) Service cannot allocate Internet sockets
✓ RestrictNamespaces=~CLONE_NEWUSER Service cannot create user namespaces
✓ RestrictAddressFamilies=~… Service cannot allocate exotic sockets
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP) Service may change file ownership/access mode/capabilities unrestricted 0.3
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER) Service may override UNIX file/IPC permission checks 0.3
✓ CapabilityBoundingSet=~CAP_NET_ADMIN Service has no network configuration privileges
✓ CapabilityBoundingSet=~CAP_RAWIO Service has no raw I/O access
✓ CapabilityBoundingSet=~CAP_SYS_MODULE Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TIME Service processes cannot change the system clock
✗ DeviceAllow= Service has no device ACL 0.3
✓ IPAddressDeny= Service blocks all IP address ranges
✓ KeyringMode= Service doesn't share key material with other services
✗ NoNewPrivileges= Service processes may acquire new privileges 0.3
✓ NotifyAccess= Service child processes cannot alter service state
✗ PrivateDevices= Service potentially has access to hardware devices 0.3
✗ PrivateMounts= Service may install system mounts 0.3
PrivateTmp= Service runs in special boot phase, option does not apply
✗ PrivateUsers= Service has access to other users 0.3
✗ ProtectControlGroups= Service may modify the control group file system 0.3
ProtectHome= Service runs in special boot phase, option does not apply
✗ ProtectKernelModules= Service may load or read kernel modules 0.3
✗ ProtectKernelTunables= Service may alter kernel tunables 0.3
ProtectSystem= Service runs in special boot phase, option does not apply
✓ RestrictAddressFamilies=~AF_PACKET Service cannot allocate packet sockets
man: systemd-analyze(1)
# systemd-analyze security systemd-journald.service
# systemd-analyze security systemd-journald.service

View Slide

Advanced
resource management

View Slide

●
Slice: resource group in cgroup hierarchy
●
-.slice (root)
●
system.slice
●
user.slice
●
machine.slice
●
Service: process (group) managed by systemd
●
Scope: group of external processes
●
user session
"cgroups made easy" man: systemd.resource-control(5)

View Slide

systemd cgroups
# systemd-cgls
Control group /:
-.slice
├─user.slice
│ └─user-1000.slice
│ ├─[email protected]
│ │ └─ ...
│ └─session-3.scope
│ ├─ kded5
│ └─ ...
├─init.scope
│ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 30
└─system.slice
├─sshd.service
│ └─1185 /usr/sbin/sshd -D
├─httpd.service
│ ├─21895 /usr/sbin/httpd -DFOREGROUND
...
# systemd-cgls
Control group /:
-.slice
├─user.slice
│ └─user-1000.slice
│ ├─[email protected]
│ │ └─ ...
│ └─session-3.scope
│ ├─ kded5
│ └─ ...
├─init.scope
│ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 30
└─system.slice
├─sshd.service
│ └─1185 /usr/sbin/sshd -D
├─httpd.service
│ ├─21895 /usr/sbin/httpd -DFOREGROUND
...
man: systemd-cgls(1)
man: cgroups(7)
$ loginctl
3 1000 heimes seat0
$ loginctl
3 1000 heimes seat0

View Slide

cgroups top
# systemd-cgtop
Control Group Tasks %CPU Memory Input/s Output/s
user.slice 1181 31,7 5.0G - -
user.slice/user-1000.slice 1181 31,7 5.0G - -
user.slice/user-1000.slice/session-3.scope 991 30,0 4.6G - -
/ 1719 28,9 6.6G - -
user.slice/use…000.slice/[email protected] 190 1,8 364.9M - -
system.slice 404 0,6 1.4G - -
system.slice/sddm.service 15 0,4 237.4M - -
system.slice/httpd.service 278 0,1 14.5M - -
system.slice/rsyslog.service 3 0,0 3.7M - -
system.slice/wpa_supplicant.service 1 0,0 4.2M - -
init.scope 1 - 21.2M - -
system.slice/ModemManager.service 3 - 3.9M - -
...
# systemd-cgtop
Control Group Tasks %CPU Memory Input/s Output/s
user.slice 1181 31,7 5.0G - -
user.slice/user-1000.slice 1181 31,7 5.0G - -
user.slice/user-1000.slice/session-3.scope 991 30,0 4.6G - -
/ 1719 28,9 6.6G - -
user.slice/use…000.slice/[email protected] 190 1,8 364.9M - -
system.slice 404 0,6 1.4G - -
system.slice/sddm.service 15 0,4 237.4M - -
system.slice/httpd.service 278 0,1 14.5M - -
system.slice/rsyslog.service 3 0,0 3.7M - -
system.slice/wpa_supplicant.service 1 0,0 4.2M - -
init.scope 1 - 21.2M - -
system.slice/ModemManager.service 3 - 3.9M - -
...
man: systemd-cgtop(1)

View Slide

Service resource control man: systemd.resource-control(5)
# httpd.service.d/custom.conf
[Service]
MemoryMax=1G # 1 GB memory
TasksMax=100 # 100 threads / process
CPUQuota=300% # 3 CPU cores
IOWeight=200 # double IO priority
IPAccounting=yes # monitor network traffic
# httpd.service.d/custom.conf
[Service]
MemoryMax=1G # 1 GB memory
TasksMax=100 # 100 threads / process
CPUQuota=300% # 3 CPU cores
IOWeight=200 # double IO priority
IPAccounting=yes # monitor network traffic
# systemctl stop httpd.service
# journalctl -f -o cat -u httpd.service
...
Stopped The Apache HTTP Server.
httpd.service: Consumed 4.927s CPU time, received 3.2M IP traffic, sent 58.3M IP
traffic.
# systemctl stop httpd.service
# journalctl -f -o cat -u httpd.service
...
Stopped The Apache HTTP Server.
httpd.service: Consumed 4.927s CPU time, received 3.2M IP traffic, sent 58.3M IP
traffic.

View Slide

Custom slice (.slice) man: systemd.slice(5)
# mydb.service
[Unit]
Slice=system-myapp.slice
# mydb.service
[Unit]
# myserver.service
[Unit]
# myserver.service
[Unit]
└─system.slice
└─system-myapp.slice
├─mydb.service
│ └─12345 mydb
└─myserver.service
└─12347 myserver
└─system.slice
└─system-myapp.slice
├─mydb.service
│ └─12345 mydb
└─myserver.service
└─12347 myserver
# system-myapp.slice
[Unit]
Description=Limit DB and app to 2 GB memory in total
[Slice]
MemoryMax=2G
# system-myapp.slice
[Unit]
Description=Limit DB and app to 2 GB memory in total
[Slice]
MemoryMax=2G

View Slide

Runtime control man: systemctl(1)
Limit slice to 1 GB
# systemctl set-property system-myapp.slice MemoryMax=1G
Limit slice to 1 GB
# systemctl set-property system-myapp.slice MemoryMax=1G
Remove limitation until next reboot
# systemctl set-property --runtime system-myapp.slice MemoryMax=
# systemctl show -p MemoryMax system-myapp.slice
MemoryMax=infinity
Remove limitation until next reboot
# systemctl set-property --runtime system-myapp.slice MemoryMax=
# systemctl show -p MemoryMax system-myapp.slice
MemoryMax=infinity
systemctl set-property httpd.service CPUWeight=200 IPAccounting=yes
systemctl set-property httpd.service CPUWeight=200 IPAccounting=yes
systemctl set-property --runtime httpd.service IPAddressDeny="10.0.0.0/8"
systemctl set-property --runtime httpd.service IPAddressDeny="10.0.0.0/8"

View Slide

application

View Slide

0) [ident] fork(), exec()
1) [p] close fds
2) [p] reset signal handlers
3) [p] sigprocmask()
4) [p] sanitize env vars
5) [p] create pipe
6) [p] fork() to create background task
7) [c] setsid()
8) [c] fork() to detach terminal
9) [c] exit() to re-parent [d] to PID 1
Unix daemons
10) [d] stdin, stdout, stderr /dev/null
→
11) [d] reset umask
12) [d] chdir(“/”)
13) [d] write PID file
14) [d] drop privileges
15) [d] notify parent [p] via pipe
16) [d] close pipe
17) [p] exit()
18) [ident] … continue
man: daemon(7)

View Slide

●
don't daemonize
●
follow LSB specification (Linux Standard Base)
●
exit codes
●
signals
●
paths
●
let systemd drop privileges
●
log to stdout / stderr
●
write a service file
How to port a service to systemd

View Slide

●
sd_notify()
●
service ready
●
watchdog
●
status information
●
socket activation
●
oneshot services for initialization / upgrades
●
sd_journal_send() for improved logging
Consider advanced features

View Slide

Journald logging man: sd_journal_send(3)
from systemd.journal import send, LOG_NOTICE
send("Hello ConFoo", PRIORITY=LOG_NOTICE, CITY="Montreal", COUNTRY="Canada")
from systemd.journal import send, LOG_NOTICE
send("Hello ConFoo", PRIORITY=LOG_NOTICE, CITY="Montreal", COUNTRY="Canada")
$ journalctl -f -o json-pretty
{
"_AUDIT_LOGINUID" : "1000",
"MESSAGE" : "Hello ConFoo",
"SYSLOG_IDENTIFIER" : "python3",
"_PID" : "91755",
"CITY" : "Montreal",
"COUNTRY" : "Canada",
"PRIORITY" : "5",
"_SELINUX_CONTEXT" : "unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023",
...
"_EXE" : "/usr/bin/python3.7",
"_SYSTEMD_CGROUP" : "/user.slice/user-1000.slice/session-3.scope",
"_UID" : "1000",
"_BOOT_ID" : "fe72a9b80a9b4b6ea9c7e1887bf827f7",
"_SYSTEMD_UNIT" : "session-3.scope",
"_COMM" : "python3"
}
$ journalctl -f -o json-pretty
{
"_AUDIT_LOGINUID" : "1000",
"MESSAGE" : "Hello ConFoo",
"SYSLOG_IDENTIFIER" : "python3",
"_PID" : "91755",
"CITY" : "Montreal",
"COUNTRY" : "Canada",
"PRIORITY" : "5",
"_SELINUX_CONTEXT" : "unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023",
...
"_EXE" : "/usr/bin/python3.7",
"_SYSTEMD_CGROUP" : "/user.slice/user-1000.slice/session-3.scope",
"_UID" : "1000",
"_BOOT_ID" : "fe72a9b80a9b4b6ea9c7e1887bf827f7",
"_SYSTEMD_UNIT" : "session-3.scope",
"_COMM" : "python3"
}

View Slide

Odds and ends

View Slide

Human readable name & documentation
[Unit]
Description=The Apache HTTP Server
Documentation=man:httpd.service(8)
[Unit]
Description=The Apache HTTP Server
Documentation=man:httpd.service(8)
$ systemctl help httpd.service
$ systemctl help $(pidof httpd)
$ systemctl help httpd.service
$ systemctl help $(pidof httpd)

View Slide

Dependencies
●
Wants
●
Requires
●
Requisites
●
BindsTo
●
PartOf
●
Conflicts
Dependencies & Ordering
start and stop order
Ordering
●
Before
●
After
[Unit]
Requires=database.service
[Unit]
[Unit]
After=database.service
[Unit]
After=database.service

View Slide

Conditionals, mounts
# don't run in battery mode
# file must exists
ConditionPathExists=/path/to/file
# at least 500 MB free memory
ConditionMemory= >500M
# don't run in battery mode
# file must exists
ConditionPathExists=/path/to/file
# at least 500 MB free memory
ConditionMemory= >500M
# /var/lib/database file system mounted
Requires=var-lib-database.mount
# /var/lib/database file system mounted
Requires=var-lib-database.mount

View Slide

●
application .target
●
application .slice
●
initial configuration .service
●
upgrade .service
Multi-service application

View Slide

Templated service (@.service) man: systemd.service(5)
# sshd-keygen.target
[email protected]
[email protected]
[email protected]
# sshd-keygen.target
[email protected]
[email protected]
[email protected]
# sshd.service
Wants=sshd-keygen.target
# sshd.service
Wants=sshd-keygen.target
# [email protected]
[Unit]
Description=OpenSSH %i Server Key Generation
[Service]
Type=oneshot
ExecStart=/usr/libexec/openssh/sshd-keygen %i
[Install]
WantedBy=sshd-keygen.target
# [email protected]
[Unit]
Description=OpenSSH %i Server Key Generation
[Service]
Type=oneshot
ExecStart=/usr/libexec/openssh/sshd-keygen %i
[Install]
WantedBy=sshd-keygen.target

View Slide

●
Don't – keep it small
●
… unless it's a multi-service container
●
check out OCI-Hooks
systemd in containers?

View Slide

●
Started on log-in first session
●
Stopped on log-off of last session
●
Linger with loginctl
User services man: systemd.unit(5)
~/.config/systemd/user/
/etc/systemd/user/
~/.local/share/systemd/user/
...
/usr/lib/systemd/user/
~/.config/systemd/user/
/etc/systemd/user/
~/.local/share/systemd/user/
...
/usr/lib/systemd/user/
$ systemctl --user daemon-reload
$ systemctl --user enable --now myuserapp.service
$ systemctl --user daemon-reload
$ systemctl --user enable --now myuserapp.service

View Slide

D-Bus scripting man: busctl(1)
# busctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1
# busctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1
# busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 \
org.freedesktop.systemd1.Manager \
GetUnit s "sssd.service"
o "/org/freedesktop/systemd1/unit/sssd_2eservice"
# busctl get-property org.freedesktop.systemd1 \
/org/freedesktop/systemd1/unit/sssd_2eservice \
org.freedesktop.systemd1.Unit ActiveState
s "active"
GetUnit s "sssd.service"
o "/org/freedesktop/systemd1/unit/sssd_2eservice"
# busctl get-property org.freedesktop.systemd1 \
/org/freedesktop/systemd1/unit/sssd_2eservice \
org.freedesktop.systemd1.Unit ActiveState
s "active"
StopUnit ss "sssd.service" "replace"
o "/org/freedesktop/systemd1/job/102829"
StopUnit ss "sssd.service" "replace"
o "/org/freedesktop/systemd1/job/102829"

View Slide

●
systemctl
●
… cat unit.service
●
systemd-delta
●
systemd-analyze
●
… plot
●
… dot
●
… verify filename
●
… security
●
systemd-tmpfiles
Useful commands

View Slide

Summary

View Slide

●
systemd is all about units
●
process: service
●
activation: time, path, socket
●
grouping: target
●
resource control: cgroups,slice, service units
●
security & sandboxing: name space, filters
Summary

View Slide

●
man: systemd(1)
●
https://www.freedesktop.org/wiki/Software/systemd/
●
The systemd for Administrators Blog Series
●
The systemd for Developers Series
●
Documentation for Developers
●
Presentations (Youtube)
●
"Demystifying systemd"
●
"The Tragedy of systemd"
●
"The Six Stages of systemd"
●
Slides: https://speakerdeck.com/tiran/
Resources

View Slide

Questions?
@ChristianHeimes
[email protected]
[email protected]
https://speakerdeck.com/tiran/

View Slide

THANK YOU
plus.google.com/+RedHat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
linkedin.com/company/red-hat

View Slide

ConFoo 2020: systemd for developers and devops

ConFoo 2020: systemd for developers and devops

Christian Heimes

More Decks by Christian Heimes

Other Decks in Programming

Featured

Transcript