Two years living your future systemd is coming, what to expect CentOS Dojo - Santa Clara - March 2014

Joe Miller Ops @ Pantheon (getpantheon.com) @miller_joe github.com/joemiller

Platform for professional web developers systemd + fedora in production for > 2 years 600,000+ systemd managed services CTO - David Strauss - systemd, core committer

“systemd is a system manager more than just an init system” - Lennart Poettering systemd manages more than just services automounts, mounts, timers (think cron), logging, devices, hostname, networking (soon), targets (runlevels), and more

systemd seeks to provide common infrastructure that is often re-implemented across applications an example is systemd’s helpers for services: no code needed for dropping privileges no code needed for proper fork()’ing no code needed for logging, just print to stdout/stderr

systemd adoption is accelerating systemd will replace init in RHEL7 (CentOS7) debian - next release ubuntu - future release http://en.wikipedia.org/wiki/Systemd (Mar 22, 2014)

but transitioning is going to be a pain? not really, lots of backwards compatibility If you like your init scripts, you can keep them but you won’t want to ● distro provides .service files for most things ● systemd can control legacy init scripts ● systemd reads legacy /etc/fstab ● journald forwards to syslog

the things we (as sysadmins) like about systemd: NO MORE FIGHTING WITH INIT SCRIPTS fast, parallel boot declarative service configs cgroups integration powerful dependency model journald (rich logging)

fast, parallel boot

services

declarative service definitions # /etc/systemd/system/cassandra.service [Unit] Description=Cassandra [Service] Type=simple User=cassandra Group=cassandra Environment=CASSANDRA_HOME=/usr/share/cassandra/ Environment=CASSANDRA_INCLUDE=/usr/share/cassandra/cassandra.in.sh Environment=MAX_HEAP_SIZE=6G ExecStart=/usr/sbin/cassandra -f LimitNOFILE=32768 CPUShares=1000 MemoryLimit=M MemorySoftLimit=M BlockIOWeight=100 [Install] WantedBy=multi-user.target /etc/systemd/system/ /usr/lib/systemd/system/

basic service settings User, Group, Supplementary Groups, Environment, EnvironmentFile, Nice Level, CPU Affinity, CPU Priority, IO Scheduling, IO Priority, OOM Score Adjust, ulimits

security settings chroot(), Set & limit capabilities(7), Read/Write directories, Read-only directories, Inaccessible directories, Private Tmp, Private Dev, SELinux Context, AppArmor Profile, Syscall Filtering, ...

advanced resource control (cgroups) CPU Accounting, CPU Shares, Memory Accounting, Memory Limiting, Block IO Weight (global, per-device), Block IO bandwidth limit Can be set per-service, or shared amongst groups of services using .slices NOTE: not all cgroups settings are available. More in the future as cgroups interface stabilizes

starting / stopping / restarting $ systemctl daemon-reload # must run this after modifying unit files $ systemctl start cassandra.service $ systemctl stop cassandra.service $ systemctl restart cassandra.service $ systemctl try-restart cassandra.service # restart only if service is running $ systemctl status cassandra.service cassandra.service - Cassandra Loaded: loaded (/etc/systemd/system/cassandra.service; enabled) Active: active (running) since Mon, 10 Mar 2014 02:38:01 +0000; 2 minutes Main PID: 21247 (java) CGroup: name=systemd:/system/cassandra.service └ 21247 java -ea -javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar Mar 26 01:00:01 valhalla1f cassandra[21247]: INFO 01:00:01,016 Started hinted handof Mar 26 01:00:01 valhalla1f cassandra[21247]: INFO 01:00:01,018 Finished hinted hando

.targets replace run-levels no more run-levels many targets can be active at once useful for grouping services or ordering startup multi-user.target is most common for your services

enabling services at boot ### /etc/systemd/system/cassandra.service [Unit] Description=Cassandra … [Install] WantedBy=multi-user.target $ systemctl enable cassandra.service ln -s '/etc/systemd/system/cassandra.service' \ '/etc/systemd/system/multi-user.target.wants/cassandra.service' $ systemctl disable cassandra.service rm '/etc/systemd/system/multi-user.target.wants/cassandra.service' $ systemctl is-enabled cassandra.service ; echo $?

listing services $ systemctl --all UNIT LOAD ACTIVE SUB DESCRIPTION proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary bloomd.service loaded inactive dead SYSV: Bloomd cassandra.service loaded active running Cassandra chronyd.service loaded inactive dead NTP client sys-fs-fuse-connections.mount loaded active mounted FUSE Control sys-kernel-config.mount loaded active mounted Configuration sys-kernel-debug.mount loaded active mounted Debug File tmp.mount loaded active mounted Temporary systemd-ask-password-console.path loaded inactive dead Dispatch core-file-cleanup.service loaded inactive dead Remove old crond.service loaded active running Command dbus.service loaded active running D-Bus System mysql.service loaded active running MySQL database netconsole.service loaded inactive dead SYSV: initia network.service loaded active exited LSB: Bring NetworkManager-dispatcher.service loaded inactive dead Network newrelic-daemon.service loaded inactive dead SYSV: Starts

listing failed services $ systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION elasticsearch.service loaded failed failed ElasticSearch great for writing monitoring checks

instantiated services single template, many instances [email protected] [Unit] Description=Getty on %I [Service] ExecStart=-/sbin/agetty --noclear %I ... $ systemctl start [email protected] $ systemctl start [email protected] $ systemctl enable [email protected] $ systemctl enable [email protected] $ systemctl --all | grep getty [email protected] loaded active running Getty on tty1 [email protected] loaded active running Getty on tty2 [email protected] loaded active running Getty on tty3

dependency management dependencies are not only between services, but all unit types, e.g: wait for remote .mount, then start a .service flexibility (loose / strong, forwards / backwards): Wants, Wanted, WantedBy, Before, After, Requires, RequiredBy, Also, PartOf, ...

dependency management example: wait for admin services to boot then start all container services ### /etc/systemd/system/containers.timer [Unit] Before=containers.target shutdown.target After=multi-user.target sysinit.target [Timer] OnActiveSec=60 Unit=containers.target [Install] WantedBy=multi-user.target ### /etc/systemd/system/containers.target [Unit] Description=Starts containers. ### /etc/systemd/system/container1.service ... [Install] WantedBy=containers.target

cgroups resource control via cgroups automatically controlled by systemd can be set per-.service or shared between multiple services using .slice units changeable at runtime

cgroups + user_1.slice |_ nginx_user_1.service |_ mariadb_user_1.service ### /etc/systemd/system/user_1.slice [Unit] Description=User One cgroup [Slice] CPUShares=512 MemoryLimit=2G ### /etc/systemd/system/nginx_user_1.service ... Slice=user_1.slice ### /etc/systemd/system/mariadb_user_1.service Slice=user_1.slice MemoryLimit=1.5G

cgroups systemd-cgtop(1) $ system-cgtop Path Tasks %CPU Memory Input/s Output/s / 230 - 7.0G - - /system.slice 3 - - - - /system.slice/atlas-nginx.service 2 - - - - /system.slice/atlas-php-fpm.service 6 - - - - /system.slice/auditd.service 1 - - - - /system.slice/avahi-daemon.service 2 - - - - /system.slice/cassandra.service 1 - - - - /system.slice/crond.service 19 - - - - /system.slice/dbus.service 1 - - - -

cgroups systemd-cgls(1) $ systemd-cgls ├─1 /usr/lib/systemd/systemd --system --deserialize 20 └─system.slice ├─tomcat.service │ ├─ 9914 /usr/bin/java -Djavax.sql.DataSource.Factory=org.apache ├─nginx.service │ ├─9776 nginx: master process /usr/sbin/ngin │ ├─9777 nginx: worker proces │ ├─9778 nginx: worker proces │ ├─9779 nginx: worker proces │ └─9780 nginx: worker proces ├─logstash_server.service │ └─10999 /usr/bin/java -server -Xms128m -Xmx128m ├─elasticsearch.service │ └─10967 /bin/java -server -Djava.net.preferIPv4Stack=true

socket activation think inetd … systemd listens to a socket on behalf of a service upon connection systemd starts the service and passes in socket fd’s systemd can listen on low ports and handoff sockets to unprivileged services

socket activation ### /etc/systemd/system/php_fpm_user1.socket [Socket] ListenStream=/home/user1/run/php-fpm.sock ### /etc/systemd/system/php_fpm_user1.service [Unit] Requires=php_fpm_user1.socket After=php_fpm_user1.socket [Service] User=user1 ExecStart=/binphp-fpm --fpm-config=/home/user1/php-fpm.conf

journald

journald is a replacement for syslog rich logging: arbitrary key=values on messages automatic logging for services stdout/stderr automatically stored in the journal optional Forward Secure Sealing for tamper- evident logging forwards to rsyslog by default (in fedora) for backwards compatibility

journald captures stdout/stderr free logging for services $ systemctl status elasticsearch elasticsearch.service - ElasticSearch Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled) Active: active (running) since Wed 2014-03-26 23:31:56 UTC; 1 day 2h ago Main PID: 10967 (java) CGroup: /system.slice/elasticsearch.service └─10967 /bin/java -server -Djava.net.preferIPv4Stack=true Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:05,143][WARN...n Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:11,254][WARN...) Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:14,773][WARN...n Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:16,312][WARN...) Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:12,847][WARN...) Mar 26 23:54:58 host1 elasticsearch[10967]: [2014-03-26 23:54:12,847][WARN...e Mar 26 23:55:33 host1 elasticsearch[10967]: Exception in thread "elasticse...e Mar 26 23:55:47 host1 elasticsearch[10967]: Exception in thread "elasticse...e

journald - journalctl(1) $ journalctl -u elasticsearch.service -f Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:05,143][WARN...n Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:11,254][WARN...) Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:14,773][WARN...n Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:16,312][WARN...) Mar 26 23:54:20 host1 elasticsearch[10967]: [2014-03-26 23:54:12,847][WARN...) Mar 26 23:54:58 host1 elasticsearch[10967]: [2014-03-26 23:54:12,847][WARN...e Mar 26 23:55:33 host1 elasticsearch[10967]: Exception in thread "elasticse...e Mar 26 23:55:47 host1 elasticsearch[10967]: Exception in thread "elasticse...e $ journalctl -u elasticsearch.service -o verbose -f Wed 2014-03-26 23:56:55.116468 UTC [s=d309a8ac4d054931ac687cbdef5c444c;i=81c235; b=e77996bef6aa452fa456b07f037b6a55;m=3c76fadbe01;t=4f58b392966b4;x=dff372bece62bee3] PRIORITY=6 SYSLOG_FACILITY=3 _HOSTNAME=host1 _TRANSPORT=stdout _SYSTEMD_SLICE=system.slice _COMM=java _EXE=/usr/java/jdk1.6.0_35/bin/java SYSLOG_IDENTIFIER=elasticsearch _SYSTEMD_CGROUP=/system.slice/elasticsearch.service _SYSTEMD_UNIT=elasticsearch.service _PID=10967 _CMDLINE=[444B blob data] MESSAGE=Exception in thread "elasticsearch[joe-onebox-f19][transport_server_worker] [T#3]" java.lang.OutOfMemoryError: Java heap space

journald - journalctl(1) `--since` argument simplifies log check utils, no need to keep state of log position between runs check-journal.rb (sensu/nagios) check https://github.com/sensu/sensu-community-plugins/blob/master/plugins/logging/check-journal.rb $ check-journal.rb --journalctl_args='-u elasticsearch.service' -q Error \ --since=-5minutes CheckJournal CRITICAL: 20 matches found for ‘Error’ in (threshold 1)

systemd-cat(1) similar to logger(1) $ crontab -l 0 1 * * * /bin/sh /usr/bin/logstash_optimize_daily.sh | \ systemd-cat -t logstash_optimize_daily ### capture both stdout and stderr: $ systemd-cat -t my_script /bin/do_something.sh

journald - rich logging adding key=vals to logs requires client library support official support: C/C++, python third party: go, node.js, ruby, ...

journald + logstash + kibana better together

journald + logstash + kibana

journald + logstash + kibana getting logs from journald -> logstash: journal2gelf - https://github. com/systemd/journal2gelf beaver (WIP) - https://github.com/joemiller/beaver (my fork, not upstream yet)

journald caveat the journal is not yet very performant, high cpu usage with high traffic not currently a good choice for high throughput streams such as a busy web server access logs

thank you q & a additional systemd reading all of the man pages, especially: systemd.service(5), systemd.exec(5), systemd.unit(5), systemd.resource- control(5) http://0pointer.de/blog/projects/socket-activated-containers.html http://dynacont.net/documentation/linux/Useful_SystemD_commands/