Who supervises supervisors?

Introduction to using systemd with Erlang

Transcript

1. @hauleth 2021
   Who supervises supervisors?
   Introduction to using systemd with Erlang
   1
   

   View Slide

2. • Hauleth - https://hauleth.dev
   
   • EEF Observability WG Member
   
   • OpenTelemetry Erlang Maintainer
   
   • Author of systemd Erlang library
   (and few other libraries)
   Who am I?
   2
   

   View Slide

3. Some knowledge
   needed
   I will not explain everything, there is no time for that
   3
   

   View Slide

4. systemd is ok
   but it has
   fl
   aws
   4
   

   View Slide

5. • What is init system
   
   • How it works on the high level
   
   • What systemd features are interesting for us
   
   • How we can use these features
   5
   

   View Slide

6. Erlang init
   App2
   App1 App3
   ?
   6
   

   View Slide

7. *nix init system
   How your system starts
   • First process ran by your OS
   
   • Run with PID 1
   
   • When it exits, everything goes down, either by shutdown or errors out
   
   • Responsible for starting everything else
   
   • Responsible for reaping zombies (in most cases)
   
   • In systemd it is the system supervisor (not all inits does that)
   7
   

   View Slide

8. Similarities between system init and Erlang init
   • When it dies, (virtual) machine dies
   
   • It is responsible for starting everything else
   
   • It reads job con
   fi
   guration and then starts required jobs
   
   • Listen on messages when to stop the (virtual) machine
   8
   

   View Slide

9. Why bother?
   9
   

   View Slide

10. "Top-level" process management
    • Ordering process startup
    
    • Restarting process when it died
    
    • Startup and shutdown hooks
    
    • Process isolation and hardening
    
    • Resource limits
    
    • System state monitoring
    10
    

    View Slide

11. Notifications and watchdog
    • Informing administrator about state of the process
    
    • Health checks whether the system is still alive
    
    • Triggering restarts from within the application
    
    • Storing opened
    fi
    le descriptors (like sockets) between application starts
    11
    

    View Slide

12. How to use it?
    12
    defmodule MyApp.Application do
    
    
    use Application
    
    
    def start(_, _) do
    
    
    children = [
    
    
    # …
    
    
    :systemd.ready()
    
    
    ]
    
    
    # …
    
    
    Supervisor.start_link(
    
    
    children,
    
    
    opts
    
    
    )
    
    
    end
    
    
    end
    [Service]
    
    
    # …
    
    
    Type=notify
    
    
    ExecStart=/path/to/rel/bin start
    

    View Slide

13. Centralised log management
    journald
    • All systems send logs to single place
    
    • No need for manual management of log rotation
    
    • Keeping metadata together with logs
    
    • Built-in log dispatching tooling
    
    • Supports both stdout and direct logging via socket
    
    • Logs stored in binary format
    13
    

    View Slide

14. How to use it?
    Version simple (well, in Erlang)
    14
    {deps, [systemd]}.
    

    View Slide

15. How to use it?
    Version hard
    15
    defp deps do
    
    
    [{:systemd, "~> 0.0"}]
    
    
    end
    defmodule MyApp.Application do
    
    
    use Application
    
    
    def start(_, _) do
    
    
    :logger.add_handlers(:systemd)
    
    
    end
    
    
    end
    

    View Slide

16. Lazy startup
    socket activation
    • Start application just when the request arrives
    
    • Keep sockets open as soon as system starts
    
    • Keep sockets open between application restarts
    
    • Use sockets from the privileged scope without superuser
    16
    

    View Slide

17. How to use it?
    17
    fds = :systemd.listen_fds()
    
    
    option =
    
    
    case fds do
    
    
    [{fd, _} | _] -> {:fd, fd}
    
    
    [fd | _] -> {:fd, fd}
    
    
    _ -> {:port, default_port}
    
    
    end
    
    
    {:ok, socket} = :gen_tcp.listen(0, [option, :inet6])
    

    View Slide

18. Demo time
    18
    

    View Slide

19. Questions?
    19
    

    View Slide