Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding fleet

Understanding fleet

Speaking at CoreOS Meetup Tokyo #1
http://coreos-meetup-tokyo.connpass.com/event/12596/

- fleet 0.9.2
- etcd 0.4.9

Seigo Uchida

April 09, 2015
Tweet

More Decks by Seigo Uchida

Other Decks in Technology

Transcript

  1. What is fleet (flt)? • distributed systemd • depends on

    systemd and etcd • low level scheduler
  2. systemd • init system • used by redhat, debian, ubuntu

    etc… • simplify for application • no need to daemonize • don’t worry about syslog
  3. systemd: 1. Create a unit file • unit: A unit

    is a configuration file that describes the properties of the process that you'd like to run
  4. fleet daemon • Every machine in the fleet cluster runs

    a single fleetd daemon • fleetd is a binary Cluster fleetd
  5. engine: lease model • a lease model to enforce that

    only one engine is running at a time • every time a reconciliation is due, an engine will attempt to take a lease on etcd engine
  6. Reconciliation • reconcile the actual state with the desired state

    • reconciliation loop • triggered periodically • triggered by certain events from etcd current state desired state
  7. engine: reconcil • gathers a snapshot of the overall state

    of the cluster • then, attempts to reconcile engine
  8. etcd as registry • the sole datastore in a fleet

    cluster • internal communication between engines and agents
  9. Cluster-level Unit State • inactive: known by fleet, but not

    assigned to a machine • loaded: assigned to a machine and loaded into systemd there, but not started • launched: loaded into systemd, and fleet has called the equivalent of systemctl start inactive loaded launched (unknown)
  10. systemd-level Unit State • ACTIVE: the high-level unit activation state,

    i.e. generalization of SUB • SUB: the low-level unit activation state, values depend on unit type
  11. Change the state via fleetctl inactive loaded launched $ fleetctl

    start inactive loaded (unknown) $ fleetctl destroy
  12. Change the state via fleetctl loaded launched $ fleetctl stop

    $ fleetctl stop $ fleetctl destroy invalid action inactive (unknown) inactive
  13. fleetctl list-unit-files • List the units that exist in the

    cluster • You can check “desire” and “current” state of the unit
  14. fleetctl submit • Upload one or more units to the

    cluster without starting them • Desire state: inactive
  15. fleetctl load • Schedule one or more units in the

    cluster • First submitting them if necessary • Desire state: loaded
  16. fleetctl start • Instruct systemd to start one or more

    units in the cluster • First submitting them if necessary • Desire state: launched
  17. fleetctl stop • Instruct systemd to stop one or more

    units in the cluster • Desire state: loaded
  18. fleetctl unload • Unschedule one or more units in the

    cluster • Desire state: inactive
  19. fleetctl destroy • Destroy one or more units in the

    cluster • Desire state: (unknown)
  20. Unit File Name: Requirements • <string>: • must not be

    an empty string • can only contain [a-zA-Z0-9:_.@-]+ <string>.<suffix> or <string>@<instance.suffix>
  21. Unit File Name: Requirements • <instance>: • can be empty

    • can only contain [a-zA-Z0-9:_.@-]* <string>.<suffix> or <string>@<instance.suffix>
  22. Unit File Name: Requirements • <suffix>: must be one of

    the following unit types: service socket device mount automount timer path <string>.<suffix> or <string>@<instance.suffix>
  23. Unit File Sections • Unit • Service • X-Fleet [Unit]

    generic_unit_option_1 generic_unit_option_2 [Service] service_specific_option_1 service_specific_option_2 service_specific_option_3 [X-Fleet] fleet_specific_option
  24. Unit Section • Used to define generic information about a

    unit • Options that are common for all unit-types are generally placed here [Unit] Description=foo service # Requirements Requires=bar.service # Dependency ordering After=bar.service
  25. Service Section • Used to set directives that are specific

    to service units • Most of the unit-types above have associated sections for unit- type-specific information • See man page: [Service] TimeoutStartSec=0 # Pre-start and Start ExecStartPre=-/usr/bin/docker kill foo ExecStartPre=-/usr/bin/docker rm foo ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo # Stop ExecStop=/usr/bin/docker stop foo http://www.freedesktop.org/software/systemd/man/systemd.unit.html
  26. X-Fleet Section • Used to set scheduling requirements for the

    unit for use with fleet • Using this section, you can require that certain conditions be true in order for a unit to be scheduled on a host [X-Fleet] Conflicts=foo.*.service MachineMetadata=location=tokyo
  27. Template Unit Files 1. Create a unit file whose name

    matches the: 2. Instantiate units by creating new units that match the instance pattern: <name>@.<suffix> <name>@<instance>.<suffix>
  28. systemd Specifiers When evaluating the [X-Fleet] section, fleet supports a

    subset of systemd's specifiers to perform variable substitution.
  29. Global units registry • Global units are not scheduled through

    the engine • Non-global units are scheduled by the engine Non-global units engine Global / Non-global units
  30. Global / Non-global units • Non-global units are scheduled by

    the engine • Global units are not scheduled through the engine (A global unit is one with Global=true in its X-Fleet section, fleet agents still check the MachineMetadata option before starting them. Other options are ignored.)
  31. Control the engine's scheduling decision • Schedule unit to specific

    machine • Schedule unit to machine with specific metadata • Schedule unit next to another unit • Schedule unit away from other unit(s)
  32. Schedule unit to specific machine registry Non-global units engine 4588e8cf

    b76e2b8a dc023323 [X-Fleet] MachineID=4588e8cf94244e4c88bfb0a7171efa72
  33. Schedule unit to machine with specific metadata registry Non-global units

    engine metadata=“region=us-west" [X-Fleet] MachineMetadata="region=us-west-1"
  34. Schedule unit next to another unit registry Non-global units engine

    foo.service [X-Fleet] MachineOf=foo.service
  35. Schedule unit away from other unit(s) registry Non-global units engine

    foo.service [X-Fleet] Conflicts=foo.service Conflicts!
  36. Schedule unit away from other unit(s) registry Non-global units engine

    [email protected] [X-Fleet] Conflicts=hello@*.service Conflicts! [email protected] Conflicts! The value of the Conflicts option is a glob pattern defining
  37. General Options • verbosity • etcd_servers • etcd_request_timeout • etcd_cafile,

    etcd_keyfile, etcd_certfile • public_ip • metadata • agent_ttl • engine_reconcil_interval See official document for each option’s detail: https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options
  38. agent_ttl • An Agent will be considered dead if it

    exceeds this amount of time to communicate with the Registry • The agent will attempt a heartbeat at half of this value agent heartbeat agent is alive? I’m alive registry
  39. Configuration Sources • an INI-formatted config file • environment variables

    The fleetd daemon uses two sources for configuration parameters:
  40. INI-formatted Config File # Lower the logging threshold. Acceptable values

    are 0, 1, and 2. A higher # value corresponds to a lower logging threshold. verbosity=0 # An Agent will be considered dead if it exceeds this amount of time to # communicate with the Registry. The agent will attempt a heartbeat at half # of this value. agent_ttl="36s" • fleet will look at for this config file by default • The flag may be passed the location to the fleetd binary /etc/fleet/fleet.conf —config
  41. Environment Variables # Enable debug logging by setting “verbosity” option

    to 1 $ FLEET_VERBOSITY=1 /usr/bin/fleetd # Increase “agent_ttl” to default 36s from 30s (default) $ FLEET_AGENT_TTL=36s /usr/bin/fleetd • Options provided in an environment variable will override the corresponding option provided in a config file • simply prefix the name of a given option with 'FLEET_', while uppercasing the rest of the name
  42. Configure via cloud-config #cloud-config coreos: fleet: verbosity: 1 agent-ttl: 36s

    public-ip: $public_ipv4 metadata: region=us-west • The coreos.fleet.* parameters allow for the configuration of fleet through environment variables.
  43. Entities • Units • Unit systems-level States • Machines The

    fleet API allows you to manage the state of the cluster using JSON over HTTP /units /state /machines https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md
  44. Unit Entity • This simply declares what should be happening;

    the backend system still has to react to the changes in this desired state (cluster-level) fleetd desire state (cluster-level state) current state (cluster-level state) /units /units
  45. Unit Entity • If you would like to get the

    systemd-level state, communicate with UnitState entities desire state (cluster-level state) current state (system-level state) /units /state fleetd