Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding fleet

Understanding fleet

Speaking at CoreOS Meetup Tokyo #1

- fleet 0.9.2
- etcd 0.4.9


Seigo Uchida

April 09, 2015



  2. @spesnova

  3. Favorites &


  5. What is fleet (flt)? • distributed systemd • depends on

    systemd and etcd • low level scheduler
  6. systemd • init system • used by redhat, debian, ubuntu

    etc… • simplify for application • no need to daemonize • don’t worry about syslog
  7. systemd: 1. Create a unit file • unit: A unit

    is a configuration file that describes the properties of the process that you'd like to run
  8. systemd: 2. Start the unit

  9. systemd: 3. Check the status

  10. systemd: 4. View the log

  11. etcd • a distributed consistent key/value store • shared configuration

    • distibuted locking • driven by Raft
  12. etcd: List keys

  13. etcd: Set / Get a key-value

  14. fleet • simple orchestration • abstraction between machines and applications

  15. fleet: Example Scheduling • 4 CoreOS machines in fleet cluster

  16. fleet: 1. Create a unit file

  17. fleet: 2. Start the unit • the unit assigned to
  18. fleet: 3. Check the status

  19. fleet: 4. View the log


  21. Job Scheduling Flow

  22. Job Scheduling Flow • fleet engine makes scheduling decisions •

    fleet agent executes units

  24. fleet daemon • Every machine in the fleet cluster runs

    a single fleetd daemon • fleetd is a binary Cluster fleetd
  25. fleetd encapsulates 2 roles engine agent fleetd

  26. fleet engine • only one engine is running at a

    time fleetd => engine
  27. engine: lease model • a lease model to enforce that

    only one engine is running at a time • every time a reconciliation is due, an engine will attempt to take a lease on etcd engine
  28. fleet agent • a single agent runs on every machines

    fleetd => agent
  29. Reconciliation • reconcile the actual state with the desired state

    • reconciliation loop • triggered periodically • triggered by certain events from etcd current state desired state
  30. engine: reconcil • gathers a snapshot of the overall state

    of the cluster • then, attempts to reconcile engine
  31. agent: reconcil • actually executing Units on systems • reporting

    the state of units to etcd agent
  32. agent: reconcil • communicates with the local systemd instance over

    D-Bus CoreOS systemd fleetd
  33. engine: algorithm • ”least-loaded" scheduling algorithm • no resource management

    for simplicity engine
  34. etcd as registry

  35. etcd as registry • the sole datastore in a fleet

    cluster • internal communication between engines and agents
  36. etcd as registry

  37. etcd as registry


  39. Cluster-level Unit State • inactive: known by fleet, but not

    assigned to a machine • loaded: assigned to a machine and loaded into systemd there, but not started • launched: loaded into systemd, and fleet has called the equivalent of systemctl start inactive loaded launched (unknown)
  40. Cluster-level Unit State • DSTATE: desired state • STATE: current

  41. systemd-level Unit State • ACTIVE: the high-level unit activation state,

    i.e. generalization of SUB • SUB: the low-level unit activation state, values depend on unit type
  42. Cluster-level Unit State

  43. Change the state via fleetctl

  44. Change the state via fleetctl inactive loaded launched $ fleetctl

    start inactive loaded (unknown) $ fleetctl destroy
  45. Change the state via fleetctl loaded launched $ fleetctl stop

    $ fleetctl stop $ fleetctl destroy invalid action inactive (unknown) inactive

  47. Listing machines and units • list-machines • list-unit-files • list-units

  48. fleetctl list-machines • Enumerate the current hosts in the cluster

  49. fleetctl list-unit-files • List the units that exist in the

    cluster • You can check “desire” and “current” state of the unit
  50. fleetctl list-units • List the current state of units in

    the cluster
  51. Scheduling a unit • submit, load, start • stop, unload,

  52. fleetctl submit • Upload one or more units to the

    cluster without starting them • Desire state: inactive
  53. fleetctl load • Schedule one or more units in the

    cluster • First submitting them if necessary • Desire state: loaded
  54. fleetctl start • Instruct systemd to start one or more

    units in the cluster • First submitting them if necessary • Desire state: launched
  55. fleetctl stop • Instruct systemd to stop one or more

    units in the cluster • Desire state: loaded
  56. fleetctl unload • Unschedule one or more units in the

    cluster • Desire state: inactive
  57. fleetctl destroy • Destroy one or more units in the

    cluster • Desire state: (unknown)
  58. Printing info for a unit • cat • status •

  59. fleetctl cat • Output the contents of a submitted unit

  60. fleetctl status • Output the contents of a submitted unit

  61. fleetctl journal • Print the journal of a unit in

    the cluster to stdout
  62. Administration • ssh

  63. fleetctl ssh • Open interactive shell on a machine in

    the cluster

  65. Unit File Name <string>.<suffix> or <string>@<instance>.<suffix> The unit name must

    be of the form:
  66. Unit File Name: Example <string>.<suffix> or <string>@<instance.suffix> foo.service bar.service foo.socket

    bar.timer foo@1.service foo@2.service
  67. Unit File Name: Requirements • <string>: • must not be

    an empty string • can only contain [a-zA-Z0-9:_.@-]+ <string>.<suffix> or <string>@<instance.suffix>
  68. Unit File Name: Requirements • <instance>: • can be empty

    • can only contain [a-zA-Z0-9:_.@-]* <string>.<suffix> or <string>@<instance.suffix>
  69. Unit File Name: Requirements • <suffix>: must be one of

    the following unit types: service socket device mount automount timer path <string>.<suffix> or <string>@<instance.suffix>
  70. Unit File Sections • Unit • Service • X-Fleet [Unit]

    generic_unit_option_1 generic_unit_option_2 [Service] service_specific_option_1 service_specific_option_2 service_specific_option_3 [X-Fleet] fleet_specific_option
  71. Unit Section • Used to define generic information about a

    unit • Options that are common for all unit-types are generally placed here [Unit] Description=foo service # Requirements Requires=bar.service # Dependency ordering After=bar.service
  72. Service Section • Used to set directives that are specific

    to service units • Most of the unit-types above have associated sections for unit- type-specific information • See man page: [Service] TimeoutStartSec=0 # Pre-start and Start ExecStartPre=-/usr/bin/docker kill foo ExecStartPre=-/usr/bin/docker rm foo ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo # Stop ExecStop=/usr/bin/docker stop foo http://www.freedesktop.org/software/systemd/man/systemd.unit.html
  73. X-Fleet Section • Used to set scheduling requirements for the

    unit for use with fleet • Using this section, you can require that certain conditions be true in order for a unit to be scheduled on a host [X-Fleet] Conflicts=foo.*.service MachineMetadata=location=tokyo
  74. fleet-specific Options in X-Fleet

  75. Template Unit Files 1. Create a unit file whose name

    matches the: 2. Instantiate units by creating new units that match the instance pattern: <name>@.<suffix> <name>@<instance>.<suffix>
  76. Template Unit Files Example

  77. systemd Specifiers When evaluating the [X-Fleet] section, fleet supports a

    subset of systemd's specifiers to perform variable substitution.
  78. systemd Specifiers Example [X-Fleet] MachineOf=%p.socket


  80. Global units registry • Global units are not scheduled through

    the engine • Non-global units are scheduled by the engine Non-global units engine Global / Non-global units
  81. Global / Non-global units • Non-global units are scheduled by

    the engine • Global units are not scheduled through the engine (A global unit is one with Global=true in its X-Fleet section, fleet agents still check the MachineMetadata option before starting them. Other options are ignored.)
  82. Control the engine's scheduling decision • Schedule unit to specific

    machine • Schedule unit to machine with specific metadata • Schedule unit next to another unit • Schedule unit away from other unit(s)
  83. Schedule unit to specific machine registry Non-global units engine 4588e8cf

    b76e2b8a dc023323 [X-Fleet] MachineID=4588e8cf94244e4c88bfb0a7171efa72
  84. Schedule unit to specific machine

  85. Schedule unit to machine with specific metadata registry Non-global units

    engine metadata=“region=us-west" [X-Fleet] MachineMetadata="region=us-west-1"
  86. Pull Request and Issue about Dynamic Metadata will be supported

    in fleet 0.10.0?
  87. Schedule unit next to another unit registry Non-global units engine

    foo.service [X-Fleet] MachineOf=foo.service
  88. Schedule unit away from other unit(s) registry Non-global units engine

    foo.service [X-Fleet] Conflicts=foo.service Conflicts!
  89. Schedule unit away from other unit(s) registry Non-global units engine

    hello@2.service [X-Fleet] Conflicts=hello@*.service Conflicts! hello@1.service Conflicts! The value of the Conflicts option is a glob pattern defining

  91. General Options • verbosity • etcd_servers • etcd_request_timeout • etcd_cafile,

    etcd_keyfile, etcd_certfile • public_ip • metadata • agent_ttl • engine_reconcil_interval See official document for each option’s detail: https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options
  92. agent_ttl • An Agent will be considered dead if it

    exceeds this amount of time to communicate with the Registry • The agent will attempt a heartbeat at half of this value agent heartbeat agent is alive? I’m alive registry
  93. Configuration Sources • an INI-formatted config file • environment variables

    The fleetd daemon uses two sources for configuration parameters:
  94. INI-formatted Config File # Lower the logging threshold. Acceptable values

    are 0, 1, and 2. A higher # value corresponds to a lower logging threshold. verbosity=0 # An Agent will be considered dead if it exceeds this amount of time to # communicate with the Registry. The agent will attempt a heartbeat at half # of this value. agent_ttl="36s" • fleet will look at for this config file by default • The flag may be passed the location to the fleetd binary /etc/fleet/fleet.conf —config
  95. Environment Variables # Enable debug logging by setting “verbosity” option

    to 1 $ FLEET_VERBOSITY=1 /usr/bin/fleetd # Increase “agent_ttl” to default 36s from 30s (default) $ FLEET_AGENT_TTL=36s /usr/bin/fleetd • Options provided in an environment variable will override the corresponding option provided in a config file • simply prefix the name of a given option with 'FLEET_', while uppercasing the rest of the name
  96. Configure via cloud-config #cloud-config coreos: fleet: verbosity: 1 agent-ttl: 36s

    public-ip: $public_ipv4 metadata: region=us-west • The coreos.fleet.* parameters allow for the configuration of fleet through environment variables.

  98. Entities • Units • Unit systems-level States • Machines The

    fleet API allows you to manage the state of the cluster using JSON over HTTP /units /state /machines https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md
  99. Unit Entity • This simply declares what should be happening;

    the backend system still has to react to the changes in this desired state (cluster-level) fleetd desire state (cluster-level state) current state (cluster-level state) /units /units
  100. Unit Entity • If you would like to get the

    systemd-level state, communicate with UnitState entities desire state (cluster-level state) current state (system-level state) /units /state fleetd

  102. Releases WIP Feature Patches 0.9.2 master 0.9.1 IUUQTHJUIVCDPNDPSFPTqFFUSFMFBTFT • New

    Features • Changes • Bugs Fixed