Understanding fleet

Speaking at CoreOS Meetup Tokyo #1
http://coreos-meetup-tokyo.connpass.com/event/12596/

- fleet 0.9.2
- etcd 0.4.9

1. FLEET IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH UNDERSTANDING @spesnova
2. @spesnova
3. Favorites &
4. What is fleet? IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
5. What is fleet (flt)? • distributed systemd • depends on systemd and etcd • low level scheduler
6. systemd • init system • used by redhat, debian, ubuntu etc… • simplify for application • no need to daemonize • don’t worry about syslog
7. systemd: 1. Create a unit file • unit: A unit is a configuration file that describes the properties of the process that you'd like to run
8. systemd: 2. Start the unit
9. systemd: 3. Check the status
10. systemd: 4. View the log
11. etcd • a distributed consistent key/value store • shared configuration • distibuted locking • driven by Raft
12. etcd: List keys
13. etcd: Set / Get a key-value
14. fleet • simple orchestration • abstraction between machines and applications
15. fleet: Example Scheduling • 4 CoreOS machines in fleet cluster
16. fleet: 1. Create a unit file
17. fleet: 2. Start the unit • the unit assigned to 172.17.8.104
18. fleet: 3. Check the status
19. fleet: 4. View the log
20. Job Scheduling Flow IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
21. Job Scheduling Flow
22. Job Scheduling Flow • fleet engine makes scheduling decisions • fleet agent executes units
23. Architecture IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
24. fleet daemon • Every machine in the fleet cluster runs a single fleetd daemon • fleetd is a binary Cluster fleetd
25. fleetd encapsulates 2 roles engine agent fleetd
26. fleet engine • only one engine is running at a time fleetd => engine
27. engine: lease model • a lease model to enforce that only one engine is running at a time • every time a reconciliation is due, an engine will attempt to take a lease on etcd engine
28. fleet agent • a single agent runs on every machines fleetd => agent
29. Reconciliation • reconcile the actual state with the desired state • reconciliation loop • triggered periodically • triggered by certain events from etcd current state desired state
30. engine: reconcil • gathers a snapshot of the overall state of the cluster • then, attempts to reconcile engine
31. agent: reconcil • actually executing Units on systems • reporting the state of units to etcd agent
32. agent: reconcil • communicates with the local systemd instance over D-Bus CoreOS systemd fleetd
33. engine: algorithm • ”least-loaded" scheduling algorithm • no resource management for simplicity engine
34. etcd as registry
35. etcd as registry • the sole datastore in a fleet cluster • internal communication between engines and agents
36. etcd as registry
37. etcd as registry
38. Unit States IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
39. Cluster-level Unit State • inactive: known by fleet, but not assigned to a machine • loaded: assigned to a machine and loaded into systemd there, but not started • launched: loaded into systemd, and fleet has called the equivalent of systemctl start inactive loaded launched (unknown)
40. Cluster-level Unit State • DSTATE: desired state • STATE: current state
41. systemd-level Unit State • ACTIVE: the high-level unit activation state, i.e. generalization of SUB • SUB: the low-level unit activation state, values depend on unit type
42. Cluster-level Unit State
43. Change the state via fleetctl
44. Change the state via fleetctl inactive loaded launched $ fleetctl start inactive loaded (unknown) $ fleetctl destroy
45. Change the state via fleetctl loaded launched $ fleetctl stop $ fleetctl stop $ fleetctl destroy invalid action inactive (unknown) inactive
46. fleetctl IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
47. Listing machines and units • list-machines • list-unit-files • list-units
48. fleetctl list-machines • Enumerate the current hosts in the cluster
49. fleetctl list-unit-files • List the units that exist in the cluster • You can check “desire” and “current” state of the unit
50. fleetctl list-units • List the current state of units in the cluster
51. Scheduling a unit • submit, load, start • stop, unload, destroy
52. fleetctl submit • Upload one or more units to the cluster without starting them • Desire state: inactive
53. fleetctl load • Schedule one or more units in the cluster • First submitting them if necessary • Desire state: loaded
54. fleetctl start • Instruct systemd to start one or more units in the cluster • First submitting them if necessary • Desire state: launched
55. fleetctl stop • Instruct systemd to stop one or more units in the cluster • Desire state: loaded
56. fleetctl unload • Unschedule one or more units in the cluster • Desire state: inactive
57. fleetctl destroy • Destroy one or more units in the cluster • Desire state: (unknown)
58. Printing info for a unit • cat • status • journal
59. fleetctl cat • Output the contents of a submitted unit
60. fleetctl status • Output the contents of a submitted unit
61. fleetctl journal • Print the journal of a unit in the cluster to stdout
62. Administration • ssh
63. fleetctl ssh • Open interactive shell on a machine in the cluster
64. Unit files IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
65. Unit File Name <string>.<suffix> or <string>@<instance>.<suffix> The unit name must be of the form:
66. Unit File Name: Example <string>.<suffix> or <string>@<instance.suffix> foo.service bar.service foo.socket bar.timer foo@1.service foo@2.service
67. Unit File Name: Requirements • <string>: • must not be an empty string • can only contain [a-zA-Z0-9:_.@-]+ <string>.<suffix> or <string>@<instance.suffix>
68. Unit File Name: Requirements • <instance>: • can be empty • can only contain [a-zA-Z0-9:_.@-]* <string>.<suffix> or <string>@<instance.suffix>
69. Unit File Name: Requirements • <suffix>: must be one of the following unit types: service socket device mount automount timer path <string>.<suffix> or <string>@<instance.suffix>
70. Unit File Sections • Unit • Service • X-Fleet [Unit] generic_unit_option_1 generic_unit_option_2 [Service] service_specific_option_1 service_specific_option_2 service_specific_option_3 [X-Fleet] fleet_specific_option
71. Unit Section • Used to define generic information about a unit • Options that are common for all unit-types are generally placed here [Unit] Description=foo service # Requirements Requires=bar.service # Dependency ordering After=bar.service
72. Service Section • Used to set directives that are specific to service units • Most of the unit-types above have associated sections for unit- type-specific information • See man page: [Service] TimeoutStartSec=0 # Pre-start and Start ExecStartPre=-/usr/bin/docker kill foo ExecStartPre=-/usr/bin/docker rm foo ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo # Stop ExecStop=/usr/bin/docker stop foo http://www.freedesktop.org/software/systemd/man/systemd.unit.html
73. X-Fleet Section • Used to set scheduling requirements for the unit for use with fleet • Using this section, you can require that certain conditions be true in order for a unit to be scheduled on a host [X-Fleet] Conflicts=foo.*.service MachineMetadata=location=tokyo
74. fleet-specific Options in X-Fleet
75. Template Unit Files 1. Create a unit file whose name matches the: 2. Instantiate units by creating new units that match the instance pattern: <name>@.<suffix> <name>@<instance>.<suffix>
76. Template Unit Files Example
77. systemd Specifiers When evaluating the [X-Fleet] section, fleet supports a subset of systemd's specifiers to perform variable substitution.
78. systemd Specifiers Example [X-Fleet] MachineOf=%p.socket
79. Unit Scheduling IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
80. Global units registry • Global units are not scheduled through the engine • Non-global units are scheduled by the engine Non-global units engine Global / Non-global units
81. Global / Non-global units • Non-global units are scheduled by the engine • Global units are not scheduled through the engine (A global unit is one with Global=true in its X-Fleet section, fleet agents still check the MachineMetadata option before starting them. Other options are ignored.)
82. Control the engine's scheduling decision • Schedule unit to specific machine • Schedule unit to machine with specific metadata • Schedule unit next to another unit • Schedule unit away from other unit(s)
83. Schedule unit to specific machine registry Non-global units engine 4588e8cf b76e2b8a dc023323 [X-Fleet] MachineID=4588e8cf94244e4c88bfb0a7171efa72
84. Schedule unit to specific machine
85. Schedule unit to machine with specific metadata registry Non-global units engine metadata=“region=us-west" [X-Fleet] MachineMetadata="region=us-west-1"
86. Pull Request and Issue about Dynamic Metadata will be supported in fleet 0.10.0?
87. Schedule unit next to another unit registry Non-global units engine foo.service [X-Fleet] MachineOf=foo.service
88. Schedule unit away from other unit(s) registry Non-global units engine foo.service [X-Fleet] Conflicts=foo.service Conflicts!
89. Schedule unit away from other unit(s) registry Non-global units engine hello@2.service [X-Fleet] Conflicts=hello@*.service Conflicts! hello@1.service Conflicts! The value of the Conflicts option is a glob pattern defining
90. Configurations IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
91. General Options • verbosity • etcd_servers • etcd_request_timeout • etcd_cafile, etcd_keyfile, etcd_certfile • public_ip • metadata • agent_ttl • engine_reconcil_interval See official document for each option’s detail: https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options
92. agent_ttl • An Agent will be considered dead if it exceeds this amount of time to communicate with the Registry • The agent will attempt a heartbeat at half of this value agent heartbeat agent is alive? I’m alive registry
93. Configuration Sources • an INI-formatted config file • environment variables The fleetd daemon uses two sources for configuration parameters:
94. INI-formatted Config File # Lower the logging threshold. Acceptable values are 0, 1, and 2. A higher # value corresponds to a lower logging threshold. verbosity=0 # An Agent will be considered dead if it exceeds this amount of time to # communicate with the Registry. The agent will attempt a heartbeat at half # of this value. agent_ttl="36s" • fleet will look at for this config file by default • The flag may be passed the location to the fleetd binary /etc/fleet/fleet.conf —config
95. Environment Variables # Enable debug logging by setting “verbosity” option to 1 $ FLEET_VERBOSITY=1 /usr/bin/fleetd # Increase “agent_ttl” to default 36s from 30s (default) $ FLEET_AGENT_TTL=36s /usr/bin/fleetd • Options provided in an environment variable will override the corresponding option provided in a config file • simply prefix the name of a given option with 'FLEET_', while uppercasing the rest of the name
96. Configure via cloud-config #cloud-config coreos: fleet: verbosity: 1 agent-ttl: 36s public-ip: $public_ipv4 metadata: region=us-west • The coreos.fleet.* parameters allow for the configuration of fleet through environment variables.
97. fleet API v1 IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
98. Entities • Units • Unit systems-level States • Machines The fleet API allows you to manage the state of the cluster using JSON over HTTP /units /state /machines https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md
99. Unit Entity • This simply declares what should be happening; the backend system still has to react to the changes in this desired state (cluster-level) fleetd desire state (cluster-level state) current state (cluster-level state) /units /units
100. Unit Entity • If you would like to get the systemd-level state, communicate with UnitState entities desire state (cluster-level state) current state (system-level state) /units /state fleetd
101. Other things IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
102. Releases WIP Feature Patches 0.9.2 master 0.9.1 IUUQTHJUIVCDPNDPSFPTqFFUSFMFBTFT • New Features • Changes • Bugs Fixed
103. end :) IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH