Understanding fleet - Speaker Deck

Slide 1

Slide 1 text

FLEET IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH UNDERSTANDING @spesnova

Slide 2

Slide 2 text

@spesnova

Slide 3

Slide 3 text

Favorites &

Slide 4

Slide 4 text

What is fleet? IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 5

Slide 5 text

What is fleet (flt)? • distributed systemd • depends on systemd and etcd • low level scheduler

Slide 6

Slide 6 text

systemd • init system • used by redhat, debian, ubuntu etc… • simplify for application • no need to daemonize • don’t worry about syslog

Slide 7

Slide 7 text

systemd: 1. Create a unit file • unit: A unit is a configuration file that describes the properties of the process that you'd like to run

Slide 8

Slide 8 text

systemd: 2. Start the unit

Slide 9

Slide 9 text

systemd: 3. Check the status

Slide 10

Slide 10 text

systemd: 4. View the log

Slide 11

Slide 11 text

etcd • a distributed consistent key/value store • shared configuration • distibuted locking • driven by Raft

Slide 12

Slide 12 text

etcd: List keys

Slide 13

Slide 13 text

etcd: Set / Get a key-value

Slide 14

Slide 14 text

fleet • simple orchestration • abstraction between machines and applications

Slide 15

Slide 15 text

fleet: Example Scheduling • 4 CoreOS machines in fleet cluster

Slide 16

Slide 16 text

fleet: 1. Create a unit file

Slide 17

Slide 17 text

fleet: 2. Start the unit • the unit assigned to 172.17.8.104

Slide 18

Slide 18 text

fleet: 3. Check the status

Slide 19

Slide 19 text

fleet: 4. View the log

Slide 20

Slide 20 text

Job Scheduling Flow IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 21

Slide 21 text

Job Scheduling Flow

Slide 22

Slide 22 text

Job Scheduling Flow • fleet engine makes scheduling decisions • fleet agent executes units

Slide 23

Slide 23 text

Architecture IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 24

Slide 24 text

fleet daemon • Every machine in the fleet cluster runs a single fleetd daemon • fleetd is a binary Cluster fleetd

Slide 25

Slide 25 text

fleetd encapsulates 2 roles engine agent fleetd

Slide 26

Slide 26 text

fleet engine • only one engine is running at a time fleetd => engine

Slide 27

Slide 27 text

engine: lease model • a lease model to enforce that only one engine is running at a time • every time a reconciliation is due, an engine will attempt to take a lease on etcd engine

Slide 28

Slide 28 text

fleet agent • a single agent runs on every machines fleetd => agent

Slide 29

Slide 29 text

Reconciliation • reconcile the actual state with the desired state • reconciliation loop • triggered periodically • triggered by certain events from etcd current state desired state

Slide 30

Slide 30 text

engine: reconcil • gathers a snapshot of the overall state of the cluster • then, attempts to reconcile engine

Slide 31

Slide 31 text

agent: reconcil • actually executing Units on systems • reporting the state of units to etcd agent

Slide 32

Slide 32 text

agent: reconcil • communicates with the local systemd instance over D-Bus CoreOS systemd ﬂeetd

Slide 33

Slide 33 text

engine: algorithm • ”least-loaded" scheduling algorithm • no resource management for simplicity engine

Slide 34

Slide 34 text

etcd as registry

Slide 35

Slide 35 text

etcd as registry • the sole datastore in a fleet cluster • internal communication between engines and agents

Slide 36

Slide 36 text

etcd as registry

Slide 37

Slide 37 text

etcd as registry

Slide 38

Slide 38 text

Unit States IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 39

Slide 39 text

Cluster-level Unit State • inactive: known by fleet, but not assigned to a machine • loaded: assigned to a machine and loaded into systemd there, but not started • launched: loaded into systemd, and fleet has called the equivalent of systemctl start inactive loaded launched (unknown)

Slide 40

Slide 40 text

Cluster-level Unit State • DSTATE: desired state • STATE: current state

Slide 41

Slide 41 text

systemd-level Unit State • ACTIVE: the high-level unit activation state, i.e. generalization of SUB • SUB: the low-level unit activation state, values depend on unit type

Slide 42

Slide 42 text

Cluster-level Unit State

Slide 43

Slide 43 text

Change the state via fleetctl

Slide 44

Slide 44 text

Change the state via fleetctl inactive loaded launched $ fleetctl start inactive loaded (unknown) $ fleetctl destroy

Slide 45

Slide 45 text

Change the state via fleetctl loaded launched $ fleetctl stop $ fleetctl stop $ fleetctl destroy invalid action inactive (unknown) inactive

Slide 46

Slide 46 text

fleetctl IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 47

Slide 47 text

Listing machines and units • list-machines • list-unit-files • list-units

Slide 48

Slide 48 text

fleetctl list-machines • Enumerate the current hosts in the cluster

Slide 49

Slide 49 text

fleetctl list-unit-files • List the units that exist in the cluster • You can check “desire” and “current” state of the unit

Slide 50

Slide 50 text

fleetctl list-units • List the current state of units in the cluster

Slide 51

Slide 51 text

Scheduling a unit • submit, load, start • stop, unload, destroy

Slide 52

Slide 52 text

fleetctl submit • Upload one or more units to the cluster without starting them • Desire state: inactive

Slide 53

Slide 53 text

fleetctl load • Schedule one or more units in the cluster • First submitting them if necessary • Desire state: loaded

Slide 54

Slide 54 text

fleetctl start • Instruct systemd to start one or more units in the cluster • First submitting them if necessary • Desire state: launched

Slide 55

Slide 55 text

fleetctl stop • Instruct systemd to stop one or more units in the cluster • Desire state: loaded

Slide 56

Slide 56 text

fleetctl unload • Unschedule one or more units in the cluster • Desire state: inactive

Slide 57

Slide 57 text

fleetctl destroy • Destroy one or more units in the cluster • Desire state: (unknown)

Slide 58

Slide 58 text

Printing info for a unit • cat • status • journal

Slide 59

Slide 59 text

fleetctl cat • Output the contents of a submitted unit

Slide 60

Slide 60 text

fleetctl status • Output the contents of a submitted unit

Slide 61

Slide 61 text

fleetctl journal • Print the journal of a unit in the cluster to stdout

Slide 62

Slide 62 text

Administration • ssh

Slide 63

Slide 63 text

fleetctl ssh • Open interactive shell on a machine in the cluster

Slide 64

Slide 64 text

Unit files IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 65

Slide 65 text

Unit File Name . or @. The unit name must be of the form:

Slide 66

Slide 66 text

Unit File Name: Example . or @ foo.service bar.service foo.socket bar.timer [email protected] [email protected]

Slide 67

Slide 67 text

Unit File Name: Requirements • : • must not be an empty string • can only contain [a-zA-Z0-9:_.@-]+ . or @

Slide 68

Slide 68 text

Unit File Name: Requirements • : • can be empty • can only contain [a-zA-Z0-9:_.@-]* . or @

Slide 69

Slide 69 text

Unit File Name: Requirements • : must be one of the following unit types: service socket device mount automount timer path . or @

Slide 70

Slide 70 text

Unit File Sections • Unit • Service • X-Fleet [Unit] generic_unit_option_1 generic_unit_option_2 [Service] service_specific_option_1 service_specific_option_2 service_specific_option_3 [X-Fleet] fleet_specific_option

Slide 71

Slide 71 text

Unit Section • Used to define generic information about a unit • Options that are common for all unit-types are generally placed here [Unit] Description=foo service # Requirements Requires=bar.service # Dependency ordering After=bar.service

Slide 72

Slide 72 text

Service Section • Used to set directives that are specific to service units • Most of the unit-types above have associated sections for unit- type-specific information • See man page: [Service] TimeoutStartSec=0 # Pre-start and Start ExecStartPre=-/usr/bin/docker kill foo ExecStartPre=-/usr/bin/docker rm foo ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo # Stop ExecStop=/usr/bin/docker stop foo http://www.freedesktop.org/software/systemd/man/systemd.unit.html

Slide 73

Slide 73 text

X-Fleet Section • Used to set scheduling requirements for the unit for use with fleet • Using this section, you can require that certain conditions be true in order for a unit to be scheduled on a host [X-Fleet] Conflicts=foo.*.service MachineMetadata=location=tokyo

Slide 74

Slide 74 text

fleet-specific Options in X-Fleet

Slide 75

Slide 75 text

Template Unit Files 1. Create a unit file whose name matches the: 2. Instantiate units by creating new units that match the instance pattern: @. @.

Slide 76

Slide 76 text

Template Unit Files Example

Slide 77

Slide 77 text

systemd Specifiers When evaluating the [X-Fleet] section, fleet supports a subset of systemd's specifiers to perform variable substitution.

Slide 78

Slide 78 text

systemd Specifiers Example [X-Fleet] MachineOf=%p.socket

Slide 79

Slide 79 text

Unit Scheduling IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 80

Slide 80 text

Global units registry • Global units are not scheduled through the engine • Non-global units are scheduled by the engine Non-global units engine Global / Non-global units

Slide 81

Slide 81 text

Global / Non-global units • Non-global units are scheduled by the engine • Global units are not scheduled through the engine (A global unit is one with Global=true in its X-Fleet section, fleet agents still check the MachineMetadata option before starting them. Other options are ignored.)

Slide 82

Slide 82 text

Control the engine's scheduling decision • Schedule unit to specific machine • Schedule unit to machine with specific metadata • Schedule unit next to another unit • Schedule unit away from other unit(s)

Slide 83

Slide 83 text

Schedule unit to specific machine registry Non-global units engine 4588e8cf b76e2b8a dc023323 [X-Fleet] MachineID=4588e8cf94244e4c88bfb0a7171efa72

Slide 84

Slide 84 text

Schedule unit to specific machine

Slide 85

Slide 85 text

Schedule unit to machine with specific metadata registry Non-global units engine metadata=“region=us-west" [X-Fleet] MachineMetadata="region=us-west-1"

Slide 86

Slide 86 text

Pull Request and Issue about Dynamic Metadata will be supported in fleet 0.10.0?

Slide 87

Slide 87 text

Schedule unit next to another unit registry Non-global units engine foo.service [X-Fleet] MachineOf=foo.service

Slide 88

Slide 88 text

Schedule unit away from other unit(s) registry Non-global units engine foo.service [X-Fleet] Conflicts=foo.service Conflicts!

Slide 89

Slide 89 text

Schedule unit away from other unit(s) registry Non-global units engine [email protected] [X-Fleet] Conflicts=hello@*.service Conflicts! [email protected] Conflicts! The value of the Conflicts option is a glob pattern defining

Slide 90

Slide 90 text

Configurations IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 91

Slide 91 text

General Options • verbosity • etcd_servers • etcd_request_timeout • etcd_cafile, etcd_keyfile, etcd_certfile • public_ip • metadata • agent_ttl • engine_reconcil_interval See official document for each option’s detail: https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options

Slide 92

Slide 92 text

agent_ttl • An Agent will be considered dead if it exceeds this amount of time to communicate with the Registry • The agent will attempt a heartbeat at half of this value agent heartbeat agent is alive? I’m alive registry

Slide 93

Slide 93 text

Configuration Sources • an INI-formatted config file • environment variables The fleetd daemon uses two sources for configuration parameters:

Slide 94

Slide 94 text

INI-formatted Config File # Lower the logging threshold. Acceptable values are 0, 1, and 2. A higher # value corresponds to a lower logging threshold. verbosity=0 # An Agent will be considered dead if it exceeds this amount of time to # communicate with the Registry. The agent will attempt a heartbeat at half # of this value. agent_ttl="36s" • fleet will look at for this config file by default • The flag may be passed the location to the fleetd binary /etc/fleet/fleet.conf —config

Slide 95

Slide 95 text

Environment Variables # Enable debug logging by setting “verbosity” option to 1 $ FLEET_VERBOSITY=1 /usr/bin/fleetd # Increase “agent_ttl” to default 36s from 30s (default) $ FLEET_AGENT_TTL=36s /usr/bin/fleetd • Options provided in an environment variable will override the corresponding option provided in a config file • simply prefix the name of a given option with 'FLEET_', while uppercasing the rest of the name

Slide 96

Slide 96 text

Configure via cloud-config #cloud-config coreos: fleet: verbosity: 1 agent-ttl: 36s public-ip: $public_ipv4 metadata: region=us-west • The coreos.fleet.* parameters allow for the configuration of fleet through environment variables.

Slide 97

Slide 97 text

fleet API v1 IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 98

Slide 98 text

Entities • Units • Unit systems-level States • Machines The fleet API allows you to manage the state of the cluster using JSON over HTTP /units /state /machines https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md

Slide 99

Slide 99 text

Unit Entity • This simply declares what should be happening; the backend system still has to react to the changes in this desired state (cluster-level) fleetd desire state (cluster-level state) current state (cluster-level state) /units /units

Slide 100

Slide 100 text

Unit Entity • If you would like to get the systemd-level state, communicate with UnitState entities desire state (cluster-level state) current state (system-level state) /units /state fleetd

Slide 101

Slide 101 text

Other things IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

Slide 102

Slide 102 text

Releases WIP Feature Patches 0.9.2 master 0.9.1 IUUQTHJUIVCDPNDPSFPTqFFUSFMFBTFT • New Features • Changes • Bugs Fixed

Slide 103

Slide 103 text

end :) IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH