Slide 1
Slide 1 text
FLEET
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
UNDERSTANDING
@spesnova
Slide 2
Slide 2 text
@spesnova
Slide 3
Slide 3 text
Favorites
&
Slide 4
Slide 4 text
What is fleet?
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 5
Slide 5 text
What is fleet (flt)?
• distributed systemd
• depends on systemd and etcd
• low level scheduler
Slide 6
Slide 6 text
systemd
• init system
• used by redhat, debian, ubuntu etc…
• simplify for application
• no need to daemonize
• don’t worry about syslog
Slide 7
Slide 7 text
systemd: 1. Create a unit file
• unit: A unit is a configuration file that describes the properties of the
process that you'd like to run
Slide 8
Slide 8 text
systemd: 2. Start the unit
Slide 9
Slide 9 text
systemd: 3. Check the status
Slide 10
Slide 10 text
systemd: 4. View the log
Slide 11
Slide 11 text
etcd
• a distributed consistent key/value store
• shared configuration
• distibuted locking
• driven by Raft
Slide 12
Slide 12 text
etcd: List keys
Slide 13
Slide 13 text
etcd: Set / Get a key-value
Slide 14
Slide 14 text
fleet
• simple orchestration
• abstraction between machines and applications
Slide 15
Slide 15 text
fleet: Example Scheduling
• 4 CoreOS machines in fleet cluster
Slide 16
Slide 16 text
fleet: 1. Create a unit file
Slide 17
Slide 17 text
fleet: 2. Start the unit
• the unit assigned to 172.17.8.104
Slide 18
Slide 18 text
fleet: 3. Check the status
Slide 19
Slide 19 text
fleet: 4. View the log
Slide 20
Slide 20 text
Job Scheduling Flow
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 21
Slide 21 text
Job Scheduling Flow
Slide 22
Slide 22 text
Job Scheduling Flow
• fleet engine makes scheduling decisions
• fleet agent executes units
Slide 23
Slide 23 text
Architecture
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 24
Slide 24 text
fleet daemon
• Every machine in the fleet cluster runs a single fleetd daemon
• fleetd is a binary
Cluster
fleetd
Slide 25
Slide 25 text
fleetd encapsulates 2 roles
engine
agent
fleetd
Slide 26
Slide 26 text
fleet engine
• only one engine is running at a time
fleetd
=> engine
Slide 27
Slide 27 text
engine: lease model
• a lease model to enforce that only one engine is running at a time
• every time a reconciliation is due, an engine will attempt to take a
lease on etcd
engine
Slide 28
Slide 28 text
fleet agent
• a single agent runs on every machines
fleetd
=> agent
Slide 29
Slide 29 text
Reconciliation
• reconcile the actual state with the desired state
• reconciliation loop
• triggered periodically
• triggered by certain events from etcd
current state desired state
Slide 30
Slide 30 text
engine: reconcil
• gathers a snapshot of the overall state of the cluster
• then, attempts to reconcile
engine
Slide 31
Slide 31 text
agent: reconcil
• actually executing Units on systems
• reporting the state of units to etcd
agent
Slide 32
Slide 32 text
agent: reconcil
• communicates with the local systemd instance over D-Bus
CoreOS
systemd
fleetd
Slide 33
Slide 33 text
engine: algorithm
• ”least-loaded" scheduling algorithm
• no resource management for simplicity
engine
Slide 34
Slide 34 text
etcd as registry
Slide 35
Slide 35 text
etcd as registry
• the sole datastore in a fleet cluster
• internal communication between
engines and agents
Slide 36
Slide 36 text
etcd as registry
Slide 37
Slide 37 text
etcd as registry
Slide 38
Slide 38 text
Unit States
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 39
Slide 39 text
Cluster-level Unit State
• inactive: known by fleet, but not assigned to a machine
• loaded: assigned to a machine and loaded into systemd there, but not
started
• launched: loaded into systemd, and fleet has called the equivalent of
systemctl start
inactive loaded launched
(unknown)
Slide 40
Slide 40 text
Cluster-level Unit State
• DSTATE: desired state
• STATE: current state
Slide 41
Slide 41 text
systemd-level Unit State
• ACTIVE: the high-level unit activation state, i.e. generalization of SUB
• SUB: the low-level unit activation state, values depend on unit type
Slide 42
Slide 42 text
Cluster-level Unit State
Slide 43
Slide 43 text
Change the state via fleetctl
Slide 44
Slide 44 text
Change the state via fleetctl
inactive loaded launched
$ fleetctl start
inactive loaded
(unknown)
$ fleetctl destroy
Slide 45
Slide 45 text
Change the state via fleetctl
loaded launched
$ fleetctl stop
$ fleetctl stop
$ fleetctl destroy
invalid action
inactive
(unknown)
inactive
Slide 46
Slide 46 text
fleetctl
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 47
Slide 47 text
Listing machines and units
• list-machines
• list-unit-files
• list-units
Slide 48
Slide 48 text
fleetctl list-machines
• Enumerate the current hosts in the cluster
Slide 49
Slide 49 text
fleetctl list-unit-files
• List the units that exist in the cluster
• You can check “desire” and “current” state of the unit
Slide 50
Slide 50 text
fleetctl list-units
• List the current state of units in the cluster
Slide 51
Slide 51 text
Scheduling a unit
• submit, load, start
• stop, unload, destroy
Slide 52
Slide 52 text
fleetctl submit
• Upload one or more units to the cluster without starting them
• Desire state: inactive
Slide 53
Slide 53 text
fleetctl load
• Schedule one or more units in the cluster
• First submitting them if necessary
• Desire state: loaded
Slide 54
Slide 54 text
fleetctl start
• Instruct systemd to start one or more units in the cluster
• First submitting them if necessary
• Desire state: launched
Slide 55
Slide 55 text
fleetctl stop
• Instruct systemd to stop one or more units in the cluster
• Desire state: loaded
Slide 56
Slide 56 text
fleetctl unload
• Unschedule one or more units in the cluster
• Desire state: inactive
Slide 57
Slide 57 text
fleetctl destroy
• Destroy one or more units in the cluster
• Desire state: (unknown)
Slide 58
Slide 58 text
Printing info for a unit
• cat
• status
• journal
Slide 59
Slide 59 text
fleetctl cat
• Output the contents of a submitted unit
Slide 60
Slide 60 text
fleetctl status
• Output the contents of a submitted unit
Slide 61
Slide 61 text
fleetctl journal
• Print the journal of a unit in the cluster to stdout
Slide 62
Slide 62 text
Administration
• ssh
Slide 63
Slide 63 text
fleetctl ssh
• Open interactive shell on a machine in the cluster
Slide 64
Slide 64 text
Unit files
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 65
Slide 65 text
Unit File Name
.
or
@.
The unit name must be of the form:
Slide 66
Slide 66 text
Unit File Name: Example
.
or
@
foo.service
bar.service
foo.socket
bar.timer
[email protected]
[email protected]
Slide 67
Slide 67 text
Unit File Name: Requirements
• :
• must not be an empty string
• can only contain [a-zA-Z0-9:_.@-]+
.
or
@
Slide 68
Slide 68 text
Unit File Name: Requirements
• :
• can be empty
• can only contain [a-zA-Z0-9:_.@-]*
.
or
@
Slide 69
Slide 69 text
Unit File Name: Requirements
• : must be one of the following unit types:
service socket
device mount automount
timer
path
.
or
@
Slide 70
Slide 70 text
Unit File Sections
• Unit
• Service
• X-Fleet
[Unit]
generic_unit_option_1
generic_unit_option_2
[Service]
service_specific_option_1
service_specific_option_2
service_specific_option_3
[X-Fleet]
fleet_specific_option
Slide 71
Slide 71 text
Unit Section
• Used to define generic information about a unit
• Options that are common for all unit-types are generally
placed here
[Unit]
Description=foo service
# Requirements
Requires=bar.service
# Dependency ordering
After=bar.service
Slide 72
Slide 72 text
Service Section
• Used to set directives that are specific to service units
• Most of the unit-types above have associated sections for unit-
type-specific information
• See man page:
[Service]
TimeoutStartSec=0
# Pre-start and Start
ExecStartPre=-/usr/bin/docker kill foo
ExecStartPre=-/usr/bin/docker rm foo
ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo
# Stop
ExecStop=/usr/bin/docker stop foo
http://www.freedesktop.org/software/systemd/man/systemd.unit.html
Slide 73
Slide 73 text
X-Fleet Section
• Used to set scheduling requirements for the unit for use with fleet
• Using this section, you can require that certain conditions be true in
order for a unit to be scheduled on a host
[X-Fleet]
Conflicts=foo.*.service
MachineMetadata=location=tokyo
Slide 74
Slide 74 text
fleet-specific Options in X-Fleet
Slide 75
Slide 75 text
Template Unit Files
1. Create a unit file whose name matches the:
2. Instantiate units by creating new units that match the
instance pattern:
@.
@.
Slide 76
Slide 76 text
Template Unit Files Example
Slide 77
Slide 77 text
systemd Specifiers
When evaluating the [X-Fleet] section, fleet supports a subset of systemd's
specifiers to perform variable substitution.
Slide 78
Slide 78 text
systemd Specifiers Example
[X-Fleet]
MachineOf=%p.socket
Slide 79
Slide 79 text
Unit Scheduling
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 80
Slide 80 text
Global units
registry
• Global units are not scheduled through the engine
• Non-global units are scheduled by the engine
Non-global units engine
Global / Non-global units
Slide 81
Slide 81 text
Global / Non-global units
• Non-global units are scheduled by the engine
• Global units are not scheduled through the engine
(A global unit is one with Global=true in its X-Fleet section, fleet agents still check the
MachineMetadata option before starting them. Other options are ignored.)
Slide 82
Slide 82 text
Control the engine's scheduling decision
• Schedule unit to specific machine
• Schedule unit to machine with specific metadata
• Schedule unit next to another unit
• Schedule unit away from other unit(s)
Slide 83
Slide 83 text
Schedule unit to specific machine
registry
Non-global units
engine 4588e8cf
b76e2b8a
dc023323
[X-Fleet]
MachineID=4588e8cf94244e4c88bfb0a7171efa72
Slide 84
Slide 84 text
Schedule unit to specific machine
Slide 85
Slide 85 text
Schedule unit to machine with specific metadata
registry
Non-global units
engine
metadata=“region=us-west"
[X-Fleet]
MachineMetadata="region=us-west-1"
Slide 86
Slide 86 text
Pull Request and Issue about Dynamic Metadata
will be supported in fleet 0.10.0?
Slide 87
Slide 87 text
Schedule unit next to another unit
registry
Non-global units
engine
foo.service
[X-Fleet]
MachineOf=foo.service
Slide 88
Slide 88 text
Schedule unit away from other unit(s)
registry
Non-global units
engine
foo.service
[X-Fleet]
Conflicts=foo.service
Conflicts!
Slide 89
Slide 89 text
Schedule unit away from other unit(s)
registry
Non-global units
engine
[email protected]
[X-Fleet]
Conflicts=hello@*.service
Conflicts!
[email protected]
Conflicts!
The value of the Conflicts option is a glob pattern defining
Slide 90
Slide 90 text
Configurations
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 91
Slide 91 text
General Options
• verbosity
• etcd_servers
• etcd_request_timeout
• etcd_cafile, etcd_keyfile, etcd_certfile
• public_ip
• metadata
• agent_ttl
• engine_reconcil_interval
See official document for each option’s detail:
https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options
Slide 92
Slide 92 text
agent_ttl
• An Agent will be considered dead if it exceeds this amount of time to
communicate with the Registry
• The agent will attempt a heartbeat at half of this value
agent
heartbeat
agent is alive?
I’m alive
registry
Slide 93
Slide 93 text
Configuration Sources
• an INI-formatted config file
• environment variables
The fleetd daemon uses two sources for configuration parameters:
Slide 94
Slide 94 text
INI-formatted Config File
# Lower the logging threshold. Acceptable values are 0, 1, and 2. A higher
# value corresponds to a lower logging threshold.
verbosity=0
# An Agent will be considered dead if it exceeds this amount of time to
# communicate with the Registry. The agent will attempt a heartbeat at half
# of this value.
agent_ttl="36s"
• fleet will look at for this config file by default
• The flag may be passed the location to the fleetd binary
/etc/fleet/fleet.conf
—config
Slide 95
Slide 95 text
Environment Variables
# Enable debug logging by setting “verbosity” option to 1
$ FLEET_VERBOSITY=1 /usr/bin/fleetd
# Increase “agent_ttl” to default 36s from 30s (default)
$ FLEET_AGENT_TTL=36s /usr/bin/fleetd
• Options provided in an environment variable will override the corresponding
option provided in a config file
• simply prefix the name of a given option with 'FLEET_', while uppercasing the
rest of the name
Slide 96
Slide 96 text
Configure via cloud-config
#cloud-config
coreos:
fleet:
verbosity: 1
agent-ttl: 36s
public-ip: $public_ipv4
metadata: region=us-west
• The coreos.fleet.* parameters allow for the configuration of fleet
through environment variables.
Slide 97
Slide 97 text
fleet API v1
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 98
Slide 98 text
Entities
• Units
• Unit systems-level States
• Machines
The fleet API allows you to manage the state of the cluster using JSON over HTTP
/units
/state
/machines
https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md
Slide 99
Slide 99 text
Unit Entity
• This simply declares what should be happening; the backend system
still has to react to the changes in this desired state (cluster-level)
fleetd
desire state
(cluster-level state)
current state
(cluster-level state)
/units /units
Slide 100
Slide 100 text
Unit Entity
• If you would like to get the systemd-level state, communicate with
UnitState entities
desire state
(cluster-level state)
current state
(system-level state)
/units /state
fleetd
Slide 101
Slide 101 text
Other things
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
Slide 102
Slide 102 text
Releases
WIP Feature
Patches
0.9.2 master
0.9.1
IUUQTHJUIVCDPNDPSFPTqFFUSFMFBTFT
• New Features
• Changes
• Bugs Fixed
Slide 103
Slide 103 text
end :)
IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH