Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding fleet

Understanding fleet

Speaking at CoreOS Meetup Tokyo #1
http://coreos-meetup-tokyo.connpass.com/event/12596/

- fleet 0.9.2
- etcd 0.4.9

Seigo Uchida

April 09, 2015
Tweet

More Decks by Seigo Uchida

Other Decks in Technology

Transcript

  1. FLEET
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH
    UNDERSTANDING
    @spesnova

    View Slide

  2. @spesnova

    View Slide

  3. Favorites
    &

    View Slide

  4. What is fleet?
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  5. What is fleet (flt)?
    • distributed systemd
    • depends on systemd and etcd
    • low level scheduler

    View Slide

  6. systemd
    • init system
    • used by redhat, debian, ubuntu etc…
    • simplify for application
    • no need to daemonize
    • don’t worry about syslog

    View Slide

  7. systemd: 1. Create a unit file
    • unit: A unit is a configuration file that describes the properties of the
    process that you'd like to run

    View Slide

  8. systemd: 2. Start the unit

    View Slide

  9. systemd: 3. Check the status

    View Slide

  10. systemd: 4. View the log

    View Slide

  11. etcd
    • a distributed consistent key/value store
    • shared configuration
    • distibuted locking
    • driven by Raft

    View Slide

  12. etcd: List keys

    View Slide

  13. etcd: Set / Get a key-value

    View Slide

  14. fleet
    • simple orchestration
    • abstraction between machines and applications

    View Slide

  15. fleet: Example Scheduling
    • 4 CoreOS machines in fleet cluster

    View Slide

  16. fleet: 1. Create a unit file

    View Slide

  17. fleet: 2. Start the unit
    • the unit assigned to 172.17.8.104

    View Slide

  18. fleet: 3. Check the status

    View Slide

  19. fleet: 4. View the log

    View Slide

  20. Job Scheduling Flow
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  21. Job Scheduling Flow

    View Slide

  22. Job Scheduling Flow
    • fleet engine makes scheduling decisions
    • fleet agent executes units

    View Slide

  23. Architecture
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  24. fleet daemon
    • Every machine in the fleet cluster runs a single fleetd daemon
    • fleetd is a binary
    Cluster
    fleetd

    View Slide

  25. fleetd encapsulates 2 roles
    engine
    agent
    fleetd

    View Slide

  26. fleet engine
    • only one engine is running at a time
    fleetd
    => engine

    View Slide

  27. engine: lease model
    • a lease model to enforce that only one engine is running at a time
    • every time a reconciliation is due, an engine will attempt to take a
    lease on etcd
    engine

    View Slide

  28. fleet agent
    • a single agent runs on every machines
    fleetd
    => agent

    View Slide

  29. Reconciliation
    • reconcile the actual state with the desired state
    • reconciliation loop
    • triggered periodically
    • triggered by certain events from etcd
    current state desired state

    View Slide

  30. engine: reconcil
    • gathers a snapshot of the overall state of the cluster
    • then, attempts to reconcile
    engine

    View Slide

  31. agent: reconcil
    • actually executing Units on systems
    • reporting the state of units to etcd
    agent

    View Slide

  32. agent: reconcil
    • communicates with the local systemd instance over D-Bus
    CoreOS
    systemd
    fleetd

    View Slide

  33. engine: algorithm
    • ”least-loaded" scheduling algorithm
    • no resource management for simplicity
    engine

    View Slide

  34. etcd as registry

    View Slide

  35. etcd as registry
    • the sole datastore in a fleet cluster
    • internal communication between
    engines and agents

    View Slide

  36. etcd as registry

    View Slide

  37. etcd as registry

    View Slide

  38. Unit States
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  39. Cluster-level Unit State
    • inactive: known by fleet, but not assigned to a machine
    • loaded: assigned to a machine and loaded into systemd there, but not
    started
    • launched: loaded into systemd, and fleet has called the equivalent of
    systemctl start
    inactive loaded launched
    (unknown)

    View Slide

  40. Cluster-level Unit State
    • DSTATE: desired state
    • STATE: current state

    View Slide

  41. systemd-level Unit State
    • ACTIVE: the high-level unit activation state, i.e. generalization of SUB
    • SUB: the low-level unit activation state, values depend on unit type

    View Slide

  42. Cluster-level Unit State

    View Slide

  43. Change the state via fleetctl

    View Slide

  44. Change the state via fleetctl
    inactive loaded launched
    $ fleetctl start
    inactive loaded
    (unknown)
    $ fleetctl destroy

    View Slide

  45. Change the state via fleetctl
    loaded launched
    $ fleetctl stop
    $ fleetctl stop
    $ fleetctl destroy
    invalid action
    inactive
    (unknown)
    inactive

    View Slide

  46. fleetctl
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  47. Listing machines and units
    • list-machines
    • list-unit-files
    • list-units

    View Slide

  48. fleetctl list-machines
    • Enumerate the current hosts in the cluster

    View Slide

  49. fleetctl list-unit-files
    • List the units that exist in the cluster
    • You can check “desire” and “current” state of the unit

    View Slide

  50. fleetctl list-units
    • List the current state of units in the cluster

    View Slide

  51. Scheduling a unit
    • submit, load, start
    • stop, unload, destroy

    View Slide

  52. fleetctl submit
    • Upload one or more units to the cluster without starting them
    • Desire state: inactive

    View Slide

  53. fleetctl load
    • Schedule one or more units in the cluster
    • First submitting them if necessary
    • Desire state: loaded

    View Slide

  54. fleetctl start
    • Instruct systemd to start one or more units in the cluster
    • First submitting them if necessary
    • Desire state: launched

    View Slide

  55. fleetctl stop
    • Instruct systemd to stop one or more units in the cluster
    • Desire state: loaded

    View Slide

  56. fleetctl unload
    • Unschedule one or more units in the cluster
    • Desire state: inactive

    View Slide

  57. fleetctl destroy
    • Destroy one or more units in the cluster
    • Desire state: (unknown)

    View Slide

  58. Printing info for a unit
    • cat
    • status
    • journal

    View Slide

  59. fleetctl cat
    • Output the contents of a submitted unit

    View Slide

  60. fleetctl status
    • Output the contents of a submitted unit

    View Slide

  61. fleetctl journal
    • Print the journal of a unit in the cluster to stdout

    View Slide

  62. Administration
    • ssh

    View Slide

  63. fleetctl ssh
    • Open interactive shell on a machine in the cluster

    View Slide

  64. Unit files
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  65. Unit File Name
    .
    or
    @.
    The unit name must be of the form:

    View Slide

  66. Unit File Name: Example
    .
    or
    @
    foo.service
    bar.service
    foo.socket
    bar.timer
    [email protected]
    [email protected]

    View Slide

  67. Unit File Name: Requirements
    • :
    • must not be an empty string
    • can only contain [a-zA-Z0-9:_.@-]+
    .
    or
    @

    View Slide

  68. Unit File Name: Requirements
    • :
    • can be empty
    • can only contain [a-zA-Z0-9:_.@-]*
    .
    or
    @

    View Slide

  69. Unit File Name: Requirements
    • : must be one of the following unit types:
    service socket
    device mount automount
    timer
    path
    .
    or
    @

    View Slide

  70. Unit File Sections
    • Unit
    • Service
    • X-Fleet
    [Unit]
    generic_unit_option_1
    generic_unit_option_2
    [Service]
    service_specific_option_1
    service_specific_option_2
    service_specific_option_3
    [X-Fleet]
    fleet_specific_option

    View Slide

  71. Unit Section
    • Used to define generic information about a unit
    • Options that are common for all unit-types are generally
    placed here
    [Unit]
    Description=foo service
    # Requirements
    Requires=bar.service
    # Dependency ordering
    After=bar.service

    View Slide

  72. Service Section
    • Used to set directives that are specific to service units
    • Most of the unit-types above have associated sections for unit-
    type-specific information
    • See man page:
    [Service]
    TimeoutStartSec=0
    # Pre-start and Start
    ExecStartPre=-/usr/bin/docker kill foo
    ExecStartPre=-/usr/bin/docker rm foo
    ExecStart=/usr/bin/docker run —name foo -p 80:80 username/foo
    # Stop
    ExecStop=/usr/bin/docker stop foo
    http://www.freedesktop.org/software/systemd/man/systemd.unit.html

    View Slide

  73. X-Fleet Section
    • Used to set scheduling requirements for the unit for use with fleet
    • Using this section, you can require that certain conditions be true in
    order for a unit to be scheduled on a host
    [X-Fleet]
    Conflicts=foo.*.service
    MachineMetadata=location=tokyo

    View Slide

  74. fleet-specific Options in X-Fleet

    View Slide

  75. Template Unit Files
    1. Create a unit file whose name matches the:
    2. Instantiate units by creating new units that match the
    instance pattern:
    @.
    @.

    View Slide

  76. Template Unit Files Example

    View Slide

  77. systemd Specifiers
    When evaluating the [X-Fleet] section, fleet supports a subset of systemd's
    specifiers to perform variable substitution.

    View Slide

  78. systemd Specifiers Example
    [X-Fleet]
    MachineOf=%p.socket

    View Slide

  79. Unit Scheduling
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  80. Global units
    registry
    • Global units are not scheduled through the engine
    • Non-global units are scheduled by the engine
    Non-global units engine
    Global / Non-global units

    View Slide

  81. Global / Non-global units
    • Non-global units are scheduled by the engine
    • Global units are not scheduled through the engine
    (A global unit is one with Global=true in its X-Fleet section, fleet agents still check the
    MachineMetadata option before starting them. Other options are ignored.)

    View Slide

  82. Control the engine's scheduling decision
    • Schedule unit to specific machine
    • Schedule unit to machine with specific metadata
    • Schedule unit next to another unit
    • Schedule unit away from other unit(s)

    View Slide

  83. Schedule unit to specific machine
    registry
    Non-global units
    engine 4588e8cf
    b76e2b8a
    dc023323
    [X-Fleet]
    MachineID=4588e8cf94244e4c88bfb0a7171efa72

    View Slide

  84. Schedule unit to specific machine

    View Slide

  85. Schedule unit to machine with specific metadata
    registry
    Non-global units
    engine
    metadata=“region=us-west"
    [X-Fleet]
    MachineMetadata="region=us-west-1"

    View Slide

  86. Pull Request and Issue about Dynamic Metadata
    will be supported in fleet 0.10.0?

    View Slide

  87. Schedule unit next to another unit
    registry
    Non-global units
    engine
    foo.service
    [X-Fleet]
    MachineOf=foo.service

    View Slide

  88. Schedule unit away from other unit(s)
    registry
    Non-global units
    engine
    foo.service
    [X-Fleet]
    Conflicts=foo.service
    Conflicts!

    View Slide

  89. Schedule unit away from other unit(s)
    registry
    Non-global units
    engine
    [email protected]
    [X-Fleet]
    Conflicts=hello@*.service
    Conflicts!
    [email protected]
    Conflicts!
    The value of the Conflicts option is a glob pattern defining

    View Slide

  90. Configurations
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  91. General Options
    • verbosity
    • etcd_servers
    • etcd_request_timeout
    • etcd_cafile, etcd_keyfile, etcd_certfile
    • public_ip
    • metadata
    • agent_ttl
    • engine_reconcil_interval
    See official document for each option’s detail:
    https://github.com/coreos/fleet/blob/master/Documentation/deployment-and-configuration.md#general-options

    View Slide

  92. agent_ttl
    • An Agent will be considered dead if it exceeds this amount of time to
    communicate with the Registry
    • The agent will attempt a heartbeat at half of this value
    agent
    heartbeat
    agent is alive?
    I’m alive
    registry

    View Slide

  93. Configuration Sources
    • an INI-formatted config file
    • environment variables
    The fleetd daemon uses two sources for configuration parameters:

    View Slide

  94. INI-formatted Config File
    # Lower the logging threshold. Acceptable values are 0, 1, and 2. A higher
    # value corresponds to a lower logging threshold.
    verbosity=0
    # An Agent will be considered dead if it exceeds this amount of time to
    # communicate with the Registry. The agent will attempt a heartbeat at half
    # of this value.
    agent_ttl="36s"
    • fleet will look at for this config file by default
    • The flag may be passed the location to the fleetd binary
    /etc/fleet/fleet.conf
    —config

    View Slide

  95. Environment Variables
    # Enable debug logging by setting “verbosity” option to 1
    $ FLEET_VERBOSITY=1 /usr/bin/fleetd
    # Increase “agent_ttl” to default 36s from 30s (default)
    $ FLEET_AGENT_TTL=36s /usr/bin/fleetd
    • Options provided in an environment variable will override the corresponding
    option provided in a config file
    • simply prefix the name of a given option with 'FLEET_', while uppercasing the
    rest of the name

    View Slide

  96. Configure via cloud-config
    #cloud-config
    coreos:
    fleet:
    verbosity: 1
    agent-ttl: 36s
    public-ip: $public_ipv4
    metadata: region=us-west
    • The coreos.fleet.* parameters allow for the configuration of fleet
    through environment variables.

    View Slide

  97. fleet API v1
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  98. Entities
    • Units
    • Unit systems-level States
    • Machines
    The fleet API allows you to manage the state of the cluster using JSON over HTTP
    /units
    /state
    /machines
    https://github.com/coreos/fleet/blob/master/Documentation/api-v1.md

    View Slide

  99. Unit Entity
    • This simply declares what should be happening; the backend system
    still has to react to the changes in this desired state (cluster-level)
    fleetd
    desire state
    (cluster-level state)
    current state
    (cluster-level state)
    /units /units

    View Slide

  100. Unit Entity
    • If you would like to get the systemd-level state, communicate with
    UnitState entities
    desire state
    (cluster-level state)
    current state
    (system-level state)
    /units /state
    fleetd

    View Slide

  101. Other things
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide

  102. Releases
    WIP Feature
    Patches
    0.9.2 master
    0.9.1
    IUUQTHJUIVCDPNDPSFPTqFFUSFMFBTFT
    • New Features
    • Changes
    • Bugs Fixed

    View Slide

  103. end :)
    IUUQVTOBWZNVTFVNPSHJNBHFT4MJEFT&Y4VSGBDF'PSDF4VSGBDF@'MFFUKQH

    View Slide