Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Glimpse of Ansible Internals

Kai Xia
November 08, 2017

A Glimpse of Ansible Internals

This is a talk I gave at a local Ansible meetup. It's about how Ansible orchestrate tasks during runtime.

Kai Xia

November 08, 2017
Tweet

Other Decks in Programming

Transcript

  1. A GLIMPSE OF ANSIBLE
    INTERNALS
    KAI

    View full-size slide

  2. ABOUT ME
    ▸ Kai Xia, Pythonista and DevOps
    ▸ Worked in a game company in China(NTES) for 7 years
    ▸ Working@ShineSolutions
    ▸ Handler: @xiaket {gmail,github,linkedin,etc}

    View full-size slide

  3. BACKGROUND
    ▸ Personal interest: reading source code helps
    ▸ Work at NTES: make Ansible work with internal CM tool
    ▸ Hacker: Learn what I can do about it

    View full-size slide

  4. AGENDA
    ▸ How ansible works(in general)
    ▸ Selected ‘How xxx works’

    View full-size slide

  5. HOW ANSIBLE
    WORKS

    View full-size slide

  6. from ansible.runner import Runner
    results = Runner(
    module_name=‘ping’,
    module_args=‘’,
    pattern=‘web*’,
    forks=10,
    ).run()
    Run Ansible 1.x using Python

    View full-size slide

  7. URL: http://docs.ansible.com/ansible/latest/dev_guide/developing_api.html

    View full-size slide

  8. WHAT’S CHANGED?
    ▸ Codebase reworked
    ▸ Smaller classes created to meet individual needs.(Runner
    has 45 init parameters, that file has 1500+ lines).
    ▸ New features introduced

    View full-size slide

  9. THE CODEBASE: META
    ▸ Ansible: 2.3.2.0
    Item Number of files Lines of code
    w/ modules 1535 424528
    w/o modules 327 53792

    View full-size slide

  10. TEXT
    THE CODEBASE: VILLAINS

    View full-size slide

  11. TEXT
    THE CODEBASE: VILLAINS(MORE EQUAL ONES)

    View full-size slide

  12. HOW ANSIBLE WORKS
    ▸ Generate a file
    ▸ scp it
    ▸ run it
    Terms and conditions apply

    View full-size slide

  13. TEXT
    PHASE 1: GATHER INFORMATION
    ▸ Playbook, inventory, roles, become, vault, templates,
    command line arguments, etc.
    ▸ DataLoader: yaml, json and vault
    ▸ VariableManager: host/group/extra/magic vars, facts
    ▸ Inventory: Represent a static/dynamic inventory

    View full-size slide

  14. TEXT
    PHASE 2: ORGANISE TODO LIST
    ▸ Play: Represent a playbook
    ▸ Block
    ▸ Task
    data = {
    'name': “A Play", 'hosts': "localhost", 'gather_facts': "no",
    'tasks': [
    {"action": {"module": "shell", "args": "ls"}, "register": “ls_cmd”},
    {"action": {"module": "debug", “args": {"msg": "{{ ls_cmd.stdout }}"}}},
    ]
    }
    play = Play.load(data, variable_manager=vm, loader=ldr)

    View full-size slide

  15. TEXT
    PHASE 3: DISPATCH & CONTROL
    ▸ TaskQueueManager: Director
    ▸ StrategyPlugin: Project manager
    ▸ PlayIterator: The overseer
    ▸ {Worker,Result}Process: The work force
    ▸ Task{Executor,Result}: the chainsaw used by the work force

    View full-size slide

  16. TEXT
    PHASE 4: THOSE 7 WORDS
    ▸ TaskExecutor will:
    ▸ Get the connection plugin(put_file, fetch_file,
    exec_command)
    ▸ Get shell plugin
    ▸ Get the action plugin/modules
    ▸ Generate the file

    View full-size slide

  17. TEXT
    THE WORKFLOW
    VariableManager
    DataLoader
    Inventory
    Play
    Block
    Task
    TaskQueueManager
    PlayIterator
    StrategyPlugin
    TaskExecutor
    ActionPlugin/Module
    ConnectionPlugin
    ShellPlugin

    View full-size slide

  18. SELECTED TOPICS

    View full-size slide

  19. TEXT
    SELECTED TOPICS
    ▸ The strategy plugin
    ▸ The raw module
    ▸ The pipelining & ssh performance

    View full-size slide

  20. TEXT
    THE STRATEGY PLUGIN
    ▸ Linear: get next task, queue it for all hosts, wait till this task is
    done, move on.
    ▸ Free: task was sent to hosts as fast as possible, with some
    fairness tweak.

    View full-size slide

  21. TEXT
    THE STRATEGY PLUGIN: COMPARED

    View full-size slide

  22. TEXT
    THE FREE STRATEGY: FAIRNESS TWEAK
    3 hosts, 2 tasks, fork = 2

    View full-size slide

  23. TEXT
    THE STRATEGY PLUGIN: ONE MORE THING
    ▸ Inject arbitrary code before and after a task.
    class StrategyModule(Linear):
    def run(self, iterator, play_context):
    for block in iterator._blocks:
    for task in block.block:
    if task.get_name() == "your module name":
    hosts = self._inventory.get_hosts(iterator._play_.hosts)
    run_your_code_here(task, hosts, play_context)
    return Linear.run(self, iterator, play_context)

    View full-size slide

  24. TEXT
    THE RAW MODULE
    ▸ An action plugin in disguise
    ▸ TRANSFERS_FILES = False
    ▸ self._low_level_execute_command(self._task.args.get('_raw_p
    arams'), executable=executable)
    ▸ rc, stdout, stderr = self._connection.exec_command(cmd,
    in_data=in_data, sudoable=sudoable)

    View full-size slide

  25. TEXT
    TRANSFER_FILES = FALSE?
    ▸ Utils: assert/debug/fail
    ▸ Helpers: add_host/group_by/include_vars/set_fact/
    set_stats
    ▸ Raw stuff: raw/service/package/win_reboot
    ▸ Networking: telnet/wait_for_connection

    View full-size slide

  26. TEXT
    SSH PERFORMANCE
    ▸ control master: retain ssh socket, feature of openssh
    ▸ pipelining: minimize len(command), reduce disk IO,
    feature of ansible

    View full-size slide

  27. TEXT
    PIPELINING IN ACTION

    View full-size slide