Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Glimpse of Ansible Internals

Kai Xia
November 08, 2017

A Glimpse of Ansible Internals

This is a talk I gave at a local Ansible meetup. It's about how Ansible orchestrate tasks during runtime.

Kai Xia

November 08, 2017
Tweet

Other Decks in Programming

Transcript

  1. ABOUT ME ▸ Kai Xia, Pythonista and DevOps ▸ Worked

    in a game company in China(NTES) for 7 years ▸ Working@ShineSolutions ▸ Handler: @xiaket {gmail,github,linkedin,etc}
  2. BACKGROUND ▸ Personal interest: reading source code helps ▸ Work

    at NTES: make Ansible work with internal CM tool ▸ Hacker: Learn what I can do about it
  3. WHAT’S CHANGED? ▸ Codebase reworked ▸ Smaller classes created to

    meet individual needs.(Runner has 45 init parameters, that file has 1500+ lines). ▸ New features introduced
  4. THE CODEBASE: META ▸ Ansible: 2.3.2.0 Item Number of files

    Lines of code w/ modules 1535 424528 w/o modules 327 53792
  5. HOW ANSIBLE WORKS ▸ Generate a file ▸ scp it

    ▸ run it Terms and conditions apply
  6. TEXT PHASE 1: GATHER INFORMATION ▸ Playbook, inventory, roles, become,

    vault, templates, command line arguments, etc. ▸ DataLoader: yaml, json and vault ▸ VariableManager: host/group/extra/magic vars, facts ▸ Inventory: Represent a static/dynamic inventory
  7. TEXT PHASE 2: ORGANISE TODO LIST ▸ Play: Represent a

    playbook ▸ Block ▸ Task data = { 'name': “A Play", 'hosts': "localhost", 'gather_facts': "no", 'tasks': [ {"action": {"module": "shell", "args": "ls"}, "register": “ls_cmd”}, {"action": {"module": "debug", “args": {"msg": "{{ ls_cmd.stdout }}"}}}, ] } play = Play.load(data, variable_manager=vm, loader=ldr)
  8. TEXT PHASE 3: DISPATCH & CONTROL ▸ TaskQueueManager: Director ▸

    StrategyPlugin: Project manager ▸ PlayIterator: The overseer ▸ {Worker,Result}Process: The work force ▸ Task{Executor,Result}: the chainsaw used by the work force
  9. TEXT PHASE 4: THOSE 7 WORDS ▸ TaskExecutor will: ▸

    Get the connection plugin(put_file, fetch_file, exec_command) ▸ Get shell plugin ▸ Get the action plugin/modules ▸ Generate the file
  10. TEXT THE WORKFLOW VariableManager DataLoader Inventory Play Block Task TaskQueueManager

    PlayIterator StrategyPlugin TaskExecutor ActionPlugin/Module ConnectionPlugin ShellPlugin
  11. TEXT SELECTED TOPICS ▸ The strategy plugin ▸ The raw

    module ▸ The pipelining & ssh performance
  12. TEXT THE STRATEGY PLUGIN ▸ Linear: get next task, queue

    it for all hosts, wait till this task is done, move on. ▸ Free: task was sent to hosts as fast as possible, with some fairness tweak.
  13. TEXT THE STRATEGY PLUGIN: ONE MORE THING ▸ Inject arbitrary

    code before and after a task. class StrategyModule(Linear): def run(self, iterator, play_context): for block in iterator._blocks: for task in block.block: if task.get_name() == "your module name": hosts = self._inventory.get_hosts(iterator._play_.hosts) run_your_code_here(task, hosts, play_context) return Linear.run(self, iterator, play_context)
  14. TEXT THE RAW MODULE ▸ An action plugin in disguise

    ▸ TRANSFERS_FILES = False ▸ self._low_level_execute_command(self._task.args.get('_raw_p arams'), executable=executable) ▸ rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  15. TEXT TRANSFER_FILES = FALSE? ▸ Utils: assert/debug/fail ▸ Helpers: add_host/group_by/include_vars/set_fact/

    set_stats ▸ Raw stuff: raw/service/package/win_reboot ▸ Networking: telnet/wait_for_connection
  16. TEXT SSH PERFORMANCE ▸ control master: retain ssh socket, feature

    of openssh ▸ pipelining: minimize len(command), reduce disk IO, feature of ansible