A Glimpse of Ansible Internals

This is a talk I gave at a local Ansible meetup. It's about how Ansible orchestrate tasks during runtime.

Transcript

1. A GLIMPSE OF ANSIBLE INTERNALS KAI
2. ABOUT ME ▸ Kai Xia, Pythonista and DevOps ▸ Worked in a game company in China(NTES) for 7 years ▸ Working@ShineSolutions ▸ Handler: @xiaket {gmail,github,linkedin,etc}
3. BACKGROUND ▸ Personal interest: reading source code helps ▸ Work at NTES: make Ansible work with internal CM tool ▸ Hacker: Learn what I can do about it
4. AGENDA ▸ How ansible works(in general) ▸ Selected ‘How xxx works’
5. HOW ANSIBLE WORKS
6. from ansible.runner import Runner results = Runner( module_name=‘ping’, module_args=‘’, pattern=‘web*’, forks=10, ).run() Run Ansible 1.x using Python
7. URL: http://docs.ansible.com/ansible/latest/dev_guide/developing_api.html
8. WHAT’S CHANGED? ▸ Codebase reworked ▸ Smaller classes created to meet individual needs.(Runner has 45 init parameters, that file has 1500+ lines). ▸ New features introduced
9. THE CODEBASE: META ▸ Ansible: 2.3.2.0 Item Number of files Lines of code w/ modules 1535 424528 w/o modules 327 53792
10. TEXT THE CODEBASE: VILLAINS
11. TEXT THE CODEBASE: VILLAINS(MORE EQUAL ONES)
12. HOW ANSIBLE WORKS ▸ Generate a file ▸ scp it ▸ run it Terms and conditions apply
13. TEXT PHASE 1: GATHER INFORMATION ▸ Playbook, inventory, roles, become, vault, templates, command line arguments, etc. ▸ DataLoader: yaml, json and vault ▸ VariableManager: host/group/extra/magic vars, facts ▸ Inventory: Represent a static/dynamic inventory
14. TEXT PHASE 2: ORGANISE TODO LIST ▸ Play: Represent a playbook ▸ Block ▸ Task data = { 'name': “A Play", 'hosts': "localhost", 'gather_facts': "no", 'tasks': [ {"action": {"module": "shell", "args": "ls"}, "register": “ls_cmd”}, {"action": {"module": "debug", “args": {"msg": "{{ ls_cmd.stdout }}"}}}, ] } play = Play.load(data, variable_manager=vm, loader=ldr)
15. TEXT PHASE 3: DISPATCH & CONTROL ▸ TaskQueueManager: Director ▸ StrategyPlugin: Project manager ▸ PlayIterator: The overseer ▸ {Worker,Result}Process: The work force ▸ Task{Executor,Result}: the chainsaw used by the work force
16. TEXT PHASE 4: THOSE 7 WORDS ▸ TaskExecutor will: ▸ Get the connection plugin(put_file, fetch_file, exec_command) ▸ Get shell plugin ▸ Get the action plugin/modules ▸ Generate the file
17. TEXT THE WORKFLOW VariableManager DataLoader Inventory Play Block Task TaskQueueManager PlayIterator StrategyPlugin TaskExecutor ActionPlugin/Module ConnectionPlugin ShellPlugin
18. SELECTED TOPICS
19. TEXT SELECTED TOPICS ▸ The strategy plugin ▸ The raw module ▸ The pipelining & ssh performance
20. TEXT THE STRATEGY PLUGIN ▸ Linear: get next task, queue it for all hosts, wait till this task is done, move on. ▸ Free: task was sent to hosts as fast as possible, with some fairness tweak.
21. TEXT THE STRATEGY PLUGIN: COMPARED
22. TEXT THE FREE STRATEGY: FAIRNESS TWEAK 3 hosts, 2 tasks, fork = 2
23. TEXT THE STRATEGY PLUGIN: ONE MORE THING ▸ Inject arbitrary code before and after a task. class StrategyModule(Linear): def run(self, iterator, play_context): for block in iterator._blocks: for task in block.block: if task.get_name() == "your module name": hosts = self._inventory.get_hosts(iterator._play_.hosts) run_your_code_here(task, hosts, play_context) return Linear.run(self, iterator, play_context)
24. TEXT THE RAW MODULE ▸ An action plugin in disguise ▸ TRANSFERS_FILES = False ▸ self._low_level_execute_command(self._task.args.get('_raw_p arams'), executable=executable) ▸ rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
25. TEXT TRANSFER_FILES = FALSE? ▸ Utils: assert/debug/fail ▸ Helpers: add_host/group_by/include_vars/set_fact/ set_stats ▸ Raw stuff: raw/service/package/win_reboot ▸ Networking: telnet/wait_for_connection
26. TEXT SSH PERFORMANCE ▸ control master: retain ssh socket, feature of openssh ▸ pipelining: minimize len(command), reduce disk IO, feature of ansible
27. TEXT PIPELINING IN ACTION
28. THANKS!