Jinja and data manipulation Dag Wieers @dagwieers LOADays 2018 – Antwerp

Who am I ? ● Born as Dag Wieërs in Flanders, Belgium – Freelance Linux consultant ● Doing Linux and Open Source since 1994 ● Working for various companies (IT, Finance, Telco, Gov) – IBM, HP, EMC, Cisco, Punch; Euroclear, AXA, BNP Paribas, ING, KBC – Proximus, Telenet; Belgian Federal Police, Belgian Constitutional Court ● Mostly as engineer/architect, but prefers hands-on too – The past 6 years this involved Ansible in various ways ● Started developing Ansible from the very start (early 2012) ● Wrote core functionality and basic modules: debug, fail, set_fact, mail, ... – Maintainer of: unarchive, xml, filetree, dense, hpilo, vmware_guest, Cisco ACI, IMC, … – Involved in: Core, Windows, VMware and Network/ACI communities

Jinja ? ● One of many Python template engines (most popular ?) ● Based on Django’s template engine ● Features include: – Sandboxed execution – Compiles to optimal python code just in time – Easy to debug – Configurable syntax ● Designed for Web-based templating in mind !

3 types of delimiters (defaults) ● Statements {% for item in items %} … {% endfor %} ● Expressions {{ items|first }} ● Comments {# If only I had more time, these examples would rock #} ● Line statements (not worth mentioning)

Jinja and Ansible ● Ansible builds on existing technology – SSH, YAML/JSON and Jinja ● Ansible is not Web-related* ! – We actually care about data types, apart from strings – NativeEnvironment accepted upstream ! ● http://jinja.pocoo.org/docs/2.10/nativetypes/ ● https://github.com/ansible/ansible/pull/32738 ● YAML and Jinja – A subtle-layered love/hate relationship – Let’s introduce a YAML ‘jinja’ type for Jinja expressions ● https://github.com/ansible/proposals/issues/101

Jinja and Ansible... ● Used for various things – Templating files (in-line or using template module) – Decision-making (e.g. when: or until:) – Data-manipulation (modeling facts or inventory data)

How to make playbooks readable ● Move complexity out of playbooks 1. Start with well-designed inventory (possibly dynamic)** 2. Use templates to reduce playbook “spaghetti” 3. Logically group into self-suficient roles 4. Modify data and simplify expressions using Jinja2 filters** 5. Use (custom) lookup_plugins to iterate over collections 6. Push complex logic into (custom) modules (→ locality)

How to organize inventories and facts ● Think about how you will be managing the data – Who’s going to maintain it – Where is it coming from – How should it be structured ● Do not model inventory data for the task at hand ! ● Do not duplicate inventory data for convenience ! ● Use Jinja for manipulating data into structures you require for the job

Example #1: Past mistakes "ansible_date_time": { "date": "2018-04-21", "day": "21", "epoch": "1524299715", "hour": "10", "iso8601": "2018-04-21T08:35:15Z", "iso8601_basic": "20180421T103515915337", "iso8601_basic_short": "20180421T103515", "iso8601_micro": "2018-04-21T08:35:15.915448Z", "minute": "35", "month": "04", "second": "15", "time": "10:35:15", "tz": "CEST", "tz_ofset": "+0200", "weekday": "zaterdag", "weekday_number": "6", "weeknumber": "16", "year": "2018" }, hw_eth0: macaddress: 00:11:22:33:44:55 macaddress_dash: 00-11-22-33-44-55 "ansible_facts": { "ansible_all_ipv4_addresses": [ "192.168.122.1" ], … "ansible_default_ipv4": {}, … "ansible_enp0s25": { "active": false, "device": "enp0s25", "features": { "esp_hw_ofload": "of [fixed]", …

Example #2: Lists vs dictionaries ● Lists - name: eth0 ipaddress: 1.2.3.4 netmask: 255.255.255.0 gateway: 1.2.3.1 mac: 00:ca:fe:ba:be:01 - name: eth1 ipaddress: 2.3.4.5 netmask: 255.255.255.0 mac: 00:de:ad:be:ef:01 ● Dictionaries eth0: ipaddress: 1.2.3.4 netmask: 255.255.255.0 gateway: 1.2.3.1 mac: 00:ca:fe:ba:be:01 eth1: ipaddress: 2.3.4.5 netmask: 255.255.255.0 mac: 00:de:ad:be:ef:01

Data manipulation filters ● Sorting lists or dicts: sort ● Turning a dict into a list (and sort): dictsort ● Modify/search lists of dicts: map ● Filter lists of dicts: select / selectattr / reject / rejectattr ● Group dicts by attribute: groupby

Problems with selecting data ● Complex data-structures (nested dicts/lists) are beautiful – Until you start using them... – Iterating over nested dicts to selected data is hard – Raising unrecoverable errors on ● missing attributes ● empty lists ● anything unsupported really ● For the “aci-model” role we wrote aci_listify – But we plan to contribute something more generic

Examples 1) Only run when the target is listed on the output of the SCVVM server’s hyperv servers when: hostname is defined and hostname in hostvars[groups['winscvmm']|first].hyperv.stdout_lines 2) Get the IP addresses of all remaining cluster nodes (i.e. exclude the current target) candid: hosts: bdsol-aci51-candid-1: ansible_host: 10.48.16.161 bdsol-aci51-candid-2: ansible_host: 10.48.16.162 bdsol-aci51-candid-3: ansible_host: 10.48.16.163 cluster_addresses: "{{ hostvars|dictsort|selectattr('1.group_names', 'issuperset', ['candid'])| map(attribute='1.ansible_host')|reject('match', '^'~ansible_host~'$')|list }}"

Extending Jinja (in Ansible) is super easy ● Jinja filters (aka. filter plugins) when: this|success == true ● Jinja Tests (aka. test plugins) when: this is successful ● Very easy, just python functions ● Accepts arguments (first argument is piped in) ● Jinja filters return any data type ● Jinja tests return strictly booleans

Jinja and taste ● Types and tests: inconsistent – https://github.com/ansible/ansible/pull/37517 ● So 0 == false but 0 is not sameas false ● Also true == 1.0 and true is number ● And ’yes’ is sequence but also {} is sequence (iterable vs sequence ?) – Upstreamed: https://github.com/pallets/jinja/pull/824 ● Functionally: incomplete – Standard filters are a weird bunch (if you’re not a web developer) – Be careful with input types ● Syntactically: unlikable – Why on earth would we reuse the pipe-paradigm again ? – Terminology: Objects vs mappings vs dicts (Why ?) / Sequence vs list vs iterable – Templates with nested expressions on nested data, never happy indentation

Typical debugging problems ● Debugging Jinja statements (online) – http://jinja.quantprogramming.com/ ● Debugging playbooks – Use “debug” module and tags (like print statements) – Use “debug” callback plugin (more output) – Use “debug” strategy plugin ● Interactive debug session for testing Jinja in real time

ERROR! A worker was found in a dead state ● Out-of-memory due to missing brackets – Problem: memory: '{{ ansible_memtotal_mb*1024*1024|filesizeformat }}' – Solution: memory: '{{ (ansible_memtotal_mb*1024*1024)|filesizeformat }}' ● This can happen to *everyone* – Imagine you expect an integer, but you get a string instead – https://github.com/ansible/ansible/issues/35344

Tell me your stories (or talk to me afterwards)