Boost your JQ IQ [JSON Parsing]

Presented at All Day DevOps (ADDO) 2020

Shows tips and tricks to using JQ effectively for automation and scripting.

Transcript

1. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) NOVEMBER 12, 2020 Rob Hirschfeld, RackN Boost your JQ IQ
2. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why JQ? Not because it’s elegant, fun or easy. JK! While APIs are JSON native, people aren’t. • Pretty Print • Validate JSON • Parse JSON • Search JSON • Build JSON
3. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) A LOT! It’s so critical, we embedded it into our golang CLI so we didn’t have to install it everywhere. If you cp & suffix “drpcli” with “jq” then it becomes a JQ parser! How much does RackN JQ?
4. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Distributed Infrastructure as Code with API-driven Provisioning All our API models and events are JSON All our states are JSON All our IaC templates are JSON (ok, we source them in YAML because readability) What does RackN do?
5. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) The Basics: How does it work?
6. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Two Reference JSON files one.json machine object { “Name”: “x”, “Uuid”: “123-XYZ”, “Params: { “foo”: “bar” } } all.json list of machine objects [ { “Name”: “test1” … }, { “Name”: “test2” … }, { “Name”: “test3” … } ]
7. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json | jq . cat one.json | jq .Name cat all.json | jq .[].Name cat all.json | jq .[0].Name cat all.json | jq .[2:4] JSON dot navigation
8. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json | jq .Params.foo cat one.json | jq .Params.gohai-inventory nope!… be careful cat one.json | jq .Params.[gohai-inventory] nope!… be more careful cat one.json | jq ‘.Params.[“gohai-inventory”]’ cat one.json | jq .Params.\“gohai-inventory\” JSON dot navigation
9. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON math cat all.json | jq length cat all.json | jq keys cat one.json | jq keys cat all.json | jq .[0] cat all.json | jq first cat all.json | jq last.Name
10. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON pipes cat one.json | jq “.Params | length” cat one.json | jq “.Params | keys” cat one.json | jq “.Params | keys | length” cat one.json | jq “.Params[‘gohai-inventory’] | keys” nope… yikes quoting! cat one.json | jq '.Params["gohai-inventory"] | keys' cat one.json | jq “.Params[\"gohai-inventory\"] | keys”
11. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why do I care about quotes? export gohai=$(cat one.json | jq -r ‘.Params | keys | last’) echo $gohai cat one.json | jq '.Params["$gohai"]' nope! cat one.json | jq “.Params[\"$gohai\"] | keys” yes!
12. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON finding cat all.json | jq ‘.[] | select(.Name==”test1”)’ Can use lots of logic! Like has, in, >, etc but, be careful of unexpected output! cat all.json | \ jq '.[] | select(.Name | startswith("test"))'
13. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON maps (arrays) this is set operations…. be careful cat all.json | \ jq 'map(.Name | startswith("test"))' not what you think! cat all.json | \ jq 'map(select(.Name | startswith("test")))'
14. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON flags cat one.json | jq .Endpoint fix that with -n cat one.json | jq -n .Endpoint cat one.json | jq -r .Name handy for bash loops: cat all.json | jq -r '.[].Uuid'
15. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) RackN Tips & Tricks
16. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bash Loop Examples all=$(cat all.json | jq -r '.[].Uuid') for uuid in $all; do echo “ID: $uuid” cat all.json \ | jq ".[] | select (.Uuid == \"$uuid\") | .Name" done
17. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bleh, Quoting! Help! --arg can be handy to remove \” quotes export name=”test1” cat all.json | \ jq --arg n "$name" \ ‘.[] | select(.Name == $n) | .Uuid’
18. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Pipe from variables export all=$(cat all.json) jq .[].Name <<< "$all" We mix in Golang Templates export params=”{{ .Machine.Params }}” jq -r .[“gohai-inventory”] <<< "$params"
19. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON (better than by hand!!) export base='{"foo": "bar"}' adds jq ‘. += {“rob”:”zehicle”}’ <<< $base removes jq 'del(."foo")' <<< "$base"
20. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) ...And Don’t Go Crazy
21. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) ...And Don’t Go Crazy In many cases, THE BEST JQ IS NO JQ
22. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Thank you! Resources: Online Dev Tooling: https://jqplay.org Reference: https://stedolan.github.com/jq This is stable stuff! Last major release was 2018
23. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) THANK YOU TO OUR SPONSORS