Boost your JQ IQ [JSON Parsing]

Presented at All Day DevOps (ADDO) 2020

Shows tips and tricks to using JQ effectively for automation and scripting.

Transcript

1. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   NOVEMBER 12, 2020
   Rob Hirschfeld, RackN
   Boost your JQ IQ
   

   View Slide

2. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   Why JQ?
   Not because it’s elegant, fun or easy. JK!
   While APIs are JSON native, people aren’t.
   ● Pretty Print
   ● Validate JSON
   ● Parse JSON
   ● Search JSON
   ● Build JSON
   

   View Slide

3. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   A LOT!
   It’s so critical, we embedded it into our golang
   CLI so we didn’t have to install it everywhere.
   If you cp & suffix “drpcli” with “jq”
   then it becomes a JQ parser!
   How much does RackN JQ?
   

   View Slide

4. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   Distributed Infrastructure as Code with
   API-driven Provisioning
   All our API models and events are JSON
   All our states are JSON
   All our IaC templates are JSON
   (ok, we source them in YAML because readability)
   What does RackN do?
   

   View Slide

5. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   The Basics: How does it work?
   

   View Slide

6. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   Two Reference JSON files
   one.json
   machine object
   {
   “Name”: “x”,
   “Uuid”: “123-XYZ”,
   “Params: {
   “foo”: “bar”
   }
   }
   all.json
   list of machine objects
   [
   { “Name”: “test1” … },
   { “Name”: “test2” … },
   { “Name”: “test3” … }
   ]
   

   View Slide

7. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   cat one.json | jq .
   cat one.json | jq .Name
   cat all.json | jq .[].Name
   cat all.json | jq .[0].Name
   cat all.json | jq .[2:4]
   JSON dot navigation
   

   View Slide

8. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   cat one.json | jq .Params.foo
   cat one.json | jq .Params.gohai-inventory
   nope!… be careful
   cat one.json | jq .Params.[gohai-inventory]
   nope!… be more careful
   cat one.json | jq ‘.Params.[“gohai-inventory”]’
   cat one.json | jq .Params.\“gohai-inventory\”
   JSON dot navigation
   

   View Slide

9. TRACK: SITE RELIABILITY ENGINEERING
   SPEAKER: Rob Hirschfeld (@zehicle)
   JSON math
   cat all.json | jq length
   cat all.json | jq keys
   cat one.json | jq keys
   cat all.json | jq .[0]
   cat all.json | jq first
   cat all.json | jq last.Name
   

   View Slide

10. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    JSON pipes
    cat one.json | jq “.Params | length”
    cat one.json | jq “.Params | keys”
    cat one.json | jq “.Params | keys | length”
    cat one.json | jq “.Params[‘gohai-inventory’] | keys”
    nope… yikes quoting!
    cat one.json | jq '.Params["gohai-inventory"] | keys'
    cat one.json | jq “.Params[\"gohai-inventory\"] | keys”
    

    View Slide

11. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Why do I care about quotes?
    export gohai=$(cat one.json | jq -r ‘.Params | keys | last’)
    echo $gohai
    cat one.json | jq '.Params["$gohai"]'
    nope!
    cat one.json | jq “.Params[\"$gohai\"] | keys”
    yes!
    

    View Slide

12. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    JSON finding
    cat all.json | jq ‘.[] | select(.Name==”test1”)’
    Can use lots of logic! Like has, in, >, etc
    but, be careful of unexpected output!
    cat all.json | \
    jq '.[] | select(.Name | startswith("test"))'
    

    View Slide

13. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Building JSON maps (arrays)
    this is set operations…. be careful
    cat all.json | \
    jq 'map(.Name | startswith("test"))'
    not what you think!
    cat all.json | \
    jq 'map(select(.Name | startswith("test")))'
    

    View Slide

14. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    JSON flags
    cat one.json | jq .Endpoint
    fix that with -n
    cat one.json | jq -n .Endpoint
    cat one.json | jq -r .Name
    handy for bash loops:
    cat all.json | jq -r '.[].Uuid'
    

    View Slide

15. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    RackN Tips & Tricks
    

    View Slide

16. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Bash Loop Examples
    all=$(cat all.json | jq -r '.[].Uuid')
    for uuid in $all; do
    echo “ID: $uuid”
    cat all.json \
    | jq ".[] | select (.Uuid == \"$uuid\") | .Name"
    done
    

    View Slide

17. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Bleh, Quoting! Help!
    --arg can be handy to remove \” quotes
    export name=”test1”
    cat all.json | \
    jq --arg n "$name" \
    ‘.[] | select(.Name == $n) | .Uuid’
    

    View Slide

18. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Pipe from variables
    export all=$(cat all.json)
    jq .[].Name <<< "$all"
    We mix in Golang Templates
    export params=”{{ .Machine.Params }}”
    jq -r .[“gohai-inventory”] <<< "$params"
    

    View Slide

19. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Building JSON
    (better than by hand!!)
    export base='{"foo": "bar"}'
    adds
    jq ‘. += {“rob”:”zehicle”}’ <<< $base
    removes
    jq 'del(."foo")' <<< "$base"
    

    View Slide

20. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    ...And Don’t Go Crazy
    

    View Slide

21. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    ...And Don’t Go Crazy
    In many cases,
    THE BEST JQ IS NO JQ
    

    View Slide

22. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    Thank you!
    Resources:
    Online Dev Tooling: https://jqplay.org
    Reference: https://stedolan.github.com/jq
    This is stable stuff!
    Last major release was 2018
    

    View Slide

23. TRACK: SITE RELIABILITY ENGINEERING
    SPEAKER: Rob Hirschfeld (@zehicle)
    THANK YOU TO OUR SPONSORS
    

    View Slide