Fine-tuning DevOps for People over Perfection

Demand for DevOps has increased in recent years as more organizations adopt cloud native technologies. Complexity has also increased and a "zero to hero" mentality leaves many people chasing perfection and FOMO.

This session focusses instead on why maybe we shouldn't adopt a technology practice and how sometimes teams can achieve the same results by **prioritizing people over ops automation & controls**. Let's look at amounts of and fine-tuning everything as code, pull requests, DevSecOps, Monitoring and more to prioritize developer well-being over optimization perfection. It can be a valid decision to deploy less and sleep better. And finally we'll examine how manual practice and discipline can be the key to superb products and experiences.

DevOps.js Conference
24 March 2022

Transcript

1. Fine-tuning DevOps for
   People over Perfection
   Julie Ng
   Engineer, Microsoft
   DevOps.js Conference
   24 March 2022
   

   View Slide

2. Disclaimer
   All opinions my own, not of my employers past and present.
   Examples not exhaustive or comprehensive. Chosen for
   human value.
   

   View Slide

3. About Me
   👩‍💻
   Making for the web (full-stack) since 1999
   🚀 Failed Entrepreneuer
   💶 Former Enterprise Architect (Allianz
   Germany)
   ☁️
   Engineer, FastTrack for Azure (Microsoft)
   

   View Slide

4. DevOps is a journey
   

   View Slide

5. Pull Requests
   Best Practice…
   

   View Slide

6. Pull Request Workflow
   git checkout main && git merge feature
   git push
   Open a Pull Request
   #signoff
   #signoff
   #signoff
   Give Up 🤷‍♀️
   & (don't) Close Pull Request
   

   View Slide

7. Slow Pull Requests
   git push
   waiting for build agent…30 sec
   waiting for build agent…30 sec
   opens Twitter
   60 minutes later…
   Build Failed. No deployment, no value add 🤷‍♀️
   

   View Slide

8. View Slide

9. View Slide

10. Stuck in a Loop?
    

    View Slide

11. Lower Velocity, Less Business Value
    

    View Slide

12. How to Deploy Everyday
    

    View Slide

13. Team Decides
    Team choosing to deploy outside of business hours?
    

    View Slide

14. DevSecOps
    

    View Slide

15. in real life…
    No one in webinar commented on the security warnings
    🤔
    

    View Slide

16. High Alert
    🚨
    There's 3 Dependabots? alerts, security updates and version updates
    🤔
    

    View Slide

17. Ok, help me
    🤖
    

    View Slide

18. OK, let's try it
    "devDependencies": {
    
    "glob-parent": ">=5.1.2"
    
    }
    $ npm update
    

    View Slide

19. 🤨
    npm WARN …
    
    npm WARN …
    
    
    
    removed 27 packages, and audited 1742 packages in 51s
    
    
    
    213 packages are looking for funding
    
    run `npm fund` for details
    
    
    
    15 vulnerabilities (7 moderate, 8 high)
    
    
    
    To address all issues, run:
    
    npm audit fix
    
    
    
    Run `npm audit` for details.
    15 Vulnerabilities?
    😧
    Dependabot said 2 high vulnerabilities
    🙁
    

    View Slide

20. $ npm audit fix
    🤨 still 15?
    glob-parent <5.1.2
    
    Severity: high
    Regular expression denial of service - https://github.com/advisories/GH
    fix available via `npm audit fix`
    
    node_modules/watchpack-chokidar2/node_modules/glob-parent
    
    chokidar 1.0.0-rc1 - 2.1.8
    
    Depends on vulnerable versions of glob-parent
    
    node_modules/watchpack-chokidar2/node_modules/chokidar
    
    watchpack-chokidar2 *
    
    Depends on vulnerable versions of chokidar
    
    # …
    
    # …
    
    
    
    15 vulnerabilities (7 moderate, 8 high)
    
    
    
    To address all issues, run:
    
    npm audit fix
    

    View Slide

21. $ npm audit fix --force
    😫 still 15
    glob-parent <5.1.2
    
    Severity: high
    Regular expression denial of service - https://github.com/advisories/GH
    fix available via `npm audit fix`
    
    node_modules/watchpack-chokidar2/node_modules/glob-parent
    
    chokidar 1.0.0-rc1 - 2.1.8
    
    Depends on vulnerable versions of glob-parent
    
    node_modules/watchpack-chokidar2/node_modules/chokidar
    
    watchpack-chokidar2 *
    
    Depends on vulnerable versions of chokidar
    
    # …
    
    
    
    15 vulnerabilities (7 moderate, 8 high)
    
    
    
    To address all issues, run:
    
    npm audit fix
    

    View Slide

22. Now What?
    Tools optimize for knowing …
    Humans optimize for learning
    

    View Slide

23. npm overrides
    but requires update to npm v8, i.e. node v17
    {
    
    "overrides": {
    
    "glob-parent": "6.0.2",
    
    "nth-check": "2.0.1"
    
    }
    
    }
    

    View Slide

24. Let's upgrade node
    Update .nvmrc
    Update Dockerfile
    node-engine in package.json
    npm run preflight && git push
    ✅
    

    View Slide

25. But CI says
    ⛔️
    Upsupported Open SSL? No Thanks.
    ℹ Compiling Client
    
    node:internal/crypto/hash:67
    
    this[kHandle] = new _Hash(algorithm, xofLen);
    
    
    
    Error: error:0308010C:digital envelope routines::unsupported
    
    at new Hash (node:internal/crypto/hash:67:19)
    
    // …
    
    at FSReqCallback.readFileAfterClose [as oncomplete] (node:internal/
    opensslErrorStack: [ 'error:03000086:digital envelope routines::initi
    library: 'digital envelope routines',
    
    reason: 'unsupported',
    
    code: 'ERR_OSSL_EVP_UNSUPPORTED'
    
    }
    
    
    
    Node.js v17.7.1
    
    The command '/bin/sh -c npm ci --production && npm run nuxt:build' retu
    

    View Slide

26. Nevermind. Revert.
    

    View Slide

27. Sorry, gotta deploy
    but thanks for the alert
    😉
    # github workflow
    
    
    
    steps:
    - run: npm audit --audit-level=moderate
    
    continue-on-error: true
    

    View Slide

28. You too Defender Scan
    Thanks for the alert
    🤗
    steps:
    - uses: Azure/container-scan@v0
    
    continue-on-error: true
    
    with:
    
    image-name: ${{ env.image-name }}:dev-${{ inputs.git-sha }}
    
    severity-threshold: WARN
    
    

    View Slide

29. Microsoft Defender Results
    

    View Slide

30. Does it even matter?
    Builds running again
    Now, let's reflect on our decision to ignore a vulnerability
    What is `glob-parent` and how is it used?
    What is the Open SSL issue and how is that used?
    What is this app even trying to do?
    🤔
    

    View Slide

31. View Slide

32. DDoS via slowing it down.
    

    View Slide

33. What are we globbing for?
    Interactive (not a checklist) App
    

    View Slide

34. View Slide

35. Headless CMS
    ---
    
    title: Mult-tenant Cluster
    
    description: Cluster is used to host multiple workloads and/or multiple
    complexity: 100
    
    security: -25
    
    cost: 0
    
    operations: 100
    
    ---
    
    
    
    Multi-tenancy is incredibly complex. For example, Ingress, which manage
    

    View Slide

36. What we're Globbing
    .
    
    ├── index.md
    
    └── requirements
    
    ├── dr.md
    
    ├── factors
    
    │ ├── dr-active-active.md
    
    │ ├── dr-active-passive.md
    
    │ ├── dr-redeploy.md
    
    │ ├── dr-wait.md
    
    │ ├── multi-tenant.md
    
    │ └── single-tenant.md
    
    └── tenancy.md
    Missing Security Context
    App has no user input
    App has a build process
    

    View Slide

37. Conclusion?
    tooling is about knowing
    tools are stupid. can't think for themselves
    but they will not teaching you thinking, learning
    tooling does not give you context
    context is experience
    can't run a marathon without first running, 5k, 10k, half
    marathon
    too many notifications
    

    View Slide

38. When Security becomes a
    todo…
    

    View Slide

39. Craftsmanship & the Art of
    DevOps
    

    View Slide

40. What does your git history look like?
    update Readme.md
    update Readme.md
    update Readme.md
    update Readme.md
    update content
    update Readme.md
    wip
    update link
    

    View Slide

41. Why does it matter?
    Common challenge - versioning, esp. with "microservices"
    

    View Slide

42. ProTip - Conventional Commits
    git commit -m "feat(aad): add owners to application objects #49"
    git commit -m "feat(azuread-provider)s: upgrade to v2 #49"
    git commit -m "feat(devops): mask tenant and user principal ids"
    git commit -m "feat(pipeline): load kv based on branch in scheduled drift
    detection (#31)"
    

    View Slide

43. Acknowledge Others
    Extra 30 seconds goes a long way for team culture
    

    View Slide

44. Document for Humans
    30 minutes right now saves hours * 10 teammates in future
    

    View Slide

45. Time to Mastery?
    Years
    

    View Slide

46. Craftsmanship is an Investment in
    yourself and your teams
    

    View Slide

47. Thank you
    More Cloud Engineering in real life
    Twitter
    YouTube.com/c/JulieNgTech
    @jng5
    

    View Slide