Apps vs Infra, Where are My Pipelines? | DevOps.js Conference

Automation of a single monolithic app is pretty straight-forward. Split it into a frontend and backend and it's still manageable. Throw in more components or infrastructure and suddenly you're scratching your head at why a build ran - or didn't run. How many pipelines do I need? How many git repos should I have? Let's walkthrough use cases from small teams who own their entire stack to organizations with central IT units that manage shared infrastructure. Learn which scenarios and criteria determine how to slice but not spaghettify your pipelines.

DevOps.js Conference, 29-30 March 2021

Transcript

1. Apps vs Infra
   Where are my Pipelines?
   Julie Ng,
   Engineer, Microsoft
   DevOps.js Conference, March 2021
   

   View Slide

2. Hi, I’m Julie
   • Engineer, FastTrack for Azure
   • Previously Enterprise Architect
   Allianz Germany (2016-2019)
   • Full-Stack Engineer, UX Designer
   • Likes: Node.js, Ruby, Open Source
   • Dislikes: Windows 😜
   • Opinions are my own
   Engineering Bootcamp, Dublin, Ireland, February 2020
   

   View Slide

3. Monorepo – CI Workflow
   Once Upon a Time
   /app
   refs/head/main dev.app.com
   refs/head/production app.com
   

   View Slide

4. Monorepo – CD Workflow
   Once Upon a Time
   /app
   npm run e2e
   Build
   n+1
   refs/head/main
   Build
   n
   push deploy
   dev.app.com
   deploy
   app.com
   OK?
   No
   Fail
   git commit Yes
   push
   refs/head/production
   

   View Slide

5. Backends for Frontends (BFFs)
   Along came Mobile Devices and…
   /frontend
   refs/head/main
   refs/head/production
   /backend
   What do you want me
   to deploy?
   • Backend
   • Frontend
   • Both?
   

   View Slide

6. • Git Push
   • Branches
   • Paths
   • Pull Requests
   • Schedule
   • Web Hooks
   Events and Triggers
   

   View Slide

7. Monorepos & Triggers
   Pipeline As Code
   Example Snippet - Jenkins
   Example Snippet – Azure Pipelines
   

   View Slide

8. 🥴
   

   View Slide

9. Split ‘em up
   Pipeline as Code
   backend.dev.Jenkinsfile backend.prod.Jenkinsfile frontend.dev.Jenkinsfile frontend.prod.Jenkinsfile
   

   View Slide

10. What About Versioning?
    /frontend
    /backend
    v1.8.2
    Version?
    v1.1
    Distributed Monolith?
    

    View Slide

11. Microservices
    Truly independent
    calculator
    subtract
    multiply
    divide
    add
    • v1.0
    • v1.1
    • v1.2
    

    View Slide

12. Which End to End Tests?
    Independent, but still challenging
    multiply
    multiply.dev.app.com
    npm run e2e
    deploy
    calculator
    Which version of e2e?
    • /refs/head/production
    • /refs/tags/v1.3
    promote
    or not
    promote?
    

    View Slide

13. So how do we solve this?
    Talk to each other 😎
    

    View Slide

14. Ingress
    Calculator to Kubernetes
    Management: All the Cool Kids are Doing it
    Ingress
    calculator
    code
    images
    charts
    calculator.com
    calculator
    Infra as Code
    Pipeline as Code
    3 Repos X n Triggers
    So many triggers, events
    and dependencies
    😬
    

    View Slide

15. PROs
    • Total Control
    • More conducive to “Learning by
    Doing” without others relying on you
    • Easier overview of possible triggers
    • Easier to version
    • High Trust à Less Security
    configuration required.
    CONs
    • More skills required – App Dev,
    DevOps, Infrastructure, esp.
    networking
    • Tightly coupled app and infra
    Lifecycles
    Monorepo – App + Infra (one team)
    

    View Slide

16. Multi-tenant Kubernetes
    What if it’s many teams?
    images
    Ingress
    Ingress
    calcula
    tor
    calculator
    add
    subtract
    multiply
    divide
    add
    subtract
    multiply
    divide calc infra
    CI/CD
    pull
    min. 7 Repos X n Triggers
    MORE triggers, events and
    dependencies(?)
    🤯
    

    View Slide

17. CI/CD at Scale - Layers
    infra
    dev.app.com
    foo.app.com
    DNS TLS Certs
    dev.app.com
    foo.app.com
    Ingress
    namespace namespace
    calculator
    pull
    TLS Certs
    team.app.com
    pull?
    pull?
    aks
    app
    Layer 0 - Infra
    Layer 1 – K8s
    Layer 2 – Apps
    dns
    dns
    fork Pull Request
    Example - InnerSource
    Multiple managed clusters
    

    View Slide

18. InnerSource Code Examples
    Terraform
    https://github.com/julie-ng
    

    View Slide

19. All the Security Configs
    Don’t Forget End-to-End Governance
    https://github.com/azure/devops-governance
    

    View Slide

20. PROs
    • App Dev Teams – less infra experience
    required if everything pre-configured
    • Loose Coupling / Independent teams
    • InnerSource Possibilities
    • …you’re doing this because it’s
    required, right?
    CONs
    • Need very experienced central IT team
    • Many Security Vectors, the more
    granular…
    • The smaller the attack surface
    • The larger the management
    overhead
    DevOps at Scale (many teams)
    

    View Slide

21. Choreography
    CI/CD is
    Easier with small teams, but still a dance 😎
    

    View Slide

22. • Triggers grow exponentially.
    • Who owns the responsibility?
    E.g. uptime. Suppliers make the rules,
    not consumers.
    • What level of complexity are you most
    comfortable with?
    • E2E Tests are critical for automated
    promotion.
    • It’s OK to promote manually.
    • TALK TO EACH OTHER
    (within and across teams)
    • In Person
    • Chat
    • Video Meetings
    • Issues, Pull Requests, etc.
    Takeaways
    

    View Slide

23. Follow Me
    GitHub.com/julie-ng
    julie.io
    

    View Slide