Developing Exceptional Architecture Design with Open Source and DevOps

The transition from long-lived models used for on-premises architecture towards a world of public cloud, serverless, and on-demand resources requires a good, hard look at how we construct conceptual and logical functional designs. Fortunately, many exceptional methodologies and workflows already exist in the open source world that we can heavily lean upon. Couple this with the face melting velocity experienced by those who pattern their release flows after the pillar characteristics of a DevOps world to start seeing a blending of on-premises and cloudy dimensions. In this talk, you’ll hear about current trends in various ecosystem industries, where the state of our art is headed, DevOps success stories, and what you can do to get started today!

Transcript

1. Developing Exceptional Architecture Design
   with Open Source and DevOps
   

   View Slide

2. Chris Wahl
   Chief Technologist, Rubrik
   chris.wahl@rubrik.com
   chriswahl
   @ChrisWahl
   

   View Slide

3. Agenda
   • What The Heck is Going On?
   • Learning from Open Source
   • Practice what you Preach
   • So, What Does This Look Like?
   • Interesting Use Cases!
   • Parting Thoughts
   

   View Slide

4. What The Heck is Going On?
   A fancy version of “Start with Why”
   

   View Slide

5. The Counter-Industrial Revolution
   “At the cusp of the 20th and 21st centuries, intangible assets overtook
   tangible assets in the economy”
   • Computerized information
   • Innovative property
   • Economic competencies
   

   View Slide

6. The Counter-Industrial Revolution
   1. The optimal scale for digital assets is global.
   2. The home has re-emerged as a workplace for teleworkers.
   3. The gig economy has sprung up at the expense of wage-based
   employment.
   4. Many firms delay IPO, preferring to remain private because they
   regard venture capitalists as better positioned to understand the
   value of their intangible assets.
   

   View Slide

7. The Counter-Industrial Revolution
   Spiral Motion
   Linear building
   Batch time
   Physical I/O
   Circular evolution
   Real-time
   Data (& memes) I/O
   

   View Slide

8. With that said …
   

   View Slide

9. Link: https://twitter.com/kevinbehr/status/1098707964721094656
   

   View Slide

10. Link: https://twitter.com/ChrisWahl/status/1080986977485406208
    

    View Slide

11. Learning from Open Source
    A sprinkle of development, a dash of chaos, and a pinch of creativity
    

    View Slide

12. Open Source Projects are Hard
    It all stems from why, how, and what.
    • How do you get folks interested?
    • What carrots or sticks do you have available?
    • How many goats can you herd at once?
    • How do you scale?
    If you don’t believe in the mission, no one else will either.
    

    View Slide

13. That should sound a bit familiar
    

    View Slide

14. Overview – 2011 Project
    • Managing a few thousand pizza boxes
    • Working on a very small team
    • Leaving behind cheeky MOTDs for the lulz
    

    View Slide

15. Challenges
    • Team of two
    • Customers: 500+ developers around the world
    • Environment would scale in really wonky ways, daily
    

    View Slide

16. View Slide

17. Looked for Inspiration Elsewhere
    • Found the OpenStack project
    • Figured Nova and Cinder would be interesting
    projects
    • Knew enough Python to get messy
    • Largely just interested in large scale
    development
    • How does the other side operate?
    • What lessons could I adapt to ops?
    

    View Slide

18. Learnings
    • My “two pizza team” was kind of nice
    • The grass wasn’t any greener, just a different shade of sad
    • DVCS is dope!
    

    View Slide

19. Learnings
    Operations often gets in the way
    • Sometimes for good – security, performance, capacity planning
    • Sometimes for bad – lack of understanding, silos, NIMBY
    

    View Slide

20. Takeaways
    • We adopted Git as our DVCS
    • Peer reviews = death to CABs!
    

    View Slide

21. Takeaways
    Contacted the facilities team to get a seating chart
    • Found where the dev leaders sat
    • Went to lunch, talked about our challenges
    • Shared knowledge and roadmap to align efforts
    • Borrowed a dev to clean up my messy code + ServiceNow workflows
    I learned that communication is always* the problem
    

    View Slide

22. *but sometimes the problem is just DNS
    

    View Slide

23. Practice what you Preach
    A different approach to operating a team
    

    View Slide

24. Some call this DevOps
    ¯\_(ツ)_/¯
    

    View Slide

25. Link: https://twitter.com/ScribblingOn/status/1101548686344114176
    

    View Slide

26. Achievement Unlocked: Developer
    

    View Slide

27. Link: https://twitter.com/srockets/status/1097642296261074945
    

    View Slide

28. Technical Debt is a tradeoff for speed
    Eventually, you need to pay it down
    

    View Slide

29. Link: https://twitter.com/tactical_intel/status/1101717575388487680
    

    View Slide

30. Infra vendors
    

    View Slide

31. So, What Does This World Look
    Like?
    How the sausage is CI/CD pipelined
    

    View Slide

32. As It Applies to Rubrik
    • 100% distributed team, globally
    • Workflows, documentation, and pipeline-driven process drive most
    everything
    • Building “all the things” in two on-prem facilities and three clouds
    (Azure, AWS, GCP) in a variety of regions
    • Default = native tools > overlays
    • We have no dedicated “ops” people
    • The work is never done, but that’s OK
    

    View Slide

33. View Slide

34. We operate as engineers
    There are some folks who are pretty good at advocating for this:
    “Netflix’s engineering culture is predicated on Freedom &
    Responsibility, the idea that everyone (and every team) at Netflix is
    entrusted with a core responsibility and they are free to operate with
    freedom to satisfy their mission.”
    This is the premise behind contractual operations
    

    View Slide

35. My team’s skill requirements:
    Learn a language
    Learn Git
    Learn REST (and GraphQL)
    Act like an adult
    

    View Slide

36. Contractual Operations
    Any system we create / operate is too complex for any one person or
    team to know all of it.
    Take API contracts as an example:
    • Machine readable definition of an API interface (e.g. Swagger).
    • Definition of the surface area of the resources that are available.
    • Describe, communicate, and collaborate around APIs.
    • These are often extremely abstract.
    

    View Slide

37. Contractual Operations: Requirements
    • Clearly define your inputs and outputs
    • Document the heck out of everything
    • Let people know when things change
    • Be creative within that realm
    • Build for scale
    • Unless it works idempotently, it’s wrong
    • Assume you will spontaneously combust at any moment
    • Embrace Service Ownership
    

    View Slide

38. This works regardless of locality or topology
    

    View Slide

39. Production Readiness
    ❑Security
    ❑Logging
    ❑Monitoring
    ❑Alerting
    ❑Observability (ephemeral)
    ❑Enabling Features
    ❑Testing
    ❑SLOs
    ❑Costs
    ❑Performance
    ❑Deployment / Build
    ❑Enablement
    

    View Slide

40. View Slide

41. View Slide

42. Musings on Building for Scale
    • Consider everything an artifact that needs a unique ID
    • Use declarative languages, DSLs, or native tooling when possible
    • Suggest changes in DVCS (Git), not by hand or dead tree (paper)
    • Style guides are dope
    • No tests = stop, just stop!
    

    View Slide

43. View Slide

44. Things My Team Likes
    • Azure DevOps is fairly solid as a build / release tool
    • Jira works for project management, Kanban boards, and getting the
    attention of other engineers ☺
    • IBM RedHat CloudForms / ManageIQ (cloud management)
    • Terraform (config management; built on our Go SDK)
    • Ansible (config management; built on our Python SDK)
    

    View Slide

45. In Conclusion
    

    View Slide

46. Thoughts
    • Communicate your plans
    • Find pain points to solve and
    innovation to copy
    • Design out in the open
    • Don’t consume anything that
    doesn’t have awesome APIs
    • Test everything
    • Assume good intent
    • Only punish those who build by
    hand (mistakes happen)
    • Version control everything
    • Pipeline everything
    • Test everything (!!!)
    

    View Slide

47. Thank you, wonderful humans!
    chris.wahl@rubrik.com
    chriswahl
    @ChrisWahl
    

    View Slide