Lessons from building an Internal Platform

In this talk, Martin Smith (Principal SRE, HashiCorp), and I look at what we learned from building an internal platform that helps developers, site reliability engineers, and governance teams to make more educated choices.

This version of the talk was given at DASH by Datadog in June 2024.

Transcript

1. Lessons from building an Internal . . . Platform
2. on-call CRITICAL VULN PAGES ALERT #423: Critical Exploit in Flarelang v1.5.1 and lower detected. now
3. on-call disrupts
4. on-call mission
5. on-call identify every on-call engineer of every service we run publicly mission
6. affected teams and codebases 100% Platform Team Mapping the impact
7. affected teams and codebases 95% Platform Team Mapping the impact
8. affected teams and codebases 63% 32% Platform Team Mapping the impact
9. find a way to reach the 63% mission
10. Principal Site Reliability Engineer, Internal Platform and Services he / him @martinb3 Martin Smith
11. find who is running the latest release of Nomad mission
12. Senior Developer Advocate, Infrastructure & Orchestration he / him @ksatirli Kerim Satirli
13. func main() { services := []string{"service-1", "service-2", "service-n"} for _, service := range services { } } // identify affected `services` internal_platform.go Collecting the data
14. affected teams and codebases Platform Team 63% 32% Mapping the impact
15. Collectors GitHub Incidents Datadog team health service health product health monitors dashboards errors / traces static collectors teams heuristics groupings Platform Team repo contents api data event driven Collecting the data
16. Rendering the data
17. Rendering the data
18. 02 Why we are building it
19. GitHub static Datadog Incidents monitors dashboards errors and traces team health service health product health teams codebases Slack channels repo metadata full source code deploy health Classifying the data
20. GitHub static Datadog Incidents logging best practices service attribution passing SLOs synthetic monitors team attribution escalation policy silenced incidents incident volume known teams Go versions SDK versions partitions code owners PR templates branch protection secret scanning Classifying the data
21. problems lots of data = lots of
22. lots of data = lots of use cases
23. "A collection of tools, services, and infrastructure components that help developers manage the entire application lifecycle." " "
24. Attribute Ensure Detect Standardize Are there errors? Has it been deployed recently? Has it been scanned for vulnerabilities? Is it on a recent version? Is it using our standard monitoring? Does it lose alerts? Does it have SLOs, and are they being met? Does it have synthetic monitoring? Are there wake-up monitors? What team owns this? Where is this in GitHub, Slack or PagerDuty? Where are the dashboards? Mapping the problems
25. How far behind is my service on Go or internal SDKs? How do I enable DR for my service? Engineering Does this service have SLOs and how is it tracking against those SLOs? Is the owner team on-call for this service? SREs Is code scanning enabled on this repository? Is this service subject to PCI controls? Governance Mapping the personas
26. open and opinionated everyone can submit data everyone can check data one-stop data vending data is ephemeral schemas as safeguards common helpers Mapping the guardrails
27. What developers need to know and have access to for their job and services. Engineering wants a Developer Platform What leaders need to be aware of related to customer experience and priorization. SRE wants a Reliability Platform What security and systems engineering teams need to understand how operators are supporting their services. Governance wants an Intelligence Platform Mapping the requirements
28. loose data team health service health product health monitors dashboard errors / traces teams heuristics groupings api data events repo contents structured data product and service teams scenarios and checks service stacks generic assets insights campaigns quality score custom checks . . . profit? better service health
29. Our definition of "service" is strongly tied to how we operate.
30. Data Integration Remediation Community Inability to tag Change over time Outreach Making truth consistency of data new tags or data types discovery of data Freshness Critical path Exporting Result Scoring freshness of data volume and scaling structure tradeoffs Problems we solve
31. 03 Looking forward
32. Automation Reliability Targeting Understanding What PCI-covered repos have a Dependabot warning? What are we doing internally when we run our products? Company-wide SRE use cases — workflows and tiering * Feeding data to AI applications could much it much easier to operate services Future direction
33. Are we able to handle unexpected situations within our SLO budget?
34. Does a more complete set of data equate to higher service quality?
35. Does the company care if a dependency is flaky across many services?
36. Platform(s) Internal Developer Platform What developers need to know and have access to for their job and services. Reliability Data Platform What leaders need to know to improve our customer experience and feature set. Service Intelligence Platform What Governance need to know to influence on how operators run their services.
37. clearly define how we run services at HashiCorp goal
38. Thank you speakerdeck.com/ksatirli