OPS213: GCP: Move Fast and Don’t Break Things Ankit Mehta, Engineering Productivity Director, Google Cloud Mithra Rajah, Software Engineer, Tools and Infrastructure, Google Cloud 

● Engineering Productivity (EngProd) makes quality software development easier and faster. ● We do this by adhering to our Guiding Principles of Driving Test Health, Avoiding Performance Regressions and Ensuring High Quality Releases. ● Anthos Engineering Productivity tools and how you can benefit from them. What are we going to talk about... 

● Innovate ● Address flaws quickly ● Better productivity ● Better Code Health ● User trust/satisfaction ● Uphold the brand ● Launch products ● Set a high bar Moving fast is good ● Innovate ● Address flaws quickly ● Better productivity ● Better Code Health But breaking things isn’t ● Erodes user safety satisfaction ● Diminishes the brand ● Hard to launch products ● Hard to set a high bar 

Balancing Velocity and Quality 

Make engineers more productive through tools, infrastructure, automation and analysis From Test Engineering to Engineering Productivity Version 1.0 Release Test Submit 

Make engineers more productive through tools, infrastructure, automation and analysis From Test Engineering to Engineering Productivity Version 2.0 Production/Stability Monitor Experiment Release Test Submit Review Author Design ML Mobile Frontends Backends + Pipelines Metrics/Insights 

Development at Google 800k+ Builds per day 2 billion+ lines of code 60,000+ code submissions per day 150M automated tests run per day 

Biweekly - twice a week Releases 

Prevent bugs - don’t just catch them Local Development Workflow Build/Test Run blaze locally. The builds, unit tests, outputs and test results run remotely. Continuous Integration Remote execution service that runs hermetic builds/tests Code Create a workspace (Git equivalent branch) 

Guiding Principles Debuggable Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Cloud Integration Testing Framework Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Performance Benchmarking Library Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Predictable Releases ● Release velocity: rate of pushing features from development to production ● Decouple feature release from binary release ● A release should be a train going out the door ● Rollback first, fix in a subsequent release Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Project Health ● Overall score = minimum of 10 dimensions -> focus ● Assessed on 5pt scale ● Assessed daily automatically ● Based on trailing 90d pH-Level-1 (configured) pH-Level-2 (improving) pH-Level-3 (acceptable) pH-Level-4 (commendable) pH-Level-5 (exemplary) tap-greeness configured 60% 74% 90% 95% tap-flakiness configured < 20% < 7% < 2% < 1% absolute- coverage ? configured 60% 75% 90% incremental- coverage ? configured 70% 80% 90% presubmit- coverage ? configured > 67% > 98% > 99% presubmit- ignored configured 20CLs 6CLs 3CLs 1CL presubmit- latency configured 35m 25m 15m 10m release- duration configured 3w 1w 3d 6h release- cherrypick- count configured 10CLs 5CLs 3CLs 1CL release- granularity configured 1000CLs 500CLs 100CLs 10CLs Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Metrics Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Guiding Principles Debuggable Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Changing Landscape Google’s software is in a single shared repository. Now the landscape everywhere is changing. The world now uses VMs, Kubernetes, Github, Scalable storage and so on. We need to evolve to match the landscape. 

Open Source Development 

● Continuous Integration for Kubernetes ● Started in ~2016. Fully replaced Jenkins - Jan 2018 ○ No longer need a Jenkins master ○ Update service daily without down time ● Test k8s on k8s! ○ Easy to maintain/deploy ○ Job config is basically a podspec ○ Respond to github webhooks, rather than a munger Prow Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Testgrid ● Grid of test results ● Fast Access: Pre-computed aggregation of test results ● Interactive: Sorting, filtering, grouping, filing bugs, finding changes ● Highly Configurable ● Integrated with cloud build Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

KIND Hermetic testing ● Local K8s cluster ● No external dependencies ● Integration testing K8s applications Kubernetes IN Docker Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

KIND Demo Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Mako ● Visualize performance data ● Automate identification of performance regressions ● Collaborate with developers around performance Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics http://mako.dev 

Jenkins X CI/CD using Prow and Knative ● Developer creates PR Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics ● Github sends PR notification to Prow ● Prow creates knative build ● Knative build runs jenkinsfile pipeline ● Prow uploads results to resultstore 

File master 1.12 1.11 api/ 49.8% 49.6% 48.1% apis/ 32.8% 32.6% 32.4% auth/ 9.8% 9.8% 9.8% capabilities/ 36.4% 36.4% 36.4% client/ 1.3% 1.3% 1.3% controller/ 30.2% 30.3% 28.1% credentialprovider/ 11.1% 11.1% 11.1% features/ 0.0% 0.0% 0.0% fieldpath/ 51.4% 51.4% 51.4% kubeapiserver/ 2.8% 2.8% 2.8% kubectl/ 0.0% 0.0% 0.0% kubelet/ 39.4% 39.0% 38.3% master/ 16.1% 16.1% 16.1% printers/ 2.1% 2.1% 2.1% probe/ 59.2% 59.2% 59.2% proxy/ 16.1% 16.1% 16.1% quota/ 6.7% 6.5% 6.9% registry/ 24.4% 24.2% 25.4% Conformance Tooling Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

Knative Conformance Coverage results display on TestGrid Drive Test Health Avoid Performance Regressions Ensure High Quality Releases Anchor with actionable metrics 

VALIDATING GCP Anthos - On to a Hybrid Cloud GitHub GitHub, Google3 Cloud Build Cloud Build GCP VSphere Lab VALIDATING ANTHOS Source Build/Test Verify Release 

Anthos Qualification ● Continuously and reliably validate Anthos binaries ● Enable teams to integrate easily ● Provide release readiness signals Development Continuous Integration Reporting Git Checkout Presubmit Conformance Tests Testgrid Periodic System Under Test Cluster Management Cluster Operators LogMon C O D E Tests Passed ? Submit Yes Presubmit No Test Results vSphere Lab Listen Create Dynamic Admin On Prem GKE Cluster Test Results 

● Engineering Productivity (EngProd) makes quality software development easier and faster. ● We do this by adhering to our Guiding Principles of Driving Test Health, Avoiding Performance Regressions and Ensuring High Quality Releases. ● Anthos Engineering Productivity tools and how you can benefit from them. To Summarize 

That’s a wrap. 

Your Feedback is Greatly Appreciated! Complete the session survey in mobile app 1-5 star rating system Open field for comments Rate icon in status bar