Google Engineering Productivity

Transcript

2. Engineering Productivity at Google Mithra Rajah Ankit Mehta
3. • Erodes User satisfaction • Diminishes the brand • Hard to launch products • Hard to set a high bar MOVING FAST IS GOOD... • Innovate • Address flaws quickly • Better productivity • Better Code Health ...BUT BREAKING THINGS ISN’T
4. Test Engineering -> Engineering Productivity Make engineers more productive through tools, infrastructure, automation and analysis Release Test Submit Production/Stability Monitor Experiment Release Test Submit Review Author Design Backends + Pipelines Frontends Mobile ML Metrics/Insights v1.0 v2.0
5. EngProd by the Numbers 2000+ 2B+ 60k+ 18:1 EngProd Engineers Lines of Code Code Submissions/Day Dev : EngProd Ratios
6. EngProd Engagement Model • Teams within Teams - ~85% ◦ Deep partnership and collaboration with product teams ◦ EngProd teams (typically 3+) embedded within product teams • Horizontal Infra Teams - ~15% ◦ Dedicated teams focused on horizontal solutions ◦ Centralized within product-area EngProd organizations
7. Balancing Velocity and Quality
8. GCP Console Ships twice a week
9. Don’t just catch Bugs, Prevent them! CODE Create a workspace (Git equivalent: branch). BUILD/TEST Run blaze locally. The builds, unit tests, outputs and test results run remotely. CONTINUOUS INTEGRATION Remote execution service that runs hermetic builds/tests Local Development Workflow
10. Guiding Principles Stable Debuggable Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics
11. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Hermetic Tests
12. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Hermetic Tests
13. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics
14. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Attack Bad Tests Slow Tests Flaky Tests 11 mins 1 min
15. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Robosheri
16. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Performance Benchmarking Library
17. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics • Decouple feature release from binary release. • A release should be a train going out the door. • Rollback first and make the fix in a subsequent release. Predictable Releases
18. • Overall score = minimum of 10 dimensions → focus on worst • Assessed on 5pt scale • Assessed daily & automatically • Based on trailing 90d Product Health Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics
19. Metrics Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics
20. Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics Priority Response SLO (initial & updates) Closure SLO P0 15-30 minutes 1 day (ASAP) to resolve or mitigate & downgrade P1 1 day 7 days P2 7 days (Initial Response); 90 days (updates) Best Effort P3 180 days Best Effort P4 365 days Best Effort Quality
21. Billing EngProd : Core Areas Developer Velocity Data Accuracy Release Velocity GCP Regions Service Integration Billing EngProd Pricing
22. ~10 billing accuracy issues per month
23. Billing EngProd Improvements • Automated Price Model changes with pre-submits & daily auto-pushes • Hermetic integration tests (tests hrs>mins) • Created pre-production environment to generate invoices verification • Automated pre-release integration tests with Console and Payments stack
24. Open Source Development
25. Productivity in Open Source • CI for Kubernetes • Started ~2016. Fully replaced Jenkins - Jan 2018 • Test k8s on k8s! ◦ Easy to maintain/deploy ◦ Job config is basically a podspec ◦ Respond to github webhooks, rather than a munger
26. Productivity in Open Source • Grid of test results • Fast Access: Pre-computed aggregation of test results • Interactive: Sorting, filtering, grouping, filing bugs, finding changes • Highly Configurable • Integrated with Cloud Build TESTGRID
27. Conformance Tooling and Infrastructure
28. Guiding Principles Stable Debuggable Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics