Share
Embed
Google Engineering Productivity
Ankit Mehta
April 03, 2019
Transcript
29 slides
-
• Erodes User satisfaction • Diminishes the brand • Hard
to launch products • Hard to set a high bar MOVING FAST IS GOOD... • Innovate • Address flaws quickly • Better productivity • Better Code Health ...BUT BREAKING THINGS ISN’T -
Test Engineering -> Engineering Productivity Make engineers more productive through
tools, infrastructure, automation and analysis Release Test Submit Production/Stability Monitor Experiment Release Test Submit Review Author Design Backends + Pipelines Frontends Mobile ML Metrics/Insights v1.0 v2.0 -
EngProd by the Numbers 2000+ 2B+ 60k+ 18:1 EngProd Engineers
Lines of Code Code Submissions/Day Dev : EngProd Ratios -
EngProd Engagement Model • Teams within Teams - ~85% ◦
Deep partnership and collaboration with product teams ◦ EngProd teams (typically 3+) embedded within product teams • Horizontal Infra Teams - ~15% ◦ Dedicated teams focused on horizontal solutions ◦ Centralized within product-area EngProd organizations -
Don’t just catch Bugs, Prevent them! CODE Create a workspace
(Git equivalent: branch). BUILD/TEST Run blaze locally. The builds, unit tests, outputs and test results run remotely. CONTINUOUS INTEGRATION Remote execution service that runs hermetic builds/tests Local Development Workflow -
Guiding Principles Stable Debuggable Drive Test Health Avoid Performance Regressions
Ensure high Quality Releases Anchor with actionable metrics -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Hermetic Tests -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Hermetic Tests -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Attack Bad Tests Slow Tests Flaky Tests 11 mins 1 min -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Robosheri -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Performance Benchmarking Library -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics • Decouple feature release from binary release. • A release should be a train going out the door. • Rollback first and make the fix in a subsequent release. Predictable Releases -
• Overall score = minimum of 10 dimensions → focus
on worst • Assessed on 5pt scale • Assessed daily & automatically • Based on trailing 90d Product Health Drive Test Health Avoid Performance Regressions Ensure high Quality Releases Anchor with actionable metrics -
Metrics Drive Test Health Avoid Performance Regressions Ensure high Quality
Releases Anchor with actionable metrics -
Drive Test Health Avoid Performance Regressions Ensure high Quality Releases
Anchor with actionable metrics Priority Response SLO (initial & updates) Closure SLO P0 15-30 minutes 1 day (ASAP) to resolve or mitigate & downgrade P1 1 day 7 days P2 7 days (Initial Response); 90 days (updates) Best Effort P3 180 days Best Effort P4 365 days Best Effort Quality -
Billing EngProd : Core Areas Developer Velocity Data Accuracy Release
Velocity GCP Regions Service Integration Billing EngProd Pricing -
Billing EngProd Improvements • Automated Price Model changes with pre-submits
& daily auto-pushes • Hermetic integration tests (tests hrs>mins) • Created pre-production environment to generate invoices verification • Automated pre-release integration tests with Console and Payments stack -
Productivity in Open Source • CI for Kubernetes • Started
~2016. Fully replaced Jenkins - Jan 2018 • Test k8s on k8s! ◦ Easy to maintain/deploy ◦ Job config is basically a podspec ◦ Respond to github webhooks, rather than a munger -
Productivity in Open Source • Grid of test results •
Fast Access: Pre-computed aggregation of test results • Interactive: Sorting, filtering, grouping, filing bugs, finding changes • Highly Configurable • Integrated with Cloud Build TESTGRID -
Guiding Principles Stable Debuggable Drive Test Health Avoid Performance Regressions
Ensure high Quality Releases Anchor with actionable metrics