Planning for Huge Scale: Designing Your Pipelines for Scalability & Resiliency

Presented at DevOps Enterprise Summit 2016, November 7-9 in San Francisco.

Installing one CI server or configuring a deployment pipeline for a specific application might be easy enough. However, as enterprises look to scale their DevOps adoption and optimize their software delivery practices across the organization (to support additional teams, product lines, application releases, processes and infrastructure) -- software delivery pipeline(s) need to scale to support enterprise workloads.

For some enterprises, this means having a pipeline that can withstand the velocity and throughput of thousands of product releases, supporting tens of thousands of developers and distributed teams, hundreds of thousands of infrastructure nodes, multitudes of inter-dependent application components, or millions of builds and test-cases.

This scale poses unique challenges and implications for your pipeline design. This talk covers best practices for analyzing and (re)designing your software delivery pipeline – regardless of your chosen tool-set or technologies. Obtain tips and tools for ensuring your pipelines and DevOps infrastructure have the right architecture and feature-set to support your software production as it scales, while also ensuring manageability, governance, security, and compliance.

Learn best practices for how to:
1) Plan for scale: how to project for the types of performance indicators/vectors you’d need to scale across.
2) How to design of your pipeline and supporting infrastructure and operations (such as data retention, artifact retrieval, monitoring, etc.).
3) Design your pipeline workflows and processes to allow reusability and standardization across the organization, while also enabling flexibility to support the needs of specific teams/apps.
4) Design your pipeline in a way that enables fast rollout- easy onboarding thousands of applications, across hundreds of teams
5) Incorporate security access controls, approval gates and compliance checks as part of your pipeline and have them standard across all releases
6) Ensure your architecture support HA, DR and business continuity.

Transcript

1. electric-cloud.com #DOES16 Planning For Huge Scale: Designing Your Pipelines for Scalability and Resiliency Avan Mathur | Product Manager amathur@electric-cloud.com | @avantika_ec
2. electric-cloud.com #DOES16 What do I Mean by Huge Scale? • A Fortune 5 Bank • Deploying to 140K endpoints • 6000s of applications on-boarded at accelerated pace • Chip Manufacturer • 100K Jobs/Day • Constant load on the system • High job throughput
3. electric-cloud.com #DOES16 Plan Ahead – Know Your End Goal • Estimate and plan for long term scale • Define your success criteria • # of deployments per window • Max duration of a deployment • Number of concurrent deployments • Benchmarking • Validate numbers with benchmarks • Testing real workloads in real environments
4. electric-cloud.com #DOES16 Metrics to Measure • CI Cycle Times • Deployment Frequency and Duration • Concurrent Deployments • Release Frequency
5. electric-cloud.com #DOES16 Focus Areas • Infrastructure • Data • Pipeline Design • Monitoring / feedback loop
6. electric-cloud.com #DOES16 Infrastructure • High Availability • Horizontal Scalability • Distribute load and components • Database sizing and reliability • Understand and plan for bottlenecks and hardware limitations
7. electric-cloud.com #DOES16 Recommended Architecture Commander Server Commander Server Clustered Orchestration Servers Web Server Database Agent Agent Agent Agent Agent Workspace Repository 1 Browser Agent Agent Agent Agent Agent Web Server Site 2 WAN Workspace Repository 2 Browser Web Server Site 3 Repository 3 Site 1 Browser Agent Agent Agent Agent Agent Workspace HA/ HS
8. electric-cloud.com #DOES16 Disaster Recovery Architecture Commander Server Commander Server Clustered Orchestration Servers Web Server Database Agent Agent Agent Agent Agent Workspace Repository 1 DR Site HA/ HS Commander Server Commander Server Clustered Orchestration Servers Web Server Database Agent Agent Agent Agent Agent Workspace Repository 1 Primary Site HA/ HS Disk Replication WAN DB Mirroring
9. electric-cloud.com #DOES16 Managing Data • Each process generates data  Metadata, log files, build & test data, monitors, artifacts … • Understand growth rates and plan accordingly  Different areas to monitor o Database o Filesystem o Artifact Repository • Establish Data Retention & Archiving Policies early
10. electric-cloud.com #DOES16 Pipeline Design: Scaling Across an Organization 1. Plan and monitor for performance at scale at every step 2. Build a generic model to standardize across apps 3. Reusable components that can be used across projects 4. Automated onboarding of new projects 5. Plan and Define Access Controls and Roles early in the process
11. electric-cloud.com #DOES16 Monitoring • Monitor key metrics  Understand patterns  Foresee upcoming peaks in usage  Plan ahead for hardware expansion or distribution • Best Practices  Unobtrusive  Automated  Not only in production  One pane of glass
12. electric-cloud.com #DOES16 Monitoring
13. electric-cloud.com #DOES16 Questions?