[Velocity Conf 2017 NY] How Twitter built a framework to improve infrastructure utilization and efficiency at scale

Twitter is powered by thousands of microservices running on an internal cloud platform consisting of a suite of multitenant platform services that offer compute, storage, messaging, monitoring, etc. as a service. These platforms have thousands of tenants and run atop hundreds of thousands of servers both on-premises and in the public cloud. The scale of diversity in Twitter’s multitenant infrastructure services makes it extremely difficult to effectively forecast capacity, compute resource utilization, and cost and drive efficiency.

Vinu Charanya explains how she and her team are building a system that captures, defines, provisions, meters, and charges infrastructure resources, redefining how systems are built atop Twitter infrastructure. The infrastructure resources include primitive bare metal servers and VMs in the public cloud and abstract resources offered by multitenant services such as a compute platform (powered by Apache Aurora and Mesos), storage (Manhattan for key-value, cache, RDBMS), and observability. Along the way, Vinu shares how Twitter used this data to better plan capacity and drive a cultural change in engineering that helped improve overall resource utilization and led to significant savings in infrastructure spending.

Transcript

1. How built a framework to improve infrastructure resource utilization at scale
2. ★ Sr. Systems Engineer @Twitter ★ Proud Member of @TwitterWomen, @WomenWhoCode I am @VinuCharanya Hello!
3. 3 1 2 3 4 History & Context Chargeback @Twitter Kite - Service Lifecycle Manager Impact & Future Work Agenda
4. History & Context
5. Thousands of MicroServices
6. Thousands of MicroServices
7. Thousands of MicroServices
9. INFRASTRUCTURE & DATACENTER MANAGEMENT CORE APPLICATION SERVICES TWEETS USERS SOCIAL GRAPH PLATFORM SERVICES SEARCH MESSAGING & QUEUES CACHE MONITORING AND ALERTING INGRESS & PROXY 
 FRAMEWORK/ LIBRARIES FINAGLE (RPC) SCALDING (Map Reduce in Scala) HERON (Streaming Compute) JVM 
 MANAGEMENT TOOLS SELF SERVE SERVICE DIRECTORY CHARGEBACK CONFIG MGMT DATA & ANALYTICS PLATFORM INTERACTIVE QUERY DATA DISCOVERY WORKFLOW MANAGEMENT INFRASTRUCTURE SERVICES MANHATTAN BLOBSTORE GRAPHSTORE TIMESERIESDB S T O R A G E MESOS/AURORA HADOOP C O M P U T E MYSQL VERTICA POSTGRES D B / D W DEPLOY
 (Workflows)
10. MESOS/AURORA HADOOP MANHATTAN 67% Number of Servers
11. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How to get visibility into resources used by individual jobs & datasets?
12. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How to attribute resource consumption
 to teams/organization?
13. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How do you incentivize the right behavior to 
 improve efficiency of resource usage?
14. Chargeback @Twitter
15. Chargeback @Twitter Ability to meter allocation & utilization of resources
16. Chargeback @Twitter Ability to meter allocation & utilization of resources per service, per project, per engineering team
17. Chargeback @Twitter Ability to meter allocation & utilization of resources per service, per project, per engineering team to improve visibility & enable accountability
18. Features Supports diverse Infra Services Chargeback @Twitter 18 Meters abstract resources at daily granularity Detailed Reports
19. 19 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources Support diverse Infrastructure and Platform Services
20. 20 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources • Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource Support diverse Infrastructure and Platform Services
21. 21 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources • Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource 2. Resource <> Client Identifier Ownership: Map of client identifier to an owner to enable accountability Support diverse Infrastructure and Platform Services
22. OFFER MEASURE COST RESOURCE CATALOG ENTITY MODEL
23. OFFER MEASURES OFFER MEASURE COST 1:N RESOURCE CATALOG ENTITY MODEL
24. PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N RESOURCE CATALOG ENTITY MODEL
25. TWITTER DC/ PUBLIC CLOUD COMPUTE CORE-DAYS $X PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N RESOURCE CATALOG ENTITY MODEL
26. TWITTER DC/ PUBLIC CLOUD COMPUTE CORE-DAYS $X PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N TWITTER DC STORAGE GB- RAM PROCESSING CLUSTER FILE ACCESSES … … GB- RAM FILE ACCESSE S … … $X $Y … $M $N … … RESOURCE CATALOG ENTITY MODEL
27. { measures: [ { "measure_id": 1, "measure_label": "core-days", "measure_unit_label": "per 1 core-day", "offering_id": 1, "offering_label": "Compute", "infrastructure_id": 1, "infrastructure_name": "Aurora" }, { "measure_id": 2, "measure_label": "machine-days", "measure_unit_label": "per 1 machine-day", "offering_id": 2, "offering_label": "zone:aquila", "infrastructure_id": 8, "infrastructure_name": "Physical Infrastructure", }, { /api/1/measures Chargeback @Twitter
28. So, how do you incentivize the right behavior to 
 improve efficiency of resource usage?
29. Pricing is one way…
30. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server
 ($X / day) Total available Cores Quota Buffer
 (Underutilized Quota) Container Size Buffer
 (Underutilized Reservation) Total Cost of Ownership for Aurora $X core-day
31. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server
 ($X / day) Total available Cores Quota Buffer
 (Underutilized Quota) Container Size Buffer
 (Underutilized Reservation) Total used Cores Total Cost of Ownership for Aurora $X core-day
32. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server
 ($X / day) Total available Cores Quota Buffer
 (Underutilized Quota) Container Size Buffer
 (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation) Total Cost of Ownership for Aurora $X core-day
33. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server
 ($X / day) Total available Cores Quota Buffer
 (Underutilized Quota) Container Size Buffer
 (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation) Cores used by platform
 for operations & maintenance Total Cost of Ownership for Aurora $X core-day
34. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server
 ($X / day) Total available Cores Quota Buffer
 (Underutilized Quota) Container Size Buffer
 (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation) Cores used by platform
 for operations & maintenance Total Cost of Ownership for Aurora $X core-day
35. Our team would be …
36. Features Supports diverse Infra/Platform Services Chargeback @Twitter 36 Meters abstract resources at daily granularity Detailed Reports
37. 37 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT Metering Pipeline (ETL Job) IDENTIFIER OWNERSHIP MAPPING Metrics Ingestor DATA FIDELITY Metering Pipeline (ETL Job)
38. 38 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT Metering Pipeline (ETL Job) IDENTIFIER OWNERSHIP MAPPING Schema(client_identifier, offering_measure, volume, metadata, timestamp) DATA FIDELITY Metering Pipeline (ETL Job)
39. 39 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT Transformer DATA FIDELITY Metering Pipeline (ETL Job)
40. 40 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT 1. Resolve Ownership DATA FIDELITY Metering Pipeline (ETL Job)
41. 41 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT 2. Cost Computation DATA FIDELITY Metering Pipeline (ETL Job)
42. 42 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG DATA FIDELITY REPORT REPORT IDENTIFIER OWNERSHIP MAPPING Data Fidelity & Reporting Metering Pipeline (ETL Job)
43. 43 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT IDENTIFIER OWNERSHIP MAPPING 1. Verify Data Integrity & Fidelity DATA FIDELITY Metering Pipeline (ETL Job)
44. 44 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT IDENTIFIER OWNERSHIP MAPPING 2. Alert when things don’t seem the way it should be DATA FIDELITY Metering Pipeline (ETL Job)
45. 45 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 EXPORT METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP DATA FIDELITY REPORT REPORT Metering Pipeline (ETL Job)
46. Features Supports diverse Infra/Platform Services Chargeback @Twitter 46 Meters abstract resources at daily granularity Detailed Reports
47. 47 Chargeback @Twitter Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends Service Owners & Developers Team Bill Per Service Allocation vs. Utilization of Resources Reports Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends
48. INFRASTRUCTURE PNL
49. 49 Chargeback @Twitter Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends Service Owners & Developers Team Bill Per Service Allocation vs. Utilization of Resources Reports
50. CHARGEBACK BILL FOR A TEAM
51. CHARGEBACK DRILLDOWN FOR A TEAM
52. Features Supports diverse Infra/Platform Services Chargeback @Twitter 52 Meters abstract resources at daily granularity Detailed Reports
53. 53 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
54. 54 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
55. 55 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
56. 56 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
57. 57 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
59. SERVICE IDENTITY MANAGER RESOURCE PROVISIONING MANAGER DASHBOARD (SINGLE PANE OF GLASS) REPORTING INFRASTRUCTURE SERVICE INFRASTRUCTURE SERVICE INFRASTRUCTURE SERVICE INFRASTRUCTURE & PLATFORM SERVICE SERVICE LIFECYCLE WORKFLOWS METADATA RESOURCE QUOTA MANAGEMENT METERING & CHARGEBACK CLIENT IDENTITY PROVIDER APIS & ADAPTERS
60. 10,000+ Client Identifiers 1,000+ Projects 100+ Teams 8 Infrastructure Services 60 Kite @Twitter
61. 61 Kite @Twitter Identity System: Built a consistent way to group client identifiers of different infrastructure services into a project and enabled ownership • Capture Org Structure: Support org structure changes, project transfer workflows to ensure up-to-date ownership of identifiers • Unify client identifier provisioning workflow: Enables single source of truth and reduces operator pain around provisioning and managing client identifiers. Client Identifier Management
62. IDENTITY ENTITY MODEL <INFRA, CLIENTID> <Aurora, tweetypie.prod.tweetypie> <Aurora, ads- prediction.prod.campaign-x>
63. IDENTITY ENTITY MODEL SERVICE/
 SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N tweetypie <Aurora, tweetypie.prod.tweetypie> ads-prediction <Aurora, ads- prediction.prod.campaign-x>
64. BUSINESS OWNER TEAM PROJECT SERVICE/
 SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N 1:N 1:N 1:N INFRASTRUCTURE TWEETYPIE tweetypie tweetypie <Aurora, tweetypie.prod.tweetypie> ADS PREDICTION prediction ads-prediction <Aurora, ads- prediction.prod.campaign-x> REVENUE IDENTITY ENTITY MODEL
65. BUSINESS OWNER TEAM PROJECT SERVICE/
 SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N 1:N 1:N 1:N INFRASTRUCTURE TWEETYPIE tweetypie tweetypie <Aurora, tweetypie.prod.tweetypie> ADS PREDICTION prediction ads-prediction <Aurora, ads- prediction.prod.campaign-x> REVENUE IDENTITY ENTITY MODEL Entities are time varying dimensions
66. Impact
67. 10,000+ Client Identifiers
68. CLAIM OWNERSHIP
69. PROJECT DISCOVERY
70. PROJECT METADATA
71. AURORA QUOTA MANAGER
72. Future Work
73. 73 Future Work Impact & Future Work 1 2 Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
75. 75 1 2 Future Work Impact & Future Work Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
76. 76 1 2 Future Work Impact & Future Work Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
77. @VinuCharanya