Today’s menu 3 • What is Cookpad Global ? • Role of Site Reliability Engineers • Paving Roads for Autonomous Teams - Challenge 1: Organization Transformation for Greater Autonomy - Challenge 2: Feasible Self-service for Autonomous Teams
Users and Developers across the globe 12 • Global Service and SRE ? • Empower high perform technology organization Global head quarter UK, Bristol 12 100 People 25 Nationalities
Users and Developers across the globe 13 • Global Service and SRE ? • Empower high perform technology organization Global head quarter UK, Bristol 13 The best people join from all over the world
Missions for SREs in Cookpad 19 • Maximize user experiences in terms of: • Service availability • Performance • Security • etc… • Build a great platform to support a growing product • Product development optimized platform • Software architects owning comprehensive knowledge for technology Role of Site Reliability Engineers
Missions for SREs in Cookpad 20 • Maximize user experiences in terms of: • Service availability • Performance • Security • etc… • Build a great platform to support a growing product • Product development optimized platform • Software architects owning comprehensive knowledge for technology Role of Site Reliability Engineers Control service availability based on various factors
SRE technology scope in Cookpad 22 4FSWJDF 1MBUGPSNT 0CTFSWBCJMJUZ &OHJOFFSJOH .JTDJOIPVTF 5PPMJOH 3FMFBTF &OHJOFFSJOH 3FTJMJFODF &OHJOFFSJOH w %JTUSJCVUFE5SBDJOH w .FUSJDT.POJUPSJOH w -PHHJOH4ZTUFN w "MFSUT.BOBHFNFOU w .-#BTFE"OPNBMZ%FUFDUJPO w %BUB"OBMZTJT w 5FBN)FBMUI7JTVBMJ[BUJPO w "84$PTU0QUJNJ[BUJPO w %FWFMPQFS'SJFOEMZ"VUI4ZTUFN w FUD w %FQMPZ1JQFMJOF w $POUJOVPVT*OUFHSBUJPO w $POUJOVPVT%FMJWFSZ w %FQMPZ4USBUFHZ w /8'BVMU*OKFDUJPO w 4QPU*OTUBODF w $JSDVJU#SFBLFS w 5ISPUUMJOH w 7.$POUBJOFS1MBUGPSNPO"84 Role of Site Reliability Engineers
23 Challenges in 2018 Attacks from China GDPR Recipe data migration EKS based staging Recruitment in UK Observability Full containerization 23 Spot instances Expense reduction Toil analysis automation
Paving Roads for Autonomous Teams 25 • Challenge 1: Organization Transformation for Greater Autonomy • Challenge 2: Feasible Self-service for Autonomous Teams
Organization Transformation for Greater Autonomy 27 • Tipping Points for Autonomous Teams • Organization Transformation: Chapter and Squad • Development style change for new team structure • Necessity of shared responsibility for service availability Challenge1: Organization Transformation for Greater Autonomy
Tipping Points for Autonomous Teams 28 • Cookpad employees in UK • 2016: 5 people • 2017: 50 people • 2018: 100 people Challenge1: Organization Transformation for Greater Autonomy 8FC J04 "OESPJE 2" 43& 1. .- Team structure in 2016, 2017
Development style change for new team structure 35 • Architecture of new feed • New development styles Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
Architecture of new feed 36 Message broker Main API Cache Feed
API DB Complete feed json/html Cache DB GET /user_id/feed List of activity primary keys in order, paginated Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
API DB Complete feed json/html Cache DB New components developed by a squad GET /user_id/feed List of activity primary keys in order, paginated Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
New development styles (Partial release in production) 38 # WIP code for new notification system # https://github.com/cookpad/xxxxxx-squad/issues/yyyyyy Rollout.add :notification_center, owner: "xxxxxx-squad" do # @developer_a, @developer_b, @developer_c, @developer_d, @developer_e Current.user&.id&.in?([AAAAAA, BBBBBB, CCCCCC, DDDDDD, EEEEEE]) end • Feature toggle (application level control) • Prototype environment (platform level control) Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure New development styles (Partial release in production) 39 # WIP code for new notification system # https://github.com/cookpad/xxxxxx-squad/issues/yyyyyy Rollout.add :notification_center, owner: "xxxxxx-squad" do # @developer_a, @developer_b, @developer_c, @developer_d, @developer_e Current.user&.id&.in?([AAAAAA, BBBBBB, CCCCCC, DDDDDD, EEEEEE]) end • Feature toggle (application level control) • Prototype environment (platform level control) Only users know answers
Feed was successful feature? 40 • Yes, feed was one of the most successful features in 2018 • New architecture • New technology stack • 100% release in production in short time Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
41 Why feed was successful? • A lot of trials, failures and improvements in short term • Developers had power and responsibility for feature developments • Feed was developed from scratch • Developers could choose appropriate technology • Introduce Streamy, Karafka (stream app frameworks) • Test Kafka, RabbitMQ, SQS, Kinesis (message brokers) Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure
Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure 42 Why feed was successful? • A lot of trials, failures and improvements in short term • Developers had power and responsibility for feature developments • Feed was developed from scratch • Developers could choose appropriate technology • Introduce Streamy, Karafka (stream app frameworks) • Test Kafka, RabbitMQ, SQS, Kinesis (message brokers) Rapid prototyping was successful
Challenge1: Organization Transformation for Greater Autonomy > Development style change for new team structure 43 Why feed was successful? • A lot of trials, failures and improvements in short term • Developers had power and responsibility for feature developments • Feed was developed from scratch • Developers could choose appropriate technology • Introduce Streamy, Karafka (stream app frameworks) • Test Kafka, RabbitMQ, SQS, Kinesis (message brokers) On the other hand …
Necessity of shared responsibility for service availability 44 Challenge1: Organization Transformation for Greater Autonomy > Necessity of shared responsibility for service availability
Challenge1: Organization Transformation for Greater Autonomy > Necessity of shared responsibility for service availability Necessity of shared responsibility for service availability 45 Too many errors SREs cannot understand…
51 Why this situation happen? • A lot of trials, failures and improvements in short term • Developers had power and responsibility for feature developments • Feed was developed from scratch • Developers could choose appropriate technology • Introduce Streamy, Karafka (stream app frameworks) • Test Kafka, RabbitMQ, SQS, Kinesis (message brokers) • No concepts of shared responsibility for service availability Challenge1: Organization Transformation for Greater Autonomy > Necessity of shared responsibility for service availability
52 (WIP) Shared responsibility as Autonomous Teams • Shared responsibility for organization sustainability • Reach consensus of service availability for each feature • Targets decided by product owners • Higher quality in emergency notifications • Alert handling by appropriate people • Another organization transformation based on ideal tech & business architectures Challenge1: Organization Transformation for Greater Autonomy > Necessity of shared responsibility for service availability
53 • Shared responsibility for organization sustainability • Reach consensus of service availability for each feature • Targets decided by product owners • Higher quality in emergency notifications • Alert handling by appropriate people • Another organization transformation based on ideal tech & business architectures Challenge1: Organization Transformation for Greater Autonomy > Necessity of shared responsibility for service availability Inverse Conway Maneuver … (WIP) Shared responsibility as Autonomous Teams
Feasible Self-service for Autonomous Teams 55 • Four Important Keys for Successful Autonomous Teams • Feasible Self-service for Developers • Our focused scope • Full-containerization • No ssh debugging Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 56 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, Deploy pipeline Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 57 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, Deploy pipeline Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 58 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, Deploy pipeline Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 59 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, Deploy pipeline Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 60 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, deploy pipeline, feature toggle Challenge2: Feasible Self-service for Autonomous Teams
Four Important Keys for Successful Autonomous Teams 61 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, deploy pipeline, feature toggle Challenge2: Feasible Self-service for Autonomous Teams Organization strategy matter
Four Important Keys for Successful Autonomous Teams 62 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, deploy pipeline, feature toggle Challenge2: Feasible Self-service for Autonomous Teams Organization strategy matter Strong leaderships across tech and business are essential
Four Important Keys for Successful Autonomous Teams 63 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, deploy pipeline, feature toggle Challenge2: Feasible Self-service for Autonomous Teams SRE squad can contribute
Four Important Keys for Successful Autonomous Teams 64 • Discipline: Common rules in organization - Technology stack, team structure • Freedom: Ownership for individual developments - Small team, technology selection, system design • Responsibility: Commitments for whole software life cycle - Design, implementation, test, deploy, service availability monitoring • Optimization: Best practices for product developments - Logging and monitoring system, deploy pipeline, feature toggle Challenge2: Feasible Self-service for Autonomous Teams SRE squad can contribute Optimized self-service mechanisms providing company-wide best practices in SRE
Feasible Self-service for Developers 65 • Low learning cost • e.g: Are you sure that developers are happy to learn and maintain k8s yaml? • Secure and painless operations in production • e.g: Are experiences provided by SREs comfortable and secure for developers? Challenge2: Feasible Self-service for Autonomous Teams
Pros of Applications on Container Platform 68 • Developers can control software version upgrade timing • SREs don’t want to maintain legacy VM based service platform • Application of in-house tools and company-wide best practices • Auto Scaling • Cost optimization (spot fleets) • Container apps deployment tool (hako) • Centralized developer console (hako-console) • Easy service mesh integration • etc … • Immutable infrastructure • version controlled applications and infrastructures • No configuration drifts Challenge2: Feasible Self-service for Autonomous Teams > Full-containerization
Pros of Applications on Container Platform 69 • Developers can control software version upgrade timing • SREs don’t want to maintain legacy VM based service platform • Application of in-house tools and company-wide best practices • Auto Scaling • Cost optimization (spot fleets) • Container apps deployment tool (hako) • Centralized developer console (hako-console) • Easy service mesh integration • etc … • Immutable infrastructure • version controlled applications and infrastructures • No configuration drifts Challenge2: Feasible Self-service for Autonomous Teams > Full-containerization
Challenge2: Feasible Self-service for Autonomous Teams > Full-containerization Progress of Full-containerization in Global 80 17/18 apps are running on containers (94 % is completed)
Challenge2: Feasible Self-service for Autonomous Teams > Full-containerization 86 %FWFMPQFST`IBQQJOFTT 43&T`IBQQJOFTT 5PUBMIBQQJOFTT IBQQZ IBQQZ VOIBQQZ VOIBQQZ Happiness Quadrant (Software Upgrade with container) Plus, SREs can focus on container platform (more best practices can be introduced)
Cons of Applications on Container Platform 88 • Additional Complexities for Developers • Lack of tools cause chaos Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
Cons of Applications on Container Platform 89 • Additional Complexities for Developers • Lack of tools create chaos SFGIUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF TMJEF
Cons of Applications on Container Platform 92 • Additional Complexities for Developers • Lack of tools cause chaos • No ssh debugging systems for Global team • Granular and chronological order metrics dashboard • Container optimized New Relic agent deployment • Short-term log collection • Safe rails console for container Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
94 Granular and chronological metrics dashboard • Before • We cannot dig errors caused by spike resource saturations • After • We can recognize errors caused by spike resource saturations • We can judge that errors should be fixed soon or not Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
Container optimized New Relic agent deployment 96 • Before • ECS cannot deploy a New Relic agent to a specific container ( We want to save ) • Agents are gone when containers are killed accidentally • After • ECS can deploy a New Relic agent to a container • Distributed locking via `consul lock` sidecar • Rack middleware that provides an endpoint to start the New Relic agent • Agents are launched in a container when agent start flag exists on shared memory Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
Short-term log collection 98 • Before • Developers have to wait for few minutes to search logs • After • Developers can check logs nearly real-time Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
Safe rails console for container 100 • Before • Developers ssh to servers and run `rails -c` (Sometimes `rails -c -s`) • Developers can run write queries in production ( historical technical debt ) • After • Developers can use REPL via web browser with safe options selected by SREs • Developers can only run read queries on designated database instance Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging
Safe rails console for container (after) 103 Challenge2: Feasible Self-service for Autonomous Teams > No ssh debugging Feasible Self-service make product development reliable and autonomous !!
Recap 104 • What is Cookpad Global ? • Role of Site Reliability Engineers • Paving Roads for Autonomous Teams - Challenge 1: Organization Transformation for Greater Autonomy - Challenge 2: Feasible Self-service for Autonomous Teams
takanabe
muziyoshiz
oracle4engineer
sekido
knsg16
karamem0
sat
simosako
wakamatsu_takumu
nishiuma
coconala_engineer
antimon2
k1low
productmarketing
afnizarnur
jensimmons
shpigford
jonyablonski
jrom
keithpitt
swwweet
addyosmani
revolveconf
chriscoyier