1 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Steps toward self-service operations in eureka SRE NEXT 2022
 2022/05/14 © 2021 eureka, Inc. All Rights Reserved.

2 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Who is me © 2021 eureka, Inc. All Rights Reserved. wapper/nari ● Site Reliability Engineer at eureka, inc. ● Favorite: VR/Hip Hop/Skate Board/Sauna ● Twitter ○ Real: @fukubaka0825 ○ VR: @wapper0825

3 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. Eureka’s current situation

4 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. Products: 2 Regions: 3 Developers: 50+

5 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Old(〜2020) Our SRE Team Practice Overview © 2021 eureka, Inc. All Rights Reserved.

6 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting New Our SRE Team Practice Overview © 2021 eureka, Inc. All Rights Reserved.

7 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Today’s topic scope © 2021 eureka, Inc. All Rights Reserved. “Self-Serive” Operation Design

8 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting ● Good “Self-Service” Operations are ○ Low Cognitive Load ○ Low Operational Load for “Users” ○ Secure and Auditable 
 Conclusion © 2021 eureka, Inc. All Rights Reserved.

9 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. What/Why/How “Self-Service” Operations

10 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting What is “Self Service” Operations? © 2021 eureka, Inc. All Rights Reserved.

11 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Why “Self Service” Operations? © 2021 eureka, Inc. All Rights Reserved.

12 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting How to build “Self Service” Operations © 2021 eureka, Inc. All Rights Reserved. Cognitive Load⬇ Operational Load⬇ Secure⬆ Auditable ⬆

13 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. 3 “Self-Service” Operations Examples in eureka 1.Infrastructure as Code(Terraform) Operation 2.Batch Container Operation 3.Incident Response Operation

14 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. 3 “Self-Service” Operations Examples in eureka 1.Infrastructure as Code(Terraform) Operation 2.Batch Container Operation 3.Incident Response Operation 👈

15 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Overview © 2021 eureka, Inc. All Rights Reserved. ● Provide IaC platform that allows developers to develop and operate infrastructure with Software Development Life Cycle (with Terraform)

16 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Policy as Code with Conftest/Rego © 2021 eureka, Inc. All Rights Reserved. ● Automatic review of semantics problems that cannot be covered by existing static analysis tools without relying on certain human review by introducing Policy as Code Operational Load⬇

17 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting User-friendly CI Notification © 2021 eureka, Inc. All Rights Reserved. ● Notify users of the results of executing Terraform and conftest commands in CI in a form that is easy for them to understand what to change and how to change it ● https://github.com/suzuki-shunsuke/tfcmt ● https://github.com/suzuki-shunsuke/github-comment Cognitive Load⬇

18 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Terraform/AWS Workshop for Developers © 2021 eureka, Inc. All Rights Reserved. ● Held workshops to raise the knowledge level of Developers' Terraform and Cloud Infrastructure Cognitive Load⬇

19 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. 3 “Self-Service” Operations Examples in eureka 1.Infrastructure as Code(Terraform) Operation 2.Batch Container Operation 3.Incident Response Operation 👈

20 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Overview © 2021 eureka, Inc. All Rights Reserved. ● Provide batch container platform for developers with AWS Fargate + Amazon Eventbridge + AWS Lambda ○ to manage batch schedule and infra computing resources with SDLC by adding simple parameters with Terraform ○ to execute adhoc batch task by using GitHub Actions

21 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting ECS Fargate worker task auto scaler with AWS Lambda © 2021 eureka, Inc. All Rights Reserved. ● Autoscaling based on current Fargate tasks and SQS depth ○ Determine the number of tasks to execute based on the difference between the “Backlog (VisibleMsg Count)” and the “Appropriate-Backlog (currently running tasks x capacity per specified task)” ● Eliminates the need for detailed capacity planning Operational Load⬇

22 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Terraform module with few required parameters © 2021 eureka, Inc. All Rights Reserved. ● Developers can easily deploy a resource by simply adding a minimum list of variables and calling it with a module ● Developers can override CPU/Memory/Task Count and other parameters as needed Cognitive Load⬇

23 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Adhoc batch task runner with GitHub Actions Workflow Dispatch © 2021 eureka, Inc. All Rights Reserved. ● Validate if the user can execute the program by using the GitHub User ID (Team ID) at the first step of the job ● Easily track history of who did what Secure⬆ Auditable ⬆

24 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting © 2021 eureka, Inc. All Rights Reserved. 3 “Self-Service” Operations Examples in eureka 1.Infrastructure as Code(Terraform) Operation 2.Batch Container Operation 3.Incident Response Operation 👈

25 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Overview © 2021 eureka, Inc. All Rights Reserved. ● Provide Incident Response platform with ChatOps interface to reduce the burden of response to incidents, shorten MTTR as much as possible, and complete Postmortems process

26 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting ChatOps to issue Incident ticket/channel © 2021 eureka, Inc. All Rights Reserved. ● Integrate with Slack, which everyone is familiar with, and make it possible to report incidents with as simple commands and steps as possible Cognitive Load⬇

27 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Add Incident Response flow to General On-boarding Process © 2021 eureka, Inc. All Rights Reserved. ● Labor-saving and continuous recognition can be ensured by having the introduction of incident response flow incorporated in the onboarding process with BOT Cognitive Load⬇

28 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Postmortem Template © 2021 eureka, Inc. All Rights Reserved. ● Postmortems can be created from templates with one click of a button on Confluence Operational Load⬇

29 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting Future Prospects © 2021 eureka, Inc. All Rights Reserved. (Quoted from O’Reilly|Seeking SRE Chapter.4) Operational Load⬇ ● Introduction of “Timeline Model” to automate incident response flow more ● Measure time between “Response” and “Mitigate” and “Repair” and Analyse them to shorten MTTR

30 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting ● Good “Self-Service” Operations are ○ Low Cognitive Load ○ Low Operational Load for “Users” ○ Secure and Auditable 
 Conclusion © 2021 eureka, Inc. All Rights Reserved.

31 © 2021 eureka, Inc. All Rights Reserved. CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy All Hands Meeting ● Self-Service Operations ● eurekaにおけるここ一年のTerraform Component Delivery Processの変化 急成長 していくProduct基盤のProductivity,Security,Privacyとの向き合い ● Terraformのレビューを自動化するために、Conftestを導入してGitHub ActionsでCIま で設定してみる ● Scaling based on Amazon SQS ● Self-Serviceとサイロ化と組織構造 / Self-Service, Siloing and Organizational Structure ● SRE を実現するための組織マネジメント / Management to achieve SRE ● Seeking SRE ● インシデントレスポンスを自動化で支援する Slack Bot で人機一体なセキュリティ対 策を実現する Reference © 2021 eureka, Inc. All Rights Reserved.

32 © 2021 eureka, Inc. All Rights Reserved.