Steps toward self-service operations in eureka

1
© 2021 eureka, Inc. All Rights Reserved.
CONFIDENTIAL INFORMATION: Not for Public Distribution - Do Not Copy
All Hands Meeting
Steps toward self-service
operations in eureka
SRE NEXT 2022 
2022/05/14

View Slide

2
All Hands Meeting
Who is me
wapper/nari
● Site Reliability Engineer at eureka, inc.
● Favorite: VR/Hip Hop/Skate Board/Sauna
● Twitter
○ Real: @fukubaka0825
○ VR: @wapper0825

View Slide

3
All Hands Meeting
Eureka’s current situation

View Slide

4
All Hands Meeting
Products: 2
Regions: 3
Developers: 50+

View Slide

5
All Hands Meeting
Old(〜2020) Our SRE Team Practice Overview

View Slide

6
All Hands Meeting
New Our SRE Team Practice Overview

View Slide

7
All Hands Meeting
Today’s topic scope
“Self-Serive”
Operation
Design

View Slide

8
All Hands Meeting
● Good “Self-Service” Operations are
○ Low Cognitive Load
○ Low Operational Load for “Users”
○ Secure and Auditable
 
Conclusion

View Slide

9
All Hands Meeting
What/Why/How “Self-Service” Operations

View Slide

10
All Hands Meeting
What is “Self Service” Operations?

View Slide

11
All Hands Meeting
Why “Self Service” Operations?

View Slide

12
All Hands Meeting
How to build “Self Service” Operations
Cognitive Load⬇
Operational Load⬇
Secure⬆
Auditable
⬆

View Slide

13
All Hands Meeting
3 “Self-Service” Operations Examples in eureka
1.Infrastructure as Code(Terraform) Operation
2.Batch Container Operation
3.Incident Response Operation

View Slide

14
All Hands Meeting
👈

View Slide

15
All Hands Meeting
Overview
● Provide IaC platform that allows developers to develop and operate
infrastructure with Software Development Life Cycle (with Terraform)

View Slide

16
All Hands Meeting
Policy as Code with Conftest/Rego
● Automatic review of semantics problems that cannot be covered by existing static
analysis tools without relying on certain human review by introducing Policy as
Code Operational Load⬇

View Slide

17
All Hands Meeting
User-friendly CI Notiﬁcation
● Notify users of the results of executing Terraform and conftest commands in CI in
a form that is easy for them to understand what to change and how to change it
● https://github.com/suzuki-shunsuke/tfcmt
● https://github.com/suzuki-shunsuke/github-comment
Cognitive Load⬇

View Slide

18
All Hands Meeting
Terraform/AWS Workshop for Developers
● Held workshops to raise the knowledge level of Developers' Terraform and Cloud
Infrastructure Cognitive Load⬇

View Slide

19
All Hands Meeting
👈

View Slide

20
All Hands Meeting
Overview
● Provide batch container platform for developers with AWS Fargate + Amazon
Eventbridge + AWS Lambda
○ to manage batch schedule and infra computing resources with SDLC by
adding simple parameters with Terraform
○ to execute adhoc batch task by using GitHub Actions

View Slide

21
All Hands Meeting
ECS Fargate worker task auto scaler with AWS Lambda
● Autoscaling based on current Fargate tasks and SQS depth
○ Determine the number of tasks to execute based on the diﬀerence between
the “Backlog (VisibleMsg Count)” and the “Appropriate-Backlog (currently
running tasks x capacity per speciﬁed task)”
● Eliminates the need for detailed capacity planning Operational Load⬇

View Slide

22
All Hands Meeting
Terraform module with few required parameters
● Developers can easily deploy a resource by simply adding a minimum list of
variables and calling it with a module
● Developers can override CPU/Memory/Task Count and other parameters as
needed
Cognitive Load⬇

View Slide

23
All Hands Meeting
Adhoc batch task runner with GitHub Actions Workﬂow Dispatch
● Validate if the user can execute the program by using the GitHub User ID (Team
ID) at the ﬁrst step of the job
● Easily track history of who did what
Secure⬆
Auditable
⬆

View Slide

24
All Hands Meeting
3.Incident Response Operation 👈

View Slide

25
All Hands Meeting
Overview
● Provide Incident Response platform with ChatOps interface to reduce the
burden of response to incidents, shorten MTTR as much as possible, and
complete Postmortems process

View Slide

26
All Hands Meeting
ChatOps to issue Incident ticket/channel
● Integrate with Slack, which everyone is familiar with, and make it possible to
report incidents with as simple commands and steps as possible Cognitive Load⬇

View Slide

27
All Hands Meeting
Add Incident Response ﬂow to General On-boarding Process
● Labor-saving and continuous recognition can be ensured by having the
introduction of incident response ﬂow incorporated in the onboarding process
with BOT Cognitive Load⬇

View Slide

28
All Hands Meeting
Postmortem Template
● Postmortems can be created from templates with one click of a button on
Conﬂuence Operational Load⬇

View Slide

29
All Hands Meeting
Future Prospects
(Quoted from O’Reilly|Seeking SRE Chapter.4)
Operational Load⬇
● Introduction of “Timeline Model” to automate incident response ﬂow more
● Measure time between “Response” and “Mitigate” and “Repair” and Analyse them
to shorten MTTR

View Slide

30
All Hands Meeting
● Good “Self-Service” Operations are
○ Low Cognitive Load
○ Low Operational Load for “Users”
○ Secure and Auditable
 
Conclusion

View Slide

31
All Hands Meeting
● Self-Service Operations
● eurekaにおけるここ一年のTerraform Component Delivery Processの変化急成長
していくProduct基盤のProductivity,Security,Privacyとの向き合い
● Terraformのレビューを自動化するために、Conftestを導入してGitHub ActionsでCIま
で設定してみる
● Scaling based on Amazon SQS
● Self-Serviceとサイロ化と組織構造 / Self-Service, Siloing and Organizational
Structure
● SRE を実現するための組織マネジメント / Management to achieve SRE
● Seeking SRE
● インシデントレスポンスを自動化で支援する Slack Bot で人機一体なセキュリティ対
策を実現する
Reference

View Slide

32

View Slide

Steps toward self-service operations in eureka

Steps toward self-service operations in eureka

fukubaka0825

More Decks by fukubaka0825

Other Decks in Technology

Featured

Transcript