Slide 1
Slide 1 text
Copyright © 2009-2017 eureka, inc. All rights reserved.
Takuya Onda / eureka, Inc.
2017-06-16 @Minami Aoyama Night #3
ߘࢹϚΠΫϩαʔϏεͷܧଓతͳσϓϩΠͱ
ߏมߋͷ࣮ݱखஈͷհ
Slide 2
Slide 2 text
Introduction
■ Takuya Onda
– eureka, Inc.
– SRE team Engineer Lead
Slide 3
Slide 3 text
Agenda
■ 1. About Us
■ 2. Introduction of our Text Moderation System
■ 3. Continuous Delivery
■ 4. Continuous Configuration Management
Slide 4
Slide 4 text
CONFIDENTIAL
Slide 5
Slide 5 text
About Us - Pairs
https://eure.jp/pairs-5million-thanks/
Slide 6
Slide 6 text
About Us - Couples
Slide 7
Slide 7 text
About Us - IAC/Match Group
Slide 8
Slide 8 text
2017.01
2016.01
2016.09
2015.05
2017 - 2015 2014 - 2008
Our History
"Couples" gets over 4 million
downloads.
"Pairs" gets over 5 million user
and 32 million matchings.
Junya Ishibashi is elected CEO,
succeeding Yu Akasaka, who
retained chairman title.
IAC/InterActiveCorp acquires
eureka.
2014.06
2014.05
2013.10
2012.10
2011.09
2010.08
2008.11
"Pairs" gets over 1 million user
and 3.3 million matchings.
eureka launches communication
app for couple, "Couples".
eureka launches online-dating
service "Pairs" in Taiwan.
eureka launches online-dating
service "Pairs" in Japan.
eureka starts smartphone and
Facebook app services.
eureka starts blog marketing
services.
eureka is founded in Tokyo.
Slide 9
Slide 9 text
About Us - People
Slide 10
Slide 10 text
1. Text Moderation System
Slide 11
Slide 11 text
How to use Pairs
Slide 12
Slide 12 text
Search Message
Like
Slide 13
Slide 13 text
Moderation
Slide 14
Slide 14 text
Before Introduction
ಋೖલ
Slide 15
Slide 15 text
Cooperative Company
Manual Text Moderation ( OK )
Pairs
ᶃ
ᶄ
ᶅ
ᶆ
Slide 16
Slide 16 text
Cooperative Company
Manual Text Moderation ( NG )
Pairs
ᶃ
ᶄ
ᶆ
ᶅ
Slide 17
Slide 17 text
Cooperative Company
Manual Text Moderation ( Service Growth )
Pairs
ᶃ
ᶄ
ᶆ
ᶅ
Slide 18
Slide 18 text
Tradeoff between Safety and Cost Expense
■ Service Safety
– Brand image can be ruined easily
– Once we lose user’s trust, service growth will be disrupted
■ Cost
– The more messages we process, the more people
we need
Slide 19
Slide 19 text
After Introduction
ಋೖޙ
Slide 20
Slide 20 text
Automated Text Moderation
Pairs
ᶃ ᶄ
ᶆ
ᶅ
Text Moderation
System
Cooperative
Company
ᶅ
Slide 21
Slide 21 text
Technical Overview
■ Machine Leaning
– Supervised Leaning By Manual Text Moderation Data
■ Microservice
– Independent Deployment Pipeline
■ Cloud
– AWS ALB / EC2 / Aurora
– CodeBuild / CodeDeploy / CodePipeline
Slide 22
Slide 22 text
Result
■ Cost Reduction
– Massive labor cost turned into much smaller AWS
operational cost
■ Service KPI
– Still under observation
Slide 23
Slide 23 text
3. Continuous Delivery
Slide 24
Slide 24 text
Why we need?
ͳͥΔͷʁ
Slide 25
Slide 25 text
Why we need Continuous Delivery
■ Reducing Time to Value
– You write code. You commit it. Then what?
– Your code doesn’t produce any benefit until you launch
■ Service Quality
– Dev : It’s not my code! Somebody caused it!
■ Quick Feedback
– We don’t know what happens until we launch new feature
Slide 26
Slide 26 text
What matters is..
ܧଓతσϦόϦΛߦ͏ʹ,,
Slide 27
Slide 27 text
Automation & Reproducibility
■ No Manual Deployment
– It should be automated
– It should be reproduced
– Your build it, You run it
Slide 28
Slide 28 text
Branch Based Deployment
■ Keep Master-branch always be releasable
– Check In = Deploy ( Should be Automated )
– Rollback = Revert & Deploy
– Don’t leave disruptive commits on master branch
Slide 29
Slide 29 text
Reliable Staging Environment
■ Test passed on staging = release immediately
– Twelve-Factor of App ( App should not care where it
runs )
– It should be same middleware / Orchestration process
Slide 30
Slide 30 text
System Architecture
ߘࢹγεςϜͷDeployͷΈ
Slide 31
Slide 31 text
System Architecture
Slide 32
Slide 32 text
System Architecture
■ Fully automated
– Deploy when you merge master-branch
– Same as Staging (When you merge stage-branch)
■ No centralized build / deploy server
– Use AWS Managed Service
– No need to monitor Build / Deploy pipeline modules
Slide 33
Slide 33 text
System Architecture (Deploy)
ᶃ
Slide 34
Slide 34 text
System Architecture (Deploy)
ᶃ
Slide 35
Slide 35 text
System Architecture (Deploy)
ᶄ
Slide 36
Slide 36 text
System Architecture (Deploy)
ᶅ
Slide 37
Slide 37 text
System Architecture
Build
ᶅ
Slide 38
Slide 38 text
System Architecture (Deploy)
ᶆ
Slide 39
Slide 39 text
System Architecture
ᶇ
Slide 40
Slide 40 text
System Architecture
Deploy
ᶆ
Slide 41
Slide 41 text
System Architecture (Auto Scaling)
Slide 42
Slide 42 text
System Architecture (Auto Scaling)
ᶃ
Slide 43
Slide 43 text
System Architecture (Auto Scaling)
ᶄ
Slide 44
Slide 44 text
System Architecture (Auto Scaling)
ᶅ
Slide 45
Slide 45 text
System Architecture (Auto Scaling)
ᶄ
Slide 46
Slide 46 text
Result
■ Fully automated deployment process
– Check In = Deploy
– No environmental differences ( if you check-in stage
branch,It`s shipped to staging environment)
■ No centralized build/deploy server
– Use AWS Managed Service
Slide 47
Slide 47 text
4. Continuous Configuration Management
Slide 48
Slide 48 text
Why we need?
ͳͥΔͷʁ
Slide 49
Slide 49 text
Why we need Continuous Configuration Management
■ Reduce security disruption
– Need frequent Middleware Update
■ Accelerate our Development
– Dev : How long do we have to keep using PHP5.xx??
Slide 50
Slide 50 text
What matters is..
ܧଓతͳߏมߋΛ͓͜ͳ͏ʹ,,
Slide 51
Slide 51 text
Code-based management & reusability
■ No Manual Configuration
– It should be managed on Code
– It should be standardized
– Is your ansible-role really reusable??
Slide 52
Slide 52 text
Scheduled(automated) & frequent apply
■ How to fight against config-changes phobia?
– When you apply your playbook last time??
– Can you really trust your playbook?
– More frequently you apply, less scared you feel
Slide 53
Slide 53 text
Treat servers as cattle
■ Be disposable
– Since we use AWS, Why not?
– Be Stateless ( Logs / Cache / Etc,,)
– Don’t modify server outside of config-management-
pipeline
Slide 54
Slide 54 text
System Architecture
ߘࢹγεςϜͷߏཧ/มߋͷΈ
Slide 55
Slide 55 text
System Architecture ( Packer Image Transition )
Slide 56
Slide 56 text
System Architecture ( Packer Image Transition )
Slide 57
Slide 57 text
System Architecture ( Packer Image Transition )
• env : prod
• domain : stage-hoge.eureka.jp
• Nginx :
• log_prefix : stage-hoge
• upstream : hoge
• mackerel_check_metrics :
• nginx
• port : 80
• log : /var/log/nginx/xxx
Slide 58
Slide 58 text
System Architecture ( Packer Image Transition )
■ Fully codenized machine-image (AMI)
– Testable ( serverspec )
■ Provisioning same middleware
– Control configuration diff by variables
– Exɿservice / region / env
Slide 59
Slide 59 text
System Architecture ( apply new configuration )
Slide 60
Slide 60 text
System Architecture ( apply new configuration )
Slide 61
Slide 61 text
System Architecture ( apply new configuration )
Slide 62
Slide 62 text
System Architecture ( apply new configuration )
Slide 63
Slide 63 text
System Architecture ( apply new configuration )
■ Terraforming & Blue Green
– Modify launch-configuration and rotate EC2
■ Scheduled rotation
– Running EC2 always be launched from clean AMI
– Avoid to leave manual config modification
■ Not Fully Automated
– Need to apply by Dev
– Pipelining between Packer-build result and Terraform
configuration (AMI ID)
Slide 64
Slide 64 text
Summary
Slide 65
Slide 65 text
Summary
■ Text Moderation System
– Use Machine Leaning
– Build as Micro Service
– Build on Cloud
■ Deploy and Configuration Management
– Why we need Continues Improvement
– Continuous Delivery
– Continuous Configuration Management
Slide 66
Slide 66 text
CONFIDENTIAL
Thank
you :)
Thank you :)