投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

Transcript

1. Copyright © 2009-2017 eureka, inc. All rights reserved. Takuya Onda / eureka, Inc. 2017-06-16 @Minami Aoyama Night #3 ౤ߘ؂ࢹϚΠΫϩαʔϏεͷܧଓతͳσϓϩΠͱ ߏ੒มߋͷ࣮ݱखஈͷ঺հ
2. Introduction ▪ Takuya Onda – eureka, Inc. – SRE team Engineer Lead
3. Agenda ▪ 1. About Us ▪ 2. Introduction of our Text Moderation System ▪ 3. Continuous Delivery ▪ 4. Continuous Configuration Management
4. CONFIDENTIAL
5. About Us - Pairs https://eure.jp/pairs-5million-thanks/
6. About Us - Couples
7. About Us - IAC/Match Group
8. 2017.01 2016.01 2016.09 2015.05 2017 - 2015 2014 - 2008 Our History "Couples" gets over 4 million downloads. "Pairs" gets over 5 million user and 32 million matchings. Junya Ishibashi is elected CEO, succeeding Yu Akasaka, who retained chairman title. IAC/InterActiveCorp acquires eureka. 2014.06 2014.05 2013.10 2012.10 2011.09 2010.08 2008.11 "Pairs" gets over 1 million user and 3.3 million matchings. eureka launches communication app for couple, "Couples". eureka launches online-dating service "Pairs" in Taiwan. eureka launches online-dating service "Pairs" in Japan. eureka starts smartphone and Facebook app services. eureka starts blog marketing services.
 eureka is founded in Tokyo.
9. About Us - People
10. 1. Text Moderation System
11. How to use Pairs
12. Search Message Like
13. Moderation
14. Before Introduction ಋೖલ
15. Cooperative Company Manual Text Moderation ( OK ) Pairs ᶃ ᶄ ᶅ ᶆ
16. Cooperative Company Manual Text Moderation ( NG ) Pairs ᶃ ᶄ ᶆ ᶅ
17. Cooperative Company Manual Text Moderation ( Service Growth ) Pairs ᶃ ᶄ ᶆ ᶅ
18. Tradeoff between Safety and Cost Expense ▪ Service Safety – Brand image can be ruined easily – Once we lose user’s trust, service growth will be disrupted ▪ Cost – The more messages we process, the more people 
 we need
19. After Introduction ಋೖޙ
20. Automated Text Moderation Pairs ᶃ ᶄ ᶆ ᶅ Text Moderation System Cooperative Company ᶅ
21. Technical Overview ▪ Machine Leaning – Supervised Leaning By Manual Text Moderation Data ▪ Microservice – Independent Deployment Pipeline ▪ Cloud – AWS ALB / EC2 / Aurora – CodeBuild / CodeDeploy / CodePipeline
22. Result ▪ Cost Reduction – Massive labor cost turned into much smaller AWS operational cost ▪ Service KPI – Still under observation
23. 3. Continuous Delivery
24. Why we need? ͳͥ΍Δͷʁ
25. Why we need Continuous Delivery ▪ Reducing Time to Value – You write code. You commit it. Then what? – Your code doesn’t produce any benefit until you launch ▪ Service Quality – Dev : It’s not my code! Somebody caused it! ▪ Quick Feedback – We don’t know what happens until we launch new feature
26. What matters is.. ܧଓతσϦόϦΛߦ͏ʹ͸,,
27. Automation & Reproducibility ▪ No Manual Deployment – It should be automated – It should be reproduced – Your build it, You run it
28. Branch Based Deployment ▪ Keep Master-branch always be releasable – Check In = Deploy ( Should be Automated ) – Rollback = Revert & Deploy – Don’t leave disruptive commits on master branch
29. Reliable Staging Environment ▪ Test passed on staging = release immediately – Twelve-Factor of App ( App should not care where it runs ) – It should be same middleware / Orchestration process
30. System Architecture ౤ߘ؂ࢹγεςϜͷDeployͷ࢓૊Έ
31. System Architecture
32. System Architecture ▪ Fully automated – Deploy when you merge master-branch – Same as Staging (When you merge stage-branch) ▪ No centralized build / deploy server – Use AWS Managed Service – No need to monitor Build / Deploy pipeline modules
33. System Architecture (Deploy) ᶃ
34. System Architecture (Deploy) ᶃ
35. System Architecture (Deploy) ᶄ
36. System Architecture (Deploy) ᶅ
37. System Architecture Build ᶅ
38. System Architecture (Deploy) ᶆ
39. System Architecture ᶇ
40. System Architecture Deploy ᶆ
41. System Architecture (Auto Scaling)
42. System Architecture (Auto Scaling) ᶃ
43. System Architecture (Auto Scaling) ᶄ
44. System Architecture (Auto Scaling) ᶅ
45. System Architecture (Auto Scaling) ᶄ
46. Result ▪ Fully automated deployment process – Check In = Deploy – No environmental differences ( if you check-in stage branch,It`s shipped to staging environment) ▪ No centralized build/deploy server – Use AWS Managed Service
47. 4. Continuous Configuration Management
48. Why we need? ͳͥ΍Δͷʁ
49. Why we need Continuous Configuration Management ▪ Reduce security disruption – Need frequent Middleware Update ▪ Accelerate our Development – Dev : How long do we have to keep using PHP5.xx??
50. What matters is.. ܧଓతͳߏ੒มߋΛ͓͜ͳ͏ʹ͸,,
51. Code-based management & reusability ▪ No Manual Configuration – It should be managed on Code – It should be standardized – Is your ansible-role really reusable??
52. Scheduled(automated) & frequent apply ▪ How to fight against config-changes phobia? – When you apply your playbook last time?? – Can you really trust your playbook? – More frequently you apply, less scared you feel
53. Treat servers as cattle ▪ Be disposable – Since we use AWS, Why not? – Be Stateless ( Logs / Cache / Etc,,) – Don’t modify server outside of config-management- pipeline
54. System Architecture ౤ߘ؂ࢹγεςϜͷߏ੒؅ཧ/มߋͷ࢓૊Έ
55. System Architecture ( Packer Image Transition )
56. System Architecture ( Packer Image Transition )
57. System Architecture ( Packer Image Transition ) • env : prod • domain : stage-hoge.eureka.jp • Nginx : • log_prefix : stage-hoge • upstream : hoge • mackerel_check_metrics : • nginx • port : 80 • log : /var/log/nginx/xxx
58. System Architecture ( Packer Image Transition ) ▪ Fully codenized machine-image (AMI) – Testable ( serverspec ) ▪ Provisioning same middleware – Control configuration diff by variables – Exɿservice / region / env
59. System Architecture ( apply new configuration )
60. System Architecture ( apply new configuration )
61. System Architecture ( apply new configuration )
62. System Architecture ( apply new configuration )
63. System Architecture ( apply new configuration ) ▪ Terraforming & Blue Green – Modify launch-configuration and rotate EC2 ▪ Scheduled rotation – Running EC2 always be launched from clean AMI – Avoid to leave manual config modification ▪ Not Fully Automated – Need to apply by Dev – Pipelining between Packer-build result and Terraform configuration (AMI ID)
64. Summary
65. Summary ▪ Text Moderation System – Use Machine Leaning – Build as Micro Service – Build on Cloud ▪ Deploy and Configuration Management – Why we need Continues Improvement – Continuous Delivery – Continuous Configuration Management
66. CONFIDENTIAL Thank you :) Thank you :)