Upgrade to Pro — share decks privately, control downloads, hide ads and more …

投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

 投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

7890032b748bfc156d75aca46db99562?s=128

takuya542

June 16, 2017
Tweet

Transcript

  1. Copyright © 2009-2017 eureka, inc. All rights reserved. Takuya Onda

    / eureka, Inc. 2017-06-16 @Minami Aoyama Night #3 ౤ߘ؂ࢹϚΠΫϩαʔϏεͷܧଓతͳσϓϩΠͱ ߏ੒มߋͷ࣮ݱखஈͷ঺հ
  2. Introduction ▪ Takuya Onda – eureka, Inc. – SRE team

    Engineer Lead
  3. Agenda ▪ 1. About Us ▪ 2. Introduction of our

    Text Moderation System ▪ 3. Continuous Delivery ▪ 4. Continuous Configuration Management
  4. CONFIDENTIAL

  5. About Us - Pairs https://eure.jp/pairs-5million-thanks/

  6. About Us - Couples

  7. About Us - IAC/Match Group

  8. 2017.01 2016.01 2016.09 2015.05 2017 - 2015 2014 - 2008

    Our History "Couples" gets over 4 million downloads. "Pairs" gets over 5 million user and 32 million matchings. Junya Ishibashi is elected CEO, succeeding Yu Akasaka, who retained chairman title. IAC/InterActiveCorp acquires eureka. 2014.06 2014.05 2013.10 2012.10 2011.09 2010.08 2008.11 "Pairs" gets over 1 million user and 3.3 million matchings. eureka launches communication app for couple, "Couples". eureka launches online-dating service "Pairs" in Taiwan. eureka launches online-dating service "Pairs" in Japan. eureka starts smartphone and Facebook app services. eureka starts blog marketing services.
 eureka is founded in Tokyo.
  9. About Us - People

  10. 1. Text Moderation System

  11. How to use Pairs

  12. Search Message Like

  13. Moderation

  14. Before Introduction ಋೖલ

  15. Cooperative Company Manual Text Moderation ( OK ) Pairs ᶃ

    ᶄ ᶅ ᶆ
  16. Cooperative Company Manual Text Moderation ( NG ) Pairs ᶃ

    ᶄ ᶆ ᶅ
  17. Cooperative Company Manual Text Moderation ( Service Growth ) Pairs

    ᶃ ᶄ ᶆ ᶅ
  18. Tradeoff between Safety and Cost Expense ▪ Service Safety –

    Brand image can be ruined easily – Once we lose user’s trust, service growth will be disrupted ▪ Cost – The more messages we process, the more people 
 we need
  19. After Introduction ಋೖޙ

  20. Automated Text Moderation Pairs ᶃ ᶄ ᶆ ᶅ Text Moderation

    System Cooperative Company ᶅ
  21. Technical Overview ▪ Machine Leaning – Supervised Leaning By Manual

    Text Moderation Data ▪ Microservice – Independent Deployment Pipeline ▪ Cloud – AWS ALB / EC2 / Aurora – CodeBuild / CodeDeploy / CodePipeline
  22. Result ▪ Cost Reduction – Massive labor cost turned into

    much smaller AWS operational cost ▪ Service KPI – Still under observation
  23. 3. Continuous Delivery

  24. Why we need? ͳͥ΍Δͷʁ

  25. Why we need Continuous Delivery ▪ Reducing Time to Value

    – You write code. You commit it. Then what? – Your code doesn’t produce any benefit until you launch ▪ Service Quality – Dev : It’s not my code! Somebody caused it! ▪ Quick Feedback – We don’t know what happens until we launch new feature
  26. What matters is.. ܧଓతσϦόϦΛߦ͏ʹ͸,,

  27. Automation & Reproducibility ▪ No Manual Deployment – It should

    be automated – It should be reproduced – Your build it, You run it
  28. Branch Based Deployment ▪ Keep Master-branch always be releasable –

    Check In = Deploy ( Should be Automated ) – Rollback = Revert & Deploy – Don’t leave disruptive commits on master branch
  29. Reliable Staging Environment ▪ Test passed on staging = release

    immediately – Twelve-Factor of App ( App should not care where it runs ) – It should be same middleware / Orchestration process
  30. System Architecture ౤ߘ؂ࢹγεςϜͷDeployͷ࢓૊Έ

  31. System Architecture

  32. System Architecture ▪ Fully automated – Deploy when you merge

    master-branch – Same as Staging (When you merge stage-branch) ▪ No centralized build / deploy server – Use AWS Managed Service – No need to monitor Build / Deploy pipeline modules
  33. System Architecture (Deploy) ᶃ

  34. System Architecture (Deploy) ᶃ

  35. System Architecture (Deploy) ᶄ

  36. System Architecture (Deploy) ᶅ

  37. System Architecture Build ᶅ

  38. System Architecture (Deploy) ᶆ

  39. System Architecture ᶇ

  40. System Architecture Deploy ᶆ

  41. System Architecture (Auto Scaling)

  42. System Architecture (Auto Scaling) ᶃ

  43. System Architecture (Auto Scaling) ᶄ

  44. System Architecture (Auto Scaling) ᶅ

  45. System Architecture (Auto Scaling) ᶄ

  46. Result ▪ Fully automated deployment process – Check In =

    Deploy – No environmental differences ( if you check-in stage branch,It`s shipped to staging environment) ▪ No centralized build/deploy server – Use AWS Managed Service
  47. 4. Continuous Configuration Management

  48. Why we need? ͳͥ΍Δͷʁ

  49. Why we need Continuous Configuration Management ▪ Reduce security disruption

    – Need frequent Middleware Update ▪ Accelerate our Development – Dev : How long do we have to keep using PHP5.xx??
  50. What matters is.. ܧଓతͳߏ੒มߋΛ͓͜ͳ͏ʹ͸,,

  51. Code-based management & reusability ▪ No Manual Configuration – It

    should be managed on Code – It should be standardized – Is your ansible-role really reusable??
  52. Scheduled(automated) & frequent apply ▪ How to fight against config-changes

    phobia? – When you apply your playbook last time?? – Can you really trust your playbook? – More frequently you apply, less scared you feel
  53. Treat servers as cattle ▪ Be disposable – Since we

    use AWS, Why not? – Be Stateless ( Logs / Cache / Etc,,) – Don’t modify server outside of config-management- pipeline
  54. System Architecture ౤ߘ؂ࢹγεςϜͷߏ੒؅ཧ/มߋͷ࢓૊Έ

  55. System Architecture ( Packer Image Transition )

  56. System Architecture ( Packer Image Transition )

  57. System Architecture ( Packer Image Transition ) • env :

    prod • domain : stage-hoge.eureka.jp • Nginx : • log_prefix : stage-hoge • upstream : hoge • mackerel_check_metrics : • nginx • port : 80 • log : /var/log/nginx/xxx
  58. System Architecture ( Packer Image Transition ) ▪ Fully codenized

    machine-image (AMI) – Testable ( serverspec ) ▪ Provisioning same middleware – Control configuration diff by variables – Exɿservice / region / env
  59. System Architecture ( apply new configuration )

  60. System Architecture ( apply new configuration )

  61. System Architecture ( apply new configuration )

  62. System Architecture ( apply new configuration )

  63. System Architecture ( apply new configuration ) ▪ Terraforming &

    Blue Green – Modify launch-configuration and rotate EC2 ▪ Scheduled rotation – Running EC2 always be launched from clean AMI – Avoid to leave manual config modification ▪ Not Fully Automated – Need to apply by Dev – Pipelining between Packer-build result and Terraform configuration (AMI ID)
  64. Summary

  65. Summary ▪ Text Moderation System – Use Machine Leaning –

    Build as Micro Service – Build on Cloud ▪ Deploy and Configuration Management – Why we need Continues Improvement – Continuous Delivery – Continuous Configuration Management
  66. CONFIDENTIAL Thank you :) Thank you :)