Upgrade to Pro — share decks privately, control downloads, hide ads and more …

投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

 投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

投稿監視マイクロサービスの継続的なデプロイと構成変更の実現手段の紹介

takuya542

June 16, 2017
Tweet

More Decks by takuya542

Other Decks in Programming

Transcript

  1. Copyright © 2009-2017 eureka, inc. All rights reserved.
    Takuya Onda / eureka, Inc.

    2017-06-16 @Minami Aoyama Night #3
    ౤ߘ؂ࢹϚΠΫϩαʔϏεͷܧଓతͳσϓϩΠͱ
    ߏ੒มߋͷ࣮ݱखஈͷ঺հ

    View full-size slide

  2. Introduction
    ■ Takuya Onda

    – eureka, Inc.

    – SRE team Engineer Lead

    View full-size slide

  3. Agenda
    ■ 1. About Us

    ■ 2. Introduction of our Text Moderation System

    ■ 3. Continuous Delivery

    ■ 4. Continuous Configuration Management

    View full-size slide

  4. CONFIDENTIAL

    View full-size slide

  5. About Us - Pairs
    https://eure.jp/pairs-5million-thanks/

    View full-size slide

  6. About Us - Couples

    View full-size slide

  7. About Us - IAC/Match Group

    View full-size slide

  8. 2017.01

    2016.01

    2016.09

    2015.05
    2017 - 2015 2014 - 2008
    Our History
    "Couples" gets over 4 million
    downloads.
    "Pairs" gets over 5 million user
    and 32 million matchings.
    Junya Ishibashi is elected CEO,
    succeeding Yu Akasaka, who
    retained chairman title.
    IAC/InterActiveCorp acquires
    eureka.
    2014.06
    2014.05
    2013.10
    2012.10
    2011.09
    2010.08
    2008.11
    "Pairs" gets over 1 million user
    and 3.3 million matchings.
    eureka launches communication
    app for couple, "Couples".
    eureka launches online-dating
    service "Pairs" in Taiwan.
    eureka launches online-dating
    service "Pairs" in Japan.
    eureka starts smartphone and
    Facebook app services.
    eureka starts blog marketing
    services.

    eureka is founded in Tokyo.

    View full-size slide

  9. About Us - People

    View full-size slide

  10. 1. Text Moderation System

    View full-size slide

  11. How to use Pairs

    View full-size slide

  12. Search Message
    Like

    View full-size slide

  13. Before Introduction
    ಋೖલ

    View full-size slide

  14. Cooperative Company
    Manual Text Moderation ( OK )
    Pairs




    View full-size slide

  15. Cooperative Company
    Manual Text Moderation ( NG )
    Pairs




    View full-size slide

  16. Cooperative Company
    Manual Text Moderation ( Service Growth )
    Pairs




    View full-size slide

  17. Tradeoff between Safety and Cost Expense
    ■ Service Safety

    – Brand image can be ruined easily

    – Once we lose user’s trust, service growth will be disrupted

    ■ Cost

    – The more messages we process, the more people 

    we need

    View full-size slide

  18. After Introduction
    ಋೖޙ

    View full-size slide

  19. Automated Text Moderation
    Pairs
    ᶃ ᶄ


    Text Moderation
    System
    Cooperative
    Company

    View full-size slide

  20. Technical Overview
    ■ Machine Leaning

    – Supervised Leaning By Manual Text Moderation Data

    ■ Microservice

    – Independent Deployment Pipeline

    ■ Cloud

    – AWS ALB / EC2 / Aurora

    – CodeBuild / CodeDeploy / CodePipeline

    View full-size slide

  21. Result
    ■ Cost Reduction

    – Massive labor cost turned into much smaller AWS
    operational cost

    ■ Service KPI

    – Still under observation

    View full-size slide

  22. 3. Continuous Delivery

    View full-size slide

  23. Why we need?
    ͳͥ΍Δͷʁ

    View full-size slide

  24. Why we need Continuous Delivery
    ■ Reducing Time to Value

    – You write code. You commit it. Then what?

    – Your code doesn’t produce any benefit until you launch

    ■ Service Quality

    – Dev : It’s not my code! Somebody caused it!

    ■ Quick Feedback

    – We don’t know what happens until we launch new feature

    View full-size slide

  25. What matters is..
    ܧଓతσϦόϦΛߦ͏ʹ͸,,

    View full-size slide

  26. Automation & Reproducibility
    ■ No Manual Deployment

    – It should be automated

    – It should be reproduced

    – Your build it, You run it

    View full-size slide

  27. Branch Based Deployment
    ■ Keep Master-branch always be releasable

    – Check In = Deploy ( Should be Automated )

    – Rollback = Revert & Deploy

    – Don’t leave disruptive commits on master branch

    View full-size slide

  28. Reliable Staging Environment
    ■ Test passed on staging = release immediately

    – Twelve-Factor of App ( App should not care where it
    runs )

    – It should be same middleware / Orchestration process

    View full-size slide

  29. System Architecture
    ౤ߘ؂ࢹγεςϜͷDeployͷ࢓૊Έ

    View full-size slide

  30. System Architecture

    View full-size slide

  31. System Architecture
    ■ Fully automated

    – Deploy when you merge master-branch

    – Same as Staging (When you merge stage-branch)

    ■ No centralized build / deploy server

    – Use AWS Managed Service

    – No need to monitor Build / Deploy pipeline modules

    View full-size slide

  32. System Architecture (Deploy)

    View full-size slide

  33. System Architecture (Deploy)

    View full-size slide

  34. System Architecture (Deploy)

    View full-size slide

  35. System Architecture (Deploy)

    View full-size slide

  36. System Architecture
    Build

    View full-size slide

  37. System Architecture (Deploy)

    View full-size slide

  38. System Architecture

    View full-size slide

  39. System Architecture
    Deploy

    View full-size slide

  40. System Architecture (Auto Scaling)

    View full-size slide

  41. System Architecture (Auto Scaling)

    View full-size slide

  42. System Architecture (Auto Scaling)

    View full-size slide

  43. System Architecture (Auto Scaling)

    View full-size slide

  44. System Architecture (Auto Scaling)

    View full-size slide

  45. Result
    ■ Fully automated deployment process

    – Check In = Deploy

    – No environmental differences ( if you check-in stage
    branch,It`s shipped to staging environment)

    ■ No centralized build/deploy server

    – Use AWS Managed Service

    View full-size slide

  46. 4. Continuous Configuration Management

    View full-size slide

  47. Why we need?
    ͳͥ΍Δͷʁ

    View full-size slide

  48. Why we need Continuous Configuration Management
    ■ Reduce security disruption

    – Need frequent Middleware Update

    ■ Accelerate our Development

    – Dev : How long do we have to keep using PHP5.xx??

    View full-size slide

  49. What matters is..
    ܧଓతͳߏ੒มߋΛ͓͜ͳ͏ʹ͸,,

    View full-size slide

  50. Code-based management & reusability
    ■ No Manual Configuration

    – It should be managed on Code

    – It should be standardized

    – Is your ansible-role really reusable??

    View full-size slide

  51. Scheduled(automated) & frequent apply
    ■ How to fight against config-changes phobia?

    – When you apply your playbook last time??

    – Can you really trust your playbook?

    – More frequently you apply, less scared you feel

    View full-size slide

  52. Treat servers as cattle
    ■ Be disposable

    – Since we use AWS, Why not?

    – Be Stateless ( Logs / Cache / Etc,,)

    – Don’t modify server outside of config-management-
    pipeline

    View full-size slide

  53. System Architecture
    ౤ߘ؂ࢹγεςϜͷߏ੒؅ཧ/มߋͷ࢓૊Έ

    View full-size slide

  54. System Architecture ( Packer Image Transition )

    View full-size slide

  55. System Architecture ( Packer Image Transition )

    View full-size slide

  56. System Architecture ( Packer Image Transition )
    • env : prod
    • domain : stage-hoge.eureka.jp
    • Nginx :
    • log_prefix : stage-hoge
    • upstream : hoge
    • mackerel_check_metrics :
    • nginx
    • port : 80
    • log : /var/log/nginx/xxx

    View full-size slide

  57. System Architecture ( Packer Image Transition )
    ■ Fully codenized machine-image (AMI)

    – Testable ( serverspec )

    ■ Provisioning same middleware

    – Control configuration diff by variables

    – Exɿservice / region / env

    View full-size slide

  58. System Architecture ( apply new configuration )

    View full-size slide

  59. System Architecture ( apply new configuration )

    View full-size slide

  60. System Architecture ( apply new configuration )

    View full-size slide

  61. System Architecture ( apply new configuration )

    View full-size slide

  62. System Architecture ( apply new configuration )
    ■ Terraforming & Blue Green

    – Modify launch-configuration and rotate EC2

    ■ Scheduled rotation

    – Running EC2 always be launched from clean AMI

    – Avoid to leave manual config modification

    ■ Not Fully Automated

    – Need to apply by Dev

    – Pipelining between Packer-build result and Terraform
    configuration (AMI ID)

    View full-size slide

  63. Summary
    ■ Text Moderation System

    – Use Machine Leaning

    – Build as Micro Service

    – Build on Cloud

    ■ Deploy and Configuration Management

    – Why we need Continues Improvement

    – Continuous Delivery

    – Continuous Configuration Management

    View full-size slide

  64. CONFIDENTIAL
    Thank
    you :)
    Thank you :)

    View full-size slide