Upgrade to Pro — share decks privately, control downloads, hide ads and more …

世界展開する大規模ウェブサービスのデプロイを支える技術 / YAPC::Asia Tokyo 2015

Hatena
August 21, 2015

世界展開する大規模ウェブサービスのデプロイを支える技術 / YAPC::Asia Tokyo 2015

Miiverse とは任天堂株式会社が運営しているウェブサービスであり、世界中の Wii U やニンテンドー3DS、そして PC やスマートデバイスから利用することができます。

AWS 上でマルチリージョン構成をとり大量のサーバを抱える Miiverse のデプロイを支える技術と運用上の工夫、そして株式会社はてなと任天堂株式会社が共同で開発する Git リポジトリの同期システムの構築を通して得られた経験をもとに、大規模なウェブサービスを素早くかつ安全に改善する方法を紹介します。

※本資料は YAPC::Asia Tokyo 2015 での発表資料となります。
http://yapcasia.org/2015/talk/show/9ec2791c-05e5-11e5-81fa-79c97d574c3a

Hatena

August 21, 2015
Tweet

More Decks by Hatena

Other Decks in Technology

Transcript

  1. 3DS/Wii U
    Miiverse

    View full-size slide

  2. Miiverse
    Miiverse
    Git

    View full-size slide

  3. Wii U 3DS
    Mii

    View full-size slide

  4. Miiverse
    API /

    View full-size slide

  5. 3
    Miiverse Amazon Web Services (AWS)
    JP/US/EU 3

    View full-size slide

  6. JAWS UG
    Developers Summit
    http://developer.hatenastaff.com/entry/2014/03/25/1
    85521
    https://speakerdeck.com/hatena/devsumi-2015-
    miiverse

    View full-size slide

  7. Capistrano2 & Git
    Git
    Git
    git pull
    Pull

    View full-size slide

  8. Capistrano2 & Git
    git pull
    Miiverse 3
    git pull
    Git git pull

    View full-size slide

  9. Git
    Git Slave
    Miiverse Git Slave
    Git Slave git pull

    View full-size slide

  10. Git
    Git Slave
    Git Slave 10 1 lsyncd inode
    rsync
    git push
    lsyncd
    rsync

    View full-size slide

  11. Git
    Git
    Git Slave
    Wii U 3DS etc...
    Git Slave
    sleep `expr $RANDOM % #{$main.sleep_divisor}` && git ...

    View full-size slide

  12. Git
    lsyncd
    JS CSS Git
    rsync git fetch

    View full-size slide

  13. Git
    Git Slave
    Git

    View full-size slide

  14. tips
    Capistrano
    role
    Mackerel API
    net-ssh

    View full-size slide

  15. tips
    Mackerel
    https://mackerel.io/

    View full-size slide

  16. tips
    Mackerel
    role Capistrano

    View full-size slide

  17. tips
    Mackerel
    Mackerel
    AutoScale
    Mackerel /
    cron kick
    EC2

    View full-size slide

  18. Miiverse
    Miiverse Pull
    Cap2 Git
    Git
    Git Slave
    Git Slave

    View full-size slide

  19. Miiverse
    Redmine

    View full-size slide

  20. Miiverse
    Redmine
    GitHub Enterprise NEW!

    View full-size slide

  21. 2 git push
    git push ghe master
    git push origin master
    Merge pull request
    GHE

    View full-size slide

  22. google/hesokuri
    ghm

    View full-size slide

  23. hesokuri
    google/hesokuri
    Distributed Git repo synchronization tool.
    Google Git

    View full-size slide

  24. ghm
    in-house Web Application for mirroring pairs
    managements
    Stores mirroring pairs
    Has Webhook API that invoke mirroring (later)
    Provides some APIs
    @
    GHE 2 GREE Tech Talk

    View full-size slide

  25. 3
    ghm Miiverse

    View full-size slide

  26. JSON over HTTP API

    View full-size slide

  27. REST API
    Git : /servers(/:id)
    : /servers/:id/promote,
    /servers/:id/demote
    : /servers/:server_id/repos(/:id)
    : /jobs/sync

    View full-size slide

  28. (1)
    GHE git push

    View full-size slide

  29. (2)
    GHE Webhook

    View full-size slide

  30. (3)
    Webhook payload

    View full-size slide

  31. (4)
    Git Slave DB

    View full-size slide

  32. (5)
    Git Slave DB

    View full-size slide

  33. (6)
    GHE git pull

    View full-size slide

  34. GHE
    1. GHE git push
    2. GHE Webhook
    3. payload
    4. Git
    5. Git
    6. GHE git pull
    7. git push

    View full-size slide

  35. HTTP POST
    github.com GHE
    Git

    View full-size slide

  36. Git
    Git
    Git
    git push --force

    View full-size slide

  37. Git Pull
    Git Pull 2
    1. commit
    2.

    View full-size slide

  38. Git Pull
    Miiverse
    Git
    3

    View full-size slide

  39. Consul + stretcher
    strecher Pull
    Hashicorp Consul
    http://tech.kayac.com/archive/10_stretcher.html
    https://github.com/fujiwara/stretcher

    View full-size slide

  40. Consul + stretcher
    1.
    Amazon S3
    2. manifest (YAML)
    Amazon S3
    3. consul event
    4. stretcher agent

    View full-size slide

  41. Consul + stretcher
    Amazon S3
    Git
    manifest
    rollback manifest

    View full-size slide

  42. Consul + stretcher
    Jenkins
    Amazon S3

    View full-size slide

  43. 1) GHE git push

    View full-size slide

  44. 2) GHE webhook Jenkins

    View full-size slide

  45. 3) Jenkins
    S3 push

    View full-size slide

  46. 4) Consul master

    View full-size slide

  47. 5) backend
    S3

    View full-size slide

  48. Git
    Cap2 & Git
    Consul + stretcher

    View full-size slide

  49. 100
    AWS
    JP/US/EU
    Miiverse

    View full-size slide

  50. Cap2 & Git 1,142
    Consul + strecher 29
    strecher 40 !

    View full-size slide

  51. Consul + strecher
    Cap2 & Git
    Consul + strecher
    deploy:update
    git pull vs S3 DL

    View full-size slide

  52. Consul + strecher
    carton install
    210
    1/10
    carton install
    Cap2 & Git

    View full-size slide

  53. 1
    AutoScale
    Consul + strecher

    View full-size slide

  54. 2
    Miiverse
    Miiverse 3
    Consul
    https://www.consul.io/docs/guides/datacenters.html

    View full-size slide

  55. Pull
    Consul + stretcher
    Miiverse
    40
    AutoScale

    View full-size slide

  56. Miiverse
    Miiverse
    Pull
    Git
    Git
    Consul + stretcher

    View full-size slide