Upgrade to Pro — share decks privately, control downloads, hide ads and more …

世界展開する大規模ウェブサービスのデプロイを支える技術 / YAPC::Asia Tokyo 2015

Hatena
August 21, 2015

世界展開する大規模ウェブサービスのデプロイを支える技術 / YAPC::Asia Tokyo 2015

Miiverse とは任天堂株式会社が運営しているウェブサービスであり、世界中の Wii U やニンテンドー3DS、そして PC やスマートデバイスから利用することができます。

AWS 上でマルチリージョン構成をとり大量のサーバを抱える Miiverse のデプロイを支える技術と運用上の工夫、そして株式会社はてなと任天堂株式会社が共同で開発する Git リポジトリの同期システムの構築を通して得られた経験をもとに、大規模なウェブサービスを素早くかつ安全に改善する方法を紹介します。

※本資料は YAPC::Asia Tokyo 2015 での発表資料となります。
http://yapcasia.org/2015/talk/show/9ec2791c-05e5-11e5-81fa-79c97d574c3a

Hatena

August 21, 2015
Tweet

More Decks by Hatena

Other Decks in Technology

Transcript

  1. View Slide

  2. View Slide

  3. Miiverse

    View Slide

  4. 3DS/Wii U
    Miiverse

    View Slide

  5. Miiverse

    View Slide

  6. Miiverse
    Miiverse
    Git

    View Slide

  7. Miiverse

    View Slide

  8. Wii U 3DS
    Mii

    View Slide

  9. View Slide

  10. View Slide

  11. Web

    View Slide

  12. Miiverse
    API /

    View Slide

  13. View Slide

  14. 3
    Miiverse Amazon Web Services (AWS)
    JP/US/EU 3

    View Slide

  15. View Slide

  16. REST API

    View Slide

  17. JAWS UG
    Developers Summit
    http://developer.hatenastaff.com/entry/2014/03/25/1
    85521
    https://speakerdeck.com/hatena/devsumi-2015-
    miiverse

    View Slide

  18. Miiverse

    View Slide

  19. Miiverse

    View Slide

  20. Capistrano2 & Git
    Git
    Git
    git pull
    Pull

    View Slide

  21. Capistrano2 & Git
    git pull
    Miiverse 3
    git pull
    Git git pull

    View Slide

  22. Git
    Git Slave
    Miiverse Git Slave
    Git Slave git pull

    View Slide

  23. Git
    Git Slave
    Git Slave 10 1 lsyncd inode
    rsync
    git push
    lsyncd
    rsync

    View Slide

  24. Git
    Git
    Git Slave
    Wii U 3DS etc...
    Git Slave
    sleep `expr $RANDOM % #{$main.sleep_divisor}` && git ...

    View Slide

  25. Git
    lsyncd
    JS CSS Git
    rsync git fetch

    View Slide

  26. Git
    Git Slave
    Git

    View Slide

  27. tips

    View Slide

  28. tips
    Capistrano
    role
    Mackerel API
    net-ssh

    View Slide

  29. tips
    Mackerel
    https://mackerel.io/

    View Slide

  30. tips
    Mackerel
    role Capistrano

    View Slide

  31. tips
    Mackerel
    Mackerel
    AutoScale
    Mackerel /
    cron kick
    EC2

    View Slide

  32. tips
    DB

    View Slide

  33. Miiverse
    Miiverse Pull
    Cap2 Git
    Git
    Git Slave
    Git Slave

    View Slide

  34. Git

    View Slide

  35. Miiverse
    Redmine

    View Slide

  36. Miiverse
    Redmine
    GitHub Enterprise NEW!

    View Slide

  37. 2 git push
    git push ghe master
    git push origin master
    Merge pull request
    GHE

    View Slide

  38. Git

    View Slide

  39. google/hesokuri
    ghm

    View Slide

  40. hesokuri
    google/hesokuri
    Distributed Git repo synchronization tool.
    Google Git

    View Slide

  41. ghm
    in-house Web Application for mirroring pairs
    managements
    Stores mirroring pairs
    Has Webhook API that invoke mirroring (later)
    Provides some APIs
    @
    GHE 2 GREE Tech Talk

    View Slide

  42. ghm

    View Slide

  43. ghm

    View Slide

  44. 3
    ghm Miiverse

    View Slide

  45. View Slide

  46. JSON over HTTP API

    View Slide

  47. REST API
    Git : /servers(/:id)
    : /servers/:id/promote,
    /servers/:id/demote
    : /servers/:server_id/repos(/:id)
    : /jobs/sync

    View Slide

  48. View Slide

  49. GHE

    View Slide

  50. (1)
    GHE git push

    View Slide

  51. (2)
    GHE Webhook

    View Slide

  52. (3)
    Webhook payload

    View Slide

  53. (4)
    Git Slave DB

    View Slide

  54. (5)
    Git Slave DB

    View Slide

  55. (6)
    GHE git pull

    View Slide

  56. (7)
    git push

    View Slide

  57. GHE
    1. GHE git push
    2. GHE Webhook
    3. payload
    4. Git
    5. Git
    6. GHE git pull
    7. git push

    View Slide

  58. HTTP POST
    github.com GHE
    Git

    View Slide

  59. Git
    Git
    Git
    git push --force

    View Slide

  60. ghm

    View Slide

  61. View Slide

  62. View Slide

  63. Git Pull
    Git Pull 2
    1. commit
    2.

    View Slide

  64. Git Pull
    Miiverse
    Git
    3

    View Slide

  65. Pull

    View Slide

  66. Consul + stretcher
    strecher Pull
    Hashicorp Consul
    http://tech.kayac.com/archive/10_stretcher.html
    https://github.com/fujiwara/stretcher

    View Slide

  67. Consul + stretcher
    1.
    Amazon S3
    2. manifest (YAML)
    Amazon S3
    3. consul event
    4. stretcher agent

    View Slide

  68. Consul + stretcher
    Amazon S3
    Git
    manifest
    rollback manifest

    View Slide

  69. Consul + stretcher
    Jenkins
    Amazon S3

    View Slide

  70. View Slide

  71. 1) GHE git push

    View Slide

  72. 2) GHE webhook Jenkins

    View Slide

  73. 3) Jenkins
    S3 push

    View Slide

  74. 4) Consul master

    View Slide

  75. 5) backend
    S3

    View Slide

  76. Git
    Cap2 & Git
    Consul + stretcher

    View Slide

  77. 100
    AWS
    JP/US/EU
    Miiverse

    View Slide

  78. Cap2 & Git 1,142
    Consul + strecher 29
    strecher 40 !

    View Slide

  79. Consul + strecher
    Cap2 & Git
    Consul + strecher
    deploy:update
    git pull vs S3 DL

    View Slide

  80. Consul + strecher
    carton install
    210
    1/10
    carton install
    Cap2 & Git

    View Slide

  81. 1
    AutoScale
    Consul + strecher

    View Slide

  82. 2
    Miiverse
    Miiverse 3
    Consul
    https://www.consul.io/docs/guides/datacenters.html

    View Slide

  83. Pull
    Consul + stretcher
    Miiverse
    40
    AutoScale

    View Slide

  84. Miiverse
    Miiverse
    Pull
    Git
    Git
    Consul + stretcher

    View Slide

  85. View Slide

  86. View Slide

  87. View Slide