世界展開する大規模ウェブサービスのデプロイを支える技術 / YAPC::Asia Tokyo 2015

Miiverse とは任天堂株式会社が運営しているウェブサービスであり、世界中の Wii U やニンテンドー3DS、そして PC やスマートデバイスから利用することができます。

AWS 上でマルチリージョン構成をとり大量のサーバを抱える Miiverse のデプロイを支える技術と運用上の工夫、そして株式会社はてなと任天堂株式会社が共同で開発する Git リポジトリの同期システムの構築を通して得られた経験をもとに、大規模なウェブサービスを素早くかつ安全に改善する方法を紹介します。

※本資料は YAPC::Asia Tokyo 2015 での発表資料となります。
http://yapcasia.org/2015/talk/show/9ec2791c-05e5-11e5-81fa-79c97d574c3a

Transcript

1. View Slide

2. View Slide

3. Miiverse
   

   View Slide

4. 3DS/Wii U
   Miiverse
   

   View Slide

5. Miiverse
   

   View Slide

6. Miiverse
   Miiverse
   Git
   

   View Slide

7. Miiverse
   

   View Slide

8. Wii U 3DS
   Mii
   

   View Slide

9. View Slide

10. View Slide

11. Web
    

    View Slide

12. Miiverse
    API /
    

    View Slide

13. View Slide

14. 3
    Miiverse Amazon Web Services (AWS)
    JP/US/EU 3
    

    View Slide

15. View Slide

16. REST API
    

    View Slide

17. JAWS UG
    Developers Summit
    http://developer.hatenastaff.com/entry/2014/03/25/1
    85521
    https://speakerdeck.com/hatena/devsumi-2015-
    miiverse
    

    View Slide

18. Miiverse
    

    View Slide

19. Miiverse
    

    View Slide

20. Capistrano2 & Git
    Git
    Git
    git pull
    Pull
    

    View Slide

21. Capistrano2 & Git
    git pull
    Miiverse 3
    git pull
    Git git pull
    

    View Slide

22. Git
    Git Slave
    Miiverse Git Slave
    Git Slave git pull
    

    View Slide

23. Git
    Git Slave
    Git Slave 10 1 lsyncd inode
    rsync
    git push
    lsyncd
    rsync
    

    View Slide

24. Git
    Git
    Git Slave
    Wii U 3DS etc...
    Git Slave
    sleep `expr $RANDOM % #{$main.sleep_divisor}` && git ...
    

    View Slide

25. Git
    lsyncd
    JS CSS Git
    rsync git fetch
    

    View Slide

26. Git
    Git Slave
    Git
    

    View Slide

27. tips
    

    View Slide

28. tips
    Capistrano
    role
    Mackerel API
    net-ssh
    

    View Slide

29. tips
    Mackerel
    https://mackerel.io/
    

    View Slide

30. tips
    Mackerel
    role Capistrano
    

    View Slide

31. tips
    Mackerel
    Mackerel
    AutoScale
    Mackerel /
    cron kick
    EC2
    

    View Slide

32. tips
    DB
    

    View Slide

33. Miiverse
    Miiverse Pull
    Cap2 Git
    Git
    Git Slave
    Git Slave
    

    View Slide

34. Git
    

    View Slide

35. Miiverse
    Redmine
    

    View Slide

36. Miiverse
    Redmine
    GitHub Enterprise NEW!
    

    View Slide

37. 2 git push
    git push ghe master
    git push origin master
    Merge pull request
    GHE
    

    View Slide

38. Git
    

    View Slide

39. google/hesokuri
    ghm
    

    View Slide

40. hesokuri
    google/hesokuri
    Distributed Git repo synchronization tool.
    Google Git
    

    View Slide

41. ghm
    in-house Web Application for mirroring pairs
    managements
    Stores mirroring pairs
    Has Webhook API that invoke mirroring (later)
    Provides some APIs
    @
    GHE 2 GREE Tech Talk
    

    View Slide

42. ghm
    

    View Slide

43. ghm
    

    View Slide

44. 3
    ghm Miiverse
    

    View Slide

45. View Slide

46. JSON over HTTP API
    

    View Slide

47. REST API
    Git : /servers(/:id)
    : /servers/:id/promote,
    /servers/:id/demote
    : /servers/:server_id/repos(/:id)
    : /jobs/sync
    

    View Slide

48. View Slide

49. GHE
    

    View Slide

50. (1)
    GHE git push
    

    View Slide

51. (2)
    GHE Webhook
    

    View Slide

52. (3)
    Webhook payload
    

    View Slide

53. (4)
    Git Slave DB
    

    View Slide

54. (5)
    Git Slave DB
    

    View Slide

55. (6)
    GHE git pull
    

    View Slide

56. (7)
    git push
    

    View Slide

57. GHE
    1. GHE git push
    2. GHE Webhook
    3. payload
    4. Git
    5. Git
    6. GHE git pull
    7. git push
    

    View Slide

58. HTTP POST
    github.com GHE
    Git
    

    View Slide

59. Git
    Git
    Git
    git push --force
    

    View Slide

60. ghm
    

    View Slide

61. View Slide

62. View Slide

63. Git Pull
    Git Pull 2
    1. commit
    2.
    

    View Slide

64. Git Pull
    Miiverse
    Git
    3
    

    View Slide

65. Pull
    

    View Slide

66. Consul + stretcher
    strecher Pull
    Hashicorp Consul
    http://tech.kayac.com/archive/10_stretcher.html
    https://github.com/fujiwara/stretcher
    

    View Slide

67. Consul + stretcher
    1.
    Amazon S3
    2. manifest (YAML)
    Amazon S3
    3. consul event
    4. stretcher agent
    

    View Slide

68. Consul + stretcher
    Amazon S3
    Git
    manifest
    rollback manifest
    

    View Slide

69. Consul + stretcher
    Jenkins
    Amazon S3
    

    View Slide

70. View Slide

71. 1) GHE git push
    

    View Slide

72. 2) GHE webhook Jenkins
    

    View Slide

73. 3) Jenkins
    S3 push
    

    View Slide

74. 4) Consul master
    

    View Slide

75. 5) backend
    S3
    

    View Slide

76. Git
    Cap2 & Git
    Consul + stretcher
    

    View Slide

77. 100
    AWS
    JP/US/EU
    Miiverse
    

    View Slide

78. Cap2 & Git 1,142
    Consul + strecher 29
    strecher 40 !
    

    View Slide

79. Consul + strecher
    Cap2 & Git
    Consul + strecher
    deploy:update
    git pull vs S3 DL
    

    View Slide

80. Consul + strecher
    carton install
    210
    1/10
    carton install
    Cap2 & Git
    

    View Slide

81. 1
    AutoScale
    Consul + strecher
    

    View Slide

82. 2
    Miiverse
    Miiverse 3
    Consul
    https://www.consul.io/docs/guides/datacenters.html
    

    View Slide

83. Pull
    Consul + stretcher
    Miiverse
    40
    AutoScale
    

    View Slide

84. Miiverse
    Miiverse
    Pull
    Git
    Git
    Consul + stretcher
    

    View Slide

85. View Slide

86. View Slide

87. View Slide