Docker をフル活用したインフラの紹介と成長し続けるためのインフラ戦略 / #abejameetup

Innovation Meetup ~Startup Infrastructure~ の発表資料です
http://abeja-innovation-meetup.connpass.com/event/36707/

Transcript

1. %PDLFSΛϑϧ׆༻ͨ͠

   Πϯϑϥͷ঺հ

   ͱ

   ੒௕͠ଓ͚ΔͨΊͷΠϯϑϥઓུ
   2016-08-03
   Innovation Meetup ~Startup Infrastructure~
   Daisuke Fujita (@dtan4)
   

   View Slide

2. Daisuke Fujita / @dtan4
   Engineer, Infrastructure Team

   @ Wantedly, Inc.
   AWS, Terraform / Terraforming, Developer Productivity
   

   View Slide

3. https://www.wantedly.com/
   

   View Slide

4. 3
   @
   https://www.wantedly.com/sync
   

   View Slide

5. Docker Λϑϧ׆༻ͨ͠

   Πϯϑϥͷ঺հ
   ੒௕͠ଓ͚ΔͨΊͷ

   Πϯϑϥઓུ
   

   View Slide

6. Docker Λϑϧ׆༻ͨ͠
   Πϯϑϥͷ঺հ
   

   View Slide

7. History of Infrastructure
   2011/09- Heroku
   
   2014/08- Docker on AWS
   2015/05- Terraform
   2016/02- CoreOS
   ૑ۀ
   

   View Slide

8. 2011/9~2014/08 Heroku
   CEO ͕ҰਓͰαʔϏε্ཱͪ͛ɺ࠷ॳ͔Β Ruby on Rails
   
   αʔϏε։ൃʹूத͢Δ
   σϓϩΠ଎͍ɺ

   ϫϯΫϦοΫͰॠ࣌ʹεέʔϧ
   
   ๛෋ͳ Addons
   

   View Slide

9. 2014/08~ Docker on AWS
   ೔ຊ <-> ถࠃ౦ւ؛ ԟ෮ʹΑΔϨΠςϯγΛແ͍ͨ͘͠
   
   Πϯϑϥઐ೚ͷΤϯδχΞ΋ join ͨ͠
   
   ౦ژϦʔδϣϯ͕ଘࡏ͢Δ AWS ΁Ҡߦ
   
   Heroku ಉ౳ͷ UX Λఏڙ͢ΔͨΊ Docker Λ࠾༻
   

   View Slide

10. 2014/08~ Docker on AWS
    Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ 

    + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI
    
    EC2 Πϯελϯε (Ubuntu 14.04) ͱ Docker image ͸

    Chef + Packer ͰϓϩϏδϣχϯά
    

    View Slide

11. 2015/05~ Terraform
    Management Console ϙνϙνͯ͠ߏங͢Δͷ͕໘౗
    
    ࡞ۀཤྺΛ࢒͍ͨ͠
    
    

    Terraform Λར༻ͯ͠ΠϯϑϥߏஙΛίʔυԽ 

    & CI ʹΑΔߏஙࣗಈԽ
    
    Terraforming ʹΑΔطଘ AWS, DNSimple ϦιʔεͷίʔυԽ
    

    View Slide

12. 2015/05~ Terraform
    

    View Slide

13. 2016/02~ CoreOS
    ϗετϚγϯͷϝϯςφϯεʹඅ΍͢޻਺͕Ͱ͔͍
    
    Chef Λۦஞ͍ͨ͠ 

    @dtan4 ͷϥοϓτοϓͱϓϩϏδϣχϯά༻ΠϯελϯεͰ͔͠ಈ͔ͳ͔ͬͨ
    
    

    ϗετ OS Λ CoreOS ʹҠߦ 

    + systemd ʹΑΔαʔϏε؅ཧ
    

    View Slide

14. CoreOS
    ίϯςφલఏͷܰྔ Linux Distribution
    
    ͢΂ͯͷΞϓϦέʔγϣϯΛ

    Docker ίϯςφ্Ͱಈ͔͢
    
    Chrome ͱಉ͡ Update System
    
    Update channel: beta

    Reboot strategy: no
    https://coreos.com/assets/images/media/Host-Diagram.png
    

    View Slide

15. Infrastructure at Wantedly
    Web, DB, Storage

    

    DNS
    AWS

    EC2, ELB, RDS, Elasticache, S3,

    Route53, VPC, Lambda
    DNSimple
    

    View Slide

16. Infrastructure at Wantedly
    Logging
    App Monitoring
    Infrastucture Monitoring
    Incident Management
    Logentries
    New Relic
    Datadog
    Pagerduty
    

    View Slide

17. Elasticsearch nginx-image-server
    Web
    Web Worker
    Scheduler
    nginx-image-server
    Elasticsearch cluster
    VPC
    Nginx
    

    View Slide

18. systemd
    journald
    Docker

    Registry
    dd-agent
    Web

    Container
    EC2 Instance
    

    View Slide

19. Web (Rails)
    Docker image ͸2ஈߏ੒
    
    ि1Ͱ intermediate image ΛϏϧυ

    σϓϩΠ࣌ʹ web image ΛϏϧυ
    bundle install
    git clone wantedly/wantedly
    Ruby
    FROM Ubuntu 14.04
    rake assets:precompile
    bundle install
    git checkout
    git fetch
    FROM intermediate
    web
    intermediate
    

    View Slide

20. Web (Rails)
    ࠩ෼ͷΈͷ fetch + bundle install Ͱ

    σϓϩΠΛߴ଎Խ
    
    intermediate ͸ Chef + Packer
    bundle install
    git clone wantedly/wantedly
    Ruby
    FROM Ubuntu 14.04
    rake assets:precompile
    bundle install
    git checkout
    git fetch
    FROM intermediate
    web
    intermediate
    

    View Slide

21. ެࣜ Elasticsearch image ϕʔεͷ Docker image
    
    ֤छϓϥάΠϯ + ಠࣗࣙॻʢਓ໊ etc...ʣ
    
    --net=host + EC2 discovery
    
    ։ൃ؀ڥͰ΋ຊ൪ͱಉ౳ͷ Elasticsearch Λ࢖͑Δ
    Elasticsearch
    

    View Slide

22. Nginx + ngx_small_light ը૾ม׵αʔό
    S3 ʹอ؅ͨ͠ը૾ΛϦαΠζ / ϒϥʔ / WebP ม׵…
    
    CloudFront Ͱ഑৴ + Ωϟογϡ
    nginx-image-server
    wantedly/nginx-image-server
    quay.io/wantedly/nginx-image-server
    Docker Image

    ެ։͍ͯ͠·͢
    

    View Slide

23. URL ύϥϝʔλͰಈతʹը૾ม׵
    
    nginx-image-server
    wantedly/nginx-image-server
    https://.../

    small_light(dw=978,dh=398,da=s,ds=s,cw=978,ch=398)

    /.../image.png
    quay.io/wantedly/nginx-image-server
    

    View Slide

24. All services on Docker
    Facebook ࢓ࣄ૬ஊ bot (Python)
    
    จॻϓϨϏϡʔ࡞੒αʔό (Golang)
    
    ࣾ಺ Slack bot (Node.js)
    
    ...

    ৽نαʔϏε͸ϦϙδτϦʹ Dockerfile Λஔ͍ͯ΋Β͏
    

    View Slide

25. Private Docker Registry (S3 backed) Λհͨ͠

    Docker image ͷ΍ΓऔΓ
    Deploy
    Web
    Build server
    $ docker push $ docker pull
    $ git fetch
    $ docker build $ docker run
    

    View Slide

26. ϗετϨϕϧͰͷ Blue-Green Deployment
    Deploy
    Web Blue
    Web Green
    if (-e /etc/nginx/switch/blue) {
    proxy_pass http://blue-container;
    }
    if (-e /etc/nginx/switch/green) {
    proxy_pass http://green-container;
    }
    

    View Slide

27. Heroku ಉ౳ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI (Capistrano)
    Deploy
    $ cap -T
    cap config # Display the config vars for an app
    cap deploy[revision] # Deploy an application
    cap deploy:diff[ref] # Display diff deployed revision and the revision to
    cap deploy:rollback # Rollback application
    cap logs:t[container_name] # Display stream logs for an container
    cap maintenance:on # Enable maintenance mode
    cap ps # Display web, worker, scheduler, run type containers
    cap ps:scale[role,num] # Scale out/in containers
    cap ps:restart:web[color,chained_task?] # Start web containers
    cap releases # Display deploy history
    cap run[command] # Run on-off commands
    cap scheduler # Display scheduling
    cap scheduler:clear # Clear scheduler config
    cap scheduler:update # Update scheduler
    cap status # Show application status
    

    View Slide

28. SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (Sap)
    Deploy
    

    View Slide

29. SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (sap)

    

    CLI ߋ৽࣌ʹ΋֤ࣗͷϚγϯͰΞοϓσʔτͯ͠΋Β͏ඞཁ
    ͕ͳ͍
    
    ֤࡞ۀͷϩάΛ Docker container ϩάͱͯ͠࢒ͤΔ & Ӿཡ

    Ͱ͖Δ
    Deploy
    

    View Slide

30. stdout ʹશϩάΛग़ྗ + journalctl ܦ༝Ͱ Logentries ʹૹ৴
    Logging
    

    View Slide

31. શϗετʹ Datadog Agent ίϯςφΛ഑ஔ͠ɺ

    ϗετͱ Docker ྆ํͷ؂ࢹΛ࣮ࢪ

    

    Monitoring
    

    View Slide

32. ੒௕͠ଓ͚ΔͨΊͷ
    Πϯϑϥઓུ
    

    View Slide

33. https://www.wantedly.com/
    

    View Slide

34. 3
    @
    https://www.wantedly.com/sync
    

    View Slide

35. Wantedly Open API
    ͸ͨΒ͘Λ໘ന͘ʯΛςʔϚʹ͜Ε·ͰαʔϏε։ൃΛల։͖ͯͨ͠
    8BOUFEMZ͕ͦͷՁ஋Λ8BOUFEMZDPNυϝΠϯҎ֎ʹ΋޿͍͖͛ͯ·͢ɻ
    ੈͷதͷҰਓͰ΋ଟ͘ͷਓ͕ɺ͸ͨΒ͘Λ໘ന͘ʯײ͡ΒΕΔੈͷதʹͳΓ·͢Α͏ʹɻ
    Ͱ͖Δ͜ͱ
    ࣗಈೖྗ
    ΫοΫύουגࣜձࣾ ΫοΫύουגࣜձࣾ
    5݄13೔ʹฐࣾΦϑΟε಺ͷεϖʔεʹͯʮʲୈ17ճʳpotatotips(iOS/Android։ൃTips
    ڞ༗ձ)ʯ http://connpass.com/event/14143/ Λ։࠵͠·ͨ͠ʂ
    18໊ͷൃදऀ͕5෼ؒͷLTʢϥΠτχϯάτʔΫɿΧϯϑΝϨϯεͳͲͰߦΘΕΔ୹͍
    ΋ͬͱݟΔ
    ஑ా୓࢘
    ϢʔβʔϑΝʔετਪਐ෦෦௕
    
    ਓ͕ϑΥϩʔத ਓ͕ϑΥϩʔத
    ϑΥϩʔ͢Δ
    ձࣾϑΟʔυϘοΫε ࿩Λฉ͖ʹߦ͘Ϙλϯ ϑΥʔϜࣗಈೖྗϘλϯ
    Wantedly্ʹ͋ΔʮձࣾϑΟʔυʯ
    Λɺϒϩά΍ίʔϙϨʔταΠτͳ
    Ͳɺ޷͖ͳϖʔδʹ؆୯ʹຒΊࠐΉ͜
    ͱ͕Ͱ͖·͢ɻखܰͳ࠾༻ϒϥϯσΟ
    ϯά΍৘ใൃ৴ʹར༻Ͱ͖·͢ɻ
    ৄ͘͠ ৄ͘͠ ৄ͘͠
    ࣗࣾαΠτ౳ͷืूཁ߲ʹɺʮ࿩Λฉ
    ͖ʹߦ͖͍ͨʯϘλϯΛઃஔ͠ɺ
    Wantedly্ͷʮ༡ͼʹ͍͘ମݧʯΛࣗ
    ࣾαΠτͰ΋ల։Ͱ͖·͢ɻ
    ࠾༻؅ཧγεςϜఏڙاۀ༷౳΁ͷಛ
    ผఏڙAPIͰ͢ɻީิऀ͕ืूʹΤϯ
    τϦ͢ΔࡍʹɺWantedly্ͷϓϩϑΟ
    ʔϧ৘ใΛ׆༻ͯ͠ΤϯτϦ͕Մೳʹ
    ͳΓ·͢ɻ
    https://www.wantedly.com/developers
    

    View Slide

36. WANTEDLY JOURNAL
    ͸ͨΒ͘Λ໘ന͘͢ΔϝσΟΞ
    γΰτͰίίϩΦυϧͬͯͳΜͩΖ͏ʁ
    8BOUFMEZ͸ɺγΰτͰίίϩΦυϧਓΛ;΍͍ͨ͠ͱߟ͍͑ͯ·͢ɻ
    Ͱ͸ɺzγΰτͰίίϩΦυϧzͱ͸Ͳ͏͍͏ঢ়ଶͳͷͰ͠ΐ͏ʁ
    ͓٬͞Μʹʮ͋Γ͕ͱ͏ʯͱݴΘΕͨͱ͖
    ໨ඪͷച্Λୡ੒ͨ͠ͱ͖
    νʔϜϝϯόʔͷসإΛݟͨͱ͖ʜ
    ͦΕ͸ɺਓʹΑͬͯҧ͏Ͱ͠ΐ͏͠ɺ
    ࣌୅΍ࠃʹΑͬͯ΋·ͬͨ͘ҟͳΔͰ͠ΐ͏ɻ
    ࠓճϦϦʔε͢Δʮ8"/5&%-:+063/"-ʯͰ͸ɺ
    8BOUFEMZΛ௨ͯ͠స৬ͨ͠ํ΍ɺීஈ͋·Γݟͨ͜ͱ͕ͳ͍اۀ΁ͷ
    ΠϯλϏϡʔΛ௨ͯ͠ɺzγΰτͰίίϩΦυϧzͱ͸Կͳͷ͔Λ୳ٻ͠·͢ɻ
    γΰτΛ୳͍ͯ͠Δਓ΋ɺͦ͏Ͱͳ͍ਓʹͱͬͯ΋ɺ
    ʮ8"/5&%-:+063/"-ʯ͕ಇ͖ํͷώϯτʹͳΕ͹޾͍Ͱ͢ɻ
    https://www.wantedly.com/journals
    

    View Slide

37. ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏
    https://www.wantedly.com/feed
    

    View Slide

38. ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏
    https://www.wantedly.com/tools
    

    View Slide

39. https://case.wantedly.com/
    

    View Slide

40. Dreams ✨
    ৽͍͠αʔϏεΛͲΜͲΜϦϦʔε͍͖͍ͯͨ͠

    ࣄۀͱͯ͠ͷαʔϏεɺϚΠΫϩαʔϏεΞʔΩςΫνϟ
    
    ಛੑʹ߹Θͤͯɺݴޠ΍ΞʔΩςΫνϟΛࣗ༝ʹ

    બ୒Ͱ͖ΔΑ͏ʹ͍ͨ͠
    
    গਓ਺ΛอͪͭͭɺαʔϏεΛεέʔϧ͍ͤͨ͞
    

    View Slide

41. Problems
    αʔϏεϦϦʔεͷͨͼʹಉ͡࡞ۀΛ܁Γฦ͢
    
    ߏஙʹ͋ͨͬͯͦΕͳΓͷ޻਺Λཁ͢Δ
    
    Wantedly ͷϝΠϯϦϙδτϦ͕ϞϊϦγοΫ
    

    View Slide

42. Problems
    ΠϯϑϥνʔϜ͕αʔϏεϦϦʔεͷϘτϧωοΫʹ

    ͳͬͯ͠·͏
    
    ৽αʔϏε΋݁ہಉ͡ίʔυϕʔεʹࡌͤͯ͠·͏
    

    View Slide

43. Solution 1: Kubernetes
    Kubernetes ΫϥελΛಋೖ͢Δ
    
    Dev ͕ΞϓϦέʔγϣϯϨΠϠΛ؅ཧ͢Δ

    Ops ͕ΞϓϦ͕Քಇ͢ΔΫϥελΛ؅ཧ͢Δ

    ʮԿॲͰԿ͕ಈ͍͍ͯΔʯͰ͸ͳ͘ʮԿ͕ಈ͍͍ͯΔ͔ʯ͚ͩ
    
    ϑϧϚωʔδυͰඞཁͳ΋ͷ (ASG, ELB, DNS, Datadog) ͕

    ࡞ΒΕΔศར͞
    

    View Slide

44. Solution 2: API
    Wantedly ͷ API Λ࡞ΓɺAPI ܦ༝Ͱ͢΂ͯͷΞΫγϣϯΛߦ͏
    
    

    View Slide

45. Solution 2: API
    React.js ϜʔϒϝϯτʹΑΔ Web ϑϩϯτΤϯυͷ෼཭
    
    Kong Λར༻ͨ͠ೝূ͖ͭ API Gateway
    
    apig ʹΑΔϞσϧఆ͔ٛΒͷ 

    REST API αʔόɺυΩϡϝϯτͷੜ੒
    
    webmock-proxy ʹΑΔαʔϏεؒ Integration Test ؆ུԽ
    wantedly/apig
    wantedly/webmock-proxy
    

    View Slide

46. ·ͱΊ
    

    View Slide

47. Recap
    Wantedly Ͱ͸ɺ2೥ऑຊ൪؀ڥΛ Docker Ͱӡ༻͍ͯ͠·͢
    
    Ϣʔβͱ։ൃऀ૒ํͷརศੑΛ௥ٻ্ͨ͠Ͱͷબ୒
    
    

    ͞ΒͳΔ੒௕ʹ޲͚ɺΑΓॊೈͳΠϯϑϥج൫Λߏஙத
    

    View Slide

48. Recap
    

    View Slide

49. View Slide