Docker をフル活用したインフラの紹介と成長し続けるためのインフラ戦略 / #abejameetup

Innovation Meetup ~Startup Infrastructure~ の発表資料です
http://abeja-innovation-meetup.connpass.com/event/36707/

Transcript

1. %PDLFS
   Λϑϧ׆༻ͨ͠
 Πϯϑϥͷ঺հ
 ͱ
 ੒௕͠ଓ͚ΔͨΊͷΠϯϑϥઓུ 2016-08-03 Innovation Meetup ~Startup Infrastructure~ Daisuke Fujita (@dtan4)
2. Daisuke Fujita / @dtan4 Engineer, Infrastructure Team
 @ Wantedly, Inc. AWS, Terraform / Terraforming, Developer Productivity
3. https://www.wantedly.com/
4. 3 @ https://www.wantedly.com/sync
5. Docker Λϑϧ׆༻ͨ͠
 Πϯϑϥͷ঺հ ੒௕͠ଓ͚ΔͨΊͷ
 Πϯϑϥઓུ
6. Docker Λϑϧ׆༻ͨ͠ Πϯϑϥͷ঺հ
7. History of Infrastructure 2011/09- Heroku 2014/08- Docker on AWS 2015/05- Terraform 2016/02- CoreOS ૑ۀ
8. 2011/9~2014/08 Heroku CEO ͕ҰਓͰαʔϏε্ཱͪ͛ɺ࠷ॳ͔Β Ruby on Rails αʔϏε։ൃʹूத͢Δ σϓϩΠ଎͍ɺ
 ϫϯΫϦοΫͰॠ࣌ʹεέʔϧ ๛෋ͳ Addons
9. 2014/08~ Docker on AWS ೔ຊ <-> ถࠃ౦ւ؛ ԟ෮ʹΑΔϨΠςϯγΛແ͍ͨ͘͠ Πϯϑϥઐ೚ͷΤϯδχΞ΋ join ͨ͠ ౦ژϦʔδϣϯ͕ଘࡏ͢Δ AWS ΁Ҡߦ Heroku ಉ౳ͷ UX Λఏڙ͢ΔͨΊ Docker Λ࠾༻
10. 2014/08~ Docker on AWS Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ 
 + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI EC2 Πϯελϯε (Ubuntu 14.04) ͱ Docker image ͸
 Chef + Packer ͰϓϩϏδϣχϯά
11. 2015/05~ Terraform Management Console ϙνϙνͯ͠ߏங͢Δͷ͕໘౗ ࡞ۀཤྺΛ࢒͍ͨ͠ 
 Terraform Λར༻ͯ͠ΠϯϑϥߏஙΛίʔυԽ 
 & CI ʹΑΔߏஙࣗಈԽ Terraforming ʹΑΔطଘ AWS, DNSimple ϦιʔεͷίʔυԽ
12. 2015/05~ Terraform
13. 2016/02~ CoreOS ϗετϚγϯͷϝϯςφϯεʹඅ΍͢޻਺͕Ͱ͔͍ Chef Λۦஞ͍ͨ͠ 
 @dtan4 ͷϥοϓτοϓͱϓϩϏδϣχϯά༻ΠϯελϯεͰ͔͠ಈ͔ͳ͔ͬͨ 
 ϗετ OS Λ CoreOS ʹҠߦ 
 + systemd ʹΑΔαʔϏε؅ཧ
14. CoreOS ίϯςφલఏͷܰྔ Linux Distribution ͢΂ͯͷΞϓϦέʔγϣϯΛ
 Docker ίϯςφ্Ͱಈ͔͢ Chrome ͱಉ͡ Update System Update channel: beta
 Reboot strategy: no https://coreos.com/assets/images/media/Host-Diagram.png
15. Infrastructure at Wantedly Web, DB, Storage
 
 DNS AWS
 EC2, ELB, RDS, Elasticache, S3,
 Route53, VPC, Lambda DNSimple
16. Infrastructure at Wantedly Logging App Monitoring Infrastucture Monitoring Incident Management Logentries New Relic Datadog Pagerduty
17. Elasticsearch nginx-image-server Web Web Worker Scheduler nginx-image-server Elasticsearch cluster VPC Nginx
18. systemd journald Docker
 Registry dd-agent Web
 Container EC2 Instance
19. Web (Rails) Docker image ͸2ஈߏ੒ ि1Ͱ intermediate image ΛϏϧυ
 σϓϩΠ࣌ʹ web image ΛϏϧυ bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout <branch> git fetch FROM intermediate web intermediate
20. Web (Rails) ࠩ෼ͷΈͷ fetch + bundle install Ͱ
 σϓϩΠΛߴ଎Խ intermediate ͸ Chef + Packer bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout <branch> git fetch FROM intermediate web intermediate
21. ެࣜ Elasticsearch image ϕʔεͷ Docker image ֤छϓϥάΠϯ + ಠࣗࣙॻʢਓ໊ etc...ʣ --net=host + EC2 discovery ։ൃ؀ڥͰ΋ຊ൪ͱಉ౳ͷ Elasticsearch Λ࢖͑Δ Elasticsearch
22. Nginx + ngx_small_light ը૾ม׵αʔό S3 ʹอ؅ͨ͠ը૾ΛϦαΠζ / ϒϥʔ / WebP ม׵… CloudFront Ͱ഑৴ + Ωϟογϡ nginx-image-server wantedly/nginx-image-server quay.io/wantedly/nginx-image-server Docker Image
 ެ։͍ͯ͠·͢
23. URL ύϥϝʔλͰಈతʹը૾ม׵ nginx-image-server wantedly/nginx-image-server https://.../
 small_light(dw=978,dh=398,da=s,ds=s,cw=978,ch=398)
 /.../image.png quay.io/wantedly/nginx-image-server
24. All services on Docker Facebook ࢓ࣄ૬ஊ bot (Python) จॻϓϨϏϡʔ࡞੒αʔό (Golang) ࣾ಺ Slack bot (Node.js) ...
 ৽نαʔϏε͸ϦϙδτϦʹ Dockerfile Λஔ͍ͯ΋Β͏
25. Private Docker Registry (S3 backed) Λհͨ͠
 Docker image ͷ΍ΓऔΓ Deploy Web Build server $ docker push $ docker pull $ git fetch $ docker build $ docker run
26. ϗετϨϕϧͰͷ Blue-Green Deployment Deploy Web Blue Web Green if (-e /etc/nginx/switch/blue) { proxy_pass http://blue-container; } if (-e /etc/nginx/switch/green) { proxy_pass http://green-container; }
27. Heroku ಉ౳ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI (Capistrano) Deploy $ cap -T cap config # Display the config vars for an app cap deploy[revision] # Deploy an application cap deploy:diff[ref] # Display diff deployed revision and the revision to cap deploy:rollback # Rollback application cap logs:t[container_name] # Display stream logs for an container cap maintenance:on # Enable maintenance mode cap ps # Display web, worker, scheduler, run type containers cap ps:scale[role,num] # Scale out/in containers cap ps:restart:web[color,chained_task?] # Start web containers cap releases # Display deploy history cap run[command] # Run on-off commands cap scheduler # Display scheduling cap scheduler:clear # Clear scheduler config cap scheduler:update # Update scheduler cap status # Show application status
28. SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (Sap) Deploy
29. SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (sap)
 
 CLI ߋ৽࣌ʹ΋֤ࣗͷϚγϯͰΞοϓσʔτͯ͠΋Β͏ඞཁ ͕ͳ͍ ֤࡞ۀͷϩάΛ Docker container ϩάͱͯ͠࢒ͤΔ & Ӿཡ
 Ͱ͖Δ Deploy
30. stdout ʹશϩάΛग़ྗ + journalctl ܦ༝Ͱ Logentries ʹૹ৴ Logging
31. શϗετʹ Datadog Agent ίϯςφΛ഑ஔ͠ɺ
 ϗετͱ Docker ྆ํͷ؂ࢹΛ࣮ࢪ
 
 Monitoring
32. ੒௕͠ଓ͚ΔͨΊͷ Πϯϑϥઓུ
33. https://www.wantedly.com/
34. 3 @ https://www.wantedly.com/sync
35. Wantedly Open API ͸ͨΒ͘Λ໘ന͘ʯΛςʔϚʹ͜Ε·ͰαʔϏε։ൃΛల։͖ͯͨ͠ 8BOUFEMZ͕ͦͷՁ஋Λ8BOUFEMZDPNυϝΠϯҎ֎ʹ΋޿͍͖͛ͯ·͢ɻ ੈͷதͷҰਓͰ΋ଟ͘ͷਓ͕ɺ͸ͨΒ͘Λ໘ന͘ʯײ͡ΒΕΔੈͷதʹͳΓ·͢Α͏ʹɻ Ͱ͖Δ͜ͱ ࣗಈೖྗ ΫοΫύουגࣜձࣾ ΫοΫύουגࣜձࣾ 5݄13೔ʹฐࣾΦϑΟε಺ͷεϖʔεʹͯʮʲୈ17ճʳpotatotips(iOS/Android։ൃTips ڞ༗ձ)ʯ http://connpass.com/event/14143/ Λ։࠵͠·ͨ͠ʂ 18໊ͷൃදऀ͕5෼ؒͷLTʢϥΠτχϯάτʔΫɿΧϯϑΝϨϯεͳͲͰߦΘΕΔ୹͍ ΋ͬͱݟΔ ஑ా
    ୓࢘ ϢʔβʔϑΝʔετਪਐ෦
    ෦௕   ਓ͕ϑΥϩʔத  ਓ͕ϑΥϩʔத ϑΥϩʔ͢Δ ձࣾϑΟʔυϘοΫε ࿩Λฉ͖ʹߦ͘Ϙλϯ ϑΥʔϜࣗಈೖྗϘλϯ Wantedly্ʹ͋ΔʮձࣾϑΟʔυʯ Λɺϒϩά΍ίʔϙϨʔταΠτͳ Ͳɺ޷͖ͳϖʔδʹ؆୯ʹຒΊࠐΉ͜ ͱ͕Ͱ͖·͢ɻखܰͳ࠾༻ϒϥϯσΟ ϯά΍৘ใൃ৴ʹར༻Ͱ͖·͢ɻ ৄ͘͠ ৄ͘͠ ৄ͘͠ ࣗࣾαΠτ౳ͷืूཁ߲ʹɺʮ࿩Λฉ ͖ʹߦ͖͍ͨʯϘλϯΛઃஔ͠ɺ Wantedly্ͷʮ༡ͼʹ͍͘ମݧʯΛࣗ ࣾαΠτͰ΋ల։Ͱ͖·͢ɻ ࠾༻؅ཧγεςϜఏڙاۀ༷౳΁ͷಛ ผఏڙAPIͰ͢ɻީิऀ͕ืूʹΤϯ τϦ͢ΔࡍʹɺWantedly্ͷϓϩϑΟ ʔϧ৘ใΛ׆༻ͯ͠ΤϯτϦ͕Մೳʹ ͳΓ·͢ɻ https://www.wantedly.com/developers
36. WANTEDLY JOURNAL ͸ͨΒ͘Λ໘ന͘͢ΔϝσΟΞ γΰτͰίίϩΦυϧͬͯͳΜͩΖ͏ʁ 8BOUFMEZ͸ɺγΰτͰίίϩΦυϧਓΛ;΍͍ͨ͠ͱߟ͍͑ͯ·͢ɻ Ͱ͸ɺ
    zγΰτͰίίϩΦυϧz
    ͱ͸Ͳ͏͍͏ঢ়ଶͳͷͰ͠ΐ͏ʁ ͓٬͞Μʹʮ͋Γ͕ͱ͏ʯͱݴΘΕͨͱ͖ ໨ඪͷച্Λୡ੒ͨ͠ͱ͖ νʔϜϝϯόʔͷসإΛݟͨͱ͖ʜ ͦΕ͸ɺਓʹΑͬͯҧ͏Ͱ͠ΐ͏͠ɺ ࣌୅΍ࠃʹΑͬͯ΋·ͬͨ͘ҟͳΔͰ͠ΐ͏ɻ ࠓճϦϦʔε͢Δʮ8"/5&%-:
    +063/"-ʯͰ͸ɺ 8BOUFEMZΛ௨ͯ͠స৬ͨ͠ํ΍ɺීஈ͋·Γݟͨ͜ͱ͕ͳ͍اۀ΁ͷ ΠϯλϏϡʔΛ௨ͯ͠ɺzγΰτͰίίϩΦυϧzͱ͸Կͳͷ͔Λ୳ٻ͠·͢ɻ γΰτΛ୳͍ͯ͠Δਓ΋ɺͦ͏Ͱͳ͍ਓʹͱͬͯ΋ɺ ʮ8"/5&%-:
    +063/"-ʯ͕ಇ͖ํͷώϯτʹͳΕ͹޾͍Ͱ͢ɻ https://www.wantedly.com/journals
37. ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏ https://www.wantedly.com/feed
38. ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏ https://www.wantedly.com/tools
39. https://case.wantedly.com/
40. Dreams ✨ ৽͍͠αʔϏεΛͲΜͲΜϦϦʔε͍͖͍ͯͨ͠
 ࣄۀͱͯ͠ͷαʔϏεɺϚΠΫϩαʔϏεΞʔΩςΫνϟ ಛੑʹ߹Θͤͯɺݴޠ΍ΞʔΩςΫνϟΛࣗ༝ʹ
 બ୒Ͱ͖ΔΑ͏ʹ͍ͨ͠ গਓ਺ΛอͪͭͭɺαʔϏεΛεέʔϧ͍ͤͨ͞
41. Problems αʔϏεϦϦʔεͷͨͼʹಉ͡࡞ۀΛ܁Γฦ͢ ߏஙʹ͋ͨͬͯͦΕͳΓͷ޻਺Λཁ͢Δ Wantedly ͷϝΠϯϦϙδτϦ͕ϞϊϦγοΫ
42. Problems ΠϯϑϥνʔϜ͕αʔϏεϦϦʔεͷϘτϧωοΫʹ
 ͳͬͯ͠·͏ ৽αʔϏε΋݁ہಉ͡ίʔυϕʔεʹࡌͤͯ͠·͏
43. Solution 1: Kubernetes Kubernetes ΫϥελΛಋೖ͢Δ Dev ͕ΞϓϦέʔγϣϯϨΠϠΛ؅ཧ͢Δ
 Ops ͕ΞϓϦ͕Քಇ͢ΔΫϥελΛ؅ཧ͢Δ
 ʮԿॲͰԿ͕ಈ͍͍ͯΔʯͰ͸ͳ͘ʮԿ͕ಈ͍͍ͯΔ͔ʯ͚ͩ ϑϧϚωʔδυͰඞཁͳ΋ͷ (ASG, ELB, DNS, Datadog) ͕
 ࡞ΒΕΔศར͞
44. Solution 2: API Wantedly ͷ API Λ࡞ΓɺAPI ܦ༝Ͱ͢΂ͯͷΞΫγϣϯΛߦ͏
45. Solution 2: API React.js ϜʔϒϝϯτʹΑΔ Web ϑϩϯτΤϯυͷ෼཭ Kong Λར༻ͨ͠ೝূ͖ͭ API Gateway apig ʹΑΔϞσϧఆ͔ٛΒͷ 
 REST API αʔόɺυΩϡϝϯτͷੜ੒ webmock-proxy ʹΑΔαʔϏεؒ Integration Test ؆ུԽ wantedly/apig wantedly/webmock-proxy
46. ·ͱΊ
47. Recap Wantedly Ͱ͸ɺ2೥ऑຊ൪؀ڥΛ Docker Ͱӡ༻͍ͯ͠·͢ Ϣʔβͱ։ൃऀ૒ํͷརศੑΛ௥ٻ্ͨ͠Ͱͷબ୒ 
 ͞ΒͳΔ੒௕ʹ޲͚ɺΑΓॊೈͳΠϯϑϥج൫Λߏஙத
48. Recap