Docker をフル活用したインフラの紹介と成長し続けるためのインフラ戦略 / #abejameetup

Slide 1

Slide 1 text

%PDLFSΛϑϧ׆༻ͨ͠  Πϯϑϥͷ঺հ  ͱ  ੒௕͠ଓ͚ΔͨΊͷΠϯϑϥઓུ 2016-08-03 Innovation Meetup ~Startup Infrastructure~ Daisuke Fujita (@dtan4)

Slide 2

Slide 2 text

Daisuke Fujita / @dtan4 Engineer, Infrastructure Team  @ Wantedly, Inc. AWS, Terraform / Terraforming, Developer Productivity

Slide 3

Slide 3 text

https://www.wantedly.com/

Slide 4

Slide 4 text

3 @ https://www.wantedly.com/sync

Slide 5

Slide 5 text

Docker Λϑϧ׆༻ͨ͠  Πϯϑϥͷ঺հ ੒௕͠ଓ͚ΔͨΊͷ  Πϯϑϥઓུ

Slide 6

Slide 6 text

Docker Λϑϧ׆༻ͨ͠ Πϯϑϥͷ঺հ

Slide 7

Slide 7 text

History of Infrastructure 2011/09- Heroku 2014/08- Docker on AWS 2015/05- Terraform 2016/02- CoreOS ૑ۀ

Slide 8

Slide 8 text

2011/9~2014/08 Heroku CEO ͕ҰਓͰαʔϏε্ཱͪ͛ɺ࠷ॳ͔Β Ruby on Rails αʔϏε։ൃʹूத͢Δ σϓϩΠ଎͍ɺ  ϫϯΫϦοΫͰॠ࣌ʹεέʔϧ ๛෋ͳ Addons

Slide 9

Slide 9 text

2014/08~ Docker on AWS ೔ຊ <-> ถࠃ౦ւ؛ ԟ෮ʹΑΔϨΠςϯγΛແ͍ͨ͘͠ Πϯϑϥઐ೚ͷΤϯδχΞ΋ join ͨ͠ ౦ژϦʔδϣϯ͕ଘࡏ͢Δ AWS ΁Ҡߦ Heroku ಉ౳ͷ UX Λఏڙ͢ΔͨΊ Docker Λ࠾༻

Slide 10

Slide 10 text

2014/08~ Docker on AWS Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ   + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI EC2 Πϯελϯε (Ubuntu 14.04) ͱ Docker image ͸  Chef + Packer ͰϓϩϏδϣχϯά

Slide 11

Slide 11 text

2015/05~ Terraform Management Console ϙνϙνͯ͠ߏங͢Δͷ͕໘౗ ࡞ۀཤྺΛ࢒͍ͨ͠   Terraform Λར༻ͯ͠ΠϯϑϥߏஙΛίʔυԽ   & CI ʹΑΔߏஙࣗಈԽ Terraforming ʹΑΔطଘ AWS, DNSimple ϦιʔεͷίʔυԽ

Slide 12

Slide 12 text

2015/05~ Terraform

Slide 13

Slide 13 text

2016/02~ CoreOS ϗετϚγϯͷϝϯςφϯεʹඅ΍͢޻਺͕Ͱ͔͍ Chef Λۦஞ͍ͨ͠   @dtan4 ͷϥοϓτοϓͱϓϩϏδϣχϯά༻ΠϯελϯεͰ͔͠ಈ͔ͳ͔ͬͨ   ϗετ OS Λ CoreOS ʹҠߦ   + systemd ʹΑΔαʔϏε؅ཧ

Slide 14

Slide 14 text

CoreOS ίϯςφલఏͷܰྔ Linux Distribution ͢΂ͯͷΞϓϦέʔγϣϯΛ  Docker ίϯςφ্Ͱಈ͔͢ Chrome ͱಉ͡ Update System Update channel: beta  Reboot strategy: no https://coreos.com/assets/images/media/Host-Diagram.png

Slide 15

Slide 15 text

Infrastructure at Wantedly Web, DB, Storage    DNS AWS  EC2, ELB, RDS, Elasticache, S3,  Route53, VPC, Lambda DNSimple

Slide 16

Slide 16 text

Infrastructure at Wantedly Logging App Monitoring Infrastucture Monitoring Incident Management Logentries New Relic Datadog Pagerduty

Slide 17

Slide 17 text

Elasticsearch nginx-image-server Web Web Worker Scheduler nginx-image-server Elasticsearch cluster VPC Nginx

Slide 18

Slide 18 text

systemd journald Docker  Registry dd-agent Web  Container EC2 Instance

Slide 19

Slide 19 text

Web (Rails) Docker image ͸2ஈߏ੒ ि1Ͱ intermediate image ΛϏϧυ  σϓϩΠ࣌ʹ web image ΛϏϧυ bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout git fetch FROM intermediate web intermediate

Slide 20

Slide 20 text

Web (Rails) ࠩ෼ͷΈͷ fetch + bundle install Ͱ  σϓϩΠΛߴ଎Խ intermediate ͸ Chef + Packer bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout git fetch FROM intermediate web intermediate

Slide 21

Slide 21 text

ެࣜ Elasticsearch image ϕʔεͷ Docker image ֤छϓϥάΠϯ + ಠࣗࣙॻʢਓ໊ etc...ʣ --net=host + EC2 discovery ։ൃ؀ڥͰ΋ຊ൪ͱಉ౳ͷ Elasticsearch Λ࢖͑Δ Elasticsearch

Slide 22

Slide 22 text

Nginx + ngx_small_light ը૾ม׵αʔό S3 ʹอ؅ͨ͠ը૾ΛϦαΠζ / ϒϥʔ / WebP ม׵… CloudFront Ͱ഑৴ + Ωϟογϡ nginx-image-server wantedly/nginx-image-server quay.io/wantedly/nginx-image-server Docker Image  ެ։͍ͯ͠·͢

Slide 23

Slide 23 text

URL ύϥϝʔλͰಈతʹը૾ม׵ nginx-image-server wantedly/nginx-image-server https://.../  small_light(dw=978,dh=398,da=s,ds=s,cw=978,ch=398)  /.../image.png quay.io/wantedly/nginx-image-server

Slide 24

Slide 24 text

All services on Docker Facebook ࢓ࣄ૬ஊ bot (Python) จॻϓϨϏϡʔ࡞੒αʔό (Golang) ࣾ಺ Slack bot (Node.js) ...  ৽نαʔϏε͸ϦϙδτϦʹ Dockerﬁle Λஔ͍ͯ΋Β͏

Slide 25

Slide 25 text

Private Docker Registry (S3 backed) Λհͨ͠  Docker image ͷ΍ΓऔΓ Deploy Web Build server $ docker push $ docker pull $ git fetch $ docker build $ docker run

Slide 26

Slide 26 text

ϗετϨϕϧͰͷ Blue-Green Deployment Deploy Web Blue Web Green if (-e /etc/nginx/switch/blue) { proxy_pass http://blue-container; } if (-e /etc/nginx/switch/green) { proxy_pass http://green-container; }

Slide 27

Slide 27 text

Heroku ಉ౳ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI (Capistrano) Deploy $ cap -T cap config # Display the config vars for an app cap deploy[revision] # Deploy an application cap deploy:diff[ref] # Display diff deployed revision and the revision to cap deploy:rollback # Rollback application cap logs:t[container_name] # Display stream logs for an container cap maintenance:on # Enable maintenance mode cap ps # Display web, worker, scheduler, run type containers cap ps:scale[role,num] # Scale out/in containers cap ps:restart:web[color,chained_task?] # Start web containers cap releases # Display deploy history cap run[command] # Run on-off commands cap scheduler # Display scheduling cap scheduler:clear # Clear scheduler config cap scheduler:update # Update scheduler cap status # Show application status

Slide 28

Slide 28 text

SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (Sap) Deploy

Slide 29

Slide 29 text

SSH + Docker ίϯςφܦ༝Ͱ Deploy ࣮ߦ (sap)    CLI ߋ৽࣌ʹ΋֤ࣗͷϚγϯͰΞοϓσʔτͯ͠΋Β͏ඞཁ ͕ͳ͍ ֤࡞ۀͷϩάΛ Docker container ϩάͱͯ͠࢒ͤΔ & Ӿཡ  Ͱ͖Δ Deploy

Slide 30

Slide 30 text

stdout ʹશϩάΛग़ྗ + journalctl ܦ༝Ͱ Logentries ʹૹ৴ Logging

Slide 31

Slide 31 text

શϗετʹ Datadog Agent ίϯςφΛ഑ஔ͠ɺ  ϗετͱ Docker ྆ํͷ؂ࢹΛ࣮ࢪ    Monitoring

Slide 32

Slide 32 text

੒௕͠ଓ͚ΔͨΊͷ Πϯϑϥઓུ

Slide 33

Slide 33 text

https://www.wantedly.com/

Slide 34

Slide 34 text

3 @ https://www.wantedly.com/sync

Slide 35

Slide 35 text

Wantedly Open API ͸ͨΒ͘Λ໘ന͘ʯΛςʔϚʹ͜Ε·ͰαʔϏε։ൃΛల։͖ͯͨ͠ 8BOUFEMZ͕ͦͷՁ஋Λ8BOUFEMZDPNυϝΠϯҎ֎ʹ΋޿͍͖͛ͯ·͢ɻ ੈͷதͷҰਓͰ΋ଟ͘ͷਓ͕ɺ͸ͨΒ͘Λ໘ന͘ʯײ͡ΒΕΔੈͷதʹͳΓ·͢Α͏ʹɻ Ͱ͖Δ͜ͱ ࣗಈೖྗ ΫοΫύουגࣜձࣾ ΫοΫύουגࣜձࣾ 5݄13೔ʹฐࣾΦϑΟε಺ͷεϖʔεʹͯʮʲୈ17ճʳpotatotips(iOS/Android։ൃTips ڞ༗ձ)ʯ http://connpass.com/event/14143/ Λ։࠵͠·ͨ͠ʂ 18໊ͷൃදऀ͕5෼ؒͷLTʢϥΠτχϯάτʔΫɿΧϯϑΝϨϯεͳͲͰߦΘΕΔ୹͍ ΋ͬͱݟΔ ஑ా୓࢘ ϢʔβʔϑΝʔετਪਐ෦෦௕ ਓ͕ϑΥϩʔத ਓ͕ϑΥϩʔத ϑΥϩʔ͢Δ ձࣾϑΟʔυϘοΫε ࿩Λฉ͖ʹߦ͘Ϙλϯ ϑΥʔϜࣗಈೖྗϘλϯ Wantedly্ʹ͋ΔʮձࣾϑΟʔυʯ Λɺϒϩά΍ίʔϙϨʔταΠτͳ Ͳɺ޷͖ͳϖʔδʹ؆୯ʹຒΊࠐΉ͜ ͱ͕Ͱ͖·͢ɻखܰͳ࠾༻ϒϥϯσΟ ϯά΍৘ใൃ৴ʹར༻Ͱ͖·͢ɻ ৄ͘͠ ৄ͘͠ ৄ͘͠ ࣗࣾαΠτ౳ͷืूཁ߲ʹɺʮ࿩Λฉ ͖ʹߦ͖͍ͨʯϘλϯΛઃஔ͠ɺ Wantedly্ͷʮ༡ͼʹ͍͘ମݧʯΛࣗ ࣾαΠτͰ΋ల։Ͱ͖·͢ɻ ࠾༻؅ཧγεςϜఏڙاۀ༷౳΁ͷಛ ผఏڙAPIͰ͢ɻީิऀ͕ืूʹΤϯ τϦ͢ΔࡍʹɺWantedly্ͷϓϩϑΟ ʔϧ৘ใΛ׆༻ͯ͠ΤϯτϦ͕Մೳʹ ͳΓ·͢ɻ https://www.wantedly.com/developers

Slide 36

Slide 36 text

WANTEDLY JOURNAL ͸ͨΒ͘Λ໘ന͘͢ΔϝσΟΞ γΰτͰίίϩΦυϧͬͯͳΜͩΖ͏ʁ 8BOUFMEZ͸ɺγΰτͰίίϩΦυϧਓΛ;΍͍ͨ͠ͱߟ͍͑ͯ·͢ɻ Ͱ͸ɺzγΰτͰίίϩΦυϧzͱ͸Ͳ͏͍͏ঢ়ଶͳͷͰ͠ΐ͏ʁ ͓٬͞Μʹʮ͋Γ͕ͱ͏ʯͱݴΘΕͨͱ͖ ໨ඪͷച্Λୡ੒ͨ͠ͱ͖ νʔϜϝϯόʔͷসإΛݟͨͱ͖ʜ ͦΕ͸ɺਓʹΑͬͯҧ͏Ͱ͠ΐ͏͠ɺ ࣌୅΍ࠃʹΑͬͯ΋·ͬͨ͘ҟͳΔͰ͠ΐ͏ɻ ࠓճϦϦʔε͢Δʮ8"/5&%-:+063/"-ʯͰ͸ɺ 8BOUFEMZΛ௨ͯ͠స৬ͨ͠ํ΍ɺීஈ͋·Γݟͨ͜ͱ͕ͳ͍اۀ΁ͷ ΠϯλϏϡʔΛ௨ͯ͠ɺzγΰτͰίίϩΦυϧzͱ͸Կͳͷ͔Λ୳ٻ͠·͢ɻ γΰτΛ୳͍ͯ͠Δਓ΋ɺͦ͏Ͱͳ͍ਓʹͱͬͯ΋ɺ ʮ8"/5&%-:+063/"-ʯ͕ಇ͖ํͷώϯτʹͳΕ͹޾͍Ͱ͢ɻ https://www.wantedly.com/journals

Slide 37

Slide 37 text

ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏ https://www.wantedly.com/feed

Slide 38

Slide 38 text

ੈքʹࣗ෼ͷ੠Λൃ৴͠Α͏ https://www.wantedly.com/tools

Slide 39

Slide 39 text

https://case.wantedly.com/

Slide 40

Slide 40 text

Dreams ✨ ৽͍͠αʔϏεΛͲΜͲΜϦϦʔε͍͖͍ͯͨ͠  ࣄۀͱͯ͠ͷαʔϏεɺϚΠΫϩαʔϏεΞʔΩςΫνϟ ಛੑʹ߹Θͤͯɺݴޠ΍ΞʔΩςΫνϟΛࣗ༝ʹ  બ୒Ͱ͖ΔΑ͏ʹ͍ͨ͠ গਓ਺ΛอͪͭͭɺαʔϏεΛεέʔϧ͍ͤͨ͞

Slide 41

Slide 41 text

Problems αʔϏεϦϦʔεͷͨͼʹಉ͡࡞ۀΛ܁Γฦ͢ ߏஙʹ͋ͨͬͯͦΕͳΓͷ޻਺Λཁ͢Δ Wantedly ͷϝΠϯϦϙδτϦ͕ϞϊϦγοΫ

Slide 42

Slide 42 text

Problems ΠϯϑϥνʔϜ͕αʔϏεϦϦʔεͷϘτϧωοΫʹ  ͳͬͯ͠·͏ ৽αʔϏε΋݁ہಉ͡ίʔυϕʔεʹࡌͤͯ͠·͏

Slide 43

Slide 43 text

Solution 1: Kubernetes Kubernetes ΫϥελΛಋೖ͢Δ Dev ͕ΞϓϦέʔγϣϯϨΠϠΛ؅ཧ͢Δ  Ops ͕ΞϓϦ͕Քಇ͢ΔΫϥελΛ؅ཧ͢Δ  ʮԿॲͰԿ͕ಈ͍͍ͯΔʯͰ͸ͳ͘ʮԿ͕ಈ͍͍ͯΔ͔ʯ͚ͩ ϑϧϚωʔδυͰඞཁͳ΋ͷ (ASG, ELB, DNS, Datadog) ͕  ࡞ΒΕΔศར͞

Slide 44

Slide 44 text

Solution 2: API Wantedly ͷ API Λ࡞ΓɺAPI ܦ༝Ͱ͢΂ͯͷΞΫγϣϯΛߦ͏

Slide 45

Slide 45 text

Solution 2: API React.js ϜʔϒϝϯτʹΑΔ Web ϑϩϯτΤϯυͷ෼཭ Kong Λར༻ͨ͠ೝূ͖ͭ API Gateway apig ʹΑΔϞσϧఆ͔ٛΒͷ   REST API αʔόɺυΩϡϝϯτͷੜ੒ webmock-proxy ʹΑΔαʔϏεؒ Integration Test ؆ུԽ wantedly/apig wantedly/webmock-proxy

Slide 46

Slide 46 text

·ͱΊ

Slide 47

Slide 47 text

Recap Wantedly Ͱ͸ɺ2೥ऑຊ൪؀ڥΛ Docker Ͱӡ༻͍ͯ͠·͢ Ϣʔβͱ։ൃऀ૒ํͷརศੑΛ௥ٻ্ͨ͠Ͱͷબ୒   ͞ΒͳΔ੒௕ʹ޲͚ɺΑΓॊೈͳΠϯϑϥج൫Λߏஙத