Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ruby on Lambdaで変わる大規模サービスの裏側 / Ruby on Lambda makes a change to Eight's backend

Sansan
March 27, 2019

Ruby on Lambdaで変わる大規模サービスの裏側 / Ruby on Lambda makes a change to Eight's backend

AWS Serverless Tech/事例セミナー
- サーバーレスで Ruby 他、いろんな言語が使えるよ
https://pages.awscloud.com/ServerlessSeminar20190327-jp.html

■登壇概要
タイトル:Ruby on Lambdaで変わる大規模サービスの裏側

登壇者:Eight事業部 エンジニアリングマネージャー 藤井洋太郎

▼Sansan Builders Box
https://buildersbox.corp-sansan.com/

Sansan

March 27, 2019
Tweet

More Decks by Sansan

Other Decks in Technology

Transcript

  1. 1ຕͷ໊͔ࢗΒ࡞ΒΕΔαϜωΠϧɺ΅͔͠ը૾ 1ຕͷ໊ࢗऔΓࠐΈ αΠζ - ද×ཪ - jpeg × webp Λߟ͑Δͱ20छఔ౓ඞཁʹ

    ΦϦδφϧ(ද) ΦϦδφϧ(ཪ) αϜωΠϧ(ද) ΅͔͠αϜωΠϧ(ද) … … … … αϜωΠϧ(ཪ) ΅͔͠αϜωΠϧ(ཪ) jpeg webp
  2. ը૾ॲཧΩϡʔͷ enqueue / dequeue ঢ়گͰݟͯΈΔ னա͗ͷϐʔΫ࣌ʹεέʔϧͰ͖͍ͯͳ͍ɻ εέʔϧΞ΢τ͢Δʹ͸όοναʔόʔΛ૿΍ ͠RailsϓϩηεΛ্ཱͪ͛ͳ͚Ε͹͍͚ͳ͍ ݱঢ়͸։ൃऀ͕खಈͰରԠ" ͔͠͠ɺ࣍ୈʹ࿛গ೥ঢ়ଶʹ…

    ྲྀೖ͕མͪண͍ͯ΋ɺॲཧྔ͸ߴ͍ঢ়ଶͰுΓ෇ ͍͍ͯΔɻ਺࣌ؒ଺ཹ/஗Ԇͯ͠͠·͍ͬͯΔɻ → ը૾औಘAPIͷύϑΥʔϚϯεྼԽʹͭͳ͕Δ enqueue dequeue
  3. - ͕ɺEight ʹͱͬͯ RubyͰॻ͚ͳ͍͜ͱ͕େ͖ͳ໰୊ Lambda ʹߦ͚Ε͹޾ͤͩͱ͍͏͜ͱ͸Θ͔͍ͬͯΔ - ໊ࢗը૾पΓͷॲཧ͸ຊମαʔϏεʹີ઀ʹඥ෇͍͍ͯΔ - ಉظతʹ໊ࢗը૾Λ৮Βͳ͍ͱ͍͚ͳ͍έʔε͕͋ΓɺϩδοΫͷଟॏ؅ཧʹ

    - Ϣʔβؒͷؔ܎ੑ΍ެ։εςʔλεʹΑ໊ͬͯࢗը૾औಘՄ൱΍݁Ռ͕ҟͳΔɻ - ը૾αʔϏε(?)ͷΑ͏ʹαʔϏεΛ੾Γग़͢ͷ͸ɺݱঢ়͔ͳΓՙ͕ॏ͍ - ͦΜͳதɺଞͷݴޠΛ࢖ͬͯͷ Lambda ࣮૷͸ϝϯςφϯεੑʹ೉͋Γ…
  4. - ᶃ gemΛ࢖ͬͨ Lambda Λ࡞Δ - ᶄ native extensionsʹґଘ͢ΔgemΛ࢖ͬͨ Lambda

    Λ࡞Δ - ᶅ ࣮ߦόΠφϦΛؚΊͨ Lambda Λ࡞Δ - ᶆ Lambda LayersΛར༻࣮ͯ͠ߦόΠφϦΛFunction͔Β੾Γ཭͢ - ᶇ FrameworkΛར༻ͨ͠ϩʔΧϧ࣮ߦ؀ڥͱσϓϩΠ؀ڥ - ᶈ CI / CD؀ڥͷߏங εςοϓ
  5. - workσΟϨΫτϦ௚Լͷ vendor/bundle Ͱಈ͘ཧ༝
 Lambda࣮ߦ؀ڥͷ؀ڥม਺(GEM_PATH)ʹσϑΥϧτͰ௥Ճ͞Ε͍ͯΔͨΊ 1. Ruby on Lambda Ͱ

    Gem Λར༻͢Δ https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/current-supported-versions.html
  6. - Lambda ͷ࣮ߦ؀ڥͱಉ౳ͷ؀ڥͰbundle(Ϗϧυ)͢Δඞཁ͕͋Δ 2. native extension ʹґଘͨ͠ gem ͷ৔߹ lambci/lambda:build-ruby2.5

    
 lambci dockerΠϝʔδΛ࢖Θͤͯ΋Β͏ https://github.com/lambci/docker-lambda lambci # rmagickͳͲnative extensionsΛඞཁͱ͢ΔgemΛGemfileʹهड़ $ vi Gemfile # docker container্Ͱbundle install. volumeϚ΢ϯτ͠ϗετ্ͷ vendor/bundle ʹΠϯετʔϧ
 $ docker run -v `pwd`:/var/task -it lambci/lambda:build-ruby2.5 bundle install --path vendor/bundle # lambda_handler.rb, vendor/bundleΛzipԽͯ͠Ξοϓϩʔυ $ zip -r zip_function ./*
  7. - Eightͷը૾ॲཧͰ͸ߴ଎ԽͷͨΊʹ webp ϑΥʔϚοτม׵Λߦ͍ͬͯΔ - ͕ɺ௨ৗͷLambda࣮ߦ؀ڥͷImageMagick͸ webp ରԠ͍ͯ͠ͳ͍…orz 3. ࣮ߦόΠφϦΛؚΊͨ

    Lambda ͷ࡞੒ ## ImageMagickͷversion֬ೝ $ convert -version
 Version: ImageMagick 6.7.8-9 2016-06-22 Q16 http://www.imagemagick.org Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC Features: OpenMP ## ରԠը૾ϑΥʔϚοτͷ֬ೝɻ $ convert -list format … VST* TGA rw- Truevision Targa image WBMP* WBMP rw- Wireless Bitmap (level 0) image WMF* WMF r-- Windows Meta File … dockerίϯςφ্ͰImageMagickΛ֬ೝ webp͕ରԠϦετʹແ͍
  8. - WebP compatibleͳImageMagickΛϏϧυ͢Δࡍͷ࣮ߦྫ 3. ࣮ߦόΠφϦΛؚΊͨ Lambda ͷ࡞੒ ## Lambdaؔ਺͕ల։͞ΕΔσΟϨΫτϦͰ࡞ۀ cd

    /var/task ## libwebpͷinstall curl -sLO https://storage.googleapis.com/downloads.webmproject.org/releases/webp/ libwebp-1.0.0.tar.gz \ && tar zxvf libwebp-1.0.0.tar.gz \ && cd libwebp-1.0.0 && ./configure --prefix=/var/task && make && make install \ && rm -rf ../libwebp-1.0.0.tar.gz ../libwebp-1.0.0 ## webp compatibleͳImageMagickͷϏϧυ curl -sLO https://www.imagemagick.org/download/releases/ImageMagick-6.9.10-34.tar.gz \ && tar zxvf ImageMagick-6.9.10-34.tar.gz \ && cd ImageMagick-6.9.10-34 && ./configure --prefix=/var/task --with-webp=yes \ && make && make install \ && rm -rf ../ImageMagick-6.9.10-34.tar.gz ../ImageMagick-6.9.10-34 - /var/task ҎԼͷϑΝΠϧ(*.rb, vendor/bundle, bin/, lib/, etc…)Λશͯύοέʔδϯά(zipԽ)͢Ε͹OK lambci dockerίϯςφ্ͰϏϧυ࡞ۀ
  9. - ͔͠͠ɺ࣮ߦόΠφϦ(ImageMagick౳)ΛؚΊͨΞοϓϩʔυ͸஍ຯʹਏ͍ ࣮ߦόΠφϦΛʮؚ·ͳ͍ʯ৔߹ (*.rb, vendor/bundle ͷΈ) ࣮ߦόΠφϦΛʮؚΉʯ৔߹ (*.rb, vendor/bundle, bin/,

    lib/, …) ໿5MB ໿50MB! - ࣮ߦόΠφϦࣗମΛߋ৽͢Δස౓͸ଟ͘ͳ͍͸͕ͣͩɺ
 ؔ਺Λमਖ਼͢Δͨͼʹຖճύοέʔδϯά & σϓϩΠ͠ͳ͚Ε͹ͳΒͳ͍ - ؅ཧ͢ΔLambda͕ҰͭͰ͋Ε͹·ͩɺͳΜͱ͔ͳΔ͔΋͠Εͳ͍͕…? ਏ͍͜ͱʹ͸มΘΒͳ͍ 3. ࣮ߦόΠφϦΛؚΊͨ Lambda ͷ࡞੒
  10. - Lambda Layers : ෳ਺ͷLambdaؔ਺ͰϥΠϒϥϦΛڞ༗Ͱ͖Δ࢓૊Έ
 ৄࡉ͸ https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/configuration-layers.html 4. Lambda LayersΛ࢖ͬͯσϓϩΠίετΛ࡟ݮ͢Δ

    - Point - Layerͷίʔυ͸ /opt ҎԼʹల։͞ΕΔ - /opt ҎԼͷύε͸ϥΠϒϥϦݕࡧύεʹ
 σϑΥϧτͰؚ·Ε͍ͯΔ ref: https://qiita.com/t_okkan/items/394a15577bd1aad46ec3
  11. - Dockerίϯςφ্ͰWebP Compatible ImageMagick LayerΛ࡞Δ 4. Lambda LayersΛ࢖ͬͯσϓϩΠίετΛ࡟ݮ͢Δ # Lambda

    Layer͕ల։͞ΕΔ /opt ҎԼͰ࡞ۀ cd /opt ## libwebpͷinstall curl -sLO https://storage.googleapis.com/downloads.webmproject.org/releases/webp/ libwebp-1.0.0.tar.gz \ && tar zxvf libwebp-1.0.0.tar.gz \ && cd libwebp-1.0.0 && ./configure --prefix=/opt && make && make install \ && rm -rf ../libwebp-1.0.0.tar.gz ../libwebp-1.0.0 ## webp compatibleͳImageMagickͷϏϧυ curl -sLO https://www.imagemagick.org/download/releases/ImageMagick-6.9.10-34.tar.gz \ && tar zxvf ImageMagick-6.9.10-34.tar.gz \ && cd ImageMagick-6.9.10-34 && ./configure --prefix=/opt --with-webp=yes \ && make && make install \ && rm -rf ../ImageMagick-6.9.10-34.tar.gz ../ImageMagick-6.9.10-34 - /opt ҎԼͷϑΝΠϧ(bin/, lib/, etc…)Λύοέʔδϯά͠LayersʹΞοϓϩʔυ͢Ε͹OK ΠϯετʔϧσΟϨΫτϦͷࢦఆΛ /var/task ͔ΒLayer͕ల։͞ΕΔ /opt ҎԼʹมߋ
  12. - , ֎෦ϥΠϒϥϦΛLayersʹҠߦ͢Δ͜ͱͰɺϏδωεϩδοΫͷ։ൃʹूத͠ ΍͘͢ͳΔ - , Lambdaؔ਺ͷσϓϩΠ࣌ʹɺ֎෦ϥΠϒϥϦΛؚΉύοέʔδϯά͕ෆཁʹ ͳΓσϓϩΠίετ͕࡟ݮ - -

    ҰํͰɺLayersࣗମͷόʔδϣϯ؅ཧ΍ɺLambdaͱͷґଘؔ܎ɾඥ෇͚Λखಈ Ͱߦ͍ͬͯ͘ͷ͸ΑΓෳࡶʹ… - ϑϨʔϜϫʔΫಋೖͷػӡ 4. Lambda LayersΛ࢖ͬͯσϓϩΠίετΛ࡟ݮ͢Δ
  13. - LambdaΛؚΉServerlessΞʔΩςΫνϟͷߏ੒؅ཧɺϩʔΧϧ࣮ߦɺσϓϩΠ͕ՄೳͳϑϨʔϜϫʔΫ - AWS SAM(Serverless Application Model) - Serverless Framework

    ͷ2୒͔ 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ AWS SAM - AWSެࣜͷαʔόʔϨεΞʔΩςΫνϟʹ
 ಛԽͨ͠ϑϨʔϜϫʔΫ - CloudFormationͰ؅ཧ - ϩʔΧϧ࣮ߦ؀ڥΛ੔උ - σϓϩΠखॱ͕एׯෳࡶͳҹ৅ - AWSҎ֎ͷΫϥ΢υʹ΋ରԠ - SAMಉ༷ɺCloudFormationͰ؅ཧ - ։ൃ͕੝Μ(ͳҹ৅) - pluginͳͲ΋๛෋Ͱ͔Ώ͍ͱ͜Ζʹख͕ಧ͘ - ࠓճ͸͢ͰʹEightͰར༻࣮੷ͷ͋Δ Serverless Framework Λ࠾༻
  14. - Serverlss Framework ͱDockerΛར༻͠ɺLambdaͱLayersΛ؅ཧ͢Δ 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ ΦεεϝͷσΟϨΫτϦߏ੒Ҋ - ᶃ Layers

    ͷ؅ཧ
 Dockefileʹهड़͠ɺDocker Image಺ʹด͡ࠐΊͯ͠·͏ɻ
 Layers ʹߋ৽͕͋ͬͨ৔߹͸Docker ImageΛ࠶Ϗϧυ͢Δɻ - ᶄ Serverless Framework
 serverless΋Docker಺Ͱ࣮ߦͰ͖ΔΑ͏ʹ - ᶅ Lambdaؔ਺, serverless.yml, vendor/bundleͷ؅ཧ
 ্ड़ͷDockerίϯςφͷ্ཱͪ͛࣌ʹ /var/task ʹ࡞ۀσΟϨΫτϦΛϚ΢ϯ τ͠ɺίϚϯυ࣮ߦ͸Dockerίϯςφ্Ͱߦ͏ɻ - gemͷ௥Ճ/ߋ৽͸͠͹͠͹͋ΔͨΊvendor/bundle͸
 docker imageʹ͸ؚΊͳ͍ & όʔδϣϯ؅ཧ΋͠ͳ͍(.gitignore)
  15. - Layers͸ Dockerfile Ͱ؅ཧͯ͠͠·͏ͷ͕٢ 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ LambdaͷDocker ImageΛϕʔεͱͨ͠ ࣗલͷDocker Image༻ҙ

    FROM lambci/lambda:build-ruby2.5 # ࡞ۀ؀ڥͷߏஙʹඞཁͳ؀ڥม਺ͷఆٛ ENV PKG_CONFIG_PATH /opt/lib/pkgconfig # Lambda layer૬౰ͷ؀ڥΛ࡞੒ WORKDIR /opt ## libwebpͷinstall ... (લड़ͱಉ༷ͷॲཧ) ## webp compatibleͳImageMagickͷϏϧυ ... (લड़ͱಉ༷ͷॲཧ) # serverless frameworkͷΠϯετʔϧ RUN curl -sL https://rpm.nodesource.com/setup_6.x | bash - \ && yum -y install nodejs && yum -y clean all \ && npm install -g serverless WORKDIR /var/task/ CMD ["/bin/bash"] ᶃlambci/lambda:build-ruby2.5 Λ
 ɹϕʔεʹͨ͠Πϝʔδ ᶄ/opt ഑ԼʹAWS Layersʹ૬౰͢Δ
 ɹϥΠϒϥϦΛϏϧυ͠഑ஔ ᶅnodeΛΠϯετʔϧ͠ɺ
 ɹserverless frameworkΛΠϯετʔϧ͢Δ Dockerfileྫ
  16. - serverless.yml Ͱͷ layers, functionsͷઃఆྫ 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ … layers: webpimagemagick:

    path: copy_layers # /opt ͷγϯϘϦοΫϦϯάΛష্ͬͨͰdeploy͢Δ description: WebP compatible ImageMagick functions: func1: handler: func1.execute layers: - {Ref: WebpimagemagickLambdaLayer} events: - s3: card-images func2: handler: func2.execute Lambdaؔ਺Ͱ࢖͏LayerΛ ඥ෇͚ͯ؅ཧ͢Δ͜ͱ͕Մೳ ࢦఆͨ͠bucketʹߋ৽͕͋ͬͨࡍʹ lambda࣮ߦͤ͞Δઃఆ
  17. - Ϗϧυͨ͠Docker Imageͷίϯςφ্Ͱ։ൃ࡞ۀ - bundle install —path vendor/bundle - gemͷΠϯετʔϧ

    - bundle exec rspec - rubyίʔυͷࣗಈςετ࣮ߦ - serverless deploy - functionͱlayersͷσϓϩΠ - serverless invoke (local) -f function໊ - Lambda functionͷ(ϩʔΧϧ)࣮ߦ 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ
  18. $ sls deploy ...snip Serverless: Uploading service aws-ruby-function.zip file to

    S3 (4.02 MB)... Serverless: Uploading service webpimagemagick.zip file to S3 (47.44 MB)... ...snip api keys: None endpoints: None functions: func1: eight-image-proccessor-dev-func1 func2: eight-image-proccessor-dev-func2 layers: webpimagemagick: arn:aws:lambda:ap-northeast-1:xxxxx:layer:webpimagemagick:9 Serverless: Removing old service artifacts from S3... 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ - deployίϚϯυΛୟ͚ͩ͘Ͱɺlambdaͱlayersͷύοέʔδϯά&σϓϩΠ͕Ͱ͖Δ - serverless(sls) deploy functionͱlayer͕ͦΕͧΕύοέʔδϯά σϓϩΠ׬ྃͨ͠functionsͱlayersͷ৘ใ
  19. def hello(event:, context:) { statusCode: 200, body: JSON.generate('Go Serverless v1.0!

    Your function executed successfully!’) } end 5. ϑϨʔϜϫʔΫΛར༻͠࠶ݱੑ/ܧଓੑΛߴΊΔ - ϩʔΧϧ؀ڥͰͷlambdaͷಈ࡞֬ೝ΋Ͱ͖Δ - serverless(sls) invoke local -f function໊ $ sls invoke local -f hello {"statusCode":200,"body":"\"Go Serverless v1.0! Your function executed successfully!\""}
  20. - ϐʔΫλΠϜͰ΋ Lambda ͕ࣗಈͰεέʔϧ͠଺ཹ͕΄΅0ʹ - ͜Ε·Ͱӡ༻ऀ͕खಈͰ΍͍ͬͯͨӡ༻ίετ΋΄΅0ʹ - ຖ೔ϐʔΫ࣌ʹൃੜ͍ͯͨ͠
 Ξϥʔτ &

    εέʔϦϯάۀ຿͔Βͷ։์ . - ଺ཹ͕ͳ͘ͳΔ = APIύϑΥʔϚϯε΋޲্ - ଺ཹ࣌ʹൃੜͯ͠͠·͏ಉظతͳը૾ม׵ॲཧʹ͔͔͍ͬͯͨ
 ਺ेʙ਺ඦ ms / req ͕ൃੜ͠ͳ͘ͳͬͨ✨ มΘͬͨ͜ͱ ࣗಈͰεέʔϧΞ΢τ/Πϯ
  21. - Lambda ΍։ൃ؀ڥͷϝϯςφϯε - ࠓճͷΑ͏ͳ Ruby on Lambda ͷ։ൃ؀ڥΛҰ౓࡞ͬͯ͠·͑͹ɺ
 ࠓޙͷLambda

    Functionͷ௥Ճɾमਖ਼ɾ࡟আ͸༰қ , - εέʔϥϏϦςΟ͕ωοΫͰରԠͰ͖ͳ͔ͬͨػೳվળ΋͠΍͘͢ - ը૾ϑΥʔϚοτ΍αΠζͷ֦ு - ը૾ͷ඼࣭ͷ࠷దԽͳͲ มΘͬͨ͜ͱ
  22. - ۀ຿ͰRubyΛϝΠϯʹ৮͍ͬͯΔํ͸ Lambda ͸ͥͻ࢖͏΂͠ - ڞ௨ॲཧΛGemԽ͢Δ͜ͱͰɺඇಉظॲཧ౳Λ༰қʹLambdaʹ͍͚࣋ͬͯΔɻ - Serverless Framework ΍

    AWS SAM Λར༻͢Δ͜ͱͰίετΛॻ͚ͣʹ
 αʔόʔϨεͳ։ൃɾCI/CD؀ڥΛߏஙͰ͖Δɻ - Native extentionʹґଘ͢Δgem΍֎෦ϥΠϒϥϦΛར༻͢Δ৔߹͸ Lambda ͱಉ ౳ͷ࣮ߦ؀ڥͰϏϧυ͠σϓϩΠ͢Δɻ - `lambci/lambda` Λϕʔεʹͨ͠Docker؀ڥͷߏஙΛ͢Δ͜ͱΛΦεεϝ͠·͢ ·ͱΊ