Upgrade to Pro — share decks privately, control downloads, hide ads and more …

紙面ビューアーを支える サーバーレスアーキテクチャ / serverless architecture supports Nikkei's paper viewer

紙面ビューアーを支える サーバーレスアーキテクチャ / serverless architecture supports Nikkei's paper viewer

Taishi Ikai

October 01, 2016
Tweet

More Decks by Taishi Ikai

Other Decks in Technology

Transcript

  1. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016

    View Slide

  2. 2
    About me
    ழࣂ େࢤ (Ikai Taishi)
    ೔ຊܦࡁ৽ฉࣾͷΤϯδχΞ
    ࢴ໘ϏϡʔΞʔͷ։ൃΛ୲౰
    Swift, Python, JavaScript Github: @ikait

    View Slide

  3. 3
    ༗ྉձһ ໿48ສਓ
    ແྉձһ ໿300ສਓ
    ඇొ࿥ ໿2500ສUB
    ೔ܦిࢠ൛ͱ͸
    2010 ೥ 3 ݄ʹ૑ץ
    ݄ؒΞΫηε 3 ԯ݅
    ຖ೔໿ 900 ຊͷهࣄΛ഑৴
    PCɾϞόΠϧαΠτɺ

    iOS, Android ΞϓϦͳͲʹରԠ

    View Slide

  4. 4
    ࢴ໘ϏϡʔΞʔʁ

    View Slide

  5. 5
    ೔ܦిࢠ൛ ࢴ໘ϏϡʔΞʔ

    View Slide

  6. 6
    ೔ຊܦࡁ৽ฉΛɺ

    εϚϗŋλϒϨοτͰಡΉ
    iOSŋAndroid ରԠ
    2016.3.15 ϦχϡʔΞϧ

    View Slide

  7. 7
    ேŋ༦ץ ͕ಡΊΔ
    ຖேɺຖ༦഑৴͞Ε·͢
    (ٳץ೔͸আ͘)
    աڈ1िؒ෼ಡΊ·͢
    ΦϓγϣϯͰɺ೔ܦ࢈ۀ৽ฉɺ
    ೔ܦ MJ ͕ಡΊ·͢

    View Slide

  8. 8
    ৽ฉͱಉ͡ϨΠΞ΢τ
    ొ࿥ͨ͠ΩʔϫʔυΛ

    ϋΠϥΠτͰ͖·͢
    ݕࡧͰ͖·͢
    μ΢ϯϩʔυ͓ͯ͘͠ͱ

    ΦϑϥΠϯͰಡΊ·͢

    View Slide

  9. 9
    ࢴ໘ϏϡʔΞʔ
    ಺੡͍ͯ͠·͢
    iOSɺAPIɺը૾ੜ੒ͳͲ
    Ϟμϯͳ։ൃϑϩʔʢ❗ʣ
    ϚωʔδυɾαʔϏεΛ
    ੵۃతʹར༻͍ͯ͘͠

    View Slide

  10. 10
    https://twitter.com/naoya_ito/status/779241231029186561

    View Slide

  11. 11
    https://github.com/serverless/serverless

    View Slide

  12. 12
    API (ೝূܥ) Ͱ͸ɺ͜ͷ Microservice ࣮ߦ؀ڥΛར༻͍ͯ͠·͢
    https://speakerdeck.com/yosukesuzuki/nikkei-microservice

    View Slide

  13. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016
    ͱϚΠΫϩαʔϏεΞʔΩςΫνϟ

    View Slide

  14. 14
    ࢴ໘ը૾͕ΞϓϦʹಧ͘·Ͱ

    View Slide

  15. 15
    ޻৔
    S3
    ฤू
    લఏ
    هࣄ

    ϨΠΞ΢τ
    ιʔε:
    ࢴ໘ը૾
    ϝλσʔλ XML
    ࠲ඪͳͲ

    View Slide

  16. 16
    S3
    SQS EC2
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue
    ΞΫηε੍ݶ
    όονॲཧͩͱ͢Δͱ..

    View Slide

  17. 17
    ओʹேץɺ༦ץͷ࣌ؒͷՔಇ͕ͩɺࢴ໘ߋ৽͸ෆఆظʹ͋Δ
    → ਺෼ִؒͰ SQS ΛϙʔϦϯά͢Δ
    ɾ Πϯελϯε͸ৗʹىಈ͓ͯ͘͠?
    ɾ εέʔϧΞ΢τ͸एׯ͕͔͔࣌ؒΔ
    ॲཧ͕ଟ͍ɺը૾ม׵(ϦαΠζɺϑΟϧλɺWebPม׵) ͸ॏΊ
    → ͍ܰॲཧ͸ϚϧνεϨουͰฒྻԽ
    → Πϯελϯε͸ෳ਺୆ىಈ͢ΔͳͲ
    ՝୊

    View Slide

  18. 18
    Lambda ͕
    ʢ౦ژϦʔδϣϯʹʣ
    ͖ͨ
    2015/06/29

    View Slide

  19. 19
    S3
    SQS EC2
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue

    View Slide

  20. 20
    S3
    SQS
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue
    Lambda

    View Slide

  21. 21
    S3
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    Lambda
    S3 ͔ΒͷΠϕϯτͰ
    Lambda ͕࣮ߦ͞ΕΔ
    ͜Μͳײ͡ʁ

    View Slide

  22. 22
    ࡞Γ௚ͨ͠

    View Slide

  23. 23
    Lambda ؔ਺ͷ։ൃ

    View Slide

  24. 24
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    ϝλσʔλม׵ / औಘ
    Lambda ্Ͱߦ͏͜ͱ
    v0.10.36

    View Slide

  25. 25
    4000px
    5296px

    View Slide

  26. 26
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    Lambda ্ͷ ImageMagick Λ
    Ͱར༻

    View Slide

  27. 27
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    Linux Ͱ੩తϏϧυͨ͠ cwebp όΠφϦ
    ΛɺΞοϓϩʔυ͢Δ zip ʹؚΊΔ
    cwebp Λ node ͔Βѻ͏ͨΊͷ

    ϥούʔϥΠϒϥϦ
    install cwebp

    View Slide

  28. 28
    όΠφϦͷύεΛίʔυ಺Ͱ؀ڥม਺ʹ௥Ճ
    JG GTFYJTUT4ZOD VTSMPDBMCJODXFCQ

    \
    DPOTU-*#@%*3QBUIKPJO @@EJSOBNF MJC

    QSPDFTTFOW1"5)A\-*#@%*3^A
    QSPDFTTFOW-%@-*#3"3:@1"5)A\-*#@%*3^A
    ^

    View Slide

  29. 29
    ϩʔΧϧͰϢχοτςετ

    View Slide

  30. 30
    Ϗϧυ / σϓϩΠ
    Lambda
    ςετ
    Babel
    ʢes6 → es5ʣ
    ɺιʔεϚοϓ
    npm Πϯετʔϧ
    σϓϩΠ (node-aws-lambda)
    npm: source-map-support
    Increment
    Version

    View Slide

  31. 31
    ಈ࡞ϑϩʔ (ུ֓)

    View Slide

  32. 32
    Source S3 Dest S3
    Order 

    Regenerate
    Dispatch
    Scheduled 

    events
    Invoke
    Tasks
    Login,
    Get
    Signed
    URL
    Request

    Signed
    URL
    API

    View Slide

  33. 33
    \
    OBNFEJWJEF
    TPVSDFYZ[QOH
    ˑEFTUJOBUJPOYZ[@QOH
    EBUB\
    SFDUBOHMF\
    Y Z X I
    ^
    ^
    ^
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    Dest S3
    Dispatch
    Tasks
    Source S3

    View Slide

  34. 34
    ֤ॲཧ୲౰ͷ Lambda ͕ S3 ͔Β
    ඞཁͳϑΝΠϧΛऔಘͯ͠ॲཧ
    Dest S3
    Dispatch
    Tasks
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    Source S3

    View Slide

  35. 35
    Dest S3
    Dispatch
    Tasks
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    ̍ΠϕϯτͰෳ਺ͷLambda 

    ΛىಈͰ͖ͳ͍ͨΊɺͻͱͭͷ

    Lambda ؔ਺ͰҰ୴ΠϕϯτΛड͚औΓɺ
    ϑΝΠϧ໊Λϕʔεʹͯ͠ɺଞͷ Lambda Λىಈ (S3ܦ༝)

    View Slide

  36. 36
    Lambda ͔Βىಈ͢Δ৔߹ɺεϩοτϧ͢Δͱ
    ࣦഊ͢Δ͔΋͠Εͳ͍ͷͰ S3 Λܦ༝͢Δ
    Dest S3
    Dispatch
    Scheduled 

    events
    Tasks

    View Slide

  37. 37
    1 ෼͓͖ʹىಈ

    ϑΝΠϧͷଘࡏ֬ೝΛߦ͍

    ഔମϦετΛߋ৽͢Δ
    ഔମϦετͷଘࡏΛ΋ͬͯɺ
    ഔମͷ४උ׬ྃΛࣔ͢
    Dest S3
    Scheduled 

    events

    View Slide

  38. 38
    API Ͱೝূ
    API ͸ϢʔβʔΛೝՄͯ͠ɺ
    CloudFront ॺ໊෇͖URL Λൃߦ
    Private Cache Distribution
    Dest S3
    Login,
    Get
    Signed
    URL
    Request

    Signed
    URL
    API

    View Slide

  39. 39
    Private Cache Distribution ύλʔϯ
    ① ೝূࡁΈϦΫΤετ GET
    https://api/sign
    ② ॺ໊
    ③ CF ΁ॺ໊Λ͚ͭͯ

    ϦΫΤετ

    View Slide

  40. 40
    Private Cache Distribution ύλʔϯ
    CloudFront Λҙࣝ͢Δ͜ͱͳ͘ɺ
    CloudFront ্ͷίϯςϯπͷऔಘ͕Մೳ
    ① ೝূࡁΈϦΫΤετ
    ② 303 (ॺ໊෇͖URL΁ͷϦμΠϨΫτ)
    ②’ ϦΫΤετ
    ॺ໊෇͖URL
    ॺ໊෇͖URLͷੜ੒

    View Slide

  41. 41
    ӡ༻

    View Slide

  42. 42
    ேץલ͸ 18000 invocations/m

    View Slide

  43. 43
    Lambda ༝དྷͷΤϥʔ͸Ίͬͨʹى͖ͳ͍

    View Slide

  44. 44
    Slack ࿈ܞ
    Քಇঢ়گΛ௨஌
    Τϥʔͷᮢ஋Λ௒͑ͨͱ͖Λ௨஌
    Notify
    Log
    Log
    Alarm
    (when exceeded threshold)
    Notify

    View Slide

  45. 45
    Source S3
    Order 

    Regenerate
    Dispatch
    Invoke
    Slash commands Ͱ API Gateway
    ͔Β Lambda ΛىಈɻΤϥʔ࣌ɺ

    QA ࣌ͳͲ͸ɺखಈͰഔମΛੜ੒
    Slack ࿈ܞ

    View Slide

  46. 46
    Lambda ͔Β Lambda ΛݺΜͩޙʹ෮ؼͰ͖ͳ͍Τϥʔ͸ݫې
    ΤϥʔΛى͜͢ͱ࠷௿ 3 ճϦτϥΠ͞ΕΔͷͰ…
    < 429: You are sending too 

    ɹmany requests. Please relax.
    time
    dying 

    lambda

    View Slide

  47. 47
    API ଆͷ E2E ςετ͕Քಇ
    ෳ਺ճ࿈ଓࣦͯ͠ഊ͢Δͱ PagerDuty ͔Β API νʔϜʹి࿩
    Lambda ͕ࢮ๢͍ͯ͠Δ৔߹͸ɺखॱʹͦͬͯϩʔΧϧͰॲཧ

    View Slide

  48. 48
    CloudFront ͕ࢮ๢ͨ͠ࡍ͸υϝΠϯͷ޲͖ઌΛ S3 ΁

    View Slide

  49. 49
    S3 ʹॻ͖ࠐ·ΕΔ CloudFront ͷϩάΛ
    Lambda Ͱύʔεͯ͠ Elasticsearch ΁౤ೖɺ
    kibana ͰՄࢹԽ
    Log S3
    Parse

    View Slide

  50. 50
    Ϣʔβʔͷ੠Λड͚औͬͯɺSlack ͱ Backlog ΁౤ߘ
    API Gateway ܦ༝ͷ Lambda ىಈͷ

    ϨΠςϯγʔ͸ؾʹͳΔ

    View Slide

  51. 51
    ৼΓฦΓ

    View Slide

  52. 52
    ։ൃ଎౓͸ɺ޲্ʁ
    Lambda ͷͨΊͷ։ൃ͸ඞཁͩͬͨ
    ΞϓϦʹԠͨ͡σʔλଆͷॊೈͳมߋ͕Մೳͩͬͨ
    αʔόʔෛՙΛؾʹ͠ͳ͍ͰΑ͘ͳͬͨʢը૾ੜ੒ɺ഑৴ʣ
    ։ൃମ੍ͷԸܙʢݱ৔Ͱٕज़తͳҙࢥܾఆ͕͋Δ͜ͱʣ
    ӡ༻ίετ͸ݮͬͨʢ΄΅ͳ͍ʣ

    View Slide

  53. 53
    αʔϏεɾঢ়گʹΑͬͯ͸ɺ୅ସΛҙࣝ͢Δඞཁ͸͋Γͦ͏
    Lambda ؔ਺͸ɺLambda Ҏ֎ͷͲ͜Ͱಈ͔ͤͦ͏ʁ
    S3, CloudFront ͕མͪͨ࣌͸ʁ
    πʔϧ͕ग़ἧͬͯɺݴޠͷબ୒ࢶ΋૿͑ͨ
    Serverless, Apex, Lamvery ͳͲ
    ͲͷαʔϏε΋҆ఆ͍ͯ͠Δ
    ࠓ͸҆৺ͯ͠৸ΒΕΔ

    View Slide

  54. 54
    ҙ֎ʹࣗ༝౓͸͋Δ
    OS ͸ Amazon Linux ͳͷͰɺ͜͜Ͱಈ͘΋ͷ͸େମಈ͘
    Linux ༻ͷ cwebp όΠφϦͰ WebP ม׵΋Ͱ͖ͨ
    Lambda ͷͨΊͷ։ൃ͕ඞཁʹͳΔ
    ಛʹɺS3 ͳͲͱ࿈ܞͯ͠ΠϕϯτϕʔεͰॲཧ͢Δ৔߹
    ΠϕϯτͷϧʔςΟϯάɺσΟεύονϟ͕ඞཁʹͳΔ৔߹΋
    Ϣχοτςετ͸Ͱ͖Δ͕ɺࣗಈͰ݁߹ςετ͕೉͍͠
    Lambda ʹ͍ͭͯ

    View Slide

  55. 55
    ࡉ͔͍πʔϧΛ࡞Δʹ͸͔ͳΓศར
    Slack ͷ Slash commands ͳͲ (API Gateway ͱͷ૊Έ߹Θͤ)
    ͪΌΜͱͰ͖͍ͯΔ
    ΠϕϯτΛ࿙Βͨ͠Γ͠ͳ͍
    ͔ͳΓ͍҆
    FaaS ͷະདྷΛײ͡Δ
    Lambda ʹ͍ͭͯ

    View Slide

  56. 56
    ؆୯ʹ࢖͑Δ PaaS ج൫ͷ४උ͕͋ͬͨ
    Elastic Beanstalk + Docker + Python + Django
    ೝূඞਢɺ༗ྉձһݶఆͷαʔϏεͳͷͰɺݎ࿚Ͱ։ൃମ੍͕
    ཱ͍֬ͯ͠Δํ͕Α͔ͬͨ
    API Gateway + Lambda ͸؅ཧͷίετ͕͔͔Γͦ͏ɺϨΠςϯ
    γʔ͕େ͖͍
    PaaS?

    View Slide

  57. 57
    εςʔτϨεΛҙࣝͤ͟ΔΛಘͳ͍
    ෭࡞༻΍ঢ়ଶอ࣋ʹཔΕͳ͍࣮૷্ͷ੍໿
    ີ݁߹ͳίʔυ͕ࣗવʹݮΓɺݟ௨͕͠෇͖΍͘͢ͳΔ
    ϑϩϯτΤϯυͷ։ൃʹ΋ੜ͖ͨ
    αʔόʔϨεͰͷ։ൃΛ௨ͯ͡

    View Slide

  58. 58
    ·ͱΊ

    View Slide

  59. 59
    Lambda + S3 + CloudWatch Ͱࢴ໘ը૾ͷม׵
    S3 + CloudFront Ͱը૾ͷ഑৴: Private cache distribution
    Lambda Ͱ WebP ͷม׵΍ը૾ॲཧΛߦͬͨ
    ݁Ռͷ଴ͪ߹ΘͤΛ͢ΔͨΊʹ ɺCloudWatch Ͱ cron తʹ
    Lambda Λ࣮ߦ
    ҰͭͷΠϕϯτͰෳ਺ͷ Lambda Λىಈ͢Δ Workaround
    Slack, Backlog, PagerDuty ͳͲͷ SaaS ΋׆༻
    αʔόʔϨεͰखͷ͔͔Βͳ͍ΠϯϑϥΛߴ଎ʹߏஙͰ͖ͨ

    View Slide

  60. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016

    View Slide