Upgrade to Pro — share decks privately, control downloads, hide ads and more …

紙面ビューアーを支える サーバーレスアーキテクチャ / serverless architecture supports Nikkei's paper viewer

紙面ビューアーを支える サーバーレスアーキテクチャ / serverless architecture supports Nikkei's paper viewer

Taishi Ikai

October 01, 2016
Tweet

More Decks by Taishi Ikai

Other Decks in Technology

Transcript

  1. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016

    View full-size slide

  2. 2
    About me
    ழࣂ େࢤ (Ikai Taishi)
    ೔ຊܦࡁ৽ฉࣾͷΤϯδχΞ
    ࢴ໘ϏϡʔΞʔͷ։ൃΛ୲౰
    Swift, Python, JavaScript Github: @ikait

    View full-size slide

  3. 3
    ༗ྉձһ ໿48ສਓ
    ແྉձһ ໿300ສਓ
    ඇొ࿥ ໿2500ສUB
    ೔ܦిࢠ൛ͱ͸
    2010 ೥ 3 ݄ʹ૑ץ
    ݄ؒΞΫηε 3 ԯ݅
    ຖ೔໿ 900 ຊͷهࣄΛ഑৴
    PCɾϞόΠϧαΠτɺ

    iOS, Android ΞϓϦͳͲʹରԠ

    View full-size slide

  4. 4
    ࢴ໘ϏϡʔΞʔʁ

    View full-size slide

  5. 5
    ೔ܦిࢠ൛ ࢴ໘ϏϡʔΞʔ

    View full-size slide

  6. 6
    ೔ຊܦࡁ৽ฉΛɺ

    εϚϗŋλϒϨοτͰಡΉ
    iOSŋAndroid ରԠ
    2016.3.15 ϦχϡʔΞϧ

    View full-size slide

  7. 7
    ேŋ༦ץ ͕ಡΊΔ
    ຖேɺຖ༦഑৴͞Ε·͢
    (ٳץ೔͸আ͘)
    աڈ1िؒ෼ಡΊ·͢
    ΦϓγϣϯͰɺ೔ܦ࢈ۀ৽ฉɺ
    ೔ܦ MJ ͕ಡΊ·͢

    View full-size slide

  8. 8
    ৽ฉͱಉ͡ϨΠΞ΢τ
    ొ࿥ͨ͠ΩʔϫʔυΛ

    ϋΠϥΠτͰ͖·͢
    ݕࡧͰ͖·͢
    μ΢ϯϩʔυ͓ͯ͘͠ͱ

    ΦϑϥΠϯͰಡΊ·͢

    View full-size slide

  9. 9
    ࢴ໘ϏϡʔΞʔ
    ಺੡͍ͯ͠·͢
    iOSɺAPIɺը૾ੜ੒ͳͲ
    Ϟμϯͳ։ൃϑϩʔʢ❗ʣ
    ϚωʔδυɾαʔϏεΛ
    ੵۃతʹར༻͍ͯ͘͠

    View full-size slide

  10. 10
    https://twitter.com/naoya_ito/status/779241231029186561

    View full-size slide

  11. 11
    https://github.com/serverless/serverless

    View full-size slide

  12. 12
    API (ೝূܥ) Ͱ͸ɺ͜ͷ Microservice ࣮ߦ؀ڥΛར༻͍ͯ͠·͢
    https://speakerdeck.com/yosukesuzuki/nikkei-microservice

    View full-size slide

  13. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016
    ͱϚΠΫϩαʔϏεΞʔΩςΫνϟ

    View full-size slide

  14. 14
    ࢴ໘ը૾͕ΞϓϦʹಧ͘·Ͱ

    View full-size slide

  15. 15
    ޻৔
    S3
    ฤू
    લఏ
    هࣄ

    ϨΠΞ΢τ
    ιʔε:
    ࢴ໘ը૾
    ϝλσʔλ XML
    ࠲ඪͳͲ

    View full-size slide

  16. 16
    S3
    SQS EC2
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue
    ΞΫηε੍ݶ
    όονॲཧͩͱ͢Δͱ..

    View full-size slide

  17. 17
    ओʹேץɺ༦ץͷ࣌ؒͷՔಇ͕ͩɺࢴ໘ߋ৽͸ෆఆظʹ͋Δ
    → ਺෼ִؒͰ SQS ΛϙʔϦϯά͢Δ
    ɾ Πϯελϯε͸ৗʹىಈ͓ͯ͘͠?
    ɾ εέʔϧΞ΢τ͸एׯ͕͔͔࣌ؒΔ
    ॲཧ͕ଟ͍ɺը૾ม׵(ϦαΠζɺϑΟϧλɺWebPม׵) ͸ॏΊ
    → ͍ܰॲཧ͸ϚϧνεϨουͰฒྻԽ
    → Πϯελϯε͸ෳ਺୆ىಈ͢ΔͳͲ
    ՝୊

    View full-size slide

  18. 18
    Lambda ͕
    ʢ౦ژϦʔδϣϯʹʣ
    ͖ͨ
    2015/06/29

    View full-size slide

  19. 19
    S3
    SQS EC2
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue

    View full-size slide

  20. 20
    S3
    SQS
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    enqueue
    polling / dequeue
    Lambda

    View full-size slide

  21. 21
    S3
    CMS
    CloudFront Client
    ιʔεऔಘ

    ϑΝΠϧ഑ஔ
    ϦαΠζ
    ը૾ϑΟϧλʔ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    ϝλσʔλऔಘ
    WebP ม׵
    JPEG ม׵
    ιʔε
    Lambda
    S3 ͔ΒͷΠϕϯτͰ
    Lambda ͕࣮ߦ͞ΕΔ
    ͜Μͳײ͡ʁ

    View full-size slide

  22. 22
    ࡞Γ௚ͨ͠

    View full-size slide

  23. 23
    Lambda ؔ਺ͷ։ൃ

    View full-size slide

  24. 24
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    ϝλσʔλม׵ / औಘ
    Lambda ্Ͱߦ͏͜ͱ
    v0.10.36

    View full-size slide

  25. 25
    4000px
    5296px

    View full-size slide

  26. 26
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    Lambda ্ͷ ImageMagick Λ
    Ͱར༻

    View full-size slide

  27. 27
    ϦαΠζ
    γϟʔϓɺίϯτϥετ
    هࣄը૾ۣܗ੾ΓऔΓ
    ࢴ໘ը૾෼ׂ
    WebP ม׵
    JPEG ม׵
    Linux Ͱ੩తϏϧυͨ͠ cwebp όΠφϦ
    ΛɺΞοϓϩʔυ͢Δ zip ʹؚΊΔ
    cwebp Λ node ͔Βѻ͏ͨΊͷ

    ϥούʔϥΠϒϥϦ
    install cwebp

    View full-size slide

  28. 28
    όΠφϦͷύεΛίʔυ಺Ͱ؀ڥม਺ʹ௥Ճ
    JG GTFYJTUT4ZOD VTSMPDBMCJODXFCQ

    \
    DPOTU-*#@%*3QBUIKPJO @@EJSOBNF MJC

    QSPDFTTFOW1"5)A\-*#@%*3^A
    QSPDFTTFOW-%@-*#3"3:@1"5)A\-*#@%*3^A
    ^

    View full-size slide

  29. 29
    ϩʔΧϧͰϢχοτςετ

    View full-size slide

  30. 30
    Ϗϧυ / σϓϩΠ
    Lambda
    ςετ
    Babel
    ʢes6 → es5ʣ
    ɺιʔεϚοϓ
    npm Πϯετʔϧ
    σϓϩΠ (node-aws-lambda)
    npm: source-map-support
    Increment
    Version

    View full-size slide

  31. 31
    ಈ࡞ϑϩʔ (ུ֓)

    View full-size slide

  32. 32
    Source S3 Dest S3
    Order 

    Regenerate
    Dispatch
    Scheduled 

    events
    Invoke
    Tasks
    Login,
    Get
    Signed
    URL
    Request

    Signed
    URL
    API

    View full-size slide

  33. 33
    \
    OBNFEJWJEF
    TPVSDFYZ[QOH
    ˑEFTUJOBUJPOYZ[@QOH
    EBUB\
    SFDUBOHMF\
    Y Z X I
    ^
    ^
    ^
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    Dest S3
    Dispatch
    Tasks
    Source S3

    View full-size slide

  34. 34
    ֤ॲཧ୲౰ͷ Lambda ͕ S3 ͔Β
    ඞཁͳϑΝΠϧΛऔಘͯ͠ॲཧ
    Dest S3
    Dispatch
    Tasks
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    Source S3

    View full-size slide

  35. 35
    Dest S3
    Dispatch
    Tasks
    S3 ΁ PUT Λ͏͚ͯɺॲཧ಺༰Λॻ͍ͨϑΝΠϧΛ഑ஔ
    ̍ΠϕϯτͰෳ਺ͷLambda 

    ΛىಈͰ͖ͳ͍ͨΊɺͻͱͭͷ

    Lambda ؔ਺ͰҰ୴ΠϕϯτΛड͚औΓɺ
    ϑΝΠϧ໊Λϕʔεʹͯ͠ɺଞͷ Lambda Λىಈ (S3ܦ༝)

    View full-size slide

  36. 36
    Lambda ͔Βىಈ͢Δ৔߹ɺεϩοτϧ͢Δͱ
    ࣦഊ͢Δ͔΋͠Εͳ͍ͷͰ S3 Λܦ༝͢Δ
    Dest S3
    Dispatch
    Scheduled 

    events
    Tasks

    View full-size slide

  37. 37
    1 ෼͓͖ʹىಈ

    ϑΝΠϧͷଘࡏ֬ೝΛߦ͍

    ഔମϦετΛߋ৽͢Δ
    ഔମϦετͷଘࡏΛ΋ͬͯɺ
    ഔମͷ४උ׬ྃΛࣔ͢
    Dest S3
    Scheduled 

    events

    View full-size slide

  38. 38
    API Ͱೝূ
    API ͸ϢʔβʔΛೝՄͯ͠ɺ
    CloudFront ॺ໊෇͖URL Λൃߦ
    Private Cache Distribution
    Dest S3
    Login,
    Get
    Signed
    URL
    Request

    Signed
    URL
    API

    View full-size slide

  39. 39
    Private Cache Distribution ύλʔϯ
    ① ೝূࡁΈϦΫΤετ GET
    https://api/sign
    ② ॺ໊
    ③ CF ΁ॺ໊Λ͚ͭͯ

    ϦΫΤετ

    View full-size slide

  40. 40
    Private Cache Distribution ύλʔϯ
    CloudFront Λҙࣝ͢Δ͜ͱͳ͘ɺ
    CloudFront ্ͷίϯςϯπͷऔಘ͕Մೳ
    ① ೝূࡁΈϦΫΤετ
    ② 303 (ॺ໊෇͖URL΁ͷϦμΠϨΫτ)
    ②’ ϦΫΤετ
    ॺ໊෇͖URL
    ॺ໊෇͖URLͷੜ੒

    View full-size slide

  41. 42
    ேץલ͸ 18000 invocations/m

    View full-size slide

  42. 43
    Lambda ༝དྷͷΤϥʔ͸Ίͬͨʹى͖ͳ͍

    View full-size slide

  43. 44
    Slack ࿈ܞ
    Քಇঢ়گΛ௨஌
    Τϥʔͷᮢ஋Λ௒͑ͨͱ͖Λ௨஌
    Notify
    Log
    Log
    Alarm
    (when exceeded threshold)
    Notify

    View full-size slide

  44. 45
    Source S3
    Order 

    Regenerate
    Dispatch
    Invoke
    Slash commands Ͱ API Gateway
    ͔Β Lambda ΛىಈɻΤϥʔ࣌ɺ

    QA ࣌ͳͲ͸ɺखಈͰഔମΛੜ੒
    Slack ࿈ܞ

    View full-size slide

  45. 46
    Lambda ͔Β Lambda ΛݺΜͩޙʹ෮ؼͰ͖ͳ͍Τϥʔ͸ݫې
    ΤϥʔΛى͜͢ͱ࠷௿ 3 ճϦτϥΠ͞ΕΔͷͰ…
    < 429: You are sending too 

    ɹmany requests. Please relax.
    time
    dying 

    lambda

    View full-size slide

  46. 47
    API ଆͷ E2E ςετ͕Քಇ
    ෳ਺ճ࿈ଓࣦͯ͠ഊ͢Δͱ PagerDuty ͔Β API νʔϜʹి࿩
    Lambda ͕ࢮ๢͍ͯ͠Δ৔߹͸ɺखॱʹͦͬͯϩʔΧϧͰॲཧ

    View full-size slide

  47. 48
    CloudFront ͕ࢮ๢ͨ͠ࡍ͸υϝΠϯͷ޲͖ઌΛ S3 ΁

    View full-size slide

  48. 49
    S3 ʹॻ͖ࠐ·ΕΔ CloudFront ͷϩάΛ
    Lambda Ͱύʔεͯ͠ Elasticsearch ΁౤ೖɺ
    kibana ͰՄࢹԽ
    Log S3
    Parse

    View full-size slide

  49. 50
    Ϣʔβʔͷ੠Λड͚औͬͯɺSlack ͱ Backlog ΁౤ߘ
    API Gateway ܦ༝ͷ Lambda ىಈͷ

    ϨΠςϯγʔ͸ؾʹͳΔ

    View full-size slide

  50. 51
    ৼΓฦΓ

    View full-size slide

  51. 52
    ։ൃ଎౓͸ɺ޲্ʁ
    Lambda ͷͨΊͷ։ൃ͸ඞཁͩͬͨ
    ΞϓϦʹԠͨ͡σʔλଆͷॊೈͳมߋ͕Մೳͩͬͨ
    αʔόʔෛՙΛؾʹ͠ͳ͍ͰΑ͘ͳͬͨʢը૾ੜ੒ɺ഑৴ʣ
    ։ൃମ੍ͷԸܙʢݱ৔Ͱٕज़తͳҙࢥܾఆ͕͋Δ͜ͱʣ
    ӡ༻ίετ͸ݮͬͨʢ΄΅ͳ͍ʣ

    View full-size slide

  52. 53
    αʔϏεɾঢ়گʹΑͬͯ͸ɺ୅ସΛҙࣝ͢Δඞཁ͸͋Γͦ͏
    Lambda ؔ਺͸ɺLambda Ҏ֎ͷͲ͜Ͱಈ͔ͤͦ͏ʁ
    S3, CloudFront ͕མͪͨ࣌͸ʁ
    πʔϧ͕ग़ἧͬͯɺݴޠͷબ୒ࢶ΋૿͑ͨ
    Serverless, Apex, Lamvery ͳͲ
    ͲͷαʔϏε΋҆ఆ͍ͯ͠Δ
    ࠓ͸҆৺ͯ͠৸ΒΕΔ

    View full-size slide

  53. 54
    ҙ֎ʹࣗ༝౓͸͋Δ
    OS ͸ Amazon Linux ͳͷͰɺ͜͜Ͱಈ͘΋ͷ͸େମಈ͘
    Linux ༻ͷ cwebp όΠφϦͰ WebP ม׵΋Ͱ͖ͨ
    Lambda ͷͨΊͷ։ൃ͕ඞཁʹͳΔ
    ಛʹɺS3 ͳͲͱ࿈ܞͯ͠ΠϕϯτϕʔεͰॲཧ͢Δ৔߹
    ΠϕϯτͷϧʔςΟϯάɺσΟεύονϟ͕ඞཁʹͳΔ৔߹΋
    Ϣχοτςετ͸Ͱ͖Δ͕ɺࣗಈͰ݁߹ςετ͕೉͍͠
    Lambda ʹ͍ͭͯ

    View full-size slide

  54. 55
    ࡉ͔͍πʔϧΛ࡞Δʹ͸͔ͳΓศར
    Slack ͷ Slash commands ͳͲ (API Gateway ͱͷ૊Έ߹Θͤ)
    ͪΌΜͱͰ͖͍ͯΔ
    ΠϕϯτΛ࿙Βͨ͠Γ͠ͳ͍
    ͔ͳΓ͍҆
    FaaS ͷະདྷΛײ͡Δ
    Lambda ʹ͍ͭͯ

    View full-size slide

  55. 56
    ؆୯ʹ࢖͑Δ PaaS ج൫ͷ४උ͕͋ͬͨ
    Elastic Beanstalk + Docker + Python + Django
    ೝূඞਢɺ༗ྉձһݶఆͷαʔϏεͳͷͰɺݎ࿚Ͱ։ൃମ੍͕
    ཱ͍֬ͯ͠Δํ͕Α͔ͬͨ
    API Gateway + Lambda ͸؅ཧͷίετ͕͔͔Γͦ͏ɺϨΠςϯ
    γʔ͕େ͖͍
    PaaS?

    View full-size slide

  56. 57
    εςʔτϨεΛҙࣝͤ͟ΔΛಘͳ͍
    ෭࡞༻΍ঢ়ଶอ࣋ʹཔΕͳ͍࣮૷্ͷ੍໿
    ີ݁߹ͳίʔυ͕ࣗવʹݮΓɺݟ௨͕͠෇͖΍͘͢ͳΔ
    ϑϩϯτΤϯυͷ։ൃʹ΋ੜ͖ͨ
    αʔόʔϨεͰͷ։ൃΛ௨ͯ͡

    View full-size slide

  57. 59
    Lambda + S3 + CloudWatch Ͱࢴ໘ը૾ͷม׵
    S3 + CloudFront Ͱը૾ͷ഑৴: Private cache distribution
    Lambda Ͱ WebP ͷม׵΍ը૾ॲཧΛߦͬͨ
    ݁Ռͷ଴ͪ߹ΘͤΛ͢ΔͨΊʹ ɺCloudWatch Ͱ cron తʹ
    Lambda Λ࣮ߦ
    ҰͭͷΠϕϯτͰෳ਺ͷ Lambda Λىಈ͢Δ Workaround
    Slack, Backlog, PagerDuty ͳͲͷ SaaS ΋׆༻
    αʔόʔϨεͰखͷ͔͔Βͳ͍ΠϯϑϥΛߴ଎ʹߏஙͰ͖ͨ

    View full-size slide

  58. ࢴ໘ϏϡʔΞʔΛࢧ͑Δ
    αʔόʔϨεΞʔΩςΫνϟ
    ழࣂ େࢤ ೔ຊܦࡁ৽ฉࣾ
    Serverless Conf Tokyo 2016

    View full-size slide