CDNの使い方 in Mercari/CDN in Mercari

CDN in Mercari/CDNの使い方 in Mercari
Fastly Yamagoya Meetup 2017

Transcript

1. CDN in Mercari
   Fastly Yamagoya Meetup 2017
   Masahiro Nagano @kazeburo
   

   View Slide

2. Me
   • Masahiro Nagano / ௕໺խ޿
   • @kazeburo
   • Mercari, Inc

   Principal Engineer

   Site Reliability Engineering (SRE) Team
   

   View Slide

3. Agenda
   • ϝϧΧϦͷ঺հ
   • ઌ೔ͷݸਓ৘ใྲྀग़Πϯγσϯτʹ͍ͭͯ
   • ϝϧΧϦͷΠϯϑϥετϥΫνϟͱCDN
   • ࠓޙͷ՝୊ɾऔΓ૊Έ
   

   View Slide

4. Mercari
   • ϑϦϚΞϓϦ
   • εϚϗͰࣸਅΛͱͬͯ؆୯ʹग़඼
   • ҆৺ɾ҆શͳܾࡁ
   • ศརͳ഑ૹ
   

   View Slide

5. Mercari
   μ΢ϯϩʔυ਺
   ྲྀ௨ֹ
   7500ສDL(JP+US)
   ݄ؒ100ԯԁҎ্
   ग़඼਺ 1೔100ສ඼Ҏ্
   

   View Slide

6. γεςϜ͔ΒΈͨϝϧΧϦ
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   IAM Add-on Example:
   IAM Add-on
   ence
   )
   Assignment/
   Task
   Requester
   Workers
   ग़඼!
   DB Search
   5-දࣔ
   ݕࡧ൓ө
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corporate
   data center
   Traditional
   server
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Amazon Mechanical Turk
   On-Demand Workforce
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   େྔͷϦΫΤετ
   ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
   User Users Client Multimedia Corp
   data c
   Mobile Client
   Internet AWS Management
   Console
   IAM Add-on Example:
   IAM Add-on
   Human Intelligence
   Tasks (HIT)
   Assignment/
   Task
   Requester
   Workers
   Amazon
   Mechanical Turk
   Non-Service Specific
   ϦΫΤετԠ౴
   DB Search
   ߪೖ!
   ϛϦඵʙ30ඵ ਺ඵʙ
   ߴ଎ʹେྔͷτϥϯβΫγϣϯΛѻ͏
   ը૾ ܾࡁ
   AI
   

   View Slide

7. Mercari
   JP US UK
   JP/US/UKͰల։த
   

   View Slide

8. CDN੾Γସ͑ʹ൐͏
   Web൛ϝϧΧϦʹ͓͚Δݸਓ৘ใྲྀग़

   Πϯγσϯτʹ͍ͭͯ
   

   View Slide

9. ଟ͘ͷօ༷ʹ͝৺഑ɺ͝໎࿭Λֻ͓͚͍ͨ͠·ͨ͠
   ਂ͓͘࿳ͼਃ͋͛͠·͢
   ଟେͳΔڠྗΛFastly༷ʹ͍͖ͨͩ·ͨ͠ɻվΊ͓ͯྱਃ্͛͠·͢
   

   View Slide

10. ൃੜͨ͠ࣄ৅
    • ϝϧΧϦWeb൛΁ͷΞΫηε଎౓޲্ͱɺηΩϡϦςΟ޲্ͷͨΊɺCDN
    ͷ੾Γସ͑Λߦ͍·ͨ͠ɻͦͷࡍʹ੾Γସ͑ઌͷCDN(Fastly)ʹ͓͚Δ
    Ωϟογϡͷಈ࡞ʹ͍ͭͯͷ೺Ѳ͕ෆ଍͓ͯ͠Γɺ͓٬͞·΁ͷϨεϙϯ
    ε͕ผͷ͓٬͞·ʹҙਤͤͣදࣔ͞Εɺ݁Ռͱͯ͠ݸਓΛಛఆͰ͖Δ৘ใ
    ΛؚΉ಺༰͕ຊਓҎ֎ʹӾཡ͞ΕΔঢ়ଶͱͳΓ·ͨ͠
    

    View Slide

11. Timeline
    • 6/22
    • 9:41ɹ CDNͷ੾Γସ͑Λ࣮ࢪʢ໰୊ൃੜʣ
    • 14:41ɹΧελϚʔαϙʔτʹ͓ͯ٬͞·͔Βͷ໰͍߹ΘͤΛ֬ೝ͠ɺࣾ಺΁ใࠂ
    • 15:05ɹCDNͷ੾Γସ͑Λதࢭ͠ɺैདྷͷCDN΁໭͢
    • 15:16ɹWeb൛ͷϝϧΧϦΛϝϯςφϯεϞʔυ΁੾Γସ͑
    • 15:38ɹCDNͷ഑৴ઃఆΛdeactivate͠ɺΞΫηεΛःஅ
    • 15:47ɹWeb൛ͷϝϧΧϦϝϯςφϯεϞʔυΛऴྃ
    • 17:55 ίʔϙϨʔταΠτʹ͓஌ΒͤΛܝࡌ
    • 20:45 Tech blogʹͯৄࡉެ։
    

    View Slide

12. Timeline
    • 6/28
    • ഑৴ઃఆΛ࠶Activate
    • Cacheͷ׬શແޮԽઃఆΛߦ͍ɺFastlyࣾͷΤϯδχΞʹϨϏϡʔΛґཔ
    • nginxͷઃఆΛߋ৽͠ɺcacheʹؔ͢ΔϔομΛมߋ
    • ࣾ಺Ͱݕূ
    • 6/29
    • FastlyࣾΤϯδχΞ଴ػͷ΋ͱɺ੾Γସ͑Λ࣮ࢪ
    

    View Slide

13. CacheແޮԽ
    • CacheΛແޮԽ͢ΔͨΊʹ͸ “Cache-Control: private” ΋͘͠͸ "Set-Cookie"
    ͕ඞཁ
    • ”Cache-Control: no-cache” ΍ “no-store” ͸ແࢹ͞ΕΔ
    • Expiresϔομ΋ར༻͞ΕΔ͕ɺ೔෇ͷղऍʹࣦഊ͋Δ͍͸աڈ೔෇ͷ৔߹
    ͸ “0ඵ” ͱͯ͠ѻΘΕΔ
    • 0ඵ͸cacheແޮͰ͸ͳ͘ʮ0ඵͷcache͕ଘࡏ͢Δʯ
    

    View Slide

14. 0ඵͷcache
    • CDN͔ΒΦϦδϯ΁ͷϦΫΤετͷॲཧதʹɺಉ͡
    URLʹରͯ͠ϦΫΤετ͕ൃੜ͢Δͱɺ࠷ॳͷϨεϙ
    ϯεΛ଴ͬͯɺ2ͭ໨Ҏ߱ͷϦΫΤετʹ΋ಉ͡Ϩε
    ϙϯε͕ฦ͞ΕΔ
    • `Request collapsing`
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    ©2011 Amazon Web Services LLC or its affiliates. All r
    User Users Client Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    origin
    CDN
    (1)
    (2)
    (3)
    (4)
    (5) (5)
    

    View Slide

15. ରࡦ
    • VCLͷઃఆมߋ
    • ෳ਺ͷbackendʹରԠ͢ΔͨΊɺPriorityΛ௿͘ઃఆ
    # ͜͜ʹbackend੾Γସ͑ͷvcl͕ඳ͍ͯ͋Δ
    # Request Condition: always Prio: 100
    if( req.url ) { ## ඞͣtrueʹͳΔ
    if (!req.http.Fastly-FF) {
    if (req.http.X-Forwarded-For) {
    set req.http.Fastly-Temp-XFF = req.http.X-Forwarded-For ", " client.ip;
    } else {
    set req.http.Fastly-Temp-XFF = client.ip
    }
    } else {
    set req.http.Fastly-Temp-XFF = req.http.X-Forwarded-For;
    }
    return(pass);
    }
    

    View Slide

16. ରࡦ
    • ϩάΛS3ʹૹ৴͠ɺlambdaʹΑΓϦΞϧλΠϜʹղੳɺmackerelͰՄࢹԽɻ
    ؂ࢹ
    • nginx/ApacheʹͯheaderΛ௥Ճ
    • header΍VCLͷมߋͷ؂ࢹ
    more_clear_headers 'Expires';
    more_set_headers "Cache-Control: private, no-cache, no-store, must-revalidate" "Pragma: no-cache";
    add_header Set-Cookie "merCtx=\"\"; HttpOnly" always;
    

    View Slide

17. https://docs.fastly.com/ja/guides/tutorials/cache-control-tutorial
    

    View Slide

18. ϝϧΧϦͷΠϯϑϥετϥΫνϟͱCDN
    

    View Slide

19. CDN in Mercari
    • ΑΓଟ͘ͷτϥϑΟοΫΛॲཧ͢ΔͨΊ
    • Πϯϑϥӡ༻ίετͷ࡟ݮͱͯ͠CDNΛར༻
    • ը૾഑৴/ಈը഑৴
    • UX/ηΩϡϦςΟͷվળͷͨΊ
    • Clientͱͷ௨৴؀ڥ࠷దԽ
    • αΠτ/APIΛؙ͝ͱCDNԽ
    

    View Slide

20. Infrastructure
    US UK
    ͦΕͧΕͷҬ಺ͷDCΛར༻
    S3 S3 S3
    ঎඼ը૾ ঎඼ը૾ ঎඼ը૾
    Frontend for Backend

    microservices
    Backend API
    JP
    

    View Slide

21. Frontend for Backend/microservices
    API Gateway
    ©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
    User Users Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Internet AWS Management
    Console
    IAM Add-on Example:
    IAM Add-on
    Non-Service Specific
    search
    personalization
    offer
    gRPC
    JSON over HTTPs
    

    View Slide

22. App Backend Infrastructure
    JP US UK
    API͸ͦΕͧΕผυϝΠϯ / CDN͸࢖༻ͯ͠ͳ͍(2017/10ݱࡏ)
    େن໛ͳτϥϑΟοΫͱͳΔ঎඼ը૾͸CDNΛར༻ͯ͠഑৴
    Client Multimedia Corporate
    data center
    Traditional
    server
    Mobile Client
    Assignment/
    Task
    Requester
    Workers
    User Users Client Multimedia Corporate
    data center
    Traditio
    serve
    Mobile Client
    Amazon Mechanical Turk
    On-Demand Workforce
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requester
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    User Users Client Multimedia
    Mobile Client
    Amazon Mechanical Turk
    On-Demand Workforce
    Human Intelligence
    Tasks (HIT)
    Assignment/
    Task
    Requ
    Workers
    Amazon
    Mechanical Turk
    Non-Service Specific
    api.example.jp api.example.com api.example.uk
    S3 S3 S3
    KQ
    DEOJ[FOFU
    VTDEOJ[FOFU
    VLDEOJ[FOFU
    CDN
    CDN CDN
    *N
    BHF'MVY
    

    View Slide

23. ImageFlux
    • ͘͞ΒΠϯλʔωοτͱpixivͷڠۀ
    • ը૾ม׵ + Cache Storage
    • WebP΁ͷରԠ
    • CDNͷCache Hit Rateվળ
    

    View Slide

24. Web Backend Infrastructure
    JP US UK
    Web͸γϯάϧυϝΠϯɻURIͰࢀর͢ΔBackendΛ੾Γସ͑Δ
    ͓٬༷͔Β࠷΋͍ۙEdgeͰ൑அ͢Δ͜ͱͰϨεϙϯελΠϜ/UX޲্
    S3 S3 S3
    https://www.mercari.com/
    /jp/ / /uk/
    

    View Slide

25. ࠓޙͷ՝୊ɾऔΓ૊Έ
    

    View Slide

26. ࠓޙͷ՝୊ɾऔΓ૊Έ
    • Security΁ͷϑΥʔΧε
    • L3-L7·Ͱେن໛ͳDoS߈ܸ͔Βͷ๷ޚ
    • WAF ͷݕূ
    • Bot੍ޚ
    • ߴ౓Խ͢ΔEdge Computing
    • Microservices Λ࣮ݱ͢Δखஈͱͯ͠ͷCDN
    • CDNͷઃఆ/VCLͷContinuous Integration
    • ίετ࠷దԽ
    

    View Slide

27. Ҏ্
    Fastlyͷօ༷ɺࠓޙͱ΋ΑΖ͓͘͠ئ͍͠·͢
    => www.mercari.com/jp/jobs/
    

    View Slide