System Integration with Fastly

The slide for Fastly Meetup #1

https://fastly-community.connpass.com/event/119123/

Transcript

1. Tatsuhiko Kubo@cubicdaiya
   Fastly Meetup #1 2019/02/20
   System Integration with Fastly
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Engineer, Tech Lead, SRE @ Mercari, Inc.
   

   View Slide

3. View Slide

4. ϝϧΧϦʹ͍ͭͯ
   • ೔ຊ࠷େͷϑϦϚΞϓϦ
   
   • 3෼Ͱ؆୯ʹग़඼
   
   • 1) ࣸਅΛࡱΔ
   
   • 2) ঎඼৘ใΛهೖ
   
   • 3) ग़඼ϘλϯΛԡ͢
   
   • ҆৺҆શͳܾࡁɾऔҾ
   
   • ΤεΫϩʔ
   
   • ಗ໊഑ૹ
   

   View Slide

5. ྦྷܭग़඼਺ͷਪҠ
   

   View Slide

6. Fastly products in Mercari
   • Full-Site Delivery
   
   • ImageOptimizer
   
   • Web Application Firewall
   
   • Enterprise Support
   
   • etc…
   

   View Slide

7. Agenda
   • FastlyͱαʔυύʔςΟͷαʔϏεɾπʔϧͷ૊Έ߹ΘͤʹΑΔΠϯςά
   Ϩʔγϣϯ
   
   • DatadogɺGoogle BigQueryɺetc…
   
   • Fastly APIʹΑΔΠϯςάϨʔγϣϯ
   
   • https://docs.fastly.com/api/
   

   View Slide

8. FastlyͱαʔυύʔςΟͷαʔϏεɾπʔϧͷ૊Έ߹Θͤ
   • Fastly͸৭ʑͳαʔϏε΍πʔϧͱ૊Έ߹ΘͤΔ͜ͱ͕Մೳ
   
   • DatadogɺAmazon S3ɺGoogle Cloud StorageɺGoogle BigQueryɺ…
   
   • ϦϞʔτϩάετϦʔϛϯά
   
   • https://docs.fastly.com/guides/streaming-logs/
   

   View Slide

9. ૊Έ߹ΘͤͷྫʢDatadog Integrationʣ
   +
   https://docs.datadoghq.com/integrations/fastly/
   

   View Slide

10. Datadog Integration with Fastly
    • FastlyͷϝτϦΫεΛDatadog্ͰදࣔɺΧελϚΠζͰ͖Δ
    
    • e.g. hit_ratio, requests, bandwidth, status_4xx, status_5xx, etc…
    
    • ෳ਺ͷϝτϦΫεΛ૊Έ߹ΘͤͯಠࣗͷϝτϦΫεΛ࡞੒͢Δ͜ͱ΋Մೳ
    
    • ᮢ஋Λઃఆͯ͠ΞϥʔτΛඈ͹͢͜ͱ΋Ͱ͖Δ
    
    • Historical Stats APIͷσʔλ͕ͦͷ··Datadog্Ͱѻ͑ΔΠϝʔδ
    

    View Slide

11. FastlyͷϝτϦΫεΛDatadog্ͰදࣔɺΧελϚΠζ
    

    View Slide

12. HTTP/2 Ratio
    (http2 ÷ requests) × 100
    

    View Slide

13. Ωϟογϡώοτ཰ͷܭࢉʢShielding͕༗ޮͳ৔߹ʣ
    Hit Ratio(True) = (1 − miss − shield
    requests − shield
    ) × 100
    miss: Number of cache misses
    
    shield: number of requests from Shield to Origin
    
    requests: Number of Requests Processed
    The truth about cache hit ratios: https://www.fastly.com/blog/truth-about-cache-hit-ratios
    

    View Slide

14. Ωϟογϡώοτ཰ͷܭࢉʢShielding͕༗ޮͳ৔߹ʣ
    Hit Ratio(True) = (1 − miss − shield
    requests − shield
    ) × 100
    

    View Slide

15. Origin Shield
    • Edge POPͱOriginͷதؒʹ഑ஔ͢ΔPOP
    
    • Edge POPͷΩϟογϡʹώοτ͠ͳ͔ͬͨ෼ΛΧόʔ
    
    • Ωϟογϡώοτ཰ͷେ͖ͳ޲্͕ݟࠐΊΔ
    
    • Documents
    
    • https://docs.fastly.com/ja/guides/performance-tuning/shielding
    
    • hit_ratio͸Edge POPͷΈͷΩϟογϡώοτ཰
    

    View Slide

16. Shieldingͷon/offͰӨڹΛड͚ΔϝτϦΫεͷྫ
    • hit_ratio
    • only Edge POP
    
    • requests
    • involve shield
    • bandwidth
    • beresp_header_bytes + beresp_body_bytes +
    resp_header_bytes + resp_body_bytes
    • resp_header_bytes, resp_body_bytes
    • involve shield_header_bytes, shield_body_bytes
    

    View Slide

17. ૊Έ߹ΘͤͷྫʢGoogle BigQuery & DataStudioʣ
    BigQuery DataStudio
    real-time
    streaming
    σʔλιʔε
    ͱͯ͠ར༻
    

    View Slide

18. ϦϞʔτϩάετϦʔϛϯά
    • Amazon S3΍Google Cloud StorageɺGoogle BigQueryΛ͸͡Ίɺ
    ৭ʑͳαʔϏεʹϩάΛసૹՄೳ
    
    • Syslog΋Մೳ
    
    • Datadog IntegrationΑΓ΋खؒ͸ଟ͍͚Ͳɺॊೈੑ͸ߴ͍
    
    • ϩάͷεΩʔϚʹVCLͷม਺͕ར༻Ͱ͖Δ
    

    View Slide

19. Fastly APIʹΑΔIntegration
    

    View Slide

20. Fastly API
    • FastlyͷػೳΛRESTfulͳAPIӽ͠ʹར༻Ͱ͖Δ
    
    • e.g. PurgeɺStatsɺConfigurationɺWAFɺetc…
    
    • Documents
    
    • https://docs.fastly.com/api/
    
    • ػೳʹΑͬͯ͸API͔Β͔͠ར༻Ͱ͖ͳ͍΋ͷ΋͋Δ
    
    • όʔδϣϯͷϩοΫɺWAF౳
    

    View Slide

21. Fastly APIΛར༻͢Δʹ͸
    • ϙʔλϧ্ͰAPIτʔΫϯΛൃߦ
    
    • ༗ޮൣғʢର৅ͱͳΔαʔϏεʣ΍ظݶɺݖݶʢRead, Write౳ʣΛઃఆ
    
    • Datadog Integration΋APIτʔΫϯΛൃߦ͢Δ͜ͱͰར༻Մೳ
    

    View Slide

22. curlͰFastly APIΛୟ͘
    $ curl \
    -X GET
    -H ‘Fastly-Key: xxx’ \
    -H ‘Accept: application/json’ \
    https://api.fastly.com/…
    

    View Slide

23. Fastly APIʹΑΔΦϖϨʔγϣϯ
    • Pros
    
    • curlͰૢ࡞Ͱ͖ͯศར
    
    • Cons
    
    • ֮͑ΒΕͳ͍ͷͰຖճެࣜυΩϡϝϯτݟͳ͕ΒAPIୟ͍ͯΔ
    
    • APIΫϥΠΞϯτΛॻ͘͜ͱʹͨ͠
    

    View Slide

24. mfc
    

    View Slide

25. mfc
    • In-house Fastly CLI at Mercari
    
    • GoͰ࣮૷
    
    • ओʹACL΍WAFܥͷΦϖϨʔγϣϯͰͨ·ʹ࢖͏
    

    View Slide

26. mfc configuration
    $ cat ~/.fastly/conf.toml
    [target]
    service = “service-A”
    [[services]]
    service = “service-A”
    apikey = “…”
    waf = “…”
    [[services]]
    service = “service-B”
    apikey = “…”
    

    View Slide

27. Usage of mfc
    •
    $ mfc
    Usage of mfc:
    config
    the utility for mfc configuration
    service
    the utility for fastly service
    acl
    the utility for fastly ACL
    waf
    the utlity fro fastly WAF
    (etc…)
    • ػೳྖҬຖʹαϒίϚϯυΛఆٛ
    
    • ACL, Service, Version౳
    
    • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹
    
    switch args[1] {
    case “config”:
    return config.NewCLI().Run(args)
    case “service”:
    return service.NewCLI().Run(args)
    case “acl”:
    return acl.NewCLI().Run(args)
    case “…”
    …
    }
    ಈ࡞Πϝʔδ
    

    View Slide

28. ACL operation
    $ mfc acl show | jq -r ‘.[].name’
    whitelist
    blacklist
    …
    $ mfc acl list -name whitelist
    $ mfc acl add -name whitelist \
    -ip x.x.x.x/32 \
    -comment “Added x to whitelist”
    ■ACLͷҰཡΛྻڍ
    ■ACLΤϯτϦͷҰཡΛྻڍ
    ■ACLʹΤϯτϦΛ௥Ճ
    $ mfc acl del -name blacklist \
    -entry-id xxx
    ■ACL͔ΒΤϯτϦΛ࡟আ
    ■ACLΛ࡞੒
    $ mfc acl create -name whitelist -version 10
    ■ ACLʹσʔλΛಉظ
    $ mfc acl sync -name blacklist \
    -provider blacklist.json
    

    View Slide

29. WAF operation
    $ mfc waf list
    …
    $ mfc acl -h
    Usage of waf:
    mfc waf list
    list all active waf objects
    mfc waf rule show
    show waf rule
    mfc waf rule status
    show and change waf rule status
    mfc waf rule vcl
    show waf rule vcl
    mfc waf ruleset show
    show waf ruleset
    mfc waf ruleset update
    update waf ruleset
    $ mfc waf rule show -id rule_id
    $ mfc waf rule status -id rule_id
    ■ WAF ObjectͷҰཡΛྻڍ
    ■ WAF Ruleͷ֓ཁΛ֬ೝ
    ■ WAF RuleͷεςʔλεΛ֬ೝ
    ■ Help
    $ mfc waf rule status -id rule_id -set disabled
    ■ WAF RuleͷεςʔλεΛdisabledʹมߋ
    ■ WAF Ruleͷ࣮૷(VCL)ΛಡΉ
    $ mfc waf rule vcl -id rule_id
    

    View Slide

30. ActiveͳόʔδϣϯΛநग़͢Δ
    $ mfc service versions
    latest: 91
    active: 90
    

    View Slide

31. ActiveͳόʔδϣϯΛநग़͢Δ
    • GET /service/service_id/version
    
    • ֘౰αʔϏεͷ͢΂ͯͷόʔδϣϯʹؔ͢Δ৘ใΛฦ͢
    
    • active?, locked?, comment, number, created_at, updated_at, …
    • refs -> https://docs.fastly.com/api/config#version
    

    View Slide

32. ActiveͳόʔδϣϯΛநग़͢Δ
    • ConfigurationܥͷAPI͸όʔδϣϯͷࢦఆΛཁٻ͢Δ΋ͷ͕ଟ͍
    
    • e.g. ACL
    
    • GET /service/service_id/version/version/acl
    
    • mfcʹΑΔૢ࡞͸activeͳόʔδϣϯʹରͯ͠ߦ͏΋ͷ͕ଟ͍
    
    • ຖճactiveͳόʔδϣϯͲΕ͚ͩͬʁΈ͍ͨͳࣄଶ͸໘౗ͳͷͰආ͚͍ͨ
    
    • e.g. mfc acl show ͸activeͳόʔδϣϯΛࣗಈͰऔಘ࣮ͯ͠ߦ͞ΕΔ
    

    View Slide

33. References
    • API Client librariesʢAPIͷΫϥΠΞϯτ΍ϥΠϒϥϦͷϦετʣ
    
    • https://docs.fastly.com/api/clients
    
    • waflyctlʢFastly WAF CLIʣ
    
    • https://github.com/fastly/waflyctl
    

    View Slide