Upgrade to Pro — share decks privately, control downloads, hide ads and more …

System Integration with Fastly

System Integration with Fastly

Tatsuhiko Kubo

February 20, 2019
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Tatsuhiko Kubo@cubicdaiya
    Fastly Meetup #1 2019/02/20
    System Integration with Fastly

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Engineer, Tech Lead, SRE @ Mercari, Inc.

    View Slide

  3. View Slide

  4. ϝϧΧϦʹ͍ͭͯ
    • ೔ຊ࠷େͷϑϦϚΞϓϦ

    • 3෼Ͱ؆୯ʹग़඼

    • 1) ࣸਅΛࡱΔ

    • 2) ঎඼৘ใΛهೖ

    • 3) ग़඼ϘλϯΛԡ͢

    • ҆৺҆શͳܾࡁɾऔҾ

    • ΤεΫϩʔ

    • ಗ໊഑ૹ

    View Slide

  5. ྦྷܭग़඼਺ͷਪҠ

    View Slide

  6. Fastly products in Mercari
    • Full-Site Delivery

    • ImageOptimizer

    • Web Application Firewall

    • Enterprise Support

    • etc…

    View Slide

  7. Agenda
    • FastlyͱαʔυύʔςΟͷαʔϏεɾπʔϧͷ૊Έ߹ΘͤʹΑΔΠϯςά
    Ϩʔγϣϯ

    • DatadogɺGoogle BigQueryɺetc…

    • Fastly APIʹΑΔΠϯςάϨʔγϣϯ

    • https://docs.fastly.com/api/

    View Slide

  8. FastlyͱαʔυύʔςΟͷαʔϏεɾπʔϧͷ૊Έ߹Θͤ
    • Fastly͸৭ʑͳαʔϏε΍πʔϧͱ૊Έ߹ΘͤΔ͜ͱ͕Մೳ

    • DatadogɺAmazon S3ɺGoogle Cloud StorageɺGoogle BigQueryɺ…

    • ϦϞʔτϩάετϦʔϛϯά

    • https://docs.fastly.com/guides/streaming-logs/

    View Slide

  9. ૊Έ߹ΘͤͷྫʢDatadog Integrationʣ
    +
    https://docs.datadoghq.com/integrations/fastly/

    View Slide

  10. Datadog Integration with Fastly
    • FastlyͷϝτϦΫεΛDatadog্ͰදࣔɺΧελϚΠζͰ͖Δ

    • e.g. hit_ratio, requests, bandwidth, status_4xx, status_5xx, etc…

    • ෳ਺ͷϝτϦΫεΛ૊Έ߹ΘͤͯಠࣗͷϝτϦΫεΛ࡞੒͢Δ͜ͱ΋Մೳ

    • ᮢ஋Λઃఆͯ͠ΞϥʔτΛඈ͹͢͜ͱ΋Ͱ͖Δ

    • Historical Stats APIͷσʔλ͕ͦͷ··Datadog্Ͱѻ͑ΔΠϝʔδ

    View Slide

  11. FastlyͷϝτϦΫεΛDatadog্ͰදࣔɺΧελϚΠζ

    View Slide

  12. HTTP/2 Ratio
    (http2 ÷ requests) × 100

    View Slide

  13. Ωϟογϡώοτ཰ͷܭࢉʢShielding͕༗ޮͳ৔߹ʣ
    Hit Ratio(True) = (1 − miss − shield
    requests − shield
    ) × 100
    miss: Number of cache misses

    shield: number of requests from Shield to Origin

    requests: Number of Requests Processed
    The truth about cache hit ratios: https://www.fastly.com/blog/truth-about-cache-hit-ratios

    View Slide

  14. Ωϟογϡώοτ཰ͷܭࢉʢShielding͕༗ޮͳ৔߹ʣ
    Hit Ratio(True) = (1 − miss − shield
    requests − shield
    ) × 100

    View Slide

  15. Origin Shield
    • Edge POPͱOriginͷதؒʹ഑ஔ͢ΔPOP

    • Edge POPͷΩϟογϡʹώοτ͠ͳ͔ͬͨ෼ΛΧόʔ

    • Ωϟογϡώοτ཰ͷେ͖ͳ޲্͕ݟࠐΊΔ

    • Documents

    • https://docs.fastly.com/ja/guides/performance-tuning/shielding

    • hit_ratio͸Edge POPͷΈͷΩϟογϡώοτ཰

    View Slide

  16. Shieldingͷon/offͰӨڹΛड͚ΔϝτϦΫεͷྫ
    • hit_ratio
    • only Edge POP

    • requests
    • involve shield
    • bandwidth
    • beresp_header_bytes + beresp_body_bytes +
    resp_header_bytes + resp_body_bytes
    • resp_header_bytes, resp_body_bytes
    • involve shield_header_bytes, shield_body_bytes

    View Slide

  17. ૊Έ߹ΘͤͷྫʢGoogle BigQuery & DataStudioʣ
    BigQuery DataStudio
    real-time
    streaming
    σʔλιʔε
    ͱͯ͠ར༻

    View Slide

  18. ϦϞʔτϩάετϦʔϛϯά
    • Amazon S3΍Google Cloud StorageɺGoogle BigQueryΛ͸͡Ίɺ
    ৭ʑͳαʔϏεʹϩάΛసૹՄೳ

    • Syslog΋Մೳ

    • Datadog IntegrationΑΓ΋खؒ͸ଟ͍͚Ͳɺॊೈੑ͸ߴ͍

    • ϩάͷεΩʔϚʹVCLͷม਺͕ར༻Ͱ͖Δ

    View Slide

  19. Fastly APIʹΑΔIntegration

    View Slide

  20. Fastly API
    • FastlyͷػೳΛRESTfulͳAPIӽ͠ʹར༻Ͱ͖Δ

    • e.g. PurgeɺStatsɺConfigurationɺWAFɺetc…

    • Documents

    • https://docs.fastly.com/api/

    • ػೳʹΑͬͯ͸API͔Β͔͠ར༻Ͱ͖ͳ͍΋ͷ΋͋Δ

    • όʔδϣϯͷϩοΫɺWAF౳

    View Slide

  21. Fastly APIΛར༻͢Δʹ͸
    • ϙʔλϧ্ͰAPIτʔΫϯΛൃߦ

    • ༗ޮൣғʢର৅ͱͳΔαʔϏεʣ΍ظݶɺݖݶʢRead, Write౳ʣΛઃఆ

    • Datadog Integration΋APIτʔΫϯΛൃߦ͢Δ͜ͱͰར༻Մೳ

    View Slide

  22. curlͰFastly APIΛୟ͘
    $ curl \
    -X GET
    -H ‘Fastly-Key: xxx’ \
    -H ‘Accept: application/json’ \
    https://api.fastly.com/…

    View Slide

  23. Fastly APIʹΑΔΦϖϨʔγϣϯ
    • Pros

    • curlͰૢ࡞Ͱ͖ͯศར

    • Cons

    • ֮͑ΒΕͳ͍ͷͰຖճެࣜυΩϡϝϯτݟͳ͕ΒAPIୟ͍ͯΔ

    • APIΫϥΠΞϯτΛॻ͘͜ͱʹͨ͠

    View Slide

  24. mfc

    View Slide

  25. mfc
    • In-house Fastly CLI at Mercari

    • GoͰ࣮૷

    • ओʹACL΍WAFܥͷΦϖϨʔγϣϯͰͨ·ʹ࢖͏

    View Slide

  26. mfc configuration
    $ cat ~/.fastly/conf.toml
    [target]
    service = “service-A”
    [[services]]
    service = “service-A”
    apikey = “…”
    waf = “…”
    [[services]]
    service = “service-B”
    apikey = “…”

    View Slide

  27. Usage of mfc

    $ mfc
    Usage of mfc:
    config
    the utility for mfc configuration
    service
    the utility for fastly service
    acl
    the utility for fastly ACL
    waf
    the utlity fro fastly WAF
    (etc…)
    • ػೳྖҬຖʹαϒίϚϯυΛఆٛ

    • ACL, Service, Version౳

    • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹

    switch args[1] {
    case “config”:
    return config.NewCLI().Run(args)
    case “service”:
    return service.NewCLI().Run(args)
    case “acl”:
    return acl.NewCLI().Run(args)
    case “…”

    }
    ಈ࡞Πϝʔδ

    View Slide

  28. ACL operation
    $ mfc acl show | jq -r ‘.[].name’
    whitelist
    blacklist

    $ mfc acl list -name whitelist
    $ mfc acl add -name whitelist \
    -ip x.x.x.x/32 \
    -comment “Added x to whitelist”
    ■ACLͷҰཡΛྻڍ
    ■ACLΤϯτϦͷҰཡΛྻڍ
    ■ACLʹΤϯτϦΛ௥Ճ
    $ mfc acl del -name blacklist \
    -entry-id xxx
    ■ACL͔ΒΤϯτϦΛ࡟আ
    ■ACLΛ࡞੒
    $ mfc acl create -name whitelist -version 10
    ■ ACLʹσʔλΛಉظ
    $ mfc acl sync -name blacklist \
    -provider blacklist.json

    View Slide

  29. WAF operation
    $ mfc waf list

    $ mfc acl -h
    Usage of waf:
    mfc waf list
    list all active waf objects
    mfc waf rule show
    show waf rule
    mfc waf rule status
    show and change waf rule status
    mfc waf rule vcl
    show waf rule vcl
    mfc waf ruleset show
    show waf ruleset
    mfc waf ruleset update
    update waf ruleset
    $ mfc waf rule show -id rule_id
    $ mfc waf rule status -id rule_id
    ■ WAF ObjectͷҰཡΛྻڍ
    ■ WAF Ruleͷ֓ཁΛ֬ೝ
    ■ WAF RuleͷεςʔλεΛ֬ೝ
    ■ Help
    $ mfc waf rule status -id rule_id -set disabled
    ■ WAF RuleͷεςʔλεΛdisabledʹมߋ
    ■ WAF Ruleͷ࣮૷(VCL)ΛಡΉ
    $ mfc waf rule vcl -id rule_id

    View Slide

  30. ActiveͳόʔδϣϯΛநग़͢Δ
    $ mfc service versions
    latest: 91
    active: 90

    View Slide

  31. ActiveͳόʔδϣϯΛநग़͢Δ
    • GET /service/service_id/version

    • ֘౰αʔϏεͷ͢΂ͯͷόʔδϣϯʹؔ͢Δ৘ใΛฦ͢

    • active?, locked?, comment, number, created_at, updated_at, …
    • refs -> https://docs.fastly.com/api/config#version

    View Slide

  32. ActiveͳόʔδϣϯΛநग़͢Δ
    • ConfigurationܥͷAPI͸όʔδϣϯͷࢦఆΛཁٻ͢Δ΋ͷ͕ଟ͍

    • e.g. ACL

    • GET /service/service_id/version/version/acl

    • mfcʹΑΔૢ࡞͸activeͳόʔδϣϯʹରͯ͠ߦ͏΋ͷ͕ଟ͍

    • ຖճactiveͳόʔδϣϯͲΕ͚ͩͬʁΈ͍ͨͳࣄଶ͸໘౗ͳͷͰආ͚͍ͨ

    • e.g. mfc acl show ͸activeͳόʔδϣϯΛࣗಈͰऔಘ࣮ͯ͠ߦ͞ΕΔ

    View Slide

  33. References
    • API Client librariesʢAPIͷΫϥΠΞϯτ΍ϥΠϒϥϦͷϦετʣ

    • https://docs.fastly.com/api/clients

    • waflyctlʢFastly WAF CLIʣ

    • https://github.com/fastly/waflyctl

    View Slide