Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ridge GAE/Go みたいなの on AWS

Ridge GAE/Go みたいなの on AWS

FUJIWARA Shunichiro

January 26, 2017
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. Ridge
    GAE/Go Έ͍ͨͳͷ on AWS
    2017.01.26 @fujiwara
    https://www.flickr.com/photos/jimgrant/2339167908

    View Slide

  2. ౻ݪ ढ़Ұ࿠
    @fujiwara
    github.com/fujiwara
    sfujiwara.hatenablog.com
    ٕज़෦

    View Slide

  3. Ridge is ͳʹ
    github.com/fujiwara/ridge
    API Gateway ɹ Lambda
    ɹ Go
    Λ࢖ͬͯ GAE/Go Έ͍ͨͳ΋ͷΛ࡞ΔϥΠϒϥϦ

    View Slide

  4. API Gateway Proxy Integration
    & Lambda ?

    View Slide

  5. API Gateway Proxy Integration & Lambda

    View Slide

  6. API Gateway Proxy Integration ͱ Lambda Λઃఆ͢Δͱ
    GET /api/hello?name=Bob HTTP/1.1
    User-Agent: curl/7.43.0
    !
    {
    "path": "/api/hello",
    "httpMethod": "GET",
    "headers": {
    "User-Agent": "curl/7.43.0",
    ...
    },
    "queryStringParameters": {
    "name": "Bob"
    },
    "body": null
    }
    API Gateway͕͏͚ͨ೚ҙͷHTTPϦΫΤετ͕
    ^ͷΑ͏ͳObjectʹͳͬͯLambdaʹ

    View Slide

  7. Lambda͕ฦͨ͠Object͕ API Gateway ʹΑͬͯ
    HTTPϨεϙϯεʹ
    {
    "statusCode": 200,
    "headers": {
    "Content-Type": "text/plain"
    },
    "body": "Hello Bob."
    }
    !
    HTTP/1.1 200 OK
    Content-Type: text/plain
    Hello Bob.

    View Slide

  8. ͭ·ΓLambdaͰ೚ҙͷWebΞϓϦ͕࡞ΕΔ
    ͚Ͳ
    • ಠࣗObjectΛಡΜͰಠࣗObjectΛฦ͢
    • routing͸… !
    • ςετ͸Ͳ͏΍ͬͯ… !
    ͋ͱ Ͱॻ͖͍ͨ

    View Slide

  9. Lambda Ͱ Go Λಈ͔͢

    View Slide

  10. http://apex.run

    View Slide

  11. ApexΛ࢖༻͢ΔͱɺAWS Lambdaؔ਺Λ؆୯ʹߏஙɺల։ɺ؅
    ཧ͢Δ͜ͱ͕Ͱ͖·͢ɻ ApexΛ࢖༻͢ΔͱɺGolangͳͲɺ
    AWS Lambda͕ωΠςΟϒʹαϙʔτ͍ͯ͠ͳ͍ݴޠΛɺϏϧ
    υʹ஫ೖ͞ΕͨNode.jsγϜΛ࢖༻ͯ͠࢖༻Ͱ͖·͢ɻ ػೳͷς
    ετɺσϓϩΠͷϩʔϧόοΫɺϝτϦοΫͷදࣔɺϩάͷς
    ʔϦϯάɺϏϧυγεςϜ΁ͷϑοΫͳͲɺ͞·͟·ͳϫʔΫ
    ϑϩʔʹؔ࿈͢Δπʔϧ͕ఏڙ͞Ε͍ͯ·͢ɻ (by Google຋༁)

    View Slide

  12. Apex Ͱ Go ͕ಈ͘࢓૊Έ
    GoͷόΠφϦΛಉࠝɺࢠϓϩηεͱͯ͠ىಈ
    STDIN / STDOUTͰϓϩηεؒ௨৴(JSON)

    View Slide

  13. Example
    package main
    type input struct {
    Name string `json:"name"`
    }
    type output struct {
    Message string `json:"message"`
    }
    func main() {
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    var in input
    if err := json.Unmarshal(event, &in); err != nil {
    return nil, err
    }
    out := output{Message: "Hello " + in.Name}
    return out, nil
    })
    }

    View Slide

  14. ؆୯ʹBuild, DeployͰ͖Δ
    $ apex deploy
    Goͷbuild
    zip ࡞੒ͯ͠ Lambda ʹ deploy
    ͜Ε͚ͩ

    View Slide

  15. ! API Gateway ͷ Object ͱ
    Go ͷ net/http Λ૬ޓม׵͢Ε͹͍͍ͷͰ͸
    ͦΕ͕ Ridge
    github.com/fujiwara/ridge

    View Slide

  16. RidgeͰॻ͔ΕͨΞϓϦͷྫ
    import (
    "net/http"
    "github.com/fujiwara/ridge"
    )
    func main() {
    var mux = http.NewServeMux()
    mux.HandleFunc("/hello", handleHello)
    ridge.Run(":8080", "/api", mux)
    }
    func handleHello(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    fmt.Fprintf(w, "Hello %s\n", r.FormValue("name"))
    }
    ridge.Run() Ҏ֎͸ී௨ͷ net/http ΞϓϦέʔγϣϯ

    View Slide

  17. ridge.Run() ͕΍Δ͜ͱ
    API Gateway Proxy Integration ͱ net/http ͷ૬ޓม׵
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    r, _ := NewRequest(event) // API GW ͷ Object Λ *http.Request ʹ͢Δ
    w := NewResponseWriter() // RidgeͷResponseWriter
    mux.ServeHTTP(w, r) // ී௨ʹHTTP Handlerʹ౉ͯ͠ॲཧ
    return w.Response(), nil // API GW͕ཁٻ͢ΔObjectʹม׵ͯ͠ฦ͢
    })

    View Slide

  18. ridge.Run() ͕΍Δ͜ͱ
    Apex(Lambda)Ͱಈ͔͞ͳ͍৔߹͸ී௨ʹhttp.Serverىಈ
    if os.Getenv("APEX_FUNCTION_NAME") != "" {
    apex.HandleFunc(...)
    } else {
    // ApexͰ͸ͳ͍৔߹
    m := http.NewServeMux()
    m.Handle(prefix+"/", http.StripPrefix(prefix, mux))
    log.Println("starting up with local httpd", address)
    log.Fatal(http.ListenAndServe(address, m))
    }

    View Slide

  19. ී௨ͷ net/http ΞϓϦέʔγϣϯ͕
    API Gateway + Lambda Ͱ΋
    net/http.Server Ͱ΋ಈ͘

    View Slide

  20. Ridge ͷ͍͍ͱ͜Ζ

    View Slide

  21. Ridge ͷ͍͍ͱ͜Ζ
    ͘͝ී௨ͷ Go ͷ net/http ΞϓϦέʔγϣϯͰ͋Δ
    • ςετ͸ී௨ʹ httptest Ͱ
    • ϩʔΧϧͰಈ͔ͤ͹ curl / ϒϥ΢βͰಈ࡞֬ೝ
    • API GatewayͰͷੑೳʹ໰୊͕͋ͬͨ৔߹(ޙड़)
    ී௨ʹEC2Ͱಈ͔ͤΔ

    View Slide

  22. Ridge ͷ͍͍ͱ͜Ζ
    API Gateway + Lambda Ͱಈ͘ͷͰ
    • ϦΫΤετ਺ʹԠͯࣗ͡ಈͰεέʔϧ
    • ϦΫΤετ͕ͳ͚Ε͹ ! ͳ͠
    • EC2ͷ؅ཧҰ੾ͳ͠
    • σϓϩΠ΋ apex deploy Ұൃ

    View Slide

  23. ͭ·Γ͜Ε͸
    GAE/Go Έ͍ͨͳ΋ͷʂ

    View Slide

  24. Ridge ͰͰ͖ͳ͍͜ͱ

    View Slide

  25. Ridge ͰͰ͖ͳ͍͜ͱ
    • ಉҰ໊ෳ਺URLҾ਺ (?foo=a&foo=b Έ͍ͨͳͷ)
    API Gateway͔ΒདྷΔ࣌఺Ͱ໊લΛkeyʹͨ͠objectͳͷͰ…
    • ಉҰ໊ෳ਺ HTTP ϔομ
    API Gatewayͷ࣌఺Ͱ(ུ)
    Proxy Integrationͷ࢓༷ʹΑΔ

    View Slide

  26. Ridge ͰͰ͖ͳ͍͜ͱ
    • όΠφϦΛฦ͢ (ը૾ͱ͔)
    API Gateway͸ઌ೔όΠφϦαϙʔτ͚ͨ͠Ͳ
    Proxy Integration͔Βฦͤͳ͍Α͏ͳ…
    ࠓޙʹظ଴

    View Slide

  27. Ridge ͰͰ͖ͳ͍͜ͱ
    • ཪͰ Goroutine Λಈ͔͠ଓ͚Δ
    Lambda͸ϦΫΤετॲཧΛऴ͑Δͱϓϩηεٳ຾
    • 1ϦΫΤετʹ10ඵҎ্ֻ͔Δॲཧ
    API GatewayͷλΠϜΞ΢τ͕ݻఆ
    Long poll / SSE తͳ͜ͱ΋ؚΉ

    View Slide

  28. RidgeͷੑೳධՁ

    View Slide

  29. RidgeͷੑೳධՁ
    RedisʹೖΕͨ༣ศ൪߸DB1 ΛҾ͍ͯJSONΛฦ͢API
    ڞ௨: ElastiCacheRedis (t2.micro)
    1. ALB(https) + EC2(t2.micro)
    2. API Gateway(https) + Lambda (128MB)
    $ curl -s "https://***/api/postal_code?code=1000001"
    {"code":"1000001","addresses":[{"address":"౦ژ౎ઍ୅ా۠ઍ୅ా","yomi":"τ΢Ωϣ΢τνϤμΫνϤμ"}]}
    1 KEN_ALL.csv

    View Slide

  30. 1. ALB + EC2 (t2.micro)
    Running 10s test @ https://***/api/postal_code?code=1000001
    4 threads and 30 connections
    Thread Stats Avg Stdev Max +/- Stdev
    Latency 10.55ms 4.22ms 222.76ms 96.48%
    Req/Sec 662.03 68.84 780.00 76.00%
    Latency Distribution
    50% 9.89ms
    75% 10.82ms
    90% 12.28ms
    99% 19.18ms
    26402 requests in 10.02s, 7.68MB read
    Requests/sec: 2634.14
    Transfer/sec: 784.46KB

    View Slide

  31. 2. API Gateway + Lambda (128MB)
    Running 10s test @ https://***/api/postal_code?code=1000001
    4 threads and 30 connections
    Thread Stats Avg Stdev Max +/- Stdev
    Latency 33.83ms 14.64ms 355.60ms 86.32%
    Req/Sec 190.01 27.50 260.00 70.50%
    Latency Distribution
    50% 31.43ms
    75% 37.48ms
    90% 46.82ms
    99% 81.43ms
    7603 requests in 10.05s, 4.30MB read
    Requests/sec: 756.37
    Transfer/sec: 438.01KB

    View Slide

  32. ൺֱͷͨΊͲͪΒ΋30ฒྻ ݶքੑೳΛܭଌ͍ͯ͠ͳ͍͜ͱʹ஫ҙ
    • EC2, Lambdaͱ΋ΞϓϦࣗମͷॲཧ͸ฏۉ 5msఔ౓
    • API Gateway, Apex, Ridge ͰͷϨΠςϯγ͕߹ܭ20msఔ౓
    (େ෦෼͕API Gateway)
    ฒྻ਺্͕͕Ε͹ req/sec ͸্͕Δ
    API Gateway͸1000req/sec2, Lambda 100ฒྻͷ੍ݶ͋Γ
    (্ݶ؇࿨ਃ੥Մೳ)
    2 ͳͷͰϕϯνͷฒྻ਺Λ্͛੾Ε͍ͯͳ͍

    View Slide

  33. Ridgeʹ޲͍͍ͯͳ͍͜ͱ
    ௿ϨΠςϯγ͕ཁٻ͞ΕΔΞϓϦέʔγϣϯ
    ͨͱ͑͹಺෦޲͚ microservice
    ͦ΋ͦ΋ API Gateway ͕ VPC Ͱಈ͔ͳ͍

    View Slide

  34. Ridge͕(ಛʹ)޲͍͍ͯΔ͜ͱ
    සൟʹΞΫηε͕ͳ͍ / ϨΠςϯγཁٻ͕γϏΞͰͳ͍
    • webhookΛड͚Δ Slack bot
    • S3ͷrepo viewer github.com/fujiwara/ridge-s3viewer
    • αʔϏε͕ऴྃͨ͠ήʔϜͷࠂ஌API
    POSTΛड͚ͯJSONΛฦ͢ͷΛEC2ͳ͠Ͱ
    ΧϠοΫͰ͸ຊ൪APIͱͯ͠౤ೖ࣮੷͋Γ

    View Slide

  35. Ridge(ͱ͍͏͔Lambda)ͷ஫ҙࣄ߲
    • (VPC Lambda ͷ৔߹) ॳճىಈ࣌ʹENIΛׂΓ౰ͯΔͷʹ
    10ඵఔ౓ඞཁ
    • ίϯςφ͕׬શʹࢭ·ͬͯ͠·͏ͱىಈ͕஗͍
    ఆظత(1ʙ5෼ִؒ)ʹ֎ܗ؂ࢹͳͲͰୟ͚͹OK
    • ϝϞϦׂ౰ྔͱCPUੑೳ͕ൺྫ͢Δ
    ࣮ࡍʹϝϞϦΛ࢖Θͳͯ͘΋CPU͕ඞཁͳΒେ͖͘
    (! ΋ൺྫ͠·͢)

    View Slide

  36. ϩάͷऔΓѻ͍

    View Slide

  37. ϩάͷऔΓѻ͍
    go-apexͰ͸࢓૊Έ্ STDOUT ʹ͸Կ΋ग़ྗͰ͖ͳ͍
    STDERR ʹग़ྗͨ͠΋ͷ͸ CloudWatch Logs ʹه࿥͞ΕΔ
    apex logs -f Ͱ΋Ӿཡ(tail)Մೳ

    View Slide

  38. ϩάͷऔΓѻ͍
    CloudWatch Logs ͸͍͍͓஋ஈ + औΓѻ͍͕ͭΒ͍
    → Lambda ʹྲྀ͢͜ͱ͕Ͱ͖Δ
    {
    "awslogs": {
    "data": "H4sIAKmNfFgAA5VQTWvbQBS891cI0aNV7dtv+aYQNRRiWiy1l9iYlfVkBPpwp
    XXTNOS/98lOwRBy6B7eYWb2zcx7DjucJnfA4umI4TK8TYt0t8ryPL3LwkU4PPY4EpxcPYL
    b4XA3DqcjMbF7nOLWdWXl4rGpDhh5nPyuc01/EeZ+RNeRkjMwMYMYdPzw8T4tsrzYVhxLU
    Vd1KS1Ih3W5R+2U2guuoUTNaMV0Kqf92Bx9M/Sfm9bjOIXLh/D+bHlZvrsyJsdwezbOfmH
    vZ+1z2FTkLwQwrY3QVjHNjUgSKbVRUlhrGZdglALOleCCYCsSQzgwAZTBN3Ql7zoqDNJKp
    aRKwDJY/Lserc+LdF0Ea/x5IumXahmANgythKgq9zICQB1ZmySRFMrUNdRaKQx+UB0qtgx
    eL7Lpw5fF28DAjJBAP5nhijHgWnFpaaqEGaNnQicJIzer9fuBxXXgdfbt6/8n3vjb0+j8O
    TN84izopo2/adoWq+CKYTMRbPwKu2F8CvLmDxLKbbC6IdD9Dl6J7xPOxhd8Lr99+fAXpgd
    o/ZQCAAA="
    }
    }
    ͳʹ͜ͷσʔλ !

    View Slide

  39. CloudWatch Logs → Lambda
    JSONจࣈྻΛgzipͯ͠base64ͨ͠΋ͷ͕ྲྀΕͯ͘Δ !
    $ jq -r .awslogs.data | base64 --decode | gzip -dc | jq .
    {
    "messageType": "DATA_MESSAGE",
    "owner": "999999999999",
    "logGroup": "/aws/lambda/ridge-test_main",
    "logStream": "2017/01/16/[$LATEST]d2eb3fdfb4814aefbce6a55c3261be60",
    "subscriptionFilters": [
    "LambdaStream_ridge-test_log"
    ],
    "logEvents": [
    {
    "id": "33106673685062739944675438880241755122532346783978881031",
    "timestamp": 1484554591801,
    "message": "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n"
    },
    {
    "id": "33106673685107341435072500126524826559077643506990841866",
    "timestamp": 1484554591803,
    "message": "REPORT RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e\t
    Duration: 1.20 ms\tBilled Duration: 100 ms \tMemory Size: 128 MB\tMax Memory Used: 18 MB\t\n"
    }
    ]
    }

    View Slide

  40. ridge.DecodeLogStream() Ͱόϥͤ·͢
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    logStream, _ := ridge.DecodeLogStream(event)
    for _, e := range logStream.LogEvents {
    /* e => {
    ID: "33106673685062739944675438880241755122532346783978881031",
    Timestamp: 1484554591801,
    Message: "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n"
    }
    */
    }
    })
    ల։ͨ͠΋ͷΛ Kinesis Streams, Firehose
    ͳͲʹ౤͛Δͷ͕Α͍ͷͰ͸

    View Slide

  41. ·ͱΊ
    API Gateway + Lambda Ͱ GAE/Go Έ͍ͨͳ3 ͜ͱ͕Ͱ͖Δ
    Ridge (github.com/fujiwara/ridge) Λ࡞Γ·ͨ͠
    ༻్ʹΑͬͯ͸طʹ
    3 ಉ͡ͱ͸͍͑·ͤΜ

    View Slide