Ridge GAE/Go みたいなの on AWS

Transcript

1. Ridge
   GAE/Go Έ͍ͨͳͷ on AWS
   2017.01.26 @fujiwara
   https://www.flickr.com/photos/jimgrant/2339167908
   

   View Slide

2. ౻ݪ ढ़Ұ࿠
   @fujiwara
   github.com/fujiwara
   sfujiwara.hatenablog.com
   ٕज़෦
   

   View Slide

3. Ridge is ͳʹ
   github.com/fujiwara/ridge
   API Gateway ɹ Lambda
   ɹ Go
   Λ࢖ͬͯ GAE/Go Έ͍ͨͳ΋ͷΛ࡞ΔϥΠϒϥϦ
   

   View Slide

4. API Gateway Proxy Integration
   & Lambda ?
   

   View Slide

5. API Gateway Proxy Integration & Lambda
   

   View Slide

6. API Gateway Proxy Integration ͱ Lambda Λઃఆ͢Δͱ
   GET /api/hello?name=Bob HTTP/1.1
   User-Agent: curl/7.43.0
   !
   {
   "path": "/api/hello",
   "httpMethod": "GET",
   "headers": {
   "User-Agent": "curl/7.43.0",
   ...
   },
   "queryStringParameters": {
   "name": "Bob"
   },
   "body": null
   }
   API Gateway͕͏͚ͨ೚ҙͷHTTPϦΫΤετ͕
   ^ͷΑ͏ͳObjectʹͳͬͯLambdaʹ
   

   View Slide

7. Lambda͕ฦͨ͠Object͕ API Gateway ʹΑͬͯ
   HTTPϨεϙϯεʹ
   {
   "statusCode": 200,
   "headers": {
   "Content-Type": "text/plain"
   },
   "body": "Hello Bob."
   }
   !
   HTTP/1.1 200 OK
   Content-Type: text/plain
   Hello Bob.
   

   View Slide

8. ͭ·ΓLambdaͰ೚ҙͷWebΞϓϦ͕࡞ΕΔ
   ͚Ͳ
   • ಠࣗObjectΛಡΜͰಠࣗObjectΛฦ͢
   • routing͸… !
   • ςετ͸Ͳ͏΍ͬͯ… !
   ͋ͱ Ͱॻ͖͍ͨ
   

   View Slide

9. Lambda Ͱ Go Λಈ͔͢
   

   View Slide

10. http://apex.run
    

    View Slide

11. ApexΛ࢖༻͢ΔͱɺAWS Lambdaؔ਺Λ؆୯ʹߏஙɺల։ɺ؅
    ཧ͢Δ͜ͱ͕Ͱ͖·͢ɻ ApexΛ࢖༻͢ΔͱɺGolangͳͲɺ
    AWS Lambda͕ωΠςΟϒʹαϙʔτ͍ͯ͠ͳ͍ݴޠΛɺϏϧ
    υʹ஫ೖ͞ΕͨNode.jsγϜΛ࢖༻ͯ͠࢖༻Ͱ͖·͢ɻ ػೳͷς
    ετɺσϓϩΠͷϩʔϧόοΫɺϝτϦοΫͷදࣔɺϩάͷς
    ʔϦϯάɺϏϧυγεςϜ΁ͷϑοΫͳͲɺ͞·͟·ͳϫʔΫ
    ϑϩʔʹؔ࿈͢Δπʔϧ͕ఏڙ͞Ε͍ͯ·͢ɻ (by Google຋༁)
    

    View Slide

12. Apex Ͱ Go ͕ಈ͘࢓૊Έ
    GoͷόΠφϦΛಉࠝɺࢠϓϩηεͱͯ͠ىಈ
    STDIN / STDOUTͰϓϩηεؒ௨৴(JSON)
    

    View Slide

13. Example
    package main
    type input struct {
    Name string `json:"name"`
    }
    type output struct {
    Message string `json:"message"`
    }
    func main() {
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    var in input
    if err := json.Unmarshal(event, &in); err != nil {
    return nil, err
    }
    out := output{Message: "Hello " + in.Name}
    return out, nil
    })
    }
    

    View Slide

14. ؆୯ʹBuild, DeployͰ͖Δ
    $ apex deploy
    Goͷbuild
    zip ࡞੒ͯ͠ Lambda ʹ deploy
    ͜Ε͚ͩ
    

    View Slide

15. ! API Gateway ͷ Object ͱ
    Go ͷ net/http Λ૬ޓม׵͢Ε͹͍͍ͷͰ͸
    ͦΕ͕ Ridge
    github.com/fujiwara/ridge
    

    View Slide

16. RidgeͰॻ͔ΕͨΞϓϦͷྫ
    import (
    "net/http"
    "github.com/fujiwara/ridge"
    )
    func main() {
    var mux = http.NewServeMux()
    mux.HandleFunc("/hello", handleHello)
    ridge.Run(":8080", "/api", mux)
    }
    func handleHello(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    fmt.Fprintf(w, "Hello %s\n", r.FormValue("name"))
    }
    ridge.Run() Ҏ֎͸ී௨ͷ net/http ΞϓϦέʔγϣϯ
    

    View Slide

17. ridge.Run() ͕΍Δ͜ͱ
    API Gateway Proxy Integration ͱ net/http ͷ૬ޓม׵
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    r, _ := NewRequest(event) // API GW ͷ Object Λ *http.Request ʹ͢Δ
    w := NewResponseWriter() // RidgeͷResponseWriter
    mux.ServeHTTP(w, r) // ී௨ʹHTTP Handlerʹ౉ͯ͠ॲཧ
    return w.Response(), nil // API GW͕ཁٻ͢ΔObjectʹม׵ͯ͠ฦ͢
    })
    

    View Slide

18. ridge.Run() ͕΍Δ͜ͱ
    Apex(Lambda)Ͱಈ͔͞ͳ͍৔߹͸ී௨ʹhttp.Serverىಈ
    if os.Getenv("APEX_FUNCTION_NAME") != "" {
    apex.HandleFunc(...)
    } else {
    // ApexͰ͸ͳ͍৔߹
    m := http.NewServeMux()
    m.Handle(prefix+"/", http.StripPrefix(prefix, mux))
    log.Println("starting up with local httpd", address)
    log.Fatal(http.ListenAndServe(address, m))
    }
    

    View Slide

19. ී௨ͷ net/http ΞϓϦέʔγϣϯ͕
    API Gateway + Lambda Ͱ΋
    net/http.Server Ͱ΋ಈ͘
    

    View Slide

20. Ridge ͷ͍͍ͱ͜Ζ
    

    View Slide

21. Ridge ͷ͍͍ͱ͜Ζ
    ͘͝ී௨ͷ Go ͷ net/http ΞϓϦέʔγϣϯͰ͋Δ
    • ςετ͸ී௨ʹ httptest Ͱ
    • ϩʔΧϧͰಈ͔ͤ͹ curl / ϒϥ΢βͰಈ࡞֬ೝ
    • API GatewayͰͷੑೳʹ໰୊͕͋ͬͨ৔߹(ޙड़)
    ී௨ʹEC2Ͱಈ͔ͤΔ
    

    View Slide

22. Ridge ͷ͍͍ͱ͜Ζ
    API Gateway + Lambda Ͱಈ͘ͷͰ
    • ϦΫΤετ਺ʹԠͯࣗ͡ಈͰεέʔϧ
    • ϦΫΤετ͕ͳ͚Ε͹ ! ͳ͠
    • EC2ͷ؅ཧҰ੾ͳ͠
    • σϓϩΠ΋ apex deploy Ұൃ
    

    View Slide

23. ͭ·Γ͜Ε͸
    GAE/Go Έ͍ͨͳ΋ͷʂ
    

    View Slide

24. Ridge ͰͰ͖ͳ͍͜ͱ
    

    View Slide

25. Ridge ͰͰ͖ͳ͍͜ͱ
    • ಉҰ໊ෳ਺URLҾ਺ (?foo=a&foo=b Έ͍ͨͳͷ)
    API Gateway͔ΒདྷΔ࣌఺Ͱ໊લΛkeyʹͨ͠objectͳͷͰ…
    • ಉҰ໊ෳ਺ HTTP ϔομ
    API Gatewayͷ࣌఺Ͱ(ུ)
    Proxy Integrationͷ࢓༷ʹΑΔ
    

    View Slide

26. Ridge ͰͰ͖ͳ͍͜ͱ
    • όΠφϦΛฦ͢ (ը૾ͱ͔)
    API Gateway͸ઌ೔όΠφϦαϙʔτ͚ͨ͠Ͳ
    Proxy Integration͔Βฦͤͳ͍Α͏ͳ…
    ࠓޙʹظ଴
    

    View Slide

27. Ridge ͰͰ͖ͳ͍͜ͱ
    • ཪͰ Goroutine Λಈ͔͠ଓ͚Δ
    Lambda͸ϦΫΤετॲཧΛऴ͑Δͱϓϩηεٳ຾
    • 1ϦΫΤετʹ10ඵҎ্ֻ͔Δॲཧ
    API GatewayͷλΠϜΞ΢τ͕ݻఆ
    Long poll / SSE తͳ͜ͱ΋ؚΉ
    

    View Slide

28. RidgeͷੑೳධՁ
    

    View Slide

29. RidgeͷੑೳධՁ
    RedisʹೖΕͨ༣ศ൪߸DB1 ΛҾ͍ͯJSONΛฦ͢API
    ڞ௨: ElastiCacheRedis (t2.micro)
    1. ALB(https) + EC2(t2.micro)
    2. API Gateway(https) + Lambda (128MB)
    $ curl -s "https://***/api/postal_code?code=1000001"
    {"code":"1000001","addresses":[{"address":"౦ژ౎ઍ୅ా۠ઍ୅ా","yomi":"τ΢Ωϣ΢τνϤμΫνϤμ"}]}
    1 KEN_ALL.csv
    

    View Slide

30. 1. ALB + EC2 (t2.micro)
    Running 10s test @ https://***/api/postal_code?code=1000001
    4 threads and 30 connections
    Thread Stats Avg Stdev Max +/- Stdev
    Latency 10.55ms 4.22ms 222.76ms 96.48%
    Req/Sec 662.03 68.84 780.00 76.00%
    Latency Distribution
    50% 9.89ms
    75% 10.82ms
    90% 12.28ms
    99% 19.18ms
    26402 requests in 10.02s, 7.68MB read
    Requests/sec: 2634.14
    Transfer/sec: 784.46KB
    

    View Slide

31. 2. API Gateway + Lambda (128MB)
    Running 10s test @ https://***/api/postal_code?code=1000001
    4 threads and 30 connections
    Thread Stats Avg Stdev Max +/- Stdev
    Latency 33.83ms 14.64ms 355.60ms 86.32%
    Req/Sec 190.01 27.50 260.00 70.50%
    Latency Distribution
    50% 31.43ms
    75% 37.48ms
    90% 46.82ms
    99% 81.43ms
    7603 requests in 10.05s, 4.30MB read
    Requests/sec: 756.37
    Transfer/sec: 438.01KB
    

    View Slide

32. ൺֱͷͨΊͲͪΒ΋30ฒྻ ݶքੑೳΛܭଌ͍ͯ͠ͳ͍͜ͱʹ஫ҙ
    • EC2, Lambdaͱ΋ΞϓϦࣗମͷॲཧ͸ฏۉ 5msఔ౓
    • API Gateway, Apex, Ridge ͰͷϨΠςϯγ͕߹ܭ20msఔ౓
    (େ෦෼͕API Gateway)
    ฒྻ਺্͕͕Ε͹ req/sec ͸্͕Δ
    API Gateway͸1000req/sec2, Lambda 100ฒྻͷ੍ݶ͋Γ
    (্ݶ؇࿨ਃ੥Մೳ)
    2 ͳͷͰϕϯνͷฒྻ਺Λ্͛੾Ε͍ͯͳ͍
    

    View Slide

33. Ridgeʹ޲͍͍ͯͳ͍͜ͱ
    ௿ϨΠςϯγ͕ཁٻ͞ΕΔΞϓϦέʔγϣϯ
    ͨͱ͑͹಺෦޲͚ microservice
    ͦ΋ͦ΋ API Gateway ͕ VPC Ͱಈ͔ͳ͍
    

    View Slide

34. Ridge͕(ಛʹ)޲͍͍ͯΔ͜ͱ
    සൟʹΞΫηε͕ͳ͍ / ϨΠςϯγཁٻ͕γϏΞͰͳ͍
    • webhookΛड͚Δ Slack bot
    • S3ͷrepo viewer github.com/fujiwara/ridge-s3viewer
    • αʔϏε͕ऴྃͨ͠ήʔϜͷࠂ஌API
    POSTΛड͚ͯJSONΛฦ͢ͷΛEC2ͳ͠Ͱ
    ΧϠοΫͰ͸ຊ൪APIͱͯ͠౤ೖ࣮੷͋Γ
    

    View Slide

35. Ridge(ͱ͍͏͔Lambda)ͷ஫ҙࣄ߲
    • (VPC Lambda ͷ৔߹) ॳճىಈ࣌ʹENIΛׂΓ౰ͯΔͷʹ
    10ඵఔ౓ඞཁ
    • ίϯςφ͕׬શʹࢭ·ͬͯ͠·͏ͱىಈ͕஗͍
    ఆظత(1ʙ5෼ִؒ)ʹ֎ܗ؂ࢹͳͲͰୟ͚͹OK
    • ϝϞϦׂ౰ྔͱCPUੑೳ͕ൺྫ͢Δ
    ࣮ࡍʹϝϞϦΛ࢖Θͳͯ͘΋CPU͕ඞཁͳΒେ͖͘
    (! ΋ൺྫ͠·͢)
    

    View Slide

36. ϩάͷऔΓѻ͍
    

    View Slide

37. ϩάͷऔΓѻ͍
    go-apexͰ͸࢓૊Έ্ STDOUT ʹ͸Կ΋ग़ྗͰ͖ͳ͍
    STDERR ʹग़ྗͨ͠΋ͷ͸ CloudWatch Logs ʹه࿥͞ΕΔ
    apex logs -f Ͱ΋Ӿཡ(tail)Մೳ
    

    View Slide

38. ϩάͷऔΓѻ͍
    CloudWatch Logs ͸͍͍͓஋ஈ + औΓѻ͍͕ͭΒ͍
    → Lambda ʹྲྀ͢͜ͱ͕Ͱ͖Δ
    {
    "awslogs": {
    "data": "H4sIAKmNfFgAA5VQTWvbQBS891cI0aNV7dtv+aYQNRRiWiy1l9iYlfVkBPpwp
    XXTNOS/98lOwRBy6B7eYWb2zcx7DjucJnfA4umI4TK8TYt0t8ryPL3LwkU4PPY4EpxcPYL
    b4XA3DqcjMbF7nOLWdWXl4rGpDhh5nPyuc01/EeZ+RNeRkjMwMYMYdPzw8T4tsrzYVhxLU
    Vd1KS1Ih3W5R+2U2guuoUTNaMV0Kqf92Bx9M/Sfm9bjOIXLh/D+bHlZvrsyJsdwezbOfmH
    vZ+1z2FTkLwQwrY3QVjHNjUgSKbVRUlhrGZdglALOleCCYCsSQzgwAZTBN3Ql7zoqDNJKp
    aRKwDJY/Lserc+LdF0Ea/x5IumXahmANgythKgq9zICQB1ZmySRFMrUNdRaKQx+UB0qtgx
    eL7Lpw5fF28DAjJBAP5nhijHgWnFpaaqEGaNnQicJIzer9fuBxXXgdfbt6/8n3vjb0+j8O
    TN84izopo2/adoWq+CKYTMRbPwKu2F8CvLmDxLKbbC6IdD9Dl6J7xPOxhd8Lr99+fAXpgd
    o/ZQCAAA="
    }
    }
    ͳʹ͜ͷσʔλ !
    

    View Slide

39. CloudWatch Logs → Lambda
    JSONจࣈྻΛgzipͯ͠base64ͨ͠΋ͷ͕ྲྀΕͯ͘Δ !
    $ jq -r .awslogs.data | base64 --decode | gzip -dc | jq .
    {
    "messageType": "DATA_MESSAGE",
    "owner": "999999999999",
    "logGroup": "/aws/lambda/ridge-test_main",
    "logStream": "2017/01/16/[$LATEST]d2eb3fdfb4814aefbce6a55c3261be60",
    "subscriptionFilters": [
    "LambdaStream_ridge-test_log"
    ],
    "logEvents": [
    {
    "id": "33106673685062739944675438880241755122532346783978881031",
    "timestamp": 1484554591801,
    "message": "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n"
    },
    {
    "id": "33106673685107341435072500126524826559077643506990841866",
    "timestamp": 1484554591803,
    "message": "REPORT RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e\t
    Duration: 1.20 ms\tBilled Duration: 100 ms \tMemory Size: 128 MB\tMax Memory Used: 18 MB\t\n"
    }
    ]
    }
    

    View Slide

40. ridge.DecodeLogStream() Ͱόϥͤ·͢
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
    logStream, _ := ridge.DecodeLogStream(event)
    for _, e := range logStream.LogEvents {
    /* e => {
    ID: "33106673685062739944675438880241755122532346783978881031",
    Timestamp: 1484554591801,
    Message: "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n"
    }
    */
    }
    })
    ల։ͨ͠΋ͷΛ Kinesis Streams, Firehose
    ͳͲʹ౤͛Δͷ͕Α͍ͷͰ͸
    

    View Slide

41. ·ͱΊ
    API Gateway + Lambda Ͱ GAE/Go Έ͍ͨͳ3 ͜ͱ͕Ͱ͖Δ
    Ridge (github.com/fujiwara/ridge) Λ࡞Γ·ͨ͠
    ༻్ʹΑͬͯ͸طʹ
    3 ಉ͡ͱ͸͍͑·ͤΜ
    

    View Slide