Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ridge - A framework like GAE/Go on AWS

Ridge - A framework like GAE/Go on AWS

Ca6281fff64797dc419b78f51f25c0a5?s=128

FUJIWARA Shunichiro
PRO

March 01, 2017
Tweet

Transcript

  1. Ridge A framework like GAE/Go on AWS 2017.03.01 golang.tokyo #4

    @fujiwara https://www.flickr.com/photos/jimgrant/2339167908
  2. ౻ݪ ढ़Ұ࿠ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦

  3. Ridge is ͳʹ github.com/fujiwara/ridge API Gateway ɹ Lambda ɹ Go

    Λ࢖ͬͯ GAE/Go Έ͍ͨͳ΋ͷΛ࡞ΔϥΠϒϥϦ
  4. API Gateway Proxy Integration & Lambda ?

  5. API Gateway Proxy Integration & Lambda

  6. API Gateway Proxy Integration ͱ Lambda Λઃఆ͢Δͱ { "path": "/api/hello",

    "httpMethod": "GET", "headers": { GET /api/hello?name=Bob HTTP/1.1 "User-Agent": "Go-http-client/1.1", User-Agent: Go-http-client/1.1 => ... }, "queryStringParameters": { "name": "Bob" }, "body": null } API Gateway ͕͏͚ͨ೚ҙͷHTTPϦΫΤετ͕ Object ʹͳͬͯ Lambda ʹ
  7. Lambda ͕ฦͨ͠ Object ͕ API Gateway ʹΑͬͯ HTTPϨεϙϯεʹ { "statusCode":

    200, "headers": { HTTP/1.1 200 OK "Content-Type": "text/plain" => Content-Type: text/plain }, "body": "Hello Bob." Hello Bob. }
  8. ͭ·ΓLambdaͰ೚ҙͷWebΞϓϦ͕࡞ΕΔ • ಠࣗObjectΛಡΜͰಠࣗObjectΛฦ͢ • routing͸… ! • ςετ͸Ͳ͏΍ͬͯ… ! •

    Ͱॻ͖͍ͨ
  9. Lambda Ͱ Go Λಈ͔͢

  10. http://apex.run

  11. ApexΛ࢖༻͢ΔͱɺAWS Lambdaؔ਺Λ؆୯ʹߏஙɺల։ɺ؅ ཧ͢Δ͜ͱ͕Ͱ͖·͢ɻ ApexΛ࢖༻͢ΔͱɺGolangͳͲɺ AWS Lambda͕ωΠςΟϒʹαϙʔτ͍ͯ͠ͳ͍ݴޠΛɺϏϧ υʹ஫ೖ͞ΕͨNode.jsγϜΛ࢖༻ͯ͠࢖༻Ͱ͖·͢ɻ ػೳͷς ετɺσϓϩΠͷϩʔϧόοΫɺϝτϦοΫͷදࣔɺϩάͷς ʔϦϯάɺϏϧυγεςϜ΁ͷϑοΫͳͲɺ͞·͟·ͳϫʔΫ

    ϑϩʔʹؔ࿈͢Δπʔϧ͕ఏڙ͞Ε͍ͯ·͢ɻ (by Google຋༁)
  12. Apex Ͱ Go ͕ಈ͘࢓૊Έ GoͷόΠφϦΛಉࠝɺࢠϓϩηεͱͯ͠ىಈ STDIN / STDOUTͰϓϩηεؒ௨৴(JSON)

  13. Example package main type input struct { Name string `json:"name"`

    } type output struct { Message string `json:"message"` } func main() { apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) { var in input if err := json.Unmarshal(event, &in); err != nil { return nil, err } out := output{Message: "Hello " + in.Name} return out, nil }) }
  14. ؆୯ʹBuild, DeployͰ͖Δ $ apex deploy Goͷbuild zip ࡞੒ͯ͠ Lambda ʹ

    deploy ͜Ε͚ͩ
  15. ! API Gateway ͷ Object ͱ Go ͷ net/http Λ૬ޓม׵͢Ε͹͍͍ͷͰ͸

    ͦΕ͕ Ridge github.com/fujiwara/ridge
  16. RidgeͰॻ͔ΕͨΞϓϦͷྫ func main() { var mux = http.NewServeMux() mux.HandleFunc("/hello", handleHello)

    ridge.Run(":8080", "/api", mux) } func handleHello(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/plain") fmt.Fprintf(w, "Hello %s\n", r.FormValue("name")) } ridge.Run() Ҏ֎͸ී௨ͷ net/http ΞϓϦέʔγϣϯ
  17. ridge.Run() ͕΍Δ͜ͱ API Gateway Proxy Integration ͱ net/http ͷ૬ޓม׵ apex.HandleFunc(func(event

    json.RawMessage, ctx *apex.Context) (interface{}, error) { r, _ := NewRequest(event) // API GW ͷ Object Λ *http.Request ʹ͢Δ w := NewResponseWriter() // RidgeͷResponseWriter mux.ServeHTTP(w, r) // ී௨ʹHTTP Handlerʹ౉ͯ͠ॲཧ return w.Response(), nil // API GW͕ཁٻ͢ΔObjectʹม׵ͯ͠ฦ͢ })
  18. -> http.Request ϔομͱΫΤϦύϥϝʔλΛ࡞Δ func (r Request) httpRequest() (*http.Request, error) {

    header := make(http.Header) for key, value := range r.Headers { header.Add(key, value) } host := header.Get("Host") header.Del("Host") v := make(url.Values) for key, value := range r.QueryStringParameters { v.Add(key, value) } uri := r.Path if len(r.QueryStringParameters) > 0 { uri = uri + "?" + v.Encode() }
  19. -> http.Request Body ͷ io.ReaderCloser Λจࣈྻ͔Β࡞Δ var contentLength int64 var

    b io.Reader if r.IsBase64Encoded { raw := make([]byte, len(r.Body)) n, err := base64.StdEncoding.Decode(raw, []byte(r.Body)) if err != nil { return nil, err } contentLength = int64(n) b = bytes.NewReader(raw[0:n]) } else { contentLength = int64(len(r.Body)) b = strings.NewReader(r.Body) } body := ioutil.NopCloser(b)
  20. -> http.Request req := http.Request{ Method: r.HTTPMethod, Proto: "HTTP/1.1", ProtoMajor:

    1, ProtoMinor: 1, Header: header, ContentLength: contentLength, Body: body, RemoteAddr: r.RequestContext.Identity["sourceIp"], Host: host, RequestURI: uri, URL: u, } return &req, nil }
  21. ResponseWriter -> http.ResponseWriter interfaceΛ࣮૷ type ResponseWriter struct { bytes.Buffer header

    http.Header statusCode int } func (w *ResponseWriter) Header() http.Header { return w.header } func (w *ResponseWriter) WriteHeader(code int) { w.statusCode = code }
  22. ResponseWriter -> func (w *ResponseWriter) Response() Response { h :=

    make(map[string]string, len(w.header)) for key := range w.header { h[key] = w.header.Get(key) } return Response{ StatusCode: w.statusCode, Headers: h, Body: w.String(), } }
  23. ridge.Run() ͕΍Δ͜ͱ Apex(Lambda)Ͱಈ͔͞ͳ͍৔߹͸ී௨ʹhttp.Serverىಈ if os.Getenv("APEX_FUNCTION_NAME") != "" { apex.HandleFunc(...) }

    else { // ApexͰ͸ͳ͍৔߹ m := http.NewServeMux() m.Handle(prefix+"/", http.StripPrefix(prefix, mux)) log.Println("starting up with local httpd", address) log.Fatal(http.ListenAndServe(address, m)) }
  24. ී௨ͷ net/http ΞϓϦέʔγϣϯ͕ API Gateway + Lambda Ͱ΋ net/http.Server Ͱ΋ಈ͘

  25. Ridge ͷ͍͍ͱ͜Ζ

  26. Ridge ͷ͍͍ͱ͜Ζ ͘͝ී௨ͷ Go ͷ net/http ΞϓϦέʔγϣϯͰ͋Δ • ςετ͸ී௨ʹ httptest

    Ͱ • ϩʔΧϧͰಈ͔ͤ͹ curl / ϒϥ΢βͰಈ࡞֬ೝ • API GatewayͰͷੑೳʹ໰୊͕͋ͬͨ৔߹(ޙड़) ී௨ʹEC2Ͱಈ͔ͤΔ
  27. Ridge ͷ͍͍ͱ͜Ζ API Gateway + Lambda Ͱಈ͘ͷͰ • ϦΫΤετ਺ʹԠͯࣗ͡ಈͰεέʔϧ •

    ϦΫΤετ͕ͳ͚Ε͹ ! ͳ͠ • EC2ͷ؅ཧҰ੾ͳ͠ • σϓϩΠ΋ apex deploy Ұൃ
  28. ͭ·Γ͜Ε͸ GAE/Go Έ͍ͨͳ΋ͷʂ

  29. Ridge ͰͰ͖ͳ͍͜ͱ

  30. Ridge ͰͰ͖ͳ͍͜ͱ • ಉҰ໊ෳ਺URLҾ਺ (?foo=a&foo=b Έ͍ͨͳͷ) API Gateway͔ΒདྷΔ࣌఺Ͱ໊લΛkeyʹͨ͠objectͳͷͰ… • ಉҰ໊ෳ਺

    HTTP ϔομ API Gatewayͷ࣌఺Ͱ(ུ) Proxy Integrationͷ࢓༷ʹΑΔ
  31. Ridge ͰͰ͖ͳ͍͜ͱ • όΠφϦΛฦ͢ (ը૾ͱ͔) API Gateway͸ઌ೔όΠφϦαϙʔτ͚ͨ͠Ͳ Proxy Integration͔Βฦͤͳ͍Α͏ͳ… ࠓޙʹظ଴

  32. Ridge ͰͰ͖ͳ͍͜ͱ • ཪͰ Goroutine Λಈ͔͠ଓ͚Δ Lambda͸ϦΫΤετॲཧΛऴ͑Δͱϓϩηεٳ຾ • 1ϦΫΤετʹ10ඵҎ্ֻ͔Δॲཧ API

    GatewayͷλΠϜΞ΢τ͕ݻఆ Long poll / SSE తͳ͜ͱ΋ؚΉ
  33. RidgeͷੑೳධՁ

  34. RidgeͷੑೳධՁ RedisʹೖΕͨ༣ศ൪߸DB1 ΛҾ͍ͯJSONΛฦ͢API ڞ௨: ElastiCacheRedis (t2.micro) 1. ALB(https) + EC2(t2.micro)

    2. API Gateway(https) + Lambda (128MB) $ curl -s "https://***/api/postal_code?code=1000001" {"code":"1000001","addresses":[{"address":"౦ژ౎ઍ୅ా۠ઍ୅ా","yomi":"τ΢Ωϣ΢τνϤμΫνϤμ"}]} 1 KEN_ALL.csv
  35. 1. ALB + EC2 (t2.micro) Running 10s test @ https://***/api/postal_code?code=1000001

    4 threads and 30 connections Thread Stats Avg Stdev Max +/- Stdev Latency 10.55ms 4.22ms 222.76ms 96.48% Req/Sec 662.03 68.84 780.00 76.00% Latency Distribution 50% 9.89ms 75% 10.82ms 90% 12.28ms 99% 19.18ms 26402 requests in 10.02s, 7.68MB read Requests/sec: 2634.14 Transfer/sec: 784.46KB
  36. 2. API Gateway + Lambda (128MB) Running 10s test @

    https://***/api/postal_code?code=1000001 4 threads and 30 connections Thread Stats Avg Stdev Max +/- Stdev Latency 33.83ms 14.64ms 355.60ms 86.32% Req/Sec 190.01 27.50 260.00 70.50% Latency Distribution 50% 31.43ms 75% 37.48ms 90% 46.82ms 99% 81.43ms 7603 requests in 10.05s, 4.30MB read Requests/sec: 756.37 Transfer/sec: 438.01KB
  37. ൺֱͷͨΊͲͪΒ΋30ฒྻ ݶքੑೳΛܭଌ͍ͯ͠ͳ͍͜ͱʹ஫ҙ • EC2, Lambdaͱ΋ΞϓϦࣗମͷॲཧ͸ฏۉ 5msఔ౓ • API Gateway, Apex,

    Ridge ͰͷϨΠςϯγ͕߹ܭ20msఔ౓ (େ෦෼͕API Gateway) ฒྻ਺্͕͕Ε͹ req/sec ͸্͕Δ API Gateway͸1000req/sec2, Lambda 100ฒྻͷ੍ݶ͋Γ (্ݶ؇࿨ਃ੥Մೳ) 2 ͳͷͰϕϯνͷฒྻ਺Λ্͛੾Ε͍ͯͳ͍
  38. Ridgeʹ޲͍͍ͯͳ͍͜ͱ ҆ఆͨ͠௿ϨΠςϯγ͕ཁٻ͞ΕΔΞϓϦέʔγϣϯ ͨͱ͑͹಺෦޲͚ microservice ͦ΋ͦ΋ API Gateway ͕ VPC Ͱಈ͔ͳ͍

  39. Ridge͕(ಛʹ)޲͍͍ͯΔ͜ͱ සൟʹΞΫηε͕ͳ͍ / ϨΠςϯγཁٻ͕γϏΞͰͳ͍ • webhookΛड͚Δ Slack bot • S3ͷrepo

    viewer github.com/fujiwara/ridge-s3viewer • αʔϏε͕ऴྃͨ͠ήʔϜͷࠂ஌API POSTΛड͚ͯJSONΛฦ͢ͷΛEC2ͳ͠Ͱ ΧϠοΫͰ͸ຊ൪APIͱͯ͠౤ೖ࣮੷͋Γ
  40. Ridge(ͱ͍͏͔Lambda)ͷ஫ҙࣄ߲ • (VPC Lambda ͷ৔߹) ॳճىಈ࣌ʹENIΛׂΓ౰ͯΔͷʹ 10ඵఔ౓ඞཁ • ίϯςφ͕׬શʹࢭ·ͬͯ͠·͏ͱىಈ͕஗͍ ఆظత(1ʙ5෼ִؒ)ʹ֎ܗ؂ࢹͳͲͰୟ͚͹OK

    • ϝϞϦׂ౰ྔͱCPUੑೳ͕ൺྫ͢Δ ࣮ࡍʹϝϞϦΛ࢖Θͳͯ͘΋CPU͕ඞཁͳΒେ͖͘ (! ΋ൺྫ͠·͢)
  41. ϩάͷऔΓѻ͍

  42. ϩάͷऔΓѻ͍ go-apexͰ͸࢓૊Έ্ STDOUT ʹ͸Կ΋ग़ྗͰ͖ͳ͍ (RidgeͰ͸ os.Stdout = os.Stderr ͯ͠໭͍ͯ͠Δ) STDERR

    ʹग़ྗͨ͠΋ͷ͸ CloudWatch Logs ʹه࿥͞ΕΔ apex logs -f Ͱ΋Ӿཡ(tail)Մೳ
  43. ϩάͷऔΓѻ͍ CloudWatch Logs ͸͍͍͓஋ஈ + औΓѻ͍͕ͭΒ͍ → Lambda ʹྲྀ͢͜ͱ͕Ͱ͖Δ {

    "awslogs": { "data": "H4sIAKmNfFgAA5VQTWvbQBS891cI0aNV7dtv+aYQNRRiWiy1l9iYlfVkBPpwp XXTNOS/98lOwRBy6B7eYWb2zcx7DjucJnfA4umI4TK8TYt0t8ryPL3LwkU4PPY4EpxcPYL b4XA3DqcjMbF7nOLWdWXl4rGpDhh5nPyuc01/EeZ+RNeRkjMwMYMYdPzw8T4tsrzYVhxLU Vd1KS1Ih3W5R+2U2guuoUTNaMV0Kqf92Bx9M/Sfm9bjOIXLh/D+bHlZvrsyJsdwezbOfmH vZ+1z2FTkLwQwrY3QVjHNjUgSKbVRUlhrGZdglALOleCCYCsSQzgwAZTBN3Ql7zoqDNJKp aRKwDJY/Lserc+LdF0Ea/x5IumXahmANgythKgq9zICQB1ZmySRFMrUNdRaKQx+UB0qtgx eL7Lpw5fF28DAjJBAP5nhijHgWnFpaaqEGaNnQicJIzer9fuBxXXgdfbt6/8n3vjb0+j8O TN84izopo2/adoWq+CKYTMRbPwKu2F8CvLmDxLKbbC6IdD9Dl6J7xPOxhd8Lr99+fAXpgd o/ZQCAAA=" } } ͳʹ͜ͷσʔλ !
  44. CloudWatch Logs → Lambda JSONจࣈྻΛgzipͯ͠base64ͨ͠΋ͷ͕ྲྀΕͯ͘Δ ! $ jq -r .awslogs.data

    | base64 --decode | gzip -dc | jq . { "messageType": "DATA_MESSAGE", "owner": "999999999999", "logGroup": "/aws/lambda/ridge-test_main", "logStream": "2017/01/16/[$LATEST]d2eb3fdfb4814aefbce6a55c3261be60", "subscriptionFilters": [ "LambdaStream_ridge-test_log" ], "logEvents": [ { "id": "33106673685062739944675438880241755122532346783978881031", "timestamp": 1484554591801, "message": "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n" }, { "id": "33106673685107341435072500126524826559077643506990841866", "timestamp": 1484554591803, "message": "REPORT RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e\t Duration: 1.20 ms\tBilled Duration: 100 ms \tMemory Size: 128 MB\tMax Memory Used: 18 MB\t\n" } ] }
  45. ridge.DecodeLogStream() Ͱόϥͤ·͢ apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) { logStream,

    _ := ridge.DecodeLogStream(event) for _, e := range logStream.LogEvents { /* e => { ID: "33106673685062739944675438880241755122532346783978881031", Timestamp: 1484554591801, Message: "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n" } */ } }) ల։ͨ͠΋ͷΛ Kinesis Streams, Firehose ͳͲʹ౤͛Δͷ͕Α͍ͷͰ͸
  46. ·ͱΊ API Gateway + Lambda Ͱ GAE/Go Έ͍ͨͳ3 ͜ͱ͕Ͱ͖Δ Ridge

    (github.com/fujiwara/ridge) Λ࡞Γ·ͨ͠ ༻్ʹΑͬͯ͸طʹ 3 ಉ͡ͱ͸͍͑·ͤΜ