Slide 1

Slide 1 text

Ridge GAE/Go Έ͍ͨͳͷ on AWS 2017.01.26 @fujiwara https://www.flickr.com/photos/jimgrant/2339167908

Slide 2

Slide 2 text

౻ݪ ढ़Ұ࿠ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦

Slide 3

Slide 3 text

Ridge is ͳʹ github.com/fujiwara/ridge API Gateway ɹ Lambda ɹ Go Λ࢖ͬͯ GAE/Go Έ͍ͨͳ΋ͷΛ࡞ΔϥΠϒϥϦ

Slide 4

Slide 4 text

API Gateway Proxy Integration & Lambda ?

Slide 5

Slide 5 text

API Gateway Proxy Integration & Lambda

Slide 6

Slide 6 text

API Gateway Proxy Integration ͱ Lambda Λઃఆ͢Δͱ GET /api/hello?name=Bob HTTP/1.1 User-Agent: curl/7.43.0 ! { "path": "/api/hello", "httpMethod": "GET", "headers": { "User-Agent": "curl/7.43.0", ... }, "queryStringParameters": { "name": "Bob" }, "body": null } API Gateway͕͏͚ͨ೚ҙͷHTTPϦΫΤετ͕ ^ͷΑ͏ͳObjectʹͳͬͯLambdaʹ

Slide 7

Slide 7 text

Lambda͕ฦͨ͠Object͕ API Gateway ʹΑͬͯ HTTPϨεϙϯεʹ { "statusCode": 200, "headers": { "Content-Type": "text/plain" }, "body": "Hello Bob." } ! HTTP/1.1 200 OK Content-Type: text/plain Hello Bob.

Slide 8

Slide 8 text

ͭ·ΓLambdaͰ೚ҙͷWebΞϓϦ͕࡞ΕΔ ͚Ͳ • ಠࣗObjectΛಡΜͰಠࣗObjectΛฦ͢ • routing͸… ! • ςετ͸Ͳ͏΍ͬͯ… ! ͋ͱ Ͱॻ͖͍ͨ

Slide 9

Slide 9 text

Lambda Ͱ Go Λಈ͔͢

Slide 10

Slide 10 text

http://apex.run

Slide 11

Slide 11 text

ApexΛ࢖༻͢ΔͱɺAWS Lambdaؔ਺Λ؆୯ʹߏஙɺల։ɺ؅ ཧ͢Δ͜ͱ͕Ͱ͖·͢ɻ ApexΛ࢖༻͢ΔͱɺGolangͳͲɺ AWS Lambda͕ωΠςΟϒʹαϙʔτ͍ͯ͠ͳ͍ݴޠΛɺϏϧ υʹ஫ೖ͞ΕͨNode.jsγϜΛ࢖༻ͯ͠࢖༻Ͱ͖·͢ɻ ػೳͷς ετɺσϓϩΠͷϩʔϧόοΫɺϝτϦοΫͷදࣔɺϩάͷς ʔϦϯάɺϏϧυγεςϜ΁ͷϑοΫͳͲɺ͞·͟·ͳϫʔΫ ϑϩʔʹؔ࿈͢Δπʔϧ͕ఏڙ͞Ε͍ͯ·͢ɻ (by Google຋༁)

Slide 12

Slide 12 text

Apex Ͱ Go ͕ಈ͘࢓૊Έ GoͷόΠφϦΛಉࠝɺࢠϓϩηεͱͯ͠ىಈ STDIN / STDOUTͰϓϩηεؒ௨৴(JSON)

Slide 13

Slide 13 text

Example package main type input struct { Name string `json:"name"` } type output struct { Message string `json:"message"` } func main() { apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) { var in input if err := json.Unmarshal(event, &in); err != nil { return nil, err } out := output{Message: "Hello " + in.Name} return out, nil }) }

Slide 14

Slide 14 text

؆୯ʹBuild, DeployͰ͖Δ $ apex deploy Goͷbuild zip ࡞੒ͯ͠ Lambda ʹ deploy ͜Ε͚ͩ

Slide 15

Slide 15 text

! API Gateway ͷ Object ͱ Go ͷ net/http Λ૬ޓม׵͢Ε͹͍͍ͷͰ͸ ͦΕ͕ Ridge github.com/fujiwara/ridge

Slide 16

Slide 16 text

RidgeͰॻ͔ΕͨΞϓϦͷྫ import ( "net/http" "github.com/fujiwara/ridge" ) func main() { var mux = http.NewServeMux() mux.HandleFunc("/hello", handleHello) ridge.Run(":8080", "/api", mux) } func handleHello(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/plain") fmt.Fprintf(w, "Hello %s\n", r.FormValue("name")) } ridge.Run() Ҏ֎͸ී௨ͷ net/http ΞϓϦέʔγϣϯ

Slide 17

Slide 17 text

ridge.Run() ͕΍Δ͜ͱ API Gateway Proxy Integration ͱ net/http ͷ૬ޓม׵ apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) { r, _ := NewRequest(event) // API GW ͷ Object Λ *http.Request ʹ͢Δ w := NewResponseWriter() // RidgeͷResponseWriter mux.ServeHTTP(w, r) // ී௨ʹHTTP Handlerʹ౉ͯ͠ॲཧ return w.Response(), nil // API GW͕ཁٻ͢ΔObjectʹม׵ͯ͠ฦ͢ })

Slide 18

Slide 18 text

ridge.Run() ͕΍Δ͜ͱ Apex(Lambda)Ͱಈ͔͞ͳ͍৔߹͸ී௨ʹhttp.Serverىಈ if os.Getenv("APEX_FUNCTION_NAME") != "" { apex.HandleFunc(...) } else { // ApexͰ͸ͳ͍৔߹ m := http.NewServeMux() m.Handle(prefix+"/", http.StripPrefix(prefix, mux)) log.Println("starting up with local httpd", address) log.Fatal(http.ListenAndServe(address, m)) }

Slide 19

Slide 19 text

ී௨ͷ net/http ΞϓϦέʔγϣϯ͕ API Gateway + Lambda Ͱ΋ net/http.Server Ͱ΋ಈ͘

Slide 20

Slide 20 text

Ridge ͷ͍͍ͱ͜Ζ

Slide 21

Slide 21 text

Ridge ͷ͍͍ͱ͜Ζ ͘͝ී௨ͷ Go ͷ net/http ΞϓϦέʔγϣϯͰ͋Δ • ςετ͸ී௨ʹ httptest Ͱ • ϩʔΧϧͰಈ͔ͤ͹ curl / ϒϥ΢βͰಈ࡞֬ೝ • API GatewayͰͷੑೳʹ໰୊͕͋ͬͨ৔߹(ޙड़) ී௨ʹEC2Ͱಈ͔ͤΔ

Slide 22

Slide 22 text

Ridge ͷ͍͍ͱ͜Ζ API Gateway + Lambda Ͱಈ͘ͷͰ • ϦΫΤετ਺ʹԠͯࣗ͡ಈͰεέʔϧ • ϦΫΤετ͕ͳ͚Ε͹ ! ͳ͠ • EC2ͷ؅ཧҰ੾ͳ͠ • σϓϩΠ΋ apex deploy Ұൃ

Slide 23

Slide 23 text

ͭ·Γ͜Ε͸ GAE/Go Έ͍ͨͳ΋ͷʂ

Slide 24

Slide 24 text

Ridge ͰͰ͖ͳ͍͜ͱ

Slide 25

Slide 25 text

Ridge ͰͰ͖ͳ͍͜ͱ • ಉҰ໊ෳ਺URLҾ਺ (?foo=a&foo=b Έ͍ͨͳͷ) API Gateway͔ΒདྷΔ࣌఺Ͱ໊લΛkeyʹͨ͠objectͳͷͰ… • ಉҰ໊ෳ਺ HTTP ϔομ API Gatewayͷ࣌఺Ͱ(ུ) Proxy Integrationͷ࢓༷ʹΑΔ

Slide 26

Slide 26 text

Ridge ͰͰ͖ͳ͍͜ͱ • όΠφϦΛฦ͢ (ը૾ͱ͔) API Gateway͸ઌ೔όΠφϦαϙʔτ͚ͨ͠Ͳ Proxy Integration͔Βฦͤͳ͍Α͏ͳ… ࠓޙʹظ଴

Slide 27

Slide 27 text

Ridge ͰͰ͖ͳ͍͜ͱ • ཪͰ Goroutine Λಈ͔͠ଓ͚Δ Lambda͸ϦΫΤετॲཧΛऴ͑Δͱϓϩηεٳ຾ • 1ϦΫΤετʹ10ඵҎ্ֻ͔Δॲཧ API GatewayͷλΠϜΞ΢τ͕ݻఆ Long poll / SSE తͳ͜ͱ΋ؚΉ

Slide 28

Slide 28 text

RidgeͷੑೳධՁ

Slide 29

Slide 29 text

RidgeͷੑೳධՁ RedisʹೖΕͨ༣ศ൪߸DB1 ΛҾ͍ͯJSONΛฦ͢API ڞ௨: ElastiCacheRedis (t2.micro) 1. ALB(https) + EC2(t2.micro) 2. API Gateway(https) + Lambda (128MB) $ curl -s "https://***/api/postal_code?code=1000001" {"code":"1000001","addresses":[{"address":"౦ژ౎ઍ୅ా۠ઍ୅ా","yomi":"τ΢Ωϣ΢τνϤμΫνϤμ"}]} 1 KEN_ALL.csv

Slide 30

Slide 30 text

1. ALB + EC2 (t2.micro) Running 10s test @ https://***/api/postal_code?code=1000001 4 threads and 30 connections Thread Stats Avg Stdev Max +/- Stdev Latency 10.55ms 4.22ms 222.76ms 96.48% Req/Sec 662.03 68.84 780.00 76.00% Latency Distribution 50% 9.89ms 75% 10.82ms 90% 12.28ms 99% 19.18ms 26402 requests in 10.02s, 7.68MB read Requests/sec: 2634.14 Transfer/sec: 784.46KB

Slide 31

Slide 31 text

2. API Gateway + Lambda (128MB) Running 10s test @ https://***/api/postal_code?code=1000001 4 threads and 30 connections Thread Stats Avg Stdev Max +/- Stdev Latency 33.83ms 14.64ms 355.60ms 86.32% Req/Sec 190.01 27.50 260.00 70.50% Latency Distribution 50% 31.43ms 75% 37.48ms 90% 46.82ms 99% 81.43ms 7603 requests in 10.05s, 4.30MB read Requests/sec: 756.37 Transfer/sec: 438.01KB

Slide 32

Slide 32 text

ൺֱͷͨΊͲͪΒ΋30ฒྻ ݶքੑೳΛܭଌ͍ͯ͠ͳ͍͜ͱʹ஫ҙ • EC2, Lambdaͱ΋ΞϓϦࣗମͷॲཧ͸ฏۉ 5msఔ౓ • API Gateway, Apex, Ridge ͰͷϨΠςϯγ͕߹ܭ20msఔ౓ (େ෦෼͕API Gateway) ฒྻ਺্͕͕Ε͹ req/sec ͸্͕Δ API Gateway͸1000req/sec2, Lambda 100ฒྻͷ੍ݶ͋Γ (্ݶ؇࿨ਃ੥Մೳ) 2 ͳͷͰϕϯνͷฒྻ਺Λ্͛੾Ε͍ͯͳ͍

Slide 33

Slide 33 text

Ridgeʹ޲͍͍ͯͳ͍͜ͱ ௿ϨΠςϯγ͕ཁٻ͞ΕΔΞϓϦέʔγϣϯ ͨͱ͑͹಺෦޲͚ microservice ͦ΋ͦ΋ API Gateway ͕ VPC Ͱಈ͔ͳ͍

Slide 34

Slide 34 text

Ridge͕(ಛʹ)޲͍͍ͯΔ͜ͱ සൟʹΞΫηε͕ͳ͍ / ϨΠςϯγཁٻ͕γϏΞͰͳ͍ • webhookΛड͚Δ Slack bot • S3ͷrepo viewer github.com/fujiwara/ridge-s3viewer • αʔϏε͕ऴྃͨ͠ήʔϜͷࠂ஌API POSTΛड͚ͯJSONΛฦ͢ͷΛEC2ͳ͠Ͱ ΧϠοΫͰ͸ຊ൪APIͱͯ͠౤ೖ࣮੷͋Γ

Slide 35

Slide 35 text

Ridge(ͱ͍͏͔Lambda)ͷ஫ҙࣄ߲ • (VPC Lambda ͷ৔߹) ॳճىಈ࣌ʹENIΛׂΓ౰ͯΔͷʹ 10ඵఔ౓ඞཁ • ίϯςφ͕׬શʹࢭ·ͬͯ͠·͏ͱىಈ͕஗͍ ఆظత(1ʙ5෼ִؒ)ʹ֎ܗ؂ࢹͳͲͰୟ͚͹OK • ϝϞϦׂ౰ྔͱCPUੑೳ͕ൺྫ͢Δ ࣮ࡍʹϝϞϦΛ࢖Θͳͯ͘΋CPU͕ඞཁͳΒେ͖͘ (! ΋ൺྫ͠·͢)

Slide 36

Slide 36 text

ϩάͷऔΓѻ͍

Slide 37

Slide 37 text

ϩάͷऔΓѻ͍ go-apexͰ͸࢓૊Έ্ STDOUT ʹ͸Կ΋ग़ྗͰ͖ͳ͍ STDERR ʹग़ྗͨ͠΋ͷ͸ CloudWatch Logs ʹه࿥͞ΕΔ apex logs -f Ͱ΋Ӿཡ(tail)Մೳ

Slide 38

Slide 38 text

ϩάͷऔΓѻ͍ CloudWatch Logs ͸͍͍͓஋ஈ + औΓѻ͍͕ͭΒ͍ → Lambda ʹྲྀ͢͜ͱ͕Ͱ͖Δ { "awslogs": { "data": "H4sIAKmNfFgAA5VQTWvbQBS891cI0aNV7dtv+aYQNRRiWiy1l9iYlfVkBPpwp XXTNOS/98lOwRBy6B7eYWb2zcx7DjucJnfA4umI4TK8TYt0t8ryPL3LwkU4PPY4EpxcPYL b4XA3DqcjMbF7nOLWdWXl4rGpDhh5nPyuc01/EeZ+RNeRkjMwMYMYdPzw8T4tsrzYVhxLU Vd1KS1Ih3W5R+2U2guuoUTNaMV0Kqf92Bx9M/Sfm9bjOIXLh/D+bHlZvrsyJsdwezbOfmH vZ+1z2FTkLwQwrY3QVjHNjUgSKbVRUlhrGZdglALOleCCYCsSQzgwAZTBN3Ql7zoqDNJKp aRKwDJY/Lserc+LdF0Ea/x5IumXahmANgythKgq9zICQB1ZmySRFMrUNdRaKQx+UB0qtgx eL7Lpw5fF28DAjJBAP5nhijHgWnFpaaqEGaNnQicJIzer9fuBxXXgdfbt6/8n3vjb0+j8O TN84izopo2/adoWq+CKYTMRbPwKu2F8CvLmDxLKbbC6IdD9Dl6J7xPOxhd8Lr99+fAXpgd o/ZQCAAA=" } } ͳʹ͜ͷσʔλ !

Slide 39

Slide 39 text

CloudWatch Logs → Lambda JSONจࣈྻΛgzipͯ͠base64ͨ͠΋ͷ͕ྲྀΕͯ͘Δ ! $ jq -r .awslogs.data | base64 --decode | gzip -dc | jq . { "messageType": "DATA_MESSAGE", "owner": "999999999999", "logGroup": "/aws/lambda/ridge-test_main", "logStream": "2017/01/16/[$LATEST]d2eb3fdfb4814aefbce6a55c3261be60", "subscriptionFilters": [ "LambdaStream_ridge-test_log" ], "logEvents": [ { "id": "33106673685062739944675438880241755122532346783978881031", "timestamp": 1484554591801, "message": "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n" }, { "id": "33106673685107341435072500126524826559077643506990841866", "timestamp": 1484554591803, "message": "REPORT RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e\t Duration: 1.20 ms\tBilled Duration: 100 ms \tMemory Size: 128 MB\tMax Memory Used: 18 MB\t\n" } ] }

Slide 40

Slide 40 text

ridge.DecodeLogStream() Ͱόϥͤ·͢ apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) { logStream, _ := ridge.DecodeLogStream(event) for _, e := range logStream.LogEvents { /* e => { ID: "33106673685062739944675438880241755122532346783978881031", Timestamp: 1484554591801, Message: "START RequestId: 1670e841-dbc4-11e6-8899-4357ff1f655e Version: $LATEST\n" } */ } }) ల։ͨ͠΋ͷΛ Kinesis Streams, Firehose ͳͲʹ౤͛Δͷ͕Α͍ͷͰ͸

Slide 41

Slide 41 text

·ͱΊ API Gateway + Lambda Ͱ GAE/Go Έ͍ͨͳ3 ͜ͱ͕Ͱ͖Δ Ridge (github.com/fujiwara/ridge) Λ࡞Γ·ͨ͠ ༻్ʹΑͬͯ͸طʹ 3 ಉ͡ͱ͸͍͑·ͤΜ