調べながらGCPやってみた話/gcpug-osaka-3

GCPUG In Osaka #3 で発表した資料。
色々サービス使ってみました。
https://gcpug-osaka.connpass.com/event/61297/

1. ௐ΂ͳ͕ΒGCP΍ͬͯ Έͨ࿩ GCPUG In Osaka #3 @plan9user (2017-07-19)
2. ࣗݾ঺հ • ໳ଟګฏ • Favorites • OS: Plan 9 • Editor: acme • Shell: rc • ϑΣϯϦϧͰಇ͍͍ͯ·͢
4. Ϟνϕʔγϣϯ • GoΛॻ͍͍ͯͨͷͰڵຯ͸ͣͬͱ͋ͬͨ • ࣮͸Compute EngineͰPlan 9͕ಈ͘ • ಈ͚ͩ͘͡Όͳͯ͘ίϯιʔϧ͕࢖͑Δ • Plan 9ͷ؅ཧʹ͸ίϯιʔϧඞਢ
5. ৘ใऩू • Google Cloud Platform Japan ެࣜϒϩά • Publickey • Google Cloud Platform Advent Calendar • ໘നͦ͏ͳαʔϏε͸υΩϡϝϯτಡΉ
6. Կ͔࿩͠·ͤΜ͔?
7. • ཉ͔ͬͨ͠αʔϏεΛ(్த·Ͱ)࡞ͬͨ࿩ • ॳΊͯͷਓ͸ • Ͳ͏͍͏αʔϏε͕͋Δͷ͔ • ࢖͍ͬͯΔਓ͸ • ڭ͍͑ͯͩ͘͞ ࠓ೔࿩͢͜ͱ
8. ͜Μͳ͜ͱ͋Γ·ͤΜ͔? • ಡΜͩ͜ͱ͸͋Δ͚ͲݕࡧͰ͖ͳ͍ • ϒοΫϚʔΫ͢ΔͷΛ๨Ε͍ͯͨ • هࣄͦͷ΋ͷ͕ͳ͘ͳͬͯ͠·ͬͨ ޙഐʮͳΜͰ͜Ε೩͑ͯΔΜͰ͔͢ʯʮͦΕ͸͔֬ݩهࣄ͕...ʯ
9. ಡΜͩهࣄશ෦࢒͍ͨ͠
10. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
11. ༻ҙ͢Δ΋ͷ • GAE/Go SDK • Go (͋Δͱศར) $ nix-env -i go-1.8.3 $ nix-env -i google-app-engine-go-sdk
12. ؀ڥΛબͿ
13. Computing Option • App Engine Standard Environment (GAE SE) • App Engine Flexible Environment (GAE FE) • Container Engine (GKE) • Compute Engine (GCE) GCP ͷίϯϐϡʔτ Φϓγϣϯ ΨΠυΛެ։ https://cloudplatform-jp.googleblog.com/2016/06/gcp_17.html
14. GAE/Go SEΛ࢖͏ • ғ͍ࠐΈͷϦεΫ͸͋Δ͚Ͳ... • ӡ༻ʹखؒΛ͔͚ͨ͘ͳ͔ͬͨ • ແྉ࿮͕͋Δͷخ͍͠ • RDB࢖͑ͳͯ͘΋ࠔΒͳ͍
15. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
16. Ϣʔβೝূ(1) • Users API (google.golang.org/appengine/user) • ͱͯ΋؆୯ʹ࢖͑Δ • Current(); CurrentOAuth(); LoginURL() • GoogleΞΧ΢ϯτ͔͠ରԠ͍ͯ͠ͳ͍ • GitHubΞΧ΢ϯτͰ΋ϩάΠϯ͍ͨ͠... • ࣗલ࣮૷ • ໘౗͍͘͞
17. Ϣʔβೝূ(2) • Cloud Identity-Aware Proxy (IAP) • Cloud Endpoints
18. Cloud Identity-Aware Proxy • ೝূ͞ΕͨϢʔβʹΞϓϦΛެ։͢Δ • GoogleΞΧ΢ϯτલఏͷΑ͏ͳͷͰະ࢖༻
19. Cloud Endpoints • REST APIΛ࡞ΔͨΊͷαʔϏε • ϩΪϯάɺϞχλϦϯάɺอޢͳͲߦ͏ • GAE SE: Cloud Endpoints Framework • ͦΕҎ֎: Extensible Service Proxy Extensible Service Proxy ͱ Endpoints Frameworks ͷൺֱ https://cloud.google.com/endpoints/docs/frameworks-extensible-service-proxy?hl=ja
20. Cloud Endpoints • Cloud Endpoints Framework
 github.com/GoogleCloudPlatform/go-endpoints/endpoints • ඪ४Ͱ͸Issuer = accounts.google.com • AuthenticatorΛ࣮૷͢Ε͹৭ʑ΍Εͦ͏
21. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
22. go-endpointsΛ࢖͏ • app.yamlʹhandlerΛ௥Ճ application: [app-name] version: 1 runtime: go api_version: go1 handlers: - url: /_ah/spi/.* script: _go_app
23. Go 1.8Λ࢖͏ • api_versionΛมߋ͢Δ(1.9.55Ҏ্) application: [app-name] version: 1 runtime: go api_version: go1.8 handlers: - url: /_ah/spi/.* script: _go_app Google App Engine for Go͕Go1.8ʹରԠͨ͠ͷͰࢼͯ͠Έͨ http://qiita.com/tenntenn/items/0b92fc089f8826fabaf1
24. go-endpointsͰAPI࣮૷ • APIͱͯ͠ѻ͏ϝιου͸ಛఆͷϧʔϧΛຬͨ͢ • JSON΁ͷม׵౳͸উखʹ΍ͬͯ͘ΕΔ type Service struct{} type ListOptions struct { Limit int `json:"limit" endpoints:"d=10"` } type ArticleList struct { Items []*Article `json:"items"` } type Article struct { URL string `json:"url"` } func (s *Service) ArticleList(c context.Context, r *ListOptions) (*ArticleList, error) { return &ArticleList{ Items: []*Article{ &Article{URL: "http://example.com/"}, }, }, nil }
25. go-endpointsͷURL Router • ࡞ͬͨAPIΛURLʹϚοϓ͢Δ func init() { srv := new(Service) api, err := endpoints.RegisterService(srv, "logs", "v1", "Reading Log API", true) if err != nil { log.Fatal(err) } m := api.MethodByName("ArticleList") if m == nil { log.Fatal("Missing ArticleList method") } info := m.Info() info.Name = "article.list" info.Path = "articles" info.HTTPMethod = "GET" info.Desc = "List most recent reading articles." endpoints.HandleHTTP() } APIαʔϏεొ࿥ URLͷઃఆ ͜ͷྫ͸ https://[app-name].appspot.com/_ah/api/logs/v1/articles ͱͳΔ
26. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
27. go-endpointsͰೝূ • infoʹɺೝূ͢ΔͨΊʹඞཁͳ஋Λઃఆ͢Δ • API Manager / ೝূ৘ใ / ΫϥΠΞϯτIDͷ࡞੒ ͰWeb Application Λબ୒ • ੜ੒͞ΕͨΫϥΠΞϯτIDͱγʔΫϨοτΛอଘ const ClientID = "xxxx" func init() { info := m.Info() info.Name = "article.list" info.Path = "articles" info.HTTPMethod = "GET" info.Desc = "List most recent reading articles." info.Scopes = []string{endpoints.EmailScope} info.Audiences = []string{"[app-name].appspot.com"} info.ClientIds = []string{ClientID, endpoints.APIExplorerClientID} endpoints.HandleHTTP() }
28. go-endpointsͰೝূ • infoͱಉ͡஋Λ࢖ͬͯCurrentUser()Λݺͼग़͢ func (s *Service) ArticleList(c context.Context, r *ListOptions) (*ArticleList, error) { scopes := []string{endpoints.EmailScope} audiences := []string{"[app-name].appspot.com"} clientIDs := []string{ClientID, endpoints.APIExplorerClientID} user, err := endpoints.CurrentUser(c, scopes, audiences, clientIDs) if err != nil { return nil, err } return &ArticleList{ Items: []*Article{ &Article{URL: "http://example.com/"}, }, }, nil }
29. API ExplorerͰςετ • https://[app-name].appspot.com/_ah/api/explorer • ೝূ͕ඞཁͱѻΘΕΔΑ͏ʹͳ͍ͬͯΔ
30. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
31. DatastoreΛ࢖͏ • NoSQLܕσʔλϕʔε • Keyͷઃܭ͕೉͍͠ • Cloud Storage΍SQLܕDB΋͋Δ͚Ͳ...
32. Datastore΁อଘ • ϢʔβIDͱهࣄͷURLͰ֊૚ʹ͍ͯ͠Δ type Page struct { URL string LastMod time.Time } func AddEntry(c context.Context, u *User, url string) error { p := datastore.NewKey(c, "User", u.ID, 0, nil) key := datastore.NewKey(c, "Page", url, 0, p) return datastore.RunInTransaction(c, func(c context.Context) error { var page Page err := datastore.Get(c, key, &page) if err != nil && err != datastore.ErrNoSuchEntity { return err } page.URL = url page.LastMod = time.Now() _, err = datastore.Put(c, key, &page) return err }, nil) }
33. Datastore΁อଘ • datastore.RunInTransaction()Ͱߋ৽͢Δ type Page struct { URL string LastMod time.Time } func AddEntry(c context.Context, u *User, url string) error { p := datastore.NewKey(c, "User", u.ID, 0, nil) key := datastore.NewKey(c, "Page", url, 0, p) return datastore.RunInTransaction(c, func(c context.Context) error { var page Page err := datastore.Get(c, key, &page) if err != nil && err != datastore.ErrNoSuchEntity { return err } page.URL = url page.LastMod = time.Now() _, err = datastore.Put(c, key, &page) return err }, nil) } ࣦഊͷ৔߹͸ϦτϥΠ͞ΕΔͷͰɺ ࠷৽ͷ஋ΛಡΈ௚͔ͯ͠Βߋ৽
34. هࣄ಺༰ͷऔಘ • Goඪ४ͷhttp.Get()͸੍ݶ͞Ε͍ͯΔ • ୅ΘΓʹurlfetch.ClientΛ࢖͏ • API࣮ߦͷཪͰऔಘ౳Λ͍ͨ͠ • GAE/Go SEͷϦΫΤετ͸60ඵͰλΠϜΞ΢τ • Task QueueΛ࢖͏ • Cloud Pub/Sub͸Subscribe͢Δଆ͕ෳ਺ͷ৔߹ʹศརͦ͏
35. Task Queue • Push QueueͱPull Queueͷ2छྨ͋Δ • GAE/Go SE͸Push Queue • ಛఆͷΤϯυϙΠϯτʹϦΫΤετ͞ΕΔ • GAE͔ΒͷΈ࣮ߦ͍ͤͨ͞
36. Task Queue༻URL௥Ճ • Task Queue͕ϦΫΤετ͢ΔURLΛ௥Ճ application: [app-name] version: 1 runtime: go api_version: go1.8 handlers: - url: /_ah/spi/.* script: _go_app - url: /_ah/queue/.* script: _go_app login: admin ؅ཧऀͷΈΞΫηεՄೳ
37. Ωϡʔͷઃఆ • queue.yamlΛ࡞੒ͯ͠ΩϡʔΛઃఆ • ৭ʑͰ͖ΔͷͰެࣜυΩϡϝϯτΛಡ΋͏ queue: - name: snap rate: 1/s bucket_size: 5
38. Task Queue࣮૷ • ΩϡʔʹೖΕ࣮ͯߦɺΤϥʔͳΒϦτϥΠ func AddTask(c context.Context, url string) error { v := url.Values{} v.Set("url", url) t := taskqueue.NewPOSTTask("/_ah/queue/snap", v) _, err := taskqueue.Add(c, t, "snap") return err } func SnapQueueHandler(w http.ResponseWriter, r *http.Request) { c := appengine.NewContext(r) url := r.FormValue("url") cli := urlfetch.Client(c) resp, err := cli.Get(uri) if err != nil { w.WriteHeader(http.StatusInternalServerError) return } ... }
39. Task Queueϋϯυϥ௥Ճ • go-endpoints͕endpoints.HandleHTTP()͍ͯ͠Δ͕ɺ
 ී௨ʹͦͷ··Goඪ४net/httpͰ௥Ճ͢Ε͹͍͍ const ClientID = "xxxx" func init() { info := m.Info() info.Name = "article.list" info.Path = "articles" info.HTTPMethod = "GET" info.Desc = "List most recent reading articles." info.Scopes = []string{endpoints.EmailScope} info.Audiences = []string{"[app-name].appspot.com"} info.ClientIds = []string{ClientID, endpoints.APIExplorerClientID} http.HandleFunc("/_ah/queue/snap", SnapQueueHandler) endpoints.HandleHTTP() }
40. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
41. Χʔιϧͷར༻ • Datastore͸೚ҙͷϖʔδʹҠಈ͢Δͷ͸޲͔ͳ͍
 ࢓༷͔Β֎ͨ͠΄͏͕͍͍ • Query.Offset()͸ಡΈඈ͹͕͢ಡΈࠐΈ͸͍ͯ͠Δ
 ಡΈࠐΈճ਺Ͱ՝ۚ͞ΕΔ... • ΧʔιϧΛ࢖ͬͯલޙͷ஋Λऔಘ͢Δͱ͍͍ • ͨͩ͠Χʔιϧ͕࡞ΒΕͨΫΤϦͱಉ͡৚͕݅ඞཁ
42. Χʔιϧͷ࢖͍ํ func FetchEntries(c context.Context, u *User, cursor string) ([]*Page, string, error) { key := datastore.NewKey(c, "User", u.ID, 0, nil) q := datastore.NewQuery("Page").Ancestor(key).Limit(20) if cursor != "" { p, err := datastore.DecodeCursor(cursor) if err != nil { return nil, "", err } q = q.Start(p) } var a []*Page t := q.Run(c) for { var page Page _, err := t.Next(&page) ... লུ ... } p, err := t.Cursor() if err != nil { return nil, "", err } return a, p.String(), nil } ಉ౳ͳΫΤϦͰऔಘͨ͠Χʔιϧ
43. ࡞ΔαʔϏεͷ࢓༷ • Ϣʔβຖʹ෼͚ͯ؅ཧ͢Δ • APIͱ࣮ͯ͠૷͢Δ • ಡΜͩهࣄΛऔࣺબ୒ͤͣશ෦อଘ͢Δ • ࣌ܥྻʹಡΜͩϦετΛऔಘ͢Δ • อଘͨ͠هࣄ͔Βશจݕࡧ͢Δ
44. Search API • Datastore͸Ωʔϫʔυݕࡧ͕ۤख • Search APIͰ୅༻͢Δ • ͨͩ͠Datastoreͱ͸ผͷ؅ཧʹͳΔ
45. • ΠϯσοΫεʹɺෳ਺ͷυΩϡϝϯτΛొ࿥ Search APIͷ࢖͍ํ import "google.golang.org/appengine/search" type Document struct { Content search.HTML LastMod time.Time } func AddContent(c context.Context, uid string, body []byte) error { doc := Document{ Content: search.HTML(body), LastMod: time.Now(), } x, err := search.Open("article_" + uid) if err != nil { return err } if _, err := x.Put(c, uri, &doc); err != nil { return err } return nil } υΩϡϝϯτΛఆٛ ΠϯσοΫε΁௥Ճ
46. ݕࡧͯ͠ΈΔ Query String https://cloud.google.com/appengine/docs/standard/go/search/query_strings
47. • ΠϯσοΫε͔ΒΩʔϫʔυݕࡧ...ͷ͸͕ͣ... Search APIͷ࢖͍ํ func Query(c context.Context, uid, expr string, cursor Cursor) ([]*Document, string, error) { x, err := search.Open("article_" + uid) if err != nil { return nil, "", err } var a []*Document if cursor != "" { options.Cursor = search.Cursor(cursor) } t := x.Search(c, expr, nil) for { var doc Document _, err := t.Next(&doc) if err == search.Done { return a, string(t.Cursor()), nil } if err != nil { return nil, "", err } a = append(a, &doc) } }
48. ·ͱΊ • ϚωʔδυαʔϏε͓खܰ • झຯͰॻ͘ίʔυͳΒ໎Θͣ࢖͏ • ϕϯμʔϩοΫΠϯ͸ؾΛ͚ͭͯ...
49. ͋Γ͕ͱ͏͍͟͝·ͨ͠