go.mercari.io/datastore はいいぞ! / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore はいいぞ! / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore の紹介です。
https://gaeja.connpass.com/event/90133/

0e797080b64e6b03ed00964cf69b5058?s=128

Masahiro Wakame

June 14, 2018
Tweet

Transcript

  1. go.mercari.io/datastore ͸͍͍ͧʂ ajn #3 Θ͔Ί ·͞ͻΖ

  2. Θ͔Ί ·͞ͻΖ @v vakame TypeScript Masahiro Wakame ٕज़ॻయ appengine/go photo

    from golang.org/doc/gopher/
  3. GAE/SE Node.js β Ͱ΋ࠓ೔ͷ࿩͸GAE/GoͰ͢

  4. ͸͡Ίʹ • ׂΓࠐΈ࣭໰׻ܴʂ • ฉ͖͍ͨ͜ͱΛฉ͍ͯ΄͍͠ • ࡞ऀͳͷͰશମ૾Λطʹ஌͍ͬͯΔ • Կ͕Θ͔ΒΜ͔Θ͔ΒΜͷͩ… ࢲ͕࡞ऀͰ͢

  5. go.mercari.io/datastore ࠓ೔ͷ࿩୊Ͱ͢

  6. ઌ೔v1.0.0ϦϦʔε ҆ఆͯ͠Δͥʂ

  7. ͦΕԿʁ

  8. ͦΕԿʁ • (AppEngine|Cloud) Datastore Wrapper • In Go! Seamless! •

    Middleware • Retry, Caching, Logging… etc • Batch operation • Single-Op Batch-Op aggregation
  9. ઃܭࢥ૝ • Cloud DatastoreͷAPIʹͳΔ΂͘دͤΔ • ༨ܭͳ͜ͱ͸ͳΔ΂͘͠ͳ͍ • ErrFieldMismatchແࢹͷσϑΥϧτԽ • RunInTransactionͷࣗಈRetryͷ࡟আ

    • PropertyLoadServerʹctx͋͛ͨ • Batchૢ࡞͸଍ͨ͠
  10. (AppEngine|Cloud) Datastore

  11. ͦ΋ͦ΋ Datastoreͬͯ2छྨ͋Δɻ ͦͷ͜ͱΛ஌͍ͬͯ·͔͢ʁ

  12. Datastore Service AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore

    GAE Instance Any Server RPC Endpoint Package Runtime Env
  13. Datastore Storage AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore

    GAE Instance Any Server RPC Endpoint Package Runtime Env ଘࡏ͢Δػೳʹ ݁ߏ͕ࠩ͋Δ ৽ چ
  14. ͡Ό͋৽࢖͑͹Α͘Ͷʁ • GAE/SE Go͔Βͷར༻ʹ੍໿͋Γ • Socket API͕ඞཁ… ΊΜͲ͍… • Cloudͷํɺͨ·ʹτϥϒͬͯΔ

    • GCPUG Slack #g-datastore_ja ࢖͍෼͚͕·ͩ·ͩඞཁͩʂ
  15. ޓ׵ੑ… • (AppEngine|Cloud) Datastore͸… • ཪͷ࣮૷͸ಉ͡ • ஌ݟ΍ઃܭ͸9ׂҎ্࢖͍ճͤΔ • ͨͩ͠ɺAPI͕ҟͳΔʂ

    • ͳΜͳΒpackage΋׬શʹผ
  16. ΑΖ͍͠ʂ ͳΒ͹Wrapperͩʂ

  17. Wrapperͷಛ௃ • Cloud Datastore(৽)ͷAPI͕ϕʔε • ࣮૷ͷ(AppEngine|Cloud)ࠩସՄ • goonϢʔβͷͨΊʹboom΋͋Γ·͢

  18. package༡ཡ • godoc root • godoc aedatastore • godoc clouddatastore

  19. ͋ΓಘΔະདྷ • AppEngine→GKE΁ͷࡌͤସ͑ • ཧ࿦্ΠέΔؾ͕͢Δ… • ࣄྫ͸ࣾ಺Ͱ΋ແ͍໛༷ • gVisorϕʔεͷGAE/SE૿͑ͯΔͬΆ͍ •

    GAE/Go΋কདྷϫϯνϟϯඍཻࢠ…ʁ
  20. ҠߦLIVE CODING • ΛɺޙͰ΍Δ͕࣌ؒ͋Δͱ͍͍ͳ͊

  21. • datastore:”,flatten” λάͷଘࡏ • Cloud DatastoreͷAPIʹ͋ΘͤͯΔ • ίʔυͷॻ͖׵͚͑ͩͰ͸μϝ • structͷλά෇͚Λݟ௚͢ʂ

    ςετ͸͔ͬ͠Γʂ ಛʹ஫ҙ͢΂͖ࠩ෼
  22. flatten? • datastore.Entity ʹ͍ͭͯ • AE → Entity͸Entity࣋ͯͳ͍ • Cloud

    → Entity͸Entity࣋ͯΔ • struct͸Entityʹม׵͞ΕΔ • nest ͨ͠struct͸EntityͷೖΕࢠʹͳΔ
  23. flatten? type AEEntity struct { "Slice.A" []string "Slice.B" []string }

    ͜ͷίʔυྫ͸ ΠϝʔδͰ͢ type Inner struct { A string B string } type Data struct { Slice []Inner } type CloudEntity struct { Slice []struct{ A string B string } }
  24. flatten? type Inner struct { A string B string }

    type Data struct { Slice []Inner } type Data struct { Slice []Inner ↩ `datastore:",flatten"` } flattenͰAEͱಉ͡ʹ
  25. ☠ ATTENTION ☠ • Cloud DatastoreͰNestͨ͠Entity࡞Δͱ AppEngine Datastore͔ΒಡΊͳ͍ • ͸ͣ(ࢼͯ͠ͳ͍)

    • ؾܰͳؾ࣋ͪͰAppEngine͔ΒGetͯ͠ CloudʹPut͢Δͱࢮ͵Մೳੑ͕͋Δ ςετ͸͔ͬ͠ΓͶʂ
  26. • KeyLoader etc ͷCloudଆಠ઎ػೳ։์ • UnitTestͰCloud Datastore Emulator • GoLandͰσόοΨ࢖͍΍͍͢

    • Local͔Βσʔλ౤ೖ͕༰қ ͔͠͠΍ͬͨ͜ͱ͸ͳ͍ʂ ศརͳ఺
  27. Middleware

  28. ͳͥཉ͍͔͠ • Middlewareͷ༻్ • RPCෆௐ࣌ͷϦτϥΠ • MemcacheͳͲͷར༻ • ϩάग़ྗ •

    େྔEntity Put࣌ͷRPC਺෼ׂ
  29. ΞϓϦͷ࣮૷ʹ ूதͯ͐͠ʂ

  30. Middlewareʹ ΍ΒͤΑ͏ʂ

  31. ࠓ͋ΔMiddleware • AE Memcache • In-Memory Cache • Automatic RPC

    Retry • Redis(Memorystore) Cache • Logger, Chaos RPC, Fishbone … • ↑ओʹςετ΍ݕূ༻ ຊ൪ར༻࣮੷ͷน
  32. Batch Operation

  33. ͜͏͍͏ίʔυ͕ॻ͚Δ type Comment struct { Message string } type Post

    struct { Content string CommentIDs []int64 `json:"-"` Comments []*Comment `datastore:"-"` } // Entityͷ४උ(ׂѪ) ……… // start fetching... posts := make([]*Post, 0) _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! bt := client.Batch() for _, post := range posts { comments := make([]*Comment, 0) for _, id := range post.CommentIDs { comment := &Comment{} bt.Get(client.IDKey("Comment", id, nil), comment, nil) comments = append(comments, comment) } post.Comments = comments } err = bt.Exec(ctx) if err != nil { panic(err) }
  34. ΋͠΋Batch͕ͳ͔ͬͨΒ // start fetching... posts := make([]*Post, 0) _, err

    = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! var commentKeys []datastore.Key for _, post := range posts { for _, commentID := range post.CommentIDs { commentKeys = append(commentKeys, client.IDKey("Comment", commentID, nil)) } } comments := make([]*Comment, len(commentKeys)) err = client.GetMulti(ctx, commentKeys, comments) if err != nil { panic(err) } for _, post := range posts { for _, commentID := range post.CommentIDs { for idx, comment := range comments { key := commentKeys[idx] if commentID == key.ID() { post.Comments = append(post.Comments, comment) break } } } } ErrNoSuchEntity͕ ࠞͬͨ͟Β͞ΒʹͭΒ͍…ʂ
  35. ErrNoSuchEntityରԠ bt.Get(key, comment, func(err error) error { if err ==

    datastore.ErrNoSuchEntity { // ignore ErrNoSuchEntity return nil } else if err != nil { return err } post.Comments = append(post.Comments, comment) return nil }) Entity୯ҐͷΤϥʔϋϯυϥ༗Γ
  36. ҆ఆ͍ͯ͠Δ͔

  37. ར༻࣮੷༗ • ϝϧΧϦ Ξος (AppEngine) • ϝϧΧϦ KYC (Cloud) •

    ٕज़ॻయ (AppEngine) • timakin͞Μ (AppEngine) ར༻ใࠂ଴ͬͯ·͢❤
  38. όάݕग़ 1݅ͷΈʂ༏लʂ

  39. ςετεΠʔτ • Goͷαϒςετͷ࢓૊Έ͕࠷ߴ • https://blog.golang.org/subtests • େྔͷςετΛ༻ҙ͢Δ • go.mercari.io/datastore/testsuite •

    AE, Cloud Ͱಉ͡ςετΛͿΜճ͢ • ↑ޓ׵ੑ
  40. Ҡߦखॱ ཁ͢Δʹطଘͱͷࠩ෼

  41. جຊతͳํ਑ • API͸Cloudͱ΄΅ಉ౳ • ϝιου໊ͱ͔ • iterator.Done Λ࢖͏ͱ͔ • datastore.MultiError

    ͱ͔ • struct͸interfaceʹஔ͖׵͑ • AEͱCloudͰ࣮૷͕ҟͳΔͨΊ
  42. godocݟͯ͘Εʂ How to migrate to this library Here's an overview

    of what you need to do to migrate your existing code. replace *datastore.Key to datastore.Key. replace *datastore.Query to datastore.Query. replace *datastore.Iterator to datastore.Iterator. from AE Datastore import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both. rewrite those using functions of datastore package to FromContext function and Client method calls. replace err.(appengine.MultiError) to err.(datastore.MultiError) . Stop using appengine.BlobKey and replace with string. replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done . replace key.IntID() to key.ID() . replace key.StringID() to key.Name() . When nesting a struct, apply `datastore:", flatten "` to the corresponding field. Delete datastore.TransactionOptions, it is not supported. If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore . from Cloud Datastore import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore . rewrite those using functions of datastore package to FromContext function and Client method calls. replace *datastore.Commit to datastore.Commit . If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore . from goon to boom replace *goon.Goon to *boom.Boom . replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) . ଍Γͳ͍ͷݟ͚ͭͨΒIssue΁ʂ
  43. ҠߦLIVE CODING github.com/vvakame/ucon-todo

  44. Welcome to contribution!

  45. ͚ͨͯ͢❤ • ɹɹɹɹɹɹ໨ࢦͤGitHub300←❤ • GCPUG SlackͰ࣭໰͢Δˡ❤❤ • Θ͔Βͳ͍͜ͱΛIssueʹ͢Δˡ❤❤❤ • ࢖ͬͯϒϩάͱ͔ॻ͘ˡ❤❤❤❤

    • Pull RequestΛ͘ΕΔˡ❤❤❤❤❤❤❤ https://gcpug.jp/join
  46. None
  47. ۙگ ๻ͷपΓͰSpanner࢖͏ͱ͜૿͖͑ͯͯͯ΍͹͍ go.mercari.io/datastore ͷग़൪͕…(´ɾТɾ`) ͱ͍͏Θ͚ͰSpannerͷΫΤϦύʔαॻ͖࢝Ί·ͨ͠