Slide 1

Slide 1 text

go.mercari.io/datastore ͸͍͍ͧʂ ajn #3 Θ͔Ί ·͞ͻΖ

Slide 2

Slide 2 text

Θ͔Ί ·͞ͻΖ @v vakame TypeScript Masahiro Wakame ٕज़ॻయ appengine/go photo from golang.org/doc/gopher/

Slide 3

Slide 3 text

GAE/SE Node.js β Ͱ΋ࠓ೔ͷ࿩͸GAE/GoͰ͢

Slide 4

Slide 4 text

͸͡Ίʹ • ׂΓࠐΈ࣭໰׻ܴʂ • ฉ͖͍ͨ͜ͱΛฉ͍ͯ΄͍͠ • ࡞ऀͳͷͰશମ૾Λطʹ஌͍ͬͯΔ • Կ͕Θ͔ΒΜ͔Θ͔ΒΜͷͩ… ࢲ͕࡞ऀͰ͢

Slide 5

Slide 5 text

go.mercari.io/datastore ࠓ೔ͷ࿩୊Ͱ͢

Slide 6

Slide 6 text

ઌ೔v1.0.0ϦϦʔε ҆ఆͯ͠Δͥʂ

Slide 7

Slide 7 text

ͦΕԿʁ

Slide 8

Slide 8 text

ͦΕԿʁ • (AppEngine|Cloud) Datastore Wrapper • In Go! Seamless! • Middleware • Retry, Caching, Logging… etc • Batch operation • Single-Op Batch-Op aggregation

Slide 9

Slide 9 text

ઃܭࢥ૝ • Cloud DatastoreͷAPIʹͳΔ΂͘دͤΔ • ༨ܭͳ͜ͱ͸ͳΔ΂͘͠ͳ͍ • ErrFieldMismatchແࢹͷσϑΥϧτԽ • RunInTransactionͷࣗಈRetryͷ࡟আ • PropertyLoadServerʹctx͋͛ͨ • Batchૢ࡞͸଍ͨ͠

Slide 10

Slide 10 text

(AppEngine|Cloud) Datastore

Slide 11

Slide 11 text

ͦ΋ͦ΋ Datastoreͬͯ2छྨ͋Δɻ ͦͷ͜ͱΛ஌͍ͬͯ·͔͢ʁ

Slide 12

Slide 12 text

Datastore Service AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore GAE Instance Any Server RPC Endpoint Package Runtime Env

Slide 13

Slide 13 text

Datastore Storage AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore GAE Instance Any Server RPC Endpoint Package Runtime Env ଘࡏ͢Δػೳʹ ݁ߏ͕ࠩ͋Δ ৽ چ

Slide 14

Slide 14 text

͡Ό͋৽࢖͑͹Α͘Ͷʁ • GAE/SE Go͔Βͷར༻ʹ੍໿͋Γ • Socket API͕ඞཁ… ΊΜͲ͍… • Cloudͷํɺͨ·ʹτϥϒͬͯΔ • GCPUG Slack #g-datastore_ja ࢖͍෼͚͕·ͩ·ͩඞཁͩʂ

Slide 15

Slide 15 text

ޓ׵ੑ… • (AppEngine|Cloud) Datastore͸… • ཪͷ࣮૷͸ಉ͡ • ஌ݟ΍ઃܭ͸9ׂҎ্࢖͍ճͤΔ • ͨͩ͠ɺAPI͕ҟͳΔʂ • ͳΜͳΒpackage΋׬શʹผ

Slide 16

Slide 16 text

ΑΖ͍͠ʂ ͳΒ͹Wrapperͩʂ

Slide 17

Slide 17 text

Wrapperͷಛ௃ • Cloud Datastore(৽)ͷAPI͕ϕʔε • ࣮૷ͷ(AppEngine|Cloud)ࠩସՄ • goonϢʔβͷͨΊʹboom΋͋Γ·͢

Slide 18

Slide 18 text

package༡ཡ • godoc root • godoc aedatastore • godoc clouddatastore

Slide 19

Slide 19 text

͋ΓಘΔະདྷ • AppEngine→GKE΁ͷࡌͤସ͑ • ཧ࿦্ΠέΔؾ͕͢Δ… • ࣄྫ͸ࣾ಺Ͱ΋ແ͍໛༷ • gVisorϕʔεͷGAE/SE૿͑ͯΔͬΆ͍ • GAE/Go΋কདྷϫϯνϟϯඍཻࢠ…ʁ

Slide 20

Slide 20 text

ҠߦLIVE CODING • ΛɺޙͰ΍Δ͕࣌ؒ͋Δͱ͍͍ͳ͊

Slide 21

Slide 21 text

• datastore:”,flatten” λάͷଘࡏ • Cloud DatastoreͷAPIʹ͋ΘͤͯΔ • ίʔυͷॻ͖׵͚͑ͩͰ͸μϝ • structͷλά෇͚Λݟ௚͢ʂ ςετ͸͔ͬ͠Γʂ ಛʹ஫ҙ͢΂͖ࠩ෼

Slide 22

Slide 22 text

flatten? • datastore.Entity ʹ͍ͭͯ • AE → Entity͸Entity࣋ͯͳ͍ • Cloud → Entity͸Entity࣋ͯΔ • struct͸Entityʹม׵͞ΕΔ • nest ͨ͠struct͸EntityͷೖΕࢠʹͳΔ

Slide 23

Slide 23 text

flatten? type AEEntity struct { "Slice.A" []string "Slice.B" []string } ͜ͷίʔυྫ͸ ΠϝʔδͰ͢ type Inner struct { A string B string } type Data struct { Slice []Inner } type CloudEntity struct { Slice []struct{ A string B string } }

Slide 24

Slide 24 text

flatten? type Inner struct { A string B string } type Data struct { Slice []Inner } type Data struct { Slice []Inner ↩ `datastore:",flatten"` } flattenͰAEͱಉ͡ʹ

Slide 25

Slide 25 text

☠ ATTENTION ☠ • Cloud DatastoreͰNestͨ͠Entity࡞Δͱ AppEngine Datastore͔ΒಡΊͳ͍ • ͸ͣ(ࢼͯ͠ͳ͍) • ؾܰͳؾ࣋ͪͰAppEngine͔ΒGetͯ͠ CloudʹPut͢Δͱࢮ͵Մೳੑ͕͋Δ ςετ͸͔ͬ͠ΓͶʂ

Slide 26

Slide 26 text

• KeyLoader etc ͷCloudଆಠ઎ػೳ։์ • UnitTestͰCloud Datastore Emulator • GoLandͰσόοΨ࢖͍΍͍͢ • Local͔Βσʔλ౤ೖ͕༰қ ͔͠͠΍ͬͨ͜ͱ͸ͳ͍ʂ ศརͳ఺

Slide 27

Slide 27 text

Middleware

Slide 28

Slide 28 text

ͳͥཉ͍͔͠ • Middlewareͷ༻్ • RPCෆௐ࣌ͷϦτϥΠ • MemcacheͳͲͷར༻ • ϩάग़ྗ • େྔEntity Put࣌ͷRPC਺෼ׂ

Slide 29

Slide 29 text

ΞϓϦͷ࣮૷ʹ ूதͯ͐͠ʂ

Slide 30

Slide 30 text

Middlewareʹ ΍ΒͤΑ͏ʂ

Slide 31

Slide 31 text

ࠓ͋ΔMiddleware • AE Memcache • In-Memory Cache • Automatic RPC Retry • Redis(Memorystore) Cache • Logger, Chaos RPC, Fishbone … • ↑ओʹςετ΍ݕূ༻ ຊ൪ར༻࣮੷ͷน

Slide 32

Slide 32 text

Batch Operation

Slide 33

Slide 33 text

͜͏͍͏ίʔυ͕ॻ͚Δ type Comment struct { Message string } type Post struct { Content string CommentIDs []int64 `json:"-"` Comments []*Comment `datastore:"-"` } // Entityͷ४උ(ׂѪ) ……… // start fetching... posts := make([]*Post, 0) _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! bt := client.Batch() for _, post := range posts { comments := make([]*Comment, 0) for _, id := range post.CommentIDs { comment := &Comment{} bt.Get(client.IDKey("Comment", id, nil), comment, nil) comments = append(comments, comment) } post.Comments = comments } err = bt.Exec(ctx) if err != nil { panic(err) }

Slide 34

Slide 34 text

΋͠΋Batch͕ͳ͔ͬͨΒ // start fetching... posts := make([]*Post, 0) _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! var commentKeys []datastore.Key for _, post := range posts { for _, commentID := range post.CommentIDs { commentKeys = append(commentKeys, client.IDKey("Comment", commentID, nil)) } } comments := make([]*Comment, len(commentKeys)) err = client.GetMulti(ctx, commentKeys, comments) if err != nil { panic(err) } for _, post := range posts { for _, commentID := range post.CommentIDs { for idx, comment := range comments { key := commentKeys[idx] if commentID == key.ID() { post.Comments = append(post.Comments, comment) break } } } } ErrNoSuchEntity͕ ࠞͬͨ͟Β͞ΒʹͭΒ͍…ʂ

Slide 35

Slide 35 text

ErrNoSuchEntityରԠ bt.Get(key, comment, func(err error) error { if err == datastore.ErrNoSuchEntity { // ignore ErrNoSuchEntity return nil } else if err != nil { return err } post.Comments = append(post.Comments, comment) return nil }) Entity୯ҐͷΤϥʔϋϯυϥ༗Γ

Slide 36

Slide 36 text

҆ఆ͍ͯ͠Δ͔

Slide 37

Slide 37 text

ར༻࣮੷༗ • ϝϧΧϦ Ξος (AppEngine) • ϝϧΧϦ KYC (Cloud) • ٕज़ॻయ (AppEngine) • timakin͞Μ (AppEngine) ར༻ใࠂ଴ͬͯ·͢❤

Slide 38

Slide 38 text

όάݕग़ 1݅ͷΈʂ༏लʂ

Slide 39

Slide 39 text

ςετεΠʔτ • Goͷαϒςετͷ࢓૊Έ͕࠷ߴ • https://blog.golang.org/subtests • େྔͷςετΛ༻ҙ͢Δ • go.mercari.io/datastore/testsuite • AE, Cloud Ͱಉ͡ςετΛͿΜճ͢ • ↑ޓ׵ੑ

Slide 40

Slide 40 text

Ҡߦखॱ ཁ͢Δʹطଘͱͷࠩ෼

Slide 41

Slide 41 text

جຊతͳํ਑ • API͸Cloudͱ΄΅ಉ౳ • ϝιου໊ͱ͔ • iterator.Done Λ࢖͏ͱ͔ • datastore.MultiError ͱ͔ • struct͸interfaceʹஔ͖׵͑ • AEͱCloudͰ࣮૷͕ҟͳΔͨΊ

Slide 42

Slide 42 text

godocݟͯ͘Εʂ How to migrate to this library Here's an overview of what you need to do to migrate your existing code. replace *datastore.Key to datastore.Key. replace *datastore.Query to datastore.Query. replace *datastore.Iterator to datastore.Iterator. from AE Datastore import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both. rewrite those using functions of datastore package to FromContext function and Client method calls. replace err.(appengine.MultiError) to err.(datastore.MultiError) . Stop using appengine.BlobKey and replace with string. replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done . replace key.IntID() to key.ID() . replace key.StringID() to key.Name() . When nesting a struct, apply `datastore:", flatten "` to the corresponding field. Delete datastore.TransactionOptions, it is not supported. If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore . from Cloud Datastore import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore . rewrite those using functions of datastore package to FromContext function and Client method calls. replace *datastore.Commit to datastore.Commit . If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore . from goon to boom replace *goon.Goon to *boom.Boom . replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) . ଍Γͳ͍ͷݟ͚ͭͨΒIssue΁ʂ

Slide 43

Slide 43 text

ҠߦLIVE CODING github.com/vvakame/ucon-todo

Slide 44

Slide 44 text

Welcome to contribution!

Slide 45

Slide 45 text

͚ͨͯ͢❤ • ɹɹɹɹɹɹ໨ࢦͤGitHub300←❤ • GCPUG SlackͰ࣭໰͢Δˡ❤❤ • Θ͔Βͳ͍͜ͱΛIssueʹ͢Δˡ❤❤❤ • ࢖ͬͯϒϩάͱ͔ॻ͘ˡ❤❤❤❤ • Pull RequestΛ͘ΕΔˡ❤❤❤❤❤❤❤ https://gcpug.jp/join

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

ۙگ ๻ͷपΓͰSpanner࢖͏ͱ͜૿͖͑ͯͯͯ΍͹͍ go.mercari.io/datastore ͷग़൪͕…(´ɾТɾ`) ͱ͍͏Θ͚ͰSpannerͷΫΤϦύʔαॻ͖࢝Ί·ͨ͠