go.mercari.io/datastore はいいぞ！ / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore の紹介です。
https://gaeja.connpass.com/event/90133/

Transcript

1. go.mercari.io/datastore ͸͍͍ͧʂ
   ajn #3
   Θ͔Ί ·͞ͻΖ
   

   View Slide

2. Θ͔Ί ·͞ͻΖ @v vakame
   TypeScript
   Masahiro Wakame
   ٕज़ॻయ
   appengine/go
   photo from golang.org/doc/gopher/
   

   View Slide

3. GAE/SE Node.js β
   Ͱ΋ࠓ೔ͷ࿩͸GAE/GoͰ͢
   

   View Slide

4. ͸͡Ίʹ
   • ׂΓࠐΈ࣭໰׻ܴʂ
   • ฉ͖͍ͨ͜ͱΛฉ͍ͯ΄͍͠
   • ࡞ऀͳͷͰશମ૾Λطʹ஌͍ͬͯΔ
   • Կ͕Θ͔ΒΜ͔Θ͔ΒΜͷͩ…
   ࢲ͕࡞ऀͰ͢
   

   View Slide

5. go.mercari.io/datastore
   ࠓ೔ͷ࿩୊Ͱ͢
   

   View Slide

6. ઌ೔v1.0.0ϦϦʔε
   ҆ఆͯ͠Δͥʂ
   

   View Slide

7. ͦΕԿʁ
   

   View Slide

8. ͦΕԿʁ
   • (AppEngine|Cloud) Datastore Wrapper
   • In Go! Seamless!
   • Middleware
   • Retry, Caching, Logging… etc
   • Batch operation
   • Single-Op Batch-Op aggregation
   

   View Slide

9. ઃܭࢥ૝
   • Cloud DatastoreͷAPIʹͳΔ΂͘دͤΔ
   • ༨ܭͳ͜ͱ͸ͳΔ΂͘͠ͳ͍
   • ErrFieldMismatchແࢹͷσϑΥϧτԽ
   • RunInTransactionͷࣗಈRetryͷ࡟আ
   • PropertyLoadServerʹctx͋͛ͨ
   • Batchૢ࡞͸଍ͨ͠
   

   View Slide

10. (AppEngine|Cloud) Datastore
    

    View Slide

11. ͦ΋ͦ΋
    Datastoreͬͯ2छྨ͋Δɻ
    ͦͷ͜ͱΛ஌͍ͬͯ·͔͢ʁ
    

    View Slide

12. Datastore Service
    AppEngine RPC Cloud RPC
    google.golang.org/
    appengine/datastore
    cloud.google.com/
    go/datastore
    GAE Instance Any Server
    RPC
    Endpoint
    Package
    Runtime
    Env
    

    View Slide

13. Datastore Storage
    AppEngine RPC Cloud RPC
    google.golang.org/
    appengine/datastore
    cloud.google.com/
    go/datastore
    GAE Instance Any Server
    RPC
    Endpoint
    Package
    Runtime
    Env
    ଘࡏ͢Δػೳʹ
    ݁ߏ͕ࠩ͋Δ
    ৽
    چ
    

    View Slide

14. ͡Ό͋৽࢖͑͹Α͘Ͷʁ
    • GAE/SE Go͔Βͷར༻ʹ੍໿͋Γ
    • Socket API͕ඞཁ… ΊΜͲ͍…
    • Cloudͷํɺͨ·ʹτϥϒͬͯΔ
    • GCPUG Slack #g-datastore_ja
    ࢖͍෼͚͕·ͩ·ͩඞཁͩʂ
    

    View Slide

15. ޓ׵ੑ…
    • (AppEngine|Cloud) Datastore͸…
    • ཪͷ࣮૷͸ಉ͡
    • ஌ݟ΍ઃܭ͸9ׂҎ্࢖͍ճͤΔ
    • ͨͩ͠ɺAPI͕ҟͳΔʂ
    • ͳΜͳΒpackage΋׬શʹผ
    

    View Slide

16. ΑΖ͍͠ʂ
    ͳΒ͹Wrapperͩʂ
    

    View Slide

17. Wrapperͷಛ௃
    • Cloud Datastore(৽)ͷAPI͕ϕʔε
    • ࣮૷ͷ(AppEngine|Cloud)ࠩସՄ
    • goonϢʔβͷͨΊʹboom΋͋Γ·͢
    

    View Slide

18. package༡ཡ
    • godoc root
    • godoc aedatastore
    • godoc clouddatastore
    

    View Slide

19. ͋ΓಘΔະདྷ
    • AppEngine→GKE΁ͷࡌͤସ͑
    • ཧ࿦্ΠέΔؾ͕͢Δ…
    • ࣄྫ͸ࣾ಺Ͱ΋ແ͍໛༷
    • gVisorϕʔεͷGAE/SE૿͑ͯΔͬΆ͍
    • GAE/Go΋কདྷϫϯνϟϯඍཻࢠ…ʁ
    

    View Slide

20. ҠߦLIVE CODING
    • ΛɺޙͰ΍Δ͕࣌ؒ͋Δͱ͍͍ͳ͊
    

    View Slide

21. • datastore:”,flatten” λάͷଘࡏ
    • Cloud DatastoreͷAPIʹ͋ΘͤͯΔ
    • ίʔυͷॻ͖׵͚͑ͩͰ͸μϝ
    • structͷλά෇͚Λݟ௚͢ʂ
    ςετ͸͔ͬ͠Γʂ
    ಛʹ஫ҙ͢΂͖ࠩ෼
    

    View Slide

22. flatten?
    • datastore.Entity ʹ͍ͭͯ
    • AE → Entity͸Entity࣋ͯͳ͍
    • Cloud → Entity͸Entity࣋ͯΔ
    • struct͸Entityʹม׵͞ΕΔ
    • nest ͨ͠struct͸EntityͷೖΕࢠʹͳΔ
    

    View Slide

23. flatten?
    type AEEntity struct {
    "Slice.A" []string
    "Slice.B" []string
    }
    ͜ͷίʔυྫ͸
    ΠϝʔδͰ͢
    type Inner struct {
    A string
    B string
    }
    type Data struct {
    Slice []Inner
    }
    type CloudEntity struct {
    Slice []struct{
    A string
    B string
    }
    }
    

    View Slide

24. flatten?
    type Inner struct {
    A string
    B string
    }
    type Data struct {
    Slice []Inner
    }
    type Data struct {
    Slice []Inner ↩
    `datastore:",flatten"`
    }
    flattenͰAEͱಉ͡ʹ
    

    View Slide

25. ☠ ATTENTION ☠
    • Cloud DatastoreͰNestͨ͠Entity࡞Δͱ
    AppEngine Datastore͔ΒಡΊͳ͍
    • ͸ͣ(ࢼͯ͠ͳ͍)
    • ؾܰͳؾ࣋ͪͰAppEngine͔ΒGetͯ͠
    CloudʹPut͢Δͱࢮ͵Մೳੑ͕͋Δ
    ςετ͸͔ͬ͠ΓͶʂ
    

    View Slide

26. • KeyLoader etc ͷCloudଆಠ઎ػೳ։์
    • UnitTestͰCloud Datastore Emulator
    • GoLandͰσόοΨ࢖͍΍͍͢
    • Local͔Βσʔλ౤ೖ͕༰қ
    ͔͠͠΍ͬͨ͜ͱ͸ͳ͍ʂ
    ศརͳ఺
    

    View Slide

27. Middleware
    

    View Slide

28. ͳͥཉ͍͔͠
    • Middlewareͷ༻్
    • RPCෆௐ࣌ͷϦτϥΠ
    • MemcacheͳͲͷར༻
    • ϩάग़ྗ
    • େྔEntity Put࣌ͷRPC਺෼ׂ
    

    View Slide

29. ΞϓϦͷ࣮૷ʹ
    ूதͯ͐͠ʂ
    

    View Slide

30. Middlewareʹ
    ΍ΒͤΑ͏ʂ
    

    View Slide

31. ࠓ͋ΔMiddleware
    • AE Memcache
    • In-Memory Cache
    • Automatic RPC Retry
    • Redis(Memorystore) Cache
    • Logger, Chaos RPC, Fishbone …
    • ↑ओʹςετ΍ݕূ༻
    ຊ൪ར༻࣮੷ͷน
    

    View Slide

32. Batch Operation
    

    View Slide

33. ͜͏͍͏ίʔυ͕ॻ͚Δ
    type Comment struct {
    Message string
    }
    type Post struct {
    Content string
    CommentIDs []int64 `json:"-"`
    Comments []*Comment `datastore:"-"`
    }
    // Entityͷ४උ(ׂѪ)
    ………
    // start fetching...
    posts := make([]*Post, 0)
    _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts)
    if err != nil {
    panic(err)
    }
    // Let's batch get!
    bt := client.Batch()
    for _, post := range posts {
    comments := make([]*Comment, 0)
    for _, id := range post.CommentIDs {
    comment := &Comment{}
    bt.Get(client.IDKey("Comment", id, nil), comment, nil)
    comments = append(comments, comment)
    }
    post.Comments = comments
    }
    err = bt.Exec(ctx)
    if err != nil {
    panic(err)
    }
    

    View Slide

34. ΋͠΋Batch͕ͳ͔ͬͨΒ
    // start fetching...
    posts := make([]*Post, 0)
    _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts)
    if err != nil {
    panic(err)
    }
    // Let's batch get!
    var commentKeys []datastore.Key
    for _, post := range posts {
    for _, commentID := range post.CommentIDs {
    commentKeys = append(commentKeys, client.IDKey("Comment", commentID, nil))
    }
    }
    comments := make([]*Comment, len(commentKeys))
    err = client.GetMulti(ctx, commentKeys, comments)
    if err != nil {
    panic(err)
    }
    for _, post := range posts {
    for _, commentID := range post.CommentIDs {
    for idx, comment := range comments {
    key := commentKeys[idx]
    if commentID == key.ID() {
    post.Comments = append(post.Comments, comment)
    break
    }
    }
    }
    }
    ErrNoSuchEntity͕
    ࠞͬͨ͟Β͞ΒʹͭΒ͍…ʂ
    

    View Slide

35. ErrNoSuchEntityରԠ
    bt.Get(key, comment, func(err error) error {
    if err == datastore.ErrNoSuchEntity {
    // ignore ErrNoSuchEntity
    return nil
    } else if err != nil {
    return err
    }
    post.Comments = append(post.Comments, comment)
    return nil
    })
    Entity୯ҐͷΤϥʔϋϯυϥ༗Γ
    

    View Slide

36. ҆ఆ͍ͯ͠Δ͔
    

    View Slide

37. ར༻࣮੷༗
    • ϝϧΧϦ Ξος (AppEngine)
    • ϝϧΧϦ KYC (Cloud)
    • ٕज़ॻయ (AppEngine)
    • timakin͞Μ (AppEngine)
    ར༻ใࠂ଴ͬͯ·͢❤
    

    View Slide

38. όάݕग़
    1݅ͷΈʂ༏लʂ
    

    View Slide

39. ςετεΠʔτ
    • Goͷαϒςετͷ࢓૊Έ͕࠷ߴ
    • https://blog.golang.org/subtests
    • େྔͷςετΛ༻ҙ͢Δ
    • go.mercari.io/datastore/testsuite
    • AE, Cloud Ͱಉ͡ςετΛͿΜճ͢
    • ↑ޓ׵ੑ
    

    View Slide

40. Ҡߦखॱ
    ཁ͢Δʹطଘͱͷࠩ෼
    

    View Slide

41. جຊతͳํ਑
    • API͸Cloudͱ΄΅ಉ౳
    • ϝιου໊ͱ͔
    • iterator.Done Λ࢖͏ͱ͔
    • datastore.MultiError ͱ͔
    • struct͸interfaceʹஔ͖׵͑
    • AEͱCloudͰ࣮૷͕ҟͳΔͨΊ
    

    View Slide

42. godocݟͯ͘Εʂ
    How to migrate to this library
    Here's an overview of what you need to do to migrate your existing code.
    
    replace *datastore.Key to datastore.Key.
    replace *datastore.Query to datastore.Query.
    replace *datastore.Iterator to datastore.Iterator.
    from AE Datastore
    
    import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both.
    rewrite those using functions of datastore package to FromContext function and Client method calls.
    replace err.(appengine.MultiError) to err.(datastore.MultiError) .
    Stop using appengine.BlobKey and replace with string.
    replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done .
    replace key.IntID() to key.ID() .
    replace key.StringID() to key.Name() .
    When nesting a struct, apply `datastore:", flatten "` to the corresponding field.
    Delete datastore.TransactionOptions, it is not supported.
    If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore .
    from Cloud Datastore
    
    import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore .
    rewrite those using functions of datastore package to FromContext function and Client method calls.
    replace *datastore.Commit to datastore.Commit .
    If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore .
    from goon to boom
    
    replace *goon.Goon to *boom.Boom .
    replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) .
    ଍Γͳ͍ͷݟ͚ͭͨΒIssue΁ʂ
    

    View Slide

43. ҠߦLIVE CODING
    github.com/vvakame/ucon-todo
    

    View Slide

44. Welcome to contribution!
    

    View Slide

45. ͚ͨͯ͢❤
    • ɹɹɹɹɹɹ໨ࢦͤGitHub300←❤
    • GCPUG SlackͰ࣭໰͢Δˡ❤❤
    • Θ͔Βͳ͍͜ͱΛIssueʹ͢Δˡ❤❤❤
    • ࢖ͬͯϒϩάͱ͔ॻ͘ˡ❤❤❤❤
    • Pull RequestΛ͘ΕΔˡ❤❤❤❤❤❤❤
    https://gcpug.jp/join
    

    View Slide

46. View Slide

47. ۙگ
    ๻ͷपΓͰSpanner࢖͏ͱ͜૿͖͑ͯͯͯ΍͹͍
    go.mercari.io/datastore ͷग़൪͕…(´ɾТɾ`)
    ͱ͍͏Θ͚ͰSpannerͷΫΤϦύʔαॻ͖࢝Ί·ͨ͠
    

    View Slide