Upgrade to Pro — share decks privately, control downloads, hide ads and more …

go.mercari.io/datastore はいいぞ! / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore はいいぞ! / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore の紹介です。
https://gaeja.connpass.com/event/90133/

Masahiro Wakame

June 14, 2018
Tweet

More Decks by Masahiro Wakame

Other Decks in Technology

Transcript

  1. go.mercari.io/datastore ͸͍͍ͧʂ
    ajn #3
    Θ͔Ί ·͞ͻΖ

    View Slide

  2. Θ͔Ί ·͞ͻΖ @v vakame
    TypeScript
    Masahiro Wakame
    ٕज़ॻయ
    appengine/go
    photo from golang.org/doc/gopher/

    View Slide

  3. GAE/SE Node.js β
    Ͱ΋ࠓ೔ͷ࿩͸GAE/GoͰ͢

    View Slide

  4. ͸͡Ίʹ
    • ׂΓࠐΈ࣭໰׻ܴʂ
    • ฉ͖͍ͨ͜ͱΛฉ͍ͯ΄͍͠
    • ࡞ऀͳͷͰશମ૾Λطʹ஌͍ͬͯΔ
    • Կ͕Θ͔ΒΜ͔Θ͔ΒΜͷͩ…
    ࢲ͕࡞ऀͰ͢

    View Slide

  5. go.mercari.io/datastore
    ࠓ೔ͷ࿩୊Ͱ͢

    View Slide

  6. ઌ೔v1.0.0ϦϦʔε
    ҆ఆͯ͠Δͥʂ

    View Slide

  7. ͦΕԿʁ

    View Slide

  8. ͦΕԿʁ
    • (AppEngine|Cloud) Datastore Wrapper
    • In Go! Seamless!
    • Middleware
    • Retry, Caching, Logging… etc
    • Batch operation
    • Single-Op Batch-Op aggregation

    View Slide

  9. ઃܭࢥ૝
    • Cloud DatastoreͷAPIʹͳΔ΂͘دͤΔ
    • ༨ܭͳ͜ͱ͸ͳΔ΂͘͠ͳ͍
    • ErrFieldMismatchແࢹͷσϑΥϧτԽ
    • RunInTransactionͷࣗಈRetryͷ࡟আ
    • PropertyLoadServerʹctx͋͛ͨ
    • Batchૢ࡞͸଍ͨ͠

    View Slide

  10. (AppEngine|Cloud) Datastore

    View Slide

  11. ͦ΋ͦ΋
    Datastoreͬͯ2छྨ͋Δɻ
    ͦͷ͜ͱΛ஌͍ͬͯ·͔͢ʁ

    View Slide

  12. Datastore Service
    AppEngine RPC Cloud RPC
    google.golang.org/
    appengine/datastore
    cloud.google.com/
    go/datastore
    GAE Instance Any Server
    RPC
    Endpoint
    Package
    Runtime
    Env

    View Slide

  13. Datastore Storage
    AppEngine RPC Cloud RPC
    google.golang.org/
    appengine/datastore
    cloud.google.com/
    go/datastore
    GAE Instance Any Server
    RPC
    Endpoint
    Package
    Runtime
    Env
    ଘࡏ͢Δػೳʹ
    ݁ߏ͕ࠩ͋Δ

    چ

    View Slide

  14. ͡Ό͋৽࢖͑͹Α͘Ͷʁ
    • GAE/SE Go͔Βͷར༻ʹ੍໿͋Γ
    • Socket API͕ඞཁ… ΊΜͲ͍…
    • Cloudͷํɺͨ·ʹτϥϒͬͯΔ
    • GCPUG Slack #g-datastore_ja
    ࢖͍෼͚͕·ͩ·ͩඞཁͩʂ

    View Slide

  15. ޓ׵ੑ…
    • (AppEngine|Cloud) Datastore͸…
    • ཪͷ࣮૷͸ಉ͡
    • ஌ݟ΍ઃܭ͸9ׂҎ্࢖͍ճͤΔ
    • ͨͩ͠ɺAPI͕ҟͳΔʂ
    • ͳΜͳΒpackage΋׬શʹผ

    View Slide

  16. ΑΖ͍͠ʂ
    ͳΒ͹Wrapperͩʂ

    View Slide

  17. Wrapperͷಛ௃
    • Cloud Datastore(৽)ͷAPI͕ϕʔε
    • ࣮૷ͷ(AppEngine|Cloud)ࠩସՄ
    • goonϢʔβͷͨΊʹboom΋͋Γ·͢

    View Slide

  18. package༡ཡ
    • godoc root
    • godoc aedatastore
    • godoc clouddatastore

    View Slide

  19. ͋ΓಘΔະདྷ
    • AppEngine→GKE΁ͷࡌͤସ͑
    • ཧ࿦্ΠέΔؾ͕͢Δ…
    • ࣄྫ͸ࣾ಺Ͱ΋ແ͍໛༷
    • gVisorϕʔεͷGAE/SE૿͑ͯΔͬΆ͍
    • GAE/Go΋কདྷϫϯνϟϯඍཻࢠ…ʁ

    View Slide

  20. ҠߦLIVE CODING
    • ΛɺޙͰ΍Δ͕࣌ؒ͋Δͱ͍͍ͳ͊

    View Slide

  21. • datastore:”,flatten” λάͷଘࡏ
    • Cloud DatastoreͷAPIʹ͋ΘͤͯΔ
    • ίʔυͷॻ͖׵͚͑ͩͰ͸μϝ
    • structͷλά෇͚Λݟ௚͢ʂ
    ςετ͸͔ͬ͠Γʂ
    ಛʹ஫ҙ͢΂͖ࠩ෼

    View Slide

  22. flatten?
    • datastore.Entity ʹ͍ͭͯ
    • AE → Entity͸Entity࣋ͯͳ͍
    • Cloud → Entity͸Entity࣋ͯΔ
    • struct͸Entityʹม׵͞ΕΔ
    • nest ͨ͠struct͸EntityͷೖΕࢠʹͳΔ

    View Slide

  23. flatten?
    type AEEntity struct {
    "Slice.A" []string
    "Slice.B" []string
    }
    ͜ͷίʔυྫ͸
    ΠϝʔδͰ͢
    type Inner struct {
    A string
    B string
    }
    type Data struct {
    Slice []Inner
    }
    type CloudEntity struct {
    Slice []struct{
    A string
    B string
    }
    }

    View Slide

  24. flatten?
    type Inner struct {
    A string
    B string
    }
    type Data struct {
    Slice []Inner
    }
    type Data struct {
    Slice []Inner ↩
    `datastore:",flatten"`
    }
    flattenͰAEͱಉ͡ʹ

    View Slide

  25. ☠ ATTENTION ☠
    • Cloud DatastoreͰNestͨ͠Entity࡞Δͱ
    AppEngine Datastore͔ΒಡΊͳ͍
    • ͸ͣ(ࢼͯ͠ͳ͍)
    • ؾܰͳؾ࣋ͪͰAppEngine͔ΒGetͯ͠
    CloudʹPut͢Δͱࢮ͵Մೳੑ͕͋Δ
    ςετ͸͔ͬ͠ΓͶʂ

    View Slide

  26. • KeyLoader etc ͷCloudଆಠ઎ػೳ։์
    • UnitTestͰCloud Datastore Emulator
    • GoLandͰσόοΨ࢖͍΍͍͢
    • Local͔Βσʔλ౤ೖ͕༰қ
    ͔͠͠΍ͬͨ͜ͱ͸ͳ͍ʂ
    ศརͳ఺

    View Slide

  27. Middleware

    View Slide

  28. ͳͥཉ͍͔͠
    • Middlewareͷ༻్
    • RPCෆௐ࣌ͷϦτϥΠ
    • MemcacheͳͲͷར༻
    • ϩάग़ྗ
    • େྔEntity Put࣌ͷRPC਺෼ׂ

    View Slide

  29. ΞϓϦͷ࣮૷ʹ
    ूதͯ͐͠ʂ

    View Slide

  30. Middlewareʹ
    ΍ΒͤΑ͏ʂ

    View Slide

  31. ࠓ͋ΔMiddleware
    • AE Memcache
    • In-Memory Cache
    • Automatic RPC Retry
    • Redis(Memorystore) Cache
    • Logger, Chaos RPC, Fishbone …
    • ↑ओʹςετ΍ݕূ༻
    ຊ൪ར༻࣮੷ͷน

    View Slide

  32. Batch Operation

    View Slide

  33. ͜͏͍͏ίʔυ͕ॻ͚Δ
    type Comment struct {
    Message string
    }
    type Post struct {
    Content string
    CommentIDs []int64 `json:"-"`
    Comments []*Comment `datastore:"-"`
    }
    // Entityͷ४උ(ׂѪ)
    ………
    // start fetching...
    posts := make([]*Post, 0)
    _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts)
    if err != nil {
    panic(err)
    }
    // Let's batch get!
    bt := client.Batch()
    for _, post := range posts {
    comments := make([]*Comment, 0)
    for _, id := range post.CommentIDs {
    comment := &Comment{}
    bt.Get(client.IDKey("Comment", id, nil), comment, nil)
    comments = append(comments, comment)
    }
    post.Comments = comments
    }
    err = bt.Exec(ctx)
    if err != nil {
    panic(err)
    }

    View Slide

  34. ΋͠΋Batch͕ͳ͔ͬͨΒ
    // start fetching...
    posts := make([]*Post, 0)
    _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts)
    if err != nil {
    panic(err)
    }
    // Let's batch get!
    var commentKeys []datastore.Key
    for _, post := range posts {
    for _, commentID := range post.CommentIDs {
    commentKeys = append(commentKeys, client.IDKey("Comment", commentID, nil))
    }
    }
    comments := make([]*Comment, len(commentKeys))
    err = client.GetMulti(ctx, commentKeys, comments)
    if err != nil {
    panic(err)
    }
    for _, post := range posts {
    for _, commentID := range post.CommentIDs {
    for idx, comment := range comments {
    key := commentKeys[idx]
    if commentID == key.ID() {
    post.Comments = append(post.Comments, comment)
    break
    }
    }
    }
    }
    ErrNoSuchEntity͕
    ࠞͬͨ͟Β͞ΒʹͭΒ͍…ʂ

    View Slide

  35. ErrNoSuchEntityରԠ
    bt.Get(key, comment, func(err error) error {
    if err == datastore.ErrNoSuchEntity {
    // ignore ErrNoSuchEntity
    return nil
    } else if err != nil {
    return err
    }
    post.Comments = append(post.Comments, comment)
    return nil
    })
    Entity୯ҐͷΤϥʔϋϯυϥ༗Γ

    View Slide

  36. ҆ఆ͍ͯ͠Δ͔

    View Slide

  37. ར༻࣮੷༗
    • ϝϧΧϦ Ξος (AppEngine)
    • ϝϧΧϦ KYC (Cloud)
    • ٕज़ॻయ (AppEngine)
    • timakin͞Μ (AppEngine)
    ར༻ใࠂ଴ͬͯ·͢❤

    View Slide

  38. όάݕग़
    1݅ͷΈʂ༏लʂ

    View Slide

  39. ςετεΠʔτ
    • Goͷαϒςετͷ࢓૊Έ͕࠷ߴ
    • https://blog.golang.org/subtests
    • େྔͷςετΛ༻ҙ͢Δ
    • go.mercari.io/datastore/testsuite
    • AE, Cloud Ͱಉ͡ςετΛͿΜճ͢
    • ↑ޓ׵ੑ

    View Slide

  40. Ҡߦखॱ
    ཁ͢Δʹطଘͱͷࠩ෼

    View Slide

  41. جຊతͳํ਑
    • API͸Cloudͱ΄΅ಉ౳
    • ϝιου໊ͱ͔
    • iterator.Done Λ࢖͏ͱ͔
    • datastore.MultiError ͱ͔
    • struct͸interfaceʹஔ͖׵͑
    • AEͱCloudͰ࣮૷͕ҟͳΔͨΊ

    View Slide

  42. godocݟͯ͘Εʂ
    How to migrate to this library
    Here's an overview of what you need to do to migrate your existing code.

    replace *datastore.Key to datastore.Key.
    replace *datastore.Query to datastore.Query.
    replace *datastore.Iterator to datastore.Iterator.
    from AE Datastore

    import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both.
    rewrite those using functions of datastore package to FromContext function and Client method calls.
    replace err.(appengine.MultiError) to err.(datastore.MultiError) .
    Stop using appengine.BlobKey and replace with string.
    replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done .
    replace key.IntID() to key.ID() .
    replace key.StringID() to key.Name() .
    When nesting a struct, apply `datastore:", flatten "` to the corresponding field.
    Delete datastore.TransactionOptions, it is not supported.
    If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore .
    from Cloud Datastore

    import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore .
    rewrite those using functions of datastore package to FromContext function and Client method calls.
    replace *datastore.Commit to datastore.Commit .
    If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore .
    from goon to boom

    replace *goon.Goon to *boom.Boom .
    replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) .
    ଍Γͳ͍ͷݟ͚ͭͨΒIssue΁ʂ

    View Slide

  43. ҠߦLIVE CODING
    github.com/vvakame/ucon-todo

    View Slide

  44. Welcome to contribution!

    View Slide

  45. ͚ͨͯ͢❤
    • ɹɹɹɹɹɹ໨ࢦͤGitHub300←❤
    • GCPUG SlackͰ࣭໰͢Δˡ❤❤
    • Θ͔Βͳ͍͜ͱΛIssueʹ͢Δˡ❤❤❤
    • ࢖ͬͯϒϩάͱ͔ॻ͘ˡ❤❤❤❤
    • Pull RequestΛ͘ΕΔˡ❤❤❤❤❤❤❤
    https://gcpug.jp/join

    View Slide

  46. View Slide

  47. ۙگ
    ๻ͷपΓͰSpanner࢖͏ͱ͜૿͖͑ͯͯͯ΍͹͍
    go.mercari.io/datastore ͷग़൪͕…(´ɾТɾ`)
    ͱ͍͏Θ͚ͰSpannerͷΫΤϦύʔαॻ͖࢝Ί·ͨ͠

    View Slide