Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Migrating from BoltDB to BadgerDB

Migrating from BoltDB to BadgerDB

This talk focuses on the technique used to migrate an application from BoltDB to BadgerDB. The technique uses wrapper interfaces and tests to keep the application in a working state during the transition. Finally, we discuss some complications that came up and one nice surprise we weren't expecting.


Marty Schoch

April 17, 2019

More Decks by Marty Schoch

Other Decks in Technology


  1. Migrating from BoltDB to BadgerDB Marty Schoch @mschoch April 2019

  2. Problem Application BoltDB BadgerDB

  3. Problem Application BoltDB

  4. Problem Application • Package Import • Exported Methods ◦ Open()

    • Types Returned ◦ *bolt.DB ◦ *bolt.Bucket ◦ *bolt.Tx BoltDB
  5. Easy Switch? BoltDB BadgerDB • Core K/V is same ◦

    Set/Get/Delete []byte • Tx support • But… • BoltDB has Bucket concept ◦ BadgerDB does not ◦ Application uses this! • Both have Cursors/Iterators ◦ But slight API differences
  6. Wrapper Interface(s) type DBWrapper interface { Batch(fn func(TxWrapper) error) error

    Begin(writable bool) (TxWrapper, error) Close() error GoString() string Info() InfoWrapper IsReadOnly() bool Path() string Stats() Stats String() string Sync() error Update(fn func(TxWrapper) error) error View(fn func(TxWrapper) error) error } wrapper.go
  7. Repeat for Other Interfaces • Bucket • Tx • Cursor

    • Stats • BucketStats
  8. Implement Wrapper for BoltDB type boltWrapper struct { db *bolt.DB

    } func openBolt(path string, mode os.FileMode) (DBWrapper, error) { db, err := bolt.Open(path, mode, …) … return &boltWrapper{db: db} } func (b *boltWrapper) Close() error { return b.db.Close() } wrapper_bolt.go
  9. Write Test func TestWrapper(t *testing.T) { // open bolt wrapper

    db, err := openBolt(...) // test behavior } wrapper_test.go • Be thorough, test as much API surface area as possible • Assert the return types and state changes
  10. Update the Application Change calls to Open Change any references

    to *bolt.DB to DBWrapper, repeat for all bolt types - db, err := bolt.Open(path, 0600, &bolt.Options{Timeout: 30*time.Second}) + db, err := openBolt(path, 30*time.Second) -func doSomething(db *bolt.Bucket) error { +func doSomething(db BucketWrapper) error {
  11. Application Works • No changes to business logic • Still

    using BoltDB • Dependency on BoltDB confined to single file, wrapper_bolt.go
  12. Implement Wrapper for BadgerDB type badgerWrapper struct { db *badger.DB

    } func openBadger(path string) (DBWrapper, error) { db, err := badger.Open(...) … return &badgerWrapper{db: db} } func (b *badgerWrapper) Close() error { return b.db.Close() } wrapper_badger.go
  13. More Complex Logic in Badger Wrapper func (b *badgerBucketWrapper) Get(key

    []byte) []byte { fullKey := b.applyPrefix(key) return b.db.Get(fullKey) }
  14. Refactor Test for Both func TestWrapper(t *testing.T) { // open

    bolt wrapper db, err := openBolt(...) // test behavior } func testWrapper(db DbWrapper, t *testing.T) { // same logic } func TestBoltWrapper(t *testing.T) { db, err := openBolt(...) if err != nil { t.Fatal(err) } testWrapper(db, t) } func TestBadgerWrapper(t *testing.T) { db, err := openBadger(...) if err != nil { t.Fatal(err) } testWrapper(db, t) }
  15. $ go test -v -run=Wrapper === RUN TestBoltWrapper --- PASS:

    TestBoltWrapper (0.04s) === RUN TestBadgerWrapper --- PASS: TestBadgerWrapper (0.12s) PASS ok github.com/mschoch/refactor0.183s Passing tests, Identical behavior
  16. Update the Application (again) - db, err := openBolt(path, 30*time.Second)

    + db, err := openBadger(path, 30*time.Second)
  17. Complications... • Comparisons against sentinel errors • BoltDB Cursors are

    bi-directional, BadgerDB Iterators are not • Cursor Limitations (beyond API) • Different Transaction Model
  18. Abstracting Sentinel Errors err = db.CreateBucket(“marty”) if err != nil

    { // } err = db.CreateBucket(“marty”) if err != bolt.ErrBucketExists { // } Most places are already OK: But, sentinel errors have to converted to wrapper version:
  19. Bi-directional Cursors/Iterators BoltDB Cursor BadgerDB Iterator

  20. Wait!?! Does our application even use this? Most cursors go

    forward, so review places that go backwards: type CursorWrapper interface { First() (key []byte, value []byte) Next() (key []byte, value []byte) //Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }
  21. Change the Wrapper Interface type CursorWrapper interface { First() (key

    []byte, value []byte) Next() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error } type ReverseCursorWrapper interface { Last() (key []byte, value []byte) Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }
  22. API Isn’t Everything type BucketWrapper interface { ... Cursor() CursorWrapper

    ReverseCursor() ReverseCursorWrapper ... }
  23. Solution Application BoltDB BadgerDB

  24. Sur s !

  25. Bi-directional Conversion Tool for Free func convert(dbIn, dbOut DBWrapper) error

    { dbIn.View(func (txIn TxWrapper) error { dbOut.Update(func (txOut TxWrapper) error { // iterate everything, read in, write out }) }) }
  26. Inspiration https://www.dotconferences.com/2016/10/katrina-owen-the-scandalous-story-of-the-dreadful-code-written-by-the-best-of-us no-op, no-op, no-op, interface maneuver

  27. Eight Simple Steps Wrapper Impl Test Run Impl Split Test

  28. Thanks Marty Schoch @mschoch marty.schoch@gmail.com References: BadgerDB - https://github.com/dgraph-io/badger BoltDB

    - https://github.com/etcd-io/bbolt