Migrating from BoltDB to BadgerDB Marty Schoch @mschoch April 2019

Problem Application BoltDB BadgerDB

Problem Application BoltDB

Problem Application ● Package Import ● Exported Methods ○ Open() ● Types Returned ○ *bolt.DB ○ *bolt.Bucket ○ *bolt.Tx BoltDB

Easy Switch? BoltDB BadgerDB ● Core K/V is same ○ Set/Get/Delete []byte ● Tx support ● But… ● BoltDB has Bucket concept ○ BadgerDB does not ○ Application uses this! ● Both have Cursors/Iterators ○ But slight API differences

Wrapper Interface(s) type DBWrapper interface { Batch(fn func(TxWrapper) error) error Begin(writable bool) (TxWrapper, error) Close() error GoString() string Info() InfoWrapper IsReadOnly() bool Path() string Stats() Stats String() string Sync() error Update(fn func(TxWrapper) error) error View(fn func(TxWrapper) error) error } wrapper.go

Repeat for Other Interfaces ● Bucket ● Tx ● Cursor ● Stats ● BucketStats

Implement Wrapper for BoltDB type boltWrapper struct { db *bolt.DB } func openBolt(path string, mode os.FileMode) (DBWrapper, error) { db, err := bolt.Open(path, mode, …) … return &boltWrapper{db: db} } func (b *boltWrapper) Close() error { return b.db.Close() } wrapper_bolt.go

Write Test func TestWrapper(t *testing.T) { // open bolt wrapper db, err := openBolt(...) // test behavior } wrapper_test.go ● Be thorough, test as much API surface area as possible ● Assert the return types and state changes

Update the Application Change calls to Open Change any references to *bolt.DB to DBWrapper, repeat for all bolt types - db, err := bolt.Open(path, 0600, &bolt.Options{Timeout: 30*time.Second}) + db, err := openBolt(path, 30*time.Second) -func doSomething(db *bolt.Bucket) error { +func doSomething(db BucketWrapper) error {

Application Works ● No changes to business logic ● Still using BoltDB ● Dependency on BoltDB confined to single file, wrapper_bolt.go

Implement Wrapper for BadgerDB type badgerWrapper struct { db *badger.DB } func openBadger(path string) (DBWrapper, error) { db, err := badger.Open(...) … return &badgerWrapper{db: db} } func (b *badgerWrapper) Close() error { return b.db.Close() } wrapper_badger.go

More Complex Logic in Badger Wrapper func (b *badgerBucketWrapper) Get(key []byte) []byte { fullKey := b.applyPrefix(key) return b.db.Get(fullKey) }

Refactor Test for Both func TestWrapper(t *testing.T) { // open bolt wrapper db, err := openBolt(...) // test behavior } func testWrapper(db DbWrapper, t *testing.T) { // same logic } func TestBoltWrapper(t *testing.T) { db, err := openBolt(...) if err != nil { t.Fatal(err) } testWrapper(db, t) } func TestBadgerWrapper(t *testing.T) { db, err := openBadger(...) if err != nil { t.Fatal(err) } testWrapper(db, t) }

$ go test -v -run=Wrapper === RUN TestBoltWrapper --- PASS: TestBoltWrapper (0.04s) === RUN TestBadgerWrapper --- PASS: TestBadgerWrapper (0.12s) PASS ok github.com/mschoch/refactor0.183s Passing tests, Identical behavior

Update the Application (again) - db, err := openBolt(path, 30*time.Second) + db, err := openBadger(path, 30*time.Second)

Complications... ● Comparisons against sentinel errors ● BoltDB Cursors are bi-directional, BadgerDB Iterators are not ● Cursor Limitations (beyond API) ● Different Transaction Model

Abstracting Sentinel Errors err = db.CreateBucket(“marty”) if err != nil { // } err = db.CreateBucket(“marty”) if err != bolt.ErrBucketExists { // } Most places are already OK: But, sentinel errors have to converted to wrapper version:

Bi-directional Cursors/Iterators BoltDB Cursor BadgerDB Iterator

Wait!?! Does our application even use this? Most cursors go forward, so review places that go backwards: type CursorWrapper interface { First() (key []byte, value []byte) Next() (key []byte, value []byte) //Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }

Change the Wrapper Interface type CursorWrapper interface { First() (key []byte, value []byte) Next() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error } type ReverseCursorWrapper interface { Last() (key []byte, value []byte) Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }

API Isn’t Everything type BucketWrapper interface { ... Cursor() CursorWrapper ReverseCursor() ReverseCursorWrapper ... }

Solution Application BoltDB BadgerDB

Sur s !

Bi-directional Conversion Tool for Free func convert(dbIn, dbOut DBWrapper) error { dbIn.View(func (txIn TxWrapper) error { dbOut.Update(func (txOut TxWrapper) error { // iterate everything, read in, write out }) }) }

Inspiration https://www.dotconferences.com/2016/10/katrina-owen-the-scandalous-story-of-the-dreadful-code-written-by-the-best-of-us no-op, no-op, no-op, interface maneuver

Eight Simple Steps Wrapper Impl Test Run Impl Split Test Run

Thanks Marty Schoch @mschoch marty.schoch@gmail.com References: BadgerDB - https://github.com/dgraph-io/badger BoltDB - https://github.com/etcd-io/bbolt