Migrating from BoltDB to BadgerDB

This talk focuses on the technique used to migrate an application from BoltDB to BadgerDB. The technique uses wrapper interfaces and tests to keep the application in a working state during the transition. Finally, we discuss some complications that came up and one nice surprise we weren't expecting.

Transcript

1. Migrating from BoltDB to BadgerDB Marty Schoch @mschoch April 2019
2. Problem Application BoltDB BadgerDB
3. Problem Application BoltDB
4. Problem Application • Package Import • Exported Methods ◦ Open() • Types Returned ◦ *bolt.DB ◦ *bolt.Bucket ◦ *bolt.Tx BoltDB
5. Easy Switch? BoltDB BadgerDB • Core K/V is same ◦ Set/Get/Delete []byte • Tx support • But… • BoltDB has Bucket concept ◦ BadgerDB does not ◦ Application uses this! • Both have Cursors/Iterators ◦ But slight API differences
6. Wrapper Interface(s) type DBWrapper interface { Batch(fn func(TxWrapper) error) error Begin(writable bool) (TxWrapper, error) Close() error GoString() string Info() InfoWrapper IsReadOnly() bool Path() string Stats() Stats String() string Sync() error Update(fn func(TxWrapper) error) error View(fn func(TxWrapper) error) error } wrapper.go
7. Repeat for Other Interfaces • Bucket • Tx • Cursor • Stats • BucketStats
8. Implement Wrapper for BoltDB type boltWrapper struct { db *bolt.DB } func openBolt(path string, mode os.FileMode) (DBWrapper, error) { db, err := bolt.Open(path, mode, …) … return &boltWrapper{db: db} } func (b *boltWrapper) Close() error { return b.db.Close() } wrapper_bolt.go
9. Write Test func TestWrapper(t *testing.T) { // open bolt wrapper db, err := openBolt(...) // test behavior } wrapper_test.go • Be thorough, test as much API surface area as possible • Assert the return types and state changes
10. Update the Application Change calls to Open Change any references to *bolt.DB to DBWrapper, repeat for all bolt types - db, err := bolt.Open(path, 0600, &bolt.Options{Timeout: 30*time.Second}) + db, err := openBolt(path, 30*time.Second) -func doSomething(db *bolt.Bucket) error { +func doSomething(db BucketWrapper) error {
11. Application Works • No changes to business logic • Still using BoltDB • Dependency on BoltDB confined to single file, wrapper_bolt.go
12. Implement Wrapper for BadgerDB type badgerWrapper struct { db *badger.DB } func openBadger(path string) (DBWrapper, error) { db, err := badger.Open(...) … return &badgerWrapper{db: db} } func (b *badgerWrapper) Close() error { return b.db.Close() } wrapper_badger.go
13. More Complex Logic in Badger Wrapper func (b *badgerBucketWrapper) Get(key []byte) []byte { fullKey := b.applyPrefix(key) return b.db.Get(fullKey) }
14. Refactor Test for Both func TestWrapper(t *testing.T) { // open bolt wrapper db, err := openBolt(...) // test behavior } func testWrapper(db DbWrapper, t *testing.T) { // same logic } func TestBoltWrapper(t *testing.T) { db, err := openBolt(...) if err != nil { t.Fatal(err) } testWrapper(db, t) } func TestBadgerWrapper(t *testing.T) { db, err := openBadger(...) if err != nil { t.Fatal(err) } testWrapper(db, t) }
15. $ go test -v -run=Wrapper === RUN TestBoltWrapper --- PASS: TestBoltWrapper (0.04s) === RUN TestBadgerWrapper --- PASS: TestBadgerWrapper (0.12s) PASS ok github.com/mschoch/refactor0.183s Passing tests, Identical behavior
16. Update the Application (again) - db, err := openBolt(path, 30*time.Second) + db, err := openBadger(path, 30*time.Second)
17. Complications... • Comparisons against sentinel errors • BoltDB Cursors are bi-directional, BadgerDB Iterators are not • Cursor Limitations (beyond API) • Different Transaction Model
18. Abstracting Sentinel Errors err = db.CreateBucket(“marty”) if err != nil { // } err = db.CreateBucket(“marty”) if err != bolt.ErrBucketExists { // } Most places are already OK: But, sentinel errors have to converted to wrapper version:
19. Bi-directional Cursors/Iterators BoltDB Cursor BadgerDB Iterator
20. Wait!?! Does our application even use this? Most cursors go forward, so review places that go backwards: type CursorWrapper interface { First() (key []byte, value []byte) Next() (key []byte, value []byte) //Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }
21. Change the Wrapper Interface type CursorWrapper interface { First() (key []byte, value []byte) Next() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error } type ReverseCursorWrapper interface { Last() (key []byte, value []byte) Prev() (key []byte, value []byte) Seek(seek []byte) (key []byte, value []byte) Close() error }
22. API Isn’t Everything type BucketWrapper interface { ... Cursor() CursorWrapper ReverseCursor() ReverseCursorWrapper ... }
23. Solution Application BoltDB BadgerDB
24. Sur s !
25. Bi-directional Conversion Tool for Free func convert(dbIn, dbOut DBWrapper) error { dbIn.View(func (txIn TxWrapper) error { dbOut.Update(func (txOut TxWrapper) error { // iterate everything, read in, write out }) }) }
26. Inspiration https://www.dotconferences.com/2016/10/katrina-owen-the-scandalous-story-of-the-dreadful-code-written-by-the-best-of-us no-op, no-op, no-op, interface maneuver
27. Eight Simple Steps Wrapper Impl Test Run Impl Split Test Run
28. Thanks Marty Schoch @mschoch marty.schoch@gmail.com References: BadgerDB - https://github.com/dgraph-io/badger BoltDB - https://github.com/etcd-io/bbolt