Go_at_MongoDB.pdf

nycgolang meetup

Transcript

2. MONGODB TOOLS Wisdom Omuya, Software Engineer Making the Move to Golang
3. PORT OF DEPARTURE
4. WHAT IS MONGODB - Document-oriented database - JSON-like data - Open-source
5. WHAT ARE MONGODB TOOLS? Several command line tools: - bsondump - mongofiles - mongooplog - mongodump - mongorestore - mongoimport - mongoexport - mongotop - mongostat
6. MOTIVATION - Looser coupling - Shorter release cycle - Performance improvements
7. FERRYING THE TOOLS
8. CASE STUDY - MONGOIMPORT - Import text data into MongoDB - Accepted formats include JSON, CSV, and TSV
9. MONGOIMPORT ARCHITECTURE
10. MONGOIMPORT PIPELINE 1 chanSize := opts.BulkBufferSize * opts.MaxInsertThreads 2 3 rawChan := make(chan bson.Raw, opts.NumDecoders) 4 docChan := make(chan bson.D, chanSize) 5 6 7 errChan := make(chan error) 8 9 // read and process from the input reader 10 go func() { 11 errChan <- Decode(opts.NumDecoders, rawChan, docChan) 12 }() 13 14 15 // insert documents into the target database 16 go func() { 17 errChan <- Insert(docChan) 18 }() 19 20 return channelQuorumError(errChan, 2)
11. TYING ‘EM UP 1 func channelQuorumError(ch <-chan error, q int) error { 2 3 for i := 0; i < q; i++ { 4 if err := <-ch; err != nil { 5 return err 6 } 7 } 8 9 return nil 10 }
12. TRAVEL VAGARIES - Transportation Mode - Booty Protection - Load Limits
13. TRANSPORTATION MODE 1 import "gopkg.in/mgo/bson" 2 3 type M map[string]interface{} 4 5 bson.M{"a": 1, "b": true} 1 import "gopkg.in/mgo/bson" 2 3 type D []DocElem 4 5 type DocElem struct { 6 Name string 7 Value interface{} 8 } 9 10 bson.D{{"a", 1}, {"b", true}}
14. BATTLE LESSON 1 “MVDS - Minimum Viable Data Structure” go tool pprof mongoimport cpu.prof
15. BOOTY PROTECTION - 1 1 func Decode(numDecoders int, rawChan chan bson.Raw, docChan chan bson.D) error { 2 var decodeWorkers []DecodeWorker 3 wg := &sync.WaitGroup{} 4 for i := 0; i < numDecoders; i++ { 5 decoder := &DecodeWorker{ 6 inChan: rawChan, 7 outChan: docChan, 8 } 9 decodeWorkers = append(decodeWorkers, decoder) 10 wg.Add(1) 11 go func() { 12 defer wg.Done() 13 err := decoder.processDocuments(); 14 if err != nil { 15 return err 16 } 17 }() 18 } 19 wg.Wait()
16. BOOTY PROTECTION - 2 1 var decodeWorkers []*DecodeWorker 2 wg := &sync.WaitGroup{} 3 mt := &sync.Mutex{} 4 decodeTomb := &tomb.Tomb{} 5 for i := 0; i < numDecoders; i++ { 6 decoder := &DecodeWorker{ 7 inChan: rawChan, 8 outChan: docChan, 9 tomb: decodeTomb, 10 } 11 decodeWorkers = append(decodeWorkers, decoder) 13 wg.Add(1) 14 go func() { 15 defer wg.Done() 16 err := decoder.processDocuments() 17 mt.Lock() 18 defer mt.Unlock() 19 if err != nil { 20 retErr = appendErr(retErr, err) 21 decoder.tomb.Kill(err) 22 } 23 }() 24 } 25 wg.Wait()
17. BOOTY PROTECTION - 3 1 var decodeWorkers []*DecodeWorker 2 wg := &sync.WaitGroup{} 3 mt := &sync.Mutex{} 4 decodeTomb := &tomb.Tomb{} 5 for i := 0; i < numDecoders; i++ { 6 decoder := &DecodeWorker{ 7 inChan: rawChan, 8 outChan: docChan, 9 tomb: decodeTomb, 10 } 11 decodeWorkers = append(decodeWorkers, decoder) 12 wg.Add(1) 13 go func(dw DecodeWorker) { 14 defer wg.Done() 15 err := dw.processDocuments() 16 mt.Lock() 17 defer mt.Unlock() 18 if err != nill { 19 retErr = err 20 dw.tomb.Kill(err) 21 } 22 }(*decoder) 23 } 24 wg.Wait() 25 return retErr
18. LESSON 2 “Continuously test your product using a race build” go build -race mongoimport.go
19. LOAD LIMITS 1 import "gopkg.in/mgo/bson" 2 chanSize := opts.BufferSize*opts.NumInserters 3 docChan := make(chan bson.D, chanSize) 1 import "gopkg.in/mgo/bson" 2 3 docChan := make(chan bson.Raw, InsertBufferSize)
20. BATTLE LESSON 3 “Buffered channels are great - provided you don’t run out of memory”
21. SEAFARER’S SURVIVAL KIT - Tomb: https://github.com/go-tomb/tomb/tree/v2 - Sync: http://golang.org/pkg/sync - Pprof: https://blog.golang.org/profiling-go- programs - Race Detector: http://blog.golang.org/race- detector
22. PORT OF ARRIVAL
23. QUESTIONS Try out our tools at: https://github.com/mongodb/mongo- tools/