Go MeetUp

Transcript

1. Reducing computation time by months by rewriting Bash scripts in Go Paris, France, MAR 24 2019 Pedro J. Ortiz Inria / Sorbonne Université Your handle
2. Reducing computation time by months by copy-pasting OpenSource Go code from the internet. But honestly it should be called
3. The Bash Script 01. The Bash script Classifier "tmp/${FILENAME}.txt" > "tmp/${FILENAME}.lid" paste "tmp/${FILENAME}.lid" "tmp/${FILENAME}.txt" | \ awk '($2 > 0.8 || ($1=="__label__hr" && $2 > 0.4)) && length() > 100 {lang = substr($1, 10); $1=""; $2=""; print $0 >> "shard/"lang".txt"}' Here is the offending piece of code I have 20TB of plain text that I want to classify by language
4. I have 56000 files, and each files takes roughly 2.5 minutes to process = 97.2 days to classify my whole dataset. Little Benchmark
5. Parallelize = Use Go 02. The Go solution go func() { // HL defer close(paths) // HL errc <- filepath.Walk(root, func(path string, info os.FileInfo, err error) error { // HL if err != nil { return err } if !info.Mode().IsRegular() { return nil } select { case paths <- path: // HL case <-done: // HL return errors.New("walk canceled") } return nil }) }() https://blog.golang.org/pipelines
6. Parallelize = Use Go 02. The Go solution var wg sync.WaitGroup maxGoroutines := 10 guard := make(chan struct{}, maxGoroutines) for path := range paths { wg.Add(1) go func(path string) { guard <- struct{}{} clasifyLines(path, files) // HLc <-guard wg.Done() }(path) } https://blog.golang.org/pipelines
7. Exec package 02. The Go solution cmd := exec.Command("classifier", "arg", "arg", "-") stdin, err := cmd.StdinPipe() cmd.Stdout = tags go func() { defer stdin.Close() for record, err := readRecord(bufin); err == nil; record, err = readRecord(bufin) { buf := bytes.NewBuffer(record.body) for line, err := buf.ReadString('\n'); err == nil; line, err = buf.ReadString('\n') { if utf8.RuneCountInString(line) > 100 && utf8.Valid([]byte(line)) { io.WriteString(stdin, line) cleanWriter.WriteString(line) } } } }() https://golang.org/pkg/os/exec/
8. After all of this I was at an estimated of 77.8 days to classify my whole dataset. Little Benchmark
9. The Bash Script Again 02. The Go solution Classifier "tmp/${FILENAME}.txt" > "tmp/${FILENAME}.lid" paste "tmp/${FILENAME}.lid" "tmp/${FILENAME}.txt" | \ awk '($2 > 0.8 || ($1=="__label__hr" && $2 > 0.4)) && length() > 100 {lang = substr($1, 10); $1=""; $2=""; print $0 >> "shard/"lang".txt"}' The offending piece of code Opening a file just to append one line?
10. Write in Parallel 02. The Go solution type langFile struct { mux sync.Mutex Writer *io.Writer } func (lf *langFile) WriteString(s string) (n int, err error) { lf.mux.Lock() defer lf.mux.Unlock() return lf.Writer.WriteString(s) }
11. Leave the files open! 02. The Go solution m := make(map[string]*langFile) for _, lang := range langs { aux, err := os.OpenFile(lang, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) defer aux.Close() if err != nil { log.Fatal(err) } m[scanner.Text()] = &langFile{Writer: bufio.NewWriter(aux)} }
12. After the additional changes I was at an estimated of 44.3 days to classify my whole dataset. Little Benchmark
13. I have 192 GB of RAM, let's use it! 02. The Go solution bufin := bufio.NewReader(in) cleanWriter := bufio.NewWriter(clean) cleanWriter.Flush() https://golang.org/pkg/bufio/
14. After sprinkling buffers everywhere I was at an estimated of 4.18 days to classify my whole dataset. And 40GB of memory consumption. Little Benchmark
15. Why not Python? But ...
16. Because Go is faster… (And I like Go better) Little Benchmark
17. Thanks