Processando análise genética em background com Go

Desde o meio do ano passado temos migrado partes de um sistema de análise genética com Go para uma versão auto contida. Um dos desafios dessa migração é como controlar os exames que devemos processar: se o processamento falhou, porque falhou, como reprocessar, retry automático, etc.

Para alguém como eu que vem do mundo Ruby, Sidekiq é quase uma unanimidade nesse quesito, mas o que tem aparecido de bom no mundo Go nos últimos tempos?

Vale a pena usar bibliotecas como GoWorker e Faktory ou devemos ir para um caminho mais híbrido com Amazon SQS, RabbitMQ, Apache Kafka e afins?

Nessa talk vamos compartilhar nossas experiências em processar mensalmente milhares de exames genéticos em Go.

Transcript

1. Processando análise genética em background com Go Marco Singer Trilha Go - TDC 2018
2. Um pouco de contexto
7. Monolítico > Microserviços > Binário
8. ~ vsa vsa stands for *Very Simple Annotator*. This package annotates variants in VCF format Usage: vsa [command] Available Commands: dx Annotate and classify variants given a VCF file help Help about any command server Run vsa server at port 9090 train Prepare features for training or standardization Flags: -h, --help help for vsa Use "vsa [command] --help" for more information about a command.
9. Simplificar a forma de processamento dos exames
10. Premissas do VSA-Worker • Receber os exames que precisamos processar • Delegar a execução para o nosso binário • Notificar em caso de erro no processamento
11. Tecnologias • RabbitMQ • Apache Kafka • Amazon SQS • Sidekiq
12. Tecnologias • RabbitMQ (exemplo) • Apache Kafka • Amazon SQS • Sidekiq
13. Tecnologias • RabbitMQ (exemplo) • Apache Kafka (canhão) • Amazon SQS • Sidekiq
14. Tecnologias • RabbitMQ (exemplo) • Apache Kafka (canhão) • Amazon SQS (não funciona offline) • Sidekiq
15. Tecnologias • RabbitMQ (exemplo) • Apache Kafka (canhão) • Amazon SQS (não funciona offline) • Sidekiq (mais uma linguagem na stack)
16. Vamos no simples
17. Faktory https://github.com/contribsys/faktory • Criador do Sidekiq • Feito em Go • Clientes em Go e Ruby "oficiais" • Usa RocksDB para persistência • Vem com um dashboard built-in • Retry automático • Pode agendar execução de um job
18. Server
19. Rodando Faktory docker pull contribsys/faktory docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory
20. Rodando Faktory docker pull contribsys/faktory docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory comunicação cliente / servidor
21. Rodando Faktory docker pull contribsys/faktory docker run --rm -it -p 7419:7419 -p 7420:7420 contribsys/faktory porta da interface web
22. http://localhost:7420
23. Criando um job
24. Payload Estrutura de um job { "jid": "123861239abnadsa", "jobtype": "SomeName", "args": [1, 2, "hello"] }
25. Payload Estrutura de um job { "jid": "123861239abnadsa", "jobtype": "SomeName", "args": [1, 2, "hello"] } ID único do job
26. Payload Estrutura de um job { "jid": "123861239abnadsa", "jobtype": "SomeName", "args": [1, 2, "hello"] } nome da função
27. Payload Estrutura de um job { "jid": "123861239abnadsa", "jobtype": "SomeName", "args": [1, 2, "hello"] } argumentos da função
28. Executando VSA através do Faktory
29. ~ vsa dx --help Annotate and classify variants given a VCF file. Usage: vsa dx [flags] Flags: --assembly string Supported assemblies: GRCh37 or GRCh38. --vcf string VCF file address.
30. Criando um job go get -u github.com/contribsys/faktory/client import ( faktory "github.com/contribsys/faktory/client" ) func send(){ client, err := faktory.Open() if err != nil { // tratamento de erro } defer client.Close() job := faktory.NewJob("dx", "GRCh37", "/path/to/file") if err := client.Push(job); err != nil { // tratamento de erro } }
31. faktory.NewJob("dx", "GRCh37", "/path/to/file")
32. faktory.NewJob("dx", "GRCh37", "/path/to/file") payload args payload jobType
33. ~ vsa dx --help Annotate and classify variants given a VCF file. Usage: vsa dx [flags] Flags: --assembly string Supported assemblies: GRCh37 or GRCh38. --vcf string VCF file address.
34. client.Push(job)
35. http://localhost:7420
36. http://localhost:7420
37. Processando um job
38. go get -u github.com/contribsys/faktory_worker_go
39. package worker import ( worker "github.com/contribsys/faktory_worker_go" )
40. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() }
41. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() svc.Register("dx", dxWorker) }
42. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() svc.Register("dx", dxWorker) } payload jobType
43. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() svc.Register("dx", dxWorker) } ?
44. // github.com/contribsys/faktory_worker_go func (mgr *Manager) Register(name string, fn Perform) { } type Perform func(ctx Context, args ...interface{}) error
45. func dxWorker(ctx worker.Context, args ...interface{}) error { }
46. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() svc.Register("dx", dxWorker) }
47. package worker import ( worker "github.com/contribsys/faktory_worker_go" ) func Run() { svc := worker.NewManager() svc.Register("dx", dxWorker) svc.Run() }
48. Juntando todos os conceitos
49. Premissas do VSA-Worker • Receber os exames que precisamos processar • Delegar a execução para o nosso binário • Notificar em caso de erro no processamento
50. main.go package main import ( // ... ) func main() { // running workers in background go func() { worker.Run() }() // configure API handlers svc := api.New() http.Handle("/", svc.Handler()) logrus.WithField("port", config.Port).Info("starting server") if err := http.ListenAndServe(":"+config.Port, nil); err != nil { logrus.WithError(err).Fatalf("something went wrong") } }
51. main.go package main import ( // ... ) func main() { // running workers in background go func() { worker.Run() }() // configure API handlers svc := api.New() http.Handle("/", svc.Handler()) logrus.WithField("port", config.Port).Info("starting server") if err := http.ListenAndServe(":"+config.Port, nil); err != nil { logrus.WithError(err).Fatalf("something went wrong") } }
52. main.go package main import ( // ... ) func main() { // running workers in background go func() { worker.Run() }() // configure API handlers svc := api.New() http.Handle("/", svc.Handler()) logrus.WithField("port", config.Port).Info("starting server") if err := http.ListenAndServe(":"+config.Port, nil); err != nil { logrus.WithError(err).Fatalf("something went wrong") } }
53. api.go package main import ( // ... ) func (s Service) Handler() *mux.Router { r := mux.NewRouter() r.PathPrefix("/api") r.HandleFunc("/v1/import", s.importHandler).Methods("POST") return r }
54. api.go package main import ( // ... ) func (s Service) Handler() *mux.Router { r := mux.NewRouter() r.PathPrefix("/api") r.HandleFunc("/v1/import", s.importHandler).Methods("POST") return r }
55. api.go type ImportInput struct { Assembly string `json:"assembly,omitempty"` Vcf string `json:"vcf,omitempty"` } func importHandler(w http.ResponseWriter, r *http.Request) { var input ImportInput json.NewDecoder(r.Body).Decode(&input) client, _ := faktory.Open() defer client.Close() job := faktory.NewJob("dx", input.Assembly, input.Vcf) client.Push(job) w.WriteHeader(http.StatusOK) }
56. api.go type ImportInput struct { Assembly string `json:"assembly,omitempty"` Vcf string `json:"vcf,omitempty"` } func importHandler(w http.ResponseWriter, r *http.Request) { var input ImportInput json.NewDecoder(r.Body).Decode(&input) client, _ := faktory.Open() defer client.Close() job := faktory.NewJob("dx", input.Assembly, input.Vcf) client.Push(job) w.WriteHeader(http.StatusOK) }
57. api.go type ImportInput struct { Assembly string `json:"assembly,omitempty"` Vcf string `json:"vcf,omitempty"` } func importHandler(w http.ResponseWriter, r *http.Request) { var input ImportInput json.NewDecoder(r.Body).Decode(&input) client, _ := faktory.Open() defer client.Close() job := faktory.NewJob("dx", input.Assembly, input.Vcf) client.Push(job) w.WriteHeader(http.StatusOK) }
58. main.go package main import ( // ... ) func main() { // running workers in background go func() { worker.Run() }() // configure API handlers svc := api.New() http.Handle("/", svc.Handler()) logrus.WithField("port", config.Port).Info("starting server") if err := http.ListenAndServe(":"+config.Port, nil); err != nil { logrus.WithError(err).Fatalf("something went wrong") } }
59. worker.go package worker import ( // ) func Run() { logrus.Info("starting workers in background mode") svc := worker.NewManager() logrus.WithField("jobType", "dx").Info("register job") svc.Register("dx", dxWorker) logrus.Info("waiting for jobs") svc.Run() }
60. func dxWorker(ctx worker.Context, args ...interface{}) error { var ( assembly = args[0].(string) vcf = args[1].(string) ) cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf) stderr, err := cmd.StderrPipe() if err != nil { return err } if err := cmd.Start(); err != nil { return err } rerr, _ := ioutil.ReadAll(stderr) if err := cmd.Wait(); err != nil { return fmt.Errorf("%s", rerr) } return nil }
61. func dxWorker(ctx worker.Context, args ...interface{}) error { var ( assembly = args[0].(string) vcf = args[1].(string) ) cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf) stderr, err := cmd.StderrPipe() if err != nil { return err } if err := cmd.Start(); err != nil { return err } rerr, _ := ioutil.ReadAll(stderr) if err := cmd.Wait(); err != nil { return fmt.Errorf("%s", rerr) } return nil }
62. func dxWorker(ctx worker.Context, args ...interface{}) error { var ( assembly = args[0].(string) vcf = args[1].(string) ) cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf) stderr, err := cmd.StderrPipe() if err != nil { return err } if err := cmd.Start(); err != nil { return err } rerr, _ := ioutil.ReadAll(stderr) if err := cmd.Wait(); err != nil { return fmt.Errorf("%s", rerr) } return nil }
63. func dxWorker(ctx worker.Context, args ...interface{}) error { var ( assembly = args[0].(string) vcf = args[1].(string) ) cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf) stderr, err := cmd.StderrPipe() if err != nil { return err } if err := cmd.Start(); err != nil { return err } rerr, _ := ioutil.ReadAll(stderr) if err := cmd.Wait(); err != nil { return fmt.Errorf("%s", rerr) } return nil }
64. http://localhost:7420
65. func dxWorker(ctx worker.Context, args ...interface{}) error { var ( assembly = args[0].(string) vcf = args[1].(string) ) cmd := exec.Command("vsa", "dx", "--assembly", assembly, "--vcf", vcf) stderr, err := cmd.StderrPipe() if err != nil { return err } if err := cmd.Start(); err != nil { return err } rerr, _ := ioutil.ReadAll(stderr) if err := cmd.Wait(); err != nil { return fmt.Errorf("%s", rerr) } return nil }
66. http://localhost:7420
67. Exemplo de call // payload.json { "assembly":"GRCh37", "vcf":"/home/marcosinger/files/FXM281-001.vcf.gz" }
68. Exemplo de call curl -XPOST \ -H "Content-Type: application/json" \ --data @payload.json \ http://host:port/api/v1/import
69. 15 exames/hora 5 máquinas c4.4xlarge Amazon SQS MongoDB Elasticsearch
70. 200 exames/hora 1 máquina t2.large Faktory MongoDB
71. Obrigado! Marco Singer @mahsinger linkedin.com/in/marcosinger github.com/marcosinger