Using Elixir to fight Covid-19

Presented at CodeBEAM V America on March 10, 2021.

Transcript

1. Using Elixir to fight Covid-19 Eric Saxby @sax @ecdysone
2. Introduction
3. Jacob Peter Gowy's The Flight of Icarus (1635–1637) {{PD-1996}}
4. Gustav Moreau's Phaethon (1878) {{PD-1996}}
5. Gustav Doré's Paradise Lost (1866) {{PD-1996}}
6. Timeline of this talk March Let’s help with Covid! (I wish this was Elixir) April/May Let’s help New York contact tracing (could we do this in Elixir?) June/July Let’s rewrite this with Broadway
7. Using Elixir to fight Covid-19 real talk
8. Luc Viatour's Total Solar Eclipse (1999) {{CC BY-SA 3.0}}
9. March 2020
10. {{PD-1996}} Gustav Moreau's Diomede devoured by his horses (1865) Personal Protective Equipment
11. {{PD-1996}} Gustav Moreau's Diomede devoured by his horses (1865) - Track hospital usage & needs - Allow supply chain to route to tomorrow’s needs - Ruby on Rails - Heroku PPE resource status
12. @jasondeblooisphotography - March 2020 : CHAOS - Large companies also solving this problem - Project purposefully shut down Retrospective
13. April 2020
14. Caravaggio's Cardsharps (1594) {{PD-1996}} - Generic platform - Busy with user workflows - Unable to add integrations Contact Tracing
15. Norman Rockwell's Radio Phone (1922) {{PD-US}} Telephony - Phoenix / LiveView - AWS ECS
16. Paolo Veronese's Arachne (Dialects) {{PD-US}} Data pipeline - NodeJS - Lambda
17. Caravaggio's Cardsharps (1594) {{PD-1996}} Shared Values - Protect PHI - As simple as possible
18. Elixir vs NodeJS John Steeple Davis's Scene of the Battle of Marathon (1900) {{PD-1996}}
19. Giovanni Domenico Tiepolo's The Procession of the Trojan Horse in Troy (1773) {{PD-1996}} Phoenix vs Rails
20. @jasondeblooisphotography - Elixir v Node v Ruby - all allow rapid development - Elixir turned out simpler resilient solutions Retrospective
21. Deeper look @ data pipeline - State database - HL7 aggregator - Each lab (possibly each location) creates its own integration - Required fields, with minimal/no validations - No built in Master Patient Index
22. Master Patient Index - Unique ID for each person in the public health system - Deduplication - Cross-reference - Each state** runs their own MPI ** or they might not
23. State database Somebody gets the file They give it to this person Who asks someone to “enrich” the data So they can get an MPI (hours later) Then they give it to this person Who undoes some of the previous steps?? Then gives it to us? So we can put the data in the Contact Tracing system every 30m literally forever 5s ?? N hours ??
24. With “post-processing” that undoes any deduplication from the MPI Only 7% successful MPI might not be unique Changes delimiter, without escaping data
25. Consequences Jacques-Louis David's Sorrow (1773) {{PD-1996}} - Massive delays, for little benefit - Multiple calls to the same people - Overloaded workers - Low trust
26. June 2020
27. Let’s look at some data
28. 555 555 1234 000 000 0000 1 {wait, is that a hospital?} {yeah, that’s definitely a hospital} 555 1234 Phone numbers
29. 123 Main St 123 Mani St {wait, is this a hospital?} {yeah, that’s definitely a hospital} Addresses
30. Names Leonardo / da / Vinci Leonardo da / / Vinci Leonardo / / Vinci Leo / / Vinci Loenardo / / Vinci
31. Leonardo / da / Vinci / M Leonardo da / / Vinci / M Leonardo / / Vinci / M Leo / / Vinci / M Leonardo / da / Vinci / F Gender
32. …with no unique constraints… …because they come from lab systems… …that sometimes recycle ids every month… …with no guarantee that different labs won’t use the same id… “Unique” identifiers
33. Can we be an MPI? Vasily Polenov's Left hand with the index finger (c1885) {{PD-1996}}
34. - Addresses - nope - Name and birth date? - Gender? - nope - Phone numbers? - nope - Accession #s - nope Vasily Polenov's Left hand with the index finger (c1885) {{PD-1996}}
35. Eugène Delacroix's Hamlet and Horatio in the Cemetary (1839) {{PD-1996}} Lets talk to people! - Data is unreliable - Duplication is terrible - Full time positions finding duplicates
36. Eugène Delacroix's Hamlet and Horatio in the Cemetary (1839) {{PD-1996}} Insights - Manual correction is fine* - * if quantity is low enough - Problem is duplicate phone calls
37. Can we be an MPI? Thomas Cole's Prometheus Bound (1846-1847) {{PD-1996}}
38. Maybe?? Why not?? Thomas Cole's Prometheus Bound (1846-1847) {{PD-1996}}
39. Odilon Redon's Icarus (c1890) {{PD-1996}} Insights - There was no MPI - “Good enough” would be much better than what’s there
40. Odilon Redon's Icarus (c1890) {{PD-1996}} Giorgio / G / Vasari / M / 1511 / ABC123 Giorgio G / / Vasari / M / 1511 / ABC123 Giorgio / G / Vasari / F / 1511 / ABC123 Gorgio / / Vasari / M / 1511 / ABC124 Giorgio / / Vasari / M / 1151 / ABC125 Giorgio / Geo / Vasari / F / 1511 / ABC126 Giorgio / G / Vasari / M / 1511 / CDE123
41. Odilon Redon's Icarus (c1890) {{PD-1996}} Giorgio / G / Vasari / M / 1511 / ABC123 Giorgio G / / Vasari / M / 1511 / ABC123 Giorgio / G / Vasari / F / 1511 / ABC123 Gorgio / / Vasari / M / 1511 / ABC124 Giorgio / / Vasari / M / 1151 / ABC125 Giorgio / Geo / Vasari / F / 1511 / ABC126 Giorgio / G / Vasari / M / 1511 / CDE123
42. Can we beat 6 hours? Ilya Repin's Leisure (1882) {{PD-1996}}
43. A brief history of streams - Enum - Jan 6, 2012 - “Start working on Enum protocol” - Stream - June 30, 2013 - “Add streams” - GenStage - Aug 26, 2015 - initial commit - Flow - Jan 17, 2017 - initial commit - Broadway - Feb 25, 2019 - Announcing Broadway
44. Broadway
47. Melchior d'Hondecoeter's The Menagerie (1690) {{PD-1996}} File Row Processor Row Batcher New Person Processor Lab Result Processor Lab Result Batcher ContactTracing Adapter Processor Batcher
48. File Row Processor Batcher New Person Processor Lab Result Processor Lab Result Batcher ContactTracing Adapter Processor Batcher
49. - Producer - handle_demand/2 - Transformer - Processor - handle_message/3 - Batcher - handle_batch/4 - Acknowledger
50. File Reader
51. Generate me a checksum :new :update
52. Naive approach
55. Back pressure
57. ????
59. File Row Processor Batcher New TestResult Records
60. DONE
61. Duplicate rows in a file - Mutexes - should have just used partition_by
62. does it blend?
63. - Old pipeline - now 3-8hrs - Broadway - 15m - +SP 200:200 - 10m - ETS cache - 25s 500k+ database queries
64. does it blend?
65. Next pipeline!
66. Unprocessed TestResults Person/Case Processor New/Updated Case Records
67. Home-grown Event Sourcing Yaaaaay
68. Halting problem Alexandre Antigna's The forced stop (1855) {{PD-1996}}
70. Next: Extract data!
71. - API calls per county - minimum page size 100 - N counties
72. handle_demand/2 handle_demand — county 1, page 1 - extra records set into a buffer on state handle_demand — read from buffer handle_demand — next page, extras on buffer - if no next page, increment county handle_demand — county 2, page 1 …etc
73. Halting problem Alexandre Antigna's The forced stop (1855) {{PD-1996}}
74. Buffering demand
77. Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher
78. Dispatcher Dispatcher Dispatcher Dispatcher Dispatcher • Consumers demand work from their dispatcher • Dispatchers demand work from producers • If the producer does not fill demand, the dispatcher may wait for more events to arrive When there is more demand than data, producers must track unmet demand
79. handle_demand/2 - handle_demand — county 1, page 1 - extra records set into a buffer on state - handle_demand — read from buffer - handle_demand — next page, extras on buffer - if no next page, increment county - county 2, page 1 - …etc Last page may not fulfill demand
80. Next: Send data to contact tracing
81. - HTTP(S) is unreliable - Don’t trust system uptime - Broadway does not come with retries - though it may be implemented in a producer
82. Hans Gude's Oban Bay (1889) {{PD-1996}} Oban
83. - Background job runner - Built-in retry - Oban Web - paid license - Quick to set up - Uses PostgreSQL for queues Hans Gude's Oban Bay (1889) {{PD-1996}}
84. Melchior d'Hondecoeter's The Menagerie (1690) {{PD-1996}} File Producer Row Processor Row Batcher Unprocessed TestResults Producer People/Case Processor Cases in CommCare Push to Oban Processor New/Updated Cases Producer Test Results (unique rows in files) People + Cases Oban Cases Updated in CT Producer Case Processor People + Cases Cases updated in CommCare
85. Broadway vs Oban
86. Testing Willy Schlobach's Test drive over the Alps (1924) {{PD-1996}}
87. Testing Willy Schlobach's Test drive over the Alps (1924) {{PD-1996}}
88. Switchover
89. Switchover
90. Timelines
91. Deployment Giovanni Domenico Tiepolo's Building of the Trojan Horse (1760) {{PD-1996}}
92. Retrospective
93. - Broadway is awesome - Its limits are fine - In the future, maybe just GenStage - Event sourcing - Our domain schemas were fine - Maybe eventstore/Kafka next time - OpenTelemetry - Endemic problems in health care - Need more individuals able to predict consequences
94. Thank you