Discovering Drugs with Kafka Streams

Recursion Pharmaceuticals is turning drug discovery into a data science problem. This entails producing and processing petabytes of microscopy images from carefully designed biological experiments. In early 2017 the data production effort in our laboratory scaled to a point where the existing naive batch processing system was not reliably processing the data. The batch approach was also introducing unwanted lag between experiment image capture time and analysis results since an entire experiment, potentially 8TB+, would not begin processing until all the images were available. This was particularly troublesome for our laboratory as they wanted real time quality control metrics on the images. All of these reasons motivated us to replace the batch processing system with a streaming approach. The original data pipeline was implemented as microservices with no central orchestrator but instead relied on implicit flow between the services. The lack of visibility and robustness made the pipeline difficult and costly to operate. We wanted to address these concerns but also avoid rewriting the existing microservices. By building on top of Kafka Streams we created a flexible, highly available, and robust pipeline which leveraged our existing microservices giving us a clear migration path. This presentation will walk you through our thought process and explain the tradeoffs between using Kafka Streams and Spark for our specific use case. We’ll dive into the details of the workflow system we created on top of Kafka Streams that orchestrates these microservices. We’ve been operating with this system since mid 2017 and the additional scale and robustness has played a key role in enabling Recursion to succeed in its mission of discovering new treatments for various diseases. The messages flowing over our Kafka Streams have already led to clinical trials in humans and will hopefully translate into meaningful impact in patients lives one day.

Transcript

1. Ben Mabey VP of Engineering @bmabey Discovering Drugs with Kafka Streams Scott Nielsen Director of Data Engineering K A F K A S U M M I T S F 2 0 1 9
2. Penn Teller
3. Penn Teller B Scott
4. Decoding Biology to Radically Improve Lives
5. © 2017 Recursion Pharmaceuticals 1000s of untreated genetic diseases Photo of our wall?
6. 0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) Moore’s Law
7. 0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) 1 10 100 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 R&D Spend / Drug (% of 2007 values) Moore’s Law
8. 0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) 1 10 100 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 R&D Spend / Drug (% of 2007 values) Moore’s Law Eroom’s Law
9. 0 10 20 30 40 50 60 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Number of Drugs Approved in US (1993-2016)
10. How can we fix this?
11. RecursionPharma.com
12. RecursionPharma.com Over 7 million per week
13. RecursionPharma.com hoechst (DNA)
14. RecursionPharma.com concanavalin A (ER)
15. RecursionPharma.com mitotracker (mitochondria)
16. RecursionPharma.com WGA (golgi apparatus, cell membrane)
17. RecursionPharma.com SYTO 14 (RNA, nucleoli)
18. RecursionPharma.com phalloidin (actin fibers)
19. RecursionPharma.com combined
20. How do these pretty pictures help?
21. Healthy child Child with rare genetic disease (Cornelia de Lange Syndrome)
22. Healthy child Healthy cells Child with rare genetic disease (Cornelia de Lange Syndrome) Genetic disease model cells (Cornelia de Lange Syndrome)
23. Healthy Disease
24. Healthy Disease Disease + Drug?
25. Public Dataset: http://rxrx.ai Nature Article Machine learning brings cell imaging promises into focus https://tinyurl.com/ml-cells
 Learn more…
26. How is this data produced?
27. 308 wells/plate
28. 4 sites/well 308 wells/plate
29. 6 channels (images)/site 7,392 images per plate 4 sites/well 308 wells/plate
30. 6 channels (images)/site 7,392 images per plate 4 sites/well 308 wells/plate ~69GB per plate
31. Experiment A Experiment B Experiment C Experiment D
32. Our “Series A” System
33. On-Premise
34. On-Premise
35. Stream images to S3 On-Premise
36. Generate thumbnails Image metrics Stream images to S3 On-Premise
37. Generate thumbnails Image metrics Stream images to S3 On-Premise
38. Generate thumbnails Image metrics Fire and forget Stream images to S3 On-Premise
39. Generate thumbnails Image metrics Fire and forget Experiment A Stream images to S3 On-Premise
40. Generate thumbnails Image metrics Fire and forget Experiment A Stream images to S3 Extract Features On-Premise Process experiments in batch
41. Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features On-Premise Process experiments in batch
42. Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch
43. Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch
44. Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch
45. Traditional, low throughput, biology
46. Traditional, low throughput, biology ~6-12 plates per week, ~400-800GB
47. © 2017 Recursion Pharmaceuticals High-throughput experiments Robots photo
51. 100 6.9TB
52. 100 6.9TB 300 20TB
53. 100 6.9TB 300 20TB Kafka Streams solution was launched
54. 100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB Kafka Streams solution was launched
57. 100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB
58. 100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB 280 TB Today
59. So what was wrong with the original system?
60. Generate thumbnails Image metrics Extract Features metrics, models, reports, etc On-Premise Process experiments in batch
61. Experiment A Experiment B Experiment C Experiment D Plates are not imaged in order
63. Migration Goals
64. Migration Goals Move orchestration and processing to cloud.
65. Migration Goals Move orchestration and processing to cloud.
66. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads.
67. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads.
68. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic.
69. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper.
70. Let’s take a look at the logical pipeline that we needed to implement…
71. Images / channel level
72. Images / channel level image level metrics
73. Images / channel level site (all channels/images) thumbnails image level metrics
74. Images / channel level site (all channels/images) thumbnails site level features image level metrics
75. Images / channel level site (all channels/images) thumbnails site level features image level metrics
76. Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics
77. well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics
78. well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics metrics
79. well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics metrics plate level features metrics
80. well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics Experiment A
81. well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc Experiment A
85. Kafka Streams was just released…
86. Kafka Streams was just released…
87. dagger workflow library written on top of Kafka Streams that orchestrates microservices
88. dagger workflow library written on top of Kafka Streams that orchestrates microservices Dagger, ya know, because it is all about the workflows represented as directed acyclic graphs, i.e. DAGs.
89. dagger workflow library written on top of Kafka Streams that orchestrates microservices
90. New workflow system in 2017?
91. New workflow system in 2017? Not Invented Here syndrome?
92. Core logic in library is ~2800 LOC New workflow system in 2017? Not Invented Here syndrome?
93. Core logic in library is ~2800 LOC All of our our DAGs, including schema, task, and workflow definition ~1700 LOC New workflow system in 2017? Not Invented Here syndrome?
94. Core logic in library is ~2800 LOC All of our our DAGs, including schema, task, and workflow definition ~1700 LOC New workflow system in 2017? Not Invented Here syndrome?
95. well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc
96. Let’s look at a small workflow using Kafka Streams initially…
97. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream
98. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
99. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
100. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
101. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
102. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
103. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable<String, ExperimentMetadata> experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream<String, ChannelLevel> images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream<String, Site> sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);
104. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic
105. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic
106. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic task output topic
107. How would you do the same workflow in dagger?
108. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream
109. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables
110. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations
111. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations Tasks
112. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations Tasks Output topics
113. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
114. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
115. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
116. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
117. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
118. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
119. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}} Specify function to be used
120. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
121. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
122. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
123. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
124. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}
125. extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} images_site stream
126. extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} images_site stream
127. extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} Inline function directly images_site stream
128. extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream
129. ( ) Dagger is a compiler
130. ( ) Kafka Streams Topology Dagger is a compiler
131. ( ) Kafka Streams Topology Dagger is a compiler
132. What would the entire pipeline look like in dagger?
133. well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc
135. Our pipeline application that uses Dagger
136. How does the whole system look like now?
137. Generate thumbnails Image metrics Extract Features metrics, models, reports, etc On-Premise Process experiments in batch
138. On-Premise
139. On-Premise Publish Image Events
140. On-Premise Publish Image Events
141. On-Premise Publish Image Events Uploader
142. On-Premise Publish Image Events Uploader dagger is used here too!
143. On-Premise Publish Image Events Uploader
144. On-Premise Publish Image Events Uploader
145. On-Premise Publish Image Events Uploader
146. On-Premise Autoscaled Workers Publish Image Events Uploader
147. On-Premise Microservices Publishers & Consumers Autoscaled Workers Publish Image Events Uploader
148. On-Premise BigQuery SQL Transform & Load Microservices Publishers & Consumers Autoscaled Workers Publish Image Events Uploader
149. Migration Goals
150. Migration Goals Move orchestration and processing to cloud. ✓
151. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. ✓ ✓
152. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. ✓ ✓ ✓
153. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper. ✓ ✓ ✓ ✓
154. Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper. ✓ ✓ ✓ ✓ EC2 and Lambda -> Google Clould preemptibles.
155. Big data, small metadata…
156. Big data, small metadata…
157. Lessons learned…
158. Early Adopter Tax
159. Missed out on mature workflow monitoring
162. On-Premise Transform & Load Uploader Easy deployment!
163. Kafka Streams App External Service task input topic task output topic Durable Log FTW
164. Thank you!
166. Come help us decode biology! @RecursionPharma @bmabey