Tweet
Tweet

Slide 1

Slide 1 text

Ben Mabey VP of Engineering @bmabey Discovering Drugs with Kafka Streams Scott Nielsen Director of Data Engineering K A F K A S U M M I T S F 2 0 1 9

Slide 2

Slide 2 text

Penn Teller

Slide 3

Slide 3 text

Penn Teller B Scott

Slide 4

Slide 4 text

Decoding Biology to Radically Improve Lives

Slide 5

Slide 5 text

© 2017 Recursion Pharmaceuticals 1000s of untreated genetic diseases Photo of our wall?

Slide 6

Slide 6 text

0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) Moore’s Law

Slide 7

Slide 7 text

0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) 1 10 100 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 R&D Spend / Drug (% of 2007 values) Moore’s Law

Slide 8

Slide 8 text

0.00001 0.0001 0.001 0.01 0.1 1 10 100 1000 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Transistor Area (% of 1970 values) 1 10 100 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 R&D Spend / Drug (% of 2007 values) Moore’s Law Eroom’s Law

Slide 9

Slide 9 text

0 10 20 30 40 50 60 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Number of Drugs Approved in US (1993-2016)

Slide 10

Slide 10 text

How can we fix this?

Slide 11

Slide 11 text

RecursionPharma.com

Slide 12

Slide 12 text

RecursionPharma.com Over 7 million per week

Slide 13

Slide 13 text

RecursionPharma.com hoechst (DNA)

Slide 14

Slide 14 text

RecursionPharma.com concanavalin A (ER)

Slide 15

Slide 15 text

RecursionPharma.com mitotracker (mitochondria)

Slide 16

Slide 16 text

RecursionPharma.com WGA (golgi apparatus, cell membrane)

Slide 17

Slide 17 text

RecursionPharma.com SYTO 14 (RNA, nucleoli)

Slide 18

Slide 18 text

RecursionPharma.com phalloidin (actin fibers)

Slide 19

Slide 19 text

RecursionPharma.com combined

Slide 20

Slide 20 text

How do these pretty pictures help?

Slide 21

Slide 21 text

Healthy child Child with rare genetic disease (Cornelia de Lange Syndrome)

Slide 22

Slide 22 text

Healthy child Healthy cells Child with rare genetic disease (Cornelia de Lange Syndrome) Genetic disease model cells (Cornelia de Lange Syndrome)

Slide 23

Slide 23 text

Healthy Disease

Slide 24

Slide 24 text

Healthy Disease Disease + Drug?

Slide 25

Slide 25 text

Public Dataset: http://rxrx.ai Nature Article Machine learning brings cell imaging promises into focus https://tinyurl.com/ml-cells
 Learn more…

Slide 26

Slide 26 text

How is this data produced?

Slide 27

Slide 27 text

308 wells/plate

Slide 28

Slide 28 text

4 sites/well 308 wells/plate

Slide 29

Slide 29 text

6 channels (images)/site 7,392 images per plate 4 sites/well 308 wells/plate

Slide 30

Slide 30 text

6 channels (images)/site 7,392 images per plate 4 sites/well 308 wells/plate ~69GB per plate

Slide 31

Slide 31 text

Experiment A Experiment B Experiment C Experiment D

Slide 32

Slide 32 text

Our “Series A” System

Slide 33

Slide 33 text

On-Premise

Slide 34

Slide 34 text

On-Premise

Slide 35

Slide 35 text

Stream images to S3 On-Premise

Slide 36

Slide 36 text

Generate thumbnails Image metrics Stream images to S3 On-Premise

Slide 37

Slide 37 text

Generate thumbnails Image metrics Stream images to S3 On-Premise

Slide 38

Slide 38 text

Generate thumbnails Image metrics Fire and forget Stream images to S3 On-Premise

Slide 39

Slide 39 text

Generate thumbnails Image metrics Fire and forget Experiment A Stream images to S3 On-Premise

Slide 40

Slide 40 text

Generate thumbnails Image metrics Fire and forget Experiment A Stream images to S3 Extract Features On-Premise Process experiments in batch

Slide 41

Slide 41 text

Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features On-Premise Process experiments in batch

Slide 42

Slide 42 text

Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch

Slide 43

Slide 43 text

Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch

Slide 44

Slide 44 text

Generate thumbnails Image metrics Fire and forget Stream images to S3 Extract Features metrics, models, reports, etc On-Premise Process experiments in batch

Slide 45

Slide 45 text

Traditional, low throughput, biology

Slide 46

Slide 46 text

Traditional, low throughput, biology ~6-12 plates per week, ~400-800GB

Slide 47

Slide 47 text

© 2017 Recursion Pharmaceuticals High-throughput experiments Robots photo

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

100 6.9TB

Slide 52

Slide 52 text

100 6.9TB 300 20TB

Slide 53

Slide 53 text

100 6.9TB 300 20TB Kafka Streams solution was launched

Slide 54

Slide 54 text

100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB Kafka Streams solution was launched

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB

Slide 58

Slide 58 text

100 6.9TB 300 20TB 700 48TB 1,300 90TB 1,700 118TB 1,900 132 TB 280 TB Today

Slide 59

Slide 59 text

So what was wrong with the original system?

Slide 60

Slide 60 text

Generate thumbnails Image metrics Extract Features metrics, models, reports, etc On-Premise Process experiments in batch

Slide 61

Slide 61 text

Experiment A Experiment B Experiment C Experiment D Plates are not imaged in order

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Migration Goals

Slide 64

Slide 64 text

Migration Goals Move orchestration and processing to cloud.

Slide 65

Slide 65 text

Migration Goals Move orchestration and processing to cloud.

Slide 66

Slide 66 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads.

Slide 67

Slide 67 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads.

Slide 68

Slide 68 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic.

Slide 69

Slide 69 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper.

Slide 70

Slide 70 text

Let’s take a look at the logical pipeline that we needed to implement…

Slide 71

Slide 71 text

Images / channel level

Slide 72

Slide 72 text

Images / channel level image level metrics

Slide 73

Slide 73 text

Images / channel level site (all channels/images) thumbnails image level metrics

Slide 74

Slide 74 text

Images / channel level site (all channels/images) thumbnails site level features image level metrics

Slide 75

Slide 75 text

Images / channel level site (all channels/images) thumbnails site level features image level metrics

Slide 76

Slide 76 text

Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics

Slide 77

Slide 77 text

well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics

Slide 78

Slide 78 text

well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics metrics

Slide 79

Slide 79 text

well level features Images / channel level site (all channels/images) thumbnails site level features image level metrics site metrics metrics plate level features metrics

Slide 80

Slide 80 text

well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics Experiment A

Slide 81

Slide 81 text

well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc Experiment A

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

Kafka Streams was just released…

Slide 86

Slide 86 text

Kafka Streams was just released…

Slide 87

Slide 87 text

dagger workflow library written on top of Kafka Streams that orchestrates microservices

Slide 88

Slide 88 text

dagger workflow library written on top of Kafka Streams that orchestrates microservices Dagger, ya know, because it is all about the workflows represented as directed acyclic graphs, i.e. DAGs.

Slide 89

Slide 89 text

dagger workflow library written on top of Kafka Streams that orchestrates microservices

Slide 90

Slide 90 text

New workflow system in 2017?

Slide 91

Slide 91 text

New workflow system in 2017? Not Invented Here syndrome?

Slide 92

Slide 92 text

Core logic in library is ~2800 LOC New workflow system in 2017? Not Invented Here syndrome?

Slide 93

Slide 93 text

Core logic in library is ~2800 LOC All of our our DAGs, including schema, task, and workflow definition ~1700 LOC New workflow system in 2017? Not Invented Here syndrome?

Slide 94

Slide 94 text

Core logic in library is ~2800 LOC All of our our DAGs, including schema, task, and workflow definition ~1700 LOC New workflow system in 2017? Not Invented Here syndrome?

Slide 95

Slide 95 text

well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc

Slide 96

Slide 96 text

Let’s look at a small workflow using Kafka Streams initially…

Slide 97

Slide 97 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream

Slide 98

Slide 98 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 99

Slide 99 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 100

Slide 100 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 101

Slide 101 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 102

Slide 102 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 103

Slide 103 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream final KTable experimentMetadata = builder.table( EXPERIMENT_METADATA_TOPIC); final KStream images = builder.stream( CHANNEL_IMAGES_TOPIC); final KStream sites = images .groupBy((exp, channel) -> channel.site()) .windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS))) .aggregate( () -> new AggState(), (site, channel, agg) -> agg.observe(channel.site(), channel.channel), (site, agg_a, agg_b) -> agg_a.merge(agg_b)) .join(experimentMetadata, (agg, expMeta) -> agg.markCompleted(expMeta.numChannels)) .filterValues(agg -> agg.isComplete()) .mapValues(agg -> agg.site()); sites.to(SITE_IMAGES_TOPIC);

Slide 104

Slide 104 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic

Slide 105

Slide 105 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic

Slide 106

Slide 106 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Kafka Streams App External Service task input topic task output topic

Slide 107

Slide 107 text

How would you do the same workflow in dagger?

Slide 108

Slide 108 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream

Slide 109

Slide 109 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables

Slide 110

Slide 110 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations

Slide 111

Slide 111 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations Tasks

Slide 112

Slide 112 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream Input topics & tables Stream operations Tasks Output topics

Slide 113

Slide 113 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 114

Slide 114 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 115

Slide 115 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 116

Slide 116 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 117

Slide 117 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 118

Slide 118 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 119

Slide 119 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}} Specify function to be used

Slide 120

Slide 120 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 121

Slide 121 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 122

Slide 122 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 123

Slide 123 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 124

Slide 124 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream {"name": "extract-site-level-features", "graph": {"images-channel": {"type": "topic-stream", "topic-name": "images_channels"} "experiment-metadata": {"type": "topic-table", "topic-name": "experiment_metadata"}, "images-site": {"type": "stream-operation", "key-schema": "long", "value-schema": "job_site_level", "inputs": ["images-channel", "experiment-metadata"], "function": "aggregations/images-site-grouping"}, "features-site": {"type": "external-task", "stream": "images-site", "task-name": "extract-features"}, "features-output": {"type": "publish", "topic-name": "extracted_features", "stream": "features-site"}}}

Slide 125

Slide 125 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} images_site stream

Slide 126

Slide 126 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} images_site stream

Slide 127

Slide 127 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic {:name "extract-site-level-features", :graph {:images-channel {:type :topic-stream, :topic-name "images_channels"}, :experiment-metadata {:type :topic-table, :topic-name "experiment_metadata"}, :images-site {:type :stream-operation, :key-schema :long, :value-schema "job_site_level", :inputs [:images-channel, :experiment-metadata], :function (fn [images-channel experiment-metadata] …), :features-site {:type :external-task, :task-name "extract-features", :stream :images-site}, :features-output {:type :publish, :stream :features-site, :topic-name "extracted_features"}}} Inline function directly images_site stream

Slide 128

Slide 128 text

extract site features images_channel topic experiment_metadata topic table extracted_features topic images_site stream

Slide 129

Slide 129 text

( ) Dagger is a compiler

Slide 130

Slide 130 text

( ) Kafka Streams Topology Dagger is a compiler

Slide 131

Slide 131 text

( ) Kafka Streams Topology Dagger is a compiler

Slide 132

Slide 132 text

What would the entire pipeline look like in dagger?

Slide 133

Slide 133 text

well level features Images / channel level site (all channels/images) thumbnails site level features experiment features image level metrics site metrics metrics plate level features metrics metrics, models, reports, etc

Slide 134

Slide 134 text

No content

Slide 135

Slide 135 text

Our pipeline application that uses Dagger

Slide 136

Slide 136 text

How does the whole system look like now?

Slide 137

Slide 137 text

Generate thumbnails Image metrics Extract Features metrics, models, reports, etc On-Premise Process experiments in batch

Slide 138

Slide 138 text

On-Premise

Slide 139

Slide 139 text

On-Premise Publish Image Events

Slide 140

Slide 140 text

On-Premise Publish Image Events

Slide 141

Slide 141 text

On-Premise Publish Image Events Uploader

Slide 142

Slide 142 text

On-Premise Publish Image Events Uploader dagger is used here too!

Slide 143

Slide 143 text

On-Premise Publish Image Events Uploader

Slide 144

Slide 144 text

On-Premise Publish Image Events Uploader

Slide 145

Slide 145 text

On-Premise Publish Image Events Uploader

Slide 146

Slide 146 text

On-Premise Autoscaled Workers Publish Image Events Uploader

Slide 147

Slide 147 text

On-Premise Microservices Publishers & Consumers Autoscaled Workers Publish Image Events Uploader

Slide 148

Slide 148 text

On-Premise BigQuery SQL Transform & Load Microservices Publishers & Consumers Autoscaled Workers Publish Image Events Uploader

Slide 149

Slide 149 text

Migration Goals

Slide 150

Slide 150 text

Migration Goals Move orchestration and processing to cloud. ✓

Slide 151

Slide 151 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. ✓ ✓

Slide 152

Slide 152 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. ✓ ✓ ✓

Slide 153

Slide 153 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper. ✓ ✓ ✓ ✓

Slide 154

Slide 154 text

Migration Goals Move orchestration and processing to cloud. Faster feedback and less bursty workloads. Preserve existing micro-services logic. Make cheaper. ✓ ✓ ✓ ✓ EC2 and Lambda -> Google Clould preemptibles.

Slide 155

Slide 155 text

Big data, small metadata…

Slide 156

Slide 156 text

Big data, small metadata…

Slide 157

Slide 157 text

Lessons learned…

Slide 158

Slide 158 text

Early Adopter Tax

Slide 159

Slide 159 text

Missed out on mature workflow monitoring

Slide 160

Slide 160 text

No content

Slide 161

Slide 161 text

No content

Slide 162

Slide 162 text

On-Premise Transform & Load Uploader Easy deployment!

Slide 163

Slide 163 text

Kafka Streams App External Service task input topic task output topic Durable Log FTW

Slide 164

Slide 164 text

Thank you!

Slide 165

Slide 165 text

No content

Slide 166

Slide 166 text

Come help us decode biology! @RecursionPharma @bmabey