Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Robust datapipelines with drake and Docker

Fa8c8d5560f95f32cb78e164ba2d222e?s=47 Tamas Szilagyi
May 16, 2018
Programming
1
120

Robust datapipelines with drake and Docker

Fa8c8d5560f95f32cb78e164ba2d222e?s=128

Tamas Szilagyi

May 16, 2018
Tweet

More Decks by Tamas Szilagyi

See All by Tamas Szilagyi
Fa8c8d5560f95f32cb78e164ba2d222e?s=48 mtoto
2
800
Fa8c8d5560f95f32cb78e164ba2d222e?s=48 mtoto
0
180

Other Decks in Programming

See All in Programming
8924d254fe8a39021d3822b5d51b6fec?s=48 wiroha
3
660
42a6a049ac8f5265f31858a9509217fb?s=48 uhooi
3
180
Fb0a40e5e7a9dc85520f1646fd41eb44?s=48 higuuu
0
170
B5ea40fde5be980a112c4ea58ad3bd39?s=48 shoheimitani
0
270
C69a66be00bc04081feb1674c9085ca2?s=48 n_makoto
2
290
2594ac7ce91fd7d9a3ce71ca7cc2d0c0?s=48 d_date
1
180
Ad449f3f4e595ade283a30c4f66010be?s=48 seto_hi
1
4.2k
Ba655e3712aaabfbca289fe136f85fe4?s=48 qnighy
0
140
4f2867b43e88bf4fbb6cf9f371dc9501?s=48 guni1192
1
310
72328eacb3f08064170cfec86b728d2a?s=48 onohiroshi1
5
21k
B3d6e998ff66c957f00da6d795f9e32c?s=48 akihiroshiota
2
530
D76231a2114896dfcc7b79ac69558b79?s=48 yosuke_furukawa
PRO
45
19k

Featured

See All Featured
Bfd6b681ec6e8c9bef82ff0521c364f7?s=48 kastner
56
2.1k
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
43
2.1k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
261
36k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
204
10k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
224
15k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
36
2.5k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
45
2.5k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
64
10k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
96
3.4k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
10
750
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
129
22k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
106
5.4k

Transcript

  1. Robust Data Pipelines
 with drake and Docker @tudosgar | tamaszilagyi.com

  2. 2 The end goal for most data analytics projects is

    to build a data product, whether it is a weekly dashboard or deploying a ML model. From getting and cleaning the data to generating plots or fitting models, each project can be split up into individual tasks. Afterwards, we want to make sure that the environment we wrote our code in can be replicated. The low-down @tudosgar | tamaszilagyi.com

  3. 3 Automation. Reproducibility. @tudosgar | tamaszilagyi.com

  4. Workflow management tool that enables you to build managed workflows.

    It is a first serious attempt to create a tool like Airflow or Luigi, but in R @tudosgar | tamaszilagyi.com building pipelines

  5. 5 Your analysis is a sequence of transformations. @tudosgar |

    tamaszilagyi.com

  6. write a plan. A plan can be simple, or very

    complex. It all depends on how you define the tasks’ dependencies. @tudosgar | tamaszilagyi.com

  7. not only targets are kept track of. By default we

    can visualise all imports, functions and transformations in our DAG, not only completed tasks. As a nice bonus we can also monitor progress for longer jobs. @tudosgar | tamaszilagyi.com

  8. make(my_plan) Checks dependencies and cache before creating plan. This means

    that on subsequent runs, only the changed tasks will rerun, leaving the rest intact. @tudosgar | tamaszilagyi.com

  9. 9 A container is kinda like a VM but..different. @tudosgar

    | tamaszilagyi.com

  10. benefits of containers - Easily reproduce your infrastructure - Runs

    independent of host OS - Consistency in production @tudosgar | tamaszilagyi.com

  11. step 1: Dockerfile A Dockerfile includes, in backwards order: -

    Your script - Package dependencies of your script - System level dependencies of these packages @tudosgar | tamaszilagyi.com

  12. step 2: build image This is the stage where we

    actually build our mini computer, ready for deployment. @tudosgar | tamaszilagyi.com

  13. step 3: run container Instantiates the container, mounts the folder

    from our host where the data resides and runs our executable as defined in the Dockerfile. @tudosgar | tamaszilagyi.com

  14. http:/ /tamaszilagyi.com/ @tudosgar @tudosgar | tamaszilagyi.com Find me

  15. Thank you. @tudosgar | tamaszilagyi.com