Building Data Pipelines in Python @ QCon London 2017

Building Data Pipelines in Python @ QCon LondonĀ 2017

Slides for my talk at QCon London 2017:
https://qconlondon.com/london-2017/presentation/building-data-pipelines-in-python

Abstract:
This talk discusses the process of building data pipelines, e.g. extraction, cleaning, integration, pre-processing of data, in general all the steps that are necessary to prepare your data for your data-driven product. In particular, the focus is on data plumbing and on the practice of going from prototype to production.

Starting from some common anti-patterns, we'll highlight the need for a workflow manager for any non-trivial project.

We'll discuss the case for Luigi as an interesting option to consider, and we'll consider where it fits in the bigger picture of deploying a data product.

Aa38bb7a9c35bc414da6ec7dcd8d7339?s=128

Marco Bonzanini

March 06, 2017
Tweet