Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
dbt Pythonモデルで実現するSnowflake活用術
trsnium
0
170
GitHub Actions × RAGでコードレビューの検証の結果
sho_000
0
270
『GO』アプリ バックエンドサーバのコスト削減
mot_techtalk
0
150
データの整合性を保つ非同期処理アーキテクチャパターン / Async Architecture Patterns
mokuo
48
17k
コミュニティ駆動 AWS CDK ライブラリ「Open Constructs Library」 / community-cdk-library
gotok365
2
140
AIの力でお手軽Chrome拡張機能作り
taiseiue
0
170
Grafana Cloudとソラカメ
devoc
0
170
Domain-Driven Transformation
hschwentner
2
1.9k
CI改善もDatadogとともに
taumu
0
120
Java Webフレームワークの現状 / java web framework at burikaigi
kishida
9
2.2k
Lottieアニメーションをカスタマイズしてみた
tahia910
0
130
pylint custom ruleで始めるレビュー自動化
shogoujiie
0
120

Featured

See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.1k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
366
25k
Building Your Own Lightsaber
phodgson
104
6.2k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
How GitHub (no longer) Works
holman
314
140k
Embracing the Ebb and Flow
colly
84
4.6k
Writing Fast Ruby
sferik
628
61k
Building a Scalable Design System with Sketch
lauravandoore
461
33k
How STYLIGHT went responsive
nonsquared
98
5.4k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]