Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
VibeCodingからAgenticWorkflowへ
Avatar for starfish719 starfish719
0
940
[PyCon KR 2026] More Variants, More Diversity for AI Accelerators
Avatar for Joongi Kim achimnol
0
120
いまどきの Codex で開発する visionOS アプリの開発スタイルについて
Avatar for kazuhiro hara karad
0
170
Hello, Hiroshima Geospatial Data! — Exploring DoboX with Python
Avatar for ぴっかりん ra0kley
0
110
How I Won Prize Money at a Hackathon Using Codex and Symphony Alpha
Avatar for yasei no otoko yasei_no_otoko
0
130
Oxlintはいいぞ（続）
Avatar for Yuji Yamaguchi yug1224
1
480
My Marp Sample
Avatar for Shota Inoue sinoue0108
0
130
信頼性の目標を誰も求めてない
Avatar for Shuma shubox
0
240
リアルな遅延を測る仕様
Avatar for kota-yata kota_yata
1
130
AIに既存システムを理解させる技術　～レガシーを見捨てないハーネスエンジニアリング入門～
Avatar for Ochtum ochtum
0
170
Pythonの実行はどこまで賢くなったのか？ CPythonとPyPyから見る最適化のしくみ
Avatar for curekoshimizu curekoshimizu
4
2.3k
AI駆動開発にグラフDBを重ねてみた
Avatar for Satoshi Kaneyasu satoshi256kbyte
1
370

Featured

See All Featured
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
57
14k
Information Architects: The Missing Link in Design Systems
Avatar for Michelle Chin soysaucechin
1
1.1k
Accessibility Awareness
Avatar for Sarah Abderemane sabderemane
1
180
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
61
10k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
27k
The Mindset for Success: Future Career Progression
Avatar for Greg Gifford greggifford
PRO
0
480
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
780
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.3k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
28
3.6k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
830
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.3k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
41k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]