Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Smart Handoff/Pickup ガイド - Claude Code セッション管理
Avatar for rasshii yukiigarashi
0
140
例外処理とどう使い分ける？Result型を使ったエラー設計 #burikaigi
Avatar for Takuma Kajikawa kajitack
16
6.1k
Rust 製のコードエディタ “Zed” を使ってみた
Avatar for NearMeの技術発表資料です nearme_tech
PRO
0
180
AIで開発はどれくらい加速したのか？AIエージェントによるコード生成を、現場の評価と研究開発の評価の両面からdeep diveしてみる
Avatar for どすこい daisuketakeda
1
2.5k
Package Management Learnings from Homebrew
Avatar for Mike McQuaid mikemcquaid
0
230
ノイジーネイバー問題を解決する 公平なキューイング
Avatar for Shoichi Ochi occhi
0
100
15年続くIoTサービスの SREエンジニアが挑む 分散トレーシング導入
Avatar for Melonps melonps
2
210
疑似コードによるプロンプト記述、どのくらい正確に実行される？
Avatar for kokuyouwind kokuyouwind
0
390
CSC307 Lecture 07
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
550
Apache Iceberg V3 and migration to V3
Avatar for Tomohiro Tanaka tomtanaka
0
160
Grafana：建立系統全知視角的捷徑
Avatar for Blueswen blueswen
0
330
Architectural Extensions
Avatar for Denys Poltorak denyspoltorak
0
290

Featured

See All Featured
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
310
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
170
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
270
Tips & Tricks on How to Get Your First Job In Tech
Avatar for Honza Javorek honzajavorek
0
440
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
250
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
62
50k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
3
220
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.9k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.8k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
196
71k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]