Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
(Extension DC 2025) Actor境界を越える技術
Avatar for Himeshi teamhimeh
1
240
CSC305 Lecture 02
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
260
非同期jobをtransaction内で 呼ぶなよ！絶対に呼ぶなよ！
Avatar for Yuto Urushima alstrocrack
0
560
Goで実践するドメイン駆動開発 AIと歩み始めた新規プロダクト開発の現在地
Avatar for imkaoru imkaoru
4
760
NetworkXとGNNで学ぶグラフデータ分析入門〜複雑な関係性を解き明かすPythonの力〜
Avatar for みずほリサーチ&テクノロジーズ株式会社 先端技術研究部 mhrtech
3
1.1k
Domain-centric? Why Hexagonal, Onion, and Clean Architecture Are Answers to the Wrong Question
Avatar for Oliver Drotbohm olivergierke
2
640
After go func(): Goroutines Through a Beginner’s Eye
Avatar for Vaibhav Gupta 97vaibhav
0
240
Swift Concurrency - 状態監視の罠
Avatar for Yuki Yasoshima objectiveaudio
2
480
CSC305 Lecture 04
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
260
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
Avatar for Samuel Williams ioquatix
3
1.2k
Local Peer-to-Peer APIはどのように使われていくのか？
Avatar for Hal hal_spidernight
2
460
LLMとPlaywright/reg-suitを活用した jQueryリファクタリングの実際
Avatar for kinocoboy kinocoboy2
4
680

Featured

See All Featured
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
19
1.2k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
697
190k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
53
9k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
Done Done
Avatar for chrislema chrislema
185
16k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
23
1.5k
Writing Fast Ruby
Avatar for Erik Berlin sferik
629
62k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
189
55k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
514
110k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
15
1.7k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
274
40k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]