Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
1

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
KiroのSpecで「五目並べ」を作ってみる
Avatar for Satoshi Kaneyasu satoshi256kbyte
1
380
AWS CDKのカスタムリソースでContinuum（旧Security Agent）を実装した話 | Implementing Continuum (formerly Security Agent) via AWS CDK Custom Resource
Avatar for アキキー | Akihisa Ikeda akihisaikeda
2
150
Railsのためのパスキー実践入門 〜マルチテナントSaaSでの意思決定〜
Avatar for utakaha utakaha
2
550
個人開発者よ、Honoを使え（おじさんver）
Avatar for tanahiro2010 tanahiro2010
2
260
Unity6 URPによる海底をテーマにしたVJシステムの解説 | Unity Shader 完全に理解した 勉強会
Avatar for がむ gam0022
0
370
AIで開発が速くなった先で、開発チームは何を目指すか
Avatar for Yuki Shiho shihochan
0
1k
パズルゲームの作り方 / how to make puzzle games
Avatar for kaityo256 kaityo256
PRO
2
290
現場と育てるRailsモデリング 〜Spreadsheetから始めたモデル設計〜
Avatar for siroemk siroemk
2
590
半永久的に提供し続けられるプライベートクラウドを目指して ― 利用者の認知負荷を抑えるAPI抽象化とハードウェア世代交代の基盤設計
Avatar for Tomofumi Kondo tomokon
0
500
When benchmarks go bad - what I learned from measuring performance wrong
Avatar for Holly Cummins hollycummins
0
220
Agentic Software Factoryに、すごく賢いIF文を。 / super smart IF statement into the Agentic Software Factory.
Avatar for r-kagaya rkaga
8
2.2k
ActionController meets Protocol Buffers
Avatar for Sorah Fukumori sorah
0
520

Featured

See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
3.1k
Money Talks: Using Revenue to Get Sh*t Done
Avatar for Nikki Halliwell nikkihalliwell
0
520
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
10k
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
22k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
3.2k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8.4k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
35
2.9k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]