Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AI巻き込み型コードレビューのススメ
Avatar for Nealle nealle
2
300
Architectural Extensions
Avatar for Denys Poltorak denyspoltorak
0
290
AIによるイベントストーミング図からのコード生成 / AI-powered code generation from Event Storming diagrams
Avatar for nrs nrslib
2
1.9k
コントリビューターによるDenoのすゝめ / Deno Recommendations by a Contributor
Avatar for petamoriken / 森建 petamoriken
0
200
Implementation Patterns
Avatar for Denys Poltorak denyspoltorak
0
290
AI によるインシデント初動調査の自動化を行う AI インシデントコマンダーを作った話
Avatar for azukiazusa azukiazusa1
1
730
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
680
なぜSQLはAIぽく見えるのか/why does SQL look AI like
Avatar for florets1 florets1
0
460
なるべく楽してバックエンドに型をつけたい！(楽とは言ってない)
Avatar for HIBIKI CUBE hibiki_cube
0
140
izumin5210のプロポーザルのネタ探し #tskaigi_msup
Avatar for izumin5210 izumin5210
1
130
Oxlintはいいぞ
Avatar for Yuji Yamaguchi yug1224
5
1.3k
Package Management Learnings from Homebrew
Avatar for Mike McQuaid mikemcquaid
0
220

Featured

See All Featured
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
5k
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
130
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
940
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
290
AI in Enterprises - Java and Open Source to the Rescue
Avatar for ivargrimstad ivargrimstad
0
1.1k
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
170
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
10
1.1k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
96
14k
Tell your own story through comics
Avatar for letsgokoyo letsgokoyo
1
810
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.8k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]