Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
130
CSC509 Lecture 08
javiergs
PRO
0
100
Webの技術スタックで マルチプラットフォームアプリ開発を可能にするElixirDesktopの紹介
thehaigo
2
910
／←このスケジュール表に立ち向かう フロントエンド開発戦略 / A front-end development strategy to tackle a single-slash schedule.
nrslib
1
590
カスタムしながら理解するGraphQL Connection
yanagii
1
1.2k
CSC305 Lecture 13
javiergs
PRO
0
130
RailsのPull requestsのレビューの時に私が考えていること
yahonda
5
1.7k
讓數據說話：用 Python、Prometheus 和 Grafana 講故事
eddie
0
350
CPython 인터프리터 구조 파헤치기 - PyCon Korea 24
kennethanceyer
0
240
PHP でアセンブリ言語のように書く技術
memory1994
PRO
1
150
Pinia Colada が実現するスマートな非同期処理
naokihaba
2
150
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
200

Featured

See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
664
120k
Building Applications with DynamoDB
mza
90
6.1k
GraphQLとの向き合い方2022年版
quramy
43
13k
Designing the Hi-DPI Web
ddemaree
280
34k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Producing Creativity
orderedlist
PRO
341
39k
Building Better People: How to give real-time feedback that sticks.
wjessup
363
19k
Documentation Writing (for coders)
carmenintech
65
4.4k
Agile that works and the tools we love
rasmusluckow
327
21k
The Invisible Side of Design
smashingmag
297
50k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Become a Pro
speakerdeck
PRO
24
5k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]