Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Deep dive into the select statement (GopherCon UK)
Avatar for Jesús Espino jespino
0
170
標準パッケージに uuid が追加された 背景から見る Go らしい意思決定 / go_127_uuid_decision
Avatar for convto convto
4
3.6k
Swift愛好会と私（ウホーイ） / Swift Fan Club and Uhooi
Avatar for uhooi uhooi
0
150
AI に Inclusive UI を書かせよう — Design Rules Skill で Compose UI を作り直す
Avatar for テオリア・テクノロジーズ theoriatec2024
1
460
What We Talk About When We Talk About XP
Avatar for seki at druby.org m_seki
2
440
AgentCore CLI で進化した AWS での AI エージェントの作り方 : 必要な機能を必要な時に
Avatar for Takahiro Kubo icoxfog417
PRO
2
220
thread_parallel_with_free-threaded_Python_and_NumPy.pdf
Avatar for Riku Sakamoto riku_sakamoto
0
200
Hono + Inertia + React で LP を構築した話
Avatar for Yuka Ooka oukayuka
2
220
DroidKaigi 2026 「個人開発という実験場: Android エンジニアが手にする4つの自由」
Avatar for SlashNephy slashnephy
0
220
Press start. Python's next generation.
Avatar for Carol Willing willingc
PRO
3
320
AIは賢い。でも実行環境は？ CLIおじさんがAI時代に伝えたいこと ~ CLIおじさんがAI時代に伝えたいこと ~
Avatar for curekoshimizu curekoshimizu
1
210
Hello, Hiroshima Geospatial Data! — Exploring DoboX with Python
Avatar for ぴっかりん ra0kley
0
180

Featured

See All Featured
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
310
Between Models and Reality
Avatar for mayunak mayunak
4
450
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.3k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.5k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
230
23k
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
2
560
We Are The Robots
Avatar for Honza Javorek honzajavorek
0
350
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
440
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
1
2.9k
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
480

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]