Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Hunting Vulnerabilities in Symfony with LLMs
Avatar for Vincent Amstoutz vinceamstoutz
0
550
ふつうのFeature Flag実践入門
Avatar for irof irof
8
4.1k
不変条件と整合性境界—ビジネスが決める設計判断と実現パターン / Invariants and Consistency Boundaries
Avatar for nrs nrslib
14
5.6k
C# and C++ Interoperability - cho-dotnetnew
Avatar for Akiko Kawai harukasao
0
290
「なぜそう決めたのか」を残し続ける仕組み ― Notion AI カスタムエージェント × Slack連携による設計判断の自動記録 - NIKKEI Tech Talk #47
Avatar for ニフティ株式会社 niftycorp
PRO
0
210
JJUG CCC 2026 Spring: JSpecify で実現する Kotlin フレンドリーな Java API 設計
Avatar for ternbusty ternbusty
1
180
その問い、本当に正しいですか？AI時代のエンジニアに必要な哲学と認知科学 / ai-philosophy-cognitive-science
Avatar for MinoDriven minodriven
11
5.9k
AI 輔助遺留系統現代化的經驗分享
Avatar for James Wang jame2408
1
840
Make SRE Operations Easier with Azure SRE Agent
Avatar for KAMEGAWA Kazushi kkamegawa
0
7.1k
並列実装の現場、2ヶ月間実務でAIを使い倒したAIもPCも私も限界が近い
Avatar for ming ming_ayami
0
130
dRuby over BLE
Avatar for makicamel makicamel
2
380
技術記事、AIに書かせるか、自分で書くか？ 〜それでも私が自分の手で書く理由〜 / #QiitaConference
Avatar for Junichi Ito jnchito
2
1.4k

Featured

See All Featured
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
Avatar for Tomoya Matsuura tomoyanonymous
2
860
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.5k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
52k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
420
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
46
8.2k
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.3k
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
3
440
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
440
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.8k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
35k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]