Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
それはエンジニアリングの糧である：AI開発のためにAIのOSSを開発する現場より / It serves as fuel for engineering: insights from the field of developing open-source AI for AI development.
Avatar for nrs nrslib
1
820
Strategy for Finding a Problem for OSS: With Real Examples
Avatar for Chikahiro Tokoro kibitan
0
130
How to stabilize UI tests using XCTest
Avatar for Akio Itaya akkeylab
0
150
コードレビューをしない選択 #でぃーぷらすトウキョウ
Avatar for Takuma Kajikawa kajitack
3
1.2k
Xdebug と IDE による デバッグ実行の仕組みを見る / Exploring-How-Debugging-Works-with-Xdebug-and-an-IDE
Avatar for shin1x1 shin1x1
0
300
Symfonyの特性(設計思想)を手軽に活かす特性(trait)
Avatar for ICKX ickx
0
110
Claude Codeログ基盤の構築
Avatar for giginet giginet
PRO
7
3.8k
飯MCP
Avatar for Yusuke Wada yusukebe
0
450
今年もTECHSCOREブログを書き続けます！
Avatar for 平奥真一（ひらおくしんいち） hiraoku101
0
210
AIと共にエンジニアとPMの “二刀流”を実現する
Avatar for Naruo Yoshiki naruogram
0
120
S3ストレージクラスの「見える」「ある」「使える」は全部違う ─ 体験から見た、仕様の深淵を覗く
Avatar for ya_ma23 ya_ma23
0
1.2k
生成 AI 時代のスナップショットテストってやつを見せてあげますよ（α版）
Avatar for ojun ojun9
0
330

Featured

See All Featured
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
35k
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
6.2k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
1
140
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.2k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
3.1k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.8k
Designing for Timeless Needs
Avatar for Cassini Nazir cassininazir
0
180
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
790
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.6k
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
160
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
4
280

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]