Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data & AI Day 2025 - You Created a Pipeline, No...
Search
Karn Wong
October 27, 2025
Technology
0
65
Data & AI Day 2025 - You Created a Pipeline, Now What?
Karn Wong
October 27, 2025
Tweet
Share
More Decks by Karn Wong
See All by Karn Wong
Pycon Thailand 2025 - ML Model Serving Optimization with ONNX
kahnwong
0
30
MFEC x Google Cloud Thailand: Betagro Bootcamp - IaC Adoption
kahnwong
0
28
{{Ops}Ver.se - Infrastructure as Code and Business Values
kahnwong
0
72
BKK.JS #23 - Intro to WASM
kahnwong
0
32
FossAsia 2025 - Take Control of Your Own Data via Self-Hosting Through Open Source Software
kahnwong
0
83
Technologista 2024 - Rust for Data - What Works and What Doesn't
kahnwong
0
180
HashiCorp User Group Thailand Meetup - Self-hosting Kubernetes at Home with Terraform
kahnwong
0
110
HashiCorp User Hub Thailand #2 - Simplify Proxmox VM Management with Terraform
kahnwong
0
100
Python Developer Day Thailand 2024 - How to Bootstrap a Python Project
kahnwong
0
170
Other Decks in Technology
See All in Technology
ヘンリー会社紹介資料(エンジニア向け) / company deck for engineer
henryofficial
0
380
Azureコストと向き合った、4年半のリアル / Four and a half years of dealing with Azure costs
aeonpeople
1
290
Okta Identity Governanceで実現する最小権限の原則 / Implementing the Principle of Least Privilege with Okta Identity Governance
tatsumin39
0
170
GraphRAG グラフDBを使ったLLM生成(自作漫画DBを用いた具体例を用いて)
seaturt1e
1
150
QA業務を変える(!?)AIを併用した不具合分析の実践
ma2ri
0
140
マルチエージェントのチームビルディング_2025-10-25
shinoyamada
0
160
AI時代の開発を加速する組織づくり - ブログでは書けなかったリアル
hiro8ma
1
310
ソースを読む時の思考プロセスの例-MkDocs
sat
PRO
1
170
OpenTelemetry が拡げる Gemini CLI の可観測性
phaya72
2
2.3k
CNCFの視点で捉えるPlatform Engineering - 最新動向と展望 / Platform Engineering from the CNCF Perspective
hhiroshell
0
140
初めてのDatabricks Apps開発
taka_aki
1
390
.NET 10のBlazorの期待の新機能
htkym
0
110
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
38
2.9k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
116
20k
Designing for humans not robots
tammielis
254
26k
Intergalactic Javascript Robots from Outer Space
tanoku
272
27k
VelocityConf: Rendering Performance Case Studies
addyosmani
333
24k
Writing Fast Ruby
sferik
630
62k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
21
1.2k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
508
140k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.7k
KATA
mclloyd
PRO
32
15k
Done Done
chrislema
185
16k
Transcript
You Created a Data Pipeline Now What? Data + AI
Day 2025-10-25
Karn Wong Loves optimization Has too much fun cranking out
benchmarks HashiCorp Ambassador & AWS Community Builder Blog & Portfolio karnwong.me Say hi at Bluesky @karnwong.me Independent Consultant
Data Pipeline Anatomy Source Transformation Output
Source Data Owner Access Location Updates Format / Schema
Data Transformation Author Read / Write Access Deployment Updates Active
Revision in Production Reproducibility
Output Data Destination Access Devs Systems Users Versioning Development vs
Production
Key Themes Role-Based Access Control (RBAC) Versioning Reproducibility
RBAC Assign permissions to groups Attach users to groups Easier
to setup and maintain Can assign ad-hoc & special permissions on a case-by-case basis
Monitoring & Observability Logs Visibility & Access Pipeline Failures When
Time to Resolution Business Impact
Monitoring & Observability Service-Level Agreement Pipeline Runtimes Disaster Recovery Server
Crashes Resources Consumption Future Scaling
Schema Evolution Lead Time Data Team in the Loop? Rollout
Strategy Deprecation To Rename or Not to Rename
Data Requests Time to Complete Slow Completion Time Shadow Data
Team Silos Governance: #@%#$%!@#!!!
Data Mesh & Data Stack Migration Reach out to me
during networking
Thank you 🙏 Download slides at: karnwong.me