Data & AI Day 2025 - You Created a Pipeline, Now What?

Transcript

1. You Created a Data Pipeline Now What? Data + AI Day 2025-10-25
2. Karn Wong Loves optimization Has too much fun cranking out benchmarks HashiCorp Ambassador & AWS Community Builder Blog & Portfolio karnwong.me Say hi at Bluesky @karnwong.me Independent Consultant
3. Data Pipeline Anatomy Source Transformation Output
4. Source Data Owner Access Location Updates Format / Schema
5. Data Transformation Author Read / Write Access Deployment Updates Active Revision in Production Reproducibility
6. Output Data Destination Access Devs Systems Users Versioning Development vs Production
7. Key Themes Role-Based Access Control (RBAC) Versioning Reproducibility
8. RBAC Assign permissions to groups Attach users to groups Easier to setup and maintain Can assign ad-hoc & special permissions on a case-by-case basis
9. Monitoring & Observability Logs Visibility & Access Pipeline Failures When Time to Resolution Business Impact
10. Monitoring & Observability Service-Level Agreement Pipeline Runtimes Disaster Recovery Server Crashes Resources Consumption Future Scaling
11. Schema Evolution Lead Time Data Team in the Loop? Rollout Strategy Deprecation To Rename or Not to Rename
12. Data Requests Time to Complete Slow Completion Time Shadow Data Team Silos Governance: #@%#$%!@#!!!
13. Data Mesh & Data Stack Migration Reach out to me during networking
14. Thank you 🙏 Download slides at: karnwong.me