Data Pipelines As Software Structures

Thesis: Data Pipelines emerge and grow to reflect collaboration between domains and are impeded by incidental coordination.

Transcript

1. Data Pipelines as Software Structures Berlin Buzzwords 2017 @brapse
3. Data Pipelines The software structures which emerge to process and disseminate information. A connected set of map reduce jobs for loading data into (a) database(s).
5. Why Data Pipelines? To integrate diverse perspectives. Enable and empower collaboration between diverse sets of domain experts.
6. How do we build them? Often Badly. Misunderstood domain. Misunderstood integration. Misunderstood coordination.
7. Thesis: Data Pipelines emerge and grow to reflect collaboration between domains and are impeded by incidental coordination.
8. Abstract
9. Evolution
10. Bounded Context
11. Storage
12. Teams
13. Nested splits Teams > Contexts > Storage
14. Mapping
15. Mapping Storage Boundaries
16. Mapping Context Boundaries
17. Mapping Team Boundaries
18. Coordination
19. Coordinating Definitions
20. Coordinating Correctness
21. Coordinating Failure
22. Concrete
24. Correctness
25. Fingerprints
26. Fingerprints
27. Fingerprints
30. Coordinate Change
31. Coordinate Change
32. Coordinate Change
33. Convergence
34. Failure
35. Retroactivity
36. Mutation
37. Mutation
38. Immutability
40. Conclusions Data pipelines enable coordination but require shared protocols to determine when and how we read and write data.
41. Emily Green Omid Aladini S e b a s t i a n O h m F ro n x Wurmus Matthias Georgi Thank You David Whiting Lorand Kasler Gavin Bell Jon Glover Erik Bartels