Mechanics of Data Pipelines

This talk focused the topic on how to model data pipelines as retroactive, immutable data structures. Essentially it covers the topic of how do you build a data pipelines for a growing organization where different teams depend on each others data and need to be able to re-process data when errors occur upstream.

Transcript

1. Who Am I Data Science & Engineering * Machine Learning * Distributed Systems * Computation Art
2. Mechanics of Data Pipelines
3. Outline * Problem motivation @ SC * Proposed properties * Mechanics * Open problems
4. SoundCloud * ~300 employees * Most working with data * Product driven * Microservices
5. Organisations which design systems ... are constrained to produce designs which are copies of the communication structures of these organisations — M. Conway Conway's law
8. Counting service * 10^6 reads/sec * 10^4 writes/sec * Maintains counts for all time
9. Challenges * Counts are subject to spam * Deleting data is painful * Can’t do full lambda
11. Royalties Reporting * Complex calculation * Interdisciplinary effort * Low frequency
12. Challenges * Wide dependencies * Multiplicative error * Reports require auditing
14. Internal Analytics * AB Testing * Ad hoc queries * Dashboards
15. Challenges * Lots of small ETL * Trans-datastore * Deep dependencies
16. TODO It’s a mess Emergent Data Pipelines * Different runtimes * Untouchable Legacy * Different processing steps * Cross dependencies
17. Failure * Transient Failure * Input set variance * Hard to test new code * Misunderstood dependency
18. TODO It’s a mess Consequence * Blocking Failures * Risk Aversion * Manual Intervention * Low coherence
19. Desired * Minimal human intervention * High coherence * Risk Tolerant * Incremental
21. Convergent * Incremental improvement * Automatic propagation * Low human intervention
22. Retroactive * Incremental re-computation * Time independence * Map input -> output
23. Persistent * Keep old computations * Comparable version * Don’t fear change
24. Entities
25. Datasource * Lambda architecture * Low complexity repairs * Inefficient
26. Partition * Date/time buckets * Time dependencies * Moderate efficiency
27. Chunk * Individual files * High incrementality * High complexity
28. Entities
29. Entities
30. Entities
31. HDFS
32. Redshift
33. Cassandra
34. Persistent
35. Retroactive
36. Convergent
37. Open Problems * Acceptance Testing * Materialising views * Data Discovery
38. Acceptance Testing * Interface focused * Risk minimisation * Continuous delivery
39. Materialised Views * Ephemeral representation * Non durable * Asynchronous
40. Data Discovery * Consumers/Producers * Service which coordinates jobs * Movable data sets
42. * Conways law is real * Design as data structure * Abstract and apply Conclusion
43. Emily Green Omid Aladini S e b a s t i a n O h m F ro n x Wurmus Matthias Georgi Thank You David Whiting Lorand Kasler Gavin Bell Jon Glover Erik Bartels