Data Contracts & Domain Ownership

What is Domain Ownership? Who owns the Data Quality? The presentation demystifies the data producer and data consumer relationship

Transcript

1. Data Contracts & Domain Ownership - Demystify Producer-Consumer(s) Relationship Ananth Packkildurai
2. Slack Data Engineer Zendesk Principal Data Engineer Creator Schemata - Data Contract Platform Author Data Engineering Weekly
3. What is Data Contract?
4. A data contract is a collaborative agreement between the people who create data (producers) and the people who use data (consumers). Data Contract defines the structure and behavior of the data, so that it can be exchanged seamlessly between different systems. What is Data Contract?
5. Data contracts are typically written in a machine-readable format, such as Protobuf, Avro, YAML or JSON. This makes it easy for computers to understand the structure of the data and how it can be used. What is Data Contract?
6. What is Data Ownership?
7. Who Own Your House?
8. House is a social Property
9. What is this has to do with Data Contract & Data Quality?
10. Data is Inherently Social in Nature
11. Typical Data Model
12. Producer - Consumer(s)
13. Data Practitioners
14. Who Owns the Data Quality?
15. Is there any better framework available than ownership?
16. RACI Matrix R (Responsible) someone who is responsible for and is the executor of a particular process C (Consulted) a person who consults and provides necessary data to implement the process A (Accountable or Approver) someone who is responsible for the result of the work I (Informed) a person who must be informed of the progress of the work
17. RACI Framework for ET (L) Data Creation Data Transformation 1 2
18. Data Creation Product Managers Developers Data Engineers Data Analyst/ Scientist/ ML Engineers Business Stakeholder/ Privacy & Governance R (Responsible) R (Responsible) C (Consulted) A (Accountable or Approver) I (Informed)
19. Data Transformation Product Managers Developers Data Engineers Data Analyst/ Scientist/ ML Engineers Business Stakeholder/ Privacy & Governance C (Consulted) C (Consulted) R (Responsible) & A (Accountable or Approver) I (Informed)
20. Data Quality always be defined from the consumer's perspective.
21. * The logos and trademarks displayed in this presentation are the property of their respective owners. The use of these logos and trademarks does not imply endorsement or sponsorship by the respective owners. Contract Testing Mock Servers API Specification Testing End-to-End Testing Acceptance Testing Consumer Driven Testing in Software Development
22. Data Quality is a Collaborative Workflow & Data Contract is the Enabler
23. https://schemata.app https://www.linkedin.com/in/ananthdurai ananth@dataengineeringweekly.com