Data Contracts & Domain Ownership

What is Domain Ownership? Who owns the Data Quality? The presentation demystifies the data producer and data consumer relationship

Transcript

1. Data Contracts & Domain
   Ownership
   - Demystify Producer-Consumer(s) Relationship
   Ananth Packkildurai
   

   View Slide

2. Slack
   Data
   Engineer
   Zendesk
   Principal Data
   Engineer
   Creator
   Schemata -
   Data Contract
   Platform
   Author
   Data
   Engineering
   Weekly
   

   View Slide

3. What is Data Contract?
   

   View Slide

4. A data contract is a collaborative agreement
   between the people who create data (producers)
   and the people who use data (consumers). Data
   Contract defines the structure and behavior of the
   data, so that it can be exchanged seamlessly
   between different systems.
   What is Data Contract?
   

   View Slide

5. Data contracts are typically written in a
   machine-readable format, such as Protobuf, Avro,
   YAML or JSON. This makes it easy for computers to
   understand the structure of the data and how it
   can be used.
   What is Data Contract?
   

   View Slide

6. What is Data Ownership?
   

   View Slide

7. Who Own Your House?
   

   View Slide

8. House is a social Property
   

   View Slide

9. What is this has to
   do with Data
   Contract & Data
   Quality?
   

   View Slide

10. Data is Inherently Social in Nature
    

    View Slide

11. Typical Data Model
    

    View Slide

12. Producer - Consumer(s)
    

    View Slide

13. Data Practitioners
    

    View Slide

14. Who Owns the
    Data Quality?
    

    View Slide

15. Is there any better
    framework
    available than
    ownership?
    

    View Slide

16. RACI Matrix
    R (Responsible) someone who is responsible
    for and is the executor of a
    particular process
    C (Consulted) a person who consults and
    provides necessary data to
    implement the process
    A (Accountable or Approver) someone who is responsible
    for the result of the work
    I (Informed) a person who must be
    informed of the progress of
    the work
    

    View Slide

17. RACI Framework for ET (L)
    Data Creation
    Data Transformation
    1
    2
    

    View Slide

18. Data Creation
    Product
    Managers
    Developers Data
    Engineers
    Data Analyst/
    Scientist/ ML
    Engineers
    Business
    Stakeholder/
    Privacy &
    Governance
    R
    (Responsible)
    R
    (Responsible)
    C (Consulted) A
    (Accountable
    or Approver)
    I (Informed)
    

    View Slide

19. Data Transformation
    Product
    Managers
    Developers Data
    Engineers
    Data Analyst/
    Scientist/ ML
    Engineers
    Business
    Stakeholder/
    Privacy &
    Governance
    C (Consulted) C (Consulted) R
    (Responsible)
    &
    A
    (Accountable
    or Approver)
    I (Informed)
    

    View Slide

20. Data Quality always
    be defined from the
    consumer's
    perspective.
    

    View Slide

21. * The logos and trademarks displayed in this presentation are the property of their respective owners. The use of these logos and trademarks does not imply endorsement or sponsorship by the
    respective owners.
    Contract
    Testing
    Mock
    Servers API
    Specification
    Testing
    End-to-End
    Testing
    Acceptance
    Testing
    Consumer
    Driven
    Testing in
    Software
    Development
    

    View Slide

22. Data Quality is a
    Collaborative
    Workflow & Data
    Contract is the
    Enabler
    

    View Slide

23. https://schemata.app
    https://www.linkedin.com/in/ananthdurai
    ananth@dataengineeringweekly.com
    

    View Slide