The Evolution of Metadata: LinkedIn's Journey [Strata NYC 2019]

The speakers examine different metadata strategies for modeling metadata, storing metadata, and then scaling the acquisition and refinement of metadata for thousands of metadata authors and producing systems. They dive into the pros and cons of each strategy and in which scenarios they think organizations should deploy them. They explore strategies including generic types versus specific types, crawling versus publish/subscribe, single source of truth versus multiple federated sources of truth, automated classification of data, lineage propagation, and more.

Transcript

1. The Evolution of Metadata:
   LinkedIn’s Journey
   Sept 25, 2019
   Shirshanka Das, Principal Staff Engineer, LinkedIn
   @shirshanka
   

   View Slide

2. LinkedIn’s Data Ecosystem
   

   View Slide

3. What is Metadata?
   What datasets do we have in our data warehouse (Hadoop, Teradata)
   How do we easily find them
   What does their schema look like
   What datasets derive from these datasets
   "Confused" by Sara Beyer (CC BY-NC-ND 2.0)
   ? ?
   

   View Slide

4. Attempt 1: Crawl Phase
   

   View Slide

5. Attempt 1: Crawl Phase
   Crawl all catalogs you can
   Parse all the logs you can
   ETL into an opinionated data model
   Build a search index + a lookup store
   Build an app to serve this info
   This is a useful product!
   We gave it a clever name: WhereHows
   And we open sourced it in 2016
   

   View Slide

6. Attempt 1: A few things we observed
   Pull-based integrations
   Central team’s burden
   Pipeline fragility, freshness
   Rigidity of model
   Hard to iterate quickly
   "Grimaces" by Fouquier (CC BY-NC-ND 2.0)
   

   View Slide

7. What is Metadata?
   What datasets do we have in our entire data
   ecosystem?
   Espresso, Kafka, Hadoop, Teradata, Search, Pinot …
   20+ data systems
   What does their schema look like?
   What datasets derive from these datasets?
   What business types are contained in these schemas?
   Who owns these datasets?
   Where are datasets being copied?
   …
   "[Katsiaryna Lenets] © 123RF.com"
   

   View Slide

8. Attempt 2: Walk Phase
   

   View Slide

9. Attempt 2: Walk Phase
   Separate REST-ful service to support this diversity
   Dataset Naming
   Generic Schema model for all kinds of data stores
   and formats
   Scalable integration patterns
   Pub-Sub using Kafka
   Support REST + Kafka ingest route
   We called it TMS: THE Metadata Store :)
   We couldn’t open source it, too coupled with our internal business concepts :(
   New extensions to the model
   Business metadata
   Ownership
   

   View Slide

10. Attempt 2: A few things we observed
    The Good
    Teams were now accountable for the quality of their metadata and their custom ETL
    Aggregate base metadata from all data platforms, overlay new metadata “aspects” on top easily
    The Not So Good
    New kind of metadata needed —> get in line behind central TMS team
    Source of truth versus “Reflection of truth” debates —> no one wants to take a dependency
    Standardized Event as interface: requires data model adapters everywhere, low incentive for producer to
    excel at their job
    

    View Slide

11. What is Metadata?
    "Thinking" by Elvin (CC BY-NC 2.0)
    

    View Slide

12. What is Metadata?
    "Thinking" by Elvin (CC BY-NC 2.0)
    

    View Slide

13. What is Metadata?
    "Thinking" by Elvin (CC BY-NC 2.0)
    Models,
    Features, …
    

    View Slide

14. Some observations about the problem
    There is value in local sub-graphs, but
    the real value lies in the global model
    Cannot execute this with a single monolith
    data model + service
    Different teams care about different sub-
    graphs
    Central team bottleneck
    Micro-services can help?
    Back to silo-ed metadata problem
    

    View Slide

15. Attempt 3: Run Together
    Distributed but collaborative authorship of model
    Single ORM-like layer to auto-generate integrations for
    CRUD operations
    Search queries
    Graph traversal
    Distributed deployment and ownership of services possible
    We’re calling this, the Generalized Metadata Architecture (GMA)
    

    View Slide

16. An Example Metadata Graph
    DATASET
    urn
    platform
    name
    fabric
    USER
    Urn
    firstName
    lastName
    PROFILE
    {
    firstName: John
    lastName: Doe
    Ldap: jdoe
    }
    GROUP
    urn
    name
    size
    OWNERSHIP
    {
    owners: [{
    type: SRE,
    user: jdoe
    }, …]
    }
    MEMBERSHIP
    {
    admin: jdoe,
    members: [{
    jdoe,
    …]
    }
    Owned By
    Has Admin
    Has Member
    1
    N
    1
    1
    1
    N
    Aspects
    Relationships
    Entity Anchors …
    SCHEMA
    {
    fields: [{
    type: integer,
    name: id
    }, …]
    }
    

    View Slide

17. Metadata Serving
    API Endpoints
    CRUD DAO
    Search DAO
    Graph DAO
    Graph DB
    Search Index
    Document Store
    Metadata Service
    

    View Slide

18. Metadata Indexing
    Metadata Service
    API Endpoints
    CRUD DAO
    Search Index
    Graph DB
    Search
    processor
    Graph
    processor
    Metadata
    Audit
    Event
    

    View Slide

19. Putting it all together
    GIT Web App Service
    CRUD Event
    Event Processor
    User, Groups
    Service
    Dataset Service
    SoT DB SoT DB
    Change Event
    Event Processor
    Change
    Event
    Hadoop
    Data Lake
    Analytics
    +
    Relevance
    Search
    

    View Slide

20. Design Decision Cheat-Sheet
    Generic versus Specific Types:
    Support strong-types layered over generic storage, model-first development
    Expose strongly-typed REST API + Graph APIs for metadata traversal
    Integration Strategy: Crawling versus Pub-Sub versus REST-ful API:
    Prefer unified “Pub-Sub + REST-ful” API, build crawlers that publish
    Single Source of Truth versus Replicated versus Federated:
    For new metadata systems, integrate as ORM layer
    For existing metadata systems, use Kafka to adapt
    Federation strategy only supports lookup cases, hard to support search and graph well
    

    View Slide

21. What about X?
    A brief survey of other similar systems in this space
    Hive Metastore: limited to Hadoop Dataset, focused on query planning
    Atlas: generic model, missing strong-typing on top
    Marquez: strongly opinionated model, focused on data pipeline metadata
    Ground: generic model, missing strong-typing on top, research prototype
    Amundsen: Web-app with a frontend service, write through to Hive Metastore
    M
    onolithic
    Services
    

    View Slide

22. Metadata Platform by the numbers
    Datasets : ~ 5M
    Dashboards: ~2K
    Features: ~3K
    Metrics: ~30K
    Schemas: ~100K
    People: 10K+
    CRUD Events: ~2M / day
    Relationship Events: ~3M / day
    

    View Slide

23. Powered by Metadata
    Metadata Platform
    ???
    

    View Slide

24. A Web App: Data Hub
    Search and Discover Data Constructs
    Find relationships, lineage, data quality, …
    Enrich metadata
    Some interesting parallels to the metadata platform in terms of extensibility
    How do we add new pages to this app in a multi-team environment?
    

    View Slide

25. Data Hub: Search
    

    View Slide

26. Data Hub: Browse
    

    View Slide

27. Data Hub: a Dataset’s page
    

    View Slide

28. Data Hub: Lineage
    

    View Slide

29. Data Hub: a Metric’s page
    

    View Slide

30. Data Management with Compliance
    Metadata Platform
    Data Access Layer
    Data
    Management
    (Purge, Export,
    …)
    Std Frameworks
    Operations
    Application Code
    Physical Data
    

    View Slide

31. Data Pipeline Operations
    

    View Slide

32. Powered by Metadata
    Metadata Platform
    Search and
    Discovery
    beyond just
    Datasets
    Data Management,
    Access
    with Compliance
    Operational
    Monitoring,
    Incremental Compute
    AI : Model, Feature
    reproducibility,
    explainability
    … and we’re just
    getting started
    

    View Slide

33. It’s open source!
    

    View Slide

34. Open Source : the details!
    Alpha release out currently at https://lnkd.in/datahub-alpha
    Check it out at: Github project wherehows, branch: datahub (https://lnkd.in/datahub-github)
    Capabilities:
    Generic Modeling Layer with CRUD on MySQL
    Search on Elastic
    Graph on Neo4j*
    Interfaces:
    The DataHub Web App
    REST-ful service from model files
    Event Processors for metadata events
    Metadata Model:
    Datasets
    People
    Integrations
    Data Catalogs (Hive, Kafka, JDBC*)
    People (LDAP)
    * coming real soon
    

    View Slide

35. Open sourcing the Data Model
    LinkedIn Internal Model Open Source Model
    

    View Slide

36. Open Source : what’s coming next!
    More Entities: [Jobs, Flows]
    More Aspects within existing entities: [e.g. ReplicationPolicy, HiveSpecification]
    More Integrations: [Calcite-compatible systems for fine-grain lineage]
    App Features
    User pages
    Social interactions
    Get engaged and
    contribute to the global model
    Build integrations with systems
    

    View Slide

37. Thank You!
    

    View Slide