Metadata Management with Semantic

データが企業の資産としてますます重要になる中、効果的なメタデータ管理が求められています。
本セッションでは、メタデータの価値を最大限に引き出す方法について、私たちが探究した結果を共有します。

具体的には、セマンティックデータモデルとグラフデータベースを活⽤し、データの可視性、関連性を高める手法を紹介します。

私たちのメタデータ管理の実践例を通じ、参加者の皆様がご自身のデータ管理戦略をより効果的にするためのヒントを得られることを期待しています。

Transcript

1. Metadata Management with Semantic LY Corporation Data Engineering Group Terouchi Daisuke
2. - The importance of metadata management - Metadata Management Challenges and Solutions - Semantic Data Models and Graph Database - Positive side effect - Future Outlook Agenda
3. The importance of metadata management
4. What is metadata? Data for Data All the information needed to make full use of our data
5. Why is metadata important? Without metadata, we cannot find, understand, manage, or utilize the data
6. What is metadata? Specific examples of metadata col_1 col_2 col_3 col_4 --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- Identifier Name Description Created At Updated At Lifecycle : t_0123456789 : shopping sales transaction : This is a sales transaction table for shopping. : 2025/6/30 14:03 : 2025/6/30 14:03 : enforced Table Service Company Organization People Lineage Policy : shopping : LY Corporation : Shopping Dept : Yamada Taro : upstream N/A, downstream table_a, table_b : shopping term
7. Metadata Management Challenges and Solutions
8. 3 Challenges in Metadata Management No Place No Method No Consistency
9. • No system exists for some metadata. • There may be no place to store certain information. • As a result, Excel or Confluence is often used. • This makes reuse difficult and leads to outdated information. No Place to Manage Metadata No Place
10. • Policies or restrictions are hard to represent as metadata. • Such information is usually managed as free text. • This makes it difficult to fully manage and use important rules. No Method to Manage Metadata No Method
11. • As we have seen, the desired metadata is not always well-organized. • As a result, searching for needed information takes time, and it is hard to find what you need easily. No Consistency to Manage Metadata No Consistency
12. Flexibility To adapt to constantly changing metadata management needs, implement a flexible system that allows you to add and build up information as requirements evolve. Support for Unstructured Information To manage information that is difficult to structure—such as policies—we need a data model that can capture and handle such unstructured metadata within the system. Solutions to Challenges in Metadata Management 2 Key Perspectives
13. Solutions to Challenges in Metadata Management Semantic Data Model Adoption From predefined metadata lists to a semantic data model Flexibility Support for Unstructured Information
14. Specific examples of metadata Traditional Expression col_1 col_2 col_3 col_4 --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- Identifier Name Description Created At Updated At Lifecycle Service Company Organization People Lineage Policy : t_0123456789 : shopping sales transaction : This is a sales transaction table for shopping. : 2025/6/30 14:03 : 2025/6/30 14:03 : enforced : shopping : LY Corporation : Shopping Dept : Yamada Taro : upstream N/A, downstream table_a, table_b : shopping term
15. Specific examples of metadata With Semantic Expression col_1 col_2 col_3 col_4 --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- Identifier Name Description Created At Updated At Lifecycle : t_0123456789 : shopping sales transaction : This is a sales transaction table for …. : 2025/6/30 14:03 : 2025/6/30 14:03 : enforced Service shopping Company LY Corporation Organization Shopping Dept People Yamada Taro Policy shopping term Dataset table_a Dataset table_b IS_OWNER_OF HAS_LINEAGE_TO PRODUCES HAS_POLICY
16. Semantic Data Models and Graph Database
17. What is a Semantic Data Model? Subject Predicate Object BELONGS_TO Semantic Data Model People Yamada Taro LY Corporation Company ORIGINATES_FROM =
18. Why Use a Graph Database? Data Visibility Subject Type subject relation Object Type object Dataset shopping_tbl HAS_LINAGE_TO Dataset table_a Dataset shopping_tbl HAS_LINAGE_TO Dataset table_b Company LY Corporation IS_OWNER_OF Dataset shopping_tbl Organization Shopping Dept IS_OWNER_OF Dataset shopping_tbl Service Shopping PRODUCES Dataset shopping_tbl Service Shopping HAS_POLICY Policy shopping term Tabular Graph vs
19. Why Use a Graph Database? Ease of Data Querying SQL GQL vs
20. Data Flow Overview Organization Data Data Platform Data BI Tools Data …etc Metadata Datalake Semantic data Graph DB ETL Copy Job : 90 Node : 6M Relation : 7M About 85% are column information. Data Ingestion Manually Original Metadata
21. Positive side effect
22. Unlocking Unexpected Value Explosive Growth in Usable Metadata DATA METADATA METADATA METADATA METADATA DATA METADATA METADATA METADATA METADATA
23. Unlocking Unexpected Value Explosive Growth in Usable Metadata DATA METADATA METADATA METADATA METADATA People Yamada Taro METADATA Organization METADATA Company ACCESSED Identifier Name Country : c_0000001 : LY Corporation : JP BELONGS_TO IS_CHILD_OF
24. Effortless Metadata Search High Compatibility with Generative AI How can I find that information? Leave it to me!
25. Effortless Metadata Search High Compatibility with Generative AI System Prompt Natural language question Generate GQL Get Metadata Answer Graph DB Input Data Model Demo
26. Future Outlook
27. Preparing for an AI-Driven Future Metadata in the Age of Generative AI Client Agent A2A A2A A2A METADATA User Discovery Agent Data Management Agent Analytics Agent Model Context Protocol