Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Catalogs - Rebuild the Broken Promise

Ananth Packkildurai
April 23, 2023
Technology
0
25

Data Catalogs - Rebuild the Broken Promise

Is Data Catalog still relevant? Why does most data catalog fail? What can we do to prevent the failures?

Ananth Packkildurai

April 23, 2023
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
27
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
340
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.1k
Murron: A Reliable Monitoring Pipeline
vananth22
0
320
The_journey_towards_Pinot.pdf
vananth22
0
190
Reliable_Event_Pipeline___scale.pdf
vananth22
0
100
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2k
Streaming data pipelines @ Slack
vananth22
2
1.9k
measuring api performance using druid
vananth22
0
1.5k

Other Decks in Technology

See All in Technology
Goで実装された高速な
仮想待合室サーバの実装と詳解
pyama86
13
5.4k
2023 CSS
nishiharatsubasa
3
2.8k
UIコンポーネントライブラリをうまく使うためにできること / components-with-designer
mottox2
2
1.1k
DMMプラットフォームに ゼロベースでSLO導入している取り組み 適切なSLI模索の軌跡
junjunjunk
2
440
変化する組織の中で運用するフロントエンドエコシステム / Frontend Ecosystem in Organizational Changes
i524
1
290
[PHPカンファレンス沖縄2023]【実践編】良いプロダクト作りのための組織育成 健全なコードは健全な組織、健全なチームから
ikezoemakoto
0
180
そのとき歴史は動かなかった 〜 JTCのウォーターフォール研究者がバックログを発明した日〜
bonotake
0
260
Traversing the GraphQL AST and Calculating Query Costs
joere
0
310
組織・プロセス・技術 - フロントエンドの生産性向上への複眼的アプローチ / 20230921
bengo4com
3
480
日経iOSプロジェクトのマルチモジュール戦略
shimastripe
2
250
レイヤードアーキテクチャにおける 例外との向き合い方
yukihiromori
0
1.2k
【新卒研修資料】基礎統計学 / Basic of statistics
brainpadpr
65
42k

Featured

See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
349
18k
Making the Leap to Tech Lead
cromwellryan
119
8.1k
4 Signs Your Business is Dying
shpigford
172
21k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
226
16k
The Invisible Side of Design
smashingmag
292
49k
The World Runs on Bad Software
bkeepers
PRO
59
6k
Designing on Purpose - Digital PM Summit 2013
jponch
108
6.1k
A designer walks into a library…
pauljervisheath
199
17k
The Art of Programming - Codeland 2020
erikaheidi
39
12k
Writing Fast Ruby
sferik
615
59k
Fontdeck: Realign not Redesign
paulrobertlloyd
74
4.6k
How to train your dragon (web standard)
notwaldorf
70
4.7k

Transcript

  1. Data Catalogs
    - Rebuild the Broken Promise
    Ananth Packkildurai

    View Slide

  2. Slack
    Data
    Engineer
    Zendesk
    Principal Data
    Engineer
    Creator
    Schemata -
    Data Contract
    Platform
    Author
    Data
    Engineering
    Weekly

    View Slide

  3. Data Catalog is an expensive
    data ingestion platform you
    never intend to build.

    View Slide

  4. How Happy are you with Data Catalog?

    View Slide

  5. What is the
    purpose of
    a Data
    Catalog?
    Data Lineage
    Data Governance
    Data Discovery
    Metadata Management
    Collaboration

    View Slide

  6. Data Engineering in a Nutshell

    View Slide

  7. Data Catalog - A Disjointed Space

    View Slide

  8. MAD Landscape - 2012

    View Slide

  9. Data Engineering Function 2012

    View Slide

  10. MAD Landscape 2023

    View Slide

  11. Don’t Strive for
    comprehensive &
    Complete
    Coverage

    View Slide

  12. Don’t Desire for
    Single Source of
    Metadata

    View Slide

  13. What should
    we start
    thinking about
    it?

    View Slide

  14. Streamline Data Creation
    Process

    View Slide

  15. Shift Left

    View Slide

  16. Data Contract as a Code

    View Slide

  17. Headless Data Catalog

    View Slide

  18. https://schemata.app
    https://www.linkedin.com/in/ananthdurai
    [email protected]

    View Slide