Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Vinitus Web: データプラットフォームの複雑性に立ち向かうデータ品質とリネージュのた...

Vinitus Web: データプラットフォームの複雑性に立ち向かうデータ品質とリネージュのためのジョブ管理システム / Ingestion Management by "Vinitus Web": Data Quality and Lineage for the complex data platform users

LINEとヤフーの合併後におけるデータ民主化とガバナンスを支援するためのbatch ingestionシステムの構築に関する洞察と、複雑なデータプラットフォームから得た経験を紹介します。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1. "CPVU6T Hoshino Kuntaro 🏠 Tokyo, Japan 👨💻 Product Manager @

    LY Corporation Joined as a new grad in 2020 Data Engineering > Data Pipeline Engineering > Batch Department 🎮 Sliver😩 @ League of Legends Macquang Huy 🏠 Hanoi, Vietnam 👨💻 Data Engineer @ LY Corporation Joined in 2022 LINE Technology Vietnam > Data Platform Dev Team 🎮 Master😎 @ League of Legends
  2. 0WFSWJFX -:%BUB1MBUGPSNBOE7JOJUVT8FC %BUB1MBUGPSN 4FSWJDF" 7JOJUVT 8FC 4FSWJDF# 4FSWJDF$ +PC0XOFS 0

    1 * * * 0 2 * * * 0 * * * * • $SFBUFTDIFEVMFEKPC • 4FFFYFDVUJPOIJTUPSZ • 3FTUBSUGBJMFEKPCT %BUB6TFST
  3. %BUB6TFST`DIBMMFOHFTJOPVS%BUB1MBUGPSN -:%BUB1MBUGPSNBOE7JOJUVT 8FC %BUB1MBUGPSN 4FSWJDF" 8IFSFUIJTEBUB DPNFTGSPN $BOXFVTFJU 🤔 %BUB6TFST

    6TFS"HSFFNFOU GPS4FSWJDF# 6TFS"HSFFNFOU GPS4FSWJDF" 4FSWJDF# 4FSWJDF$ 🇹🇭 🇯🇵 🇹🇼
  4. %BUB6TFST`DIBMMFOHFTJOPVS%BUB1MBUGPSN -:%BUB1MBUGPSNBOE7JOJUVT 8FC %BUB1MBUGPSN 4FSWJDF" )PXJTUIFEBUB JOHFTUFE 🤔 %BUB6TFST 4FSWJDF#

    4FSWJDF$ .Z42- .POHP%# CAST(`col` AS CHAR) $.name $.spec.color → `name` → `color` /dt=20250630 JST or UTC?
  5. %BUB6TFST`DIBMMFOHFTJOPVS%BUB1MBUGPSN %BUB6TFST 6TFS"HSFFNFOU 6TFS"HSFFNFOU or 🇹🇭 🇯🇵 🇹🇼 8IJDIPOFʜ .Z42-

    .POHP%# or EUTUPSFT d+45 0S d65$ Uasge Rules Data Structure
  6. )PX7JOJUVT 8FC)FMQT%BUB6TFST *OHFTUJPOKPCJOGPSNBUJPOGPSCFUUFS%BUB-JOFBHF %BUB1MBUGPSN 4FSWJDF" 4FSWJDF# 4FSWJDF$ 0 1 *

    * * 0 2 * * * 0 * * * * %BUB-JOFBHFJOGPSNBUJPO %BUB6TFST $PMVNOTBSF$"45FEMJLFʜ 8IFSFEBUBDPNFTGSPN 8IPJTUIFKPCPXOFS )PXJTUIFEBUBQSPDFTTFE
  7. )PX7JOJUVT8FCIFMQT+PC0XOFST 1SPWJEFBGVMMZNBOBHFE&5-XPSLGMPXFOHJOF +PCDSFBUJPOUJNF 4BWF  .BOVBMPQFSBUJPOT 3FNPWF  $VTUPNNFUSJDT 1SPWJEF

     +PCBT $POGJH 4DIFNB FWPMVUJPO 'BJMFEUBTLT #BDLGJMM 5BTLMFWFM NPOJUPSJOH 'SPNIPVSTà NJOT
  8. +PCBT$POGJH .BLJOH+PC$SFBUJPO&BTZ (FOFSBUF $BMMCBDL %"(QBSTJOH +PC0XOFSXPSLMPBE 7JOJUVTUBLFTDBSFPGUIFSFTU 7JOJUVT "JSGMPX +PCBT$POGJH

    ,FZ1PJOU 6TFSTDSFBUFBOJOHFTUJPOKPCUISPVHI6* 7JOJUVT 1SPWJEJOHBGVMMZNBOBHFE&5-XPSLGMPXFOHJOF /POFYQFSUVTFSTDBODSFBUFJOHFTUJPOKPCT JONJOVUFT 
  9. 8FQSPWJEFQSFQSPDFTTJOH UBTL🤩 à 'VMMZTVQQPSUBMMTDFOBSJPT %BJMZ0QFSBUJPOT  4DIFNB&WPMVUJPO %BUBTDIFNBTDIBOHFPWFSUJNF .PEJGZJOHEBUBUZQFT %FMFUJOHDPMVNOT

    "EEJOHDPMVNOT $IBOHJOHQBSUJUJPOT $3&"5&5&.1 5"#-& */4&35EBUB'30. 0SJHJOBM5BCMFUP 5FNQ5BCMF #BDLVQPSJHJOBM EBUBUP)%'4 %30103*(*/"- 5"#-& $3&"5&03*(*/"- 5"#-& OFXTDIFNB .PWFEBUB'30. 5FNQ5BCMFUP 0SJHJOBM5BCMF !#J.POUIMZ 3FNPWFCBDLVQEBUB !@)PVST %BUBJTEFMFUFEDPNQMFUFMZ %BUBJTNPWFEUP 5SBTIEJSFDUPSZ 7JOJUVT"JSGMPX )BEPPQ WJFXGTJVHPW\%#^*6#BDLVQ\5"#-&^\5*.&^