Vinitus Web: データプラットフォームの複雑性に立ち向かうデータ品質とリネージュのためのジョブ管理システム / Ingestion Management by "Vinitus Web": Data Quality and Lineage for the complex data platform users

LINEとヤフーの合併後におけるデータ民主化とガバナンスを支援するためのbatch ingestionシステムの構築に関する洞察と、複雑なデータプラットフォームから得た経験を紹介します。

Transcript

1. *OHFTUJPO
   .BOBHFNFOU
   CZ
   7JOJUVT
   8FC
    %BUB
   2VBMJUZ
   BOE
   -JOFBHF
   GPS
   UIF
   DPNQMFY
   EBUB
   QMBUGPSN
   VTFST -:
   $PSQPSBUJPO )PTIJOP
   ,VOUBSP .BDRVBOH )VZ
2. "CPVU
   6T Hoshino Kuntaro 🏠 Tokyo, Japan 👨💻 Product Manager @ LY Corporation Joined as a new grad in 2020 Data Engineering > Data Pipeline Engineering > Batch Department 🎮 Sliver😩 @ League of Legends Macquang Huy 🏠 Hanoi, Vietnam 👨💻 Data Engineer @ LY Corporation Joined in 2022 LINE Technology Vietnam > Data Platform Dev Team 🎮 Master😎 @ League of Legends
3. 0WFSWJFX -:
   %BUB
   1MBUGPSN
   BOE
   7JOJUVT
   8FC %BUB
   1MBUGPSN 4FSWJDF
   " 4FSWJDF
   # 4FSWJDF
   $ 🤔 %BUB
   6TFST “Ingestion” copying data to the data platform
4. 0WFSWJFX -:
   %BUB
   1MBUGPSN
   BOE
   7JOJUVT
   8FC %BUB
   1MBUGPSN 4FSWJDF
   " 7JOJUVT 8FC 4FSWJDF
   # 4FSWJDF
   $ +PC
   0XOFS 0 1 * * * 0 2 * * * 0 * * * * • $SFBUF
   TDIFEVMFE
   KPC • 4FF
   FYFDVUJPO
   IJTUPSZ • 3FTUBSU
   GBJMFE
   KPCT %BUB
   6TFST
5.  %BUB
   6TFST`
   DIBMMFOHFT
   JO
   PVS
   %BUB
   1MBUGPSN  )PX
   7JOJUVT
   8FC
   )FMQT
   %BUB
   6TFST  )PX
   7JOJUVT
   8FC
   )FMQT
   +PC
   0XOFST  3FDBQ
   
   /FYU "HFOEB
6.  %BUB
   6TFST`
   DIBMMFOHFT
   JO
   PVS
   %BUB
   1MBUGPSN • 5IF
   JODSFBTFE
   FGGPSU
   EVF
   UP
   EJWFSTF
   EBUB
   PSJHJOT  )PX
   7JOJUVT 8FC
   )FMQT
   %BUB
   6TFST  )PX
   7JOJUVT
   8FC
   )FMQT
   +PC
   0XOFST  3FDBQ
   
   /FYU "HFOEB
7. %BUB
   6TFST`
   DIBMMFOHFT
   JO
   PVS
   %BUB
   1MBUGPSN -:
   %BUB
   1MBUGPSN
   BOE
   7JOJUVT 8FC %BUB
   1MBUGPSN 4FSWJDF
   " 8IFSF
   UIJT
   EBUB
   DPNFT
   GSPN $BO
   XF
   VTF
   JU
   🤔 %BUB
   6TFST 6TFS
   "HSFFNFOU GPS
   4FSWJDF
   # 6TFS
   "HSFFNFOU GPS
   4FSWJDF
   " 4FSWJDF
   # 4FSWJDF
   $ 🇹🇭 🇯🇵 🇹🇼
8. %BUB
   6TFST`
   DIBMMFOHFT
   JO
   PVS
   %BUB
   1MBUGPSN -:
   %BUB
   1MBUGPSN
   BOE
   7JOJUVT 8FC %BUB
   1MBUGPSN 4FSWJDF
   " )PX
   JT
   UIF
   EBUB
   JOHFTUFE
   🤔 %BUB
   6TFST 4FSWJDF
   # 4FSWJDF
   $ .Z42- .POHP%# CAST(`col` AS CHAR) $.name $.spec.color → `name` → `color` /dt=20250630 JST or UTC?
9. %BUB
   6TFST`
   DIBMMFOHFT
   JO
   PVS
   %BUB
   1MBUGPSN %BUB
   6TFST 6TFS
   "HSFFNFOU 6TFS
   "HSFFNFOU or 🇹🇭 🇯🇵 🇹🇼 8IJDI
   POFʜ .Z42- .POHP%# or EU
   TUPSFT 
   
   d
   
   
   +45 0S 
   
   d
   
   
   65$ Uasge Rules Data Structure
10.  %BUB
    6TFST`
    DIBMMFOHFT
    JO
    PVS
    %BUB
    1MBUGPSN  )PX
    7JOJUVT
    8FC
    )FMQT
    %BUB
    6TFST  )PX
    7JOJUVT
    8FC
    )FMQT
    +PC
    0XOFST  3FDBQ
    
    /FYU "HFOEB
11.  &OBCMF
    VTFST
    UP
    RVJDLMZ
    GJOE
    UIF
    PSJHJO
    PG
    UIF
    EBUB • )FMQ
    GJOEJOH
    SVMFT
    BOE
    OFDFTTBSZ
    BQQSPWBMT  &OTVSF
    DPOTJTUFODZ
    JO
    EBUB
    TUSVDUVSF • /P
    OFFE
    UP
    XPSSZ
    BCPVU
    EBUBCBTF
    EJGGFSFODFT )PX
    7JOJUVT 8FC
    )FMQT
    %BUB
    6TFST Detailed Data Lineage information Sensible defaults in job settings
12. )PX
    7JOJUVT 8FC
    )FMQT
    %BUB
    6TFST *OHFTUJPO
    KPC
    JOGPSNBUJPO
    GPS
    CFUUFS
    %BUB
    -JOFBHF %BUB
    1MBUGPSN 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ 0 1 * * * 0 2 * * * 0 * * * * %BUB
    -JOFBHF
    JOGPSNBUJPO %BUB
    6TFST $PMVNOT
    BSF
    $"45FE
    MJLFʜ 8IFSF
    EBUB
    DPNFT
    GSPN 8IP
    JT
    UIF
    KPC
    PXOFS )PX
    JT
    UIF
    EBUB
    QSPDFTTFE
13. %BUB
    $POTJTUFODZ
    %BUB
    2VBMJUZ 
    4PVSDF
    $PMVNO
    BOE
    %FTUJOBUJPO
    $PMVNO
    JT
    UP 
    4FOTJCMF
    EFGBVMUT
    CBTFE
    PO
    UIF
    TPVSDF
    DPMVNO UP
    CF
    UIF
    BQQSPQSJBUF
    GPSNBU
14. /P
    
    "CTPMVUFMZ
    OPU
    🤩 %PFT
    UIJT
    NFBO
    NPSF
    XPSL
    GPS
    +PC
    0XOFS
    
15.  %BUB
    6TFST`
    DIBMMFOHFT
    JO
    PVS
    %BUB
    1MBUGPSN  )PX
    7JOJUVT
    8FC
    )FMQT
    %BUB
    6TFST  )PX
    7JOJUVT
    8FC
    )FMQT
    +PC
    0XOFST  3FDBQ
    
    /FYU "HFOEB
16. )PX
    7JOJUVT
    8FC
    IFMQT
    +PC
    0XOFST
    1SPWJEF
    B
    GVMMZ
    NBOBHFE
    &5-
    XPSLGMPX
    FOHJOF +PC
    DSFBUJPO
    UJNF 4BWF  .BOVBM
    PQFSBUJPOT 3FNPWF  $VTUPN
    NFUSJDT 1SPWJEF  +PCBT $POGJH 4DIFNB
    FWPMVUJPO 'BJMFE
    UBTLT #BDLGJMM 5BTLMFWFM
    NPOJUPSJOH 'SPN
    
    IPVST
    à 
    NJOT
17. +PCBT$POGJH .BLJOH
    +PC
    $SFBUJPO
    &BTZ (FOFSBUF $BMMCBDL %"(
    QBSTJOH +PC
    0XOFS
    XPSLMPBE 7JOJUVT
    UBLFT
    DBSF
    PG
    UIF
    SFTU 7JOJUVT
    "JSGMPX +PCBT$POGJH ,FZ
    1PJOU 6TFST
    DSFBUF
    BO
    JOHFTUJPO
    KPC
    UISPVHI
    6* 7JOJUVT 1SPWJEJOH
    B
    GVMMZ
    NBOBHFE
    &5-
    XPSLGMPX
    FOHJOF /POFYQFSU
    VTFST
    DBO
    DSFBUF
    JOHFTUJPO
    KPCT
    JO
    NJOVUFT
    
18. +PCBT$POGJH 8IBU`T
    *OTJEF UBTL@DPODVSSFODZ
     OVN1BSUJUJPOT
     QBSUJUJPO$PMVNO
    JE OVN1BSUJUJPOT
     QBSUJUJPO$PMVNO
    DPVOUSZ
19. 8F
    QSPWJEF
    QSFQSPDFTTJOH UBTL
    🤩 à 'VMMZ
    TVQQPSU
    BMM
    TDFOBSJPT %BJMZ
    0QFSBUJPOT 
    4DIFNB
    &WPMVUJPO %BUB
    TDIFNBT
    DIBOHF
    PWFS
    UJNF
     .PEJGZJOH
    EBUB
    UZQFT %FMFUJOH
    DPMVNOT "EEJOH
    DPMVNOT $IBOHJOH
    QBSUJUJPOT $3&"5&
    5&.1 5"#-& */4&35
    EBUB
    '30. 0SJHJOBM5BCMF
    UP 5FNQ5BCMF #BDLVQ
    PSJHJOBM
    EBUB
    UP
    )%'4 %301
    03*(*/"- 5"#-& $3&"5&
    03*(*/"- 5"#-& OFX
    TDIFNB .PWF
    EBUB
    '30.
    5FNQ5BCMF
    UP
    0SJHJOBM
    5BCMF !#J.POUIMZ 3FNPWF
    CBDLVQ
    EBUB !@)PVST %BUB
    JT
    EFMFUFE
    DPNQMFUFMZ %BUB
    JT
    NPWFE
    UP 5SBTI
    EJSFDUPSZ 7JOJUVT
    "JSGMPX )BEPPQ WJFXGTJVHPW\%#^*6#BDLVQ\5"#-&^\5*.&^
20. &SSPS DMBTTJGJDBUJPO
    🤩 #BTFE
    PO
    UIF
    MPH
    VTFST
    EFDJEF
    GVSUIFS
    BDUJPOT %BJMZ
    0QFSBUJPOT 
    )BOEMJOH
    'BJMFE
    5BTLT 5BTLT
    BSF
    GBJMFE
     +%#$FSSPS 7BMJEBUFFSSPS 'JYFE
    CZ
    NZTFMG ,TFSSPS 5BTLFSSPS 3BJTFE
    BO
    JORVJSZ
21. #BDLpMM
    GFBUVSF
    🤩 "
    IFBWZ
    PQFSBUPS
    GPS
    pMMJOH
    EBUB
    GPS
    QBTU
    QFSJPET )PX
    &BDI
    CBDLpMM
    SFRVFTU
    XJMM
    CF
    TFOU
    UP
    "JSqPX &BDI
    CBDLpMM
    SFRVFTU
    XJMM
    CF
    IBOEMFE
    JO
    EFEJDBUFE
    QPE %BJMZ
    0QFSBUJPOT 
    #BDLGJMM
    4VQQPSU 8BOU
    EBUB
    GPS
    MPOH
    QBTU
    QFSJPET
     HTTP requests Airflow Backfill
22. 5BTL
    MFWFM
    NPOJUPSJOH $VTUPN
    .FUSJDT &YQPSU
    MPHT &YFDVUF
     0SDIFTUSBUF &YQPTF
    NFUSJDT NFUSJDT $IFDLJOH MPHT "MFSU 5BTL
    3VO TUBUTE 7JOJUVT
    "JSGMPX
23. 5BTL
    MFWFM
    NPOJUPSJOH 8IBU
    :PV
    4FF General information Failed jobs information Identify long running jobs Error classification
24.  %BUB
    6TFST`
    DIBMMFOHFT
    JO
    PVS
    %BUB
    1MBUGPSN  )PX
    7JOJUVT
    8FC
    )FMQT
    %BUB
    6TFST  )PX
    7JOJUVT
    8FC
    )FMQT
    +PC
    0XOFST  3FDBQ
    
    /FYU "HFOEB
25. • 3FEVDF
    UIF
    DPTU
    EVSJOH
    EBUB
    VTF
    CZ
    QSPWJEJOH
    %BUB
    -JOFBHF
    BOE
    %BUB
    $POTJTUFODZ • 3FEVDF
    UIF
    CVSEFO
    PO
    UIF
    KPC
    PXOFS
    XIFO
    NBJOUBJOJOH
    EBUB
    JO
    BO
    BDDFTTJCMF
    TUBUF 3FDBQ
    
    /FYU
    8JUI
    7JOJUVT
    8FC • 4VQQPSU
    UP
    VOEFSTUBOE
    UIF
    %BUB
    -JOFBHF
    JOGPSNBUJPO • -BCFMT
    GPS
    EBUB
    "* • 3FEVDF
    KPC
    PXOFSbT
    CVSEFO • %BUB
    BT
    1SPEVDU
    %BUB
    4-0 8IBU`T
    OFYU