Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Iceberg Case Study in LY Corporation

Apache Iceberg Case Study in LY Corporation

LY Corporation have introduced Apache Iceberg into our data platform.
In this slide, I will show our use cases of Apache Iceberg and its challenges.

LINEヤフーでは、Apache Icebergをデータプラットフォームに導入しました。
このスライドでは、LINEヤフーにおけるApache Icebergのユースケースとその課題を紹介します。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1. ˜-:$PSQPSBUJPO 5BTVLV0,6%" 4FOJPS.BOBHFS 1SPEVDU4USBUFHZEFQU %BUB1JQFMJOF&OHJOFFSJOHEJWJTJPO -:$PSQPSBUJPO  *OUSPEVDUJPO  +PJO-*/&$PSQ

    BTOFXHSBE   %#"GPS-*/&(".&   )BEPPQ&5-KPCEFWFMPQNFOU   %BUB1MBUGPSN&OHJOFFS   )BEPPQ.JHSBJPO1SPK-FBE   *6%BUB$POOFDUUFBN&.1E.  /FX DPNQBOZ -:$PSQPSBUJPO  /PX 4FOJPS. %BUB&OHJOFFSJOH(SPVQ
  2. ˜-:$PSQPSBUJPO  .JTTJPO  $SFBUFBOBNB[JOHMJGFQMBUGPSN UIBUCSJOHT808UPPVSVTFST 'SPNXIFOZPVXBLFVQVOUJMZPVGBMMBTMFFQ BTUIF GPVOEBUJPOGPSZPVSEBJMZMJGF XFBJNUPTVSQSJTF

     JOTQJSFBOENBLFZPVSMJGFCFUUFS 8FIBWFTUFBEJMZFYQBOEFEPVSQSFTFODFJOUIFCVTJOFTTMBOETDBQFBOE UIFOVNCFSPGTFSWJDFTXFPGGFSPVSVTFST XIJDIJOUVSOIBTDBVTFEPVS JNQBDUPOPVSVTFST`MJWFTUPHSPX .PWJOHGPSXBSE XFXJMMLFFQDIBMMFOHJOHPVSTFMWFTUPVTFPVS UFDIOPMPHZBOETFSWJDFTUPDSFBUFOFXBOEVOFYQFDUFEWBMVFJOPVS VTFST`EBJMZMJWFT IUUQTXXXMZDPSQDPKQKBDPNQBOZNJTTJPO
  3. ˜-:$PSQPSBUJPO /VNFSJDBM0WFSWJFXPG-:$PSQPSBUJPO  /PPG6TFST  #VTJOFTT3FTVMUT  4FSWJDFT1SPWJEFEJO  /PPG&NQMPZFFT

     /PPG(SPVQ$PNQBOJFT   5IFTVNPG:BIPP+"1"/NPOUIMZMPHHFEJOVTFS*%T -*/&NPOUIMZBDUJWFVTFST ."6 1BZ1BZ SFHJTUFSFEVTFST BOE BOOVBMCVZFSTPG;0;0508/CVTJOFTT FYDMVEJOH:BIPP+"1"/4IPQQJOH "MMGJHVSFTBSFBTPG.BSDI    ': $VNVMBUJWFOPPGVTFSTPGUIF (SPVQ 0WFS NJMMJPO :BIPP+"1"/  NJMMJPO .POUIMZMPHHFEJOVTFST -*/&  NJMMJPO .POUIMZBDUJWFVTFST 3FWFOVF +1: USJMMJPO "EKVTUFE&#*5%" +1:  CJMMJPO $POTPMJEBUFE 0WFS   4UBOEBMPOF 0WFS     "TPG"VHVTU  "TPG.BSDI  "TPG.BSDI BQQSPY  DPVOUSJFTBOESFHJPOT 
  4. ˜-:$PSQPSBUJPO • .PUJWBUJPO • &WFSDIBOHJOHEBUBSFBMUFEMBXT • .BOZQSPEVDUTTFSWJDFT .BOZQSJWBDZQPMJDJFT • "DDJEFOUSFDPWFSZ

    • 8IBUXFXBOUUPBDIJFWF • 5PEFMFUFGFXSPXTDPMVNOTXJUISFBTPOBCMFPQFSBUJPOBMDPTU • "QQSPBDI3FMBUFEGFBUVSF • .FSHF0O3FBEX1PTJUJPOEFMFUF • 1BSBMMFMSFBECZ"$*% • *OQMBDFUBCMFFWPMVUJPO  6TF$BTF1BSUJBM%BUB%FMFUJPO
  5. ˜-:$PSQPSBUJPO "MMEBUBGJMFTTIPVMECFSFXSJUUFO  6TF$BTF1BSUJBM%BUB%FMFUJPO " # $ %  

      %&-&5&'30.ECUCM 8)&3&SPX@JE %BUB GJMF %BUB GJMF %BUB GJMF %BUB GJMF "-5&35"#-&ECUCM %301$0-6./$
  6. ˜-:$PSQPSBUJPO 5BCMFEBUB  6TF$BTF1BSUJBM%BUB%FMFUJPO " # $ %  

      %&-&5&'30.ECUCM 8)&3&SPX@JE %BUB GJMF %BUB GJMF %BUB GJMF %BUB GJMF .FUBEBUB GJMF %JGGFSFODF GJMF "-5&35"#-&ECUCM %301$0-6./$
  7. ˜-:$PSQPSBUJPO • .PUJWBUJPO •  IPVSEFMBZVOUJMTUSFBNJOHEBUBJTBWBJMBCMF • 4DBMBCJMJUZJTTVFGPSGVMMUFYUTFBSDIFOHJOF • 8IBUXFXBOUUPBDIJFWF

    • 5PNBLFPVSTUSFBNJOHEBUBBWBJMBCMFJOGFXNJOVUFT • "QQSPBDI3FMBUFEGFBUVSF • 4BGFUZ$PNQBDUJPOUIBOLTUP"$*% • 4NBMMGJMFCZFWFSZNJOVUF $PNQBDUJPO  6TF$BTF3FEVDFEBUBMBUFODZ
  8. ˜-:$PSQPSBUJPO • .PUJWBUJPO • 3BQJEVQEBUFCVTJOFTTSFRVJSFNFOU • *OJUJBMDPTUUPEFGJOFBUBCMFTDIFNB • 8IBUXFXBOUUPBDIJFWF •

    5PVQEBUFUBCMFTDIFNBFBTJMZXJUIPMEEBUBTFUDPNQBUJCJMJUZ • "QQSPBDI3FMBUFEGFBUVSF • 4DIFNBFWPMVUJPO  6TF$BTF4DIFNB'MFYJCJMJUZ
  9. ˜-:$PSQPSBUJPO • .PUJWBUJPO • 5FOUIPVTBOETQBSUJUJPOTJOBTJOHMFUBCMF • )JHIMPBECZCBERVFSJFTXJUIXSPOHQBSUJUJPODPOEJUJPO • 8IBUXFXBOUUPBDIJFWF •

    5PSFEVDFIJWFNFUBTUPSFMPBE • "QQSPBDI3FMBUFEGFBUVSF • .FUBGJMFTJO)%'4  6TF$BTF.FUBTUPSFTDBMBCJMJUZ
  10. ˜-:$PSQPSBUJPO • .PUJWBUJPO • *OUSPEVDF"QBDIF*DFCFSHCFOFGJUJOUPFYJTUJOHEBUBQMBUGPSN • .JOJNJ[FTJEFFGGFDUUPEBUBSFMBUFEXPSL • 1BJO1PJOUT •

    " MPUPG)JWFUBCMFT • "MPUPGDPNQPOFOU • "MPUPGTUBLFIPMEFST • "MPUPGBDDFTTQBUUFSO • 1MBO • *OQMBDFNJHSBUJPO • 1MBUGPSNMFWFM"QBDIF*DFCFSHTUBOEBSEJ[BUJPO • %JTU$Q BMUFSOBUJWF  $IBMMFOHF.JHSBUJPO
  11. ˜-:$PSQPSBUJPO • .PUJWBUJPO • ,FFQ)BEPPQDMVTUFSIFBMUIZ • .BLFQMBUGPSNFBTZUPVTFGPSFWFSZPOF • 1BJO1PJOUT •

    -FBSOJOHDVSWFGPS"QBDIF*DFCFSHNBJOUFOBODFQSPDFTT • 8IBUUBCMFSFRVJSFTBEEJUJPOBMNBJOUFOBODFPQFSBUPO • )PXUPSFEVDFUIFOVNCFSPGGJMFT • )PXUPQSVOFPVUEBUFEGJMFT • 1MBO • *DFCFSH5BCMF0QUJNJ[FS  $IBMMFOHF5BCMF.BJOUFOBODF
  12. ˜-:$PSQPSBUJPO • -:$PSQPSBUJPOOPXUSJFTUPJOUSPEVDF"QBDIF*DFCFSHJOUP FYJTUJOH&#TDBMFQMBUGPSN • "QBDIF*DFCFSHXJMMCSJOHVTBMPUPGCFOFGJUTMJLF • #VTJOFTTGMFYJCJMJUZ • 4ZTUFNTUBCJMJUZ

    • 4USBUFHJDTDBMBCJMJUZ • -:$PSQPSBUJPOLFFQDIBMMFOHJOHUPNPEFSOJ[FPVSEBUB QMBUGPSNDPOUJOVPVTMZ XIJDI"QBDIF*DFCFSHIFMQT 8IZBOE8IBU  4VNNBSZ