Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to "Spark Connect" and Implementat...

Introduction to "Spark Connect" and Implementation of Managed Spark as a Service

「Spark Connect」プロトコルについて紹介し、Spark Connectを活用した社内管理型Spark as a Serviceの実装アーキテクチャおよび使用例を共有します。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1. $IBMMFOHFTPGUSBEJUJPOBM4QBSL Single Point of Failure - Limited Fault Tolerance -

    Resource Bottlenecks Network Dependency - Network partitions or instability disrupts execution - Clients Need Direct Access - Scheduling Restrictions Difficult Multi-Tenancy - Cluster endpoint must be protected 5JHIUDPVQMJOHPG$MJFOUBOE%SJWFS
  2. 4QBSLWT4QBSL$POOFDU TQBSL TQBSLDPOOFDU DMJFOUMPDBUJPO JODMVTUFS PVUDMVTUFS DMJFOUSFRVJSFNFOU GVMMTQBSL DMVTUFSBDDFTTBOE +7.CBTFE

    UIJODMJFOUNJOJNBMH31$ DMJFOUXJUI%BUB'SBNF -JCSBSZ OFUXPSLUPQPMPHZ NVTUIBWFEJSFDUOFUXPSL BDDFTTUPUIFDMVTUFS SFNPUFBDDFTTWJBH31$ TDBMBCJMJUZ ESJWFS DMJFOUIBOEMFTBMM TDIFEVMJOHBOEDPPSEJOBUJPO ESJWFSSVOTPODMVTUFSDMJFOUT BSFTUBUFMFTT GBVMUUPMFSBODF DMJFOUDSBTIDBVTFTKPCMPTU DMJFOUDBOEJTDPOOFDUBOE SFDPOOFDU
  3. 4QBSLWT4QBSL$POOFDU DPOU TQBSL TQBSLDPOOFDU MBOHVBHF MJNJUFEUPTQBSLTVQQPSUFE MBOHVBHFT+7. 1Z4QBSL 3 BOZ

    FY KBWBTDSJQU HP D SVTU DMPVE OBUJWF UJHIUDPVQMJOHBOEESJWFS SFRVJSFNFOUTMJNJUGMFYJCJMJUZ EFTJHOFEGPSDMPVEOBUJWF  DPOUBJOFSJ[FEBOETFSWFSMFTT TFDVSJUZ FYQPTJOHDMVTUFSFOEQPJOUTDBO SBJTFTFDVSJUZDPODFSOT DBOTFDVSFH31$ DIBOOFMXJUI DFOUSBMJ[FEDPOUSPM WFSTJPOJOH DMJFOUBOETFSWFSNVTUCFUJHIUMZ WFSTJPONBUDIFE DMJFOUBOETFSWFSDBOFWPMWF JOEFQFOEFOUMZ
  4.  1SJWBUF'JFMETPG%BUB'SBNF $PMVNO 4QBSL4FTTJPO FUD  &YFDVUJPO"1*T3%%  &OWJSPONFOU.BOJQVMBUJPO"1*T4QBSL$POUFYU 4QBSL$POGJHVSBUJPO

    6OTVQQPSUFE"1*T IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNMIPXTQBSLDPOOFDUDMJFOUBQQMJDBUJPOTEJGGFSGSPNDMBTTJD TQBSLBQQMJDBUJPO
  5.  &YUFOTJCJMJUZ  4QBSLBTB4FSWJDFJO$MPVE&OWJSPONFOU  .VMUJMBOHVBHF$MJFOUT3VTU (P +BWBTDSJQU FUD 

    *OUFSBDUJWJUZ  /PUFCPPLT+VQZUFS 74$PEF  *OUFHSBUFE%FWFMPQNFOU&OWJSPONFOU  .JOJNBMJUZ  8FC6*PS#*5PPMT  .PCJMFPS.JDSPTFSWJDFT  .$1TFSWFST 6TF$BTFT
  6.  &YUFOTJCJMJUZ  4QBSLBTB4FSWJDFJO$MPVE&OWJSPONFOU  .VMUJMBOHVBHF$MJFOUT3VTU (P +BWBTDSJQU FUD 

    *OUFSBDUJWJUZ  /PUFCPPLT+VQZUFS 74$PEF  *OUFHSBUFE%FWFMPQNFOU&OWJSPONFOU  .JOJNBMJUZ  8FC6*PS#*5PPMT  .PCJMFPS.JDSPTFSWJDFT  .$1TFSWFST 6TF$BTFT
  7.  *OUFHSBUFXJUIJOIPVTFQMBUGPSNT  "DDFTT$POUSPM  $VTUPNJ[FTQBSLDMJFOUQFSVTFSTFTTJPO  QZUIPO3TDBMBMJCSBSJFT  DMJFOUESJWFSTQFDT$16

    .FNPSZ (16 FUD  $VTUPNJ[F4QBSLJUTFMG  JOIPVTFQBUDIFT  GJOFUVOFTQBSLQSPQFSUJFT  SFNPUFTIVGGMFTFSWJDFT 3FRVJSFNFOUT
  8. 4QBSL$POOFDUWT"MUFSOBUJWFT TQBSLDPOOFDU MJWZ LZVVCJ BCTUSBDUJPO MPX NFEJVN IJHI QSPUPDPM H31$

    3&45 +%#$0%#$ MBOHVBHF BOZ QZUIPO TDBMB KBWB  3 42- TQBSLWFSTJPO  Yr Y TUBOEBMPOF Yr Y JOUFHSBUFE
  9. 4QBSL.VMUJUFOBODZ TQBSLDPOOFDU MJWZ LZVVCJ CVJMUJO ❌ /P ⚠ 1BSUJBM ✅

    :FT 4FTTJPO *TPMBUJPO ✅ :FT ✅ :FT ✅ :FT 4FTTJPO .BOBHFNFOU ❌ /P ⚠ #BTJD ✅ 'VMM 6TFS "VUIFOUJDBUJPO ⚠ -JNJUFE ⚠ 1MVHHBCMF ✅ 4USPOH 3FTPVSDF 2VPUB ❌ /P ❌ /P ✅ :FT
  10. 4QBSL&YUFOTJCJMJUZ TQBSLDPOOFDU MJWZ LZVVCJ -BOHVBHF ✅ "OZ ⚠ +7. 1ZUIPO

    3  42- ❌ 42- $VTUPN4QBSL ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN 4FTTJPO ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN6TFS "VUIFOUJDBUJPO ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN 3FTPVSDF2VPUB ✅ :FT ⚠ -JNJUFE ❌ /P
  11.  4FSWF4QBSL%SJWFSJOTUBODFXJUI4QBSL$POOFDU"EEPO  4VQQPSU4QBSL$SFEFOUJBMTLFSCFSPT LFZUBC  4VQQPSUSFTPVSDFRVPUBT  4VQQPSUSPMFNBOBHFNFOU 

    4VQQPSUBVEJUMPHHJOH  )PSJ[POUBMMZ4DBMBCMF è,VCFSOFUFTXJUINVMUJQMFOBNFTQBDFT &OWPZ"%4 $%4 &%4 3FRVJSFNFOU3VOUJNF
  12. *%&

  13. .$1