Introduction to "Spark Connect" and Implementation of Managed Spark as a Service

*OUSPEVDUJPOUP4QBSL$POOFDUBOE *NQMFNFOUBUJPOPG.BOBHFE4QBSLBTB 4FSWJDF %BUB1MBUGPSN&OHJOFFSJOH .BSL-FF 이승 李承

"QBDIF4QBSL$POOFDU "QBDIF4QBSLBTB4FSWJDF %&.04QBSLBTB4FSWJDF %&.04QBSL$MJFOU 3FDBQ
"HFOEB

4QBSL$POOFDU

"QBDIF4QBSLä JTBNVMUJMBOHVBHFFOHJOFGPSFYFDVUJOHEBUBFOHJOFFSJOH EBUB TDJFODF BOENBDIJOFMFBSOJOHPOTJOHMFOPEFNBDIJOFTPSDMVTUFST ,FZ'FBUVSFT #BUDITUSFBNJOHEBUB 42-BOBMZUJDT
%BUBTDJFODFBUTDBMF .BDIJOFMFBSOJOH "QBDIF4QBSL IUUQTTQBSLBQBDIFPSH

4QBSL$MVTUFS IUUQTTQBSLBQBDIFPSHEPDTMBUFTUDMVTUFSPWFSWJFXIUNM

$IBMMFOHFTPGUSBEJUJPOBM4QBSL Single Point of Failure - Limited Fault Tolerance -
Resource Bottlenecks Network Dependency - Network partitions or instability disrupts execution - Clients Need Direct Access - Scheduling Restrictions Difficult Multi-Tenancy - Cluster endpoint must be protected 5JHIUDPVQMJOHPG$MJFOUBOE%SJWFS

3FNPUFDPOOFDUJWJUZUP4QBSLDMVTUFST CZ%BUB'SBNF "1*BOEVOSFTPMWFEMPHJDBMQMBOT WJBH31$ XJUI"QBDIF"SSPXFODPEFESPXCBUDIFT $BOCFFNCFEEFEFWFSZXIFSF BQQMJDBUJPOTFSWFST
*%&T OPUFCPPLTBOEQSPHSBNNJOHMBOHVBHFT "CPVU4QBSL$POOFDU %FDPVQMFMBZFSCFUXFFO$MJFOUBOE%SJWFS

4QBSL$POOFDUBSDIJUFDUVSF IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNM

4QBSL$POOFDU8PSLGMPX IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNM

4QBSLWT4QBSL$POOFDU TQBSL TQBSLDPOOFDU DMJFOUMPDBUJPO JODMVTUFS PVUDMVTUFS DMJFOUSFRVJSFNFOU GVMMTQBSL DMVTUFSBDDFTTBOE +7.CBTFE
UIJODMJFOUNJOJNBMH31$ DMJFOUXJUI%BUB'SBNF -JCSBSZ OFUXPSLUPQPMPHZ NVTUIBWFEJSFDUOFUXPSL BDDFTTUPUIFDMVTUFS SFNPUFBDDFTTWJBH31$ TDBMBCJMJUZ ESJWFS DMJFOUIBOEMFTBMM TDIFEVMJOHBOEDPPSEJOBUJPO ESJWFSSVOTPODMVTUFSDMJFOUT BSFTUBUFMFTT GBVMUUPMFSBODF DMJFOUDSBTIDBVTFTKPCMPTU DMJFOUDBOEJTDPOOFDUBOE SFDPOOFDU

4QBSLWT4QBSL$POOFDU DPOU TQBSL TQBSLDPOOFDU MBOHVBHF MJNJUFEUPTQBSLTVQQPSUFE MBOHVBHFT+7. 1Z4QBSL 3 BOZ
FY KBWBTDSJQU HP D SVTU DMPVE OBUJWF UJHIUDPVQMJOHBOEESJWFS SFRVJSFNFOUTMJNJUGMFYJCJMJUZ EFTJHOFEGPSDMPVEOBUJWF DPOUBJOFSJ[FEBOETFSWFSMFTT TFDVSJUZ FYQPTJOHDMVTUFSFOEQPJOUTDBO SBJTFTFDVSJUZDPODFSOT DBOTFDVSFH31$ DIBOOFMXJUI DFOUSBMJ[FEDPOUSPM WFSTJPOJOH DMJFOUBOETFSWFSNVTUCFUJHIUMZ WFSTJPONBUDIFE DMJFOUBOETFSWFSDBOFWPMWF JOEFQFOEFOUMZ

1Z4QBSL%BUB'SBNF 'VODUJPOTBOE$PMVNO 4DBMB%BUBTFU GVODUJPOT $PMVNO $BUBMPHBOE,FZ7BMVF(SPVQFE%BUBTFU 6TFS%FGJOFE'VODUJPOT 4USFBNJOH"1*T%BUB4USFBN3FBEFS %BUB4USFBN8SJUFS 4USFBNJOH2VFSZ
BOE 4USFBNJOH2VFSZ-JTUFOFS 4VQQPSUFE"1*T IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNMXIBUJTTVQQPSUFE

1SJWBUF'JFMETPG%BUB'SBNF $PMVNO 4QBSL4FTTJPO FUD &YFDVUJPO"1*T3%% &OWJSPONFOU.BOJQVMBUJPO"1*T4QBSL$POUFYU 4QBSL$POGJHVSBUJPO
6OTVQQPSUFE"1*T IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNMIPXTQBSLDPOOFDUDMJFOUBQQMJDBUJPOTEJGGFSGSPNDMBTTJD TQBSLBQQMJDBUJPO

&YUFOTJCJMJUZ 4QBSLBTB4FSWJDFJO$MPVE&OWJSPONFOU .VMUJMBOHVBHF$MJFOUT3VTU (P +BWBTDSJQU FUD
*OUFSBDUJWJUZ /PUFCPPLT+VQZUFS 74$PEF *OUFHSBUFE%FWFMPQNFOU&OWJSPONFOU .JOJNBMJUZ 8FC6*PS#*5PPMT .PCJMFPS.JDSPTFSWJDFT .$1TFSWFST 6TF$BTFT

QPXFSFECZ4QBSL$POOFDU 4QBSLBTB4FSWJDF

*OUFHSBUFXJUIJOIPVTFQMBUGPSNT "DDFTT$POUSPM $VTUPNJ[FTQBSLDMJFOUQFSVTFSTFTTJPO QZUIPO3TDBMBMJCSBSJFT DMJFOUESJWFSTQFDT$16
.FNPSZ (16 FUD $VTUPNJ[F4QBSLJUTFMG JOIPVTFQBUDIFT GJOFUVOFTQBSLQSPQFSUJFT SFNPUFTIVGGMFTFSWJDFT 3FRVJSFNFOUT

4QBSL$POOFDUWT"MUFSOBUJWFT TQBSLDPOOFDU MJWZ LZVVCJ BCTUSBDUJPO MPX NFEJVN IJHI QSPUPDPM H31$
3&45 +%#$0%#$ MBOHVBHF BOZ QZUIPO TDBMB KBWB 3 42- TQBSLWFSTJPO Yr Y TUBOEBMPOF Yr Y JOUFHSBUFE

4QBSL.VMUJUFOBODZ TQBSLDPOOFDU MJWZ LZVVCJ CVJMUJO ❌ /P ⚠ 1BSUJBM ✅
:FT 4FTTJPO *TPMBUJPO ✅ :FT ✅ :FT ✅ :FT 4FTTJPO .BOBHFNFOU ❌ /P ⚠ #BTJD ✅ 'VMM 6TFS "VUIFOUJDBUJPO ⚠ -JNJUFE ⚠ 1MVHHBCMF ✅ 4USPOH 3FTPVSDF 2VPUB ❌ /P ❌ /P ✅ :FT

4QBSL&YUFOTJCJMJUZ TQBSLDPOOFDU MJWZ LZVVCJ -BOHVBHF ✅ "OZ ⚠ +7. 1ZUIPO
3 42- ❌ 42- $VTUPN4QBSL ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN 4FTTJPO ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN6TFS "VUIFOUJDBUJPO ✅ :FT ⚠ -JNJUFE ❌ /P $VTUPN 3FTPVSDF2VPUB ✅ :FT ⚠ -JNJUFE ❌ /P

4FSWF4QBSL%SJWFSJOTUBODFXJUI4QBSL$POOFDU"EEPO 4VQQPSU4QBSL$SFEFOUJBMTLFSCFSPT LFZUBC 4VQQPSUSFTPVSDFRVPUBT 4VQQPSUSPMFNBOBHFNFOU
4VQQPSUBVEJUMPHHJOH )PSJ[POUBMMZ4DBMBCMF 3FRVJSFNFOU3VOUJNF

4FSWF4QBSL%SJWFSJOTUBODFXJUI4QBSL$POOFDU"EEPO 4VQQPSU4QBSL$SFEFOUJBMTLFSCFSPT LFZUBC 4VQQPSUSFTPVSDFRVPUBT 4VQQPSUSPMFNBOBHFNFOU
4VQQPSUBVEJUMPHHJOH )PSJ[POUBMMZ4DBMBCMF è,VCFSOFUFTXJUINVMUJQMFOBNFTQBDFT &OWPZ"%4 $%4 &%4 3FRVJSFNFOU3VOUJNF

8IJMF4QBSL$POOFDUEPFTOPUIBWFCVJMUJOBVUIFOUJDBUJPO JUJTEFTJHOFEUP XPSLTFBNMFTTMZXJUIZPVSFYJTUJOHBVUIFOUJDBUJPOJOGSBTUSVDUVSF*UTH31$ )551JOUFSGBDFBMMPXTGPSUIFVTFPGBVUIFOUJDBUJOHQSPYJFT XIJDINBLFTJU QPTTJCMFUPTFDVSF4QBSL$POOFDUXJUIPVUIBWJOHUPJNQMFNFOUBVUIFOUJDBUJPO MPHJDJO4QBSLEJSFDUMZ 3FRVJSFNFOU"VUI IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNMXIBUJTTVQQPSUFE

8IJMF4QBSL$POOFDUEPFTOPUIBWFCVJMUJOBVUIFOUJDBUJPO JUJTEFTJHOFEUP XPSLTFBNMFTTMZXJUIZPVSFYJTUJOHBVUIFOUJDBUJPOJOGSBTUSVDUVSF*UTH31$ )551JOUFSGBDFBMMPXTGPSUIFVTFPGBVUIFOUJDBUJOHQSPYJFT XIJDINBLFTJU QPTTJCMFUPTFDVSF4QBSL$POOFDUXJUIPVUIBWJOHUPJNQMFNFOUBVUIFOUJDBUJPO MPHJDJO4QBSLEJSFDUMZ è&OWPZY%4 &YU"VUI[ GJMUFS
3FRVJSFNFOU"VUI IUUQTTQBSLBQBDIFPSHEPDTMBUFTUTQBSLDPOOFDUPWFSWJFXIUNMXIBUJTTVQQPSUFE

&OWPZY%4

4FMGNBOBHF4QBSL%SJWFSJOTUBODFT 4FMGNBOBHF$SFEFOUJBMUPBDDFTT4QBSL%SJWFSJOTUBODFT 3FRVJSFNFOU0QFSBUJPO

4FMGNBOBHF4QBSL%SJWFSJOTUBODFT 4FMGNBOBHF$SFEFOUJBMUPBDDFTT4QBSL%SJWFSJOTUBODFT è#BTJD8FCTJUF 3FRVJSFNFOU0QFSBUJPO

.BOBHFNFOU$POTPMF

)JCBOB"SDIJUFDUVSF

%&.04QBSLBTB4FSWJDF

)JCBOB

%&.04QBSL$MJFOU

+VQZUFS

4QBSL$POOFDUJT BH31$CBTFEDMJFOUTFSWFSQSPUPDPMUIBUFOBCMFTSFNPUF JOUFSBDUJWFBDDFTTUP"QBDIF4QBSLVTJOHUIFOBUJWF%BUB'SBNF BOE42- "1*T 4QBSL$POOFDU

4QBSL$POOFDUJT BH31$CBTFEDMJFOUTFSWFSQSPUPDPMUIBUFOBCMFTSFNPUF JOUFSBDUJWFBDDFTTUP"QBDIF4QBSLVTJOHUIFOBUJWF%BUB'SBNF BOE42- "1*T 4QBSL$POOFDUDBO CFVTFEXJUIFYUFSOBM
KPCUSBDLFS KPCTDIFEVMFSBOE VTFSTFTTJPONBOBHFNFOU 4QBSL$POOFDU

Introduction to "Spark Connect" and Implementat...

Introduction to "Spark Connect" and Implementation of Managed Spark as a Service

Transcript

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Featured