Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Do Flink on Web with FLOW

Do Flink on Web with FLOW

We present a web service named FLOW to let users do FLink On Web. FLOW aims to minimize the effort of handwriting streaming applications similar in spirit to Hortonworks Stream Analytics Manager, StreamAnalytix, and Nussknacker by letting users drag and drop graphical icons representing streaming operators on GUI.

FLOW builds on Flink Table API and lets users assemble graphical icons associated with not only basic SQL operations but also advanced SQL operations like window aggregation, temporal join, and pattern recognition (MATCH_RECOGNIZE clause). Its data preview function enables to observe how sample data changes before and after applying each operation on screen. In addition, FLOW shows the sample data as time-series charts and geographical maps by interacting with Elasticsearch and Kibana. Therefore, domain experts with basic knowledge of SQL can design their streaming applications easily on GUI without understanding of Flink DataStream API and Flink CEP library.

In this talk, we first present what motivates the development of FLOW, then show how FLOW can be used to figure out the "Popular Places" exercise in its own style, and lastly explain how FLOW leverages Flink Table API.

Avatar for eastcirclek

eastcirclek

October 08, 2019
Tweet

More Decks by eastcirclek

Other Decks in Programming

Transcript

  1. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  2. )PXXFVTFEUPXPSL 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL %PNBJOFYQFSUT 5BSHFUTZTUFNT

    3FRVJSFNFOU 3FGJOFSZFOHJOFFST &YQFOTJWFSFGJOFSZFRVJQNFOU %SJWJOHTDPSFTFSWJDF (FOFSBUJOHBMBSNT"4"1 JOSFBMUJNF (FOFSBUJOHTDPSFT"4"1 JOSFBMUJNF .PCJMJUZTFSWJDFQMBOOFST
  3. /FFETGPSSFBMUJNFTUSFBNQSPDFTTJOH 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST )PXUPQSPDFTT SFBMUJNF TUSFBNEBUB
  4. /POOFHMJHJCMFEJTUBODFCFUXFFOEPNBJOFYQFSUTBOE'MJOL 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST NJMMJPO LJMPNFUFST   LJMPNFUFST
  5. %BUBFOHJOFFSTCSJEHFUIFHBQ 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %BUBFOHJOFFST %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST
  6. %BUBFOHJOFFSTCSJEHFUIFHBQ 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %BUBFOHJOFFST %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU 5SBJOJOH5SBOTGFS .BJOUFOBODF .BJOUFOBODF 5SBJOJOH5SBOTGFS 7FSZJOFGGJDJFOU
  7. -FUEPNBJOFYQFSUTEP'MJOL EJSFDUMZWJB'-08 "QBDIF'MJOL 42-5BCMF"1* %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT

    TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO 7BSJPVTEPNBJOTPG4, .PCJMJUZ *P5 5FMDP &OFSHZ 4FNJDPOEVDUPS &DPNNFSDF .FEJB
  8. '-08 "CTUSBDUJPOMBZFSUPIJEF UIFEFUBJMTPG'MJOL BQQEFWFMPQNFOU -FUEPNBJOFYQFSUTEP'MJOL EJSFDUMZWJB'-08 "QBDIF'MJOL 42-5BCMF"1* %BUB4USFBN"1* 1SPDFTT'VODUJPO

    EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO 7BSJPVTEPNBJOTPG4, .PCJMJUZ *P5 5FMDP &OFSHZ 4FNJDPOEVDUPS &DPNNFSDF .FEJB (SBQIJDBMVTFSJOUFSGBDF UPCVJME TUSFBNQSPDFTTJOHQJQFMJOFTPG 42-PQFSBUJPOT BOEDPOOFDUPST %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU 42- 42- 42- 42-
  9. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNPm JEFOUJGZJOHQPQVMBSQMBDFTXJUI'-08  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  10. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB 3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF

    4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF                   
  11. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB        

              3FNPWJOH FWFOUTUIBUEPOPU TUBSUPSFOEJO/FX:PSL$JUZ
  12.          

         %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB N N .BQQJOHDPPSEJOBUFTPGFBDI SFDPSE JOUPBHSJEDFMM               
  13. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSFm IPX'-08 JOUFSBDUTXJUI'MJOL  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  14. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD 4QFD 4DIFNB

    1SFWJFX 4DIFNB 1SFWJFX ,BGLB TPVSDF 4QFD 1SFWJFX-PBEFST ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 ,BGLB TPVSDF 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065
  15. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF

    1SFWJFX-PBEFST 4QFD ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD 4QFD CVUBMTPQBSFOUT TDIFNBQSFWJFX ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 1SFWJFX 4DIFNB OPUDPNQVUFEZFU 'JMUFS OPUPOMZTQFD
  16. -PDBM4USFBN&OWJSPONFOU FOW 'MJOL1SFWJFX-PBEFS 4USFBN5BCMF&OWJSPONFOU U&OW WBMQBSFOU5BCMFU&OWSFHJTUFS5BCMF  ,BGLB TPVSDF 4DIFNB

    1SFWJFX SFHJTUFSBMMLOPXO6%'JOTUBODFT U&OWSFHJTUFS'VODUJPO JT*O/:$ OFX(FP6UJMT*T*O/:$ U&OWSFHJTUFS'VODUJPO UP$FMM*E OFX(FP6UJMTUP$FMM*E U&OWSFHJTUFS'VODUJPO UP$PPSET OFX(FP6UJMT5P$PPSET XXXIBMMPXFFODPTUVNFTDPNBEVMUQJHHZCBDLSJEFPODPTUVNFIUNM '-08 '-08QJHHZCBDLTPO'MJOL GPSTDIFNBQSFWJFX DPNQVUBUJPO WBMFOW4USFBN&YFDVUJPO&OWJSPONFOUDSFBUF-PDBM4USFBN&OWJSPONFOU WBMU&OW4USFBN5BCMF&OWJSPONFOUDSFBUF FOW 'MJOL1SFWJFX -PBEFS -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 WBMUBCMFQBSFOU5BCMFGJMUFS JT*O/:$ TUBSU-PO TUBSU-BU JT*O/:$ TUBSU-PO TUBSU-BU   'JMUFS 4QFD UPHFUUIFSFTVMUJOUIJTUISFBE U&OWUP"QQFOE4USFBN UBCMF 3PXDMBTT BEE4JOL OFX$PMMFDU4JOL FOWFYFDVUF SFUVSOBOE SFUVSO $PMMFDU4JOLWBMVFT UBCMFHFU4DIFNB 4DIFNB 1SFWJFX ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD */ 5BCMFT 'VODUJPOT QBSFOU5BCMF UBCMF JT*O/:$ GMPBU GMPBU ªCPPMFBO UP$FMM*E GMPBU GMPBU ªJOU UP$PPSEJOUª GMPBU GMPBU
  17. ,BGLB TPVSDF 4DIFNB 1SFWJFX 4DIFNB 1SFWJFX 'JMUFS 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH

    'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF 1SFWJFX-PBEFST 4QFD ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD 4DIFNB 1SFWJFX 1SFWJFX 4DIFNB 4QFD 'JMUFS ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065
  18. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF 1SFWJFX 4DIFNB

    4QFD 'JMUFS 1SFWJFX 4DIFNB 4QFD 4FMFDU 1SFWJFX 4DIFNB 4QFD 8JOEPX 1SFWJFX 4DIFNB 4QFD 42- RVFSZ 1SFWJFX-PBEFST 4QFD 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD 4DIFNB 1SFWJFX 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 1BSFOU 4DIFNB 1SFWJFX 4DIFNB 1SFWJFX $IJME
  19. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF

    1SFWJFX 4DIFNB 4QFD 'JMUFS 1SFWJFX 4DIFNB 4QFD 4FMFDU 1SFWJFX 4DIFNB 4QFD 8JOEPX 1SFWJFX 4DIFNB 4QFD 42- RVFSZ 1SFWJFX 4DIFNB 4QFD &44JOL 1SPKFDU(FOFSBUPS CZKBWBQPFUGSFFNBSLFS 4DIFNB 4QFD ,BGLB TPVSDF 4DIFNB 4QFD 'JMUFS 4DIFNB 4QFD 4FMFDU 4DIFNB 4QFD 8JOEPX 4DIFNB 4QFD 42- RVFSZ 4DIFNB 4QFD &44JOL .BWFOQSPKFDU 'MJOLBQQMJDBUJPO
  20. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  21. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  22. 4VNNBSZ $POOFDUPST  ,BGLBTPVSDF  ,BGLBTJOL  &MBTUJDTFBSDITJOL $IBSUCBDLFOE 

    &MBTUJDTFBSDI ,JCBOB  NFUBUSPOEJTDPWFSZ TJTUFSQSPKFDU 42-PQFSBUJPOT  4FMFDU  'JMUFS  8JOEPX  42-RVFSZ  1BUUFSOSFDPHOJUJPO  5FNQPSBMKPJO 1SPKFDUDPEFHFOFSBUJPO  .BWFO  (SBEMF 6%'TVQQPSU  4DBMBSGVODUJPOT  5BCMFGVODUJPOT  "HHSFHBUFGVODUJPOT  5BCMFBHHSFHBUFGVODUJPOT %BUBQSFWJFX  5BCMF  5JNFTFSJFTDIBSU  (FPHSBQIJDBMNBQ 'VUVSFXPSL  .PSFDPOOFDUPSTPQFSBUJPOT  +PCNBOBHFNFOU  #BUDIKPCTVQQPSU