Do Flink on Web with FLOW

Do Flink on Web with FLOW

We present a web service named FLOW to let users do FLink On Web. FLOW aims to minimize the effort of handwriting streaming applications similar in spirit to Hortonworks Stream Analytics Manager, StreamAnalytix, and Nussknacker by letting users drag and drop graphical icons representing streaming operators on GUI.

FLOW builds on Flink Table API and lets users assemble graphical icons associated with not only basic SQL operations but also advanced SQL operations like window aggregation, temporal join, and pattern recognition (MATCH_RECOGNIZE clause). Its data preview function enables to observe how sample data changes before and after applying each operation on screen. In addition, FLOW shows the sample data as time-series charts and geographical maps by interacting with Elasticsearch and Kibana. Therefore, domain experts with basic knowledge of SQL can design their streaming applications easily on GUI without understanding of Flink DataStream API and Flink CEP library.

In this talk, we first present what motivates the development of FLOW, then show how FLOW can be used to figure out the "Popular Places" exercise in its own style, and lastly explain how FLOW leverages Flink Table API.

E420b084b2caeb7abcddedf008877c1a?s=128

eastcirclek

October 08, 2019
Tweet

Transcript

  1. %P'MJOL PO8FC XJUI'-08 %POHXPO,JN FBTUDJSDMFL!TLDPN )BFNFF 1BSL IBFNFFQBSL!TLDPN

  2. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  3. 4,5'MJOL'PSXBSEUBMLT 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL

  4. )PXXFVTFEUPXPSL 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL %PNBJOFYQFSUT 5BSHFUTZTUFNT

    3FRVJSFNFOU 3FGJOFSZFOHJOFFST &YQFOTJWFSFGJOFSZFRVJQNFOU %SJWJOHTDPSFTFSWJDF (FOFSBUJOHBMBSNT"4"1 JOSFBMUJNF (FOFSBUJOHTDPSFT"4"1 JOSFBMUJNF .PCJMJUZTFSWJDFQMBOOFST
  5. /FFETGPSSFBMUJNFTUSFBNQSPDFTTJOH 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST )PXUPQSPDFTT SFBMUJNF TUSFBNEBUB
  6. /POOFHMJHJCMFEJTUBODFCFUXFFOEPNBJOFYQFSUTBOE'MJOL 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST NJMMJPO LJMPNFUFST   LJMPNFUFST
  7. %BUBFOHJOFFSTCSJEHFUIFHBQ 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %BUBFOHJOFFST %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST
  8. %BUBFOHJOFFSTCSJEHFUIFHBQ 'MJOL'PSXBSE 1SFEJDUJWF.BJOUFOBODF XJUI 'MJOL 'MJOL'PSXBSE 3FBMUJNFESJWJOHTDPSF TFSWJDFVTJOH'MJOL "QBDIF'MJOL 42-5BCMF"1*

    %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO %BUBFOHJOFFST %PNBJOFYQFSUT 3FGJOFSZFOHJOFFST .PCJMJUZTFSWJDFQMBOOFST %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU 5SBJOJOH5SBOTGFS .BJOUFOBODF .BJOUFOBODF 5SBJOJOH5SBOTGFS 7FSZJOFGGJDJFOU
  9. -FUEPNBJOFYQFSUTEP'MJOL EJSFDUMZWJB'-08 "QBDIF'MJOL 42-5BCMF"1* %BUB4USFBN"1* 1SPDFTT'VODUJPO EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT

    TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO 7BSJPVTEPNBJOTPG4, .PCJMJUZ *P5 5FMDP &OFSHZ 4FNJDPOEVDUPS &DPNNFSDF .FEJB
  10. '-08 "CTUSBDUJPOMBZFSUPIJEF UIFEFUBJMTPG'MJOL BQQEFWFMPQNFOU -FUEPNBJOFYQFSUTEP'MJOL EJSFDUMZWJB'-08 "QBDIF'MJOL 42-5BCMF"1* %BUB4USFBN"1* 1SPDFTT'VODUJPO

    EZOBNJDUBCMFT TUSFBNT XJOEPXT FWFOUT TUBUF UJNF +7.MBOHVBHFT *%& 1SPKFDUNBOBHFNFOU %BUBWJTVBMJ[BUJPO 7BSJPVTEPNBJOTPG4, .PCJMJUZ *P5 5FMDP &OFSHZ 4FNJDPOEVDUPS &DPNNFSDF .FEJB (SBQIJDBMVTFSJOUFSGBDF UPCVJME TUSFBNQSPDFTTJOHQJQFMJOFTPG 42-PQFSBUJPOT BOEDPOOFDUPST %BUB %PNBJO LOPXMFEHF 3FRVJSFNFOU 42- 42- 42- 42-
  11. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNPm JEFOUJGZJOHQPQVMBSQMBDFTXJUI'-08  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  12. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB 3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF

    4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF  3JEF*E  5BYJ*E  %SJWFS*E  *T4UBSU 5SVF 4UBSU$PPSE   &OE$PPSE   1BTTFOHFS$PVOU  &WFOU5JNF                   
  13. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB        

              3FNPWJOH FWFOUTUIBUEPOPU TUBSUPSFOEJO/FX:PSL$JUZ
  14.          

         %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB N N .BQQJOHDPPSEJOBUFTPGFBDI SFDPSE JOUPBHSJEDFMM               
  15. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB $SFBUJOHBTMJEJOHXJOEPXPGTJ[F NJOVUFT UIBUTMJEFTCZNJOVUFT     

             
  16. %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB N N    $PVOUJOH UIFOVNCFSPGFWFOUT JOFBDIHSJEDFMMQFSXJOEPX

                  
  17. N N %FNP4DFOBSJP*EFOUJGZJOHQPQVMBSQMBDFTJO/:$ ,BGLB  *EFOUJGZJOHUIFDFMMTXIPTFDPVOUJT PSNPSFBTBQPQVMBSQMBDF   

                 
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. 5 min 5 min 5 min

  49. None
  50. None
  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSFm IPX'-08 JOUFSBDUTXJUI'MJOL  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  59. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 4QFD 4QFD 4DIFNB 1SFWJFX FYQFDUFEUPCFSFUVSOFE

    BTSFTQPOTF 1SFWJFX 4DIFNB OPUDPNQVUFEZFU ,BGLB TPVSDF
  60. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD 4QFD 4DIFNB

    1SFWJFX 4DIFNB 1SFWJFX ,BGLB TPVSDF 4QFD 1SFWJFX-PBEFST ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 ,BGLB TPVSDF 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065
  61. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF

    1SFWJFX-PBEFST 4QFD ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD 4QFD CVUBMTPQBSFOUT TDIFNBQSFWJFX ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 1SFWJFX 4DIFNB OPUDPNQVUFEZFU 'JMUFS OPUPOMZTQFD
  62. -PDBM4USFBN&OWJSPONFOU FOW 'MJOL1SFWJFX-PBEFS 4USFBN5BCMF&OWJSPONFOU U&OW WBMQBSFOU5BCMFU&OWSFHJTUFS5BCMF  ,BGLB TPVSDF 4DIFNB

    1SFWJFX SFHJTUFSBMMLOPXO6%'JOTUBODFT U&OWSFHJTUFS'VODUJPO JT*O/:$ OFX(FP6UJMT*T*O/:$ U&OWSFHJTUFS'VODUJPO UP$FMM*E OFX(FP6UJMTUP$FMM*E U&OWSFHJTUFS'VODUJPO UP$PPSET OFX(FP6UJMT5P$PPSET XXXIBMMPXFFODPTUVNFTDPNBEVMUQJHHZCBDLSJEFPODPTUVNFIUNM '-08 '-08QJHHZCBDLTPO'MJOL GPSTDIFNBQSFWJFX DPNQVUBUJPO WBMFOW4USFBN&YFDVUJPO&OWJSPONFOUDSFBUF-PDBM4USFBN&OWJSPONFOU WBMU&OW4USFBN5BCMF&OWJSPONFOUDSFBUF FOW 'MJOL1SFWJFX -PBEFS -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 WBMUBCMFQBSFOU5BCMFGJMUFS JT*O/:$ TUBSU-PO TUBSU-BU JT*O/:$ TUBSU-PO TUBSU-BU   'JMUFS 4QFD UPHFUUIFSFTVMUJOUIJTUISFBE U&OWUP"QQFOE4USFBN UBCMF 3PXDMBTT BEE4JOL OFX$PMMFDU4JOL FOWFYFDVUF SFUVSOBOE SFUVSO $PMMFDU4JOLWBMVFT UBCMFHFU4DIFNB 4DIFNB 1SFWJFX ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD */ 5BCMFT 'VODUJPOT QBSFOU5BCMF UBCMF JT*O/:$ GMPBU GMPBU ªCPPMFBO UP$FMM*E GMPBU GMPBU ªJOU UP$PPSEJOUª GMPBU GMPBU
  63. ,BGLB TPVSDF 4DIFNB 1SFWJFX 4DIFNB 1SFWJFX 'JMUFS 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH

    'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF 1SFWJFX-PBEFST 4QFD ,BGLB TPVSDF 4DIFNB 1SFWJFX 'JMUFS 4QFD 4DIFNB 1SFWJFX 1SFWJFX 4DIFNB 4QFD 'JMUFS ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065
  64. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF 1SFWJFX 4DIFNB

    4QFD 'JMUFS 1SFWJFX 4DIFNB 4QFD 4FMFDU 1SFWJFX 4DIFNB 4QFD 8JOEPX 1SFWJFX 4DIFNB 4QFD 42- RVFSZ 1SFWJFX-PBEFST 4QFD 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD 4DIFNB 1SFWJFX 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT ,BGLB1SFWJFX -PBEFS ,BGLB DPOTVNFS ,BGLB TPVSDF 4QFD */ 4DIFNB 1SFWJFX 065 'MJOL1SFWJFX -PBEFS 1BSFOU 4DIFNB 1SFWJFX $IJME 4QFD */ -PDBM'MJOL NJOJDMVTUFS 4DIFNB 1SFWJFX 065 1BSFOU 4DIFNB 1SFWJFX 4DIFNB 1SFWJFX $IJME
  65. 0WFSBMMBSDIJUFDUVSFPG'-08 3&45GVM8FC4FSWFS CZ4QSJOH 'SPOUFOE*OUFSGBDF CZ"OHVMBSKT 1SFWJFX 4DIFNB 4QFD ,BGLB TPVSDF

    1SFWJFX 4DIFNB 4QFD 'JMUFS 1SFWJFX 4DIFNB 4QFD 4FMFDU 1SFWJFX 4DIFNB 4QFD 8JOEPX 1SFWJFX 4DIFNB 4QFD 42- RVFSZ 1SFWJFX 4DIFNB 4QFD &44JOL 1SPKFDU(FOFSBUPS CZKBWBQPFUGSFFNBSLFS 4DIFNB 4QFD ,BGLB TPVSDF 4DIFNB 4QFD 'JMUFS 4DIFNB 4QFD 4FMFDU 4DIFNB 4QFD 8JOEPX 4DIFNB 4QFD 42- RVFSZ 4DIFNB 4QFD &44JOL .BWFOQSPKFDU 'MJOLBQQMJDBUJPO
  66. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  67. 4VQQPSUFEPQFSBUJPOTDPOOFDUPST #BTJD42-PQFSBUJPOT &YUFSOBMDPOOFDUPST "EWBODFE42-PQFSBUJPOT

  68. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  69. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  70. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  71. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  72. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  73. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  74. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  75. 1BUUFSOSFDPHOJUJPO ."5$)@3&$0(/*;&DMBVTF

  76. 4VQQPSUFEPQFSBUJPOTDPOOFDUPST #BTJD42-PQFSBUJPOT &YUFSOBMDPOOFDUPST "EWBODFE42-PQFSBUJPOT

  77. 5FNQPSBMKPJO

  78. 5FNQPSBMKPJO

  79. 5FNQPSBMKPJO

  80. 5FNQPSBMKPJO

  81. T A B L E O F C O N

    T E N T S %P'MJOL PO8FCXJUI'-08  .PUJWBUJPO  %FNP  "SDIJUFDUVSF  4VQQPSUFEPQFSBUJPOTDPOOFDUPST  4VNNBSZ
  82. 4VNNBSZ $POOFDUPST  ,BGLBTPVSDF  ,BGLBTJOL  &MBTUJDTFBSDITJOL $IBSUCBDLFOE 

    &MBTUJDTFBSDI ,JCBOB  NFUBUSPOEJTDPWFSZ TJTUFSQSPKFDU 42-PQFSBUJPOT  4FMFDU  'JMUFS  8JOEPX  42-RVFSZ  1BUUFSOSFDPHOJUJPO  5FNQPSBMKPJO 1SPKFDUDPEFHFOFSBUJPO  .BWFO  (SBEMF 6%'TVQQPSU  4DBMBSGVODUJPOT  5BCMFGVODUJPOT  "HHSFHBUFGVODUJPOT  5BCMFBHHSFHBUFGVODUJPOT %BUBQSFWJFX  5BCMF  5JNFTFSJFTDIBSU  (FPHSBQIJDBMNBQ 'VUVSFXPSL  .PSFDPOOFDUPSTPQFSBUJPOT  +PCNBOBHFNFOU  #BUDIKPCTVQQPSU