世界初！Go言語とMySQL UDFで実現する超高速データ前処理/分析基盤
Google BigQuery MLと同じ発想で、MySQL上に文字列データ前処理関数群を作って運用まで回していた。という話をさせていただきました。
2018.7.27 Developers Summit 2018 Summer C-8 Session https://event.shoeisha.jp/devsumi/20180727/session/1771/
mecompany Inc.12018728ੈքॳʂ(Pݴޠͱ.Z42-6%'Ͱ࣮ݱ͢Δߴσʔλલॲཧੳج൫%FWFMPQFST4VNNJU4VNNFS$4FTTJPOEFWTVNJ$ాޱେथϛʔΧϯύχʔגࣜձࣾ(PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%
View Slide
22018728lલॲཧ൛(PPHMF#JH2VFSZ .-zΛ.Z42-্ʹ(PݴޠͰ࣮ͯ͠ӡ༻·Ͱճ͍ͯ͠·ͨ͠%FWFMPQFST4VNNJU4VNNFS$4FTTJPOEFWTVNJ$ాޱେथϛʔΧϯύχʔגࣜձࣾ(PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%mecompany Inc.
2018728mecompany Inc.3ϓϩϑΟʔϧాޱେथ %BJLJ5BOPHVDIJ !EQSPKFDUϛʔΧϯύχʔגࣜձࣾ(PMBOH0VUTUBOEJOH%BUBFOHJOFFS (0%ྵࡉιϑτϋε 7#/&54*FS8FCαʔϏεاۀྲྀ࿘ͷຽओݴޠ $+BWBػցֶश *P5τϥΠΞϧࢀը 1ZUIPO(PMBOH લॲཧϥΠϒϥϦ։ൃͳͲ"LJCBHP(PMBOH5PLZP$PEFGPS+BQBOϦʔμʔक़%FW-07& 1VC50$G& #PPUDBNQ8"$"5&ϛʔΧϯύχʔגࣜձࣾ
ϓϩϑΟʔϧmecompany Inc. 48C:%#G+K$/&/0B -!8XVWV WX)Y(,J'EZSYYSXQ XeQ3 I ¡A>#¢P* 9 ¡bci¢WRVVVW_51N?D M6Fldgeanmjg"[email protected]B" VYS\Z[]S^[]_VYS\\\]SZ_XXt||y`UUvrpxvyow~Tvrt||y`UU{p}ruTvrt||y`UU{p}ruqo|oTvrt||y`UU{p}ruZvzTvr[email protected]?51 U7DU? =2H // /fl / / ; / 4O?.< kbmdg kbmdgQsxzQhj2018 7)28(
mecompany Inc. 5ҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔεϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ2018728ϓϩϑΟʔϧ
<3Smecompany Inc. 67> QP#?y!9/w (AzR'_Uw4KwF6_HXZ^\Wkxg* Y^]r`xmoic_\'_%Xy!9/w (Az'U[email protected] ;[email protected];3uxbx;3S MJ O2(SCUEL ProjectpxlnxVL$=NdxqfI8G&G;wMwOdxqdxwwA1DsvoahM0WEBWEBWEBSCUEL Projectpxlnxkxg58)+:T]HP2018"7-28,ϓϩϑΟʔϧ
5-Kࠓͷʜmecompany Inc. 707 IH8r2)p #:sr2)p #:s"M
6.I݁ʜmecompany Inc. 818 HG9k3*i #;lk3*i #;l"K=Z[^f: 5)?:5.gjXj5.I DB F-#SCUEL Projectcj`ajLC 7EYjd[[email protected][email protected]5iDiFYjdYjii;,>ehbW]D+WEBWEBWEBSCUEL Projectcj`aj_j\/2$%4JUHP20187(28'&_j\U 0PRMQNOVSST
2018728mecompany Inc.9σʔλ։ൃͷݱ
σʔλ։ൃͷݱmecompany Inc. 10 ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλ&YDFM $47ͷଞ 1%' 8FCαΠτ ࢴʢཧʣ͕͋Δ 2018728ࢴͱઓ͏ͨΊʹ0$3Λͬͨ͋ͱޡೝࣝσʔλͱઓΘͳ͚Ε͍͚ͳ͍ɻ
mecompany Inc. 11 ެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλதʹ zόουσʔλzଘࡏ͢Δ 2018728• ޡࣈࣈ• දهΏΕ• ҟମࣈ• ༣ศ൪߸ؒҧ͍• ి൪߸ ͳ͔ͥ&ϝʔϧΞυϨε• ʙࢢʙ۠ ͷ͕۠ൈ͚͍ͯΔσʔλ։ൃͷݱ
!mecompany Inc. 12 / -0/ -0"2018728લॲཧରσʔλ ԯ߲ʢेສϨίʔυ ºສ߲ʣ%'+.*+.*,.)ࠃ๏ਓ ສϨίʔυެతػ͔ؔΒ։ࣔ͞Ε͍ͯΔσʔλΛ౷߹͠l͑Δzσʔλʹ͢ΔͨΊɺલॲཧ͕ඞཁ#($&+.*,.)ଞ͔ࣾΒߪೖͨ͠σʔλϕʔε ԯϨίʔυσʔλ։ൃͷݱ
mecompany Inc. 13ެతͳެ։ใ͔Β z͑Δzσʔλʹ্ཱͯ͛ͨҩྍػؔɾༀہɾհޢࣄۀॴͷࢪઃɾ๏ਓσʔλϕʔεϥΠηϯεͷൢചɾϝϯςφϯεΛߦ͍ͬͯ·͢ɻ2018728σʔλ։ൃͷݱ
!mecompany Inc. 14 / -0ԯʙԯ݅ͷσʔλલॲཧ͕ఆৗۀ/ -0"2018728%'+.*+.*,.)#($&+.*,.)• ຖมΘΔ੍มߋʹै• σʔλͷׂɺ౷߹ɺഇࢭ͕ৗతʹൃੜ• ໊শมߋɺࢢ۠ொଜ߹ซͳͲॅॴมߋൃੜ͢Δσʔλ։ൃͷݱ
!σʔλ։ൃͷ࣮ࡍmecompany Inc. 15 / -0ߴɺߴਫ਼ɺशಘ༰қͳલॲཧ͕ඞཁ/ -0"2018728%'+.*+.*,.)#($&+.*,.)• ࣄۀ͕֦େ͢Δͱѻ͏ྖҬ߲͕૿͑Δ• ৽ϝϯόʔͰ͙͢ʹΩϟονΞοϓͰ͖ΔΈ͕ඞཁ• σʔλੳɺઃܭɺ࣮શͯͰಉ͡ΈΛ͍ੜ࢈ੑΛߴΊ͍ͨ
2018728mecompany Inc.16ͳͥ.Z42-6%'ʢϢʔβʔఆٛؔʣͳͷ͔
ͳͥ.Z42-6%'ͳͷ͔mecompany Inc. 17 42-͕ෆಘҙͳจࣈܕσʔλͷॲཧΛͲ͏ʹ͔͍ͨ͠ 2018728• JGOVMM SFQMBDF͕Ԇʑͱଓ͘42-• ઍߦͷετΞυϓϩγʔδϟ• ૿͑ΔҰํͷσʔλྔʹॲཧ͕͍͔ͳ͍
ͳͥ.Z42-6%'ͳͷ͔mecompany Inc. 18 42-͕ෆಘҙͳจࣈܕσʔλͷॲཧΛ6%'ʢϢʔβʔఆٛؔʣͰ͏ 2018728• ৽ϝϯόʔͰ͙͢ʹΩϟονΞοϓͰ͖ΔΈ͕ඞཁˠࣾશମͷڞ௨εΩϧ͕42-• σʔλੳɺઃܭɺ࣮શͯͰಉ͡ΈΛ͍͍ͨˠ6%'ͳΒ42-ͷͳ͔Ͱѻ͏͜ͱ͕Ͱ͖Δ• 42-͕ෆಘҙͳจࣈܕσʔλͷՃΛͲ͏ʹ͔͍ͨ͠ˠจࣈܕσʔλॲཧ༻ͷ6%'Λఆٛ͢Δ
2018728mecompany Inc.19ͳͥ(Pݴޠͳͷ͔
ͳͥ(Pݴޠͳͷ͔mecompany Inc. 20ػցֶशͳΒɺࠓͷ࣌͝ੈͩͱ1ZUIPOػցֶश #*σʔλੳͷ͞ΒʹલʹདྷΔॲཧͳͷͰ1ZUIPOͰͳͯ͘Α͍2018728
ͳͥ(Pݴޠͳͷ͔mecompany Inc. 21$$ͱͷੑ͕ߴ͘ɺॻ͖͍͢2018728• .Z42-6%'௨ৗ $$Ͱॻ͔ΕΔ• $$ͱͷม෦͕ଞݴޠͱൺͯൺֱతॻ͖͍͢ʢ'PSFJHOGVODUJPOJOUFSGBDF''*(PݴޠͰDHPʣ• ϝϞϦཧ($͕ͬͯ͘ΕΔ• ϥΠϒϥϦ͍͍۩߹ʹἧͬͯΔ
ͳͥ(Pݴޠͳͷ͔mecompany Inc. 22ѻ͏σʔλͱॲཧͷཁ݅ʹϚονͨ͠2018728• όΠφϦʹ࣮ͯ͠ߦͰ͖Δ• αϩήʔτϖΞͱ͔ؾʹ͠ͳͯ͘Α͍• $$ͱಉͷͰɺ3VCZ1ZUIPOͱಉͷѻ͍͕͢͞ಘΒΕΔͷ͕(Pݴޠ• จࣈܕσʔλͷॲཧ͕ओͳతͳͷͰɺ1BOEBT/VNQZΛ͏ඞཁੑ͕ͳ͔ͬͨ
ͳͥ(Pݴޠͳͷ͔mecompany Inc. 23কདྷతͳ֦ுੑ2018728• $-*8FC"1*ϑϨʔϜϫʔΫͷϥΠϒϥϦ͕ἧ͍ͬͯΔ• $MFBO"SDIJUFDUVSFͰཱͯΕɺ$-*8FC"1*ͳͲͷల։༰қ6%'&YUFSOBM*OUFSGBDFTͱͯ͠ѻ͏͜ͱ͕Ͱ͖Δɻ
2018728mecompany Inc.24։ൃՌ
։ൃՌmecompany Inc. 252018728։ൃͨ͠൚༻จࣈྻॲཧ6%'● [email protected] OVMM͍͍۩߹ʹѻ͑ΔDPODBU● TQMJUΧϯϚ۠ΓσʔλͷY൪͚ͩΛऔΓग़͢● 8FCαΠτεΫϨΠϐϯά݁Ռͷॲཧ● ࣈɾΞϧϑΝϕοτɾΧλΧφͷશ֯֯ม● ࣈɾه߸ͷҟମࣈม● ͻΒ͕ͳΧλΧφ૬ޓมͳͲଟ
։ൃՌmecompany Inc. 262018728ྫʣ OVMM͍͍۩߹ʹѻ͑ΔDPODBU[email protected] QMBDFσϒαϛՆ ޚϊਫιϥγςΟσϒαϛౙ OVMMOVMM ϕϧαʔϧ౦ژຊڮselect concat(ifnull(event_name, ’’),ifnull(place, ‘’)) from event;select concat_nullable(event_name,place) from event;
։ൃՌmecompany Inc. 272018728ॲཧ8FCαΠτεΫϨΠϐϯά݁Ռͷॲཧ● εΫϨΠϐϯά݁Ռͷ+40/σʔλԯ݅Λ࣌ؒͰॲཧࣈɾΞϧϑΝϕοτɾΧλΧφͷશ֯֯ม● 1P$ͷ࣌ͰετΞυϓϩγʔδϟ͔ΒഒߴԽ ʢ݄ʣ● ݱࡏετΞυϓϩγʔδϟ͔ΒഒߴԽ
2018728mecompany Inc.28ߋͳΔ։ൃՌ
ߋͳΔ։ൃՌmecompany Inc. 29.Z42-6%'ͷཪଆͰ ී௨ͷγεςϜ͕ಈ͘2018728• ཪͰ 3FEJT ʹॲཧ݁ՌΩϟογϡΛ࣋ͨͤΔ• ଞ%#42-ΛྲྀͤΔ• ଞࣾ8FC"1*ୟ͚Δ6%' 3FEJT ଞࣾ8FC"1* ଞ%#શͯ &YUFSOBM*OUFSGBDFT$MFBO"SDIJUFDUVSF͍͍ͧ.Z42-6%'lͳͷʹz
ߋͳΔ։ൃՌmecompany Inc. 302018728l߲zʹಛԽͨ͠จࣈྻॲཧ6%'• ి൪߸ ͳ͔ͥ&ϝʔϧΞυϨε• ࢢ֎ہ൪͕ൈ͚͍ͯΔͳͲͷόουి൪߸σʔλ͔Βਖ਼͍͠ి൪߸Λநग़͢ΔޡࣈࣈɺදهΏΕɺҟମࣈɺ༣ศ൪߸ؒҧ͍ɺʙࢢʙ۠ ͷ͕۠ൈ͚͍ͯΔͳͲͷόουॅॴσʔλΛͳΜͱ͔͍ͨ͠ॅॴਖ਼نԽి൪߸ਖ਼نԽ
ߋͳΔ։ൃՌmecompany Inc. 312018728l߲zʹಛԽͨ͠จࣈྻॲཧ6%'΄΅ਖ਼͍͠ి൪߸Λநग़Մೳ• ࣌ؒ͋ͨΓສ݅ΛॲཧՄೳ• ࠃ࠷ߴͷิਫ਼Λୡ˞ϥϯμϜʹநग़ͨ͠σʔλ݅Λଞࣾͷॅॴਖ਼نԽ݁Ռͱൺֱॅॴਖ਼نԽి൪߸ਖ਼نԽ͜͜·Ͱհͨ͠ཁૉٕज़ͷ্ʹΓཱ͍ͬͯΔ
2018728mecompany Inc.32͜ΕΒΛؾܰʹѻ͑Δ
6.I݁mecompany Inc. 3318 HG9k3*i #;lk3*i #;l"K=Z[^f: 5)?:5.gjXj5.I DB F-#SCUEL Projectcj`ajLC 7EYjd[[email protected][email protected]5iDiFYjdYjii;,>ehbW]D+WEBWEBWEBSCUEL Projectcj`aj_j\/2$%4JUHP20187(28'&_j\U 0PRMQNOVSST
dproject21
jacopen
hanhan1978
jordihofc
toshiaizawa
mizunohiroaki
odasho
tsubakimoto_s
arthur1
k1low
harukasakihara
sat
renjikari
morganepeng
rocio
destraynor
robhawkes
jennybc
swwweet
marcduiker
lara
shpigford
andyhume
jakevdp
zakiwarfel