DMMでのPrestoおよび周辺エコシステムの運用事例

Presto Meetup 201706にて、発表した弊社のPresto運用事例紹介です。
https://techplay.jp/event/621143

Transcript

1. %..DPNϥϘ
   ౻Ҫ
   ྄ଠ 
   1SFTUP
   .FFUVQ %..Ͱͷ1SFTUP͓Αͼ
   पลΤίγεςϜͷӡ༻ࣄྫ
2. 2 Copyright © since 1998 DMM All Rights Reserved. ࣗݾ঺հ Ϗοάσʔλج൫ͷӡ༻อक
   1SFTUPྺɿ೥Ґ
   (P
   +BWB
   3VCZ
   1)1 ౻Ҫ
   ྄ଠ
   %..DPN
   ϥϘ
   γεςϜຊ෦
   Ϗοάσʔλ෦
3. 3 Copyright © since 1998 DMM All Rights Reserved. %..DPNʹ͍ͭͯ %..DPNͱάϧʔϓʹ͍ͭͯ
4. 4 Copyright © since 1998 DMM All Rights Reserved. αʔϏεʹ͍ͭͯ %..DPN͸೥͔Βଓ͘࿝ฮ αʔϏεαΠτͰ͢ɻ࣌୅ͷχʔζ ʹ߹Θͤͨଟ࠼ͳίϯςϯπΛɺ ೥ؒഓͬͨಠࣗϓϥοτϑΥʔϜͰ ҆ఆతʹఏڙ͍ͯ͠·͢ɻ
   "CPVU
   %..DPN Ҏ্ͷ෯޿͍αʔϏεΛల։
5. 5 Copyright © since 1998 DMM All Rights Reserved. ৽ΦϑΟε /FX
   P⒏DF
6. 6 Copyright © since 1998 DMM All Rights Reserved. ΞδΣϯμ w γεςϜߏ੒
   w ֤ίϯϙʔωϯτͷ໾ׂ
   w 1SFTUPϦιʔεͷ׆༻
   w ·ͱΊ
   
7. 7 Copyright © since 1998 DMM All Rights Reserved. γεςϜߏ੒
8. 8 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤ 1SFTUP ;FQQFMJO 1SFTUP 3FEBTI &MBTUJDTFBSDI
   LJCBOB )BEPPQ CBUDI NBSJBEC BQJ !
9. 9 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤ
   
   BOBMZUJDT
    1SFTUP ;FQQFMJO 1SFTUP 3FEBTI &MBTUJDTFBSDI
   LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w ΞφϦετ͕೚ҙͷΫΤϦΛ ࣮ߦ͢Δ؀ڥ
   w 1SFTUPΫΤϦ͸SFEBTI
   
   [FQQFMJO͔Β࣮ߦ͢Δ
   w ௕ظՔಇ͍ͯ͠ΔΫΤϦ͸ɺ ΤϯδχΞ͕؂ࢹ͠ɺঢ়گ ʹԠ͡Ωϟϯηϧ͢Δ ෼ੳ؀ڥ
10. 10 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤ
    
    CBUDI
     1SFTUP [FQQFMJO 1SFTUP SFEBTI &MBTUJDTFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w γεςϜ͔Βൃߦ͞ΕΔΫΤ Ϧͷ࣮ߦ؀ڥ
    w IBEPPQ্ͷσʔλΛBQJܦ ༝ͰଞγεςϜʹ࿈ܞ͢Δ ͨΊͷநग़ॲཧΛߦ͏
 IBEPPQʹର͢Δ&5-͸ߦ Θͳ͍
    w ఆܕతͳΫΤϦ͕ϝΠϯ όον؀ڥ
11. 11 Copyright © since 1998 DMM All Rights Reserved. ֤ίϯϙʔωϯτͷ໾ׂ
12. 12 Copyright © since 1998 DMM All Rights Reserved. ;FQQFMJO 1SFTUP 3FEBTI FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! )BEPPQ %..άϧʔϓશମͷϏοάσʔλج൫ $%)
    
    
    5#
    
    
    1#
    
    ECT
    
    
    UBCMFT
    NPTUMZ
    TUPSFE
    BT
    03$
    
    TOBQQZ
13. 13 Copyright © since 1998 DMM All Rights Reserved. 1SFTUP
    
    BOBMZUJDT
    QSFTUP
     ෼ੳ༻Ϋϥελ 1SFTUP
    
    
    DPPSE
    
    XPSLFS
    
    
    
    RVFSZ
    
    EBZ
    CZ
    BOBMZTU
    
    FOHJOFFS
    FWFSZ
    
    NJO
    GPS
    DIFDL
    TMPX
    RVFSZ
    TFOE
    RVFSZ
    MPH
    CZ
    qVFOUE 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
14. 14 Copyright © since 1998 DMM All Rights Reserved. MPHHJOH
    RVFSZ
    MPH
    PO
    1SFTUP 1SFTUP ΞυϗοΫΫΤϦΛՄࢹԽ͍ͨ͠ IUUQTHJUIVCDPN[[QSFTUPBVEJU
15. 15 Copyright © since 1998 DMM All Rights Reserved. 3FEBTI
    
    #*
    UPPM
     #*πʔϧ
    3FEBTI
     C
    JOUFHSBUF
    HPPHMF
    PBVUI ;FQQFMJO 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! 3FEBTI
16. 16 Copyright © since 1998 DMM All Rights Reserved. JOUSPEVDF
    VTFS
    UP
    1SFTUP 3FEBTI QBUDIΛ౰ͯͯΫΤϦͷ࣮ߦϢʔβʔ൑ผ IUUQTHJUIVCDPNHFUSFEBTISFEBTIJTTVFT
17. 17 Copyright © since 1998 DMM All Rights Reserved. "QBDIF
    ;FQQFMJO
    
    OPUFCPPL
     ΞφϦετ༻OPUFCPPL 7FSTJPO
    4/"14)05
    
    VTF
    1SFTUP
    WJB
    KECD OPU
    TQBSL
    OPU
    ZFU
    JOUFHSBUF
    HPPHMF
    PBVUI 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
18. 18 Copyright © since 1998 DMM All Rights Reserved. JOUSPEVDF
    VTFS
    UP
    QSFTUP "QBDIF
    ;FQQFMJO ;FQQFMJOͰར༻͍ͯ͠ΔBQBDIF
    TIJSPͱQBDKͷ૊Έ߹ ΘͤʹΑΓɺ(PPHMF
    0"VUIΛར༻Մೳ
19. 19 Copyright © since 1998 DMM All Rights Reserved. &',
    
    FMBTUJD
    TFBSDI
    
    qVFOUE
    
    LJCBOB
    
     ෼ੳΫΤϦͷՄࢹԽ ར༻ঢ়گͷϞχλϦϯά
    -VDFOF
    2VFSZʹΑΓɺҟৗΫΤϦΛݕࡧ 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
20. 20 Copyright © since 1998 DMM All Rights Reserved. NPOJUPS
    
    
    
    TUBUJTUJDT &', QSFTUPͷར༻ঢ়گΛϞχλϦϯά
21. 21 Copyright © since 1998 DMM All Rights Reserved. 1SFTUP
    
    CBUDI
    QSFTUP
     όον༻Ϋϥελ 1SFTUP
    
    
    DPPSE
    
    XPSLFS
    ʙ
    RVFSZ
    
    EBZ
    OFFET
    IJHI
    BWBJMBCJMJUZ 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
22. 22 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
23. 23 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
24. 24 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
25. 25 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
26. 26 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ# DPPSEJOBUPSͷϑΣΠϧΦʔόʔ΋ରԠՄೳ "
27. 27 Copyright © since 1998 DMM All Rights Reserved. CBUDI ֤छσʔλఏڙͷͨΊͷόονॲཧ $SFBUF
    UBCMF
    BT
    TFMFDU
    Ͱ
    NBSJBECʹσʔλసૹ
    1SFTUP
    DPPSEJOBUPSͷ੾Γସ͑Λߟྀ͠ɺϦτϥΠΛ࣮૷ 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
28. 28 Copyright © since 1998 DMM All Rights Reserved. $5"4
    CZ
    XJUI
    CMPDL CBUDI with test as ( select 1 num ) create table hive.adhoc.ctas_test_ng as select * from test ; create table hive.adhoc.ctas_test_ok as with test as ( select 1 num ) select * from test ; XJUI۟ΛDSFBUFจͰ࢖͏৔߹͸ɺઌʹDSFBUFΛॻ͘ /(
    0,
    with test as ( select 2 num ) insert into hive.adhoc.ctas_test_ok select * from test ; insert into hive.adhoc.ctas_test_ok with test as ( select 2 num ) select * from test ;
29. 29 Copyright © since 1998 DMM All Rights Reserved. ͦͷଞ TMBDL
    
    1SFTUPͰTMPX
    RVFSZͷݕ஌͓ΑͼDBODFM TZTUFNSVOUJNFRVFSJFTΛࢀর͠ɺ͖͍͠஋Ҏ্ͷ௕࣌ؒ Քಇ͍ͯ͠ΔΫΤϦΛݕ஌
    ˠ
    #TMBDL
    ର৅ΫΤϦΛ֬ೝ͠ɺDBODFM 1SFTUP FMBTUJD
    TFBSDI
    LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
30. 30 Copyright © since 1998 DMM All Rights Reserved. DIFDL
    
    LJMM
    TMPX
    RVFSZ TMBDL
    XFC
    
    )VCPU ࣌ؒͷ͔͔͍ͬͯΔฅΫΤϦͷݕ஌ఀࢭ
31. 31 Copyright © since 1998 DMM All Rights Reserved. 1SFTUPϦιʔεͷ׆༻ ໷ؒଳͷXPSLFSϊʔυར༻཰޲্
32. 32 Copyright © since 1998 DMM All Rights Reserved. ֤1SFTUPΫϥελͷར༻ঢ়گ ෼ੳ1SFTUPΫϥελ
    DPPSE
    
    XPSLFS
    ΄΅೔தͷΈར༻ όον1SFTUPΫϥελ
    DPPSE
    
    XPSLFS
    ओʹ໷ؒʹར༻ ໷ؒଳ͸෼ੳ1SFTUPΫϥελΛόον༻్ʹస༻͍ͨ͠
33. 33 Copyright © since 1998 DMM All Rights Reserved. TXJUDI
    DPPSEJOBUPS
    CZ
    )"1SPYZ ೔தଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" ෼ੳ όον
34. 34 Copyright © since 1998 DMM All Rights Reserved. TXJUDI
    DPPSEJOBUPS
    CZ
    )"1SPYZ ໷ؒଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" )"1SPYZ
    $
    )"1SPYZʹϓϩΩγͯ͠
 ෼ੳ1SFTUPͷXPSLFSΛόονʹͿΒԼ͛ ෼ੳ όον
35. 35 Copyright © since 1998 DMM All Rights Reserved. TXJUDI
    DPPSEJOBUPS
    CZ
    )"1SPYZ ˞஫ҙ఺˞ OPEFFOWJSPONFOU͕ҧ͏XPSLFS͸ೝࣝ͞Εͳ͍
    ˠ
    QSPQFSUJFTΛมߋ͠ɺ࠶ىಈ͕ඞཁ # node.properties node.environment=ANALYTICS_PROD node.id=xxx node.data-dir=/var/presto/data # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data ෼ੳ # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data όον
36. 36 Copyright © since 1998 DMM All Rights Reserved. ·ͱΊ
37. 37 Copyright © since 1998 DMM All Rights Reserved. ·ͱΊ #*πʔϧͱ1SFTUPΛ࿈ܞ 1SFTUPͷར༻ঢ়گΛ೺Ѳ͠ɺΤϯδχΞ͕ద੾ʹରॲՄೳ )"1SPYZʹΑΓ1SFTUP
    DPPSEJOBUPSͷ410'Λআڈ CBUDIͳͲγεςϜ༻్ͷΫϥελͷՄ༻ੑ޲্ ֤Ϋϥελͷར༻ঢ়گʹԠ͡XPSLFSͷৼΓ෼͚ Ϧιʔεͷޮ཰తͳར༻
38. ͍͞͝ʹʜ
39. 39 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO 1SFTUP SFEBTI FMBTUJD
    TFBSDI
    LJCBOB IBEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! QSFTUP
    
    BOBMZUJDT
    QSFTUP
     ΞυϗοΫΫΤϦ༻Ϋϥελ 
    DPSE
    
    XPSLFS
    
    
    
    RVFSZ
    
    EBZ
    CZ
    BOBMZTU
    
    FOHJOFFS
    FWFSZ
    
    NJO
    GPS
    DIFDL
    TMPX
    RVFSZ
    TFOE
    RVFSZ
    MPH
    CZ
    qVFOUE
40. 40 Copyright © since 1998 DMM All Rights Reserved. ֤QSFTUPΫϥελͷར༻ঢ়گ ෼ੳ1SFTUPΫϥελ
    DPPSE
    
    XPSLFS
    ΄΅೔தͷΈར༻ όον1SFTUPΫϥελ
    DPPSE
    
    XPSLFS
    ओʹ໷ؒʹར༻ ໷ؒଳ͸෼ੳ1SFTUPΫϥελΛόον༻్ʹస༻͍ͨ͠
41. 41 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO QSFTUP SFEBTI FMBTUJD
    TFBSDI
    LJCBOB IBEPPQ CBUDI NBSJBEC BQJ QSFTUP ! QSFTUP
    
    BOBMZUJDT
    QSFTUP
     ΞυϗοΫΫΤϦ༻Ϋϥελ 
    DPSE
    
    XPSLFS
    
    
    
    RVFSZ
    
    EBZ
    CZ
    BOBMZTU
    
    FOHJOFFS
    FWFSZ
    
    NJO
    GPS
    DIFDL
    TMPX
    RVFSZ
    TFOE
    RVFSZ
    MPH
    CZ
    qVFOUE ࡏ੶ΞφϦετɿਓ
    ʘ ?P? ʗ
    ʺΤϯδχΞ
42. We are Hiring! Ϗοάσʔλ෦Ͱ͸ΤϯδχΞͷΈͰͳ͘ɺΞφϦετ΋ੵۃืूதͰ͢ʂ