DMMでのPrestoおよび周辺エコシステムの運用事例

Slide 1

Slide 1 text

%..DPNϥϘ౻Ҫ྄ଠ 1SFTUP.FFUVQ %..Ͱͷ1SFTUP͓Αͼ पลΤίγεςϜͷӡ༻ࣄྫ

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

9 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤBOBMZUJDT 1SFTUP ;FQQFMJO 1SFTUP 3FEBTI &MBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w ΞφϦετ͕೚ҙͷΫΤϦΛ ࣮ߦ͢Δ؀ڥ w 1SFTUPΫΤϦ͸SFEBTI [FQQFMJO͔Β࣮ߦ͢Δ w ௕ظՔಇ͍ͯ͠ΔΫΤϦ͸ɺ ΤϯδχΞ͕؂ࢹ͠ɺঢ়گ ʹԠ͡Ωϟϯηϧ͢Δ ෼ੳ؀ڥ

Slide 10

Slide 10 text

10 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤCBUDI 1SFTUP [FQQFMJO 1SFTUP SFEBTI &MBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w γεςϜ͔Βൃߦ͞ΕΔΫΤ Ϧͷ࣮ߦ؀ڥ w IBEPPQ্ͷσʔλΛBQJܦ ༝ͰଞγεςϜʹ࿈ܞ͢Δ ͨΊͷநग़ॲཧΛߦ͏  IBEPPQʹର͢Δ&5-͸ߦ Θͳ͍ w ఆܕతͳΫΤϦ͕ϝΠϯ όον؀ڥ

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

13 Copyright © since 1998 DMM All Rights Reserved. 1SFTUPBOBMZUJDTQSFTUP ෼ੳ༻Ϋϥελ 1SFTUP DPPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

17 Copyright © since 1998 DMM All Rights Reserved. "QBDIF;FQQFMJOOPUFCPPL ΞφϦετ༻OPUFCPPL 7FSTJPO4/"14)05 VTF1SFTUPWJBKECD OPUTQBSL OPUZFUJOUFHSBUFHPPHMFPBVUI 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 18

Slide 18 text

Slide 19

Slide 19 text

19 Copyright © since 1998 DMM All Rights Reserved. &',FMBTUJDTFBSDIqVFOUELJCBOB ෼ੳΫΤϦͷՄࢹԽ ར༻ঢ়گͷϞχλϦϯά -VDFOF2VFSZʹΑΓɺҟৗΫΤϦΛݕࡧ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 20

Slide 20 text

Slide 21

Slide 21 text

21 Copyright © since 1998 DMM All Rights Reserved. 1SFTUPCBUDIQSFTUP όον༻Ϋϥελ 1SFTUP DPPSE XPSLFS ʙRVFSZEBZ OFFETIJHIBWBJMBCJMJUZ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 22

Slide 22 text

22 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#

Slide 23

Slide 23 text

23 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#

Slide 24

Slide 24 text

24 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#

Slide 25

Slide 25 text

25 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#

Slide 26

Slide 26 text

26 Copyright © since 1998 DMM All Rights Reserved. NVMUJ DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ# DPPSEJOBUPSͷϑΣΠϧΦʔόʔ΋ରԠՄೳ "

Slide 27

Slide 27 text

27 Copyright © since 1998 DMM All Rights Reserved. CBUDI ֤छσʔλఏڙͷͨΊͷόονॲཧ $SFBUFUBCMFBTTFMFDUͰNBSJBECʹσʔλసૹ 1SFTUPDPPSEJOBUPSͷ੾Γସ͑Λߟྀ͠ɺϦτϥΠΛ࣮૷ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 28

Slide 28 text

28 Copyright © since 1998 DMM All Rights Reserved. $5"4CZ XJUICMPDL CBUDI with test as ( select 1 num ) create table hive.adhoc.ctas_test_ng as select * from test ; create table hive.adhoc.ctas_test_ok as with test as ( select 1 num ) select * from test ; XJUI۟ΛDSFBUFจͰ࢖͏৔߹͸ɺઌʹDSFBUFΛॻ͘ /( 0, with test as ( select 2 num ) insert into hive.adhoc.ctas_test_ok select * from test ; insert into hive.adhoc.ctas_test_ok with test as ( select 2 num ) select * from test ;

Slide 29

Slide 29 text

29 Copyright © since 1998 DMM All Rights Reserved. ͦͷଞ TMBDL1SFTUPͰTMPXRVFSZͷݕ஌͓ΑͼDBODFM TZTUFNSVOUJNFRVFSJFTΛࢀর͠ɺ͖͍͠஋Ҏ্ͷ௕࣌ؒ Քಇ͍ͯ͠ΔΫΤϦΛݕ஌ˠ#TMBDL ର৅ΫΤϦΛ֬ೝ͠ɺDBODFM 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

33 Copyright © since 1998 DMM All Rights Reserved. TXJUDI DPPSEJOBUPS CZ)"1SPYZ ೔தଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" ෼ੳ όον

Slide 34

Slide 34 text

34 Copyright © since 1998 DMM All Rights Reserved. TXJUDI DPPSEJOBUPS CZ)"1SPYZ ໷ؒଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" )"1SPYZ$)"1SPYZʹϓϩΩγͯ͠  ෼ੳ1SFTUPͷXPSLFSΛόονʹͿΒԼ͛ ෼ੳ όον

Slide 35

Slide 35 text

35 Copyright © since 1998 DMM All Rights Reserved. TXJUDI DPPSEJOBUPS CZ)"1SPYZ ˞஫ҙ఺˞ OPEFFOWJSPONFOU͕ҧ͏XPSLFS͸ೝࣝ͞Εͳ͍ ˠQSPQFSUJFTΛมߋ͠ɺ࠶ىಈ͕ඞཁ # node.properties node.environment=ANALYTICS_PROD node.id=xxx node.data-dir=/var/presto/data # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data ෼ੳ # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data όον

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

͍͞͝ʹʜ

Slide 39

Slide 39 text

39 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO 1SFTUP SFEBTI FMBTUJDTFBSDI LJCBOB IBEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! QSFTUPBOBMZUJDTQSFTUP ΞυϗοΫΫΤϦ༻Ϋϥελ DPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE

Slide 40

Slide 40 text

Slide 41

Slide 41 text

41 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO QSFTUP SFEBTI FMBTUJDTFBSDI LJCBOB IBEPPQ CBUDI NBSJBEC BQJ QSFTUP ! QSFTUPBOBMZUJDTQSFTUP ΞυϗοΫΫΤϦ༻Ϋϥελ DPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE ࡏ੶ΞφϦετɿਓʘ ?P? ʗ ʺΤϯδχΞ

Slide 42

Slide 42 text

We are Hiring! Ϗοάσʔλ෦Ͱ͸ΤϯδχΞͷΈͰͳ͘ɺΞφϦετ΋ੵۃืूதͰ͢ʂ