Elasticsearch Cluster Deployment Model

Slide 1

Slide 1 text

‹#› Kosho Owa, Solutions Architect, Elastic May 2016 &MBTUJDTFBSDIΫϥελʔ σϓϩΠϝϯτϞσϧ

Slide 2

Slide 2 text

2 γϯάϧΫϥελʔ

Slide 3

Slide 3 text

։ൃ؀ڥ 3 ES node Apps ES clients, Kibana, Logstash, Beats, User apps and etc.

Slide 4

Slide 4 text

Cluster ϓϩμΫγϣϯͰͷ࠷খߏ੒ 4 ES node Apps ES node ES node ES clients, Kibana, Logstash, Beats, User apps and etc.

Slide 5

Slide 5 text

Cluster ઐ೚Ϛελʔϊʔυ 5 Data node Apps Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master Node discovery.zen.minimum_master_nodes: 1 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 1 node.master: false

Slide 6

Slide 6 text

Cluster ৑௕Ϛελʔϊʔυ 6 Data node Apps Master node Data node Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master-eligible Nodes discovery.zen.minimum_master_nodes: 2 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 2 node.master: false

Slide 7

Slide 7 text

Cluster ΫϥΠΞϯτϊʔυ 7 Data node Apps Master node Data node Data node Master node Master node Data node Data node Client node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Client Node node.master: false node.data: false

Slide 8

Slide 8 text

Cluster *OHFTUϊʔυ GSPN7 8 Data node Apps Master node Data node Data node Master node Master node Data node Data node Client node Data shippers Ingest node Logstash, Beats and etc. Kibana, User apps and etc. # config/elasticsearch.yml # Client/Ingest Node node.master: false node.data: false

Slide 9

Slide 9 text

Cluster -PHTUBTI 9 Data shippers/ sources Logstash nodes ES nodes Logstash, Beats, syslog, file and etc.

Slide 10

Slide 10 text

Cluster Cluster Message queues Cluster ଱σʔλαʔδ 10 Data shippers Logstash nodes ES nodes Logstash, Beats and etc. Redis, Kafka or RabbitMQ

Slide 11

Slide 11 text

Cluster &4)BEPPQ 11 ES nodes ES Hadoop HDFS Front-ends Storm, Spark, MapReduce, Cascading, Hive and Pig

Slide 12

Slide 12 text

12 Tribeϊʔυ

Slide 13

Slide 13 text

5SJCFϊʔυͱ͸ • ෳ਺ͷΫϥελʔʹ·͕ͨΔΞΫηεΛఏڙ͢ΔΫϥΠΞϯτ ‒ ಡΈॻ͖ ‒ ౷߹తͳαʔν • $MVTUFS4UBUFΠϯσοΫεͷϚʔδ • ղܾ͢Δ໰୊ ‒ ஍ཧతʹ཭Εͨ৔ॴʹଘࡏ͢ΔΫϥελʔͷӡ༻ ‒ σʔλαΠϩ ෳ਺ͷσʔλ؅ཧऀ 13

Slide 14

Slide 14 text

Cluster 5SJCFϊʔυ 14 Apps Tribe node ES nodes Cluster ES nodes Kibana, User apps and etc.

Slide 15

Slide 15 text

੍ݶࣄ߲ • ΠϯσοΫε໊͸͢΂ͯͷΫϥελʔʹΘͨͬͯϢχʔΫͰ͋Δ͜ͱ ‒ ಉҰ໊শͷΠϯσοΫεΛϚʔδ͢ΔΑ͏ͳಈ࡞͸ߦΘΕͳ͍ ‒ USJCFPO@DPOGMJDUͷઃఆʹΑΓɺBOZ ESPQ΋͘͠͸QSFGFS@Λબ୒Մೳ • ΫϥελʔϫΠυͰߦΘΕΔϨϕϧͷॻ͖ࠐΈΦϖϨʔγϣϯ͸ෆڐՄ ‒ $SFBUFJOEFY ‒ *OEFYNBQQJOHT TFUUJOHTͷมߋ 15

Slide 16

Slide 16 text

ઃఆྫ 16 # config/elasticsearch.yml node.name: tribe marvel.enabled: false tribe: on_conflicts: drop ldn: cluster.name: london discovery.zen.ping.unicast.hosts: [“london-master1:9300", “london-master2:9300”, “london- master3:9300”] tyo: cluster.name: tokyo discovery.zen.ping.unicast.hosts: [“tokyo-master1:9300", “tokyo-master2:9300”, “tokyo- master3:9300”]

Slide 17

Slide 17 text

ϓϥάΠϯͱͷ࿈ܞ • .BSWFM ‒ :PVNBZJODMVEFFYDMVEFNPOJUPSJOHEBUBGSPNUIF5SJCFOPEF • 4IJFME ‒ *OTUBMMJOH4IJFMEJTOFDFTTBSZXIFOFWFSZPVSDMVTUFSTBSFQSPUFDUFECZ4IJFME 17 • .BSWFM%PDVNFOUBUJPO$POGJHVSJOH.BSWFM$POGJHVSJOHB5SJCF/PEFUP8PSLXJUI.BSWFM IUUQTXXXFMBTUJDDPHVJEFFONBSWFMDVSSFOUDPOGJHVSBUJPOIUNMUSJCFDPOGJH • 4IJFME*OTUBMMJOH4IJFMEIUUQTXXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUJOTUBMMJOH TIJFMEIUNMUSJCFOPEF

Slide 18

Slide 18 text

18 ϞχλϦϯάઐ༻ Ϋϥελʔ

Slide 19

Slide 19 text

γϯάϧΫϥελʔ 19 Elasticsearch Cluster ES node marvel-agent ES node marvel-agent ES node marvel-agent Kibana marvel-ui

Slide 20

Slide 20 text

ϞχλϦϯάઐ༻Ϋϥελʔ 20 Cluster ES node marvel-agent ES node marvel-agent ES node marvel-agent Monitoring Cluster ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]

Slide 21

Slide 21 text

.BSWFMͷදࣔྫ 21 ϓϩμΫγϣϯΫϥελʔͱϞχλϦϯάΫϥ ελʔ͕දࣔ͞ΕΔ 1MVHJO$IPPTFSΑΓ.BSWFMΛબ୒

Slide 22

Slide 22 text

ΠϯετʔϧNBSWFMBHFOU 22 $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y

Slide 23

Slide 23 text

ΠϯετʔϧNBSWFMVJ 23 $ cd /opt/kibana $ sudo bin/kibana plugin --install elasticsearch/marvel Installing marvel … Plugin installation complete

Slide 24

Slide 24 text

ϓϥάΠϯͱͷ࿈ܞ • 8BUDIFS ‒ $MVTUFSIFBMUIDIBOHFT )JHINFNPSZVTBHF )JHIDQVVTBHF )JHIGJMFEFTDSJQUPS VTBHF )JHIGJFMEEBUBDBDIFVTBHF /PEFTKPJOJOHPSMFBWJOHUIFDMVTUFS • 4IJFME - ,JCBOBΞΫηε࣌ͷύεϫʔυઃఆ - 3#"$ ໾ׂϕʔεͷΞΫηε੍ޚ FHDMVTUFSNPOJUPSJOHPOMZVTFST - NBSWFMBHFOUͱNPOJUPSJOHDMVTUFSؒͷ௨৴ͷ҉߸Խ 24 • 8BUDIFS&YBNQMF8BUDIFT8BUDIJOH.BSWFM%BUBIUUQTXXXFMBTUJDDPHVJEFFO XBUDIFSDVSSFOUXBUDIJOHNBSWFMEBUBIUNM • 4IJFME$POGJHVSJOH$MJFOUTBOE*OUFHSBUJPOT6TJOH.BSWFMXJUI4IJFMEIUUQT XXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUNBSWFMIUNM