Share
Embed
Elasticsearch Cluster Deployment Model
Elasticsearch クラスターの構成例を紹介します。
- 開発環境
- 最小プロダクション環境
- 専任マスターノード
- クライアントノード
- Ingest ノード
- Logstash
- ES Hadoop
- Tribe ノード
- モニタリング専用クラスター
Kosho Owa
May 30, 2016
Transcript
24 slides
-
։ൃڥ 3 ES node Apps ES clients, Kibana, Logstash, Beats,
User apps and etc. -
Cluster ϓϩμΫγϣϯͰͷ࠷খߏ 4 ES node Apps ES node ES node
ES clients, Kibana, Logstash, Beats, User apps and etc. -
Cluster ઐϚελʔϊʔυ 5 Data node Apps Master node Data node
Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master Node discovery.zen.minimum_master_nodes: 1 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 1 node.master: false -
Cluster Ϛελʔϊʔυ 6 Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master-eligible Nodes discovery.zen.minimum_master_nodes: 2 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 2 node.master: false -
Cluster ΫϥΠΞϯτϊʔυ 7 Data node Apps Master node Data node
Data node Master node Master node Data node Data node Client node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Client Node node.master: false node.data: false -
Cluster *OHFTUϊʔυ GSPN7 8 Data node Apps Master node Data
node Data node Master node Master node Data node Data node Client node Data shippers Ingest node Logstash, Beats and etc. Kibana, User apps and etc. # config/elasticsearch.yml # Client/Ingest Node node.master: false node.data: false -
Cluster -PHTUBTI 9 Data shippers/ sources Logstash nodes ES nodes
Logstash, Beats, syslog, file and etc. -
Cluster Cluster Message queues Cluster σʔλαʔδ 10 Data shippers Logstash
nodes ES nodes Logstash, Beats and etc. Redis, Kafka or RabbitMQ -
Cluster &4)BEPPQ 11 ES nodes ES Hadoop HDFS Front-ends Storm,
Spark, MapReduce, Cascading, Hive and Pig -
5SJCFϊʔυͱ • ෳͷΫϥελʔʹ·͕ͨΔΞΫηεΛఏڙ͢ΔΫϥΠΞϯτ ‒ ಡΈॻ͖ ‒ ౷߹తͳαʔν • $MVTUFS4UBUFΠϯσοΫεͷϚʔδ •
ղܾ͢Δ ‒ ཧతʹΕͨॴʹଘࡏ͢ΔΫϥελʔͷӡ༻ ‒ σʔλαΠϩ ෳͷσʔλཧऀ 13 -
Cluster 5SJCFϊʔυ 14 Apps Tribe node ES nodes Cluster ES
nodes Kibana, User apps and etc. -
੍ݶࣄ߲ • ΠϯσοΫε໊ͯ͢ͷΫϥελʔʹΘͨͬͯϢχʔΫͰ͋Δ͜ͱ ‒ ಉҰ໊শͷΠϯσοΫεΛϚʔδ͢ΔΑ͏ͳಈ࡞ߦΘΕͳ͍ ‒ USJCFPO@DPOGMJDUͷઃఆʹΑΓɺBOZ ESPQ͘͠QSFGFS@<USJCF/BNF>ΛબՄೳ • ΫϥελʔϫΠυͰߦΘΕΔϨϕϧͷॻ͖ࠐΈΦϖϨʔγϣϯෆڐՄ
‒ $SFBUFJOEFY ‒ *OEFYNBQQJOHT TFUUJOHTͷมߋ 15 -
ઃఆྫ 16 # config/elasticsearch.yml node.name: tribe marvel.enabled: false tribe: on_conflicts:
drop ldn: cluster.name: london discovery.zen.ping.unicast.hosts: [“london-master1:9300", “london-master2:9300”, “london- master3:9300”] tyo: cluster.name: tokyo discovery.zen.ping.unicast.hosts: [“tokyo-master1:9300", “tokyo-master2:9300”, “tokyo- master3:9300”] -
ϓϥάΠϯͱͷ࿈ܞ • .BSWFM ‒ :PVNBZJODMVEFFYDMVEFNPOJUPSJOHEBUBGSPNUIF5SJCFOPEF • 4IJFME ‒ *OTUBMMJOH4IJFMEJTOFDFTTBSZXIFOFWFSZPVSDMVTUFSTBSFQSPUFDUFECZ4IJFME 17
• .BSWFM%PDVNFOUBUJPO$POGJHVSJOH.BSWFM$POGJHVSJOHB5SJCF/PEFUP8PSLXJUI.BSWFM IUUQTXXXFMBTUJDDPHVJEFFONBSWFMDVSSFOUDPOGJHVSBUJPOIUNMUSJCFDPOGJH • 4IJFME*OTUBMMJOH4IJFMEIUUQTXXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUJOTUBMMJOH TIJFMEIUNMUSJCFOPEF -
γϯάϧΫϥελʔ 19 Elasticsearch Cluster ES node marvel-agent ES node marvel-agent
ES node marvel-agent Kibana marvel-ui -
ϞχλϦϯάઐ༻Ϋϥελʔ 20 Cluster ES node marvel-agent ES node marvel-agent ES
node marvel-agent Monitoring Cluster ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…] -
ΠϯετʔϧNBSWFMBHFOU 22 $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license
-> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -
ΠϯετʔϧNBSWFMVJ 23 $ cd /opt/kibana $ sudo bin/kibana plugin --install
elasticsearch/marvel Installing marvel … Plugin installation complete -
ϓϥάΠϯͱͷ࿈ܞ • 8BUDIFS ‒ $MVTUFSIFBMUIDIBOHFT )JHINFNPSZVTBHF )JHIDQVVTBHF )JHIGJMFEFTDSJQUPS VTBHF )JHIGJFMEEBUBDBDIFVTBHF
/PEFTKPJOJOHPSMFBWJOHUIFDMVTUFS • 4IJFME - ,JCBOBΞΫηε࣌ͷύεϫʔυઃఆ - 3#"$ ׂϕʔεͷΞΫηε੍ޚ FHDMVTUFSNPOJUPSJOHPOMZVTFST - NBSWFMBHFOUͱNPOJUPSJOHDMVTUFSؒͷ௨৴ͷ҉߸Խ 24 • 8BUDIFS&YBNQMF8BUDIFT8BUDIJOH.BSWFM%BUBIUUQTXXXFMBTUJDDPHVJEFFO XBUDIFSDVSSFOUXBUDIJOHNBSWFMEBUBIUNM • 4IJFME$POGJHVSJOH$MJFOUTBOE*OUFHSBUJPOT6TJOH.BSWFMXJUI4IJFMEIUUQT XXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUNBSWFMIUNM