Save 37% off PRO during our Black Friday Sale! »

Elasticsearch Cluster Deployment Model

Elasticsearch Cluster Deployment Model

Elasticsearch クラスターの構成例を紹介します。
- 開発環境
- 最小プロダクション環境
- 専任マスターノード
- クライアントノード
- Ingest ノード
- Logstash
- ES Hadoop
- Tribe ノード
- モニタリング専用クラスター

34cbde72de5f384380d5489543294dc5?s=128

Kosho Owa

May 30, 2016
Tweet

Transcript

  1. ‹#› Kosho Owa, Solutions Architect, Elastic May 2016 &MBTUJDTFBSDIΫϥελʔ σϓϩΠϝϯτϞσϧ

  2. 2 γϯάϧΫϥελʔ

  3. ։ൃ؀ڥ 3 ES node Apps ES clients, Kibana, Logstash, Beats,

    User apps and etc.
  4. Cluster ϓϩμΫγϣϯͰͷ࠷খߏ੒ 4 ES node Apps ES node ES node

    ES clients, Kibana, Logstash, Beats, User apps and etc.
  5. Cluster ઐ೚Ϛελʔϊʔυ 5 Data node Apps Master node Data node

    Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master Node discovery.zen.minimum_master_nodes: 1 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 1 node.master: false
  6. Cluster ৑௕Ϛελʔϊʔυ 6 Data node Apps Master node Data node

    Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master-eligible Nodes discovery.zen.minimum_master_nodes: 2 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 2 node.master: false
  7. Cluster ΫϥΠΞϯτϊʔυ 7 Data node Apps Master node Data node

    Data node Master node Master node Data node Data node Client node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Client Node node.master: false node.data: false
  8. Cluster *OHFTUϊʔυ GSPN7 8 Data node Apps Master node Data

    node Data node Master node Master node Data node Data node Client node Data shippers Ingest node Logstash, Beats and etc. Kibana, User apps and etc. # config/elasticsearch.yml # Client/Ingest Node node.master: false node.data: false
  9. Cluster -PHTUBTI 9 Data shippers/ sources Logstash nodes ES nodes

    Logstash, Beats, syslog, file and etc.
  10. Cluster Cluster Message queues Cluster ଱σʔλαʔδ 10 Data shippers Logstash

    nodes ES nodes Logstash, Beats and etc. Redis, Kafka or RabbitMQ
  11. Cluster &4)BEPPQ 11 ES nodes ES Hadoop HDFS Front-ends Storm,

    Spark, MapReduce, Cascading, Hive and Pig
  12. 12 Tribeϊʔυ

  13. 5SJCFϊʔυͱ͸ • ෳ਺ͷΫϥελʔʹ·͕ͨΔΞΫηεΛఏڙ͢ΔΫϥΠΞϯτ ‒ ಡΈॻ͖ ‒ ౷߹తͳαʔν • $MVTUFS4UBUFΠϯσοΫεͷϚʔδ •

    ղܾ͢Δ໰୊ ‒ ஍ཧతʹ཭Εͨ৔ॴʹଘࡏ͢ΔΫϥελʔͷӡ༻ ‒ σʔλαΠϩ ෳ਺ͷσʔλ؅ཧऀ 13
  14. Cluster 5SJCFϊʔυ 14 Apps Tribe node ES nodes Cluster ES

    nodes Kibana, User apps and etc.
  15. ੍ݶࣄ߲ • ΠϯσοΫε໊͸͢΂ͯͷΫϥελʔʹΘͨͬͯϢχʔΫͰ͋Δ͜ͱ ‒ ಉҰ໊শͷΠϯσοΫεΛϚʔδ͢ΔΑ͏ͳಈ࡞͸ߦΘΕͳ͍ ‒ USJCFPO@DPOGMJDUͷઃఆʹΑΓɺBOZ ESPQ΋͘͠͸QSFGFS@<USJCF/BNF>Λબ୒Մೳ • ΫϥελʔϫΠυͰߦΘΕΔϨϕϧͷॻ͖ࠐΈΦϖϨʔγϣϯ͸ෆڐՄ

    ‒ $SFBUFJOEFY ‒ *OEFYNBQQJOHT TFUUJOHTͷมߋ 15
  16. ઃఆྫ 16 # config/elasticsearch.yml node.name: tribe marvel.enabled: false tribe: on_conflicts:

    drop ldn: cluster.name: london discovery.zen.ping.unicast.hosts: [“london-master1:9300", “london-master2:9300”, “london- master3:9300”] tyo: cluster.name: tokyo discovery.zen.ping.unicast.hosts: [“tokyo-master1:9300", “tokyo-master2:9300”, “tokyo- master3:9300”]
  17. ϓϥάΠϯͱͷ࿈ܞ • .BSWFM ‒ :PVNBZJODMVEFFYDMVEFNPOJUPSJOHEBUBGSPNUIF5SJCFOPEF • 4IJFME ‒ *OTUBMMJOH4IJFMEJTOFDFTTBSZXIFOFWFSZPVSDMVTUFSTBSFQSPUFDUFECZ4IJFME 17

    • .BSWFM%PDVNFOUBUJPO$POGJHVSJOH.BSWFM$POGJHVSJOHB5SJCF/PEFUP8PSLXJUI.BSWFM IUUQTXXXFMBTUJDDPHVJEFFONBSWFMDVSSFOUDPOGJHVSBUJPOIUNMUSJCFDPOGJH • 4IJFME*OTUBMMJOH4IJFMEIUUQTXXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUJOTUBMMJOH TIJFMEIUNMUSJCFOPEF
  18. 18 ϞχλϦϯάઐ༻ Ϋϥελʔ

  19. γϯάϧΫϥελʔ 19 Elasticsearch Cluster ES node marvel-agent ES node marvel-agent

    ES node marvel-agent Kibana marvel-ui
  20. ϞχλϦϯάઐ༻Ϋϥελʔ 20 Cluster ES node marvel-agent ES node marvel-agent ES

    node marvel-agent Monitoring Cluster ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]
  21. .BSWFMͷදࣔྫ 21  ϓϩμΫγϣϯΫϥελʔͱϞχλϦϯάΫϥ ελʔ͕දࣔ͞ΕΔ  1MVHJO$IPPTFSΑΓ.BSWFMΛબ୒

  22. ΠϯετʔϧNBSWFMBHFOU 22 $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license

    -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y
  23. ΠϯετʔϧNBSWFMVJ 23 $ cd /opt/kibana $ sudo bin/kibana plugin --install

    elasticsearch/marvel Installing marvel … Plugin installation complete
  24. ϓϥάΠϯͱͷ࿈ܞ • 8BUDIFS ‒ $MVTUFSIFBMUIDIBOHFT )JHINFNPSZVTBHF )JHIDQVVTBHF )JHIGJMFEFTDSJQUPS VTBHF )JHIGJFMEEBUBDBDIFVTBHF

    /PEFTKPJOJOHPSMFBWJOHUIFDMVTUFS • 4IJFME - ,JCBOBΞΫηε࣌ͷύεϫʔυઃఆ - 3#"$ ໾ׂϕʔεͷΞΫηε੍ޚ FHDMVTUFSNPOJUPSJOHPOMZVTFST - NBSWFMBHFOUͱNPOJUPSJOHDMVTUFSؒͷ௨৴ͷ҉߸Խ 24 • 8BUDIFS&YBNQMF8BUDIFT8BUDIJOH.BSWFM%BUBIUUQTXXXFMBTUJDDPHVJEFFO XBUDIFSDVSSFOUXBUDIJOHNBSWFMEBUBIUNM • 4IJFME$POGJHVSJOH$MJFOUTBOE*OUFHSBUJPOT6TJOH.BSWFMXJUI4IJFMEIUUQT XXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUNBSWFMIUNM