Elasticsearch Cluster Deployment Model

Elasticsearch クラスターの構成例を紹介します。
- 開発環境
- 最小プロダクション環境
- 専任マスターノード
- クライアントノード
- Ingest ノード
- Logstash
- ES Hadoop
- Tribe ノード
- モニタリング専用クラスター

Transcript

1. ‹#› Kosho Owa, Solutions Architect, Elastic May 2016 &MBTUJDTFBSDI
   Ϋϥελʔ
   σϓϩΠϝϯτϞσϧ
2. 2 γϯάϧΫϥελʔ
3. ։ൃ؀ڥ 3 ES node Apps ES clients, Kibana, Logstash, Beats, User apps and etc.
4. Cluster ϓϩμΫγϣϯͰͷ࠷খߏ੒ 4 ES node Apps ES node ES node ES clients, Kibana, Logstash, Beats, User apps and etc.
5. Cluster ઐ೚Ϛελʔϊʔυ 5 Data node Apps Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master Node discovery.zen.minimum_master_nodes: 1 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 1 node.master: false
6. Cluster ৑௕Ϛελʔϊʔυ 6 Data node Apps Master node Data node Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master-eligible Nodes discovery.zen.minimum_master_nodes: 2 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 2 node.master: false
7. Cluster ΫϥΠΞϯτϊʔυ 7 Data node Apps Master node Data node Data node Master node Master node Data node Data node Client node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Client Node node.master: false node.data: false
8. Cluster *OHFTU
   ϊʔυ
   GSPN
   7 8 Data node Apps Master node Data node Data node Master node Master node Data node Data node Client node Data shippers Ingest node Logstash, Beats and etc. Kibana, User apps and etc. # config/elasticsearch.yml # Client/Ingest Node node.master: false node.data: false
9. Cluster -PHTUBTI 9 Data shippers/ sources Logstash nodes ES nodes Logstash, Beats, syslog, file and etc.
10. Cluster Cluster Message queues Cluster ଱σʔλαʔδ 10 Data shippers Logstash nodes ES nodes Logstash, Beats and etc. Redis, Kafka or RabbitMQ
11. Cluster &4
    )BEPPQ 11 ES nodes ES Hadoop HDFS Front-ends Storm, Spark, MapReduce, Cascading, Hive and Pig
12. 12 Tribeϊʔυ
13. 5SJCFϊʔυͱ͸ • ෳ਺ͷΫϥελʔʹ·͕ͨΔΞΫηεΛఏڙ͢ΔΫϥΠΞϯτ
    ‒ ಡΈॻ͖
    ‒ ౷߹తͳαʔν
    • $MVTUFS
    4UBUF
    ΠϯσοΫεͷϚʔδ
    • ղܾ͢Δ໰୊
    ‒ ஍ཧతʹ཭Εͨ৔ॴʹଘࡏ͢ΔΫϥελʔͷӡ༻
    ‒ σʔλαΠϩ
    ෳ਺ͷσʔλ؅ཧऀ 13
14. Cluster 5SJCFϊʔυ 14 Apps Tribe node ES nodes Cluster ES nodes Kibana, User apps and etc.
15. ੍ݶࣄ߲ • ΠϯσοΫε໊͸͢΂ͯͷΫϥελʔʹΘͨͬͯϢχʔΫͰ͋Δ͜ͱ
    ‒ ಉҰ໊শͷΠϯσοΫεΛϚʔδ͢ΔΑ͏ͳಈ࡞͸ߦΘΕͳ͍
    ‒ USJCFPO@DPOGMJDU
    ͷઃఆʹΑΓɺBOZ
    ESPQ
    ΋͘͠͸
    QSFGFS@<USJCF/BNF>
    Λબ୒Մೳ
    • ΫϥελʔϫΠυͰߦΘΕΔϨϕϧͷॻ͖ࠐΈΦϖϨʔγϣϯ͸ෆڐՄ
    ‒ $SFBUF
    JOEFY
    ‒ *OEFY
    NBQQJOHT
    TFUUJOHT
    ͷมߋ 15
16. ઃఆྫ 16 # config/elasticsearch.yml node.name: tribe marvel.enabled: false tribe: on_conflicts: drop ldn: cluster.name: london discovery.zen.ping.unicast.hosts: [“london-master1:9300", “london-master2:9300”, “london- master3:9300”] tyo: cluster.name: tokyo discovery.zen.ping.unicast.hosts: [“tokyo-master1:9300", “tokyo-master2:9300”, “tokyo- master3:9300”]
17. ϓϥάΠϯͱͷ࿈ܞ • .BSWFM
    ‒ :PV
    NBZ
    JODMVEFFYDMVEF
    NPOJUPSJOH
    EBUB
    GSPN
    UIF
    5SJCF
    OPEF
    • 4IJFME
    ‒ *OTUBMMJOH
    4IJFME
    JT
    OFDFTTBSZ
    XIFOFWFS
    ZPVS
    DMVTUFST
    BSF
    QSPUFDUFE
    CZ
    4IJFME 17 • .BSWFM
    %PDVNFOUBUJPO
    
    $POGJHVSJOH
    .BSWFM
    
    $POGJHVSJOH
    B
    5SJCF
    /PEF
    UP
    8PSL
    XJUI
    .BSWFM
    
    
    IUUQTXXXFMBTUJDDPHVJEFFONBSWFMDVSSFOUDPOGJHVSBUJPOIUNMUSJCFDPOGJH • 4IJFME
    
    *OTUBMMJOH
    4IJFME
    
    IUUQTXXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUJOTUBMMJOH TIJFMEIUNMUSJCFOPEF
18. 18 ϞχλϦϯάઐ༻ Ϋϥελʔ
19. γϯάϧΫϥελʔ 19 Elasticsearch Cluster ES node marvel-agent ES node marvel-agent ES node marvel-agent Kibana marvel-ui
20. ϞχλϦϯάઐ༻Ϋϥελʔ 20 Cluster ES node marvel-agent ES node marvel-agent ES node marvel-agent Monitoring Cluster ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]
21. .BSWFM
    ͷදࣔྫ 21  ϓϩμΫγϣϯΫϥελʔͱϞχλϦϯάΫϥ ελʔ͕දࣔ͞ΕΔ  1MVHJO
    $IPPTFS
    ΑΓ
    .BSWFM
    Λબ୒
22. Πϯετʔϧ
    
    NBSWFMBHFOU 22 $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y
23. Πϯετʔϧ
    
    NBSWFMVJ 23 $ cd /opt/kibana $ sudo bin/kibana plugin --install elasticsearch/marvel Installing marvel … Plugin installation complete
24. ϓϥάΠϯͱͷ࿈ܞ • 8BUDIFS
    ‒ $MVTUFS
    IFBMUI
    DIBOHFT
    )JHI
    NFNPSZ
    VTBHF
    )JHI
    DQV
    VTBHF
    )JHI
    GJMF
    EFTDSJQUPS
    VTBHF
    )JHI
    GJFMEEBUB
    DBDIF
    VTBHF
    /PEFT
    KPJOJOH
    PS
    MFBWJOH
    UIF
    DMVTUFS
    • 4IJFME
    - ,JCBOB
    ΞΫηε࣌ͷύεϫʔυઃఆ
    - 3#"$
    ໾ׂϕʔεͷΞΫηε੍ޚ
    
    FH
    DMVTUFS
    NPOJUPSJOH
    POMZ
    VTFST
    - NBSWFMBHFOU
    ͱ
    NPOJUPSJOH
    DMVTUFS
    ؒͷ௨৴ͷ҉߸Խ 24 • 8BUDIFS
    
    &YBNQMF
    8BUDIFT
    
    8BUDIJOH
    .BSWFM
    %BUB
    
    IUUQTXXXFMBTUJDDPHVJEFFO XBUDIFSDVSSFOUXBUDIJOHNBSWFMEBUBIUNM
    • 4IJFME
    
    $POGJHVSJOH
    $MJFOUT
    BOE
    *OUFHSBUJPOT
    
    6TJOH
    .BSWFM
    XJUI
    4IJFME
    
    IUUQT XXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUNBSWFMIUNM