Elasticsearchの始め方

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
September 02, 2016

 Elasticsearchの始め方

2016/09/02 JJUG ナイトセミナーでの発表資料。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

September 02, 2016
Tweet

Transcript

  1. ‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic

    Stack)ͷ࢝Ίํ
  2. ‹#›

  3. ΞδΣϯμ • Elastic stackͷ঺հ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3

  4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Marvel, Shield, Watcher, Graph
 Professional services: Support & development subscriptions ‒ Trainings 4
  5. ࣭໰ͦͷ1 • ElasticsearchΛ஌͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͕ɺ৮ͬͨ͜ͱ͸ͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •

    ຊ൪؀ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
  6. ࣭໰ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛ࢖ͬͨ͜ͱ͕͋Δ • ࢖ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ

    • Kibana • X-Pack 6
  7. ࣭໰ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana

    • Elasticsearch • X-Pack 7
  8. 8 ElasticελοΫ

  9. ElasticελοΫʢOpen Sourceʣ 9 Kibana      Elasticsearch

         Logstash Beats
  10. ElasticελοΫ 10 Elastic Cloud      

    X-Pack Kibana     Elasticsearch !  "  Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
  11. ‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch

  12. ϑϦʔϫʔυݕࡧ 12

  13. ߜΓࠐΈ 13

  14. ϋΠϥΠτ 14

  15. ιʔτ 15

  16. ϖʔδϯά 16

  17. ूܭ 17

  18. αδΣετ 18

  19. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

  20. 20 Elasticsearch is the backbone across all of Wikimedia’s sites,

    powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
  21. ‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch

  22. Elastic stackʹΑΔσʔλ෼ੳ 22 σʔλ Import/Parse
 /Export Store/Search Visualize

  23. 23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア

    アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
  24. KibanaͰՄࢹԽ 24

  25. 25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,

    and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
  26. beats

  27. Capture the Packet Packetbeat

  28. Capture the Packet Packetbeat

  29. Welcome to 1998 winlogbeat

  30. Now winlogbeat

  31. elasticsearch-hadoop 31 - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC
  32. 32 Elasticsearchͷ
 ࢝Ίํ

  33. Elasticsearch in 10 seconds • ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ • Javaͷόʔδϣϯʹ஫ҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு΋༰қʢPluginػߏͳͲʣ 33
  34. ‹#› Πϯετʔϧ͔Βىಈ·Ͱ

  35. μ΢ϯϩʔυͱىಈʢ2016/09/02࣌఺ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $

    cd elasticsearch-2.4.0 $ ./bin/elasticsearch
  36. ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"

    : "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
  37. RPM or Deb 37

  38. 38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •

    Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
  39. 39 Google Cloud Platform • Google Compute Engine্ʹ਺ΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine

  40. 40 AWS? • Elastic Cloud

  41. ‹#› ݕࡧͱͯ͠ͷElasticsearch

  42. ‹#› ؆୯ͳCRUD

  43. σʔλొ࿥ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  44. σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
  45. σʔλ࡟আ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
  46. ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"

    : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
  47. ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{

    "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
  48. ‹#› ෼ࢄߏ੒ εέʔϧ

  49. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 49
  50. γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
  51. γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3
  52. ࣗಈతͳ෼ࢄ 52 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3
  53. ‹#› શจݕࡧͱ͸ʁ

  54. શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ 54

  55. ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ

    ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 55
  56. ༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒

    จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 56
  57. υΩϡϝϯτͷొ࿥ 57 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

  58. υΩϡϝϯτͷొ࿥ 58 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
  59. υΩϡϝϯτͷొ࿥ 59 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
  60. ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
  61. ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  62. ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  63. ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  64. ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  65. ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  66. ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  67. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 67
  68. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 68
  69. N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ

    • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 69
  70. ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ

    ೳɻ 70 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
  71. N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒

    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
  72. ‹#› ղੳͱͯ͠ͷElasticsearch

  73. ‹#› aggregation

  74. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ

    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 74
  75. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {

    "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
  76. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":

    "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
  77. ‹#› ΫϥΠΞϯτϥΠϒϥϦ

  78. ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •

    Python • .NET • JavaScript • Groovy 78
  79. ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go

    • Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
  80. 80 Elasticsearchͷ ଓ͚ํ

  81. ‹#› ؀ڥपΓ

  82. ࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000΋͘͠͸64,000͕ਪ঑ • ϝϞϦपΓ • SwapΛOff •

    ώʔϓ͸ES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-Xmxʹಉ஋Λࢦఆʣ • ώʔϓͷϝϞϦ͸Ϛγϯͷ൒෼ҎԼ • ωοτϫʔΫ • σϑΥϧτ͸localhostͷΈ 82
  83. ‹#› ৘ใऩू

  84. ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
  85. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

    ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 85
  86. meetup.comʢษڧձʣ 86

  87. ͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •

    https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ఺ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
  88. ‹#› ৘ใڞ༗

  89. ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙਌ձʹ΋ࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճ౴ʣ • Issue/Pull Requestͷ࡞੒

    • υΩϡϝϯτͷमਖ਼ͱ͔΋͋Γ·͢Αʂ 89
  90. Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/

    We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co