Elasticsearchの始め方

2016/09/02 JJUG ナイトセミナーでの発表資料。

Transcript

1. ‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic Stack)ͷ࢝Ίํ
2. ‹#›
3. ΞδΣϯμ • Elastic stackͷ঺հ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Marvel, Shield, Watcher, Graph
 Professional services: Support & development subscriptions ‒ Trainings 4
5. ࣭໰ͦͷ1 • ElasticsearchΛ஌͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͕ɺ৮ͬͨ͜ͱ͸ͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ • ຊ൪؀ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
6. ࣭໰ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛ࢖ͬͨ͜ͱ͕͋Δ • ࢖ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ • Kibana • X-Pack 6
7. ࣭໰ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana • Elasticsearch • X-Pack 7
8. 8 ElasticελοΫ
9. ElasticελοΫʢOpen Sourceʣ 9 Kibana    
   Elasticsearch    
   
   Logstash Beats
10. ElasticελοΫ 10 Elastic Cloud  
    
    
     X-Pack Kibana    
    Elasticsearch !  "
    
    Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
11. ‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
12. ϑϦʔϫʔυݕࡧ 12
13. ߜΓࠐΈ 13
14. ϋΠϥΠτ 14
15. ιʔτ 15
16. ϖʔδϯά 16
17. ूܭ 17
18. αδΣετ 18
19. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO
20. 20 Elasticsearch is the backbone across all of Wikimedia’s sites, powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
21. ‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
22. Elastic stackʹΑΔσʔλ෼ੳ 22 σʔλ Import/Parse
 /Export Store/Search Visualize
23. 23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
24. KibanaͰՄࢹԽ 24
25. 25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search, and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
26. beats
27. Capture the Packet Packetbeat
28. Capture the Packet Packetbeat
29. Welcome to 1998 winlogbeat
30. Now winlogbeat
31. elasticsearch-hadoop 31 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC
32. 32 Elasticsearchͷ
 ࢝Ίํ
33. Elasticsearch in 10 seconds • ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ • Javaͷόʔδϣϯʹ஫ҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு΋༰қʢPluginػߏͳͲʣ 33
34. ‹#› Πϯετʔϧ͔Βىಈ·Ͱ
35. μ΢ϯϩʔυͱىಈʢ2016/09/02࣌఺ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $ cd elasticsearch-2.4.0 $ ./bin/elasticsearch
36. ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name" : "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
37. RPM or Deb 37
38. 38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ • Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39. 39 Google Cloud Platform • Google Compute Engine্ʹ਺ΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40. 40 AWS? • Elastic Cloud
41. ‹#› ݕࡧͱͯ͠ͷElasticsearch
42. ‹#› ؆୯ͳCRUD
43. σʔλొ࿥ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
44. σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
45. σʔλ࡟আ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
46. ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
47. ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
48. ‹#› ෼ࢄߏ੒ εέʔϧ
49. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 49
50. γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
51. γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
52. ࣗಈతͳ෼ࢄ 52 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
53. ‹#› શจݕࡧͱ͸ʁ
54. શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ 54
55. ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 55
56. ༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 56
57. υΩϡϝϯτͷొ࿥ 57 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
58. υΩϡϝϯτͷొ࿥ 58 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
59. υΩϡϝϯτͷొ࿥ 59 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
60. ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
61. ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
62. ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
63. ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
64. ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
65. ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
66. ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
67. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 
 • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 67
68. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 
 εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 68
69. N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 69
70. ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ ೳɻ 70 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
71. N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
72. ‹#› ղੳͱͯ͠ͷElasticsearch
73. ‹#› aggregation
74. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 74
75. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
76. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
77. ‹#› ΫϥΠΞϯτϥΠϒϥϦ
78. ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl • Python • .NET • JavaScript • Groovy 78
79. ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go • Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80. 80 Elasticsearchͷ ଓ͚ํ
81. ‹#› ؀ڥपΓ
82. ࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000΋͘͠͸64,000͕ਪ঑ • ϝϞϦपΓ • SwapΛOff • ώʔϓ͸ES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-Xmxʹಉ஋Λࢦఆʣ • ώʔϓͷϝϞϦ͸Ϛγϯͷ൒෼ҎԼ • ωοτϫʔΫ • σϑΥϧτ͸localhostͷΈ 82
83. ‹#› ৘ใऩू
84. ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
85. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 85
86. meetup.comʢษڧձʣ 86
87. ͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog • https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ఺ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
88. ‹#› ৘ใڞ༗
89. ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙਌ձʹ΋ࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճ౴ʣ • Issue/Pull Requestͷ࡞੒ • υΩϡϝϯτͷमਖ਼ͱ͔΋͋Γ·͢Αʂ 89
90. Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co