Elasticsearchの始め方

Transcript

1. ‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic

   Stack)ͷ࢝Ίํ

2. ‹#›

3. ΞδΣϯμ • Elastic stackͷ঺հ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3

4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

   ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Marvel, Shield, Watcher, Graph
 Professional services: Support & development subscriptions ‒ Trainings 4

5. ࣭໰ͦͷ1 • ElasticsearchΛ஌͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͕ɺ৮ͬͨ͜ͱ͸ͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •

   ຊ൪؀ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5

6. ࣭໰ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛ࢖ͬͨ͜ͱ͕͋Δ • ࢖ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ

   • Kibana • X-Pack 6

7. ࣭໰ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana

   • Elasticsearch • X-Pack 7

8. 8 ElasticελοΫ

9. ElasticελοΫʢOpen Sourceʣ 9 Kibana    
   Elasticsearch

      
   
   Logstash Beats

10. ElasticελοΫ 10 Elastic Cloud  
    
    
    

    X-Pack Kibana    
    Elasticsearch !  "
    
    Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph

11. ‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch

12. ϑϦʔϫʔυݕࡧ 12

13. ߜΓࠐΈ 13

14. ϋΠϥΠτ 14

15. ιʔτ 15

16. ϖʔδϯά 16

17. ूܭ 17

18. αδΣετ 18

19. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

20. 20 Elasticsearch is the backbone across all of Wikimedia’s sites,

    powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering

21. ‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch

22. Elastic stackʹΑΔσʔλ෼ੳ 22 σʔλ Import/Parse
 /Export Store/Search Visualize

23. 23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア

    アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム

24. KibanaͰՄࢹԽ 24

25. 25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,

    and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer

26. beats

27. Capture the Packet Packetbeat

28. Capture the Packet Packetbeat

29. Welcome to 1998 winlogbeat

30. Now winlogbeat

31. elasticsearch-hadoop 31 - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC

32. 32 Elasticsearchͷ
 ࢝Ίํ

33. Elasticsearch in 10 seconds • ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ • Javaͷόʔδϣϯʹ஫ҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு΋༰қʢPluginػߏͳͲʣ 33

34. ‹#› Πϯετʔϧ͔Βىಈ·Ͱ

35. μ΢ϯϩʔυͱىಈʢ2016/09/02࣌఺ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $

    cd elasticsearch-2.4.0 $ ./bin/elasticsearch

36. ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"

    : "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }

37. RPM or Deb 37

38. 38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •

    Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/

39. 39 Google Cloud Platform • Google Compute Engine্ʹ਺ΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine

40. 40 AWS? • Elastic Cloud

41. ‹#› ݕࡧͱͯ͠ͷElasticsearch

42. ‹#› ؆୯ͳCRUD

43. σʔλొ࿥ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

44. σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

45. σʔλ࡟আ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source

46. ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"

    : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]

47. ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{

    "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0

48. ‹#› ෼ࢄߏ੒ εέʔϧ

49. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 49

50. γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

51. γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3

52. ࣗಈతͳ෼ࢄ 52 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3

53. ‹#› શจݕࡧͱ͸ʁ

54. શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ 54

55. ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ

    ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 55

56. ༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒

    จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 56

57. υΩϡϝϯτͷొ࿥ 57 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

58. υΩϡϝϯτͷొ࿥ 58 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

59. υΩϡϝϯτͷొ࿥ 59 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

60. ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

61. ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

62. ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

63. ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

64. ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

65. ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

66. ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

67. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 67

68. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 68

69. N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ

    • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 69

70. ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ

    ೳɻ 70 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

71. N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒

    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

72. ‹#› ղੳͱͯ͠ͷElasticsearch

73. ‹#› aggregation

74. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ

    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 74

75. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {

    "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

76. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":

    "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

77. ‹#› ΫϥΠΞϯτϥΠϒϥϦ

78. ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •

    Python • .NET • JavaScript • Groovy 78

79. ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go

    • Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x

80. 80 Elasticsearchͷ ଓ͚ํ

81. ‹#› ؀ڥपΓ

82. ࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000΋͘͠͸64,000͕ਪ঑ • ϝϞϦपΓ • SwapΛOff •

    ώʔϓ͸ES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-Xmxʹಉ஋Λࢦఆʣ • ώʔϓͷϝϞϦ͸Ϛγϯͷ൒෼ҎԼ • ωοτϫʔΫ • σϑΥϧτ͸localhostͷΈ 82

83. ‹#› ৘ใऩू

84. ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84

85. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

    ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 85

86. meetup.comʢษڧձʣ 86

87. ͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •

    https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ఺ • http://code46.hatenablog.com/entry/2014/01/21/115620 87

88. ‹#› ৘ใڞ༗

89. ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙਌ձʹ΋ࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճ౴ʣ • Issue/Pull Requestͷ࡞੒

    • υΩϡϝϯτͷमਖ਼ͱ͔΋͋Γ·͢Αʂ 89

90. Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/

    We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co