Elasticsearchの始め方

by Jun Ohtani

Slide 1

Slide 1 text

‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic Stack)ͷ࢝Ίํ

Slide 2

Slide 2 text

‹#›

Slide 3

Slide 3 text

ΞδΣϯμ • Elastic stackͷ঺հ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3

Slide 4

Slide 4 text

about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info  • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats   Marvel, Shield, Watcher, Graph  Professional services: Support & development subscriptions ‒ Trainings 4

Slide 5

Slide 5 text

࣭໰ͦͷ1 • ElasticsearchΛ஌͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͕ɺ৮ͬͨ͜ͱ͸ͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ • ຊ൪؀ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5

Slide 6

Slide 6 text

࣭໰ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛ࢖ͬͨ͜ͱ͕͋Δ • ࢖ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ • Kibana • X-Pack 6

Slide 7

Slide 7 text

࣭໰ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana • Elasticsearch • X-Pack 7

Slide 8

Slide 8 text

8 ElasticελοΫ

Slide 9

Slide 9 text

ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch Logstash Beats

Slide 10

Slide 10 text

ElasticελοΫ 10 Elastic Cloud X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph

Slide 11

Slide 11 text

‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch

Slide 12

Slide 12 text

ϑϦʔϫʔυݕࡧ 12

Slide 13

Slide 13 text

ߜΓࠐΈ 13

Slide 14

Slide 14 text

ϋΠϥΠτ 14

Slide 15

Slide 15 text

ιʔτ 15

Slide 16

Slide 16 text

ϖʔδϯά 16

Slide 17

Slide 17 text

ूܭ 17

Slide 18

Slide 18 text

αδΣετ 18

Slide 19

Slide 19 text

elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ    Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

Slide 20

Slide 20 text

20 Elasticsearch is the backbone across all of Wikimedia’s sites, powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering

Slide 21

Slide 21 text

‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch

Slide 22

Slide 22 text

Elastic stackʹΑΔσʔλ෼ੳ 22 σʔλ Import/Parse  /Export Store/Search Visualize

Slide 23

Slide 23 text

23 収集、リッチ化、転送ログおよび数値指標データセンサーおよびデバイスデータ Webおよびソーシャルデータデータストアおよびストリーム分析 Elasticsearch + 任意のデータストアアラート Watcher+任意の通知ツール監視 Marvel+任意の監視ツールアーカイブ Hadoop+任意のクラウドストレージプラットフォーム

Slide 24

Slide 24 text

KibanaͰՄࢹԽ 24

Slide 25

Slide 25 text

25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search, and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer

Slide 26

Slide 26 text

beats

Slide 27

Slide 27 text

Capture the Packet Packetbeat

Slide 28

Slide 28 text

Capture the Packet Packetbeat

Slide 29

Slide 29 text

Welcome to 1998 winlogbeat

Slide 30

Slide 30 text

Now winlogbeat

Slide 31

Slide 31 text

elasticsearch-hadoop 31 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC

Slide 32

Slide 32 text

32 Elasticsearchͷ  ࢝Ίํ

Slide 33

Slide 33 text

Elasticsearch in 10 seconds • ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ • Javaͷόʔδϣϯʹ஫ҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு΋༰қʢPluginػߏͳͲʣ 33

Slide 34

Slide 34 text

‹#› Πϯετʔϧ͔Βىಈ·Ͱ

Slide 35

Slide 35 text

μ΢ϯϩʔυͱىಈʢ2016/09/02࣌఺ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $ cd elasticsearch-2.4.0 $ ./bin/elasticsearch

Slide 36

Slide 36 text

ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name" : "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }

Slide 37

Slide 37 text

RPM or Deb 37

Slide 38

Slide 38 text

38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ • Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/

Slide 39

Slide 39 text

39 Google Cloud Platform • Google Compute Engine্ʹ਺ΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine

Slide 40

Slide 40 text

40 AWS? • Elastic Cloud

Slide 41

Slide 41 text

‹#› ݕࡧͱͯ͠ͷElasticsearch

Slide 42

Slide 42 text

‹#› ؆୯ͳCRUD

Slide 43

Slide 43 text

σʔλొ࿥ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

Slide 44

Slide 44 text

σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

Slide 45

Slide 45 text

σʔλ࡟আ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source

Slide 46

Slide 46 text

ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]

Slide 47

Slide 47 text

ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0

Slide 48

Slide 48 text

‹#› ෼ࢄߏ੒ εέʔϧ

Slide 49

Slide 49 text

Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ  RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ  ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্  σʔλϑϩʔ੍ޚ 49

Slide 50

Slide 50 text

γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

Slide 51

Slide 51 text

γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3

Slide 52

Slide 52 text

ࣗಈతͳ෼ࢄ 52 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3

Slide 53

Slide 53 text

‹#› શจݕࡧͱ͸ʁ

Slide 54

Slide 54 text

શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ  ʢWikipediaΑΓʣ 54

Slide 55

Slide 55 text

༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 55

Slide 56

Slide 56 text

༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 56

Slide 57

Slide 57 text

υΩϡϝϯτͷొ࿥ 57 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

Slide 58

Slide 58 text

υΩϡϝϯτͷొ࿥ 58 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

Slide 59

Slide 59 text

υΩϡϝϯτͷొ࿥ 59 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

Slide 60

Slide 60 text

ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

Slide 61

Slide 61 text

ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 62

Slide 62 text

ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 63

Slide 63 text

ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 64

Slide 64 text

ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 65

Slide 65 text

ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 66

Slide 66 text

ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 67

Slide 67 text

୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.     • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ    67

Slide 68

Slide 68 text

୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.     εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ  Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 68

Slide 69

Slide 69 text

N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 69

Slide 70

Slide 70 text

ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ ೳɻ 70 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

Slide 71

Slide 71 text

N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

Slide 72

Slide 72 text

‹#› ղੳͱͯ͠ͷElasticsearch

Slide 73

Slide 73 text

‹#› aggregation

Slide 74

Slide 74 text

Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ  ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 74

Slide 75

Slide 75 text

ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

Slide 76

Slide 76 text

ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

Slide 77

Slide 77 text

‹#› ΫϥΠΞϯτϥΠϒϥϦ

Slide 78

Slide 78 text

ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl • Python • .NET • JavaScript • Groovy 78

Slide 79

Slide 79 text

ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go • Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x

Slide 80

Slide 80 text

80 Elasticsearchͷ ଓ͚ํ

Slide 81

Slide 81 text

‹#› ؀ڥपΓ

Slide 82

Slide 82 text

࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000΋͘͠͸64,000͕ਪ঑ • ϝϞϦपΓ • SwapΛOff • ώʔϓ͸ES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-Xmxʹಉ஋Λࢦఆʣ • ώʔϓͷϝϞϦ͸Ϛγϯͷ൒෼ҎԼ • ωοτϫʔΫ • σϑΥϧτ͸localhostͷΈ 82

Slide 83

Slide 83 text

‹#› ৘ใऩू

Slide 84

Slide 84 text

ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84

Slide 85

Slide 85 text

ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛  αʔό/ΠϯϑϥΤϯδχΞ  ɹཆ੒ಡຊɹϩάऩू 85

Slide 86

Slide 86 text

meetup.comʢษڧձʣ 86

Slide 87

Slide 87 text

͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog • https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ఺ • http://code46.hatenablog.com/entry/2014/01/21/115620 87

Slide 88

Slide 88 text

‹#› ৘ใڞ༗

Slide 89

Slide 89 text

ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙਌ձʹ΋ࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճ౴ʣ • Issue/Pull Requestͷ࡞੒ • υΩϡϝϯτͷमਖ਼ͱ͔΋͋Γ·͢Αʂ 89

Slide 90

Slide 90 text

Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co