Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchの始め方
Search
Jun Ohtani
September 02, 2016
Technology
9
37k
Elasticsearchの始め方
2016/09/02 JJUG ナイトセミナーでの発表資料。
Jun Ohtani
September 02, 2016
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1.1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.9k
What's new in Elastic Stack 6.3
johtani
2
2.2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
140
Intro Elastic Stack at Telemetry WG
johtani
0
250
What's new in Elastic Stack 6.1?
johtani
0
660
Other Decks in Technology
See All in Technology
Shirankedo NOCで見えてきたeduroam/OpenRoaming運用ノウハウと課題 - BAKUCHIKU BANBAN #2
marokiki
0
150
定期的な価値提供だけじゃない、スクラムが導くチームの共創化 / 20251004 Naoki Takahashi
shift_evolve
PRO
3
310
pprof vs runtime/trace (FlightRecorder)
task4233
0
170
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
980
職種別ミートアップで社内から盛り上げる アウトプット文化の醸成と関係強化/ #DevRelKaigi
nishiuma
2
140
ユニットテストに対する考え方の変遷 / Everyone should watch his live coding
mdstoy
0
130
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
350
実装で解き明かす並行処理の歴史
zozotech
PRO
1
350
AI時代だからこそ考える、僕らが本当につくりたいスクラムチーム / A Scrum Team we really want to create in this AI era
takaking22
6
3.5k
Function calling機能をPLaMo2に実装するには / PFN LLMセミナー
pfn
PRO
0
930
【Oracle Cloud ウェビナー】クラウド導入に「専用クラウド」という選択肢、Oracle AlloyとOCI Dedicated Region とは
oracle4engineer
PRO
3
110
バイブコーディングと継続的デプロイメント
nwiizo
2
430
Featured
See All Featured
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Docker and Python
trallard
46
3.6k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
850
Rails Girls Zürich Keynote
gr2m
95
14k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
RailsConf 2023
tenderlove
30
1.2k
Why Our Code Smells
bkeepers
PRO
339
57k
Statistics for Hackers
jakevdp
799
220k
How to train your dragon (web standard)
notwaldorf
96
6.3k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Done Done
chrislema
185
16k
Transcript
‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic
Stack)ͷ࢝Ίํ
‹#›
ΞδΣϯμ • Elastic stackͷհ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Marvel, Shield, Watcher, Graph Professional services: Support & development subscriptions ‒ Trainings 4
࣭ͦͷ1 • ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍ • ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •
ຊ൪ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
࣭ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ • ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ
• Kibana • X-Pack 6
࣭ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana
• Elasticsearch • X-Pack 7
8 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 10 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
ϑϦʔϫʔυݕࡧ 12
ߜΓࠐΈ 13
ϋΠϥΠτ 14
ιʔτ 15
ϖʔδϯά 16
ूܭ 17
αδΣετ 18
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
20 Elasticsearch is the backbone across all of Wikimedia’s sites,
powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
Elastic stackʹΑΔσʔλੳ 22 σʔλ Import/Parse /Export Store/Search Visualize
23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア
アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
KibanaͰՄࢹԽ 24
25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,
and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
beats
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
elasticsearch-hadoop 31 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
32 Elasticsearchͷ ࢝Ίํ
Elasticsearch in 10 seconds • ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ • JavaͷόʔδϣϯʹҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு༰қʢPluginػߏͳͲʣ 33
‹#› Πϯετʔϧ͔Βىಈ·Ͱ
μϯϩʔυͱىಈʢ2016/09/02࣌ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $
cd elasticsearch-2.4.0 $ ./bin/elasticsearch
ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"
: "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
RPM or Deb 37
38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •
Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39 Google Cloud Platform • Google Compute Engine্ʹΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40 AWS? • Elastic Cloud
‹#› ݕࡧͱͯ͠ͷElasticsearch
‹#› ؆୯ͳCRUD
σʔλొ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"
: false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
‹#› ࢄߏ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ 49
γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ 52 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
‹#› શจݕࡧͱʁ
શจݕࡧͱʁ • શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ ʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ 54
༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ݅ɺݕࡧࣜ 55
༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ 56
υΩϡϝϯτͷొ 57 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ υΩϡϝϯτͷొ
υΩϡϝϯτͷొ 58 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ ୯ޠʹׂ
υΩϡϝϯτͷొ 59 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ ୯ޠʹׂ ୯ޠ͔Βidͷྻ͕ Ҿ͚ΔΑ͏ʹ
ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ݅ೖྗ ΧπΦɹαβΤ
ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
• ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ 67
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
εϖʔε͕ΕͱΘ͔Δ • ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ 68
N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞ • N-Gram ‒ NจࣈͣͭจষΛ۠Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ 69
ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆ
ೳɻ 70 ΧπΦαβΤͷఋ ΧπΦ ͷ αβΤ ఋ
N-Gram • ϝϦοτɿ ‒ ະޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
ࢺใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦαβΤͷఋ Χπ πΦ Φ α αβ βΤ Τͷ ͷఋ
‹#› ղੳͱͯ͠ͷElasticsearch
‹#› aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 74
ྫɿݴޠ͓ΑͼҬͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
‹#› ΫϥΠΞϯτϥΠϒϥϦ
ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •
Python • .NET • JavaScript • Groovy 78
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go
• Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80 Elasticsearchͷ ଓ͚ํ
‹#› ڥपΓ
࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000͘͠64,000͕ਪ • ϝϞϦपΓ • SwapΛOff •
ώʔϓES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-XmxʹಉΛࢦఆʣ • ώʔϓͷϝϞϦϚγϯͷҎԼ • ωοτϫʔΫ • σϑΥϧτlocalhostͷΈ 82
‹#› ใऩू
ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •
ॻ੶ʢຊޠʣ ‒ ElasticSearchServerຊޠ൛ αʔό/ΠϯϑϥΤϯδχΞ ɹཆಡຊɹϩάऩू 85
meetup.comʢษڧձʣ 86
͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •
https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
‹#› ใڞ༗
ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙ձʹࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ • Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ 89
Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/
We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co