Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchの始め方
Search
Jun Ohtani
September 02, 2016
Technology
9
37k
Elasticsearchの始め方
2016/09/02 JJUG ナイトセミナーでの発表資料。
Jun Ohtani
September 02, 2016
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.8k
What's new in Elastic Stack 6.3
johtani
2
2.2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
130
Intro Elastic Stack at Telemetry WG
johtani
0
250
What's new in Elastic Stack 6.1?
johtani
0
640
Other Decks in Technology
See All in Technology
ポストコロナ時代の SaaS におけるコスト削減の意義
izzii
1
470
助けて! XからWaylandに移行しないと新しいGNOMEが使えなくなっちゃう 2025-07-12
nobutomurata
2
200
ロールが細分化された組織でSREは何をするか?
tgidgd
1
420
[SRE NEXT 2025] すみずみまで暖かく照らすあなたの太陽でありたい
carnappopper
2
470
伴走から自律へ: 形式知へと導くSREイネーブリングによる プロダクトチームの信頼性オーナーシップ向上 / SRE NEXT 2025
visional_engineering_and_design
3
460
振り返りTransit Gateway ~VPCをいい感じでつなげるために~
masakiokuda
3
210
AWS 怖い話 WAF編 @fillz_noh #AWSStartup #AWSStartup_Kansai
fillznoh
0
130
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
VS CodeとGitHub Copilotで爆速開発!アップデートの波に乗るおさらい会 / Rapid Development with VS Code and GitHub Copilot: Catch the Latest Wave
yamachu
3
460
“日本一のM&A企業”を支える、少人数SREの効率化戦略 / SRE NEXT 2025
genda
1
270
United™️ Airlines®️ Customer®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedguide
0
800
全部AI、全員Cursor、ドキュメント駆動開発 〜DevinやGeminiも添えて〜
rinchsan
10
5.1k
Featured
See All Featured
The Invisible Side of Design
smashingmag
301
51k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Raft: Consensus for Rubyists
vanstee
140
7k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Speed Design
sergeychernyshev
32
1k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Scaling GitHub
holman
460
140k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
Gamification - CAS2011
davidbonilla
81
5.4k
Building Adaptive Systems
keathley
43
2.7k
The Language of Interfaces
destraynor
158
25k
Transcript
‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic
Stack)ͷ࢝Ίํ
‹#›
ΞδΣϯμ • Elastic stackͷհ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Marvel, Shield, Watcher, Graph Professional services: Support & development subscriptions ‒ Trainings 4
࣭ͦͷ1 • ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍ • ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •
ຊ൪ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
࣭ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ • ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ
• Kibana • X-Pack 6
࣭ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana
• Elasticsearch • X-Pack 7
8 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 10 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
ϑϦʔϫʔυݕࡧ 12
ߜΓࠐΈ 13
ϋΠϥΠτ 14
ιʔτ 15
ϖʔδϯά 16
ूܭ 17
αδΣετ 18
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
20 Elasticsearch is the backbone across all of Wikimedia’s sites,
powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
Elastic stackʹΑΔσʔλੳ 22 σʔλ Import/Parse /Export Store/Search Visualize
23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア
アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
KibanaͰՄࢹԽ 24
25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,
and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
beats
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
elasticsearch-hadoop 31 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
32 Elasticsearchͷ ࢝Ίํ
Elasticsearch in 10 seconds • ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ • JavaͷόʔδϣϯʹҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு༰қʢPluginػߏͳͲʣ 33
‹#› Πϯετʔϧ͔Βىಈ·Ͱ
μϯϩʔυͱىಈʢ2016/09/02࣌ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $
cd elasticsearch-2.4.0 $ ./bin/elasticsearch
ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"
: "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
RPM or Deb 37
38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •
Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39 Google Cloud Platform • Google Compute Engine্ʹΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40 AWS? • Elastic Cloud
‹#› ݕࡧͱͯ͠ͷElasticsearch
‹#› ؆୯ͳCRUD
σʔλొ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"
: false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
‹#› ࢄߏ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ 49
γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ 52 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
‹#› શจݕࡧͱʁ
શจݕࡧͱʁ • શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ ʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ 54
༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ݅ɺݕࡧࣜ 55
༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ 56
υΩϡϝϯτͷొ 57 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ υΩϡϝϯτͷొ
υΩϡϝϯτͷొ 58 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ ୯ޠʹׂ
υΩϡϝϯτͷొ 59 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ ୯ޠʹׂ ୯ޠ͔Βidͷྻ͕ Ҿ͚ΔΑ͏ʹ
ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ݅ೖྗ ΧπΦɹαβΤ
ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
• ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ 67
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
εϖʔε͕ΕͱΘ͔Δ • ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ 68
N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞ • N-Gram ‒ NจࣈͣͭจষΛ۠Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ 69
ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆ
ೳɻ 70 ΧπΦαβΤͷఋ ΧπΦ ͷ αβΤ ఋ
N-Gram • ϝϦοτɿ ‒ ະޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
ࢺใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦαβΤͷఋ Χπ πΦ Φ α αβ βΤ Τͷ ͷఋ
‹#› ղੳͱͯ͠ͷElasticsearch
‹#› aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 74
ྫɿݴޠ͓ΑͼҬͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
‹#› ΫϥΠΞϯτϥΠϒϥϦ
ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •
Python • .NET • JavaScript • Groovy 78
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go
• Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80 Elasticsearchͷ ଓ͚ํ
‹#› ڥपΓ
࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000͘͠64,000͕ਪ • ϝϞϦपΓ • SwapΛOff •
ώʔϓES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-XmxʹಉΛࢦఆʣ • ώʔϓͷϝϞϦϚγϯͷҎԼ • ωοτϫʔΫ • σϑΥϧτlocalhostͷΈ 82
‹#› ใऩू
ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •
ॻ੶ʢຊޠʣ ‒ ElasticSearchServerຊޠ൛ αʔό/ΠϯϑϥΤϯδχΞ ɹཆಡຊɹϩάऩू 85
meetup.comʢษڧձʣ 86
͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •
https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
‹#› ใڞ༗
ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙ձʹࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ • Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ 89
Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/
We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co