Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchの始め方
Search
Jun Ohtani
September 02, 2016
Technology
9
37k
Elasticsearchの始め方
2016/09/02 JJUG ナイトセミナーでの発表資料。
Jun Ohtani
September 02, 2016
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1.1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.8k
What's new in Elastic Stack 6.3
johtani
2
2.2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
140
Intro Elastic Stack at Telemetry WG
johtani
0
250
What's new in Elastic Stack 6.1?
johtani
0
660
Other Decks in Technology
See All in Technology
2つのフロントエンドと状態管理
mixi_engineers
PRO
3
120
Android Audio: Beyond Winning On It
atsushieno
0
2.4k
品質視点から考える組織デザイン/Organizational Design from Quality
mii3king
0
210
バイブスに「型」を!Kent Beckに学ぶ、AI時代のテスト駆動開発
amixedcolor
2
580
OCI Oracle Database Services新機能アップデート(2025/06-2025/08)
oracle4engineer
PRO
0
180
新規プロダクトでプロトタイプから正式リリースまでNext.jsで開発したリアル
kawanoriku0
1
190
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
自作JSエンジンに推しプロポーザルを実装したい!
sajikix
1
190
まずはマネコンでちゃちゃっと作ってから、それをCDKにしてみよか。
yamada_r
2
120
EncryptedSharedPreferences が deprecated になっちゃった!どうしよう! / Oh no! EncryptedSharedPreferences has been deprecated! What should I do?
yanzm
0
480
【NoMapsTECH 2025】AI Edge Computing Workshop
akit37
0
230
テストを軸にした生き残り術
kworkdev
PRO
0
210
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Become a Pro
speakerdeck
PRO
29
5.5k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1.1k
Designing for Performance
lara
610
69k
Visualization
eitanlees
148
16k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Designing Experiences People Love
moore
142
24k
What's in a price? How to price your products and services
michaelherold
246
12k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Transcript
‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic
Stack)ͷ࢝Ίํ
‹#›
ΞδΣϯμ • Elastic stackͷհ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Marvel, Shield, Watcher, Graph Professional services: Support & development subscriptions ‒ Trainings 4
࣭ͦͷ1 • ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍ • ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •
ຊ൪ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
࣭ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ • ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ
• Kibana • X-Pack 6
࣭ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana
• Elasticsearch • X-Pack 7
8 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 10 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
ϑϦʔϫʔυݕࡧ 12
ߜΓࠐΈ 13
ϋΠϥΠτ 14
ιʔτ 15
ϖʔδϯά 16
ूܭ 17
αδΣετ 18
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
20 Elasticsearch is the backbone across all of Wikimedia’s sites,
powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
Elastic stackʹΑΔσʔλੳ 22 σʔλ Import/Parse /Export Store/Search Visualize
23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア
アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
KibanaͰՄࢹԽ 24
25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,
and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
beats
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
elasticsearch-hadoop 31 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
32 Elasticsearchͷ ࢝Ίํ
Elasticsearch in 10 seconds • ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ • JavaͷόʔδϣϯʹҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு༰қʢPluginػߏͳͲʣ 33
‹#› Πϯετʔϧ͔Βىಈ·Ͱ
μϯϩʔυͱىಈʢ2016/09/02࣌ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $
cd elasticsearch-2.4.0 $ ./bin/elasticsearch
ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"
: "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
RPM or Deb 37
38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •
Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39 Google Cloud Platform • Google Compute Engine্ʹΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40 AWS? • Elastic Cloud
‹#› ݕࡧͱͯ͠ͷElasticsearch
‹#› ؆୯ͳCRUD
σʔλొ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"
: false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
‹#› ࢄߏ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ 49
γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ 52 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
‹#› શจݕࡧͱʁ
શจݕࡧͱʁ • શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ ʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ 54
༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ݅ɺݕࡧࣜ 55
༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ 56
υΩϡϝϯτͷొ 57 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ υΩϡϝϯτͷొ
υΩϡϝϯτͷొ 58 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ ୯ޠʹׂ
υΩϡϝϯτͷొ 59 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ ୯ޠʹׂ ୯ޠ͔Βidͷྻ͕ Ҿ͚ΔΑ͏ʹ
ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ݅ೖྗ ΧπΦɹαβΤ
ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
• ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ 67
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
εϖʔε͕ΕͱΘ͔Δ • ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ 68
N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞ • N-Gram ‒ NจࣈͣͭจষΛ۠Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ 69
ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆ
ೳɻ 70 ΧπΦαβΤͷఋ ΧπΦ ͷ αβΤ ఋ
N-Gram • ϝϦοτɿ ‒ ະޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
ࢺใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦαβΤͷఋ Χπ πΦ Φ α αβ βΤ Τͷ ͷఋ
‹#› ղੳͱͯ͠ͷElasticsearch
‹#› aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 74
ྫɿݴޠ͓ΑͼҬͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
‹#› ΫϥΠΞϯτϥΠϒϥϦ
ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •
Python • .NET • JavaScript • Groovy 78
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go
• Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80 Elasticsearchͷ ଓ͚ํ
‹#› ڥपΓ
࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000͘͠64,000͕ਪ • ϝϞϦपΓ • SwapΛOff •
ώʔϓES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-XmxʹಉΛࢦఆʣ • ώʔϓͷϝϞϦϚγϯͷҎԼ • ωοτϫʔΫ • σϑΥϧτlocalhostͷΈ 82
‹#› ใऩू
ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •
ॻ੶ʢຊޠʣ ‒ ElasticSearchServerຊޠ൛ αʔό/ΠϯϑϥΤϯδχΞ ɹཆಡຊɹϩάऩू 85
meetup.comʢษڧձʣ 86
͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •
https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
‹#› ใڞ༗
ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙ձʹࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ • Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ 89
Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/
We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co