Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchの始め方
Search
Jun Ohtani
September 02, 2016
Technology
9
36k
Elasticsearchの始め方
2016/09/02 JJUG ナイトセミナーでの発表資料。
Jun Ohtani
September 02, 2016
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.6k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
950
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
870
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.6k
What's new in Elastic Stack 6.3
johtani
2
2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.3k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
100
Intro Elastic Stack at Telemetry WG
johtani
0
210
What's new in Elastic Stack 6.1?
johtani
0
540
Other Decks in Technology
See All in Technology
コンテナ・K8s研修 - 前半 コンテナ基礎・ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
How to Think Like a Performance Engineer
csswizardry
4
590
LINE WORKSへ簡単通知!Incoming Webhookアプリの紹介
mmclsntr
0
110
VPoEの視点から見た、ヘンリーがサーバーサイドKotlinを使う理由 / Why Server-side Kotlin 2024
cho0o0
1
420
ペパボのオブザーバビリティ研修2024 説明資料
kesompochy
0
1.1k
Flutter研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
What is DRE? - Road to SRE NEXT@広島
chanyou0311
3
630
CTOから見た事業開発とプロダクト開発 / My Perspective on Business and Product Development as CTO
keisuke69
4
960
「我々はどこに向かっているのか」を問い続けるための仕組みづくり / Establishing a System for Continuous Inquiry about where we are
daitasu
0
170
MySQLのロックの種類とその競合
yoku0825
6
1.6k
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
CEL(Common Expression Language)で書いた条件にマッチしたIAM Policyを見つける / iam-policy-finder
fujiwara3
0
710
Featured
See All Featured
Faster Mobile Websites
deanohume
303
30k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
277
13k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
24
1.8k
A designer walks into a library…
pauljervisheath
201
24k
How GitHub Uses GitHub to Build GitHub
holman
471
290k
Happy Clients
brianwarren
94
6.6k
Code Review Best Practice
trishagee
58
16k
Designing Experiences People Love
moore
136
23k
The Pragmatic Product Professional
lauravandoore
29
6.1k
How to Ace a Technical Interview
jacobian
274
23k
Typedesign – Prime Four
hannesfritz
37
2.2k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
360
22k
Transcript
‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic
Stack)ͷ࢝Ίํ
‹#›
ΞδΣϯμ • Elastic stackͷհ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Marvel, Shield, Watcher, Graph Professional services: Support & development subscriptions ‒ Trainings 4
࣭ͦͷ1 • ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍ • ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •
ຊ൪ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
࣭ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ • ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ
• Kibana • X-Pack 6
࣭ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana
• Elasticsearch • X-Pack 7
8 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 10 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
ϑϦʔϫʔυݕࡧ 12
ߜΓࠐΈ 13
ϋΠϥΠτ 14
ιʔτ 15
ϖʔδϯά 16
ूܭ 17
αδΣετ 18
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
20 Elasticsearch is the backbone across all of Wikimedia’s sites,
powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
Elastic stackʹΑΔσʔλੳ 22 σʔλ Import/Parse /Export Store/Search Visualize
23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア
アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
KibanaͰՄࢹԽ 24
25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,
and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
beats
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
elasticsearch-hadoop 31 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
32 Elasticsearchͷ ࢝Ίํ
Elasticsearch in 10 seconds • ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ • JavaͷόʔδϣϯʹҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு༰қʢPluginػߏͳͲʣ 33
‹#› Πϯετʔϧ͔Βىಈ·Ͱ
μϯϩʔυͱىಈʢ2016/09/02࣌ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $
cd elasticsearch-2.4.0 $ ./bin/elasticsearch
ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"
: "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
RPM or Deb 37
38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •
Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39 Google Cloud Platform • Google Compute Engine্ʹΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40 AWS? • Elastic Cloud
‹#› ݕࡧͱͯ͠ͷElasticsearch
‹#› ؆୯ͳCRUD
σʔλొ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"
: false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
‹#› ࢄߏ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ 49
γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ 52 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
‹#› શจݕࡧͱʁ
શจݕࡧͱʁ • શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ ʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ 54
༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ݅ɺݕࡧࣜ 55
༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ 56
υΩϡϝϯτͷొ 57 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ υΩϡϝϯτͷొ
υΩϡϝϯτͷొ 58 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ ୯ޠʹׂ
υΩϡϝϯτͷొ 59 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ ୯ޠʹׂ ୯ޠ͔Βidͷྻ͕ Ҿ͚ΔΑ͏ʹ
ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ݅ೖྗ ΧπΦɹαβΤ
ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
• ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ 67
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
εϖʔε͕ΕͱΘ͔Δ • ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ 68
N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞ • N-Gram ‒ NจࣈͣͭจষΛ۠Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ 69
ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆ
ೳɻ 70 ΧπΦαβΤͷఋ ΧπΦ ͷ αβΤ ఋ
N-Gram • ϝϦοτɿ ‒ ະޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
ࢺใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦαβΤͷఋ Χπ πΦ Φ α αβ βΤ Τͷ ͷఋ
‹#› ղੳͱͯ͠ͷElasticsearch
‹#› aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 74
ྫɿݴޠ͓ΑͼҬͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
‹#› ΫϥΠΞϯτϥΠϒϥϦ
ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •
Python • .NET • JavaScript • Groovy 78
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go
• Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80 Elasticsearchͷ ଓ͚ํ
‹#› ڥपΓ
࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000͘͠64,000͕ਪ • ϝϞϦपΓ • SwapΛOff •
ώʔϓES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-XmxʹಉΛࢦఆʣ • ώʔϓͷϝϞϦϚγϯͷҎԼ • ωοτϫʔΫ • σϑΥϧτlocalhostͷΈ 82
‹#› ใऩू
ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •
ॻ੶ʢຊޠʣ ‒ ElasticSearchServerຊޠ൛ αʔό/ΠϯϑϥΤϯδχΞ ɹཆಡຊɹϩάऩू 85
meetup.comʢษڧձʣ 86
͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •
https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
‹#› ใڞ༗
ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙ձʹࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ • Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ 89
Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/
We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co