#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
‹#›2017/02/07Evangelist at ElasticJun Ohtani @johtaniElastic StackΛར༻ͯ͠σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
View Slide
‹#›
ΞδΣϯμ• ؾ͖ͮΛݟ͚ͭΔͱʁ• Ϣʔεέʔεͷհ• Elastic stackհ• BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack• σϞ3
about• Me, Jun Ohtani / Technical Advocate‒ lucene-gosenίϛολʔ‒ ElasticSearch Serverຊޠ൛ͷ༁‒ http://blog.johtani.info • Elasticsearch, founded in 2012‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions‒ Trainings & Consulting4
༷ʑͳϢʔεέʔε5ؾ͖ͮΛݟ͚ͭΔͱʁ
Search andanalytics, it allstarted hereMore than 60% of our customershave a search or analytics usecase
7
8
Logs Logs Logs, many devices, many systemsMore than 40% of our customers use our products for operational log analysis
We collect more than1.2 TB logs every day from ourinfrastructure, web servers, andapplications.10
11We handle more than3 Billion daily eventswhile meeting our all of our datasecurity requirements.
Sniff sniff sniff, find the bad actors in your data200% YoY growth in securityuse cases with our products
We analyze piles of data:13B AMP queries/day600B emails/day16B web requests/day13
14We mine and analyze4 billion events everyday to detect security hacksand threats.1
The Elastic Stack: A foundation to solvemany use cases75% of our customers useour products for more thanone use case SEARCHSECURITCUSTOM APPSMETRICSOPERATIONAL ANALYTICSLOG ANALYSIS
Operational analyticsFlight telemetry analysisAnomaly resolutionInternal search engine16
17Enterprise searchIntranet searchReal-time log analyticsLegal contract repositoryTrade tracking applicationHR recruiting application
18ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ19Kibana Elasticsearch Logstash Beats
ElasticελοΫ20Elastic Cloud X-PackKibana Elasticsearch!" Logstash Beats+
Ingest
22Logstash
Logstash in 10 seconds• ϩάɾσʔλͷऩूɾཧ• ऩूɺύʔεɾՃɺૹग़• ΦʔϓϯιʔεɿApache License 2.0• Ruby app (JRuby)23
Logstash architecture24Input OutputFilter? ?collect and split alter and enrich store and visualize
ઃఆ25input {…}filter {…}output {…}
ઃఆɿinput26input {file {path => “/Users/johtani/sample/*_log"start_position => "beginning"}}
1ߦ1σʔλ189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101Firefox/5.0"27
ઃఆɿfilter28filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ύʔε29189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"{…"@timestamp": "2015-04-10T09:07:49.325Z","clientip": "189.120.xx.xx","ident": "-","auth": "-","timestamp": "02/Dec/2014:12:18:29 +0900","verb": "GET","request": "/manager/html",…"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter30filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
IP͔ΒҢܦͳͲ༩31"clientip": "189.120.xx.xx","clientip": "189.120.xx.xx","geoip": {"ip": “189.120.xxx.xxx”,…"country_name": "Brazil","continent_code": "SA","region_name": "27","city_name": "São Paulo","latitude":
ઃఆɿoutput32output {elasticsearch {hosts => ["localhost"]index => “demo_access_log-%{+YYYY.MM.dd}”}}
ܰྔσʔλγούʔ33Beats
To tail aFilefilebeat
Capture thePacketPacketbeat
Welcometo 1998winlogbeat
Nowwinlogbeat
Store, Search & Analytics
41Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ44
ߜΓࠐΈ45
ϋΠϥΠτ46
ιʔτ47
ϖʔδϯά48
ूܭ49
αδΣετ50
Elasticsearch in 10 seconds• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON• Φʔϓϯιʔε: Apache License 2.0• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ• JavaͰ࣮ɻ֦ு༰қ51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ• 1.0͔Βಋೖ• FacetΑΓڧྗͳूܭͳͲ͕Մೳ• ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ• େ͖͘2छྨ• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά• Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ54
ྫɿݴޠ͓ΑͼҬͷूܭ55curl -XGET twitter-2014.08.22/_search -d '{"aggs": {"lang": {"terms": {"field": "lang" },"aggs": {"place": {"terms": {"field": “place.full_name", "size": 10}}}}}}
ྫɿݴޠ͓ΑͼҬͷूܭ56"aggregations": {"lang": {"buckets": [{…}, {"key": "ja","doc_count": 980145,"place": {"buckets": [{ "key": "ژࢢ෬ݟ۠, ژ","doc_count":252 },{ "key": "ઍా۠, ౦ژ","doc_count": 39 },…
elasticsearch-hadoop57-• D E H• PD ecdER• g D• CH• Ca M DMSD FERC
The Window into the Elastic Stack
59KibanaͰՄࢹԽ
Kibana 5• ElasticsearchͷσʔλΛՄࢹԽ• Node.js server & JavaScript• Apache License 2.0• Elastic Stackͷ૭ͷׂ• ༷ʑͳGUIΛPluginͱ͍ͯެ։• MarvelɺSenseɺTimelionͳͲ60
Kibana 561
X-Pack 5.0:Extending the Elastic Stack
Security
X-Pack : Securityͷಛ• User Authentication‒ LDAP/Active Directory/ϑΝΠϧϕʔε• Authorization‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ• ηΩϡΞͳ௨৴‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά• ࠪϩά65
Alerting
X-Pack : Alertingͷಛ• ΫΤϦʹΑΔWatch‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ• ݅ͷઃఆ‒ ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ• εέδϡʔϧ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ• ΞΫγϣϯͷఆٛ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ• ཤྺͷอଘ67
Graph
Graphͷಛ• σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ• KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔσʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠शݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73σϞDemo
ࢀߟαΠτ• Ϣʔεέʔε• https://www.elastic.co/use-cases• DiscussʢWebϑΥʔϥϜʣ• https://discuss.elastic.co• Elastic{ON}ͷϏσΦͱࢿྉ• https://www.elastic.co/elasticon/videos• αϙʔτϝχϡʔ• https://www.elastic.co/subscriptions74
75March 7-9, 2017• Pier 48• San Francisco, CA• 2,500 attendees3rd Annual Elastic User ConferenceTopics• Latest Roadmap• Ask Me Anything Booth• 70+ Sessions• 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢bit.ly/bigdata-tokyo-elastic
Thanks for listening!Q & AWe’re hiring!https://www.elastic.co/about/careers/We’re helping!https://www.elastic.co/subscriptionshttp://training.elastic.co
johtani
coconala_engineer
yamamuteki
sat
niftycorp
masasuzu
ufried
kei_s
upura
shibuiwilliam
naohachi89
eileencodes
shpigford
morganepeng
tammielis
mongodb
lara
sferik
hursman
brad_frost
jcasabona
myddelton