Elastic{ON} 2017 Closing Keynote: Cause Award Honorees and Q&A with Elastic Founders

Leah Sutton l VP Global HR l Elastic
Adam Thompson l Executive Director l eHealth Africa
Paul Michael Privateer, Ph.D l Founding Director l NoSchoolViolence.org
Michael Paley l President l IST Research
David Cavitt l Sr. TechOps Engineer l IST Research
Marty Messer l VP Customer Care l Elastic
Steven Schuurman l Co-Founder & CEO l Elastic
Simon Willnauer l Co-Founder & Elasticsearch Tech Lead l Elastic
Uri Boness l Co-Founder & Commercial Products Tech Lead l Elastic

Transcript

2. Leah Sutton Cause Award Honorees
3. Fighting human trafficking and slave trade Trevor Ellermann, Sr. TechOps Engineer
4. IST Research Inc. 3/9/2017 www.istresearch.com Elasticsearch Finds a Pulse Michael Paley, President David Cavitt, Senior TechOps Engineer
5. • Introduction to IST Research • Our Path to the Elastic Stack • Lessons Learned: “Mistakes were made” • Production deployment of Pulse • Future use of Elastic Stack at IST Research Overview
6. Human security is the combination of threats associated with war, genocide, and the displacement of populations. At a minimum, human security means freedom from violence and from the fear of violence.
7. This is a sample image Pulse helps you understand hard to reach communities by providing insight into their thinking and engaging them in two- way conversations
8. Ebola Monitoring Afghan Teacher Pay Labor Trafficking Monitoring and Evaluation Sex Trafficking Civil Affairs
14. Evolution of ELK @ IST Research Evolution of the Elastic Stack @ IST Research
15. Finding Elasticsearch - Log Analysis Tool It’s not just for search! Debugging large scale distributed applications is hard Need for aggregate log data Why not all data?
16. The Glory that is Kibana Area Charts Pie Charts Data Tables Line Charts Metrics Tile Maps Vertical Bar Charts Tag Clouds Heatmap Charts
17. •Long time intervals •Queries running across all Indices •Not being specific on the data type *Star* is *the Enemy*
18. 1.Disabling Shard Allocation 2.Rolling Restart 3.Waiting, Bueller, Bueller… 4.Enabling Shard Allocation 5.Relocating Shards...Bueller 6.Cluster is Green? NOPE 7.Unassigned Shards 8.Forcing Shard Allocation 9.Cluster is finally Green 10.Back to Business With Great Power Comes Great Responsibility #!/bin/bash curl -XPOST 'es.istresearch.com:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : ".kibana", "shard" : 0, "node" : "esnode-9-master_data", "allow_primary" : true } } ] }'
19. Our Cluster Then... Getting Smart • 10 nodes - m4.2xlarge • 4 vCPU 32GB RAM each • Master/Data Combo • No Snapshots • RAM/Storage Ratio of 1/90 GB Our Cluster Now... • 18 nodes • 3 Master Nodes ◦ 2 vCPU 8GB RAM • 15 Data Nodes ◦ 8 vCPU 64GB RAM • Snapshots ◦ Monthly Indices Archived in S3/Glacier ◦ Active Snapshots every 30 min for 48 hr • Rolling Indices using Curator • RAM/Storage Ratio of 1/30 GB
20. Searchable Data Logs Deployments • 10 kilobytes per document • 300 million docs per month • 1,500+ fields • 500 million logs per day • 5-20 node ES clusters • 30+ servers for ingestion pipeline • 15 - 30 Terabytes worth of data In Production
21. The Pipeline 1 Search - ES 2 Log aggregation - Logstash 3 Visualization - Kibana 4 Index Maintenance - Curator 5 Percolation
22. All the Percolators in the House... ? Why did we collect this?
23. Did you just percolate on my query? Index It Data Collection API Collection Criteria Index Percolator Search Data Index Tag each Document with User Info Social Media Web
24. Collection Criteria Query { "query":{ "bool" : { "must" : { "match_all" : {} }, "filter" : { "bool" : { "should" : [ { "geo_shape": { "geo_shape": { "shape": { "type": "envelope", "coordinates" : [[-10.75, 46.8], [10.75, 37.8]] }, "relation": "intersects" } } } ] } } } }, "status":"active", "rule_type":"locations", "tag":"test", "description":"Tweets from a specified area", "rule_value":"-10.75,37.8,10.75,46.8", "appid":"test-app", "date_added":"2016-05-10 16:58:34", "campaign_id":"default", "project_id":"default" }
25. Example Collected Data Document { "query" : { "percolate" : { "field" : "query", "document_type" : "document", "document" : { "text": "Vermut-Swing a La Rosaleda #tempsdeflors #girona @girona_cat aavv.devesa_guell @ Associaci\u00f3 de\u2026 https://t.co/xfcB3yj4bK https://www.instagram.com/p/BFJeFpHpEjY/ instagram.com/p/BFJeFpHpEjY/ tempsdeflors girona LoLites", "geo_point": { "lat": 41.9857903, "lon": 2.8190701 }, "user_id": "131253506" } } } }
26. Snapshots and Backups
27. #!/bin/bash DATE=`date +%Y-%m-%d-%H-%M` URL="localhost:9200/_snapshot/ist-es-snapshots/prod-active-snapshots-$DATE" echo "`date`: Creating Snapshot..." >> /var/log/cron_snapshots.log echo `date` >> /var/log/cron_snapshots.log curl -s -XPUT $URL -d' { "indices": "pulse-data-2016.02,pulse-rules-2016.06.29", "ignore_unavailable": true, "include_global_state": false }' >> /var/log/cron_snapshots.log Active Snapshot Code
28. •Inclusion of X-Pack •Monitoring Cluster - being able to troubleshoot our ES Cluster after it crashes •Machine Learning •Managed Data Clusters •Managed Logging Clusters The Future of the Elastic Stack @ IST Research
29. Thanks to Elastic!
30. Paul Privateer, Founding Director Reducing school violence and bullying
31. Break Ups, Bad Grades, and Badass Leather Coats: Elasticsearch Helping NoSchoolViolence.org Reduce School Violence Paul Michael Privateer, Ph.D.
32. 32 there can be no keener revelation of a society's soul than the way in which it treats its children nelson mandela
33. 33 { Who } { Why } { What } { When} 0111001 01110101 01100101 01110011 01110100 01101001 0110111 01101110
34. NoSchoolViolence.org a non-profit national initiative public health epidemic adolescent behavior exploring school violence with a new lens Who
35. Goals The Lantern® App correlate search terms related to observed adolescent behaviors with relative risk levels of specific types of school violence get the Lantern® on desktops use results to develop “smart” intervention sessions reduce school violence before it happens vs. the current “response” approach
36. Why?
37. Why? 2016: 757,000 recorded violent school incidents 2001-2016: 515,000 average of violent school incidences 1990-2016: 2125 weapons fatalities (Virginia Tech/UTexas 32, Newtown 27, Columbine 13) National Center for Education Statistics ENOUGH! WE’VE HAD ENOUGH
38. Effects of School Violence: The Future At Risk
39. What | How + NSV Lantern® = Behavior Query ≈ 12 Form of Violence Proven Value of Big Data Behavioral Studies: Medicine, Crime, Consumer, Infrastructure, Law etc.
40. 40 Demo The Lantern®
41. 41 Why Elasticsearch? UK/MIT Connection Developing The Lantern® - Proof of Concept • indexing of research docs in Elasticsearch mode; • each doc parsed in Python and split into several fields; • built Lantern® as a Python Flask app; • Lantern® passes search terms into Elasticsearch index as a bool should-match filter; • Lantern® uses count-weighted match rate; • submission results are relayed in the form of violence assessment table
42. When 42 2017 Initiatives Goals: grow the database and create more effective Lantern® functionalities • create a Seattle national office • hire social scientists /data science team/school outreach reps • develop sponsorship program • launch crowdfunding initiatives • seek donors (expertise/funding) • help with growing a 501 (3)
43. Next – The Near Future 43 Expand App to • adolescent health, • academic success, • drug abuse, • relationship successes, • sexuality • negotiation skills
44. Conclusion • protecting school children is everyone’s business • NoSchoolViolence.org must grow to meet the epidemic levels of school violence • we need the community - Elasticsearch users - the tech community - corporate sponsors looking to enhance their social responsibility - educators, mental health specialists parents to test and apply the Lantern® 44
45. NoSchoolViolence.org The Lantern® Seattle | Washington A Seattle Data For Good Project 800-215-0415 contact@noschoolviolence.org 45
46. www.elastic.co
47. Adam Thompson, Executive Director Monitoring and tracking ebola cases
48. eHealth Africa March 9, 2017 @eHealth_Africa Using Elastic Stack to combat Ebola Adam Thompson, Executive Director
49. Agenda 4 9 1 eHealth Africa 2 Ebola outbreak 2014 3 Call Center application 4 Use of Elastic Stack 5 Q&A
50. eHealth Africa (eHA) builds stronger health systems through the design & implementation of data- driven solutions that respond to local needs and provide underserved communities with tools to lead healthier lives
51. 51 This is a sample image About eHA • Founded in 2009 • Almost 800 employees • Offices in Nigeria, Sierra Leone, Liberia, Guinea, the US and Germany • Develops mobile- and desktop-based solutions => Manage patient information => Track system-level data => Analyze programmatic outcomes
52. 52 Health Delivery Public Health Emergency Management Five programs for … Systems Disease Surveillance Laboratory & Diagnostic Nutrition & Food Security
53. • Hit Liberia, Sierra Leone and Guinea • Ebola control: isolating any infected individuals and tracking down those who have come into contact with them • eHA partnered with the ministries of health of all three countries to help them rapidly scale up existing call centers Ebola outbreak in West Africa 2014 5 3
54. EMERGENCY HOTLINES SEVERELY LIMITED •NUMBER OFTEN ENGAGED •PAPER-BASED DATA COLLECTION
55. 5 5
56. • Several reliable lines and dozens of operators working at the same time • Captures information on incoming calls • Generates daily reports on call volume, geographical distribution of call origins and alert types • Captures information about outcome of actions • Information used to make evidence-based recommendations, e.g. refocusing country teams on the ground to cover Ebola hotspots, locate and isolate infected individuals • In 2015, over 5,000,000 calls were managed by the application in Guinea, Sierra Leone and Liberia Call Center application 5 6
57. 5 7
58. 58 DEMO TO BE INSERTED
59. 59 AngularJS CouchDB Elasticsearch Kibana • allows only primitive indexing • enables full-text searches on the call records • enables sorting of records via multiple criteria • visualizes calls and alerts in real time • analyzes call volume, reported cases vs. expected Technical details
60. 6 0
61. Adam Thompson adam@eHealthAfrica.org Questions?
62. www.elastic.co
63. Q&A with Elastic Founders Steven Schuurman, Uri Boness, Simon Willnauer Moderated by Marty Messer