Elastic{ON} Tour 2018 Munich : car2go

Elastic{ON} Tour Munich - February 1, 2018

Using Elasticsearch to drive car2go's realtime business insights from their connected cars

Car2go is an always-on business offering mobility service with cars to customers living in urban areas. Customers and cars constitute an IoT service generating data which must be processed and analyzed in real-time. E.g. vehicle connectivity and condition, position data, reservation and payment, registration and validation. Elasticsearch was introduced to all development teams as an offering, and as a result high quality data analysis can be generated based on systems inside status to all parts of the organization. Using DevOps methods each team is able to implement, modify and visualize data effectively. This gives a fast understanding of capacity, errors and business opportunities in real-time.

Erik Bak-Mikkelsen | Head of Cloud Operations | car2go
Marius Svechla | DevOps Engineer | car2go

Transcript

1. 1 Erik Bak-Mikkelsen Marius Svechla, @msvechla 02/01/2018, elastic on Munich, car2go Using Elasticsearch to drive car2go's realtime business insights from their connected cars
2. 2 E Amsterdam Berlin Florence Frankfurt Hamburg E Madrid Milan Munich Rhineland Rome E Stuttgart Turin Vienna Chongqing Austin Brooklyn (NYC) Calgary Columbus Denver Montreal Portland Seattle Toronto Vancouver Washington D.C. Every 1.3 SECOND a CAR2GO is RENTED by one of our 2.9 MILLION MEMBERS around the world WE OPERATE THE LARGEST FREE-FLOATING FLEET IN URBAN CENTERS AROUND THE WORLD
3. 3 A Brief Look at Some of our Challenges • Observing rentals per city • Discover business opportunities • Monitoring car connectivity • Relocating vehicles • Acquiring new customers • Analyzing fraud cases • … • Physical / virtual servers • Containers (Docker / Kubernetes) • Capacity planning / Updates • Microservice Architecture • Debugging • Intrusion detection • … • Distributed Teams • Location Managers • Corporate audits • Regulations • … Business Technical Company
4. 4 Common Denominator of our Challenges A data-driven environment • Systems producing large amount of data • Aggregating distributed information in a single point of truth • Mastering the analysis of our big data environment as a key to success • Every implementation should follow a clear business need
5. 5 Why the elastic stack? Reasons for choosing the elastic stack • Open source / core • Easy to get started • Great community • High availability • Visualizations for every use-case • Enterprise-level features – LDAP Authentication – Field-level permissions – Machine Learning – Reporting...
6. 6 Milestones Along our Journey How we went from zero to a multi-cluster / datacenter setup 1 2 3 4 5 2014 First Tests at Daimler Internal Datacenter ELK v1.x Q1 2016 Dual-Datacenter Setup Single Cluster Across DCs Upgrade to ELK v2.x Q3 2016 Architecture Redesign Multi-Cluster / Tribe / Hot-Warm ELK v2.x Q1 2017 Beats Upgrade to v5.x Elastic Stack v5.x Q3 2017 Platinum License Security, Monitoring, Reporting, ML … Elastic Stack v5.x
7. 7 7 Our Elastic Landscape Today Elastic fully integrated
8. 8 Elastic at car2go technical overview Beats Baremetal Virtual Redis Messaging Queue Nodes (X) Logstash Indexer Elasticsearch Kibana X-pack Authentication X-pack LDAP Instances (X) Master Nodes (3) SearchNodes (3) Data Nodes - Hot (3) Data Nodes - Warm (3) Redis Redis Cluster: amsterdam-prod Cluster: frankfurt-prod Cluster: xyz Nodes (X) Tribenode HAProxy HAProxy HAProxy Logstash Docker Containers Kubernetes Orchestrated Containers fluentd Filebeat Tools ES Curator / ElastAlert / Other Fraudster
9. 9 Elastic Stack as a Self- Service No handover, full-throttle DevOps • Infrastructure as Code • Simplifying usage for our developers • Sane defaults and templates for Logstash, Beats, elastalert / Watcher…
10. 10 Ingesting a new Type of Log to Elasticsearch Via infrastructure as code self-service file { '/etc/logstash/conf.d/curator_logstash.conf': owner => logstash, content => ' input { file { type => "curator" path => ["/var/log/curator.log"] codec => json add_field => {"_project" => "curator" } add_field => {"_rollover" => "MONTHLY" } sincedb_path => "/var/lib/logstash/.sincedb_curator" } }', notify => Service['logstash'], }
11. 11 Logstash Tangibles Beats Containers 500 1K 3K ES Clusters Elasticsearch Nodes Hot & Warm Data 6 100 35 TB
12. 12 How are we using this data to tackle our challenges? Analyzing terabytes of data to gain insights
13. 13 Operations: How are we doing right now?
14. 14 Operations: Analyzing affected locations during outages
15. 15 Operations: Vehicle Saturation of a City
16. 16 Technical Overview: Vehicle Events
17. 17 Insights: Customer License Validation
18. 18 18 File With 1.4 Billion Hacked And Leaked Passwords Found On The Dark Web Headline from Forbes Magazine, Dec. 2017
19. 19 Intrusion Detection: Login Attacks
20. 20 Intrusion Detection: Machine Learning
21. 21 Intrusion Detection: Drill- Down
22. 22 Best Practices What we learned from running the Elastic Stack in production for more than 3 years
23. 23 If it hurts, do it more often! DevOps Mantra
24. 24 Gather metrics of your cluster and improve Embrace transparency! • Keep an eye on index stats • Collect cluster stats • Container logs via rsyslog • Automate tasks with curator • X-Pack Monitoring for detailed recovery information
25. 25 25 Use rsyslog logging- driver to ship container logs via Logstash
26. 26 26 Use Elasticsearch Curator to automate common cluster tasks
27. 27 Outlook Future projects and planned improvements
28. 28 Outlook • Elasticsearch beyond logging for upcoming microservices • Machine Learning deep-dive • Tribenode to Cross-Cluster Search • Elastalert to Watcher / ML • Fluentd to Filebeat for Kubernetes logs Future projects and improvement ideas
29. 29 PROUD TO SHARE Thank you Any Questions?