Benoit Calvez | Sr. Developer Thierno Barry | Technical Architect Electric {search} from 0 to 400 volts Blue Solutions by Polyconseil November 9, 2016 

by Agenda •  Deployment automation with Ansible •  Use case #1: Network events •  Use case #2: Charging point events •  Use case #3: Primary telemetry storage •  Use case #4: Business events storage •  What’s next? 2 

by Thierno Barry I’m a devOPS barryib @barryib _nodes/dzen,barryib/info 3 Benoit Calvez I’m a DEVops dzen @dzen 

by 4 100% electric car sharing (Personal & Enterprise) 4000 cars, 6000 charge points on 1100 stations (Paris) Opened in Paris, Lyon, Bordeaux, London, Indianapolis and Turin by … soon in Singapore 

by Requirements and Challenges Collect them all… 5 

by Requirements and Challenges 6 

by Automated Deployment From Ansible with ❤… 7 

by 8 Automate Deployments with Ansible •  Use the official ansible-elasticsearch role https://github.com/elastic/ansible-elasticsearch •  Custom playbooks and roles to –  Activate offline resources (RAM and CPU) –  Activate Elasticsearch –  Configure security features (formerly Shield) in X- Pack –  Deploy templates –  Install and configure Curator –  Deploy backup scripts ❤ 

by 9 Automate Deployments with Ansible •  Wrote custom Ansible’s modules •  Pushing back to ansible-modules-extras –  Manage Elasticsearch plugins (Ansible 2.2 and 2.1.2) –  Manage Kibana plugins (Ansible 2.2) –  Manage users and roles using security features (formerly Shield) in X-Pack (on-going) ❤ 

by Our Use Cases You know, for {search} and more… 10 

by Charging Point Events (BCA) 11 Syslog Collectd Security (formerly Shield) Monitoring (formerly Marvel) 

by Network Events with Netflow 12 ne8low sflow Routeurs Switchs dvSwitchs 

by Network Events with Netflow (Kibana’s Dashboard) 13 

by 14 Numbers * Numbers per node BCA Netflow Nodes 3 6 Documents 1.5 M per day 80 M per day Indices 1 per day 1 per hour RAM 16 Go 64 Go HDD 300 Go 300 Go 

by 15 Learning Curve •  First clusters we ran •  Automatization with Ansible •  Machine sizing (java) •  Cluster sizing •  Supervision / Metrology 

by Primary Telemetry Storage (Bluecap) 16 Bluecap Charging Points Electric Vehicles BaBeries POP POP WSS AMQP AMQP 

by Primary Telemetry Storage (GSM Signal Quality) 17 

by Primary Telemetry Storage 18 

by Business Events Storage (Loglady / Columbo) 19 Loglady / Columbo APP #1 APP #2 APP #3 APP #N … AMQP AMQP 

by Business Events Storage (Columbo) 20 Keep calm and watch Columbo 

by Business Events Storage (Columbo) 21 * Made with ❤ by WiFirst 

by 22 Numbers * Numbers per node Bluecap Loglady/Columbo Nodes 3 3 Documents 3 M per day 15 M per day Indices 1 per day No time based indices RAM 32 Go 64 Go HDD 300 Go 300 Go 

by 23 Learning Curve •  Re-indexation, the hard way •  Rolling upgrade •  Snapshots, thanks to Curator ❤ •  Timelion ❤ 

by Lessons Learned 24 •  Automate everything you can §  Use automation tools like Ansible, Puppet, Chef, etc. •  For number of shards, start big §  To avoid unnecessary re-sharding when adding node •  Curator 4 is awesome §  Snapshot / Restore §  Closing / Opening indices §  Deleting indices 

by Lessons Learned 25 •  Keep re-indexation in mind and industrialize it §  Use Logstash §  How do you update your documents? o  Build your own re-indexer? §  Use SSD •  Don’t forget monitoring (Marvel vs Prometheus vs Sensu) 

by What’s next? To 400 volts… and beyond 26 

by What’s next Technical •  Build our own Re-indexer §  async? §  In dedicated cluster? •  Upgrade to Elastic Stack 5 §  Pipelines with ingest nodes? •  Run it on our kubernetes? 27 Business •  Reporting •  Track batteries and hardware fatigue •  Use telemetry data as feedback for hardware design •  Turn technical data into business insight… •  … and new business opportunities 

by Time for Questions Ask us anything 28 

www.elastic.co 29 

Except where otherwise noted, this work is licensed under hBp://creaUvecommons.org/licenses/by-nd/4.0/ CreaUve Commons and the double C in a circle are registered trademarks of CreaUve Commons in the United States and other countries. Third party marks and brands are the property of their respecUve holders.