Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Electricsearch from 0 to 400 volts at Blue-Solutions by Polyconseil

Elastic Co
November 09, 2016

Electricsearch from 0 to 400 volts at Blue-Solutions by Polyconseil

At Blue-Solutions we use Elasticsearch to store and index telemetry data from thousands of blue devices such as electric vehicles, charging points, batteries, etc. to send data to our backend applications, generate alerts and gain insight on their use. In this talk we will describe how we fully automated Elasticsearch cluster deployments (on-premise) using Ansible and how we drill down into the data in our blue devices even further.

Benoit Calvez | Sr. Developer | Polyconseil
Thierno Barry | Technical Architect | Polyconseil

Elastic Co

November 09, 2016
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Benoit Calvez | Sr. Developer Thierno Barry | Technical Architect

    Electric {search} from 0 to 400 volts Blue Solutions by Polyconseil November 9, 2016
  2. by Agenda •  Deployment automation with Ansible •  Use case

    #1: Network events •  Use case #2: Charging point events •  Use case #3: Primary telemetry storage •  Use case #4: Business events storage •  What’s next? 2
  3. by 4 100% electric car sharing (Personal & Enterprise) 4000

    cars, 6000 charge points on 1100 stations (Paris) Opened in Paris, Lyon, Bordeaux, London, Indianapolis and Turin by … soon in Singapore
  4. by 8 Automate Deployments with Ansible •  Use the official

    ansible-elasticsearch role https://github.com/elastic/ansible-elasticsearch •  Custom playbooks and roles to –  Activate offline resources (RAM and CPU) –  Activate Elasticsearch –  Configure security features (formerly Shield) in X- Pack –  Deploy templates –  Install and configure Curator –  Deploy backup scripts ❤
  5. by 9 Automate Deployments with Ansible •  Wrote custom Ansible’s

    modules •  Pushing back to ansible-modules-extras –  Manage Elasticsearch plugins (Ansible 2.2 and 2.1.2) –  Manage Kibana plugins (Ansible 2.2) –  Manage users and roles using security features (formerly Shield) in X-Pack (on-going) ❤
  6. by 14 Numbers * Numbers per node BCA Netflow Nodes

    3 6 Documents 1.5 M per day 80 M per day Indices 1 per day 1 per hour RAM 16 Go 64 Go HDD 300 Go 300 Go
  7. by 15 Learning Curve •  First clusters we ran • 

    Automatization with Ansible •  Machine sizing (java) •  Cluster sizing •  Supervision / Metrology
  8. by Business Events Storage (Loglady / Columbo) 19 Loglady /

    Columbo APP #1 APP #2 APP #3 APP #N … AMQP AMQP
  9. by 22 Numbers * Numbers per node Bluecap Loglady/Columbo Nodes

    3 3 Documents 3 M per day 15 M per day Indices 1 per day No time based indices RAM 32 Go 64 Go HDD 300 Go 300 Go
  10. by 23 Learning Curve •  Re-indexation, the hard way • 

    Rolling upgrade •  Snapshots, thanks to Curator ❤ •  Timelion ❤
  11. by Lessons Learned 24 •  Automate everything you can § 

    Use automation tools like Ansible, Puppet, Chef, etc. •  For number of shards, start big §  To avoid unnecessary re-sharding when adding node •  Curator 4 is awesome §  Snapshot / Restore §  Closing / Opening indices §  Deleting indices
  12. by Lessons Learned 25 •  Keep re-indexation in mind and

    industrialize it §  Use Logstash §  How do you update your documents? o  Build your own re-indexer? §  Use SSD •  Don’t forget monitoring (Marvel vs Prometheus vs Sensu)
  13. by What’s next Technical •  Build our own Re-indexer § 

    async? §  In dedicated cluster? •  Upgrade to Elastic Stack 5 §  Pipelines with ingest nodes? •  Run it on our kubernetes? 27 Business •  Reporting •  Track batteries and hardware fatigue •  Use telemetry data as feedback for hardware design •  Turn technical data into business insight… •  … and new business opportunities
  14. Except where otherwise noted, this work is licensed under hBp://creaUvecommons.org/licenses/by-nd/4.0/

    CreaUve Commons and the double C in a circle are registered trademarks of CreaUve Commons in the United States and other countries. Third party marks and brands are the property of their respecUve holders.