Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Warp 10: Collect, store and manipulate sensor data - BreizhCamp 2016

Warp 10: Collect, store and manipulate sensor data - BreizhCamp 2016

Horacio Gonzalez

March 24, 2016

More Decks by Horacio Gonzalez

Other Decks in Technology


  1. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Time series storage and analysis Image:

    Hamza Fessi and ABC Bourse Not suited for your vanilla SQL RDBMS One simple example: moving averages...
  2. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Warp 10 : A software platform

    for IoT Warp 10 is a software platform that • Ingests and stores data • Manipulates and analyzes data • Is dedicated to data from sensors, meters, IoT and any real or virtual probe
  3. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Warp 10 General Synoptic Stockage Architecture

    Language, Functions, Algorithms Application access Vizualisation Real Time
  4. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Warp 10 GTS Input Format 57

    bytes But size isn't the most important reason parsing time is way more important XML or even JSON parsing is slow and costly Warp 10 GTS input format isn't
  5. @FinistSeb @LostInBrittany #BzhCmp #Warp10 timestamp (us by default) latitude:longitude (WGS84)

    elevation (millimeters) classname* labels (key=value) value* (long, double, boolean or string) * mandatory fields Warp 10 GTS Input Format
  6. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Warp 10 on a cluster 3

    millions of datapoints per second (our current record on input traffic)
  7. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Manipulating GTS To be scalable, analysis

    must be done in Warp 10 platform, not in user's computer
  8. @FinistSeb @LostInBrittany #BzhCmp #Warp10 A true GTS analysis toolbox ◦

    Hundreds of functions ◦ Manipulation frameworks ◦ Analysis workflow Manipulating GTS
  9. @FinistSeb @LostInBrittany #BzhCmp #Warp10 GTS manipulation Why not a simple

    REST API? • One endpoint by function? • How to chain a workflow analysis? REST API not suitable for complex manipulations
  10. @FinistSeb @LostInBrittany #BzhCmp #Warp10 GTS manipulation Why not a SQL

    dialect? • How do you do a simple moving average in SQL? • How do you geo-time fencing in SQL? SQL is not adapted to (G)TS analysis!
  11. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Geo-Time Series functions Geo mapping (WKT)

    Horizontal & vertical speed Horizontal & vertical distance Haversine ...
  12. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Fuel prices data 16 297 448

    metrics 11 379 fuel stations 42 885 Geo Time Series
  13. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Calculate the average with WarpScript BUCKETIZE

    framework Put the data of a GTS into regularly spaced buckets
  14. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Calculate the average with WarpScript REDUCE

    framework Apply a function on a set of GTS tick by tick
  15. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Basic analysis Mean of the last

    available diesel fuel prices in France Image: LEGO Ideas
  16. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Mean of those last prices align

    ticks with BUCKETIZE framework compute the average with REDUCE
  17. @FinistSeb @LostInBrittany #BzhCmp #Warp10 Geo-filtering points of GTS MAPPER framework

    Apply a function on values of a GTS that fall into a sliding window