Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paris Time Series Meetup #5 - FLoWS

Paris Time Series Meetup #5 - FLoWS

Edition 5 du Paris Time Series Meetup où Mathias Herberts, CTO de SenX et éditeur de la solution Warp10, nous présente FLoWS, le langage alternatif à WarpScript.

TimeSeriesFr

July 01, 2020
Tweet

More Decks by TimeSeriesFr

Other Decks in Technology

Transcript

  1. How you will
    Finally Love WarpScript
    and embrace True Time Series Analytics
    Virtual
    Mathias Herberts - CTO
    2020-07-01

    View Slide

  2. View Slide

  3. The analytics environment (library + language) of
    The Most Advanced Time Series Platform
    What is WarpScript?

    View Slide

  4. Origins

    View Slide

  5. Paradigm
    ■ Dynamically typed concatenative language, Turing complete
    ■ Functional by nature, everything is a function
    ■ Powerful macro mechanism with early/late binding
    ■ Fully extensible
    ■ Sandboxed
    ■ Designed for Geo Time Series manipulation

    View Slide

  6. In practice
    ■ Implemented around a stack data structure
    ■ Functions consume their parameters off the stack
    ■ Results are produced on the stack
    ■ Availability of variables and registers
    ■ Postfix notation (RPN - Reverse Polish Notation)
    ■ Interpreted with a commercially available compiler

    View Slide

  7. Why ?

    View Slide

  8. WarpScript adoption
    ■ Younger users struggle learning WarpScript
    ■ Yet they don’t really know what is slowing them down
    ■ They blame things which are not the root cause of their struggle
    ■ Result is they don’t persevere
    ■ And miss lots of great opportunities
    ■ Took us some time to really understand what needed to be changed

    View Slide

  9. What causes trouble?
    ■ The RPN is being blamed: 1 2 3 4 5 FUNC
    ■ But what is disturbing seems to be that you need to know how FUNC behaves
    ■ Clear parameter identification makes things clearer: 1 2 (3,4,5)FUNC
    ■ Still no clue about what FUNC produces
    ■ Mandatory assignment makes it better: (a,b) = (3,4,5)FUNC
    ■ This is conceptually what FLoWS brings to WarpScript

    View Slide

  10. Quick comparison
    ■ FLoWS uses prefix notation for function calls
    ■ Function parameters are specified in parentheses
    ■ Function results are either ignored, used as single values, assigned or returned
    ■ Some minor syntax additions
    ■ Those simple changes make it more accessible while preserving everything

    View Slide

  11. All of !

    View Slide

  12. Comments
    Comments are important to understand code!
    // C++ Style comments
    /*
    C-Style comments
    */

    View Slide

  13. Numbers
    FLoWS supports LONGs and DOUBLEs
    42 // LONG
    3.14 // DOUBLE
    1.0E-12 // DOUBLE

    View Slide

  14. Booleans
    true // Not false
    false // Not true
    maybe // Syntax error

    View Slide

  15. Character strings
    Percent encoded STRINGs using UTF-8, enclosed in single or double quotes
    'Hello'
    "%F0%9F%96%96" //
    'Multiline
    Strings'

    View Slide

  16. Lists
    Comma separated expressions
    [ 'Hello', 3.1415, 42 ]
    [ 'Hello', 3.1415,
    42 // Works on multiple lines too
    ]

    View Slide

  17. Maps
    Comma separated list of colon separated pairs of expressions (Key:Value)
    { 'A':65, '@':64, 64:'@' }
    { '@':64, 64:'@',
    'A':65 // Works on multiple lines too
    }

    View Slide

  18. Accessing map and list elements
    Assuming the previous list and map are stored in variables list and map
    map['A'] // 65
    map[64] // '@'
    list[0] // 'Hello'

    View Slide

  19. Function calls
    Comma separated list of expressions as function parameters
    F(1,2,'A',b)// F is the function name
    G() // Parameterless function call
    // Functions can return 0-N values

    View Slide

  20. Assignments
    Assignments assign values to variables
    A = 12
    (x, y) = F(1) // F MUST return two values
    M[0][1] = 3.14 // Assign to list/map element

    View Slide

  21. Macros
    Macros are sequences of statements
    M = (a,b,c) -> { // 3 parameters
    ...
    }

    View Slide

  22. Return values
    Macros can return values just like functions, return as last statement
    M = (a,b,c) -> { // 3 parameters
    return F(a,b), G(c)
    } // return 0 to N values

    View Slide

  23. Strict return values
    Expected number of return values can be enforced
    M = (a,b,c) -> 2 { // 2 return values
    return F(a,b), G(c)
    } // Error if more than 2 return values

    View Slide

  24. Macro calls
    As for functions, comma separated list of expressions as parameters
    @M(1,2,3) // @ prefix like in WarpScript
    (x,y) = @M(1,2,3) // Assign return values

    View Slide

  25. View Slide


  26. View Slide

  27. Full compatibility
    ■ All WarpScript functions, including from extensions, are available in FLoWS
    ■ FLoWS macros are WarpScript macros
    ■ WarpScript macros can be used as FLoWS macros
    ■ Variables are interchangeable, bang variables are supported: !$a ⇒ a!
    ■ FLoWS can call WarpScript and vice versa, macro resolution works the same
    ■ FLoWS code can be transpiled into WarpScript

    View Slide

  28. Minor differences
    ■ FLoWS macros have an enforced fixed number of parameters
    ■ Using some asynchronous transfer of control functions can be tricky
    ⇒ RETURN NRETURN STOP BREAK CONTINUE
    ■ Some exotic names cannot be used as is but can via EVAL
    ■ FLoWS makes extensive use of HIDE and SHOW to manage the stack
    ■ Variables are scoped to a macro and its descendants, no global variables

    View Slide

  29. Calling FLoWS from WarpScript
    The FLoWS WarpScript extension provides the FLOWS function
    A = 1
    '>
    FLOWS
    $A // Pushes 1 into the pipeline

    View Slide

  30. Calling FLoWS from WarpScript
    42 'B' STORE
    A = B // Variable B from WarpScript is available in FLoWS
    '>
    FLOWS
    $A // Pushes 42 onto the stack

    View Slide

  31. Calling WarpScript from FLoWS
    Simply use the EVAL function
    B = 42
    EVAL('$B "A" STORE') // WarpScript can access B, FLoWS will see A
    return A // returns 42

    View Slide

  32. Transpiling FLoWS
    The FLoWS WarpScript extension also provides the FLOWS-> function
    return (a,b) -> { return F(b,a) }
    '>
    FLOWS-> // Generates a macro of the FLoWS code
    'L1:9-1:34' SECTION
    SAVE '# 0' STORE
    // Storing macro parameters
    2 FLOWS.ASSERTDEPTH
    [ 'a' 'b' ] STORE
    // F(...)
    'L1:27-1:32' SECTION
    0 HIDE '# 2' STORE
    $b
    $a
    F
    '# 2' LOAD SHOW
    '# 0' LOAD RESTORE
    %> // END Macro definition #0
    %>

    View Slide

  33. Roadmap

    View Slide

  34. Availability
    ■ Immediately in the Warp 10 sandbox
    ■ As a WarpScript extension later this summer
    ■ Needs the latest Warp 10 code for HIDE/SHOW functions. Will need 2.7.0+
    ■ License not yet finalized
    ⇒ Most likely Commons Clause or BSL like

    View Slide

  35. Evolution
    ■ Documentation will provide FLoWS syntax for all functions
    ■ Examples will be adapted opportunistically
    ■ Optional dedicated endpoint will be created for Warp 10 instances
    ■ WarpStudio and VS Code support to come
    ■ Tutorials will be written to accelerate onboarding

    View Slide

  36. in Action

    View Slide

  37. Typical WarpScript flow
    NOW = NOW()
    GTS = FETCH([ 'TOKEN', '~.*', {}, NOW, -100 ])
    BUCKETIZED = BUCKETIZE([ GTS, bucketizer.last(), NOW, m(1), 0 ])
    return BUCKETIZED

    View Slide

  38. Use of macros
    // Inline anonymous macro
    return LMAP([ 1,2,3,4 ], (n) -> { return *(n,n) }, false)
    // Macro stored in a variable
    SQUARE = (n) -> { return *(n,n) }
    return LMAP([1,2,3,4], SQUARE, false)
    // Remote macro loaded by the WarpFleet resolver
    return @senx/geo/circle(48.0,-4.5,100)

    View Slide

  39. Conditionals
    (value,text) = IFTE(
    cond, // Assumed to contain a boolean
    () -> { return 1, 'true' }, // return two elements for value,text
    () -> { return 0, 'false' } // return two elements for value,text
    )

    View Slide

  40. Loops
    L = []
    FOR(1,10, (i) -> { +!(L,*(i,i)) })
    return L

    View Slide

  41. A more elaborate example
    ■ Trajectory of a vehicle timestamp,latitude,longitude
    ■ Determine at each timestamp the traveled distance since start

    View Slide

  42. Solution using the most popular time series platform
    import "math"
    import "experimental"
    planetRadiusKm = 6371.0
    // helper function to convert degrees to radians
    degreesToRadians = (tables=
    tables
    |> map(fn: (r) => ({ r with _value: r._value * math.pi / 180.0 }))
    // let's call all latitude values
    LATRAW = from(bucket: "my_bucket")
    |> range($range)
    |> filter(fn: (r) =>
    r._measurement == "gps" and
    r._field == "latitude"
    )
    |> degreesToRadians()
    |> aggregateWindow(every: $__interval, fn: mean)
    |> fill(column: "_value", usePrevious: true)
    // let's create the differences of all latitude values and shift them by one
    LATDIFF = from(bucket: "my_bucket")
    |> range($range)
    |> filter(fn: (r) =>
    r._measurement == "gps" and
    r._field == "latitude"
    )
    |> degreesToRadians()
    |> aggregateWindow(every: $__interval, fn: mean)
    |> difference(nonNegative: false, columns: ["_value"])
    |> timeShift(duration: -$__interval, columns: ["_start", "_stop", "_time"])
    |> fill(value: 0.0)
    // let's join both lat tables together,
    // so we have current and previous latitudes in one row
    LAT = join(tables: {raw: LATRAW, diff: LATDIFF}, on: ["_time"])
    |> sort()
    |> map(fn: (r) => ({
    _time: r._time,
    lat_curr: r._value_raw,
    lat_last: r._value_raw + r._value_diff
    }))
    // let's do this stuff again for the longitude values
    LONRAW = from(bucket: "my_bucket")
    |> range($range)
    |> filter(fn: (r) =>
    r._measurement == "gps" and
    r._field == "longitude"
    )
    |> degreesToRadians()
    |> aggregateWindow(every: $__interval, fn: mean)
    |> fill(column: "_value", usePrevious: true)
    LONDIFF = from(bucket: "my_bucket")
    |> range($range)
    |> filter(fn: (r) =>
    r._measurement == "gps" and
    r._field == "longitude"
    )
    |> degreesToRadians()
    |> aggregateWindow(every: $__interval, fn: mean)
    |> difference(nonNegative: false, columns: ["_value"])
    |> timeShift(duration: -$__interval, columns: ["_start", "_stop", "_time"])
    |> fill(value: 0.0)
    LON = join(tables: {raw: LONRAW, diff: LONDIFF}, on: ["_time"])
    |> sort()
    |> map(fn: (r) => ({
    _time: r._time,
    lon_curr: r._value_raw,
    lon_last: r._value_raw + r._value_diff
    }))
    // let's join lats and lons together, filter out NaNs (ugly),
    // apply haversine formula and accumulate the sums to get travel-distance
    join(tables: {lat:LAT, lon:LON}, on: ["_time"])
    |> filter(fn: (r) =>
    r.lat_curr >= -90.0 and
    r.lon_curr >= -90.0 and
    r.lat_last >= -90.0 and
    r.lon_last >= -90.0
    )
    |> map(fn: (r) => ({
    _time: r._time,
    _field: "travel-distance",
    _value: ( 2.0 * math.atan2(
    x: math.sqrt(x: (math.sin(x: (r.lat_curr - r.lat_last)/2.0) *
    math.sin(x: (r.lat_curr - r.lat_last)/2.0)) + (math.sin(x: (r.lon_curr -
    r.lon_last)/2.0) * math.sin(x: (r.lon_curr - r.lon_last)/2.0)) *
    math.cos(x: r.lat_curr) * math.cos(x: r.lat_last)),
    y: math.sqrt(x: 1.0 - (math.sin(x: (r.lat_curr - r.lat_last)/2.0) *
    math.sin(x: (r.lat_curr - r.lat_last)/2.0)) + (math.sin(x: (r.lon_curr -
    r.lon_last)/2.0) * math.sin(x: (r.lon_curr - r.lon_last)/2.0)) *
    math.cos(x: r.lat_curr) * math.cos(x: r.lat_last)))
    ) * planetRadiusKm * 1000.0
    }))
    |> cumulativeSum(columns: ["_value"])

    View Slide

  43. Solution using
    GTS = FETCH([ 'TOKEN', 'gps', {}, NOW(), h(2) ])
    DIST = MAP([ GTS, mapper.hdist(), 1, 0, 0 ])
    return MAP([ DIST, mapper.sum(), MAXLONG(), 0 0 ])

    View Slide

  44. The Surprise

    View Slide

  45. View Slide

  46. The Lounge supports both WarpScript and
    /flows L = []
    FOR(1,10, (i) -> { +!(L,*(i,i)) })
    return L
    /warpscript []
    1 10 FOR
    Execution is performed on the sandbox.senx.io instance

    View Slide

  47. Takeaways

    View Slide

  48. Takeaways
    ■ FLoWS is a novel and friendly syntax for using the WarpScript library
    ■ It is fully compatible with WarpScript
    ■ It is usable where WarpScript is, including in all integrations (Spark, Pig, …)
    ■ WarpScript code can be generated from FLoWS code
    ■ Deployed on the Warp 10 sandbox, available as an extension soon
    ■ Accessible from the Warp 10 Lounge which has just opened ⇒ lounge.warp10.io

    View Slide