$30 off During Our Annual Pro Sale. View Details »

Advanced Bonsai Workshop #1: From databases to data streams

glopesdev
January 18, 2016

Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at:
https://www.youtube.com/watch?v=I8qwJXVghQ0

glopesdev

January 18, 2016
Tweet

More Decks by glopesdev

Other Decks in Programming

Transcript

  1. Advanced Bonsai Workshop
    https://bitbucket.org/horizongir/bonsai
    January 18-21, 2016
    Bonsai = Rx + VPL + Packages

    View Slide

  2. Be Reactive
    Ask Questions

    View Slide

  3. Workshop Programme
    1. From databases to data streams
    2. Bonsai toolboxes and visual editor
    3. Observable combinators: An algebra for data streams
    4. Representing discrete states with data streams

    View Slide

  4. From Databases to
    Data Streams
    January 18, 2016
    Advanced Bonsai Workshop

    View Slide

  5. There are (almost) as many data structures as
    there are algorithms
    Array 1D
    Array 2D
    Array 3D
    Linked List
    Tree
    Graph

    View Slide

  6. Many bulk operations over data are independent
    of the details of the data structure
    Ex:

    View Slide

  7. Many bulk operations over data are independent
    of the details of the data structure
    Ex:

    View Slide

  8. Many bulk operations over data are independent
    of the details of the data structure
    Ex:

    View Slide

  9. Enumerable streams
    interface IEnumerable
    {
    IEnumerator GetEnumerator();
    }
    interface IEnumerator : IDisposable
    {
    bool MoveNext();
    T Current { get; }
    }

    View Slide

  10. Programming with streams
    // sum of all numbers
    var sum = numbers.Sum();
    // sum of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Sum();
    // sum of the square of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();

    View Slide

  11. Where

    View Slide

  12. Select

    View Slide

  13. Sum

    View Slide

  14. Programming with streams
    // sum of all numbers
    var sum = numbers.Sum();
    // sum of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Sum();
    // sum of the square of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();

    View Slide

  15. A data table is a stream of rows
    Name Age Height Weight
    João 30 1.60 60
    Maria 24 1.70 54
    Pedro 27 1.90 70
    Ana 44 1.55 62

    View Slide

  16. Streams don’t even need a data structure
    class RandomStream : IEnumerable
    {
    public IEnumerator GetEnumerator() { return new RandomIterator(); }
    class RandomIterator : IEnumerator
    {
    int current;
    Random random = new Random();
    public int Current { get { return current; } }
    public bool MoveNext()
    {
    current = random.Next();
    return true;
    }
    public void Dispose() { }
    }
    }

    View Slide

  17. Iterator blocks provide convenient syntax for
    creating enumerable streams
    IEnumerable Generator()
    {
    yield return 0;
    yield return 1;
    yield return 2;
    }

    View Slide

  18. Iterator blocks provide convenient syntax for
    creating enumerable streams
    IEnumerable RandomNumbers()
    {
    var random = new Random();
    while (true)
    {
    yield return random.Next();
    }
    }

    View Slide

  19. Not every stream is waiting to be picked

    View Slide

  20. The real world doesn’t wait for you!
    Hardware
    Interrupts
    User-Interface
    Events
    Button click
    Monitoring
    Systems

    View Slide

  21. Enumerable streams
    interface IEnumerable
    {
    IEnumerator GetEnumerator();
    }
    interface IEnumerator : IDisposable
    {
    bool MoveNext();
    T Current { get; }
    }

    View Slide

  22. Observable streams
    interface IObservable
    {
    IDisposable Subscribe(IObserver observer);
    }
    interface IObserver
    {
    void OnCompleted();
    void OnNext(T value);
    void OnError(Exception error);
    }

    View Slide

  23. Pull streams vs Push streams

    View Slide

  24. Programming with observable streams
    // sum of all measurements
    var sum = measurements.Sum();
    // sum of all measurements greater than 10
    var sum = measurements.Where(x => x > 10)
    .Sum();
    // sum of the square of all measurements greater than 10
    var sum = measurements.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();

    View Slide

  25. Where

    View Slide

  26. Select

    View Slide

  27. Sum

    View Slide

  28. Single value sequence
    public class ValueSource : Source
    {
    public int Value { get; set; }
    public override IObservable Generate()
    {
    return Observable.Return(Value);
    }
    }

    View Slide

  29. Single value asynchronous sequence
    public class AsyncValue : Source
    {
    public int Value { get; set; }
    public override IObservable Generate()
    {
    return Observable.Start(() =>
    {
    Thread.Sleep(1000);
    return Value;
    });
    }
    }

    View Slide

  30. Multi-value sequence
    public class EnumerableSource : Source
    {
    public override IObservable Generate()
    {
    return new[] { 0, 1, 2, 3, 4 }.ToObservable();
    }
    }

    View Slide

  31. Multi-value asynchronous sequence
    (generator)
    public class AsyncEnumerable : Source
    {
    IEnumerable Generator()
    {
    for (int i = 0; i < 5; i++)
    {
    Thread.Sleep(1000);
    yield return i;
    }
    }
    public override IObservable Generate()
    {
    return Generator().ToObservable(Scheduler.Default);
    }
    }

    View Slide

  32. Polling a device
    public class CameraSource : Source
    {
    IEnumerable FrameCapture()
    {
    using (var camera = Capture.CreateCameraCapture(0))
    {
    while (true)
    {
    var image = camera.QueryFrame();
    yield return image;
    }
    }
    }
    public override IObservable Generate()
    {
    return FrameCapture().ToObservable(Scheduler.Default);
    }
    }

    View Slide

  33. A simple transform operator
    public class ScaleTransform : Transform
    {
    public int Value { get; set; }
    public override IObservable Process(IObservable source)
    {
    return source.Select(x => x * Value);
    }
    }

    View Slide

  34. Transform with mutable state (memory)
    public class AccumulatorTransform : Transform
    {
    public override IObservable Process(IObservable source)
    {
    return Observable.Defer(() =>
    {
    var accumulator = 0;
    return source.Select(x => accumulator += x);
    });
    }
    }

    View Slide

  35. Encapsulating processing side-effects in a Sink
    public class FileSink : Sink
    {
    public string FileName { get; set; }
    public override IObservable Process(IObservable source)
    {
    return Observable.Using(
    () => new StreamWriter(FileName),
    stream => source.Do(x => stream.WriteLine(x)));
    }
    }

    View Slide

  36. Workshop Project Assignment
    Code your own Source, Transform or Sink for Bonsai
    • Web Sockets
    • DAQ Boards
    • Cameras
    • Joysticks
    • Tracking
    • Spike Sorting
    • Classifiers
    • File formats (NEV,HDF5)
    • Visualizations

    View Slide