Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at:
https://www.youtube.com/watch?v=I8qwJXVghQ0

Transcript

1. Advanced Bonsai Workshop
   https://bitbucket.org/horizongir/bonsai
   January 18-21, 2016
   Bonsai = Rx + VPL + Packages
   

   View Slide

2. Be Reactive
   Ask Questions
   

   View Slide

3. Workshop Programme
   1. From databases to data streams
   2. Bonsai toolboxes and visual editor
   3. Observable combinators: An algebra for data streams
   4. Representing discrete states with data streams
   

   View Slide

4. From Databases to
   Data Streams
   January 18, 2016
   Advanced Bonsai Workshop
   

   View Slide

5. There are (almost) as many data structures as
   there are algorithms
   Array 1D
   Array 2D
   Array 3D
   Linked List
   Tree
   Graph
   

   View Slide

6. Many bulk operations over data are independent
   of the details of the data structure
   Ex:
   

   View Slide

7. Many bulk operations over data are independent
   of the details of the data structure
   Ex:
   

   View Slide

8. Many bulk operations over data are independent
   of the details of the data structure
   Ex:
   

   View Slide

9. Enumerable streams
   interface IEnumerable
   {
   IEnumerator GetEnumerator();
   }
   interface IEnumerator : IDisposable
   {
   bool MoveNext();
   T Current { get; }
   }
   

   View Slide

10. Programming with streams
    // sum of all numbers
    var sum = numbers.Sum();
    // sum of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Sum();
    // sum of the square of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();
    

    View Slide

11. Where
    

    View Slide

12. Select
    

    View Slide

13. Sum
    

    View Slide

14. Programming with streams
    // sum of all numbers
    var sum = numbers.Sum();
    // sum of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Sum();
    // sum of the square of all numbers greater than 10
    var sum = numbers.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();
    

    View Slide

15. A data table is a stream of rows
    Name Age Height Weight
    João 30 1.60 60
    Maria 24 1.70 54
    Pedro 27 1.90 70
    Ana 44 1.55 62
    

    View Slide

16. Streams don’t even need a data structure
    class RandomStream : IEnumerable
    {
    public IEnumerator GetEnumerator() { return new RandomIterator(); }
    class RandomIterator : IEnumerator
    {
    int current;
    Random random = new Random();
    public int Current { get { return current; } }
    public bool MoveNext()
    {
    current = random.Next();
    return true;
    }
    public void Dispose() { }
    }
    }
    

    View Slide

17. Iterator blocks provide convenient syntax for
    creating enumerable streams
    IEnumerable Generator()
    {
    yield return 0;
    yield return 1;
    yield return 2;
    }
    

    View Slide

18. Iterator blocks provide convenient syntax for
    creating enumerable streams
    IEnumerable RandomNumbers()
    {
    var random = new Random();
    while (true)
    {
    yield return random.Next();
    }
    }
    

    View Slide

19. Not every stream is waiting to be picked
    

    View Slide

20. The real world doesn’t wait for you!
    Hardware
    Interrupts
    User-Interface
    Events
    Button click
    Monitoring
    Systems
    

    View Slide

21. Enumerable streams
    interface IEnumerable
    {
    IEnumerator GetEnumerator();
    }
    interface IEnumerator : IDisposable
    {
    bool MoveNext();
    T Current { get; }
    }
    

    View Slide

22. Observable streams
    interface IObservable
    {
    IDisposable Subscribe(IObserver observer);
    }
    interface IObserver
    {
    void OnCompleted();
    void OnNext(T value);
    void OnError(Exception error);
    }
    

    View Slide

23. Pull streams vs Push streams
    

    View Slide

24. Programming with observable streams
    // sum of all measurements
    var sum = measurements.Sum();
    // sum of all measurements greater than 10
    var sum = measurements.Where(x => x > 10)
    .Sum();
    // sum of the square of all measurements greater than 10
    var sum = measurements.Where(x => x > 10)
    .Select(x => x * x)
    .Sum();
    

    View Slide

25. Where
    

    View Slide

26. Select
    

    View Slide

27. Sum
    

    View Slide

28. Single value sequence
    public class ValueSource : Source
    {
    public int Value { get; set; }
    public override IObservable Generate()
    {
    return Observable.Return(Value);
    }
    }
    

    View Slide

29. Single value asynchronous sequence
    public class AsyncValue : Source
    {
    public int Value { get; set; }
    public override IObservable Generate()
    {
    return Observable.Start(() =>
    {
    Thread.Sleep(1000);
    return Value;
    });
    }
    }
    

    View Slide

30. Multi-value sequence
    public class EnumerableSource : Source
    {
    public override IObservable Generate()
    {
    return new[] { 0, 1, 2, 3, 4 }.ToObservable();
    }
    }
    

    View Slide

31. Multi-value asynchronous sequence
    (generator)
    public class AsyncEnumerable : Source
    {
    IEnumerable Generator()
    {
    for (int i = 0; i < 5; i++)
    {
    Thread.Sleep(1000);
    yield return i;
    }
    }
    public override IObservable Generate()
    {
    return Generator().ToObservable(Scheduler.Default);
    }
    }
    

    View Slide

32. Polling a device
    public class CameraSource : Source
    {
    IEnumerable FrameCapture()
    {
    using (var camera = Capture.CreateCameraCapture(0))
    {
    while (true)
    {
    var image = camera.QueryFrame();
    yield return image;
    }
    }
    }
    public override IObservable Generate()
    {
    return FrameCapture().ToObservable(Scheduler.Default);
    }
    }
    

    View Slide

33. A simple transform operator
    public class ScaleTransform : Transform
    {
    public int Value { get; set; }
    public override IObservable Process(IObservable source)
    {
    return source.Select(x => x * Value);
    }
    }
    

    View Slide

34. Transform with mutable state (memory)
    public class AccumulatorTransform : Transform
    {
    public override IObservable Process(IObservable source)
    {
    return Observable.Defer(() =>
    {
    var accumulator = 0;
    return source.Select(x => accumulator += x);
    });
    }
    }
    

    View Slide

35. Encapsulating processing side-effects in a Sink
    public class FileSink : Sink
    {
    public string FileName { get; set; }
    public override IObservable Process(IObservable source)
    {
    return Observable.Using(
    () => new StreamWriter(FileName),
    stream => source.Do(x => stream.WriteLine(x)));
    }
    }
    

    View Slide

36. Workshop Project Assignment
    Code your own Source, Transform or Sink for Bonsai
    • Web Sockets
    • DAQ Boards
    • Cameras
    • Joysticks
    • Tracking
    • Spike Sorting
    • Classifiers
    • File formats (NEV,HDF5)
    • Visualizations
    

    View Slide