Advanced Bonsai Workshop #1: From databases to data streams

Cff520b66669064bbfefdc9109731160?s=47 glopesdev
January 18, 2016

Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at:
https://www.youtube.com/watch?v=I8qwJXVghQ0

Cff520b66669064bbfefdc9109731160?s=128

glopesdev

January 18, 2016
Tweet

Transcript

  1. 3.

    Workshop Programme 1. From databases to data streams 2. Bonsai

    toolboxes and visual editor 3. Observable combinators: An algebra for data streams 4. Representing discrete states with data streams
  2. 5.

    There are (almost) as many data structures as there are

    algorithms Array 1D Array 2D Array 3D Linked List Tree Graph
  3. 10.

    Programming with streams // sum of all numbers var sum

    = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
  4. 11.
  5. 12.
  6. 13.

    Sum

  7. 14.

    Programming with streams // sum of all numbers var sum

    = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
  8. 15.

    A data table is a stream of rows Name Age

    Height Weight João 30 1.60 60 Maria 24 1.70 54 Pedro 27 1.90 70 Ana 44 1.55 62
  9. 16.

    Streams don’t even need a data structure class RandomStream :

    IEnumerable<int> { public IEnumerator<int> GetEnumerator() { return new RandomIterator(); } class RandomIterator : IEnumerator<int> { int current; Random random = new Random(); public int Current { get { return current; } } public bool MoveNext() { current = random.Next(); return true; } public void Dispose() { } } }
  10. 17.

    Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int>

    Generator() { yield return 0; yield return 1; yield return 2; }
  11. 18.

    Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int>

    RandomNumbers() { var random = new Random(); while (true) { yield return random.Next(); } }
  12. 22.

    Observable streams interface IObservable<T> { IDisposable Subscribe(IObserver<T> observer); } interface

    IObserver<T> { void OnCompleted(); void OnNext(T value); void OnError(Exception error); }
  13. 24.

    Programming with observable streams // sum of all measurements var

    sum = measurements.Sum(); // sum of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Sum(); // sum of the square of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Select(x => x * x) .Sum();
  14. 25.
  15. 26.
  16. 27.

    Sum

  17. 28.

    Single value sequence public class ValueSource : Source<int> { public

    int Value { get; set; } public override IObservable<int> Generate() { return Observable.Return(Value); } }
  18. 29.

    Single value asynchronous sequence public class AsyncValue : Source<int> {

    public int Value { get; set; } public override IObservable<int> Generate() { return Observable.Start(() => { Thread.Sleep(1000); return Value; }); } }
  19. 30.

    Multi-value sequence public class EnumerableSource : Source<int> { public override

    IObservable<int> Generate() { return new[] { 0, 1, 2, 3, 4 }.ToObservable(); } }
  20. 31.

    Multi-value asynchronous sequence (generator) public class AsyncEnumerable : Source<int> {

    IEnumerable<int> Generator() { for (int i = 0; i < 5; i++) { Thread.Sleep(1000); yield return i; } } public override IObservable<int> Generate() { return Generator().ToObservable(Scheduler.Default); } }
  21. 32.

    Polling a device public class CameraSource : Source<IplImage> { IEnumerable<IplImage>

    FrameCapture() { using (var camera = Capture.CreateCameraCapture(0)) { while (true) { var image = camera.QueryFrame(); yield return image; } } } public override IObservable<IplImage> Generate() { return FrameCapture().ToObservable(Scheduler.Default); } }
  22. 33.

    A simple transform operator public class ScaleTransform : Transform<int, int>

    { public int Value { get; set; } public override IObservable<int> Process(IObservable<int> source) { return source.Select(x => x * Value); } }
  23. 34.

    Transform with mutable state (memory) public class AccumulatorTransform : Transform<int,

    int> { public override IObservable<int> Process(IObservable<int> source) { return Observable.Defer(() => { var accumulator = 0; return source.Select(x => accumulator += x); }); } }
  24. 35.

    Encapsulating processing side-effects in a Sink public class FileSink :

    Sink<int> { public string FileName { get; set; } public override IObservable<int> Process(IObservable<int> source) { return Observable.Using( () => new StreamWriter(FileName), stream => source.Do(x => stream.WriteLine(x))); } }
  25. 36.

    Workshop Project Assignment Code your own Source, Transform or Sink

    for Bonsai • Web Sockets • DAQ Boards • Cameras • Joysticks • Tracking • Spike Sorting • Classifiers • File formats (NEV,HDF5) • Visualizations