Advanced Bonsai Workshop #1: From databases to data streams

Cff520b66669064bbfefdc9109731160?s=47 glopesdev
January 18, 2016

Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at:
https://www.youtube.com/watch?v=I8qwJXVghQ0

Cff520b66669064bbfefdc9109731160?s=128

glopesdev

January 18, 2016
Tweet

Transcript

  1. Advanced Bonsai Workshop https://bitbucket.org/horizongir/bonsai January 18-21, 2016 Bonsai = Rx

    + VPL + Packages
  2. Be Reactive Ask Questions

  3. Workshop Programme 1. From databases to data streams 2. Bonsai

    toolboxes and visual editor 3. Observable combinators: An algebra for data streams 4. Representing discrete states with data streams
  4. From Databases to Data Streams January 18, 2016 Advanced Bonsai

    Workshop
  5. There are (almost) as many data structures as there are

    algorithms Array 1D Array 2D Array 3D Linked List Tree Graph
  6. Many bulk operations over data are independent of the details

    of the data structure Ex:
  7. Many bulk operations over data are independent of the details

    of the data structure Ex:
  8. Many bulk operations over data are independent of the details

    of the data structure Ex:
  9. Enumerable streams interface IEnumerable<T> { IEnumerator<T> GetEnumerator(); } interface IEnumerator<T>

    : IDisposable { bool MoveNext(); T Current { get; } }
  10. Programming with streams // sum of all numbers var sum

    = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
  11. Where

  12. Select

  13. Sum

  14. Programming with streams // sum of all numbers var sum

    = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
  15. A data table is a stream of rows Name Age

    Height Weight João 30 1.60 60 Maria 24 1.70 54 Pedro 27 1.90 70 Ana 44 1.55 62
  16. Streams don’t even need a data structure class RandomStream :

    IEnumerable<int> { public IEnumerator<int> GetEnumerator() { return new RandomIterator(); } class RandomIterator : IEnumerator<int> { int current; Random random = new Random(); public int Current { get { return current; } } public bool MoveNext() { current = random.Next(); return true; } public void Dispose() { } } }
  17. Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int>

    Generator() { yield return 0; yield return 1; yield return 2; }
  18. Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int>

    RandomNumbers() { var random = new Random(); while (true) { yield return random.Next(); } }
  19. Not every stream is waiting to be picked

  20. The real world doesn’t wait for you! Hardware Interrupts User-Interface

    Events Button click Monitoring Systems
  21. Enumerable streams interface IEnumerable<T> { IEnumerator<T> GetEnumerator(); } interface IEnumerator<T>

    : IDisposable { bool MoveNext(); T Current { get; } }
  22. Observable streams interface IObservable<T> { IDisposable Subscribe(IObserver<T> observer); } interface

    IObserver<T> { void OnCompleted(); void OnNext(T value); void OnError(Exception error); }
  23. Pull streams vs Push streams

  24. Programming with observable streams // sum of all measurements var

    sum = measurements.Sum(); // sum of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Sum(); // sum of the square of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Select(x => x * x) .Sum();
  25. Where

  26. Select

  27. Sum

  28. Single value sequence public class ValueSource : Source<int> { public

    int Value { get; set; } public override IObservable<int> Generate() { return Observable.Return(Value); } }
  29. Single value asynchronous sequence public class AsyncValue : Source<int> {

    public int Value { get; set; } public override IObservable<int> Generate() { return Observable.Start(() => { Thread.Sleep(1000); return Value; }); } }
  30. Multi-value sequence public class EnumerableSource : Source<int> { public override

    IObservable<int> Generate() { return new[] { 0, 1, 2, 3, 4 }.ToObservable(); } }
  31. Multi-value asynchronous sequence (generator) public class AsyncEnumerable : Source<int> {

    IEnumerable<int> Generator() { for (int i = 0; i < 5; i++) { Thread.Sleep(1000); yield return i; } } public override IObservable<int> Generate() { return Generator().ToObservable(Scheduler.Default); } }
  32. Polling a device public class CameraSource : Source<IplImage> { IEnumerable<IplImage>

    FrameCapture() { using (var camera = Capture.CreateCameraCapture(0)) { while (true) { var image = camera.QueryFrame(); yield return image; } } } public override IObservable<IplImage> Generate() { return FrameCapture().ToObservable(Scheduler.Default); } }
  33. A simple transform operator public class ScaleTransform : Transform<int, int>

    { public int Value { get; set; } public override IObservable<int> Process(IObservable<int> source) { return source.Select(x => x * Value); } }
  34. Transform with mutable state (memory) public class AccumulatorTransform : Transform<int,

    int> { public override IObservable<int> Process(IObservable<int> source) { return Observable.Defer(() => { var accumulator = 0; return source.Select(x => accumulator += x); }); } }
  35. Encapsulating processing side-effects in a Sink public class FileSink :

    Sink<int> { public string FileName { get; set; } public override IObservable<int> Process(IObservable<int> source) { return Observable.Using( () => new StreamWriter(FileName), stream => source.Do(x => stream.WriteLine(x))); } }
  36. Workshop Project Assignment Code your own Source, Transform or Sink

    for Bonsai • Web Sockets • DAQ Boards • Cameras • Joysticks • Tracking • Spike Sorting • Classifiers • File formats (NEV,HDF5) • Visualizations