Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at:
https://www.youtube.com/watch?v=I8qwJXVghQ0

Transcript

1. Advanced Bonsai Workshop https://bitbucket.org/horizongir/bonsai January 18-21, 2016 Bonsai = Rx + VPL + Packages
2. Be Reactive Ask Questions
3. Workshop Programme 1. From databases to data streams 2. Bonsai toolboxes and visual editor 3. Observable combinators: An algebra for data streams 4. Representing discrete states with data streams
4. From Databases to Data Streams January 18, 2016 Advanced Bonsai Workshop
5. There are (almost) as many data structures as there are algorithms Array 1D Array 2D Array 3D Linked List Tree Graph
6. Many bulk operations over data are independent of the details of the data structure Ex:
7. Many bulk operations over data are independent of the details of the data structure Ex:
8. Many bulk operations over data are independent of the details of the data structure Ex:
9. Enumerable streams interface IEnumerable<T> { IEnumerator<T> GetEnumerator(); } interface IEnumerator<T> : IDisposable { bool MoveNext(); T Current { get; } }
10. Programming with streams // sum of all numbers var sum = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
11. Where
12. Select
13. Sum
14. Programming with streams // sum of all numbers var sum = numbers.Sum(); // sum of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Sum(); // sum of the square of all numbers greater than 10 var sum = numbers.Where(x => x > 10) .Select(x => x * x) .Sum();
15. A data table is a stream of rows Name Age Height Weight João 30 1.60 60 Maria 24 1.70 54 Pedro 27 1.90 70 Ana 44 1.55 62
16. Streams don’t even need a data structure class RandomStream : IEnumerable<int> { public IEnumerator<int> GetEnumerator() { return new RandomIterator(); } class RandomIterator : IEnumerator<int> { int current; Random random = new Random(); public int Current { get { return current; } } public bool MoveNext() { current = random.Next(); return true; } public void Dispose() { } } }
17. Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int> Generator() { yield return 0; yield return 1; yield return 2; }
18. Iterator blocks provide convenient syntax for creating enumerable streams IEnumerable<int> RandomNumbers() { var random = new Random(); while (true) { yield return random.Next(); } }
19. Not every stream is waiting to be picked
20. The real world doesn’t wait for you! Hardware Interrupts User-Interface Events Button click Monitoring Systems
21. Enumerable streams interface IEnumerable<T> { IEnumerator<T> GetEnumerator(); } interface IEnumerator<T> : IDisposable { bool MoveNext(); T Current { get; } }
22. Observable streams interface IObservable<T> { IDisposable Subscribe(IObserver<T> observer); } interface IObserver<T> { void OnCompleted(); void OnNext(T value); void OnError(Exception error); }
23. Pull streams vs Push streams
24. Programming with observable streams // sum of all measurements var sum = measurements.Sum(); // sum of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Sum(); // sum of the square of all measurements greater than 10 var sum = measurements.Where(x => x > 10) .Select(x => x * x) .Sum();
25. Where
26. Select
27. Sum
28. Single value sequence public class ValueSource : Source<int> { public int Value { get; set; } public override IObservable<int> Generate() { return Observable.Return(Value); } }
29. Single value asynchronous sequence public class AsyncValue : Source<int> { public int Value { get; set; } public override IObservable<int> Generate() { return Observable.Start(() => { Thread.Sleep(1000); return Value; }); } }
30. Multi-value sequence public class EnumerableSource : Source<int> { public override IObservable<int> Generate() { return new[] { 0, 1, 2, 3, 4 }.ToObservable(); } }
31. Multi-value asynchronous sequence (generator) public class AsyncEnumerable : Source<int> { IEnumerable<int> Generator() { for (int i = 0; i < 5; i++) { Thread.Sleep(1000); yield return i; } } public override IObservable<int> Generate() { return Generator().ToObservable(Scheduler.Default); } }
32. Polling a device public class CameraSource : Source<IplImage> { IEnumerable<IplImage> FrameCapture() { using (var camera = Capture.CreateCameraCapture(0)) { while (true) { var image = camera.QueryFrame(); yield return image; } } } public override IObservable<IplImage> Generate() { return FrameCapture().ToObservable(Scheduler.Default); } }
33. A simple transform operator public class ScaleTransform : Transform<int, int> { public int Value { get; set; } public override IObservable<int> Process(IObservable<int> source) { return source.Select(x => x * Value); } }
34. Transform with mutable state (memory) public class AccumulatorTransform : Transform<int, int> { public override IObservable<int> Process(IObservable<int> source) { return Observable.Defer(() => { var accumulator = 0; return source.Select(x => accumulator += x); }); } }
35. Encapsulating processing side-effects in a Sink public class FileSink : Sink<int> { public string FileName { get; set; } public override IObservable<int> Process(IObservable<int> source) { return Observable.Using( () => new StreamWriter(FileName), stream => source.Do(x => stream.WriteLine(x))); } }
36. Workshop Project Assignment Code your own Source, Transform or Sink for Bonsai • Web Sockets • DAQ Boards • Cameras • Joysticks • Tracking • Spike Sorting • Classifiers • File formats (NEV,HDF5) • Visualizations