710

# Advanced Bonsai Workshop #1: From databases to data streams

The first part of a workshop series on the Bonsai programming language. These lecture slides include a general introduction to data stream and reactive programming using Rx, as well as a tutorial on how to create your own Bonsai nodes.

A video presentation of these slides is available at: January 18, 2016

## Transcript

https://bitbucket.org/horizongir/bonsai
January 18-21, 2016
Bonsai = Rx + VPL + Packages

2. Be Reactive

3. Workshop Programme
1. From databases to data streams
2. Bonsai toolboxes and visual editor
3. Observable combinators: An algebra for data streams
4. Representing discrete states with data streams

4. From Databases to
Data Streams
January 18, 2016

5. There are (almost) as many data structures as
there are algorithms
Array 1D
Array 2D
Array 3D
Tree
Graph

6. Many bulk operations over data are independent
of the details of the data structure
Ex:

7. Many bulk operations over data are independent
of the details of the data structure
Ex:

8. Many bulk operations over data are independent
of the details of the data structure
Ex:

9. Enumerable streams
interface IEnumerable
{
IEnumerator GetEnumerator();
}
interface IEnumerator : IDisposable
{
bool MoveNext();
T Current { get; }
}

10. Programming with streams
// sum of all numbers
var sum = numbers.Sum();
// sum of all numbers greater than 10
var sum = numbers.Where(x => x > 10)
.Sum();
// sum of the square of all numbers greater than 10
var sum = numbers.Where(x => x > 10)
.Select(x => x * x)
.Sum();

11. Where

12. Select

13. Sum

14. Programming with streams
// sum of all numbers
var sum = numbers.Sum();
// sum of all numbers greater than 10
var sum = numbers.Where(x => x > 10)
.Sum();
// sum of the square of all numbers greater than 10
var sum = numbers.Where(x => x > 10)
.Select(x => x * x)
.Sum();

15. A data table is a stream of rows
Name Age Height Weight
João 30 1.60 60
Maria 24 1.70 54
Pedro 27 1.90 70
Ana 44 1.55 62

16. Streams don’t even need a data structure
class RandomStream : IEnumerable
{
public IEnumerator GetEnumerator() { return new RandomIterator(); }
class RandomIterator : IEnumerator
{
int current;
Random random = new Random();
public int Current { get { return current; } }
public bool MoveNext()
{
current = random.Next();
return true;
}
public void Dispose() { }
}
}

17. Iterator blocks provide convenient syntax for
creating enumerable streams
IEnumerable Generator()
{
yield return 0;
yield return 1;
yield return 2;
}

18. Iterator blocks provide convenient syntax for
creating enumerable streams
IEnumerable RandomNumbers()
{
var random = new Random();
while (true)
{
yield return random.Next();
}
}

19. Not every stream is waiting to be picked

20. The real world doesn’t wait for you!
Hardware
Interrupts
User-Interface
Events
Button click
Monitoring
Systems

21. Enumerable streams
interface IEnumerable
{
IEnumerator GetEnumerator();
}
interface IEnumerator : IDisposable
{
bool MoveNext();
T Current { get; }
}

22. Observable streams
interface IObservable
{
IDisposable Subscribe(IObserver observer);
}
interface IObserver
{
void OnCompleted();
void OnNext(T value);
void OnError(Exception error);
}

23. Pull streams vs Push streams

24. Programming with observable streams
// sum of all measurements
var sum = measurements.Sum();
// sum of all measurements greater than 10
var sum = measurements.Where(x => x > 10)
.Sum();
// sum of the square of all measurements greater than 10
var sum = measurements.Where(x => x > 10)
.Select(x => x * x)
.Sum();

25. Where

26. Select

27. Sum

28. Single value sequence
public class ValueSource : Source
{
public int Value { get; set; }
public override IObservable Generate()
{
return Observable.Return(Value);
}
}

29. Single value asynchronous sequence
public class AsyncValue : Source
{
public int Value { get; set; }
public override IObservable Generate()
{
return Observable.Start(() =>
{
return Value;
});
}
}

30. Multi-value sequence
public class EnumerableSource : Source
{
public override IObservable Generate()
{
return new[] { 0, 1, 2, 3, 4 }.ToObservable();
}
}

31. Multi-value asynchronous sequence
(generator)
public class AsyncEnumerable : Source
{
IEnumerable Generator()
{
for (int i = 0; i < 5; i++)
{
yield return i;
}
}
public override IObservable Generate()
{
return Generator().ToObservable(Scheduler.Default);
}
}

32. Polling a device
public class CameraSource : Source
{
IEnumerable FrameCapture()
{
using (var camera = Capture.CreateCameraCapture(0))
{
while (true)
{
var image = camera.QueryFrame();
yield return image;
}
}
}
public override IObservable Generate()
{
return FrameCapture().ToObservable(Scheduler.Default);
}
}

33. A simple transform operator
public class ScaleTransform : Transform
{
public int Value { get; set; }
public override IObservable Process(IObservable source)
{
return source.Select(x => x * Value);
}
}

34. Transform with mutable state (memory)
public class AccumulatorTransform : Transform
{
public override IObservable Process(IObservable source)
{
return Observable.Defer(() =>
{
var accumulator = 0;
return source.Select(x => accumulator += x);
});
}
}

35. Encapsulating processing side-effects in a Sink
public class FileSink : Sink
{
public string FileName { get; set; }
public override IObservable Process(IObservable source)
{
return Observable.Using(
() => new StreamWriter(FileName),
stream => source.Do(x => stream.WriteLine(x)));
}
}

36. Workshop Project Assignment
Code your own Source, Transform or Sink for Bonsai
• Web Sockets
• DAQ Boards
• Cameras
• Joysticks
• Tracking
• Spike Sorting
• Classifiers
• File formats (NEV,HDF5)
• Visualizations