Brain-Computer Interfaces - Science Fiction or Reality?

Brain-Computer Interface (BCI) is a direct communication pathway between the brain and an external device. It seems to be new technology, but it has its origins in 1924! Today, we have technologies, which lie on the border between biology and computer science like neuroprosthetics, electromyography and of course BCI. During the talk, we will start the journey through different brain imaging techniques, but in the end, we will focus on electroencephalography (EEG). BCI is no longer expensive technology available only in laboratories. Anyone can have an access to it. Such technology gives us the possibility to create human-computer interaction of the future. Additionally, we can develop communication interfaces for people with illnesses like LIS and it may be the only hope for them. During this talk, we will discuss not only a theory but also practical examples, available software and working pieces of code for the Android platform. If you care about transhumanism, you cannot neglect BCI.

This presentation was shown during the Øredev 2018 conference in Malmö, Sweden on 19th of November 2018.

Transcript

1. Brain-Computer Interfaces
   Science Fiction or Reality?
   Piotr Wittchen
   wittchen.io
   

   View Slide

2. Evolution of communication tools
   2
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. Leap motion
   8
   

   View Slide

9. What’s next?
   9
   

   View Slide

10. View Slide

11. A Brain-Computer Interface (BCI) is a direct
    communication pathway between the brain and
    an external device.
    11
    

    View Slide

12. View Slide

13. In 1924 Hans Berger was the first
    to record human brain activity. He
    was able to identify oscillatory
    activity in the brain, such as the
    alpha wave (8–12 Hz).
    13
    

    View Slide

14. What can we do with BCI?
    14
    

    View Slide

15. Help people with Locked-In Syndrome (LIS)
    15
    

    View Slide

16. Create tools for rehabilitation
    16
    

    View Slide

17. Train meditation
    17
    

    View Slide

18. 18
    Gathering secure information
    

    View Slide

19. Develop new ways of communication
    19
    

    View Slide

20. View Slide

21. View Slide

22. MindFlix
    https://www.youtube.com/watch?v=cyMqFEJSI_U
    22
    

    View Slide

23. Art and Music
    Eunoia
    https://vimeo.com/65175792
    23
    

    View Slide

24. Does anyone care about BCI today?
    24
    

    View Slide

25. People talk about BCI at the conferences
    25
    

    View Slide

26. Facebook is doing research regarding BCI
    26
    

    View Slide

27. View Slide

28. View Slide

29. Similar technologies
    29
    

    View Slide

30. BCI vs. Neuroprosthetics
    Neuroprosthetics connect
    the nervous system to a
    device, whereas BCIs
    connect the brain with a
    computer system
    30
    

    View Slide

31. Electromyography (EMG)
    Technique for evaluating and recording
    the electrical activity produced by
    skeletal muscles.
    31
    

    View Slide

32. Brain imaging techniques
    32
    

    View Slide

33. Magnetoencephalography
    MEG is the technique for mapping brain activity by
    recording magnetic fields produced by electrical
    currents occurring naturally in the brain, using very
    sensitive magnetometers.
    33
    

    View Slide

34. Positron Emission Tomography
    PET is the nuclear medical imaging
    technique that produces a 3D image
    or picture of functional processes in
    the body.
    34
    

    View Slide

35. Single-Photon Emission Computed Tomography
    SPECT is the nuclear medicine tomographic
    imaging technique using gamma rays.
    35
    

    View Slide

36. Functional Magnetic Resonance Imaging
    FMRI is the Magnetic Resonance Imaging procedure that
    measures brain activity by detecting associated changes in
    blood flow.
    36
    

    View Slide

37. Functional Near Infrared
    Using FNIR, brain activity is measured through hemodynamic
    responses (blood movement) associated with neuron
    behavior. Mentioned responses can be detected with Infrared
    (IR) radiation.
    37
    

    View Slide

38. Electroencephalography
    EEG is recording of the electrical activity
    of the brain along the scalp
    38
    

    View Slide

39. How does EEG work?
    After placing electrodes on the skull, we can register changes
    of electric potential on the surface of the skin. These changes
    are caused by activity of neurons of the cortex.
    39
    

    View Slide

40. View Slide

41. 41
    Source: Nunez P., Srinivasan R.: Electroencephalogram, Scholarpedia, 2007
    

    View Slide

42. 42
    EEG Voltage recorded
    from the electrode
    electrode
    amplitude
    frequency
    phase
    Frequency component
    time
    Source: Nunez P., Srinivasan R.: Electroencephalogram, Scholarpedia, 2007
    

    View Slide

43. EEG rhythmic activity frequency bands
    ● Delta (up to 4 Hz) - in babies or slow-wave sleep by adults
    ● Theta (4-8 Hz) - young children or drowsiness by older children and adults,
    idling
    ● Alpha (8-13 Hz) - relaxed/reflecting, closing eyes, pathologically: coma
    ● Beta (>13-30 Hz) - active, busy, concentration or anxious thinking,
    pathologically: Benzodiazepines
    ● Gamma (30-100+ Hz) - perception that combines different senses (e.g. sound
    and sight), short-term memory matching of recognized objects
    ● Mu (8-13 Hz) - shows rest-state motor neurons
    43
    

    View Slide

44. Selected affordable EEG hardware available today
    Muse Emotiv NeuroSky
    44
    

    View Slide

45. NeuroSky MindWave Mobile
    ● Bluetooth v. 3.0 class 2 (10 meters range)
    ● Wireless pairing
    ● Static headset ID for pairing purposes
    ● 8-hours battery runtime (uses one AAA battery -
    1.5 V)
    ● iOS and Android support
    ● Measures raw brainwaves
    ● Processing and output of EEG power spectrum
    (Alpha, Beta, etc.)
    ● Processing and output proprietary eSense
    meter for mediation, attention and other future
    meters
    ● Blink detection
    ● Signal quality analysis
    45
    

    View Slide

46. Applications
    46
    

    View Slide

47. 47
    EEG Analyzer
    Google Play: https://play.google.com/store/apps/details?id=com.pwittchen.eeganalyzer
    

    View Slide

48. EEG Controller
    48
    

    View Slide

49. BrainWave Visualiser
    https://play.google.com/store/apps/details?id=com.neurosky.unitythinkgear
    49
    

    View Slide

50. Brain Bits
    50
    https://github.com/dashersw/brain-bits
    

    View Slide

51. Muse
    51
    https://play.google.com/store/apps/details?id=com.interaxon.muse
    

    View Slide

52. How to write software for the NeuroSky?
    52
    

    View Slide

53. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    53
    

    View Slide

54. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    54
    

    View Slide

55. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    55
    

    View Slide

56. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    56
    

    View Slide

57. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    57
    

    View Slide

58. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    58
    

    View Slide

59. Gathering data from NeuroSky with ThinkGear
    TGDevice device = new TGDevice(BluetoothAdapter.getDefaultAdapter(), new Handler() {
    @Override public void handleMessage(Message msg) {
    super.handleMessage(msg);
    if(msg == TGDevice.MSG_STATE_CHANGE) {
    if(msg.arg1 == TGDevice.STATE_CONNECTED) {
    device.start();
    }
    // handle rest of the messages here...
    }
    }
    });
    }
    device.connect();
    59
    

    View Slide

60. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    60
    

    View Slide

61. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    61
    

    View Slide

62. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    62
    

    View Slide

63. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    63
    

    View Slide

64. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    64
    

    View Slide

65. Using NeuroSky Android SDK
    NeuroSky neuroSky = new NeuroSky(new ExtendedDeviceMessageListener() {
    @Override public void onStateChange(State state) {
    // handle state change...
    }
    @Override public void onSignalChange(Signal signal) {
    // handle signal change...
    }
    @Override public void onBrainWavesChange(Set brainWaves) {
    // handle brain waves change...
    }
    });
    neuroSky.connect();
    65
    

    View Slide

66. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    66
    

    View Slide

67. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    67
    

    View Slide

68. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    68
    

    View Slide

69. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    69
    

    View Slide

70. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    70
    

    View Slide

71. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    71
    

    View Slide

72. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    72
    

    View Slide

73. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    73
    

    View Slide

74. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    74
    

    View Slide

75. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    75
    

    View Slide

76. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    76
    

    View Slide

77. Using NeuroSky Android SDK with RxJava
    RxNeuroSky neuroSky = new RxNeuroSky();
    neuroSky
    .stream()
    .subscribeOn(Schedulers.computation())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(brainEvent -> {
    handleStateChange(brainEvent.state());
    handleSignalChange(brainEvent.signal());
    handleBrainWavesChange(brainEvent.brainWaves());
    });
    neuroSky
    .connect()
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(
    () -> showMessage("connecting..."),
    throwable -> { /* handle an error... */ }
    );
    77
    

    View Slide

78. View Slide

79. Let’s see the live demo!
    What can go wrong?
    79
    

    View Slide

80. Source code, examples and links
    https://github.com/pwittchen/neurosky-android-sdk (brand new project)
    https://github.com/pwittchen/EEGReader (old project using ThinkGear only)
    http://developer.neurosky.com/ (official website of the NeuroSky)
    http://developer.choosemuse.com/sdk (official Muse developer website)
    http://wittchen.io/tags/bci/ (a few articles about BCI on my blog)
    80
    

    View Slide

81. Summary
    ● Direct communication between human brain and the computer is no longer
    science-fiction
    ● We have affordable devices using EEG technology today
    ● We are able to write Brain-Computer Interfaces without having scientific
    knowledge about it
    ● We can develop new ways of communication with BCI
    ● We can create data acquisition tools for doctors and scientists
    ● We can develop tools supporting meditation
    ● We can develop tools for rehabilitation and improving concentration
    ● We can create communication tools for people with illnesses like LIS
    ● BCI can be improved with Machine Learning techniques
    81
    

    View Slide

82. Brain-Computer Interfaces
    Science Fiction or Reality?
    Piotr Wittchen
    wittchen.io
    get presentation slides at:
    wittchen.io/talks
    Thank you for attention! Questions?
    

    View Slide