KotlinConf 2019 | Bridge the physical world: Kotlin/Native on Raspberry Pi

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
December 05, 2019

KotlinConf 2019 | Bridge the physical world: Kotlin/Native on Raspberry Pi

With Kotlin/Native, we can now compile Kotlin code to run on various platforms, including Raspberry Pi. This cross-platform ability has drawn the attention of many developers. Through the process of building a hand game robot which can play rock paper scissors with human beings, this talk aims to show you the possibility of using Kotlin to control GPIO pins on a Raspberry Pi and other experiments such as performing machine learning operations in Kotlin using TensorFlow backend, thanks to the interoperability with C libraries.

Video: https://youtu.be/KfdNiMP0emE
Github: https://github.com/jinqian/kotlin-native-chifoumi

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

December 05, 2019
Tweet

Transcript

  1. 3.

    •Context •Rock-Paper-Scissor Hand Robot •Kotlin/Native as an alternative •Project setup

    •Step by step •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip 1 - Move the project into IntelliJ •Tip 2 - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?!) •Demo - Kotlin/Native on Raspberry Pi in actions! 3 Agenda
  2. 5.

    5

  3. 6.
  4. 7.
  5. 8.
  6. 9.

    • Data collection tool: raw data for model training •

    Hardware components: robot arms + game control • Application running on Android Things • Game logic and interactions • Image capture with camera preview • Gesture recognition with TensorFlow 9 Break Down the Project
  7. 10.
  8. 11.
  9. 13.

    • Android Things Starter Kit - NXP i.MX7D (~200€) •

    16-Channel PWM/Servo Driver (~12€) • Power Supply or battery (~5€) • 5 servomotors (~12€) • Cardboard + wood sticks (~0€) • Glue Gun (~10€) • Electronic jumper + Resistors + LED + Button + Breadboard (~10€) 13 Hardware (Android Things Version)
  10. 16.
  11. 17.
  12. 22.

    Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the

    Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.
  13. 23.

    Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source

    Code C C++ Java Kotlin … Machine Code ARM Sparc X86 PowerPC … Compiler Design
  14. 24.

    Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend

    LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design
  15. 27.

    Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32,

    iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64, mingwX86 • WebAssembly: wasm32 • watchOS, tvOS… ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets 27
  16. 28.

    Architecture •ARM (*Originally: Acorn RISC Machine) •ARM (Advanced RISC Machines)

    •x86 (Successors to Intel’s 8086 processor) •RISC (Reduced Instruction Set Computing) •CISC (Complex Instruction Set Computing) Source: https://github.com/raspberrypi/firmware 28
  17. 29.

    Raspberry Pi 3 Model B+ Spec • BCM2837 1.2 GHz

    64-bit quad processor based on the ARMv8 Cortex-A53 • 32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode” 29
  18. 30.
  19. 33.

    Key Components •konanc: Kotlin/Native compiler •cinterop: produce Kotlin binding for

    native libraries Source: https://kotlinlang.org/docs/reference/native/c_interop.html 33
  20. 34.

    C Interoperability • Create a .def file describing what to

    include into bindings • Use the cinterop tool to produce Kotlin bindings • Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 34 .def .h cinterop .klib
  21. 35.

    • Pointers and arrays => CPointer<T>? • Enums => Kotlin

    enum or integral values • Structs / unions => types having fields available via the dot notation (i.e. kotlinConfInstance.speakers) • typedef => typealias Source: https://kotlinlang.org/docs/reference/native/c_interop.html 35 Basic Interop Types C Type Kotlin Type Signed, unsigned integral and floating point Kotlin counterpart with the same width Pointers and arrays CPointer<T>? Enums Kotlin enum or integral values Structs / Unions Types having fields available via the dot notation (i.e. kotlinConfInstance.speakers) typedef typealias
  22. 36.
  23. 38.

    • Android Things Starter Kit - NXP i.MX7D (~200€) •

    Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display • … 38 Hardware (Raspberry Pi Version)
  24. 39.
  25. 41.

    February 2019: Me trying to cross compile Kotlin/Native application on

    my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.
  26. 42.

    exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64

    host at org.jetbrains.kotlin.backend.konan.KonanConfig.<init>(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …
  27. 46.

    Compile Manually $ konanc hello.kt -o hello -target raspberrypi $

    scp hello.kexe pi@{IP-RASPBERRY-PI}:~/dir/ hello.kexe
  28. 47.
  29. 50.

    pigpio pigpio: a library for the Raspberry which allows control

    of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio 50
  30. 51.

    Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO

    access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/ 51
  31. 52.

    • Create a pigpio.def file • Use the cinterop tool

    to produce Kotlin bindings • ⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation • Run Kotlin/Native compiler on an application to produce the final executable Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ 52 Make it work! Thank you Hadi Hariri!!
  32. 54.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi
  33. 55.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi -def pigpio -> the .def file we just created
  34. 56.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi -compiler-option -I`pwd`/include —> indicate the path of the header file
  35. 57.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi -o pigpio -> the output name for the Kotlin binding
  36. 58.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi -target raspberrypi -> our target platform
  37. 59.

    Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option

    -I`pwd`/include \ -o pigpio \ -target raspberrypi pigpio.klib pigpio-build/
  38. 60.

    pigpio: Basic API in C int gpioInitialise(void); int gpioSetMode(unsigned gpio,

    unsigned mode); int gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros); 60
  39. 61.

    Generated Kotlin Binding @kotlinx.cinterop.internal.CCall fun gpioInitialise(): Int @kotlinx.cinterop.internal.CCall fun gpioSetMode(gpio:

    UInt, mode: UInt): Int @kotlinx.cinterop.internal.CCall fun gpioWrite(gpio: UInt, level: UInt): Int @kotlinx.cinterop.internal.CCall fun gpioSleep(timetype: UInt, seconds: Int, micros: Int): Int
  40. 62.

    Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port,

    PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }
  41. 63.

    $ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $

    sudo make install Install pigpio Retrieve libpigpio.so for cross compilation
  42. 64.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi
  43. 65.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -library pigpio -> indicate the Kotlin binding to use
  44. 66.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -linker-options -> link the library libpigpio.so
  45. 67.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -o led -> indicate the output executable name
  46. 68.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -target raspberrypi -> still our target!
  47. 69.

    Generate executable $ konanc led.kt \ -library pigpio \ -linker-options

    "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi led.kexe
  48. 71.

    • kotlin-multiplatform plugin • Still experimental, DSL keeps changing •

    Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 71 Move into IntelliJ
  49. 72.

    build.gradle Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html plugins { id 'kotlin-multiplatform' version '1.3.60' }

    kotlin { linuxArm32Hfp("chifoumi") { binaries { executable { entryPoint “chifoumi.robot.main" // libpigpio.so are compiled on raspberry pi and copied here linkerOpts("-Lsrc/include/pigpio", "-lpigpio") } } compilations.main.cinterops { pigpio { includeDirs ‘src/include/pigpio' } } } }
  50. 74.

    • Cross compilation repetitive scp • Always automate! ✨ •

    Step 1: passwordless SSH with your Raspberry Pi • Step 2: write a gradle task which does the job • Step 3: Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md 74 Deploy on Pi with Gradle
  51. 75.

    task deployOnPi Source: https://gradle-ssh-plugin.github.io/ remotes { pi { host =

    ‘IP address ip of your pi' user = 'pi user' identity = file(‘/path/to/.ssh/id_rsa’) } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/app.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }
  52. 78.

    •The executable will launch a continuous loop… •How do we

    interact with the application? •With a physical input! •So how can we receive the callback?
  53. 79.

    pigpio: Set Alert Function // Register a function to be

    called (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick); 79
  54. 81.

    Generated Kotlin Binding Source: https://kotlinlang.org/docs/reference/type-aliases.html typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int,

    platform.posix.uint32_t) -> Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );
  55. 82.

    • How to convert a Kotlin function to a pointer

    to a C function? • staticCFunction(::kotlinFunction) to the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html 82 Mapping Function Pointer from C
  56. 83.

    Button Callback val onButtonPressed = staticCFunction<Int, Int, UInt, Unit> {

    gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }
  57. 85.

    I²C: I-squared-C • I2C bus connects simple peripheral devices with

    small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c 85
  58. 87.
  59. 88.

    PWM: Pulse Width Modulation • Pulse Width Modulation (PWM) is

    a common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) • Frequency: expressed in Hz • Period: the time each cycle takes and is the inverse of frequency • Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm 88
  60. 90.

    • PCA9685 16-Channel 12 Bit PWM Servo Driver • Adafruit

    C++ lib: not interoperable with Kotlin Native • Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3 90 In search of the C binding
  61. 91.

    $ git clone https://github.com/edlins/libPCA9685 $ cd libPCA9685 && mkdir build

    && cd build $ cmake .. $ make $ sudo make install Install libPCA9685 Retrieve libPCA9685.so for cross compilation
  62. 92.

    pigpio: Initialize PWM // Open the I2C bus device and

    assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq); 92
  63. 93.

    PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C(

    adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )
  64. 94.

    pigpio: Set PWM Values // set all PWM channels from

    two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals); // set a single PWM channel with a 16-bit ON val and a 16-bit OFF val int PCA9685_setPWMVal(int fd, unsigned char addr, unsigned char reg, unsigned int on, unsigned int off); // set all PWM channels with one 16-bit ON val and one 16-bit OFF val int PCA9685_setAllPWM(int fd, unsigned char addr, unsigned int on, unsigned int off); 94
  65. 95.

    pigpio: Set PWM Values // set all PWM channels from

    two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals); // set a single PWM channel with a 16-bit ON val and a 16-bit OFF val int PCA9685_setPWMVal(int fd, unsigned char addr, unsigned char reg, unsigned int on, unsigned int off); // set all PWM channels with one 16-bit ON val and one 16-bit OFF val int PCA9685_setAllPWM(int fd, unsigned char addr, unsigned int on, unsigned int off); 95
  66. 96.

    Generated Kotlin Binding CValuesRef<UIntVar>? public fun PCA9685_setPWMVals( fd: Int, addr:

    UByte, onVals: CValuesRef<UIntVar>?, offVals: CValuesRef<UIntVar>? ): Int
  67. 98.

    • Get a NativePlacement instance from the memScoped {…} block

    receiver • Create an array of UIntVar instance with the allocArray<T> extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/ 98 Explicit Native Memory Allocation
  68. 99.

    Set PWM On/Off values (1/2) memScoped { val onValues =

    allocArray<UIntVar>(_PCA9685_CHANS) val offValues = allocArray<UIntVar>(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }
  69. 100.

    • The CValuesRef<T> is designed to support C array literals

    without explicit native memory allocation. • Construct the immutable self-contained sequence of C values: • ${type}Array.toCValues(), where type is the Kotlin primitive type • cValuesOf(vararg elements: ${type}), where type is a primitive or pointer • Array<CPointer<T>?>.toCValues(), List<CPointer<T>?>.toCValues() Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 100 Oh Wait, Simpler Solution!
  70. 101.

    Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for

    (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )
  71. 103.

    Understand raspicam • 4 Applications are provided: raspistill, raspivid, raspiyuv

    and raspividyuv • Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* • No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/ 103
  72. 104.

    • POSIX bindings to the rescue • Execute command line

    from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html 104 Reinvent the wheel? Nope.
  73. 107.

    • TensorFlow for C is supported on: • Linux, 64-bit,

    x86 • MacOS X, Version 10.12.6 (Sierra) or higher • Windows, 64-bit x86 • No official C library for Raspberry Pi Source: https://www.tensorflow.org/install/lang_c 107 TensorFlow for C
  74. 109.
  75. 110.

    No documentation ¯\_(ツ)_/¯ Unofficial binaries https://github.com/PINTO0309/Tensorflow-bin *“The behavior is unconfirmed

    because I do not have C language implementation skills.” Source: https://github.com/PINTO0309/Tensorflow-bin 110 TensorFlow for C
  76. 112.
  77. 113.
  78. 114.
  79. 116.

    • ⚠ Experimental ⚠ • Tip of the iceberg •

    Debugging is hard • Fine tuning on the sampling rate is needed • C and cross compilation knowledges are essential • It was fun exploring! 116 What I learned so far
  80. 117.

    • Software is getting incredibly complex and interfacing between languages

    is increasingly important • Gain low-level control without giving up high-level conveniences • The Openness of Kotlin/Native (for Web Assembly even) 117 Why?
  81. 118.

    • Both using LLVM as a backend • Different memory

    management systems • Kotlin/Native: automated, reference counter with a cycle collector • Rust: manual, with smart pointers • Maturity level 118 Kotlin/Native v.s. Rust?
  82. 119.

    • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native Snake

    by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native •Int ptr in Kotlin/Native 119 References