Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bridge The Physical World: Kotlin Native on Raspberry Pi

June 07, 2019

Bridge The Physical World: Kotlin Native on Raspberry Pi

With Kotlin/Native, we can now compile Kotlin code to run on various platforms, including Raspberry Pi. This cross-platform ability has drawn the attention of many developers. Through the process of building a hand game robot which can play rock paper scissors with human beings, this talk aims to show you the possibility of using Kotlin to control GPIO pins on a Raspberry Pi and other experiments such as performing machine learning operations in Kotlin using TensorFlow backend, thanks to the interoperability with C libraries.

Video: https://www.youtube.com/watch?v=73Ng_zsOsM8
Github: https://github.com/jinqian/kotlin-native-chifumi

#Kotlin #KotlinNative #RaspberryPi


June 07, 2019

More Decks by jinqian

Other Decks in Technology


  1. Bridge The Physical World: Kotlin/Native on Raspberry Pi Qian Jin

    | @bonbonking Conference for Kotliners 2019 Photo Credit: Harrison Broadbent on Unsplash
  2. Agenda •Rock-Paper-Scissor Hand Robot (with Android Things) •Kotlin/Native as an

    alternative •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip - Move the project into IntelliJ •Tip - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?) •Demo - Kotlin/Native on Raspberry Pi in actions!
  3. Break Down the Project •Data collection tool: raw data for

    model training •Hardware components: robot arms + game control •Application running on Android Things •Game logic and interactions •Image capture with camera preview •Gesture recognition with TensorFlow
  4. Data Collection Tool •Web mobile app built with Node.js +

    Express.js •Deployed with Firebase Hosting / Function / Storage
  5. Hardware (Android Things Version) •Android Things Starter Kit - NXP

    i.MX7D (~200€) •16-Channel PWM/Servo Driver (~12€) •Power Supply or battery (~5€) •5 servomotors (~12€) •Cardboard + wood sticks (~0€) •Glue Gun (~10€) •Electronic jumper + Resistors + LED + Button + Breadboard (~10€)
  6. Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the

    Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.
  7. Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source

    Code C C++ Java Kotlin … Machine Code ARM Sparc X86
 PowerPC … Compiler Design
  8. Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend

    LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design
  9. Terminology ARM (Acorn RISC Machine) ARM (Advanced RISC Machines) RISC

    (Reduced Instruction Set Computing) CISC (Complex Instruction Set Computing) MIPS (Millions of Instructions Per Second) x86 ABI (Abstraction Binary Interface) FPU (Floating-point Unit) Source: https://github.com/raspberrypi/firmware
  10. Raspberry Pi 3 Model B+ Spec •BCM2837 1.2 GHz 64-bit

    quad processor based on the ARMv8 Cortex-A53 •32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode”
  11. Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32,

    iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64 • WebAssembly: wasm32 •ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets
  12. Key Components •konanc: Kotlin/Native compiler •cinterop: produce kotlin binding for

    native library Source: https://kotlinlang.org/docs/reference/native/c_interop.html
  13. C Interoperability •Create a .def file describing what to include

    into bindings •Use the cinterop tool to produce Kotlin bindings •Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md .def .h cinterop .klib
  14. Hardware (Raspberry Pi Version) •Android Things Starter Kit - NXP

    i.MX7D (~200€) •Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display •…
  15. February 2019: Me trying to cross compile Kotlin/Native application on

    my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.
  16. exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64

    host at org.jetbrains.kotlin.backend.konan.KonanConfig.<init>(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …
  17. fun main() { println("Hello Kotlin/Native from Qian’s Macbook!") } Source:

    https://hadihariri.com/2017/04/09/kotlin-on-raspberry-pi/ hello.kt
  18. $ konanc hello.kt -target raspberrypi -o hello $ scp hello.kexe

    pi@{IP-RASPBERRY-PI}:~/dir/ Compile with command line
  19. pigpio pigpio: a library for the Raspberry which allows control

    of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio
  20. Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO

    access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/
  21. Make it work! •Create a pigpio.def file •Generate Kotlin stub

    for the library •⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation •Compile the code by linking the library using linkerOpts option Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ / Thank you Hadi Hariri!!
  22. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -copt (-compilerOpts) -> additional compiler options -I`pwd`/pigpio -> the location of the header file
  23. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -o pigpio.bc -> the output file for kotlin stub
  24. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -target -> the target platform is raspberry pi
  25. pigpio: Basics int gpioInitialise(void); int gpioSetMode(unsigned gpio, unsigned mode); int

    gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros);
  26. Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port,

    PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }
  27. $ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $

    sudo make install Build the library on Pi libpigpio.so
  28. Generate the binary led.kexe $ konanc led.kt -target raspberrypi -library

    pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led
  29. Move into IntelliJ •Articles are generally out of date •No

    much documentation at this moment •Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
  30. kotlin { targets { fromPreset(kotlin.presets.linuxArm32Hfp, 'chifumi') { compilations.main.outputKinds 'EXECUTABLE' compilations.main.entryPoint

    'chifumi.robot.main' compilations.main.cinterops { pigpio { includeDirs 'src/include/pigpio' } } // libpigpio.so are compiled on raspberry pi and copied here compilations.main.linkerOpts '-lpigpio -Lsrc/include/pigpio' } } } build.gradle
  31. Deploy on Pi with Gradle •Cross compilation 0 repetitive scp

    •Always automate! ✨ •Step 1: passwordless SSH with your Raspberry Pi •Step 2: write a gradle task which does the job •Step 3: 24 Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md
  32. Source: https://gradle-ssh-plugin.github.io/ task deployOnPi remotes { pi { host =

    'address ip of your pi' user = 'pi user' identity = file('/Users/foo/.ssh/id_rsa') } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/main.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }
  33. •The executable will launch a continuous loop •How do we

    interact with the application? •With a physical input! •So how can we receive the callback?
  34. pigpio: Set Alert // Register a function to be called

    (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick);
  35. Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int,

    platform.posix.uint32_t) -> Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );
  36. Mapping Function Pointer from C •From typedef to typealias •How

    to convert a Kotlin function to a pointer to a C function? •staticCFunction(::kotlinFunction) for the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html
  37. Button Callback val onButtonPressed = staticCFunction<Int, Int, UInt, Unit> {

    gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }
  38. I²C: I-squared-C • I2C bus connects simple peripheral devices with

    small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c
  39. PWM: Pulse Width Modulation •Pulse Width Modulation (PWM) is a

    common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) •Frequency: expressed in Hz •Period: the time each cycle takes and is the inverse of frequency •Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm
  40. In search of the C binding •PCA9685 16-Channel 12 Bit

    PWM Servo Driver •Adafruit C++ lib => not supported in Kotlin/Native •Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3
  41. pigpio: Initialize PWM // Open the I2C bus device and

    assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq);
  42. PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C(

    adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )
  43. pigpio: Set PWM Values // Set all PWM channels from

    two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals);
  44. Generate Kotlin Stub CValuesRef<UIntVar>? public fun PCA9685_setPWMVals( fd: kotlin.Int, addr:

    kotlin.UByte, onVals: CValuesRef<UIntVar>?, offVals: CValuesRef<UIntVar>? ): Int
  45. Pass Int* to C from Kotlin/Native •Get a NativePlacement instance

    from the memScoped {…} block receiver •Create an array of UIntVar instance with the allocArray<T> extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/
  46. Set PWM On/Off values (1/2) memScoped { val onValues =

    allocArray<UIntVar>(_PCA9685_CHANS) val offValues = allocArray<UIntVar>(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }
  47. Oh Wait, Simpler Solution! •${type}Array.toCValues(), where type is the Kotlin

    primitive type •Array<CPointer<T>?>.toCValues() •List<CPointer<T>?>.toCValues() •cValuesOf(vararg elements: ${type}), where type is a primitive or pointer Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
  48. Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for

    (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )
  49. Understand raspicam •4 Applications are provided: raspistill, raspivid, raspiyuv and

    raspividyuv •Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* •No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/
  50. Reinvent the wheel? Nope. •POSIX bindings to the rescue •Execute

    command line from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html
  51. TensorFlow •There is a Kotlin/Native sample for TensorFlow! •But only

    for MacOS 64-bit & Linux 64-bit… •It works okay! 9 Source: https://juliuskunze.com/tensorflow-in-kotlin-native.html
  52. Tensorflow for C? •TensorFlow for C is supported on the

    following systems: •Linux, 64-bit, x86 •MacOS X, Version 10.12.6 (Sierra) or higher •Windows, 64-bit x86 Source: https://www.tensorflow.org/install/lang_c
  53. Tensorflow for Java? •TensorFlow for Java is supported on the

    following systems: •Ubuntu 16.04 or higher; 64-bit, x86 •macOS 10.12.6 (Sierra) or higher •Windows 7 or higher; 64-bit, x86 Source: https://www.tensorflow.org/install/lang_java
  54. What I learned so far •⚠ Experimental ⚠ • Tip

    of the iceberg •= C and cross compilation knowledges are essential • Fine tunings are needed •> Always read the doc! • It was fun exploring!
  55. References • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native

    Snake by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native