Bridge The Physical World: Kotlin Native on Raspberry Pi

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
June 07, 2019

Bridge The Physical World: Kotlin Native on Raspberry Pi

With Kotlin/Native, we can now compile Kotlin code to run on various platforms, including Raspberry Pi. This cross-platform ability has drawn the attention of many developers. Through the process of building a hand game robot which can play rock paper scissors with human beings, this talk aims to show you the possibility of using Kotlin to control GPIO pins on a Raspberry Pi and other experiments such as performing machine learning operations in Kotlin using TensorFlow backend, thanks to the interoperability with C libraries.

Video: https://www.youtube.com/watch?v=73Ng_zsOsM8
Github: https://github.com/jinqian/kotlin-native-chifumi

#Kotlin #KotlinNative #RaspberryPi

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

June 07, 2019
Tweet

Transcript

  1. Bridge The Physical World: Kotlin/Native on Raspberry Pi Qian Jin

    | @bonbonking Conference for Kotliners 2019 Photo Credit: Harrison Broadbent on Unsplash
  2. Android Developer, GDE for IoT Made in China ! Live

    in Paris " @bonbonking
  3. Agenda •Rock-Paper-Scissor Hand Robot (with Android Things) •Kotlin/Native as an

    alternative •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip - Move the project into IntelliJ •Tip - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?) •Demo - Kotlin/Native on Raspberry Pi in actions!
  4. Rock-Paper-Scissor hand game robot #$%& with Android Things

  5. Source:

  6. Source:

  7. None
  8. None
  9. Break Down the Project •Data collection tool: raw data for

    model training •Hardware components: robot arms + game control •Application running on Android Things •Game logic and interactions •Image capture with camera preview •Gesture recognition with TensorFlow
  10. Data Collection Tool •Web mobile app built with Node.js +

    Express.js •Deployed with Firebase Hosting / Function / Storage
  11. None
  12. None
  13. GDPR? Not compliant…

  14. Hardware (Android Things Version) •Android Things Starter Kit - NXP

    i.MX7D (~200€) •16-Channel PWM/Servo Driver (~12€) •Power Supply or battery (~5€) •5 servomotors (~12€) •Cardboard + wood sticks (~0€) •Glue Gun (~10€) •Electronic jumper + Resistors + LED + Button + Breadboard (~10€)
  15. Source: https://developer.android.com/things

  16. Source: https://www.adafruit.com/product/815 16-Channel 12-bit PWM/Servo Driver (I2C interface/PCA9685)

  17. None
  18. Source:

  19. Source: https://commons.wikimedia.org/wiki/File:Android_robot_skateboarding.svg

  20. Source: https://android-developers.googleblog.com/2019/02/an-update-on-android-things.html #

  21. Source: https://www.iconfinder.com/icons/1172126/android_crying_emoji_mobile_mood_robot_sad_icon

  22. Kotlin/Native (Explained to a cat)

  23. Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the

    Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.
  24. Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source

    Code C C++ Java Kotlin … Machine Code ARM Sparc X86
 PowerPC … Compiler Design
  25. Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend

    LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design
  26. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR Kotlin/Native as a LLVM Backend LLVM IR

    0101 0101 …… IR: Intermediate Representation
  27. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR LLVM IR 0101 0101 …… IR: Intermediate

    Representation Kotlin/Native as a LLVM Backend
  28. Terminology ARM (Acorn RISC Machine) ARM (Advanced RISC Machines) RISC

    (Reduced Instruction Set Computing) CISC (Complex Instruction Set Computing) MIPS (Millions of Instructions Per Second) x86 ABI (Abstraction Binary Interface) FPU (Floating-point Unit) Source: https://github.com/raspberrypi/firmware
  29. Raspberry Pi 3 Model B+ Spec •BCM2837 1.2 GHz 64-bit

    quad processor based on the ARMv8 Cortex-A53 •32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode”
  30. My Raspberry Pi 3B armhf: ARM Hard Float

  31. Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32,

    iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64 • WebAssembly: wasm32 •ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets
  32. Source: https://en.wikipedia.org/wiki/Cross_compiler Host Machine Cross Compiler Source Code Target Machine

    Target compatible binary Cross Compilation
  33. Source: https://en.wikipedia.org/wiki/Cross_compiler MacBook Pro (macosX64) Cross Compiler Raspberry Pi (linuxArm32Hfp)

    Target compatible binary Cross Compilation Source Code
  34. Key Components •konanc: Kotlin/Native compiler •cinterop: produce kotlin binding for

    native library Source: https://kotlinlang.org/docs/reference/native/c_interop.html
  35. C Interoperability •Create a .def file describing what to include

    into bindings •Use the cinterop tool to produce Kotlin bindings •Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md .def .h cinterop .klib
  36. None
  37. My Setup Macbook + Raspberry Pi 3 (Model B)

  38. Hardware (Raspberry Pi Version) •Android Things Starter Kit - NXP

    i.MX7D (~200€) •Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display •…
  39. Source:

  40. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C LED GPIO Button

    GPIO Ground
  41. None
  42. February 2019: Me trying to cross compile Kotlin/Native application on

    my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.
  43. exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64

    host at org.jetbrains.kotlin.backend.konan.KonanConfig.<init>(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …
  44. Source: https://github.com/JetBrains/kotlin-native/releases

  45. Step 0: Hello, Kotlin/Native!

  46. fun main() { println("Hello Kotlin/Native from Qian’s Macbook!") } Source:

    https://hadihariri.com/2017/04/09/kotlin-on-raspberry-pi/ hello.kt
  47. $ konanc hello.kt -target raspberrypi -o hello $ scp hello.kexe

    pi@{IP-RASPBERRY-PI}:~/dir/ Compile with command line
  48. Source:

  49. Step 1: Basic GPIO GPIO (General Purpose Input/Output)

  50. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC LED GPIO Ground

  51. pigpio pigpio: a library for the Raspberry which allows control

    of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio
  52. Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO

    access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/
  53. Make it work! •Create a pigpio.def file •Generate Kotlin stub

    for the library •⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation •Compile the code by linking the library using linkerOpts option Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ / Thank you Hadi Hariri!!
  54. pigpio.def headers = pigpio.h

  55. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub
  56. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -copt (-compilerOpts) -> additional compiler options -I`pwd`/pigpio -> the location of the header file
  57. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -o pigpio.bc -> the output file for kotlin stub
  58. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi

    Generate Kotlin Stub -target -> the target platform is raspberry pi
  59. pigpio: Basics int gpioInitialise(void); int gpioSetMode(unsigned gpio, unsigned mode); int

    gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros);
  60. Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port,

    PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }
  61. $ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $

    sudo make install Build the library on Pi libpigpio.so
  62. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio"

    -o led Generate the binary
  63. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio"

    -o led Generate the binary
  64. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio"

    -o led Generate the binary
  65. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio"

    -o led Generate the binary
  66. Generate the binary led.kexe $ konanc led.kt -target raspberrypi -library

    pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led
  67. Tip 1: Move the project into IntelliJ

  68. Move into IntelliJ •Articles are generally out of date •No

    much documentation at this moment •Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
  69. kotlin { targets { fromPreset(kotlin.presets.linuxArm32Hfp, 'chifumi') { compilations.main.outputKinds 'EXECUTABLE' compilations.main.entryPoint

    'chifumi.robot.main' compilations.main.cinterops { pigpio { includeDirs 'src/include/pigpio' } } // libpigpio.so are compiled on raspberry pi and copied here compilations.main.linkerOpts '-lpigpio -Lsrc/include/pigpio' } } } build.gradle
  70. Tip 2: Deploy on Pi with Gradle SSH plugin

  71. Deploy on Pi with Gradle •Cross compilation 0 repetitive scp

    •Always automate! ✨ •Step 1: passwordless SSH with your Raspberry Pi •Step 2: write a gradle task which does the job •Step 3: 24 Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md
  72. Source: https://gradle-ssh-plugin.github.io/ task deployOnPi remotes { pi { host =

    'address ip of your pi' user = 'pi user' identity = file('/Users/foo/.ssh/id_rsa') } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/main.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }
  73. Step 2: Button Callback (Mapping Function Pointer from C)

  74. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC Button GPIO Ground

  75. •The executable will launch a continuous loop •How do we

    interact with the application? •With a physical input! •So how can we receive the callback?
  76. pigpio: Set Alert // Register a function to be called

    (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick);
  77. Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int,

    platform.posix.uint32_t) -> Unit>>
  78. Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int,

    platform.posix.uint32_t) -> Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );
  79. Mapping Function Pointer from C •From typedef to typealias •How

    to convert a Kotlin function to a pointer to a C function? •staticCFunction(::kotlinFunction) for the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html
  80. Button Callback val onButtonPressed = staticCFunction<Int, Int, UInt, Unit> {

    gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }
  81. Step 3: Control Servomotors

  82. I²C: I-squared-C • I2C bus connects simple peripheral devices with

    small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c
  83. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C Ground

  84. Source:

  85. PWM: Pulse Width Modulation •Pulse Width Modulation (PWM) is a

    common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) •Frequency: expressed in Hz •Period: the time each cycle takes and is the inverse of frequency •Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm
  86. Source: https://developer.android.com/things/sdk/pio/pwm

  87. In search of the C binding •PCA9685 16-Channel 12 Bit

    PWM Servo Driver •Adafruit C++ lib => not supported in Kotlin/Native •Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3
  88. pigpio: Initialize PWM // Open the I2C bus device and

    assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq);
  89. PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C(

    adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )
  90. pigpio: Set PWM Values // Set all PWM channels from

    two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals);
  91. Generate Kotlin Stub CValuesRef<UIntVar>? public fun PCA9685_setPWMVals( fd: kotlin.Int, addr:

    kotlin.UByte, onVals: CValuesRef<UIntVar>?, offVals: CValuesRef<UIntVar>? ): Int
  92. Understand CValuesRef Source: https://kotlinlang.org/api/latest/jvm/stdlib/kotlinx.cinterop/-c-values-ref/index.html

  93. Pass Int* to C from Kotlin/Native •Get a NativePlacement instance

    from the memScoped {…} block receiver •Create an array of UIntVar instance with the allocArray<T> extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/
  94. Set PWM On/Off values (1/2) memScoped { val onValues =

    allocArray<UIntVar>(_PCA9685_CHANS) val offValues = allocArray<UIntVar>(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }
  95. Oh Wait, Simpler Solution! •${type}Array.toCValues(), where type is the Kotlin

    primitive type •Array<CPointer<T>?>.toCValues() •List<CPointer<T>?>.toCValues() •cValuesOf(vararg elements: ${type}), where type is a primitive or pointer Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
  96. Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for

    (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )
  97. Step 4: Capture Image with Pi Camera

  98. Understand raspicam •4 Applications are provided: raspistill, raspivid, raspiyuv and

    raspividyuv •Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* •No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/
  99. Reinvent the wheel? Nope. •POSIX bindings to the rescue •Execute

    command line from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html
  100. Code platform.posix.system("raspistill -t 2000 -o photo.jpg")

  101. Run TensorFlow With Kotlin/Native (on raspberry pi?)

  102. TensorFlow •There is a Kotlin/Native sample for TensorFlow! •But only

    for MacOS 64-bit & Linux 64-bit… •It works okay! 9 Source: https://juliuskunze.com/tensorflow-in-kotlin-native.html
  103. Tensorflow for C? •TensorFlow for C is supported on the

    following systems: •Linux, 64-bit, x86 •MacOS X, Version 10.12.6 (Sierra) or higher •Windows, 64-bit x86 Source: https://www.tensorflow.org/install/lang_c
  104. Tensorflow for Java? •TensorFlow for Java is supported on the

    following systems: •Ubuntu 16.04 or higher; 64-bit, x86 •macOS 10.12.6 (Sierra) or higher •Windows 7 or higher; 64-bit, x86 Source: https://www.tensorflow.org/install/lang_java
  105. KEEP CALM AND BUILD TENSORFLOW LITE FROM SOURCE(?)

  106. $./tensorflow/lite/tools/make/download_dependencies.sh $./tensorflow/lite/tools/make/build_aarch64_lib.sh Source: https://www.tensorflow.org/lite/guide/build_arm64, https://github.com/tensorflow/tensorflow/pull/29017 Cross Compiling TF Lite

  107. None
  108. Demo: Kotlin/Native on Raspberry Pi in actions

  109. None
  110. Conclusion ☕

  111. What I learned so far •⚠ Experimental ⚠ • Tip

    of the iceberg •= C and cross compilation knowledges are essential • Fine tunings are needed •> Always read the doc! • It was fun exploring!
  112. References • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native

    Snake by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native
  113. Thank you! Github Repo: https://github.com/jinqian/kotlin-native-chifumi