Bridge The Physical World: Kotlin Native on Raspberry Pi

With Kotlin/Native, we can now compile Kotlin code to run on various platforms, including Raspberry Pi. This cross-platform ability has drawn the attention of many developers. Through the process of building a hand game robot which can play rock paper scissors with human beings, this talk aims to show you the possibility of using Kotlin to control GPIO pins on a Raspberry Pi and other experiments such as performing machine learning operations in Kotlin using TensorFlow backend, thanks to the interoperability with C libraries.

Video: https://www.youtube.com/watch?v=73Ng_zsOsM8
Github: https://github.com/jinqian/kotlin-native-chifumi

#Kotlin #KotlinNative #RaspberryPi

Transcript

1. Bridge The Physical World: Kotlin/Native on Raspberry Pi Qian Jin | @bonbonking Conference for Kotliners 2019 Photo Credit: Harrison Broadbent on Unsplash
2. Android Developer, GDE for IoT Made in China ! Live in Paris " @bonbonking
3. Agenda •Rock-Paper-Scissor Hand Robot (with Android Things) •Kotlin/Native as an alternative •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip - Move the project into IntelliJ •Tip - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?) •Demo - Kotlin/Native on Raspberry Pi in actions!
4. Rock-Paper-Scissor hand game robot #$%& with Android Things
5. Source:
6. Source:
9. Break Down the Project •Data collection tool: raw data for model training •Hardware components: robot arms + game control •Application running on Android Things •Game logic and interactions •Image capture with camera preview •Gesture recognition with TensorFlow
10. Data Collection Tool •Web mobile app built with Node.js + Express.js •Deployed with Firebase Hosting / Function / Storage
13. GDPR? Not compliant…
14. Hardware (Android Things Version) •Android Things Starter Kit - NXP i.MX7D (~200€) •16-Channel PWM/Servo Driver (~12€) •Power Supply or battery (~5€) •5 servomotors (~12€) •Cardboard + wood sticks (~0€) •Glue Gun (~10€) •Electronic jumper + Resistors + LED + Button + Breadboard (~10€)
15. Source: https://developer.android.com/things
16. Source: https://www.adafruit.com/product/815 16-Channel 12-bit PWM/Servo Driver (I2C interface/PCA9685)
18. Source:
19. Source: https://commons.wikimedia.org/wiki/File:Android_robot_skateboarding.svg
20. Source: https://android-developers.googleblog.com/2019/02/an-update-on-android-things.html #
21. Source: https://www.iconfinder.com/icons/1172126/android_crying_emoji_mobile_mood_robot_sad_icon
22. Kotlin/Native (Explained to a cat)
23. Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.
24. Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source Code C C++ Java Kotlin … Machine Code ARM Sparc X86
 PowerPC … Compiler Design
25. Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design
26. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR Kotlin/Native as a LLVM Backend LLVM IR 0101 0101 …… IR: Intermediate Representation
27. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR LLVM IR 0101 0101 …… IR: Intermediate Representation Kotlin/Native as a LLVM Backend
28. Terminology ARM (Acorn RISC Machine) ARM (Advanced RISC Machines) RISC (Reduced Instruction Set Computing) CISC (Complex Instruction Set Computing) MIPS (Millions of Instructions Per Second) x86 ABI (Abstraction Binary Interface) FPU (Floating-point Unit) Source: https://github.com/raspberrypi/firmware
29. Raspberry Pi 3 Model B+ Spec •BCM2837 1.2 GHz 64-bit quad processor based on the ARMv8 Cortex-A53 •32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode”
30. My Raspberry Pi 3B armhf: ARM Hard Float
31. Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32, iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64 • WebAssembly: wasm32 •ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets
32. Source: https://en.wikipedia.org/wiki/Cross_compiler Host Machine Cross Compiler Source Code Target Machine Target compatible binary Cross Compilation
33. Source: https://en.wikipedia.org/wiki/Cross_compiler MacBook Pro (macosX64) Cross Compiler Raspberry Pi (linuxArm32Hfp) Target compatible binary Cross Compilation Source Code
34. Key Components •konanc: Kotlin/Native compiler •cinterop: produce kotlin binding for native library Source: https://kotlinlang.org/docs/reference/native/c_interop.html
35. C Interoperability •Create a .def file describing what to include into bindings •Use the cinterop tool to produce Kotlin bindings •Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md .def .h cinterop .klib
37. My Setup Macbook + Raspberry Pi 3 (Model B)
38. Hardware (Raspberry Pi Version) •Android Things Starter Kit - NXP i.MX7D (~200€) •Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display •…
39. Source:
40. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C LED GPIO Button GPIO Ground
42. February 2019: Me trying to cross compile Kotlin/Native application on my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.
43. exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64 host at org.jetbrains.kotlin.backend.konan.KonanConfig.<init>(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …
44. Source: https://github.com/JetBrains/kotlin-native/releases
45. Step 0: Hello, Kotlin/Native!
46. fun main() { println("Hello Kotlin/Native from Qian’s Macbook!") } Source: https://hadihariri.com/2017/04/09/kotlin-on-raspberry-pi/ hello.kt
47. $ konanc hello.kt -target raspberrypi -o hello $ scp hello.kexe pi@{IP-RASPBERRY-PI}:~/dir/ Compile with command line
48. Source:
49. Step 1: Basic GPIO GPIO (General Purpose Input/Output)
50. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC LED GPIO Ground
51. pigpio pigpio: a library for the Raspberry which allows control of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio
52. Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/
53. Make it work! •Create a pigpio.def file •Generate Kotlin stub for the library •⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation •Compile the code by linking the library using linkerOpts option Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ / Thank you Hadi Hariri!!
54. pigpio.def headers = pigpio.h
55. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub
56. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -copt (-compilerOpts) -> additional compiler options -I`pwd`/pigpio -> the location of the header file
57. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -o pigpio.bc -> the output file for kotlin stub
58. $ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -target -> the target platform is raspberry pi
59. pigpio: Basics int gpioInitialise(void); int gpioSetMode(unsigned gpio, unsigned mode); int gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros);
60. Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port, PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }
61. $ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $ sudo make install Build the library on Pi libpigpio.so
62. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary
63. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary
64. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary
65. $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary
66. Generate the binary led.kexe $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led
67. Tip 1: Move the project into IntelliJ
68. Move into IntelliJ •Articles are generally out of date •No much documentation at this moment •Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
69. kotlin { targets { fromPreset(kotlin.presets.linuxArm32Hfp, 'chifumi') { compilations.main.outputKinds 'EXECUTABLE' compilations.main.entryPoint 'chifumi.robot.main' compilations.main.cinterops { pigpio { includeDirs 'src/include/pigpio' } } // libpigpio.so are compiled on raspberry pi and copied here compilations.main.linkerOpts '-lpigpio -Lsrc/include/pigpio' } } } build.gradle
70. Tip 2: Deploy on Pi with Gradle SSH plugin
71. Deploy on Pi with Gradle •Cross compilation 0 repetitive scp •Always automate! ✨ •Step 1: passwordless SSH with your Raspberry Pi •Step 2: write a gradle task which does the job •Step 3: 24 Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md
72. Source: https://gradle-ssh-plugin.github.io/ task deployOnPi remotes { pi { host = 'address ip of your pi' user = 'pi user' identity = file('/Users/foo/.ssh/id_rsa') } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/main.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }
73. Step 2: Button Callback (Mapping Function Pointer from C)
74. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC Button GPIO Ground
75. •The executable will launch a continuous loop •How do we interact with the application? •With a physical input! •So how can we receive the callback?
76. pigpio: Set Alert // Register a function to be called (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick);
77. Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int, platform.posix.uint32_t) -> Unit>>
78. Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int, platform.posix.uint32_t) -> Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );
79. Mapping Function Pointer from C •From typedef to typealias •How to convert a Kotlin function to a pointer to a C function? •staticCFunction(::kotlinFunction) for the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html
80. Button Callback val onButtonPressed = staticCFunction<Int, Int, UInt, Unit> { gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }
81. Step 3: Control Servomotors
82. I²C: I-squared-C • I2C bus connects simple peripheral devices with small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c
83. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C Ground
84. Source:
85. PWM: Pulse Width Modulation •Pulse Width Modulation (PWM) is a common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) •Frequency: expressed in Hz •Period: the time each cycle takes and is the inverse of frequency •Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm
86. Source: https://developer.android.com/things/sdk/pio/pwm
87. In search of the C binding •PCA9685 16-Channel 12 Bit PWM Servo Driver •Adafruit C++ lib => not supported in Kotlin/Native •Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3
88. pigpio: Initialize PWM // Open the I2C bus device and assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq);
89. PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C( adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )
90. pigpio: Set PWM Values // Set all PWM channels from two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals);
91. Generate Kotlin Stub CValuesRef<UIntVar>? public fun PCA9685_setPWMVals( fd: kotlin.Int, addr: kotlin.UByte, onVals: CValuesRef<UIntVar>?, offVals: CValuesRef<UIntVar>? ): Int
92. Understand CValuesRef Source: https://kotlinlang.org/api/latest/jvm/stdlib/kotlinx.cinterop/-c-values-ref/index.html
93. Pass Int* to C from Kotlin/Native •Get a NativePlacement instance from the memScoped {…} block receiver •Create an array of UIntVar instance with the allocArray<T> extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/
94. Set PWM On/Off values (1/2) memScoped { val onValues = allocArray<UIntVar>(_PCA9685_CHANS) val offValues = allocArray<UIntVar>(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }
95. Oh Wait, Simpler Solution! •${type}Array.toCValues(), where type is the Kotlin primitive type •Array<CPointer<T>?>.toCValues() •List<CPointer<T>?>.toCValues() •cValuesOf(vararg elements: ${type}), where type is a primitive or pointer Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md
96. Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )
97. Step 4: Capture Image with Pi Camera
98. Understand raspicam •4 Applications are provided: raspistill, raspivid, raspiyuv and raspividyuv •Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* •No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/
99. Reinvent the wheel? Nope. •POSIX bindings to the rescue •Execute command line from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html
100. Code platform.posix.system("raspistill -t 2000 -o photo.jpg")
101. Run TensorFlow With Kotlin/Native (on raspberry pi?)
102. TensorFlow •There is a Kotlin/Native sample for TensorFlow! •But only for MacOS 64-bit & Linux 64-bit… •It works okay! 9 Source: https://juliuskunze.com/tensorflow-in-kotlin-native.html
103. Tensorflow for C? •TensorFlow for C is supported on the following systems: •Linux, 64-bit, x86 •MacOS X, Version 10.12.6 (Sierra) or higher •Windows, 64-bit x86 Source: https://www.tensorflow.org/install/lang_c
104. Tensorflow for Java? •TensorFlow for Java is supported on the following systems: •Ubuntu 16.04 or higher; 64-bit, x86 •macOS 10.12.6 (Sierra) or higher •Windows 7 or higher; 64-bit, x86 Source: https://www.tensorflow.org/install/lang_java
105. KEEP CALM AND BUILD TENSORFLOW LITE FROM SOURCE(?)
106. $./tensorflow/lite/tools/make/download_dependencies.sh $./tensorflow/lite/tools/make/build_aarch64_lib.sh Source: https://www.tensorflow.org/lite/guide/build_arm64, https://github.com/tensorflow/tensorflow/pull/29017 Cross Compiling TF Lite
108. Demo: Kotlin/Native on Raspberry Pi in actions
110. Conclusion ☕
111. What I learned so far •⚠ Experimental ⚠ • Tip of the iceberg •= C and cross compilation knowledges are essential • Fine tunings are needed •> Always read the doc! • It was fun exploring!
112. References • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native Snake by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native
113. Thank you! Github Repo: https://github.com/jinqian/kotlin-native-chifumi