KotlinConf 2019 | Bridge the physical world: Kotlin/Native on Raspberry Pi

With Kotlin/Native, we can now compile Kotlin code to run on various platforms, including Raspberry Pi. This cross-platform ability has drawn the attention of many developers. Through the process of building a hand game robot which can play rock paper scissors with human beings, this talk aims to show you the possibility of using Kotlin to control GPIO pins on a Raspberry Pi and other experiments such as performing machine learning operations in Kotlin using TensorFlow backend, thanks to the interoperability with C libraries.

Video: https://youtu.be/KfdNiMP0emE
Github: https://github.com/jinqian/kotlin-native-chifoumi

Transcript

1. Copenhagen Denmark BRIDGE THE PHYSICAL WORLD: KOTLIN/NATIVE ON RASPBERRY PI QIAN JIN @bonbonking
2. Android Developer, GDE for IoT Made in China Live in Paris @bonbonking
3. •Context •Rock-Paper-Scissor Hand Robot •Kotlin/Native as an alternative •Project setup •Step by step •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip 1 - Move the project into IntelliJ •Tip 2 - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?!) •Demo - Kotlin/Native on Raspberry Pi in actions! 3 Agenda
4. Rock-Paper-Scissor hand game robot ✌✊ with Android Things
6. Source:
7. Source:
9. • Data collection tool: raw data for model training • Hardware components: robot arms + game control • Application running on Android Things • Game logic and interactions • Image capture with camera preview • Gesture recognition with TensorFlow 9 Break Down the Project
12. GDPR? Not compliant…
13. • Android Things Starter Kit - NXP i.MX7D (~200€) • 16-Channel PWM/Servo Driver (~12€) • Power Supply or battery (~5€) • 5 servomotors (~12€) • Cardboard + wood sticks (~0€) • Glue Gun (~10€) • Electronic jumper + Resistors + LED + Button + Breadboard (~10€) 13 Hardware (Android Things Version)
14. Source: https://developer.android.com/things
15. Source: https://www.adafruit.com/product/815 16-Channel 12-bit PWM/Servo Driver (I2C interface/PCA9685)
17. Source:
18. Source: https://commons.wikimedia.org/wiki/File:Android_robot_skateboarding.svg
19. Source: https://android-developers.googleblog.com/2019/02/an-update-on-android-things.html
20. Source: https://www.iconfinder.com/icons/1172126/android_crying_emoji_mobile_mood_robot_sad_icon
21. Kotlin/Native (Explained to a cat)
22. Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.
23. Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source Code C C++ Java Kotlin … Machine Code ARM Sparc X86 PowerPC … Compiler Design
24. Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design
25. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR Kotlin/Native as a LLVM Backend LLVM IR 0101 0101 …… IR: Intermediate Representation
26. Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR LLVM IR 0101 0101 …… IR: Intermediate Representation Kotlin/Native as a LLVM Backend
27. Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32, iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64, mingwX86 • WebAssembly: wasm32 • watchOS, tvOS… ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets 27
28. Architecture •ARM (*Originally: Acorn RISC Machine) •ARM (Advanced RISC Machines) •x86 (Successors to Intel’s 8086 processor) •RISC (Reduced Instruction Set Computing) •CISC (Complex Instruction Set Computing) Source: https://github.com/raspberrypi/firmware 28
29. Raspberry Pi 3 Model B+ Spec • BCM2837 1.2 GHz 64-bit quad processor based on the ARMv8 Cortex-A53 • 32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode” 29
30. My Raspberry Pi 3B *armhf (ARM Hard Float) ✅ Among target preset (linuxArm32Hfp)
31. Source: https://en.wikipedia.org/wiki/Cross_compiler Host Machine Cross Compiler Source Code Target Machine Target compatible binary Cross Compilation
32. Source: https://en.wikipedia.org/wiki/Cross_compiler MacBook Pro (macosX64) Cross Compiler Raspberry Pi (linuxArm32Hfp) Target compatible binary Cross Compilation Source Code
33. Key Components •konanc: Kotlin/Native compiler •cinterop: produce Kotlin binding for native libraries Source: https://kotlinlang.org/docs/reference/native/c_interop.html 33
34. C Interoperability • Create a .def file describing what to include into bindings • Use the cinterop tool to produce Kotlin bindings • Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 34 .def .h cinterop .klib
35. • Pointers and arrays => CPointer<T>? • Enums => Kotlin enum or integral values • Structs / unions => types having fields available via the dot notation (i.e. kotlinConfInstance.speakers) • typedef => typealias Source: https://kotlinlang.org/docs/reference/native/c_interop.html 35 Basic Interop Types C Type Kotlin Type Signed, unsigned integral and floating point Kotlin counterpart with the same width Pointers and arrays CPointer<T>? Enums Kotlin enum or integral values Structs / Unions Types having fields available via the dot notation (i.e. kotlinConfInstance.speakers) typedef typealias
37. My Setup Macbook + Raspberry Pi 3 (Model B)
38. • Android Things Starter Kit - NXP i.MX7D (~200€) • Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display • … 38 Hardware (Raspberry Pi Version)
39. Source:
40. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C LED GPIO Button GPIO Ground
41. February 2019: Me trying to cross compile Kotlin/Native application on my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.
42. exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64 host at org.jetbrains.kotlin.backend.konan.KonanConfig.<init>(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …
43. Source: https://github.com/JetBrains/kotlin-native/releases
44. Step 0: Hello, Kotlin/Native!
45. hello.kt Source: https://hadihariri.com/2017/04/09/kotlin-on-raspberry-pi/ fun main() { println("Hello Kotlin/Native from Qian’s Macbook!") }
46. Compile Manually $ konanc hello.kt -o hello -target raspberrypi $ scp hello.kexe pi@{IP-RASPBERRY-PI}:~/dir/ hello.kexe
47. Source:
48. Step 1: Basic GPIO (General Purpose Input/Output)
49. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC LED GPIO Ground
50. pigpio pigpio: a library for the Raspberry which allows control of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio 50
51. Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/ 51
52. • Create a pigpio.def file • Use the cinterop tool to produce Kotlin bindings • ⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation • Run Kotlin/Native compiler on an application to produce the final executable Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ 52 Make it work! Thank you Hadi Hariri!!
53. pigpio.def headers = pigpio.h
54. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi
55. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi -def pigpio -> the .def file we just created
56. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi -compiler-option -I`pwd`/include —> indicate the path of the header file
57. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi -o pigpio -> the output name for the Kotlin binding
58. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi -target raspberrypi -> our target platform
59. Generate Kotlin Binding $ cinterop \ -def pigpio.def \ -compiler-option -I`pwd`/include \ -o pigpio \ -target raspberrypi pigpio.klib pigpio-build/
60. pigpio: Basic API in C int gpioInitialise(void); int gpioSetMode(unsigned gpio, unsigned mode); int gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros); 60
61. Generated Kotlin Binding @kotlinx.cinterop.internal.CCall fun gpioInitialise(): Int @kotlinx.cinterop.internal.CCall fun gpioSetMode(gpio: UInt, mode: UInt): Int @kotlinx.cinterop.internal.CCall fun gpioWrite(gpio: UInt, level: UInt): Int @kotlinx.cinterop.internal.CCall fun gpioSleep(timetype: UInt, seconds: Int, micros: Int): Int
62. Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port, PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }
63. $ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $ sudo make install Install pigpio Retrieve libpigpio.so for cross compilation
64. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi
65. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -library pigpio -> indicate the Kotlin binding to use
66. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -linker-options -> link the library libpigpio.so
67. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -o led -> indicate the output executable name
68. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi -target raspberrypi -> still our target!
69. Generate executable $ konanc led.kt \ -library pigpio \ -linker-options "-L`pwd`/lib -lpigpio" \ -o led \ -target raspberrypi led.kexe
70. Tip 1: Move the project into IntelliJ
71. • kotlin-multiplatform plugin • Still experimental, DSL keeps changing • Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 71 Move into IntelliJ
72. build.gradle Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html plugins { id 'kotlin-multiplatform' version '1.3.60' } kotlin { linuxArm32Hfp("chifoumi") { binaries { executable { entryPoint “chifoumi.robot.main" // libpigpio.so are compiled on raspberry pi and copied here linkerOpts("-Lsrc/include/pigpio", "-lpigpio") } } compilations.main.cinterops { pigpio { includeDirs ‘src/include/pigpio' } } } }
73. Tip 2: Deploy on Pi with Gradle SSH plugin
74. • Cross compilation repetitive scp • Always automate! ✨ • Step 1: passwordless SSH with your Raspberry Pi • Step 2: write a gradle task which does the job • Step 3: Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md 74 Deploy on Pi with Gradle
75. task deployOnPi Source: https://gradle-ssh-plugin.github.io/ remotes { pi { host = ‘IP address ip of your pi' user = 'pi user' identity = file(‘/path/to/.ssh/id_rsa’) } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/app.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }
76. Step 2: Button Callback (Mapping Function Pointer from C)
77. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC Button GPIO Ground
78. •The executable will launch a continuous loop… •How do we interact with the application? •With a physical input! •So how can we receive the callback?
79. pigpio: Set Alert Function // Register a function to be called (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick); 79
80. Generated Kotlin Binding Source: https://kotlinlang.org/docs/reference/type-aliases.html typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int, platform.posix.uint32_t) -> Unit>>
81. Generated Kotlin Binding Source: https://kotlinlang.org/docs/reference/type-aliases.html typealias gpioAlertFunc_t = CPointer<CFunction<(Int, Int, platform.posix.uint32_t) -> Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );
82. • How to convert a Kotlin function to a pointer to a C function? • staticCFunction(::kotlinFunction) to the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html 82 Mapping Function Pointer from C
83. Button Callback val onButtonPressed = staticCFunction<Int, Int, UInt, Unit> { gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }
84. Step 3: Control Servomotors
85. I²C: I-squared-C • I2C bus connects simple peripheral devices with small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c 85
86. Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C Ground
87. Source:
88. PWM: Pulse Width Modulation • Pulse Width Modulation (PWM) is a common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) • Frequency: expressed in Hz • Period: the time each cycle takes and is the inverse of frequency • Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm 88
89. Source: https://developer.android.com/things/sdk/pio/pwm
90. • PCA9685 16-Channel 12 Bit PWM Servo Driver • Adafruit C++ lib: not interoperable with Kotlin Native • Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3 90 In search of the C binding
91. $ git clone https://github.com/edlins/libPCA9685 $ cd libPCA9685 && mkdir build && cd build $ cmake .. $ make $ sudo make install Install libPCA9685 Retrieve libPCA9685.so for cross compilation
92. pigpio: Initialize PWM // Open the I2C bus device and assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq); 92
93. PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C( adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )
94. pigpio: Set PWM Values // set all PWM channels from two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals); // set a single PWM channel with a 16-bit ON val and a 16-bit OFF val int PCA9685_setPWMVal(int fd, unsigned char addr, unsigned char reg, unsigned int on, unsigned int off); // set all PWM channels with one 16-bit ON val and one 16-bit OFF val int PCA9685_setAllPWM(int fd, unsigned char addr, unsigned int on, unsigned int off); 94
95. pigpio: Set PWM Values // set all PWM channels from two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals); // set a single PWM channel with a 16-bit ON val and a 16-bit OFF val int PCA9685_setPWMVal(int fd, unsigned char addr, unsigned char reg, unsigned int on, unsigned int off); // set all PWM channels with one 16-bit ON val and one 16-bit OFF val int PCA9685_setAllPWM(int fd, unsigned char addr, unsigned int on, unsigned int off); 95
96. Generated Kotlin Binding CValuesRef<UIntVar>? public fun PCA9685_setPWMVals( fd: Int, addr: UByte, onVals: CValuesRef<UIntVar>?, offVals: CValuesRef<UIntVar>? ): Int
97. Source: https://kotlinlang.org/api/latest/jvm/stdlib/kotlinx.cinterop/-c-values-ref/index.html 97 Understand CValuesRef
98. • Get a NativePlacement instance from the memScoped {…} block receiver • Create an array of UIntVar instance with the allocArray<T> extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/ 98 Explicit Native Memory Allocation
99. Set PWM On/Off values (1/2) memScoped { val onValues = allocArray<UIntVar>(_PCA9685_CHANS) val offValues = allocArray<UIntVar>(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }
100. • The CValuesRef<T> is designed to support C array literals without explicit native memory allocation. • Construct the immutable self-contained sequence of C values: • ${type}Array.toCValues(), where type is the Kotlin primitive type • cValuesOf(vararg elements: ${type}), where type is a primitive or pointer • Array<CPointer<T>?>.toCValues(), List<CPointer<T>?>.toCValues() Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md 100 Oh Wait, Simpler Solution!
101. Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )
102. Step 4: Capture Image with Pi Camera
103. Understand raspicam • 4 Applications are provided: raspistill, raspivid, raspiyuv and raspividyuv • Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* • No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/ 103
104. • POSIX bindings to the rescue • Execute command line from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html 104 Reinvent the wheel? Nope.
105. Code platform.posix.system("raspistill -t 2000 -o photo.jpg")
106. Run TensorFlow With Kotlin/Native (on raspberry pi?!)
107. • TensorFlow for C is supported on: • Linux, 64-bit, x86 • MacOS X, Version 10.12.6 (Sierra) or higher • Windows, 64-bit x86 • No official C library for Raspberry Pi Source: https://www.tensorflow.org/install/lang_c 107 TensorFlow for C
108. KEEP CALM AND BUILD TENSORFLOW FROM SOURCE
110. No documentation ¯\_(ツ)_/¯ Unofficial binaries https://github.com/PINTO0309/Tensorflow-bin *“The behavior is unconfirmed because I do not have C language implementation skills.” Source: https://github.com/PINTO0309/Tensorflow-bin 110 TensorFlow for C
111. Demo: Kotlin/Native on Raspberry Pi in action
115. Conclusion ☕
116. • ⚠ Experimental ⚠ • Tip of the iceberg • Debugging is hard • Fine tuning on the sampling rate is needed • C and cross compilation knowledges are essential • It was fun exploring! 116 What I learned so far
117. • Software is getting incredibly complex and interfacing between languages is increasingly important • Gain low-level control without giving up high-level conveniences • The Openness of Kotlin/Native (for Web Assembly even) 117 Why?
118. • Both using LLVM as a backend • Different memory management systems • Kotlin/Native: automated, reference counter with a cycle collector • Rust: manual, with smart pointers • Maturity level 118 Kotlin/Native v.s. Rust?
119. • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native Snake by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native •Int ptr in Kotlin/Native 119 References
120. Github: https://github.com/jinqian/kotlin-native-chifoumi
121. #KotlinConf THANK YOU AND REMEMBER TO VOTE Qian Jin @bonbonking