Tweet
Tweet

Slide 1

Slide 1 text

Bridge The Physical World: Kotlin/Native on Raspberry Pi Qian Jin | @bonbonking Conference for Kotliners 2019 Photo Credit: Harrison Broadbent on Unsplash

Slide 2

Slide 2 text

Android Developer, GDE for IoT Made in China ! Live in Paris " @bonbonking

Slide 3

Slide 3 text

Agenda •Rock-Paper-Scissor Hand Robot (with Android Things) •Kotlin/Native as an alternative •Step 0 - Hello Kotlin Native •Step 1 - Basic GPIO: Blink the LED •Tip - Move the project into IntelliJ •Tip - Deploy with Gradle SSH plugin •Step 2 - Button Callback •Step 3 - Control servomotors •Step 4 - Capture image with pi camera •Step 5 - Run TensorFlow with Kotlin/Native (on raspberry pi?) •Demo - Kotlin/Native on Raspberry Pi in actions!

Slide 4

Slide 4 text

Rock-Paper-Scissor hand game robot #$%& with Android Things

Slide 5

Slide 5 text

Source:

Slide 6

Slide 6 text

Source:

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Break Down the Project •Data collection tool: raw data for model training •Hardware components: robot arms + game control •Application running on Android Things •Game logic and interactions •Image capture with camera preview •Gesture recognition with TensorFlow

Slide 10

Slide 10 text

Data Collection Tool •Web mobile app built with Node.js + Express.js •Deployed with Firebase Hosting / Function / Storage

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

GDPR? Not compliant…

Slide 14

Slide 14 text

Hardware (Android Things Version) •Android Things Starter Kit - NXP i.MX7D (~200€) •16-Channel PWM/Servo Driver (~12€) •Power Supply or battery (~5€) •5 servomotors (~12€) •Cardboard + wood sticks (~0€) •Glue Gun (~10€) •Electronic jumper + Resistors + LED + Button + Breadboard (~10€)

Slide 15

Slide 15 text

Source: https://developer.android.com/things

Slide 16

Slide 16 text

Source: https://www.adafruit.com/product/815 16-Channel 12-bit PWM/Servo Driver (I2C interface/PCA9685)

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Source:

Slide 19

Slide 19 text

Source: https://commons.wikimedia.org/wiki/File:Android_robot_skateboarding.svg

Slide 20

Slide 20 text

Source: https://android-developers.googleblog.com/2019/02/an-update-on-android-things.html #

Slide 21

Slide 21 text

Source: https://www.iconfinder.com/icons/1172126/android_crying_emoji_mobile_mood_robot_sad_icon

Slide 22

Slide 22 text

Kotlin/Native (Explained to a cat)

Slide 23

Slide 23 text

Source: https://kotlinlang.org/docs/reference/native-overview.html Kotlin/Native is an LLVM based backend for the Kotlin compiler and native implementation of the Kotlin standard library. It compiles Kotlin code to native binaries, which can run without a virtual machine.

Slide 24

Slide 24 text

Source: https://en.wikipedia.org/wiki/Compiler Front End Middle End (Optimizer) Back End Source Code C C++ Java Kotlin … Machine Code ARM Sparc X86
 PowerPC … Compiler Design

Slide 25

Slide 25 text

Source: https://www.aosabook.org/en/llvm.html LLVM Optimizer LLVM X86 Backend LLVM PowerPC Backend LLVM ARM Backend Clang C/C++/ObjC Front End Kotlin Front End Rust Front End LLVM IR LLVM IR LLVM's Implementation of the Three-Phase Design

Slide 26

Slide 26 text

Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR Kotlin/Native as a LLVM Backend LLVM IR 0101 0101 …… IR: Intermediate Representation

Slide 27

Slide 27 text

Source: https://github.com/JetBrains/kotlin-native/tree/master/backend.native IR LLVM IR 0101 0101 …… IR: Intermediate Representation Kotlin/Native as a LLVM Backend

Slide 28

Slide 28 text

Terminology ARM (Acorn RISC Machine) ARM (Advanced RISC Machines) RISC (Reduced Instruction Set Computing) CISC (Complex Instruction Set Computing) MIPS (Millions of Instructions Per Second) x86 ABI (Abstraction Binary Interface) FPU (Floating-point Unit) Source: https://github.com/raspberrypi/firmware

Slide 29

Slide 29 text

Raspberry Pi 3 Model B+ Spec •BCM2837 1.2 GHz 64-bit quad processor based on the ARMv8 Cortex-A53 •32-bit Raspbian version, with a 64-bit version later to come if "there is value in moving to 64-bit mode”

Slide 30

Slide 30 text

My Raspberry Pi 3B armhf: ARM Hard Float

Slide 31

Slide 31 text

Kotlin/Native target presets • Android: androidNativeArm32, androidNativeArm64 • iOS: iosArm32, iosArm64, iosX64 • Linux: linuxArm32Hfp, linuxMips32, linuxMipsel32, linuxX64 • MacOS: macosX64 • Windows: mingwX64 • WebAssembly: wasm32 •ℹ Linux MIPS targets (linuxMips32 and linuxMipsel32) require a Linux host. Other Linux targets can be built on any supported host. Source: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#supported-platforms https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#using-kotlinnative-targets

Slide 32

Slide 32 text

Source: https://en.wikipedia.org/wiki/Cross_compiler Host Machine Cross Compiler Source Code Target Machine Target compatible binary Cross Compilation

Slide 33

Slide 33 text

Source: https://en.wikipedia.org/wiki/Cross_compiler MacBook Pro (macosX64) Cross Compiler Raspberry Pi (linuxArm32Hfp) Target compatible binary Cross Compilation Source Code

Slide 34

Slide 34 text

Key Components •konanc: Kotlin/Native compiler •cinterop: produce kotlin binding for native library Source: https://kotlinlang.org/docs/reference/native/c_interop.html

Slide 35

Slide 35 text

C Interoperability •Create a .def file describing what to include into bindings •Use the cinterop tool to produce Kotlin bindings •Run Kotlin/Native compiler on an application to produce the final executable Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md .def .h cinterop .klib

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

My Setup Macbook + Raspberry Pi 3 (Model B)

Slide 38

Slide 38 text

Hardware (Raspberry Pi Version) •Android Things Starter Kit - NXP i.MX7D (~200€) •Raspberry Pi 3B+ (~40€) + Pi Camera Module V2 (~25€) + Display •…

Slide 39

Slide 39 text

Source:

Slide 40

Slide 40 text

Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C LED GPIO Button GPIO Ground

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

February 2019: Me trying to cross compile Kotlin/Native application on my MacBook for the very first time when there were only 3 articles from 2017 which talk about the subject.

Slide 43

Slide 43 text

exception: java.lang.IllegalStateException: Target linux_arm32_hfp is not available on the macos_x64 host at org.jetbrains.kotlin.backend.konan.KonanConfig.(KonanConfig.kt:47) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:60) at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:35) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:96) at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52) …

Slide 44

Slide 44 text

Source: https://github.com/JetBrains/kotlin-native/releases

Slide 45

Slide 45 text

Step 0: Hello, Kotlin/Native!

Slide 46

Slide 46 text

fun main() { println("Hello Kotlin/Native from Qian’s Macbook!") } Source: https://hadihariri.com/2017/04/09/kotlin-on-raspberry-pi/ hello.kt

Slide 47

Slide 47 text

$ konanc hello.kt -target raspberrypi -o hello $ scp hello.kexe pi@{IP-RASPBERRY-PI}:~/dir/ Compile with command line

Slide 48

Slide 48 text

Source:

Slide 49

Slide 49 text

Step 1: Basic GPIO GPIO (General Purpose Input/Output)

Slide 50

Slide 50 text

Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC LED GPIO Ground

Slide 51

Slide 51 text

pigpio pigpio: a library for the Raspberry which allows control of the General Purpose Input Outputs (GPIO), works on all versions of the Pi. Source: https://github.com/joan2937/pigpio

Slide 52

Slide 52 text

Wiring Pi Wiring Pi: WiringPi is a PIN based GPIO access library written in C for the BCM2835, BCM2836 and BCM2837 SoC devices used in all Raspberry Pi. Source: http://wiringpi.com/

Slide 53

Slide 53 text

Make it work! •Create a pigpio.def file •Generate Kotlin stub for the library •⚠ Build the library on your Raspberry Pi and copy the .so to your Mac for cross compilation •Compile the code by linking the library using linkerOpts option Source: https://hadihariri.com/2017/05/28/raspberry-pi-starter-kit-and-kotlin-native/ / Thank you Hadi Hariri!!

Slide 54

Slide 54 text

pigpio.def headers = pigpio.h

Slide 55

Slide 55 text

$ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub

Slide 56

Slide 56 text

$ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -copt (-compilerOpts) -> additional compiler options -I`pwd`/pigpio -> the location of the header file

Slide 57

Slide 57 text

$ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -o pigpio.bc -> the output file for kotlin stub

Slide 58

Slide 58 text

$ cinterop -def pigpio.def -copt -I`pwd`/pigpio -o pigpio.bc -target raspberrypi Generate Kotlin Stub -target -> the target platform is raspberry pi

Slide 59

Slide 59 text

pigpio: Basics int gpioInitialise(void); int gpioSetMode(unsigned gpio, unsigned mode); int gpioWrite(unsigned gpio, unsigned level); int gpioSleep(unsigned timetype, int seconds, int micros);

Slide 60

Slide 60 text

Led.kt fun blinkLed() { val port = GPIO_LED.toUInt() gpioInitialise() gpioSetMode(port, PI_OUTPUT) println("Start blinking”) while (true) { gpioWrite(port, 0) gpioSleep(PI_TIME_RELATIVE, 0, 500000) gpioWrite(port, 1) gpioSleep(PI_TIME_RELATIVE, 0, 500000) } }

Slide 61

Slide 61 text

$ git clone https://github.com/joan2937/pigpio $ cd pigpio $ make $ sudo make install Build the library on Pi libpigpio.so

Slide 62

Slide 62 text

$ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary

Slide 63

Slide 63 text

$ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary

Slide 64

Slide 64 text

$ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary

Slide 65

Slide 65 text

$ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led Generate the binary

Slide 66

Slide 66 text

Generate the binary led.kexe $ konanc led.kt -target raspberrypi -library pigpio.bc -linker-options "-L`pwd`/pigpio/rpi -lpigpio" -o led

Slide 67

Slide 67 text

Tip 1: Move the project into IntelliJ

Slide 68

Slide 68 text

Move into IntelliJ •Articles are generally out of date •No much documentation at this moment •Samples use the most recent APIs Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md

Slide 69

Slide 69 text

kotlin { targets { fromPreset(kotlin.presets.linuxArm32Hfp, 'chifumi') { compilations.main.outputKinds 'EXECUTABLE' compilations.main.entryPoint 'chifumi.robot.main' compilations.main.cinterops { pigpio { includeDirs 'src/include/pigpio' } } // libpigpio.so are compiled on raspberry pi and copied here compilations.main.linkerOpts '-lpigpio -Lsrc/include/pigpio' } } } build.gradle

Slide 70

Slide 70 text

Tip 2: Deploy on Pi with Gradle SSH plugin

Slide 71

Slide 71 text

Deploy on Pi with Gradle •Cross compilation 0 repetitive scp •Always automate! ✨ •Step 1: passwordless SSH with your Raspberry Pi •Step 2: write a gradle task which does the job •Step 3: 24 Source: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md

Slide 72

Slide 72 text

Source: https://gradle-ssh-plugin.github.io/ task deployOnPi remotes { pi { host = 'address ip of your pi' user = 'pi user' identity = file('/Users/foo/.ssh/id_rsa') } } task deployOnPi { doLast { ssh.run { session(remotes.pi) { def sourceFile = "$buildDir/path/to/main.kexe" def destinationDir = "/path/on/pi" put from: sourceFile, into: destinationDir } } } }

Slide 73

Slide 73 text

Step 2: Button Callback (Mapping Function Pointer from C)

Slide 74

Slide 74 text

Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC Button GPIO Ground

Slide 75

Slide 75 text

•The executable will launch a continuous loop •How do we interact with the application? •With a physical input! •So how can we receive the callback?

Slide 76

Slide 76 text

pigpio: Set Alert // Register a function to be called (a callback) when the specified GPIO changes state. int gpioSetAlertFunc(unsigned user_gpio, gpioAlertFunc_t f); typedef void (*gpioAlertFunc_t) (int gpio, int level, uint32_t tick);

Slide 77

Slide 77 text

Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer Unit>>

Slide 78

Slide 78 text

Source: https://kotlinlang.org/docs/reference/type-aliases.html Generated Kotlin Stub typealias gpioAlertFunc_t = CPointer Unit>> typedef void (*gpioAlertFunc_t) ( int gpio, int level, uint32_t tick );

Slide 79

Slide 79 text

Mapping Function Pointer from C •From typedef to typealias •How to convert a Kotlin function to a pointer to a C function? •staticCFunction(::kotlinFunction) for the rescue! Source: https://kotlinlang.org/docs/tutorials/native/mapping-function-pointers-from-c.html

Slide 80

Slide 80 text

Button Callback val onButtonPressed = staticCFunction { gpio, level, tick -> when (level) { 0 -> println("Button Pressed down, level 0") 1 -> println("Button Released, level 1") 2 -> println("Button GPIO timeout, no level change") } } fun setupButton() { val buttonPort = GPIO_BUTTON.toUInt() gpioSetMode(buttonPort, PI_INPUT) gpioSetAlertFunc(buttonPort, onButtonPressed) }

Slide 81

Slide 81 text

Step 3: Control Servomotors

Slide 82

Slide 82 text

I²C: I-squared-C • I2C bus connects simple peripheral devices with small data payloads • I2C devices connect using a 3-Wire interface consisting of: • Shared clock signal (SCL) • Shared data line (SDA) • Common ground reference (GND) Source: https://developer.android.com/things/sdk/pio/i2c

Slide 83

Slide 83 text

Source: https://www.raspberrypi-spy.co.uk/2014/07/raspberry-pi-b-gpio-header-details-and-pinout/ VCC SDA1 I2C SCL1 I2C Ground

Slide 84

Slide 84 text

Source:

Slide 85

Slide 85 text

PWM: Pulse Width Modulation •Pulse Width Modulation (PWM) is a common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display) •Frequency: expressed in Hz •Period: the time each cycle takes and is the inverse of frequency •Duty cycle: expressed as a percentage Source: https://developer.android.com/things/sdk/pio/pwm

Slide 86

Slide 86 text

Source: https://developer.android.com/things/sdk/pio/pwm

Slide 87

Slide 87 text

In search of the C binding •PCA9685 16-Channel 12 Bit PWM Servo Driver •Adafruit C++ lib => not supported in Kotlin/Native •Small but essential: https://github.com/edlins/ libPCA9685 Source: https://discuss.kotlinlang.org/t/kotlin-native-with-c-libraries/2490/3

Slide 88

Slide 88 text

pigpio: Initialize PWM // Open the I2C bus device and assign the default slave address int PCA9685_openI2C(unsigned char adpt, unsigned char addr); // Initialize a PWM device with the frequency int PCA9685_initPWM(int fd, unsigned char addr, unsigned int freq);

Slide 89

Slide 89 text

PWM Initialisation val adapterNumber = 1 val fileDescriptor = PCA9685_openI2C( adapterNumber.toUByte(), I2C_SERVO_ADDRESS.toUByte() ) PCA9685_initPWM( fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), PWM_FREQUENCY.toUInt() )

Slide 90

Slide 90 text

pigpio: Set PWM Values // Set all PWM channels from two arrays of ON and OFF vals in one transaction int PCA9685_setPWMVals(int fd, unsigned char addr, unsigned int* onVals, unsigned int* offVals);

Slide 91

Slide 91 text

Generate Kotlin Stub CValuesRef? public fun PCA9685_setPWMVals( fd: kotlin.Int, addr: kotlin.UByte, onVals: CValuesRef?, offVals: CValuesRef? ): Int

Slide 92

Slide 92 text

Understand CValuesRef Source: https://kotlinlang.org/api/latest/jvm/stdlib/kotlinx.cinterop/-c-values-ref/index.html

Slide 93

Slide 93 text

Pass Int* to C from Kotlin/Native •Get a NativePlacement instance from the memScoped {…} block receiver •Create an array of UIntVar instance with the allocArray extension function in the NativePlacement instance Source: https://jonnyzzz.com/blog/2019/01/14/kn-intptr/

Slide 94

Slide 94 text

Set PWM On/Off values (1/2) memScoped { val onValues = allocArray(_PCA9685_CHANS) val offValues = allocArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues, offValues) }

Slide 95

Slide 95 text

Oh Wait, Simpler Solution! •${type}Array.toCValues(), where type is the Kotlin primitive type •Array?>.toCValues() •List?>.toCValues() •cValuesOf(vararg elements: ${type}), where type is a primitive or pointer Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md

Slide 96

Slide 96 text

Set PWM On/Off values (2/2) val onValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { onValues[i] = SERVO_ON_ANGLE.toUInt() } val offValues = UIntArray(_PCA9685_CHANS) for (i in 0 until _PCA9685_CHANS - 1) { offValues[i] = SERVO_OFF_ANGLE.toUInt() } PCA9685_setPWMVals(fileDescriptor, I2C_SERVO_ADDRESS.toUByte(), onValues.toCValues(), offValues.toCValues() )

Slide 97

Slide 97 text

Step 4: Capture Image with Pi Camera

Slide 98

Slide 98 text

Understand raspicam •4 Applications are provided: raspistill, raspivid, raspiyuv and raspividyuv •Using MMAL (Multi-Media Abstraction Layer) API which runs over OpenMAX* •No really simple C API as I wanted *OpenMAX: non-proprietary and royalty-free cross-platform set of C-language programming interfaces Source: https://www.raspberrypi.org/documentation/usage/camera/raspicam/

Slide 99

Slide 99 text

Reinvent the wheel? Nope. •POSIX bindings to the rescue •Execute command line from your Kotlin/Native application Source: https://kotlinlang.org/docs/reference/native/platform_libs.html

Slide 100

Slide 100 text

Code platform.posix.system("raspistill -t 2000 -o photo.jpg")

Slide 101

Slide 101 text

Run TensorFlow With Kotlin/Native (on raspberry pi?)

Slide 102

Slide 102 text

TensorFlow •There is a Kotlin/Native sample for TensorFlow! •But only for MacOS 64-bit & Linux 64-bit… •It works okay! 9 Source: https://juliuskunze.com/tensorflow-in-kotlin-native.html

Slide 103

Slide 103 text

Tensorflow for C? •TensorFlow for C is supported on the following systems: •Linux, 64-bit, x86 •MacOS X, Version 10.12.6 (Sierra) or higher •Windows, 64-bit x86 Source: https://www.tensorflow.org/install/lang_c

Slide 104

Slide 104 text

Tensorflow for Java? •TensorFlow for Java is supported on the following systems: •Ubuntu 16.04 or higher; 64-bit, x86 •macOS 10.12.6 (Sierra) or higher •Windows 7 or higher; 64-bit, x86 Source: https://www.tensorflow.org/install/lang_java

Slide 105

Slide 105 text

KEEP CALM AND BUILD TENSORFLOW LITE FROM SOURCE(?)

Slide 106

Slide 106 text

$./tensorflow/lite/tools/make/download_dependencies.sh $./tensorflow/lite/tools/make/build_aarch64_lib.sh Source: https://www.tensorflow.org/lite/guide/build_arm64, https://github.com/tensorflow/tensorflow/pull/29017 Cross Compiling TF Lite

Slide 107

Slide 107 text

No content

Slide 108

Slide 108 text

Demo: Kotlin/Native on Raspberry Pi in actions

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

Conclusion ☕

Slide 111

Slide 111 text

What I learned so far •⚠ Experimental ⚠ • Tip of the iceberg •= C and cross compilation knowledges are essential • Fine tunings are needed •> Always read the doc! • It was fun exploring!

Slide 112

Slide 112 text

References • Talks: • KotlinConf 2018 - Live Coding Kotlin/Native Snake by Dmitry Kandalov • KotlinConf 2018 - Making Noise with Kotlin Native by Josh Skeen • Articles: •Kotlin on Raspberry Pi •Raspberry Pi Starter Kit and Kotlin/Native •Raspberry Pi GPIO with Kotlin Native

Slide 113

Slide 113 text

Thank you! Github Repo: https://github.com/jinqian/kotlin-native-chifumi