Stream Processors in Kotlin - KotlinConf 2018 edition

My talk at the KotlinConf 2018 (https://kotlinconf.com/schedule/#date=5-october&session=41150).

Excerpt:
At trivago, we use Apache Kafka a lot. We recently migrated a good part of our Kafka-related Java code to Kotlin. In this talk we focus on how extension functions and the closure syntax make things pretty when using Kafka Streams.

Find the code to the talk here: https://github.com/xenji/kotlinconf-2018-iot-code

Transcript

1. Writing Stream Processors in Kotlin KotlinConf 2018 - Amsterdam
2. Who of you uses ?
3. A quick look over the shoulder 3
4. Kafka in a Nutshell Source: Wikipedia
5. 5 What is so special about Kafka? • Persistent • Time and Log compaction • Guaranteed order per partition
6. Messages • Have a byte[] key • Have a byte[] value • Contain Metadata like a timestamp and the partition
7. Image from http://kafka.apache.org/documentation/
8. Event Streams • null-key and a bare or serialized value • Compaction of the topic by time • Examples: Clickstreams, Logs, IoT events
9. Image from http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html Log Compaction
10. Stream Processors • Imagine you could do Kotlin Sequences on an infinite list of messages • Stateful • Parallel • And the output is one or more Kafka topic(s)
11. What now? • I do home automation, a lot • I use homee, which aggregates multiple protocols (Z-Wave, ZigBee, etc) • It has no official API, yet • But … They (unintentionally) made one of their repos public on Github • And now I have an API Blueprint file ☺ • Guess what? It’s a websocket!
12. Let’s get started • The websocket issues text messages when something happens • Attribute Changes of any connected device (motion, temperature, light, door opens, etc.) • System Events (user created, system update, etc.) • We use Kafka Connect to build what is called a “source” and pipe these events into Kafka.
13. Let’s look at some code (no live demo, sorry)
14. Now, why Kotlin? The Java code did not look that bad…
15. An engineering view • Less verbose, better to read • Extension functions enable composable, high level abstractions • Extensions can be called from the Java folks • One can write libraries with business abstractions, reusable within the company, that blend into the original API
16. A team leads view • Onboarding people coming from non-JVM languages to Kotlin is easier than Java • The education material available is very rich, based on current teaching standards • People love the language and the feeling of productivity • Unmodifiable / Immutable helps a lot of people to write more stable code, • resulting in better outcome and higher motivation to go on with learning • because “it just works es expected” • You can write good code even without knowing the language in-depth
17. Kotlin at trivago • More than 80% of our Kafka related source code is Kotlin • Kafka Connect Sinks, Transforms, Converters • Stream Processors • Custom Solutions, based on Spring Boot 2, Spring Kafka, Spring Integration • My current team writes client facing REST and GRPC services based on Spring Boot 2 entirely in Kotlin • Our recently relaunched Android App is written entirely in Kotlin • Some of our B2B focused teams started to rewrite their PHP applications in Kotlin
19. Curious about how we use Apache Kafka? Kafka and Debezium at trivago (code.talks 2017) https://speakerdeck.com/xenji/kafka-and-debezium-at-trivago-code-dot-talks-2017-edition