Stateful & Reactive Streaming Applications Without a Database

Time and again we should move out of our comfort zone and take the opportunity to experiment with new ways to build applications. Based on a simple to understand example we will look at a different, for some of us unconventional and radical way to build modern data-centric applications. For that purpose, we are going to write a stateful streaming application on top of Apache Kafka and integrate with Spring Boot 2.0 in order to provide a reactive WebAPI which allows clients to consume data changes in near real-time. All of this without explicitly using or managing an external traditional database.
The session aims to contribute towards a better understanding of how to architect and implement continuous, reactive applications on top of new technology stacks. It's particularly interesting for developers and architects but in general targeting
anyone audacious enough to take part in this whirlwind tour.

* Find the recording of this talk on YouTube
https://www.youtube.com/watch?v=5V4GR3ZgdMY

* Find the full source code used for the example on GitHub
https://github.com/hpgrahsl/wearedevs-2018

Transcript

1. Stateful & Reactive Streaming Applications without a Database Apache Kafka Streams ❤ Spring Boot 2.0
2. Who is this guy? • Hans-Peter Grahsl • living and working in Graz, Austria • technical trainer at Netconomy • independent consultant & engineer • associate lecturer • highly irregular speaker :) Twitter @hpgrahsl https://github.com/hpgrahsl @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May
3. Apache Kafka @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 3
4. much more than messaging • Apache Kafka is offering 3 key capabilities • publish / subscribe to streams of records • (permanently) store streams of records • process streams of records in near real-time > horizontally scalable & fault-tolerant < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 4
5. Producer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 5
6. Consumer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 6
7. Connect API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 7
8. Streams API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 8
9. @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 9
10. STREAMING PLATFORM @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 10
11. Kafka Streams API • stream processing with a library only approach • lightweight Java applications • NO(!) external streaming frameworks or additional clusters needed • e.g. Strom, Spark, Flink, Samza, ... • Processor API (low-level) & Streams DSL (high-level) • configurable delivery semantics / guarantees @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 11
12. writing code NOT (!) managing clusters @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 12
13. KSQL: newest kid on the block • open source SQL streaming engine • extremely low entry barrier • declarative (mostly ANSI) SQL-like language and semantics • very expressive: no coding required • works on top of KStream API • joins, aggregations, windowing • UD(A)Fs @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 13
14. example ? !! ...hmmm... @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 14
16. example: near real-time EMOJI TRACKING
17. What's needed? @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 17
18. emoji tracking | step 1 store ingest a subset of public live tweets via Twitter Stream API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 18
19. emoji tracking | step 2 process extract emojis, group & count them, maintain top N @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 19
20. emoji tracking | step 3 query single emoji count, all emoji counts, top N emojis @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 20
21. emoji tracking | step 4 notify consumable near-realtime change streams of emoji counts @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 21
22. Let's do it! @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 22
24. example: step 1 ingest tweets • make use of Kafka Connect • e.g. this community connector https://github.com/jcustenborder/kafka-connect-twitter • configure the connector (JSON) • manage connector via REST API create | pause | resume | delete | status @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 24
25. { "name": "tweets-twitter-source", "config": { "connector.class": "c.g.j.k.c.t.TwitterSourceConnector", "twitter.oauth.accessToken": "...", "twitter.oauth.consumerSecret": ...", "twitter.oauth.consumerKey": "...", "twitter.oauth.accessTokenSecret": "...", "kafka.status.topic": "tweets", "process.deletes": false, "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": false, "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "filter.keywords": "..." } } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 25
27. ! ! ! Look ... No Code! ! ! ! @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 27
28. example: step 2 process tweets • make use of high-level streaming DSL • group emojis and count them • update top N emoji counts • map tweets to emoji occurrences • only a few lines of Java @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 28
29. calculate emoji counts It all starts with raw data... ! this is some twitter " status text including EMOJIS @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 29
30. calculate emoji counts Operation Key Value ID ! this is some twitter " status text including EMOJIS # # .map(...) ID [ !, ", #, # ] @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 30
31. calculate emoji counts Operation Key Value .flatMapValues(...) ID ! ID " ID # ID # @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 31
32. calculate emoji counts Operation Key Value .map(...) ! " " " " " # " " # " " .groupByKey(...).count(...) @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 32
33. result: continuously updated KTable holding emoji counts Key Value ... ... ! 1 " 1 # 2 ... ... @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 33
34. maps 1:1 to KStreams API KTable<String, Long> emojiCounts = tweets.map((id,tweet) -> KeyValue.pair(id, EmojiUtils...)) .flatMapValues(emojis -> emojis) .map((id,emoji) -> KeyValue.pair(emoji, "")) .groupByKey(...) .count(...); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 34
36. example: step 3 query results • access to state stores with interactive queries • KStreams offers all needed metadata • ! RPC integration is left as developer exercise > Reactive WebAPI powered by Spring Boot 2.0 < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 36
37. rest controller @RestController @RequestMapping("interactive/queries/") @CrossOrigin(origins = "*") public class StateStoreController { private final StateStoreService service; //... } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 37
38. rest controller methods @GetMapping("emojis/{code}") public Mono<ResponseEntity<EmojiCount>> getEmoji(@PathVariable String code) { return service.querySingleEmojiCount(code); } @GetMapping("emojis") public Flux<EmojiCount> getEmojis() { return service.queryAllEmojiCounts(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 38
39. state store access in service //GET STREAMS METADATA FOR KEY i.e. EMOJI StreamsMetadata metadata = kafkaStreams.metadataForKey( "your-store-name", emoji, Serializer... ); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 39
40. state store access in service //CHECK IF STATE IS LOCALLY AVAILABLE AND SERVE RESULT if(itsMe.equals(metadata.hostInfo())) { ReadOnlyKeyValueStore<String,Long> kvStoreEmojiCounts = kafkaStreams.store("your-store-name", QueryableStoreTypes.keyValueStore()); Long count = kvStoreEmojiCounts.get(emoji); return Mono.just( new ResponseEntity<>(new EmojiCount(...),HttpStatus.OK) ); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 40
41. state store access in service //OTHERWISE REDIRECT CLIENT String location = String.format("http://%s:%d/.../%s", metadata.host(),metadata.port(),emoji); return Mono.just(ResponseEntity.status(HttpStatus.FOUND) .location(URI.create(location)).build()); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 41
44. example: step 4 real-time notifications • reactively consume from Kafka changelog topics • stream results to clients using server-sent-events > Project Reactor's reactor-kafka < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 44
45. notifications via SSE @GetMapping(path = "emojis/updates/notify", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<EmojiCount> getEmojiCountsStream() { return service.consumeEmojiCountsStream(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 45
46. ! LIVE DASHBOARD
47. ! mission accomplished @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 47
48. session materials source https://github.com/hpgrahsl/wearedevs-2018 slides https://speakerdeck.com/hpgrahsl/wearedevs-2018 @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 48
49. THANK YOU Q&A ? @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 49