Stateful & Reactive Streaming Apps without Databases @MeetUp OpenBlend SLO 2019

as presented at OpenBlend - Slovenian Java User Group

Abstract:

Time and again we should move out of our comfort zone and take the opportunity to experiment with new ways to build applications. Based on an easy to understand example we will look at a different, for some of us unconventional and radical way to build modern data-centric applications. For that purpose, we are going to discuss a stateful streaming application on top of Apache Kafka and integrate with Spring Boot 2.0 in order to provide a reactive WebAPI which allows clients to consume data changes in near real-time. All of this without explicitly using or managing an external database.

GitHub Repository: https://github.com/hpgrahsl/meetup-openblend-slo-2019

Transcript

1. Stateful & Reactive Streaming Apps without Databases Kafka Streams ❤ Spring Boot @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia
2. $ whoami • Hans-Peter Grahsl • working & living in Graz • technical trainer at • independent consultant & engineer • associate lecturer • " irregular conference speaker @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 2
3. challenges in today's data architectures • rising number of apps producing + consuming data • need to integrate ever more data sources • heterogeneous environments all over the place • traditional technologies may struggle to cope with this @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 3
4. challenges may lead to a GIANT MESS @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 4
5. Apache Kafka @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 5
6. STREAMING PLATFORM @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 6
7. much more than messaging • Apache Kafka is offering 3 key capabilities • publish / subscribe to streams of records • (permanently) store streams of records • process streams of records in near real-time fault-tolerance & horizontal scalability @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 7
8. Producer API @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 8
9. Consumer API @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 9
10. Connect API @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 10
11. Streams API @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 11
12. Kafka Streams API • stream processing with a library only approach • lightweight applications • build however & deploy wherever you like • NO(!) additional clusters or frameworks e.g. • Processor API & Streams DSL • configurable delivery guarantees @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 12
13. writing applications NOT (!) managing clusters @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 13
14. KSQL • SQL only (not embedded) • NO(!) coding skills required • extremely low entry barrier • familiar syntax and semantics • concise and expressive • joins, aggregations, windowing • UD(A)Fs and UDTFs still pending... • built on top of Kafka Streams @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 14
15. skyrocketing developer productivity @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 15
16. liberating streaming for the masses @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 16
17. @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 17
18. central nervous system @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 18
19. example ? hmmmm... @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 19
21. example: near real-time Emoji Tracking
22. HOW TO build this? @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 22
23. emoji tracking | step 1 store ingest subset of public live tweets from Twitter @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 23
24. emoji tracking | step 2 process extract emojis - group & count them - maintain top N @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 24
25. emoji tracking | step 3 query single emoji count - all emoji counts - top N emojis @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 25
26. emoji tracking | step 4 notify consumable near-realtime change streams of updates @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 26
27. Let's do it! @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 27
29. example: step 1 ingest tweets • using Kafka Connect • e.g. this community connector https://github.com/jcustenborder/kafka- connect-twitter • configure the connector (JSON) • manage connector via REST-like API create | pause | resume | delete | status @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 29
30. { "name": "tweets-twitter-source", "config": { "connector.class": "c.g.j.k.c.t.TwitterSourceConnector", "twitter.oauth.accessToken": "...", "twitter.oauth.consumerSecret": ...", "twitter.oauth.consumerKey": "...", "twitter.oauth.accessTokenSecret": "...", "kafka.status.topic": "tweets", "process.deletes": false, "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": false, "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "filter.keywords": "..." } } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 30
32. ! ! ! ! NO CODE! ! ! ! ! @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 32
33. example: step 2 process tweets • using Kafka Streams high-level DSL • grouping and counting emojis • updating top N emoji counts • map tweets to emoji occurrences • only a few lines of Java @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 33
34. calculate emoji counts • It all starts with tweets like this... ! this is a twitter status " text with ## five emojis @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 34
35. calculate emoji counts Key Value raw input ID ! this is a twitter ! status " text with ## five emojis extract emoji list ID [!,!,",#,#] @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 35
36. calculate emoji counts Key Value flatten the list ID ! ID ! ID " ID # ID # @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 36
37. calculate emoji counts Key Value set keys to values ! "" ! "" " "" # "" # "" finally group & count by key @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 37
38. result: continuously updated KTable with emoji counts Key Value ! 2 " 1 # 2 ... ... @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 38
39. 1:1 mapping to KStreams API KTable<String, Long> emojiCounts = tweets.map((id,tweet) -> KeyValue.pair(id, EmojiUtils...)) .flatMapValues(emojis -> emojis) .map((id,emoji) -> KeyValue.pair(emoji, "")) .groupByKey(...).count(...); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia
41. example: step 3 query results • access to state stores with interactive queries • KStreams offers all needed metadata • ! RPC integration left for developers > Reactive WebAPI powered by Spring Boot 2.0 < @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 41
42. REST controller @RestController @RequestMapping("interactive/queries/") @CrossOrigin(origins = "*") public class StateStoreController { private final StateStoreService service; [...] } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 42
43. REST controller methods @GetMapping("emojis/{code}") public Mono<ResponseEntity<EmojiCount>> getEmoji(@PathVariable String code) { return service.querySingleEmojiCount(code); } @GetMapping("emojis") public Flux<EmojiCount> getEmojis() { return service.queryAllEmojiCounts(); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 43
44. state store access in service StreamsMetadata metadata = kafkaStreams.metadataForKey( "your-store-name", emoji, Serializer... ); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 44
45. state store access in service if(itsMe.equals(metadata.hostInfo())) { ReadOnlyKeyValueStore<String,Long> kvStoreEmojiCounts = kafkaStreams.store("your-store-name", QueryableStoreTypes.keyValueStore()); Long count = kvStoreEmojiCounts.get(emoji); return Mono.just( new ResponseEntity<>(new EmojiCount(...),HttpStatus.OK) ); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 45
46. state store access in service String location = String.format("http://%s:%d/.../%s", metadata.host(),metadata.port(),emoji); return Mono.just( ResponseEntity.status(HttpStatus.FOUND) .location(URI.create(location)).build() ); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 46
49. example: step 4 real-time notifications • reactively consume from changelog topics • stream any changes to clients using SSE > Project Reactor's reactor-kafka < @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 49
50. notifications via SSE @GetMapping(path = "emojis/updates/notify", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<EmojiCount> getEmojiCountsStream() { return service.consumeEmojiCountsStream(); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 50
51. ! LIVE ! DASHBOARD
52. mission accomplished @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 52
53. source https://github.com/hpgrahsl/meetup-openblend-slo-2019 slides https://speakerdeck.com/hpgrahsl/stateful-and-reactive- streaming-apps-without-databases-at-meetup-openblend- slo-2019 @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 53
54. THANKS Q&A?