Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stateful & Reactive Streaming Applications Without a Database

Stateful & Reactive Streaming Applications Without a Database

Time and again we should move out of our comfort zone and take the opportunity to experiment with new ways to build applications. Based on a simple to understand example we will look at a different, for some of us unconventional and radical way to build modern data-centric applications. For that purpose, we are going to write a stateful streaming application on top of Apache Kafka and integrate with Spring Boot 2.0 in order to provide a reactive WebAPI which allows clients to consume data changes in near real-time. All of this without explicitly using or managing an external traditional database.
The session aims to contribute towards a better understanding of how to architect and implement continuous, reactive applications on top of new technology stacks. It's particularly interesting for developers and architects but in general targeting
anyone audacious enough to take part in this whirlwind tour.

* Find the recording of this talk on YouTube
https://www.youtube.com/watch?v=5V4GR3ZgdMY

* Find the full source code used for the example on GitHub
https://github.com/hpgrahsl/wearedevs-2018

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

May 17, 2018
Tweet

Transcript

  1. Stateful & Reactive Streaming Applications without a Database Apache Kafka

    Streams ❤ Spring Boot 2.0
  2. Who is this guy? • Hans-Peter Grahsl • living and

    working in Graz, Austria • technical trainer at Netconomy • independent consultant & engineer • associate lecturer • highly irregular speaker :) Twitter @hpgrahsl https://github.com/hpgrahsl @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May
  3. Apache Kafka @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 3
  4. much more than messaging • Apache Kafka is offering 3

    key capabilities • publish / subscribe to streams of records • (permanently) store streams of records • process streams of records in near real-time > horizontally scalable & fault-tolerant < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 4
  5. Producer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 5
  6. Consumer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 6
  7. Connect API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 7
  8. Streams API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 8
  9. @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16

    - 18 May 9
  10. STREAMING PLATFORM @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 10
  11. Kafka Streams API • stream processing with a library only

    approach • lightweight Java applications • NO(!) external streaming frameworks or additional clusters needed • e.g. Strom, Spark, Flink, Samza, ... • Processor API (low-level) & Streams DSL (high-level) • configurable delivery semantics / guarantees @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 11
  12. writing code NOT (!) managing clusters @hpgrahsl | #WeAreDevelopers World

    Congress 2018, Austria Center Vienna 16 - 18 May 12
  13. KSQL: newest kid on the block • open source SQL

    streaming engine • extremely low entry barrier • declarative (mostly ANSI) SQL-like language and semantics • very expressive: no coding required • works on top of KStream API • joins, aggregations, windowing • UD(A)Fs @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 13
  14. example ? !! ...hmmm... @hpgrahsl | #WeAreDevelopers World Congress 2018,

    Austria Center Vienna 16 - 18 May 14
  15. None
  16. example: near real-time EMOJI TRACKING

  17. What's needed? @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center

    Vienna 16 - 18 May 17
  18. emoji tracking | step 1 store ingest a subset of

    public live tweets via Twitter Stream API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 18
  19. emoji tracking | step 2 process extract emojis, group &

    count them, maintain top N @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 19
  20. emoji tracking | step 3 query single emoji count, all

    emoji counts, top N emojis @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 20
  21. emoji tracking | step 4 notify consumable near-realtime change streams

    of emoji counts @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 21
  22. Let's do it! @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria

    Center Vienna 16 - 18 May 22
  23. None
  24. example: step 1 ingest tweets • make use of Kafka

    Connect • e.g. this community connector https://github.com/jcustenborder/kafka-connect-twitter • configure the connector (JSON) • manage connector via REST API create | pause | resume | delete | status @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 24
  25. { "name": "tweets-twitter-source", "config": { "connector.class": "c.g.j.k.c.t.TwitterSourceConnector", "twitter.oauth.accessToken": "...", "twitter.oauth.consumerSecret":

    ...", "twitter.oauth.consumerKey": "...", "twitter.oauth.accessTokenSecret": "...", "kafka.status.topic": "tweets", "process.deletes": false, "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": false, "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "filter.keywords": "..." } } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 25
  26. None
  27. ! ! ! Look ... No Code! ! ! !

    @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 27
  28. example: step 2 process tweets • make use of high-level

    streaming DSL • group emojis and count them • update top N emoji counts • map tweets to emoji occurrences • only a few lines of Java @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 28
  29. calculate emoji counts It all starts with raw data... !

    this is some twitter " status text including EMOJIS @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 29
  30. calculate emoji counts Operation Key Value ID ! this is

    some twitter " status text including EMOJIS # # .map(...) ID [ !, ", #, # ] @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 30
  31. calculate emoji counts Operation Key Value .flatMapValues(...) ID ! ID

    " ID # ID # @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 31
  32. calculate emoji counts Operation Key Value .map(...) ! " "

    " " " # " " # " " .groupByKey(...).count(...) @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 32
  33. result: continuously updated KTable holding emoji counts Key Value ...

    ... ! 1 " 1 # 2 ... ... @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 33
  34. maps 1:1 to KStreams API KTable<String, Long> emojiCounts = tweets.map((id,tweet)

    -> KeyValue.pair(id, EmojiUtils...)) .flatMapValues(emojis -> emojis) .map((id,emoji) -> KeyValue.pair(emoji, "")) .groupByKey(...) .count(...); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 34
  35. None
  36. example: step 3 query results • access to state stores

    with interactive queries • KStreams offers all needed metadata • ! RPC integration is left as developer exercise > Reactive WebAPI powered by Spring Boot 2.0 < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 36
  37. rest controller @RestController @RequestMapping("interactive/queries/") @CrossOrigin(origins = "*") public class StateStoreController

    { private final StateStoreService service; //... } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 37
  38. rest controller methods @GetMapping("emojis/{code}") public Mono<ResponseEntity<EmojiCount>> getEmoji(@PathVariable String code) {

    return service.querySingleEmojiCount(code); } @GetMapping("emojis") public Flux<EmojiCount> getEmojis() { return service.queryAllEmojiCounts(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 38
  39. state store access in service //GET STREAMS METADATA FOR KEY

    i.e. EMOJI StreamsMetadata metadata = kafkaStreams.metadataForKey( "your-store-name", emoji, Serializer... ); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 39
  40. state store access in service //CHECK IF STATE IS LOCALLY

    AVAILABLE AND SERVE RESULT if(itsMe.equals(metadata.hostInfo())) { ReadOnlyKeyValueStore<String,Long> kvStoreEmojiCounts = kafkaStreams.store("your-store-name", QueryableStoreTypes.keyValueStore()); Long count = kvStoreEmojiCounts.get(emoji); return Mono.just( new ResponseEntity<>(new EmojiCount(...),HttpStatus.OK) ); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 40
  41. state store access in service //OTHERWISE REDIRECT CLIENT String location

    = String.format("http://%s:%d/.../%s", metadata.host(),metadata.port(),emoji); return Mono.just(ResponseEntity.status(HttpStatus.FOUND) .location(URI.create(location)).build()); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 41
  42. None
  43. None
  44. example: step 4 real-time notifications • reactively consume from Kafka

    changelog topics • stream results to clients using server-sent-events > Project Reactor's reactor-kafka < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 44
  45. notifications via SSE @GetMapping(path = "emojis/updates/notify", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public

    Flux<EmojiCount> getEmojiCountsStream() { return service.consumeEmojiCountsStream(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 45
  46. ! LIVE DASHBOARD

  47. ! mission accomplished @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria

    Center Vienna 16 - 18 May 47
  48. session materials source https://github.com/hpgrahsl/wearedevs-2018 slides https://speakerdeck.com/hpgrahsl/wearedevs-2018 @hpgrahsl | #WeAreDevelopers World

    Congress 2018, Austria Center Vienna 16 - 18 May 48
  49. THANK YOU Q&A ? @hpgrahsl | #WeAreDevelopers World Congress 2018,

    Austria Center Vienna 16 - 18 May 49
  50. None