Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stateful & Reactive Streaming Apps without Databases @MeetUp OpenBlend SLO 2019

Stateful & Reactive Streaming Apps without Databases @MeetUp OpenBlend SLO 2019

as presented at OpenBlend - Slovenian Java User Group

Abstract:

Time and again we should move out of our comfort zone and take the opportunity to experiment with new ways to build applications. Based on an easy to understand example we will look at a different, for some of us unconventional and radical way to build modern data-centric applications. For that purpose, we are going to discuss a stateful streaming application on top of Apache Kafka and integrate with Spring Boot 2.0 in order to provide a reactive WebAPI which allows clients to consume data changes in near real-time. All of this without explicitly using or managing an external database.

GitHub Repository: https://github.com/hpgrahsl/meetup-openblend-slo-2019

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

February 21, 2019
Tweet

Transcript

  1. Stateful & Reactive Streaming Apps without Databases Kafka Streams ❤

    Spring Boot @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia
  2. $ whoami • Hans-Peter Grahsl • working & living in

    Graz • technical trainer at • independent consultant & engineer • associate lecturer • " irregular conference speaker @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 2
  3. challenges in today's data architectures • rising number of apps

    producing + consuming data • need to integrate ever more data sources • heterogeneous environments all over the place • traditional technologies may struggle to cope with this @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 3
  4. challenges may lead to a GIANT MESS @hpgrahsl | OpenBlend

    Java User Group, 21st Feb. 2019, Slovenia 4
  5. Apache Kafka @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 5
  6. STREAMING PLATFORM @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 6
  7. much more than messaging • Apache Kafka is offering 3

    key capabilities • publish / subscribe to streams of records • (permanently) store streams of records • process streams of records in near real-time fault-tolerance & horizontal scalability @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 7
  8. Producer API @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 8
  9. Consumer API @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 9
  10. Connect API @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 10
  11. Streams API @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 11
  12. Kafka Streams API • stream processing with a library only

    approach • lightweight applications • build however & deploy wherever you like • NO(!) additional clusters or frameworks e.g. • Processor API & Streams DSL • configurable delivery guarantees @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 12
  13. writing applications NOT (!) managing clusters @hpgrahsl | OpenBlend Java

    User Group, 21st Feb. 2019, Slovenia 13
  14. KSQL • SQL only (not embedded) • NO(!) coding skills

    required • extremely low entry barrier • familiar syntax and semantics • concise and expressive • joins, aggregations, windowing • UD(A)Fs and UDTFs still pending... • built on top of Kafka Streams @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 14
  15. skyrocketing developer productivity @hpgrahsl | OpenBlend Java User Group, 21st

    Feb. 2019, Slovenia 15
  16. liberating streaming for the masses @hpgrahsl | OpenBlend Java User

    Group, 21st Feb. 2019, Slovenia 16
  17. @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia

    17
  18. central nervous system @hpgrahsl | OpenBlend Java User Group, 21st

    Feb. 2019, Slovenia 18
  19. example ? hmmmm... @hpgrahsl | OpenBlend Java User Group, 21st

    Feb. 2019, Slovenia 19
  20. None
  21. example: near real-time Emoji Tracking

  22. HOW TO build this? @hpgrahsl | OpenBlend Java User Group,

    21st Feb. 2019, Slovenia 22
  23. emoji tracking | step 1 store ingest subset of public

    live tweets from Twitter @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 23
  24. emoji tracking | step 2 process extract emojis - group

    & count them - maintain top N @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 24
  25. emoji tracking | step 3 query single emoji count -

    all emoji counts - top N emojis @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 25
  26. emoji tracking | step 4 notify consumable near-realtime change streams

    of updates @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 26
  27. Let's do it! @hpgrahsl | OpenBlend Java User Group, 21st

    Feb. 2019, Slovenia 27
  28. None
  29. example: step 1 ingest tweets • using Kafka Connect •

    e.g. this community connector https://github.com/jcustenborder/kafka- connect-twitter • configure the connector (JSON) • manage connector via REST-like API create | pause | resume | delete | status @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 29
  30. { "name": "tweets-twitter-source", "config": { "connector.class": "c.g.j.k.c.t.TwitterSourceConnector", "twitter.oauth.accessToken": "...", "twitter.oauth.consumerSecret":

    ...", "twitter.oauth.consumerKey": "...", "twitter.oauth.accessTokenSecret": "...", "kafka.status.topic": "tweets", "process.deletes": false, "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": false, "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "filter.keywords": "..." } } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 30
  31. None
  32. ! ! ! ! NO CODE! ! ! ! !

    @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 32
  33. example: step 2 process tweets • using Kafka Streams high-level

    DSL • grouping and counting emojis • updating top N emoji counts • map tweets to emoji occurrences • only a few lines of Java @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 33
  34. calculate emoji counts • It all starts with tweets like

    this... ! this is a twitter status " text with ## five emojis @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 34
  35. calculate emoji counts Key Value raw input ID ! this

    is a twitter ! status " text with ## five emojis extract emoji list ID [!,!,",#,#] @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 35
  36. calculate emoji counts Key Value flatten the list ID !

    ID ! ID " ID # ID # @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 36
  37. calculate emoji counts Key Value set keys to values !

    "" ! "" " "" # "" # "" finally group & count by key @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 37
  38. result: continuously updated KTable with emoji counts Key Value !

    2 " 1 # 2 ... ... @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 38
  39. 1:1 mapping to KStreams API KTable<String, Long> emojiCounts = tweets.map((id,tweet)

    -> KeyValue.pair(id, EmojiUtils...)) .flatMapValues(emojis -> emojis) .map((id,emoji) -> KeyValue.pair(emoji, "")) .groupByKey(...).count(...); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia
  40. None
  41. example: step 3 query results • access to state stores

    with interactive queries • KStreams offers all needed metadata • ! RPC integration left for developers > Reactive WebAPI powered by Spring Boot 2.0 < @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 41
  42. REST controller @RestController @RequestMapping("interactive/queries/") @CrossOrigin(origins = "*") public class StateStoreController

    { private final StateStoreService service; [...] } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 42
  43. REST controller methods @GetMapping("emojis/{code}") public Mono<ResponseEntity<EmojiCount>> getEmoji(@PathVariable String code) {

    return service.querySingleEmojiCount(code); } @GetMapping("emojis") public Flux<EmojiCount> getEmojis() { return service.queryAllEmojiCounts(); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 43
  44. state store access in service StreamsMetadata metadata = kafkaStreams.metadataForKey( "your-store-name",

    emoji, Serializer... ); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 44
  45. state store access in service if(itsMe.equals(metadata.hostInfo())) { ReadOnlyKeyValueStore<String,Long> kvStoreEmojiCounts =

    kafkaStreams.store("your-store-name", QueryableStoreTypes.keyValueStore()); Long count = kvStoreEmojiCounts.get(emoji); return Mono.just( new ResponseEntity<>(new EmojiCount(...),HttpStatus.OK) ); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 45
  46. state store access in service String location = String.format("http://%s:%d/.../%s", metadata.host(),metadata.port(),emoji);

    return Mono.just( ResponseEntity.status(HttpStatus.FOUND) .location(URI.create(location)).build() ); @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 46
  47. None
  48. None
  49. example: step 4 real-time notifications • reactively consume from changelog

    topics • stream any changes to clients using SSE > Project Reactor's reactor-kafka < @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 49
  50. notifications via SSE @GetMapping(path = "emojis/updates/notify", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public

    Flux<EmojiCount> getEmojiCountsStream() { return service.consumeEmojiCountsStream(); } @hpgrahsl | OpenBlend Java User Group, 21st Feb. 2019, Slovenia 50
  51. ! LIVE ! DASHBOARD

  52. mission accomplished @hpgrahsl | OpenBlend Java User Group, 21st Feb.

    2019, Slovenia 52
  53. source https://github.com/hpgrahsl/meetup-openblend-slo-2019 slides https://speakerdeck.com/hpgrahsl/stateful-and-reactive- streaming-apps-without-databases-at-meetup-openblend- slo-2019 @hpgrahsl | OpenBlend Java

    User Group, 21st Feb. 2019, Slovenia 53
  54. THANKS Q&A?