Stateful & Reactive Streaming Applications without a Database Apache Kafka Streams ❤ Spring Boot 2.0 

Who is this guy? • Hans-Peter Grahsl • living and working in Graz, Austria • technical trainer at Netconomy • independent consultant & engineer • associate lecturer • highly irregular speaker :) Twitter @hpgrahsl https://github.com/hpgrahsl @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 

Apache Kafka @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 3 

much more than messaging • Apache Kafka is offering 3 key capabilities • publish / subscribe to streams of records • (permanently) store streams of records • process streams of records in near real-time > horizontally scalable & fault-tolerant < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 4 

Producer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 5 

Consumer API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 6 

Connect API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 7 

Streams API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 8 

@hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 9 

STREAMING PLATFORM @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 10 

Kafka Streams API • stream processing with a library only approach • lightweight Java applications • NO(!) external streaming frameworks or additional clusters needed • e.g. Strom, Spark, Flink, Samza, ... • Processor API (low-level) & Streams DSL (high-level) • configurable delivery semantics / guarantees @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 11 

writing code NOT (!) managing clusters @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 12 

KSQL: newest kid on the block • open source SQL streaming engine • extremely low entry barrier • declarative (mostly ANSI) SQL-like language and semantics • very expressive: no coding required • works on top of KStream API • joins, aggregations, windowing • UD(A)Fs @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 13 

example ? !! ...hmmm... @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 14 

No content

example: near real-time EMOJI TRACKING 

What's needed? @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 17 

emoji tracking | step 1 store ingest a subset of public live tweets via Twitter Stream API @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 18 

emoji tracking | step 2 process extract emojis, group & count them, maintain top N @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 19 

emoji tracking | step 3 query single emoji count, all emoji counts, top N emojis @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 20 

emoji tracking | step 4 notify consumable near-realtime change streams of emoji counts @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 21 

Let's do it! @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 22 

No content

example: step 1 ingest tweets • make use of Kafka Connect • e.g. this community connector https://github.com/jcustenborder/kafka-connect-twitter • configure the connector (JSON) • manage connector via REST API create | pause | resume | delete | status @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 24 

{ "name": "tweets-twitter-source", "config": { "connector.class": "c.g.j.k.c.t.TwitterSourceConnector", "twitter.oauth.accessToken": "...", "twitter.oauth.consumerSecret": ...", "twitter.oauth.consumerKey": "...", "twitter.oauth.accessTokenSecret": "...", "kafka.status.topic": "tweets", "process.deletes": false, "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": false, "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": false, "filter.keywords": "..." } } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 25 

No content

! ! ! Look ... No Code! ! ! ! @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 27 

example: step 2 process tweets • make use of high-level streaming DSL • group emojis and count them • update top N emoji counts • map tweets to emoji occurrences • only a few lines of Java @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 28 

calculate emoji counts It all starts with raw data... ! this is some twitter " status text including EMOJIS @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 29 

calculate emoji counts Operation Key Value ID ! this is some twitter " status text including EMOJIS # # .map(...) ID [ !, ", #, # ] @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 30 

calculate emoji counts Operation Key Value .flatMapValues(...) ID ! ID " ID # ID # @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 31 

calculate emoji counts Operation Key Value .map(...) ! " " " " " # " " # " " .groupByKey(...).count(...) @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 32 

result: continuously updated KTable holding emoji counts Key Value ... ... ! 1 " 1 # 2 ... ... @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 33 

maps 1:1 to KStreams API KTable emojiCounts = tweets.map((id,tweet) -> KeyValue.pair(id, EmojiUtils...)) .flatMapValues(emojis -> emojis) .map((id,emoji) -> KeyValue.pair(emoji, "")) .groupByKey(...) .count(...); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 34 

No content

example: step 3 query results • access to state stores with interactive queries • KStreams offers all needed metadata • ! RPC integration is left as developer exercise > Reactive WebAPI powered by Spring Boot 2.0 < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 36 

rest controller @RestController @RequestMapping("interactive/queries/") @CrossOrigin(origins = "*") public class StateStoreController { private final StateStoreService service; //... } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 37 

rest controller methods @GetMapping("emojis/{code}") public Mono> getEmoji(@PathVariable String code) { return service.querySingleEmojiCount(code); } @GetMapping("emojis") public Flux getEmojis() { return service.queryAllEmojiCounts(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 38 

state store access in service //GET STREAMS METADATA FOR KEY i.e. EMOJI StreamsMetadata metadata = kafkaStreams.metadataForKey( "your-store-name", emoji, Serializer... ); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 39 

state store access in service //CHECK IF STATE IS LOCALLY AVAILABLE AND SERVE RESULT if(itsMe.equals(metadata.hostInfo())) { ReadOnlyKeyValueStore kvStoreEmojiCounts = kafkaStreams.store("your-store-name", QueryableStoreTypes.keyValueStore()); Long count = kvStoreEmojiCounts.get(emoji); return Mono.just( new ResponseEntity<>(new EmojiCount(...),HttpStatus.OK) ); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 40 

state store access in service //OTHERWISE REDIRECT CLIENT String location = String.format("http://%s:%d/.../%s", metadata.host(),metadata.port(),emoji); return Mono.just(ResponseEntity.status(HttpStatus.FOUND) .location(URI.create(location)).build()); @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 41 

No content

No content

example: step 4 real-time notifications • reactively consume from Kafka changelog topics • stream results to clients using server-sent-events > Project Reactor's reactor-kafka < @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 44 

notifications via SSE @GetMapping(path = "emojis/updates/notify", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux getEmojiCountsStream() { return service.consumeEmojiCountsStream(); } @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 45 

! LIVE DASHBOARD 

! mission accomplished @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 47 

session materials source https://github.com/hpgrahsl/wearedevs-2018 slides https://speakerdeck.com/hpgrahsl/wearedevs-2018 @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 48 

THANK YOU Q&A ? @hpgrahsl | #WeAreDevelopers World Congress 2018, Austria Center Vienna 16 - 18 May 49 

No content