Slide 1

Slide 1 text

Let's write a production-ready Kafka Streams app before the end of this talk! Florian Hussonnois, Co-founder, Data Streaming Engineer @StreamThoughts @fhussonnois 1

Slide 2

Slide 2 text

“Streaming technologies, the final frontier. These are the voyages of the Kafka Streams Users. Their 45-minutes mission: to explore strange new words, to seek out new pitfalls and new practices, to boldly go where no developer has gone before.”

Slide 3

Slide 3 text

. Co-founder, Data Streaming Engineer @StreamThoughts Organizer Paris Apache Kafka Meetup Confluent Community Catalyst Apache Kafka Streams contributor Open Source Technology Enthusiastic 3 @fhussonnois About me

Slide 4

Slide 4 text

. 4 Scottify The Starfleet media services provider db-users events-user-activity User Transponder App light speed streaming platform cross-universe topics Our Kafka Streams App KafkaStreams REST API state-full application db-albums What genre of music does each member listen to? (The Federation) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 5

Slide 5 text

// Create StreamsBuilder. StreamsBuilder builder = new StreamsBuilder(); // Consume all user events from input topic event-user-activity. KStream events = builder .stream( "event-user-activity", consumedEarliestWithValueSerde(newJsonSerde(UserEvent.class)) ); // Filter records to only keep events of type MUSIC_LISTEN_START. KStream songListenedEvents = allUserEvents .filter((userId, event) -> event.isOfType(UserEventType.MUSIC_LISTEN_START) .mapValues(SongListenedEvent::parseEventPayload); 5 Consume & Filter User Events 1 Input Record key=031, value={"data":"Damage, Inc; Master of Puppets","userId":"031", "type":"MUSIC_LISTEN_START"}

Slide 6

Slide 6 text

// Previous code is omitted for clarity KStream songListenedEvents = ... // Create all GlobalKTables for albums and users. GlobalKTable albums = createGlobalKTable( builder, "db-albums", "Albums", Album.class); GlobalKTable users = createGlobalKTable( builder, "db-users", "Users", User.class); // Join events with Albums and Users global state stores KGroupedStream> groupedStreams = songListenedEvents .leftJoin(users, (userId, event) -> userId, /*keyValueMapper */ Tuple::of) /*ValueJoiner */ .leftJoin(albums, (userId, tuple) -> tuple.left().album, /*keyValueMapper */ (tuple, album) -> Tuple.of(tuple.right(), album)) /*ValueJoiner */ .groupByKey); 6 Enrich song listened events 2

Slide 7

Slide 7 text

// Previous code is omitted for clarity KGroupedStream> groupedStreams = ... // Do aggregate KTable kTable = groupedStreams .aggregate( UserListenCountPerGenre::new, /* initializer */ /* aggregator */ (userId, tuple, aggregate) -> aggregate.update(tuple.left(), tuple.right()), /* materialized store */ Materialized.as("UserSongsListenedByGenre") .withValueSerde(newJsonSerde(UserListenCountPerGenre.class))) // Streams results to sink topic kTable.toStream() .to( "agg-listened-genres-by-user", Produced.with(Serdes.String(), newJsonSerde(UserListenCountPerGenre.class)) ); 7 Aggregate songs listened by genre 3

Slide 8

Slide 8 text

Topology topology = builder.build(); Properties streamsProps = new Properties(); streamsProps.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-app-id"); streamsProps.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); streamsProps.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class); streamsProps.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class); KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProps); kafkaStreams.start(); Runtime.getRuntime().addShutdownHook(new Thread(kafkaStreams::close)); 8 Running Streams Topology 4 Output Record key=031, value={ "userName": "James Tiberius Kirk", "listenedPerGenre": { "Alternative hip hop": 49, "Rock": 119, "Metal": 34} }

Slide 9

Slide 9 text

Writing a Kafka Streams application is usually not so hard (at least as much as your business logic). But writing an app for production can be MORE COMPLEX 9

Slide 10

Slide 10 text

. ● Handle streams exceptions ● Monitor Kafka Streams states & topic-partitions assignments ● Handle state stores lifecycle and access ● Write streams code to be testable and reusable ● Externalize streams configuration ● Manage properly the initialization of a Kafka Streams instance 10 Some principles to design a Kafka Streams application for production © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 11

Slide 11 text

KafkaStreams.start() “Live long, and prosper process - Spoke” 11

Slide 12

Slide 12 text

. KafkaStreams.start() Deep dive 12 StreamThread-0 StreamThread-1 GlobalStreamThread configuration num.stream.threads=2 ❏ Before starting KafkaStreams creates and configure all internal threads 0 state=CREATED state=CREATED state=CREATED KafkaStreams.state() = CREATED db-albums db-users P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 13

Slide 13 text

. db-albums db-users KafkaStreams.start() Deep dive 13 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ KafkaStreams will start the GlobalStreamThread 1 start() state=CREATED GlobalState Maintainer global-consumer state=CREATED state=CREATED StateRestore Callback GlobalStore KafkaStreams.state() = REBALANCING P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 14

Slide 14 text

. db-users KafkaStreams.start() Deep dive 14 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ Once all global state stores are restored, consumer is assigned to all partitions from all source topics. 1 start() state=RUNNING GlobalState Maintainer GlobalStore global-consumer state=CREATED state=CREATED Processor KafkaStreams.state() = REBALANCING db-albums P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 15

Slide 15 text

. db-albums db-users KafkaStreams.start() Deep dive 15 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ KafkaStreams starts all StreamThreads. ❏ Each consumer subscribes to source topics and starts pollings. 2 start() state=RUNNING GlobalState Maintainer Store global-consumer state=PARTITIONS_ REVOKED state=PARTITIONS_ REVOKED Processor P0 P1 consumer consumer (subscribe) (subscribe) start() consumer-group KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 16

Slide 16 text

. db-albums db-users KafkaStreams.start() Deep dive 16 GlobalStreamThread ❏ Part of rebalancing protocol, each StreamThread is assigned to Tasks (i.e topic-partitions) 2 state=RUNNING GlobalState Maintainer Store global-consumer Processor StreamThread-0 StreamThread-1 state=PARTITIONS_ ASSIGNED state=PARTITIONS_ ASSIGNED P0 P1 consumer consumer (assigned) (assigned) Store Task 0_1 Store Task 0_0 KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 17

Slide 17 text

. KafkaStreams.start() Deep dive 17 GlobalStreamThread ❏ Each Task is restored. ❏ One consumer is dedicated, per Thread, to restore state-stores 3 state=RUNNING GlobalState Maintainer Store global-consumer Processor P0 P1 KafkaStreams.state() = REBALANCING event-user-activity db-albums db-users StreamThread-0 StreamThread-1 state=PARTITIONS_ ASSIGNED state=PARTITIONS_ ASSIGNED consumer consumer (assigned) (assigned) Task 0_1 Task 0_0 Store Store restore-consumer restore-consumer StateRestore Callback StateRestore Callback changelog-store-p0 changelog-store-p1 State Store Recovering © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 18

Slide 18 text

. db-albums db-users KafkaStreams.start() Deep dive 18 GlobalStreamThread ❏ StreamThread are running when all Tasks (active/standby) are restored. 4 state=RUNNING GlobalState Maintainer GlobalStore global-consumer Processor StreamThread-0 StreamThread-1 state=RUNNING state=RUNNING P0 P1 consumer consumer (assigned) (assigned) Store Task 0_1 Store Task 0_0 KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 19

Slide 19 text

KafkaStreams.state() == State.RUNNING (i.e our streams application is now up and running) 19

Slide 20

Slide 20 text

. Issue #1 Messages are NOT processed while state stores are recovering 20 20 ❏ A StreamTask can’t start processing message until all its states stores are fully recovered. ❏ This guarantees the consistency of returned data. source topic last committed offset 1 1 2 changelog topic P0 P0 Task 0 internal restore consumer position 1 1 1 1 ← kv.get( ) This can’t happen ! Availability vs Consistency (state-data) State kv.put( , 2 ) current position

Slide 21

Slide 21 text

. Issue #1 Messages are NOT processed while state stores are recovering 21 21 ❏ A StreamTask can’t start processing message until all its states stores are fully recovered. ❏ This guarantees the consistency of returned data. source topic last committed offset 1 1 2 changelog topic P0 P0 Task 0 internal restore consumer position 1 1 1 1 ← kv.get( ) This can’t happen! Availability vs Consistency (state-data) State kv.put( , 2 ) As a result, long recovering process can significantly increase consumer lags

Slide 22

Slide 22 text

. Monitoring State Store Recovering Using a Global listener 22 StateRestoreListener listener = new StateRestoreListener() { @Override public void onRestoreStart( TopicPartition topicPartition, String storeName, long startingOffset, long endingOffset) { } @Override public void onBatchRestored( TopicPartition topicPartition, String storeName, long batchEndOffset, long numRestored) { } @Override public void onRestoreEnd( TopicPartition topicPartition, String storeName, long totalRestored) { } }; streams.setGlobalStateRestoreListener(globalListener); Listen to all state stores restoration © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 23

Slide 23 text

. Monitoring State Store Recovering Using Consumer Metrics Can help to monitor recovering performance : MBean: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+),topic=([-.w]+) The average number of : ❏ bytes consumed per second for a topic: bytes-consumed-rate ❏ bytes fetched per request for a topic: fetch-size-avg ❏ records in each request for a topic: records-per-request-avg ❏ records consumed per second for a topic: records-consumed-rate 23 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 24

Slide 24 text

. Issue #2 Things may be blocking StreamsBuilder builder = new StreamsBuilder(); //... builder.globalTable("input-topic"); Topology topology = builder.build(); new KafkaStreams(topology, streamsConfig).start(); Depending on your code, this can block your entire application, and perhaps lead to an application crash (or timeout). 24 https://issues.apache.org/jira/browse/KAFKA-7380 This can actually block! © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 25

Slide 25 text

. Issue #2 ...a better way final KafkaStreams streams = new KafkaStreams(topology, streamsConfig); ExecutorService executor = Executors.newSingleThreadExecutor(r -> { final Thread thread = new Thread(r, "streams-starter"); thread.setDaemon(false); return thread; }); CompletableFuture.supplyAsync(() -> { streams.start(); return streams.state(); }, executor); 25 Set as user-thread, indeed internal StreamsThreads will inherit from this one. © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 26

Slide 26 text

How to turn our application into distributed queryable KV store ? “Change is the essential process of all existence. — Spock” 26

Slide 27

Slide 27 text

. Interactive Queries In 30 seconds 27 ❏ IQ allows direct access to local states ❏ Read-only access ❏ Simplifies architecture by removing external DB needs StreamThread-0 StreamThread-1 consumer consumer Store Task 0_1 Store Task 0_0 KafkaStreams API User Service API UI REST © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 28

Slide 28 text

. Interactive Queries In 30 seconds 28 StreamThread-0 StreamThread-1 consumer consumer Store Task 0_1 Store Task 0_0 KafkaStreams API User Service API UI 28 ReadOnlyKeyValueStore store = streams.store( "Album", QueryableStoreTypes.keyValueStore() ); Album value = store.get("Ok Computer"); ❏ Simple API that works for DSL and Processor API

Slide 29

Slide 29 text

. Issue #1 State stores can’t be queried while recovering 29 29 BAD streams.start(); ReadOnlyKeyValueStore store; while (true) { try { store = streams.store( "Albums", QueryableStoreTypes.keyValueStore() ); break; } catch (InvalidStateStoreException ignored) { // wait...store not ready yet Time.SYSTEM.sleep(Duration.ofSeconds(1)); } } // do something useful with the store… ❏ Developers have to check for state-store availability ⚠ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore ❏ State may have migrated

Slide 30

Slide 30 text

. Issue #1 State stores can’t be queried while recovering 30 30 OK streams.start(); // Global state store can be queried from here if (streams.state().isRunning() /* rebalancing*/ ) { ReadOnlyKeyValueStore store = streams.store( “Albums” QueryableStoreTypes.keyValueStore() ); // do something useful with the store... } // Wait for all StreamThreads to be ready while (streams.state() != KafkaStreams.State.RUNNING) { Time.SYSTEM.sleep(Duration.ofSeconds(1)); } // all local state stores are now recovered ❏ Developers have to check for state-store availability ⚠ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore ❏ State may have migrated

Slide 31

Slide 31 text

. Issue #1 State stores can’t be queried while recovering 31 31 OK streams.start(); // Global state store can be queried from here if (streams.state().isRunning() /* rebalancing*/ ) { ReadOnlyKeyValueStore store = streams.store( "globalStoreName", QueryableStoreTypes.keyValueStore() ); // do something useful with the store... } // Wait for all StreamThreads to be ready while (streams.state() != KafkaStreams.State.RUNNING) { Time.SYSTEM.sleep(Duration.ofSeconds(1)); } // all local state stores are now recovered ❏ Developers have to check for state-store availability ❏ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore This only works for single streams instance!

Slide 32

Slide 32 text

. Scaling our Application Distributed States 32 StreamThread-0 StreamThread-0 P0 P1 consumer consumer Store Task 0_1 Task 0_0 source topic K V Kirk Metal Saru Classical Store Spock Rock Picard Electro K V ❏ Tasks are spread across instances/threads ❏ Each instance own a state-store sub-set application.server= localhost:8080 application.server= localhost:8082 Instance 1 (JVM) Instance 2 (JVM) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 33

Slide 33 text

. Issue #2 Discovery API 33 33 As developers you will have to : ❏ Query local instance for metadata. ❏ Discover which instance has the data you are looking for. ❏ Query either local or remote state store. Caveats: A lot of boilerplate code But bad things may happens... // Discover which instance hosts the the key-value StreamsMetadata metadata = streams.metadataForKey( "UserSongsListenedByGenre", "Picard", Serdes.String().serializer() ); // Check if key-value is hosted by the local instance if (isLocalHost(metadata.hostInfo())) { ReadOnlyKeyValueStore store = kafkstreamsaStreams.store( "UserSongsListenedByGenre", QueryableStoreTypes.keyValueStore() ); store.get("Picard"); } else forwardInteractiveQuery(metadata.hostInfo());

Slide 34

Slide 34 text

. Issue #2 Scaling Up or Down 34 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro Instance 2 (JVM) StreamThread-0 application.server= localhost:8082 STARTING consumer © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 35

Slide 35 text

. Issue #2 Scaling Up or Down 35 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock application.server= localhost:8082 Task 0_1 NEW Task Migration Recovering... © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 36

Slide 36 text

. © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent. Issue #2 Scaling Up or Down 36 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock application.server= localhost:8082 Task 0_1 NEW Task Migration Recovering... Code will throw an InvalidStateStoreException (usually transient failure)

Slide 37

Slide 37 text

. Issue #3 Instance Failure 37 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 crash / network outage © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 38

Slide 38 text

. Issue #3 Instance Failure 38 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 session.timeout.ms (default 10 seconds) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 39

Slide 39 text

. Issue #3 Instance Failure 39 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 java.net.ConnectException (while state is not re-assigned) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 40

Slide 40 text

40 Your app must handle state stores unavailability, not the client querying it! 40

Slide 41

Slide 41 text

Handling Exceptions “It is possible to commit no mistakes and still lose. That is not weakness, that is life.” – Jean-Luc Picard 41

Slide 42

Slide 42 text

. 42 Unexpected Messages or how to lose your app... ! ! ! db_users event_user_activity User Transponder App The bad guys KafkaStreams REST API Our application ! ! db_albums (Klingon Empire) (The Federation) ! DeserializationException © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 43

Slide 43 text

. Solution #1 Skip or Fail Built-in mechanisms 43 default.deserialization.exception.handler ❏ CONTINUE: continue with processing ❏ FAIL: fail the processing and stop Two available implementations : ❏ LogAndContinueExceptionHandler ❏ LogAndFailExceptionHandler Not really suitable for production. Cannot monitor efficiently corrupted messages © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 44

Slide 44 text

. Solution #2 Dead Letter Topic 44 44 Solution #3 Sentinel Value DeserializationExceptionHandler Deserializer ! ! ! ! Handler ? Source Topic Topology ! ! Source Topic SafeDeserializer Inner Deserializer (null)(null) Catch any exception thrown during deserialization and return a default value (e.g: null, “N/A”, etc). (skip) Dead Letter topic Send corrupted messages to a special topic.

Slide 45

Slide 45 text

. Best Practices How to send corrupted messages ❏ Never change the schema/format of the corrupted message. ❏ Send the original message as it is in the DLT. ❏ Use Kafka Headers to trace exception cause and origin. 45 Kafka Message raw key raw value original topic / partition / offset exception trace app info (id, host, version) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 46

Slide 46 text

Ok, Let's summarize! 46

Slide 47

Slide 47 text

. Business Value vs Effort 47 Topology Definition Business Value High Kafka Streams Management IQ Error Handling logic Monitoring / Health-check Security Configuration Externalization Low Effort Low/Medium High Streams Lifecycle Kafka Streams Application © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 48

Slide 48 text

. Business Value vs Effort 48 Topology Definition Business Value High Kafka Streams Management IQ Error Handling logic Monitoring / Health-check Security Configuration Externalization Low Effort Low/Medium High Streams Lifecycle Kafka Streams Application Eventually, sooner or later, you'll write your own Kafka Streams framework to wrap all that stuff! © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 49

Slide 49 text

Consistency Without common practices and libraries, teams have to (re)write new classes for handling errors, Kafka Streams startup and Interactive Queries for each project , etc 49 Updatability Kafka Streams is evolving fast, with new features, bug fixes and optimizations. Maintaining multiple applications up-to-date with the latest version can be challenging. We never build a single Kafka Streams apps, but dozens each running across multiple instances. Operability Kafka Streams can be challenging for operations teams. Make easy to monitor streams instance using standard API, across all projects, will help to keep the system running smoothly.

Slide 50

Slide 50 text

50 (Azkarra is Basque word for "Fast") 50

Slide 51

Slide 51 text

. Yet Another Micro-Framework a lightweight framework that makes easy to create production-ready Kafka Streams applications. ❏ Open-source since 2019 November under Apache-2.0 License. ❏ Written in Java. Add a star to the GitHub project, it only takes 5 seconds ^^ 51 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 52

Slide 52 text

. Key Features Developer Friendly Production-Ready Secured Easy to learn API Configuration Externalization REST APIs for Interactive Queries SSL/TLS, Basic Authentication Built-in Healthchecks REST Endpoints for Metrics (JSON / Prometheus) Error handling Logics 52

Slide 53

Slide 53 text

. How to use It ? 53 io.streamthoughts azkarra-streams 0.6.1 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 54

Slide 54 text

. Concept TopologyProvider 54 54 public interface TopologyProvider extends Provider { /** * Supplies a new Kafka Streams {@link Topology} * instance. * * @return the {@link Topology} instance. */ Topology get(); /** * Returns the version of the supplied * {@link Topology}. * * @return the string version. */ @Override String version(); } A simple interface to implement ❏ Each topology must be versioned. ❏ Use to provide the Topology instance. 1

Slide 55

Slide 55 text

. Concept StreamsExecutionEnvironment 55 // Configure environment specific properties Conf config = … // Create a new environment to manage lifecycle of one or many KafkaStreams instances. StreamsExecutionEnvironment env = DefaultStreamsExecutionEnvironment.create("dev-env") .setConfiguration(config) .setKafkaStreamsFactory(/**/) .setRocksDBConfig(RocksDBConfig.withStatsEnable()) .setApplicationIdBuilder(/**/) .addGlobalStateListener(/**/) .addStateListener(/**/) // Register the topology env.addTopology(CountUserListenMusicPerGenreTopology::new, Executed.as("scottify-streams")); 2 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 56

Slide 56 text

. Concept StreamsLifecycleInterceptor 56 public interface StreamsLifecycleInterceptor { /** * This method is executed before starting the streams instance. */ default void onStart(StreamsLifecycleContext context, StreamsLifecycleChain chain) { chain.execute(); } /** * This method is executed before stopping the streams instance. */ default void onStop(StreamsLifecycleContext context, StreamsLifecycleChain chain) { chain.execute(); } } AutoCreateTopicsInterceptor WaitForSourceTopicsInterceptor Built-in interceptors 3 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 57

Slide 57 text

. Concept AzkarraContext 57 Registering Custom ExecutionEnvironment context.addExecutionEnvironment(env) Using a default ExecutionEnvironment context.addTopology( CountUserListenMusicPerGenreTopology.class, "dev-env" Executed.as("scottify-streams") ) // Configure context specific properties Conf ctxConf = … AzkarraContext context = DefaultAzkarraContext.create(ctxConf) 4 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 58

Slide 58 text

. Concept AzkarraApplication 58 //Provide configuration externalization (using Lightbend Config library). Conf appConf = AzkarraConf.create("application") new AzkarraApplication() .setConfiguration(appConf) .setContext(context) .setBannerMode(Banner.Mode.OFF) .enableHttpServer(true) .setRegisterShutdownHook(true) .run(args); 58 Provides additional features on top-off AzkarraContext : ❏ Embedded HTTP Server ❏ Component Scan (scan current package for classes annotated @Component - e.g: TopologyProvider) ❏ Application auto-configuration 5

Slide 59

Slide 59 text

. Interactive Queries HTTP Endpoint 59 Request $ POST /api/v1/applications/:application_id /stores/:store_name { "set_options": { "query_timeout_ms": 1000, "retries": 100, "retry_backoff_ms": 100, "remote_access_allowed": true }, "type": "key_value", "query": { "get": { "key": "001" } } } 59 Response { "took": 1, "timeout": false, "server": "localhost:8080", "status": "SUCCESS", "total": 1, "result": { "success": [ { "server": "localhost:8080", "remote": false, "records": [ { "key": "001", "value": { "name": "James Tiberius Kirk", "gender": "Male", "species": "Human", "key": "001" } } ] ...

Slide 60

Slide 60 text

DEMO

Slide 61

Slide 61 text

. Azkarra WebUI 61

Slide 62

Slide 62 text

. Azkarra WebUI 62

Slide 63

Slide 63 text

. Azkarra WebUI 63

Slide 64

Slide 64 text

. Additional Resources Code Source ❏ https://github.com/streamthoughts/demo-kafka-streams-scottify Azkarra Streams ❏ https://streamthoughts.github.io/azkarra-streams/ ❏ https://medium.com/streamthoughts/introducing-azkarra-streams-the-first-micro-framework-for-apache-kafka-streams-e1 3605f3a3a6 ❏ https://dev.to/fhussonnois/create-kafka-streams-applications-faster-than-ever-before-via-azkarra-streams-3nng 64 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 65

Slide 65 text

. Images & Icons Images Photo by Bryan Goff on Unsplash Photo by Stefan Cosma on Unsplash Photo by Jordan Whitfield on Unsplash Icons https://en.wikipedia.org/wiki/Starfleet https://en.wikipedia.org/wiki/Klingon 65 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.

Slide 66

Slide 66 text

Thank you @streamthoughts www.streamthoughts.io [email protected]