Let's write a production-ready Kafka Streams app before the end of this talk!

While it's easy to get started with Kafka Streams, building a streaming application with the minimal features required for going into production is usually another story! If you plan to build a complete event-driven architecture based on many Kafka Streams microservices, you will have to know; how to handle processing failures and bad records, how to query Kafka Streams states stores, how to monitor and operate instances... Yes, it's starting to do a lot of things, doesn’t it? And sooner or later, you will probably build and maintain in-house libraries to standardize all that stuff across your projects.

In this talk, I propose to show you how to easily build a Kafka Streams application for production in just a few minutes. But before that,
we’ll explore some commons practices used to develop Kafka Streams applications. We'll review the things you have to be careful while developing. Then, I will introduce Azkarra Streams, an open-source lightweight Java framework that lets you focus on writing topologies code that matters for your business, not boilerplate code for running them!

Transcript

1. Let's write a production-ready Kafka Streams app before the end of this talk! Florian Hussonnois, Co-founder, Data Streaming Engineer @StreamThoughts @fhussonnois 1
2. “Streaming technologies, the final frontier. These are the voyages of the Kafka Streams Users. Their 45-minutes mission: to explore strange new words, to seek out new pitfalls and new practices, to boldly go where no developer has gone before.”
3. . Co-founder, Data Streaming Engineer @StreamThoughts Organizer Paris Apache Kafka Meetup Confluent Community Catalyst Apache Kafka Streams contributor Open Source Technology Enthusiastic 3 @fhussonnois About me
4. . 4 Scottify The Starfleet media services provider db-users events-user-activity User Transponder App light speed streaming platform cross-universe topics Our Kafka Streams App KafkaStreams REST API state-full application db-albums What genre of music does each member listen to? (The Federation) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
5. // Create StreamsBuilder. StreamsBuilder builder = new StreamsBuilder(); // Consume all user events from input topic event-user-activity. KStream<String, UserEvent> events = builder .stream( "event-user-activity", consumedEarliestWithValueSerde(newJsonSerde(UserEvent.class)) ); // Filter records to only keep events of type MUSIC_LISTEN_START. KStream<String, SongListenedEvent> songListenedEvents = allUserEvents .filter((userId, event) -> event.isOfType(UserEventType.MUSIC_LISTEN_START) .mapValues(SongListenedEvent::parseEventPayload); 5 Consume & Filter User Events 1 Input Record key=031, value={"data":"Damage, Inc; Master of Puppets","userId":"031", "type":"MUSIC_LISTEN_START"}
6. // Previous code is omitted for clarity KStream<String, SongListenedEvent> songListenedEvents = ... // Create all GlobalKTables for albums and users. GlobalKTable<String, Album> albums = createGlobalKTable( builder, "db-albums", "Albums", Album.class); GlobalKTable<String, User> users = createGlobalKTable( builder, "db-users", "Users", User.class); // Join events with Albums and Users global state stores KGroupedStream<String, Tuple<User, Album>> groupedStreams = songListenedEvents .leftJoin(users, (userId, event) -> userId, /*keyValueMapper */ Tuple::of) /*ValueJoiner */ .leftJoin(albums, (userId, tuple) -> tuple.left().album, /*keyValueMapper */ (tuple, album) -> Tuple.of(tuple.right(), album)) /*ValueJoiner */ .groupByKey); 6 Enrich song listened events 2
7. // Previous code is omitted for clarity KGroupedStream<String, Tuple<User, Album>> groupedStreams = ... // Do aggregate KTable<String, UserListenedSongsByGenre> kTable = groupedStreams .aggregate( UserListenCountPerGenre::new, /* initializer */ /* aggregator */ (userId, tuple, aggregate) -> aggregate.update(tuple.left(), tuple.right()), /* materialized store */ Materialized.as("UserSongsListenedByGenre") .withValueSerde(newJsonSerde(UserListenCountPerGenre.class))) // Streams results to sink topic kTable.toStream() .to( "agg-listened-genres-by-user", Produced.with(Serdes.String(), newJsonSerde(UserListenCountPerGenre.class)) ); 7 Aggregate songs listened by genre 3
8. Topology topology = builder.build(); Properties streamsProps = new Properties(); streamsProps.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-app-id"); streamsProps.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); streamsProps.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class); streamsProps.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class); KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProps); kafkaStreams.start(); Runtime.getRuntime().addShutdownHook(new Thread(kafkaStreams::close)); 8 Running Streams Topology 4 Output Record key=031, value={ "userName": "James Tiberius Kirk", "listenedPerGenre": { "Alternative hip hop": 49, "Rock": 119, "Metal": 34} }
9. Writing a Kafka Streams application is usually not so hard (at least as much as your business logic). But writing an app for production can be MORE COMPLEX 9
10. . • Handle streams exceptions • Monitor Kafka Streams states & topic-partitions assignments • Handle state stores lifecycle and access • Write streams code to be testable and reusable • Externalize streams configuration • Manage properly the initialization of a Kafka Streams instance 10 Some principles to design a Kafka Streams application for production © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
11. KafkaStreams.start() “Live long, and prosper process - Spoke” 11
12. . KafkaStreams.start() Deep dive 12 StreamThread-0 StreamThread-1 GlobalStreamThread configuration num.stream.threads=2 ❏ Before starting KafkaStreams creates and configure all internal threads 0 state=CREATED state=CREATED state=CREATED KafkaStreams.state() = CREATED db-albums db-users P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
13. . db-albums db-users KafkaStreams.start() Deep dive 13 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ KafkaStreams will start the GlobalStreamThread 1 start() state=CREATED GlobalState Maintainer global-consumer state=CREATED state=CREATED StateRestore Callback GlobalStore KafkaStreams.state() = REBALANCING P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
14. . db-users KafkaStreams.start() Deep dive 14 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ Once all global state stores are restored, consumer is assigned to all partitions from all source topics. 1 start() state=RUNNING GlobalState Maintainer GlobalStore global-consumer state=CREATED state=CREATED Processor KafkaStreams.state() = REBALANCING db-albums P0 P1 event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
15. . db-albums db-users KafkaStreams.start() Deep dive 15 StreamThread-0 StreamThread-1 GlobalStreamThread ❏ KafkaStreams starts all StreamThreads. ❏ Each consumer subscribes to source topics and starts pollings. 2 start() state=RUNNING GlobalState Maintainer Store global-consumer state=PARTITIONS_ REVOKED state=PARTITIONS_ REVOKED Processor P0 P1 consumer consumer (subscribe) (subscribe) start() consumer-group KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
16. . db-albums db-users KafkaStreams.start() Deep dive 16 GlobalStreamThread ❏ Part of rebalancing protocol, each StreamThread is assigned to Tasks (i.e topic-partitions) 2 state=RUNNING GlobalState Maintainer Store global-consumer Processor StreamThread-0 StreamThread-1 state=PARTITIONS_ ASSIGNED state=PARTITIONS_ ASSIGNED P0 P1 consumer consumer (assigned) (assigned) Store Task 0_1 Store Task 0_0 KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
17. . KafkaStreams.start() Deep dive 17 GlobalStreamThread ❏ Each Task is restored. ❏ One consumer is dedicated, per Thread, to restore state-stores 3 state=RUNNING GlobalState Maintainer Store global-consumer Processor P0 P1 KafkaStreams.state() = REBALANCING event-user-activity db-albums db-users StreamThread-0 StreamThread-1 state=PARTITIONS_ ASSIGNED state=PARTITIONS_ ASSIGNED consumer consumer (assigned) (assigned) Task 0_1 Task 0_0 Store Store restore-consumer restore-consumer StateRestore Callback StateRestore Callback changelog-store-p0 changelog-store-p1 State Store Recovering © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
18. . db-albums db-users KafkaStreams.start() Deep dive 18 GlobalStreamThread ❏ StreamThread are running when all Tasks (active/standby) are restored. 4 state=RUNNING GlobalState Maintainer GlobalStore global-consumer Processor StreamThread-0 StreamThread-1 state=RUNNING state=RUNNING P0 P1 consumer consumer (assigned) (assigned) Store Task 0_1 Store Task 0_0 KafkaStreams.state() = REBALANCING event-user-activity © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
19. KafkaStreams.state() == State.RUNNING (i.e our streams application is now up and running) 19
20. . Issue #1 Messages are NOT processed while state stores are recovering 20 20 ❏ A StreamTask can’t start processing message until all its states stores are fully recovered. ❏ This guarantees the consistency of returned data. source topic last committed offset 1 1 2 changelog topic P0 P0 Task 0 internal restore consumer position 1 1 1 1 ← kv.get( ) This can’t happen ! Availability vs Consistency (state-data) State kv.put( , 2 ) current position
21. . Issue #1 Messages are NOT processed while state stores are recovering 21 21 ❏ A StreamTask can’t start processing message until all its states stores are fully recovered. ❏ This guarantees the consistency of returned data. source topic last committed offset 1 1 2 changelog topic P0 P0 Task 0 internal restore consumer position 1 1 1 1 ← kv.get( ) This can’t happen! Availability vs Consistency (state-data) State kv.put( , 2 ) As a result, long recovering process can significantly increase consumer lags
22. . Monitoring State Store Recovering Using a Global listener 22 StateRestoreListener listener = new StateRestoreListener() { @Override public void onRestoreStart( TopicPartition topicPartition, String storeName, long startingOffset, long endingOffset) { } @Override public void onBatchRestored( TopicPartition topicPartition, String storeName, long batchEndOffset, long numRestored) { } @Override public void onRestoreEnd( TopicPartition topicPartition, String storeName, long totalRestored) { } }; streams.setGlobalStateRestoreListener(globalListener); Listen to all state stores restoration © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
23. . Monitoring State Store Recovering Using Consumer Metrics Can help to monitor recovering performance : MBean: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+),topic=([-.w]+) The average number of : ❏ bytes consumed per second for a topic: bytes-consumed-rate ❏ bytes fetched per request for a topic: fetch-size-avg ❏ records in each request for a topic: records-per-request-avg ❏ records consumed per second for a topic: records-consumed-rate 23 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
24. . Issue #2 Things may be blocking StreamsBuilder builder = new StreamsBuilder(); //... builder.globalTable("input-topic"); Topology topology = builder.build(); new KafkaStreams(topology, streamsConfig).start(); Depending on your code, this can block your entire application, and perhaps lead to an application crash (or timeout). 24 https://issues.apache.org/jira/browse/KAFKA-7380 This can actually block! © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
25. . Issue #2 ...a better way final KafkaStreams streams = new KafkaStreams(topology, streamsConfig); ExecutorService executor = Executors.newSingleThreadExecutor(r -> { final Thread thread = new Thread(r, "streams-starter"); thread.setDaemon(false); return thread; }); CompletableFuture.supplyAsync(() -> { streams.start(); return streams.state(); }, executor); 25 Set as user-thread, indeed internal StreamsThreads will inherit from this one. © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
26. How to turn our application into distributed queryable KV store ? “Change is the essential process of all existence. — Spock” 26
27. . Interactive Queries In 30 seconds 27 ❏ IQ allows direct access to local states ❏ Read-only access ❏ Simplifies architecture by removing external DB needs StreamThread-0 StreamThread-1 consumer consumer Store Task 0_1 Store Task 0_0 KafkaStreams API User Service API UI REST © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
28. . Interactive Queries In 30 seconds 28 StreamThread-0 StreamThread-1 consumer consumer Store Task 0_1 Store Task 0_0 KafkaStreams API User Service API UI 28 ReadOnlyKeyValueStore<String, Album> store = streams.store( "Album", QueryableStoreTypes.keyValueStore() ); Album value = store.get("Ok Computer"); ❏ Simple API that works for DSL and Processor API
29. . Issue #1 State stores can’t be queried while recovering 29 29 BAD streams.start(); ReadOnlyKeyValueStore<String, Album> store; while (true) { try { store = streams.store( "Albums", QueryableStoreTypes.keyValueStore() ); break; } catch (InvalidStateStoreException ignored) { // wait...store not ready yet Time.SYSTEM.sleep(Duration.ofSeconds(1)); } } // do something useful with the store… ❏ Developers have to check for state-store availability ⚠ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore ❏ State may have migrated
30. . Issue #1 State stores can’t be queried while recovering 30 30 OK streams.start(); // Global state store can be queried from here if (streams.state().isRunning() /* rebalancing*/ ) { ReadOnlyKeyValueStore<String, Long> store = streams.store( “Albums” QueryableStoreTypes.keyValueStore() ); // do something useful with the store... } // Wait for all StreamThreads to be ready while (streams.state() != KafkaStreams.State.RUNNING) { Time.SYSTEM.sleep(Duration.ofSeconds(1)); } // all local state stores are now recovered ❏ Developers have to check for state-store availability ⚠ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore ❏ State may have migrated
31. . Issue #1 State stores can’t be queried while recovering 31 31 OK streams.start(); // Global state store can be queried from here if (streams.state().isRunning() /* rebalancing*/ ) { ReadOnlyKeyValueStore<String, Long> store = streams.store( "globalStoreName", QueryableStoreTypes.keyValueStore() ); // do something useful with the store... } // Wait for all StreamThreads to be ready while (streams.state() != KafkaStreams.State.RUNNING) { Time.SYSTEM.sleep(Duration.ofSeconds(1)); } // all local state stores are now recovered ❏ Developers have to check for state-store availability ❏ Please don’t use infinite loop to wait for a state store to be ready. Caveats: ❏ StreamThread could be DEAD ❏ Bad store name is provided ❏ Useless for GlobalStateStore This only works for single streams instance!
32. . Scaling our Application Distributed States 32 StreamThread-0 StreamThread-0 P0 P1 consumer consumer Store Task 0_1 Task 0_0 source topic K V Kirk Metal Saru Classical Store Spock Rock Picard Electro K V ❏ Tasks are spread across instances/threads ❏ Each instance own a state-store sub-set application.server= localhost:8080 application.server= localhost:8082 Instance 1 (JVM) Instance 2 (JVM) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
33. . Issue #2 Discovery API 33 33 As developers you will have to : ❏ Query local instance for metadata. ❏ Discover which instance has the data you are looking for. ❏ Query either local or remote state store. Caveats: A lot of boilerplate code But bad things may happens... // Discover which instance hosts the the key-value StreamsMetadata metadata = streams.metadataForKey( "UserSongsListenedByGenre", "Picard", Serdes.String().serializer() ); // Check if key-value is hosted by the local instance if (isLocalHost(metadata.hostInfo())) { ReadOnlyKeyValueStore<String, Long> store = kafkstreamsaStreams.store( "UserSongsListenedByGenre", QueryableStoreTypes.keyValueStore() ); store.get("Picard"); } else forwardInteractiveQuery(metadata.hostInfo());
34. . Issue #2 Scaling Up or Down 34 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro Instance 2 (JVM) StreamThread-0 application.server= localhost:8082 STARTING consumer © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
35. . Issue #2 Scaling Up or Down 35 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock application.server= localhost:8082 Task 0_1 NEW Task Migration Recovering... © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
36. . © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent. Issue #2 Scaling Up or Down 36 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Task 0_1 Store K V Kirk Metal Saru Classical Spock Rock Picard Electro StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock application.server= localhost:8082 Task 0_1 NEW Task Migration Recovering... Code will throw an InvalidStateStoreException (usually transient failure)
37. . Issue #3 Instance Failure 37 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 crash / network outage © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
38. . Issue #3 Instance Failure 38 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 session.timeout.ms (default 10 seconds) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
39. . Issue #3 Instance Failure 39 StreamThread-0 P0 P1 consumer source topic Instance 1 (JVM) application.server= localhost:8080 Task 0_0 Store K V Kirk Metal Saru Classical StreamThread-0 consumer Instance 2 (JVM) changelog topic Store K V Spock Rock Picard Electro application.server= localhost:8082 Task 0_1 java.net.ConnectException (while state is not re-assigned) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
40. 40 Your app must handle state stores unavailability, not the client querying it! 40
41. Handling Exceptions “It is possible to commit no mistakes and still lose. That is not weakness, that is life.” – Jean-Luc Picard 41
42. . 42 Unexpected Messages or how to lose your app... ! ! ! db_users event_user_activity User Transponder App The bad guys KafkaStreams REST API Our application ! ! db_albums (Klingon Empire) (The Federation) ! DeserializationException © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
43. . Solution #1 Skip or Fail Built-in mechanisms 43 default.deserialization.exception.handler ❏ CONTINUE: continue with processing ❏ FAIL: fail the processing and stop Two available implementations : ❏ LogAndContinueExceptionHandler ❏ LogAndFailExceptionHandler Not really suitable for production. Cannot monitor efficiently corrupted messages © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
44. . Solution #2 Dead Letter Topic 44 44 Solution #3 Sentinel Value DeserializationExceptionHandler Deserializer<T> ! ! ! ! Handler ? Source Topic Topology ! ! Source Topic SafeDeserializer Inner Deserializer (null)(null) Catch any exception thrown during deserialization and return a default value (e.g: null, “N/A”, etc). (skip) Dead Letter topic Send corrupted messages to a special topic.
45. . Best Practices How to send corrupted messages ❏ Never change the schema/format of the corrupted message. ❏ Send the original message as it is in the DLT. ❏ Use Kafka Headers to trace exception cause and origin. 45 Kafka Message raw key raw value original topic / partition / offset exception trace app info (id, host, version) © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
46. Ok, Let's summarize! 46
47. . Business Value vs Effort 47 Topology Definition Business Value High Kafka Streams Management IQ Error Handling logic Monitoring / Health-check Security Configuration Externalization Low Effort Low/Medium High Streams Lifecycle Kafka Streams Application © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
48. . Business Value vs Effort 48 Topology Definition Business Value High Kafka Streams Management IQ Error Handling logic Monitoring / Health-check Security Configuration Externalization Low Effort Low/Medium High Streams Lifecycle Kafka Streams Application Eventually, sooner or later, you'll write your own Kafka Streams framework to wrap all that stuff! © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
49. Consistency Without common practices and libraries, teams have to (re)write new classes for handling errors, Kafka Streams startup and Interactive Queries for each project , etc 49 Updatability Kafka Streams is evolving fast, with new features, bug fixes and optimizations. Maintaining multiple applications up-to-date with the latest version can be challenging. We never build a single Kafka Streams apps, but dozens each running across multiple instances. Operability Kafka Streams can be challenging for operations teams. Make easy to monitor streams instance using standard API, across all projects, will help to keep the system running smoothly.
50. 50 (Azkarra is Basque word for "Fast") 50
51. . Yet Another Micro-Framework a lightweight framework that makes easy to create production-ready Kafka Streams applications. ❏ Open-source since 2019 November under Apache-2.0 License. ❏ Written in Java. Add a star to the GitHub project, it only takes 5 seconds ^^ 51 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
52. . Key Features Developer Friendly Production-Ready Secured Easy to learn API Configuration Externalization REST APIs for Interactive Queries SSL/TLS, Basic Authentication Built-in Healthchecks REST Endpoints for Metrics (JSON / Prometheus) Error handling Logics 52
53. . How to use It ? 53 <dependency> <groupId>io.streamthoughts</groupId> <artifactId>azkarra-streams</artifactId> <version>0.6.1</version> </dependency> © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
54. . Concept TopologyProvider 54 54 public interface TopologyProvider extends Provider<Topology> { /** * Supplies a new Kafka Streams {@link Topology} * instance. * * @return the {@link Topology} instance. */ Topology get(); /** * Returns the version of the supplied * {@link Topology}. * * @return the string version. */ @Override String version(); } A simple interface to implement ❏ Each topology must be versioned. ❏ Use to provide the Topology instance. 1
55. . Concept StreamsExecutionEnvironment 55 // Configure environment specific properties Conf config = … // Create a new environment to manage lifecycle of one or many KafkaStreams instances. StreamsExecutionEnvironment env = DefaultStreamsExecutionEnvironment.create("dev-env") .setConfiguration(config) .setKafkaStreamsFactory(/**/) .setRocksDBConfig(RocksDBConfig.withStatsEnable()) .setApplicationIdBuilder(/**/) .addGlobalStateListener(/**/) .addStateListener(/**/) // Register the topology env.addTopology(CountUserListenMusicPerGenreTopology::new, Executed.as("scottify-streams")); 2 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
56. . Concept StreamsLifecycleInterceptor 56 public interface StreamsLifecycleInterceptor { /** * This method is executed before starting the streams instance. */ default void onStart(StreamsLifecycleContext context, StreamsLifecycleChain chain) { chain.execute(); } /** * This method is executed before stopping the streams instance. */ default void onStop(StreamsLifecycleContext context, StreamsLifecycleChain chain) { chain.execute(); } } AutoCreateTopicsInterceptor WaitForSourceTopicsInterceptor Built-in interceptors 3 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
57. . Concept AzkarraContext 57 Registering Custom ExecutionEnvironment context.addExecutionEnvironment(env) Using a default ExecutionEnvironment context.addTopology( CountUserListenMusicPerGenreTopology.class, "dev-env" Executed.as("scottify-streams") ) // Configure context specific properties Conf ctxConf = … AzkarraContext context = DefaultAzkarraContext.create(ctxConf) 4 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
58. . Concept AzkarraApplication 58 //Provide configuration externalization (using Lightbend Config library). Conf appConf = AzkarraConf.create("application") new AzkarraApplication() .setConfiguration(appConf) .setContext(context) .setBannerMode(Banner.Mode.OFF) .enableHttpServer(true) .setRegisterShutdownHook(true) .run(args); 58 Provides additional features on top-off AzkarraContext : ❏ Embedded HTTP Server ❏ Component Scan (scan current package for classes annotated @Component - e.g: TopologyProvider) ❏ Application auto-configuration 5
59. . Interactive Queries HTTP Endpoint 59 Request $ POST /api/v1/applications/:application_id /stores/:store_name { "set_options": { "query_timeout_ms": 1000, "retries": 100, "retry_backoff_ms": 100, "remote_access_allowed": true }, "type": "key_value", "query": { "get": { "key": "001" } } } 59 Response { "took": 1, "timeout": false, "server": "localhost:8080", "status": "SUCCESS", "total": 1, "result": { "success": [ { "server": "localhost:8080", "remote": false, "records": [ { "key": "001", "value": { "name": "James Tiberius Kirk", "gender": "Male", "species": "Human", "key": "001" } } ] ...
60. DEMO
61. . Azkarra WebUI 61
62. . Azkarra WebUI 62
63. . Azkarra WebUI 63
64. . Additional Resources Code Source ❏ https://github.com/streamthoughts/demo-kafka-streams-scottify Azkarra Streams ❏ https://streamthoughts.github.io/azkarra-streams/ ❏ https://medium.com/streamthoughts/introducing-azkarra-streams-the-first-micro-framework-for-apache-kafka-streams-e1 3605f3a3a6 ❏ https://dev.to/fhussonnois/create-kafka-streams-applications-faster-than-ever-before-via-azkarra-streams-3nng 64 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
65. . Images & Icons Images Photo by Bryan Goff on Unsplash Photo by Stefan Cosma on Unsplash Photo by Jordan Whitfield on Unsplash Icons https://en.wikipedia.org/wiki/Starfleet https://en.wikipedia.org/wiki/Klingon 65 © streamthoughts. All rights reserved. Not to be reproduced in any form without prior written consent.
66. Thank you @streamthoughts www.streamthoughts.io contact@streamthoughts.io