Slide 34
Slide 34 text
Horizontal Scaling Story in KStreams(KIP-878)
Problem: as business expands, you might want to increase the parallelism of the streams
processing. Users will want to increase the number of partitions of input topics. However, internal
topics(changelog, repartition) will not automatically increase, and today, KStreams application will
crash upon detecting a mismatch of partition number between internal topics and input topics.
KIP-878: Support auto scaling of internal topics. This works well if your application can be,
● Statically partitioned or stateless: stateless is straightforward. In KStreams, your state
store(e.g. RocksDB) is backed by internal topics, thus, bound to a partition. Upon
autoscaling, the pre-existing state will not move. Choose a partition strategy that works
well for your use case, such that you can drive tasks(without pre-existing state) to the
newly created partitions, while existing keys remains sticky.
● Upfront over-provisioned for stateful processing, while the KIP is in progress.