Introducing_FastKafka.pdf

Transcript

1. Introducing FastKafka Tvrtko Sternak PyCon DE / PyData 2023
2. Motivation
3. model
4. prediction = model.predict()
5. app = FastApi() @app.get() async def on_input_data(msg: InputData) -> Prediction: prediction = model.predict(msg) return prediction
6. app = FastApi() @app.get() async def on_input_data(msg: InputData) -> Prediction: prediction = model.predict(msg) return prediction decodes input using InputData.parse_raw() encodes output using Prediction.json() handles message routing
7. app = some_kafka_framework.Application() @app.consume() async def on_input_data(msg: InputData): p = model.predict(msg) to_prediction(p) @app.produce() async def to_prediction(p: int) -> Prediction: return Prediction(p=p)
8. What we need • Easy to write code • Testable • Automatic documentation generation • High performance • Well maintained
9. aiokafka + + FastKafka =
10. Consuming
11. app = fastkafka.FastKafka() @app.consume() async def on_input_data(msg: InputData): # some processing decodes consumed message using InputData.parse_raw() consumes messages from input_data topic
12. app = fastkafka.FastKafka() @app.consume(auto_offset_reset=“earliest”) async def on_input_data(msg: InputData): # some processing override any parameter from the underlying AIOKafkaConsumer
14. app = fastkafka.FastKafka() @app.consume(decoder=“avro”) async def on_input_data(msg: InputData): # some processing decodes consumed message using Avro decoder
15. Producing
16. @kafka_app.produces() def to_predictions(score: float) -> Prediction: return Prediction(score=score) encodes produced message using Prediction.json() produces messages to prediction topic
17. @kafka_app.produces(encoder=“avro”) def to_predictions(score: float) -> Prediction: return Prediction(score=score) encodes produced message using Avro encoder
18. What about the keys?
19. @kafka_app.produces() def to_predictions(score: float) -> KafkaEvent[Prediction]: p = Prediction(score=score) return KafkaEvent(p, key=b”my_key”) encapsulates Prediction class in order to use key or partition de fi nes key of the produced message
20. Testing
21. from fastkafka.testing import Tester async with Tester(kafka_app) as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) )
22. from fastkafka.testing import Tester async with Tester(kafka_app) as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) ) Tester(kafka_app) creates mirrors of consumers and producers: kafka_app.on_input_data → tester.to_input_data kafka_app.to_predictions → tester.on_predictions
23. from fastkafka.testing import Tester async with Tester(kafka_app) as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) ) Tester(kafka_app) creates mockups for assertions: kafka_app.on_input_data → tester.awaited_mocks.on_input_data kafka_app.to_predictions → tester.awaited_mocks.on_predictions
24. from fastkafka.testing import Tester async with Tester(kafka_app) as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) ) Tester(kafka_app) runs an Instance of Kafka in background!
25. The problem: you don’t want to use Kafka broker
26. Tester runs with in-memory Kafka broker by default • Fully-functional Kafka broker running in memory • Fast setup & teardown needed for unit-testing • No side e ff ects
27. from fastkafka.testing import Tester async with Tester(kafka_app) as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) ) Tester(kafka_app) runs an Instance of Kafka in background Tester(kafka_app) runs an Instance of in memory Kafka mockup
28. But I want the real Kafka
29. async with Tester(kafka_app).using_local_redpanda() as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) ) async with Tester(kafka_app).using_local_kafka() as tester: await tester.to_input_data(msg) await tester.awaited_mocks.on_predictions.assert_awaited_with( Prediction(p=0.97) )
30. Documentation generation
31. $ fastkafka docs generate application:kafka_app $ fastkafka docs serve application:kafka_app
36. Serving
37. $ fastkafka run application:kafka_app Symbol kafka_app of type FastKafka to import Python module application (or local fi le application.py) with fast_kafka symbol de fi ned
38. $ fastkafka run —num-workers=2 application:kafka_app easily scales horizontally using multiple processes
39. Benchmarking
40. app = fastkafka.FastKafka() @app.consume() @app.benchmark() async def on_input_data(msg: InputData): # some processing
41. Performance penalty No computation apart from a simple JSON encoding/decoding • Worst case performance penalty: ~30% msg/s FastKafka AIOKafka Consuming ~147.000 ~212.000 Producing ~47.000 ~66.000
42. Future steps
43. 🐝🐝🐝 We were busy lately 🐝🐝🐝
44. What we plan to implement • Add transactional logic: @app.process() • Add support for other messaging protocols (RabbitMQ…) • Add support for other libraries (Con fl uent…) • Performance improvements • Possibly rewrite (parts of) in Rust • Last but not least, community feedback
46. Star, comment and subscribe! • GitHub: https://github.com/airtai/fastkafka • Discord: https://bit.ly/fastkafka-discord