Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.

Transcript

1. Practical Tips & Tricks for Apache Kafka Messages Integration Francesco Tisiot - Developer Advocate @ftisiot
2. @ftisiot
3. @ftisiot Libraries
4. @ftisiot kafka-python pip install kafka-python Consume Admin Produce
5. @ftisiot confluent-kafka-python pip install confluent-kafka Schema Registry Avro Karapace
6. @ftisiot Connect
7. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234', ) How many brokers?
8. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', )
9. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', security_protocol='SSL', ssl_cafile='ca.pem', ssl_certfile='service.cert', ssl_keyfile='service.key' ) SSL
10. @ftisiot Produce
11. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', ) Minimal Setup
12. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode(‘ascii'), ) Data Conversion https://aiven.io/blog/tips-for-designing-payloads
13. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, ) Confirmation [0,1,'all']
14. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, retries=5, ) What do we do in case of Errors?
15. @ftisiot
16. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, retries=5, max_in_flight_requests_per_connection=1 ) Avoids out of order messages
17. @ftisiot producer.send( 'my_shop_topic', key=b'Milan', value= { 'order_id': 1, 'product_name': 'Nutella Jar', 'quantity': 12 } ) batch_size
18. @ftisiot producer.flush(timeout=10)
19. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing to Non Existing Topic auto.create.topics.enable=true
20. @ftisiot Admin
21. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678' ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
22. @ftisiot Consume
23. @ftisiot consumer = KafkaConsumer( bootstrap_servers= 'kafka:1234,kafka2:5678' ) consumer.subscribe( topics=['testA'] ) for message in consumer: print(message.value)
24. @ftisiot auto_offset_reset='earliest'
25. @ftisiot consumer.subscribe( topics=['testA','testB'] )
26. @ftisiot consumer.subscribe( pattern='test*' ) consumer.subscribe( topics=['testA','testB'] ) metadata_max_age_ms
27. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', )
28. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), )
29. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), auto_offset_reset='earliest', metadata_max_age_ms=10000, )
30. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )
31. @ftisiot subscribe() assign()
32. @ftisiot consumer.subscribe(topics=['test']) group_id='snakes'
33. @ftisiot consumer.assign( [TopicPartition('test', 0)] )
34. @ftisiot Connect Produce Admin Consume
35. @ftisiot Resources https://aiven.io https://kafka-python.readthedocs.io/ https://aiven.io/blog/create-your-own-data-stream-for- kafka-with-python-and-faker kafka-summit-2021 500$