Practical Tips & Tricks for Apache Kafka Messages Integration Francesco Tisiot - Developer Advocate @ftisiot

@ftisiot

@ftisiot Libraries

@ftisiot kafka-python pip install kafka-python Consume Admin Produce

@ftisiot confluent-kafka-python pip install confluent-kafka Schema Registry Avro Karapace

@ftisiot Connect

@ftisiot KafkaProducer( bootstrap_servers='kafka:1234', ) How many brokers?

@ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', )

@ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', security_protocol='SSL', ssl_cafile='ca.pem', ssl_certfile='service.cert', ssl_keyfile='service.key' ) SSL

@ftisiot Produce

@ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', ) Minimal Setup

@ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode(‘ascii'), ) Data Conversion https://aiven.io/blog/tips-for-designing-payloads

@ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, ) Confirmation [0,1,'all']

@ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, retries=5, ) What do we do in case of Errors?

@ftisiot

@ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1, retries=5, max_in_flight_requests_per_connection=1 ) Avoids out of order messages

@ftisiot producer.send( 'my_shop_topic', key=b'Milan', value= { 'order_id': 1, 'product_name': 'Nutella Jar', 'quantity': 12 } ) batch_size

@ftisiot producer.flush(timeout=10)

@ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing to Non Existing Topic auto.create.topics.enable=true

@ftisiot Admin

@ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678' ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )

@ftisiot Consume

@ftisiot consumer = KafkaConsumer( bootstrap_servers= 'kafka:1234,kafka2:5678' ) consumer.subscribe( topics=['testA'] ) for message in consumer: print(message.value)

@ftisiot auto_offset_reset='earliest'

@ftisiot consumer.subscribe( topics=['testA','testB'] )

@ftisiot consumer.subscribe( pattern='test*' ) consumer.subscribe( topics=['testA','testB'] ) metadata_max_age_ms

@ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', )

@ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), )

@ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), auto_offset_reset='earliest', metadata_max_age_ms=10000, )

@ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')), auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )

@ftisiot subscribe() assign()

@ftisiot consumer.subscribe(topics=['test']) group_id='snakes'

@ftisiot consumer.assign( [TopicPartition('test', 0)] )

@ftisiot Connect Produce Admin Consume

@ftisiot Resources https://aiven.io https://kafka-python.readthedocs.io/ https://aiven.io/blog/create-your-own-data-stream-for- kafka-with-python-and-faker kafka-summit-2021 500$