Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Tips & Tricks for Apache Kafka Messages Integration

September 15, 2021

Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.


September 15, 2021

More Decks by FTisiot

Other Decks in Technology


  1. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing

    to Non Existing Topic auto.create.topics.enable=true
  2. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678'

    ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
  3. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )