Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Tips & Tricks for Apache Kafka Messages Integration

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
September 15, 2021

Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

September 15, 2021
Tweet

Transcript

  1. Practical Tips & Tricks for Apache Kafka Messages Integration Francesco

    Tisiot - Developer Advocate @ftisiot
  2. @ftisiot

  3. @ftisiot Libraries

  4. @ftisiot kafka-python pip install kafka-python Consume Admin Produce

  5. @ftisiot confluent-kafka-python pip install confluent-kafka Schema Registry Avro Karapace

  6. @ftisiot Connect

  7. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234', ) How many brokers?

  8. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', )

  9. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', security_protocol='SSL', ssl_cafile='ca.pem', ssl_certfile='service.cert', ssl_keyfile='service.key' ) SSL

  10. @ftisiot Produce

  11. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', ) Minimal Setup

  12. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode(‘ascii'), )

    Data Conversion https://aiven.io/blog/tips-for-designing-payloads
  13. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    ) Confirmation [0,1,'all']
  14. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    retries=5, ) What do we do in case of Errors?
  15. @ftisiot

  16. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    retries=5, max_in_flight_requests_per_connection=1 ) Avoids out of order messages
  17. @ftisiot producer.send( 'my_shop_topic', key=b'Milan', value= { 'order_id': 1, 'product_name': 'Nutella

    Jar', 'quantity': 12 } ) batch_size
  18. @ftisiot producer.flush(timeout=10)

  19. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing

    to Non Existing Topic auto.create.topics.enable=true
  20. @ftisiot Admin

  21. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678'

    ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
  22. @ftisiot Consume

  23. @ftisiot consumer = KafkaConsumer( bootstrap_servers= 'kafka:1234,kafka2:5678' ) consumer.subscribe( topics=['testA'] )

    for message in consumer: print(message.value)
  24. @ftisiot auto_offset_reset='earliest'

  25. @ftisiot consumer.subscribe( topics=['testA','testB'] )

  26. @ftisiot consumer.subscribe( pattern='test*' ) consumer.subscribe( topics=['testA','testB'] ) metadata_max_age_ms

  27. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', )

  28. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    )
  29. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, )
  30. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )
  31. @ftisiot subscribe() assign()

  32. @ftisiot consumer.subscribe(topics=['test']) group_id='snakes'

  33. @ftisiot consumer.assign( [TopicPartition('test', 0)] )

  34. @ftisiot Connect Produce Admin Consume

  35. @ftisiot Resources https://aiven.io https://kafka-python.readthedocs.io/ https://aiven.io/blog/create-your-own-data-stream-for- kafka-with-python-and-faker kafka-summit-2021 500$