Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Tips & Tricks for Apache Kafka Messages Integration

FTisiot
September 15, 2021

Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.

FTisiot

September 15, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. Practical Tips & Tricks for Apache Kafka Messages Integration Francesco

    Tisiot - Developer Advocate @ftisiot
  2. @ftisiot

  3. @ftisiot Libraries

  4. @ftisiot kafka-python pip install kafka-python Consume Admin Produce

  5. @ftisiot confluent-kafka-python pip install confluent-kafka Schema Registry Avro Karapace

  6. @ftisiot Connect

  7. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234', ) How many brokers?

  8. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', )

  9. @ftisiot KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', security_protocol='SSL', ssl_cafile='ca.pem', ssl_certfile='service.cert', ssl_keyfile='service.key' ) SSL

  10. @ftisiot Produce

  11. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', ) Minimal Setup

  12. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode(‘ascii'), )

    Data Conversion https://aiven.io/blog/tips-for-designing-payloads
  13. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    ) Confirmation [0,1,'all']
  14. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    retries=5, ) What do we do in case of Errors?
  15. @ftisiot

  16. @ftisiot producer = KafkaProducer( bootstrap_servers='kafka:1234,kafka2:5678', value_serializer= lambda v: json.dumps(v).encode('ascii'), acks=1,

    retries=5, max_in_flight_requests_per_connection=1 ) Avoids out of order messages
  17. @ftisiot producer.send( 'my_shop_topic', key=b'Milan', value= { 'order_id': 1, 'product_name': 'Nutella

    Jar', 'quantity': 12 } ) batch_size
  18. @ftisiot producer.flush(timeout=10)

  19. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing

    to Non Existing Topic auto.create.topics.enable=true
  20. @ftisiot Admin

  21. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678'

    ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
  22. @ftisiot Consume

  23. @ftisiot consumer = KafkaConsumer( bootstrap_servers= 'kafka:1234,kafka2:5678' ) consumer.subscribe( topics=['testA'] )

    for message in consumer: print(message.value)
  24. @ftisiot auto_offset_reset='earliest'

  25. @ftisiot consumer.subscribe( topics=['testA','testB'] )

  26. @ftisiot consumer.subscribe( pattern='test*' ) consumer.subscribe( topics=['testA','testB'] ) metadata_max_age_ms

  27. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', )

  28. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    )
  29. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, )
  30. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )
  31. @ftisiot subscribe() assign()

  32. @ftisiot consumer.subscribe(topics=['test']) group_id='snakes'

  33. @ftisiot consumer.assign( [TopicPartition('test', 0)] )

  34. @ftisiot Connect Produce Admin Consume

  35. @ftisiot Resources https://aiven.io https://kafka-python.readthedocs.io/ https://aiven.io/blog/create-your-own-data-stream-for- kafka-with-python-and-faker kafka-summit-2021 500$