Using Apache Kafka from Go

Using Apache Kafka from Go

At Uber, we use Kafka and Golang a lot. In this talk, I go over the basic concepts in Kafka and the libraries we use to publish and consume messages.

Eb44761e0fb3a5ec8e23ec28048dd7a5?s=128

Nikolay Stoitsev

February 19, 2019
Tweet

Transcript

  1. Using Apache Kafka from Go Nikolay Stoitsev, Software Engineer

  2. Trillion+ Messages/Day PBs Data Volume Tens of Thousands Topics

  3. Message Message Message Message Message Message Topic

  4. Partition - ordered, immutable sequence Message Message Message Message Message

    Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
  5. Offset Message Message Message Message Message Partition 0 Offset: 1

  6. Offset Message Message Message Message Message Partition 0 Offset: 2

  7. Offset Message Message Message Message Message Partition 0 Offset: 2

    Offset: 3 Consumer: 1 Consumer: 2
  8. Kafka Cluster

  9. Broker Broker Broker

  10. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2

    P2 P2 Partitioned and Replicated
  11. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2

    P2 P2 Producers Producer Producer
  12. Consumer Consumer Consumer Consumer Consumer Broker Broker Broker P0 P0

    P0 P1 P1 P1 P2 P2 P2 Producers Producer Producer Consumer Group 1 Consumer Group 2
  13. None
  14. Latency vs. Durability

  15. At-most-once cluster for logging

  16. At-least-one cluster for message bus

  17. Publishing to lossy kafka for logging

  18. log.Printf("failed to fetch URL %s", url)

  19. Structured logging

  20. None
  21. https://github.com/uber-go/zap

  22. import "go.uber.org/zap"

  23. SugaredLogger

  24. Logger

  25. None
  26. None
  27. None
  28. None
  29. None
  30. Publishing to lossless kafka

  31. Avro

  32. https://github.com/confluentinc/kafka-rest

  33. *net/http.Client

  34. Consume from kafka

  35. https://github.com/Shopify/sarama

  36. AsyncProducer and SyncProducer

  37. None
  38. None
  39. https://github.com/bsm/sarama-cluster

  40. None
  41. None
  42. Heartbeat

  43. Handles rebalancing

  44. Handles coordinator failure

  45. Thanks! Nikolay Stoitsev