Kafka Will Get The Message Across, Guaranteed.

D6ccd6409910643d05ddaea3b2cd6f13?s=47 David Zuelke
December 02, 2016

Kafka Will Get The Message Across, Guaranteed.

Presentation given at SymfonyCon 2016 in Berlin, Germany.

D6ccd6409910643d05ddaea3b2cd6f13?s=128

David Zuelke

December 02, 2016
Tweet

Transcript

  1. 3.
  2. 6.
  3. 10.
  4. 12.
  5. 21.
  6. 23.

    WAL

  7. 26.
  8. 28.

    1 create document: "foo", data: "…" 2 update document: "foo",

    data: "…" 3 create document: "bar", data: "…" 4 remove document: "foo"
  9. 29.
  10. 32.
  11. 37.
  12. 40.

    1 2 3 4 5 6 7 8 … producer

    writes consumer reads
  13. 42.

    1 2 3 4 5 6 7 8 … producer

    writes consumerB reads consumerA reads
  14. 44.

    P0 1 2 3 4 5 6 7 … P1

    1 2 3 4 … P2 1 2 3 4 5 6 7 8 … P3 1 2 3 4 5 6 …
  15. 47.
  16. 49.
  17. 50.
  18. 55.

    BASIC PRODUCER $rk = new RdKafka\Producer(); $rk->addBrokers("127.0.0.1"); $topic = $rk->newTopic("test");

    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Unassigned partition, let Kafka choose"); $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Yay consistent hashing", $user->getId()); $topic->produce(1, 0, "This will always be sent to partition 1");
  19. 56.
  20. 57.
  21. 61.
  22. 62.

    BASIC CONSUMER $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); }
  23. 63.

    AT-MOST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk

    = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); $topic->offsetStore($msg->partition, $msg->offset); do_something($msg); }
  24. 64.

    AT-LEAST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk

    = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); $topic->offsetStore($msg->partition, $msg->offset); }
  25. 67.

    THE BYZANTINE GENERALS "together we can beat the monsters. let's

    both attack at 07:00?" "confirm, we attack at 07:00" ☠
  26. 68.
  27. 69.

    • LinkedIn • Yahoo • Twitter • Netflix • Square

    • Spotify • Pinterest • Uber • Goldman Sachs • Tumblr • PayPal • Airbnb • Mozilla • Cisco • Etsy • Foursquare • Shopify • CloudFlare
  28. 71.
  29. 75.

    IoT

  30. 77.
  31. 80.