Apache Kafka with PHP App

Apache Kafka with PHP App

PHPの話

17d4ef53b432ebf7c566fd6a11345570?s=128

yuuki takezawa

April 24, 2018
Tweet

Transcript

  1. Apache Kafka With PHP App yuuki takezawa

  2. profile • ஛ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ • PHP,

    Hack, Go, Scala • Hadoop, Spark, HBase • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake
  3. author

  4. None
  5. PHPͰ΋ར༻Ͱ͖ΔͷΛ ͝ଘ஌Ͱ͔͢ʁ

  6. Rdkafka https://github.com/arnaud-lb/ php-rdkafka

  7. Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row

    Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
  8. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );
  9. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ
  10. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ
  11. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');
  12. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
  13. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏ $POTVNFSઃఆ஋Λࢦఆ
  14. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }
  15. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔
  16. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ
  17. Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ


    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
  18. ฐ࣮ࣾྫ

  19. Example

  20. Example: Produce Ϣʔβʔͷ࣌ܥྻߦಈϩά͕ QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ

  21. Example: Kafka "QBDIF,BGLB  "QBDIF;PPLFFQFS  QBSUJUJPO ϲ݄͘Β͍Ͱԯઍສ͘Β͍
 ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

  22. Example: Consume σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴ ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

  23. Example: Cassandra $BTTBOESB $MVTUFS ͪ͜Β΋ԯͪΐͬͱ͘Β͍ ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ τϥϒϧͳ͠

  24. ൃల • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋
 • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ
 RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋
 Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ
 Event

    Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ
  25. ൃల • kafka + Spark Streaming͕ڧྗ͗͢Δ
 • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ
 ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍
 •

    ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ
  26. ͦͷޙ

  27. ·ͱΊ • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋ • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋


    • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏
 • Need KSQL REST API Document!