Apache Kafka with PHP App

Transcript

1. Apache Kafka With PHP App yuuki takezawa

2. profile • ஛ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ • PHP,

   Hack, Go, Scala • Hadoop, Spark, HBase • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake

3. author

4. None

5. PHPͰ΋ར༻Ͱ͖ΔͷΛ ͝ଘ஌Ͱ͔͢ʁ

6. Rdkafka https://github.com/arnaud-lb/ php-rdkafka

7. Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row

   Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ

8. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

   $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );

9. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

   $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ

10. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ

11. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');

12. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO
    1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏

13. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO
    1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏ $POTVNFSઃఆ஋Λࢦఆ

14. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }

15. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔

16. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ

17. Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ


    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ

18. ฐ࣮ࣾྫ

19. Example

20. Example: Produce Ϣʔβʔͷ࣌ܥྻߦಈϩά͕
    QIQ
    SELBGLBܦ༝ͰૹΒΕͯ͘Δ

21. Example: Kafka "QBDIF
    ,BGLB
    
    "QBDIF
    ;PPLFFQFS
    
    QBSUJUJPO
    
    ϲ݄͘Β͍Ͱԯઍສ͘Β͍
 ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

22. Example: Consume σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ
    QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴
    ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

23. Example: Cassandra $BTTBOESB
    
    $MVTUFS
    ͪ͜Β΋ԯͪΐͬͱ͘Β͍
    ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ
    τϥϒϧͳ͠

24. ൃల • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋
 • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ
 RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋
 Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ
 Event

    Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ

25. ൃల • kafka + Spark Streaming͕ڧྗ͗͢Δ
 • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ
 ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍
 •

    ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ

26. ͦͷޙ

27. ·ͱΊ • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋ • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋


    • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏
 • Need KSQL REST API Document!