Apache Kafka with PHP App

PHPの話

Transcript

1. Apache Kafka With PHP App yuuki takezawa
2. profile • ஛ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ • PHP, Hack, Go, Scala • Hadoop, Spark, HBase • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake
3. author
5. PHPͰ΋ར༻Ͱ͖ΔͷΛ ͝ଘ஌Ͱ͔͢ʁ
6. Rdkafka https://github.com/arnaud-lb/ php-rdkafka
7. Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
8. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );
9. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ
10. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ
11. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');
12. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO
    1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
13. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO
    1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏ $POTVNFSઃఆ஋Λࢦఆ
14. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }
15. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔
16. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ
17. Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ
 ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
18. ฐ࣮ࣾྫ
19. Example
20. Example: Produce Ϣʔβʔͷ࣌ܥྻߦಈϩά͕
    QIQ
    SELBGLBܦ༝ͰૹΒΕͯ͘Δ
21. Example: Kafka "QBDIF
    ,BGLB
    
    "QBDIF
    ;PPLFFQFS
    
    QBSUJUJPO
    
    ϲ݄͘Β͍Ͱԯઍສ͘Β͍
 ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠
22. Example: Consume σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ
    QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴
    ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ
23. Example: Cassandra $BTTBOESB
    
    $MVTUFS
    ͪ͜Β΋ԯͪΐͬͱ͘Β͍
    ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ
    τϥϒϧͳ͠
24. ൃల • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋
 • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ
 RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋
 Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ
 Event Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ
25. ൃల • kafka + Spark Streaming͕ڧྗ͗͢Δ
 • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ
 ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍
 • ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ
26. ͦͷޙ
27. ·ͱΊ • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋ • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋
 • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏
 • Need KSQL REST API Document!