Apache Kafka with PHP App

PHPの話

Transcript

1. Apache Kafka With PHP App
   yuuki takezawa
   

   View Slide

2. profile
   • ஛ᖒ ༗و / ytake
   • גࣜձࣾΞΠελΠϧ
   • PHP, Hack, Go, Scala
   • Hadoop, Spark, HBase
   • twitter https://twitter.com/ex_takezawa
   • facebook https://www.facebook.com/yuuki.takezawa
   • github https://github.com/ytake
   

   View Slide

3. author
   

   View Slide

4. View Slide

5. PHPͰ΋ར༻Ͱ͖ΔͷΛ
   ͝ଘ஌Ͱ͔͢ʁ
   

   View Slide

6. Rdkafka
   https://github.com/arnaud-lb/
   php-rdkafka
   

   View Slide

7. Rdkafka
   • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠

   ΤΫεςϯγϣϯ
   • High Level API & Row Level API
   • PHP5/7ରԠ
   • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
   

   View Slide

8. Rdkafka Producer࣮૷
   $rk = new RdKafka\Producer();
   $rk->setLogLevel(LOG_DEBUG);
   $rk->addBrokers(“127.0.0.1");
   $topic = $rk->newTopic("test");
   $topic->produce(
   RD_KAFKA_PARTITION_UA,
   0,
   json_encode(['message' => ‘phpcon'])
   );
   

   View Slide

9. Rdkafka Producer࣮૷
   $rk = new RdKafka\Producer();
   $rk->setLogLevel(LOG_DEBUG);
   $rk->addBrokers(“127.0.0.1");
   $topic = $rk->newTopic("test");
   $topic->produce(
   RD_KAFKA_PARTITION_UA,
   0,
   json_encode(['message' => ‘phpcon'])
   );
   $MVTUFSͷ৔߹͸ɺ

   ΧϯϚ۠੾ΓͰࢦఆ
   

   View Slide

10. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ
    5PQJD໊
    1BSUJUJPOΛࢦఆ
    

    View Slide

11. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    

    View Slide

12. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
    

    View Slide

13. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
    $POTVNFSઃఆ஋Λࢦఆ
    

    View Slide

14. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    

    View Slide

15. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    

    View Slide

16. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    ਖ਼ৗʹड৴ͨ͠৔߹ʹ

    ೚ҙͷॲཧΛ࣮ߦ
    

    View Slide

17. Rdkafka Consumerͷઃఆ
    • heartbeatͱλΠϜΞ΢τ

    heartbeat.interval.ms͸

    session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘

    • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ
    • group.idʹΑΔॲཧͷฒྻԽ

    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
    

    View Slide

18. ฐ࣮ࣾྫ
    

    View Slide

19. Example
    

    View Slide

20. Example: Produce
    Ϣʔβʔͷ࣌ܥྻߦಈϩά͕
    QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ
    

    View Slide

21. Example: Kafka
    "QBDIF,BGLB
    "QBDIF;PPLFFQFS
    QBSUJUJPO
    ϲ݄͘Β͍Ͱԯઍສ͘Β͍

    ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠
    

    View Slide

22. Example: Consume
    σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ
    QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴
    ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ
    
    

    View Slide

23. Example: Cassandra
    $BTTBOESB$MVTUFS
    ͪ͜Β΋ԯͪΐͬͱ͘Β͍
    ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ
    τϥϒϧͳ͠
    

    View Slide

24. ൃల
    • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋

    • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ

    RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋

    Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ

    Event Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ
    

    View Slide

25. ൃల
    • kafka + Spark Streaming͕ڧྗ͗͢Δ

    • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ

    ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍

    • ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ
    

    View Slide

26. ͦͷޙ
    

    View Slide

27. ·ͱΊ
    • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋
    • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ
    • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋

    • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏

    • Need KSQL REST API Document!
    

    View Slide