Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Kafka with PHP App

Apache Kafka with PHP App

PHPの話

yuuki takezawa

April 24, 2018
Tweet

More Decks by yuuki takezawa

Other Decks in Programming

Transcript

  1. Apache Kafka With PHP App
    yuuki takezawa

    View Slide

  2. profile
    • ஛ᖒ ༗و / ytake
    • גࣜձࣾΞΠελΠϧ
    • PHP, Hack, Go, Scala
    • Hadoop, Spark, HBase
    • twitter https://twitter.com/ex_takezawa
    • facebook https://www.facebook.com/yuuki.takezawa
    • github https://github.com/ytake

    View Slide

  3. author

    View Slide

  4. View Slide

  5. PHPͰ΋ར༻Ͱ͖ΔͷΛ
    ͝ଘ஌Ͱ͔͢ʁ

    View Slide

  6. Rdkafka
    https://github.com/arnaud-lb/
    php-rdkafka

    View Slide

  7. Rdkafka
    • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠

    ΤΫεςϯγϣϯ
    • High Level API & Row Level API
    • PHP5/7ରԠ
    • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ

    View Slide

  8. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );

    View Slide

  9. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ

    View Slide

  10. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ
    5PQJD໊
    1BSUJUJPOΛࢦఆ

    View Slide

  11. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');

    View Slide

  12. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏

    View Slide

  13. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
    $POTVNFSઃఆ஋Λࢦఆ

    View Slide

  14. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }

    View Slide

  15. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔

    View Slide

  16. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    ਖ਼ৗʹड৴ͨ͠৔߹ʹ

    ೚ҙͷॲཧΛ࣮ߦ

    View Slide

  17. Rdkafka Consumerͷઃఆ
    • heartbeatͱλΠϜΞ΢τ

    heartbeat.interval.ms͸

    session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘

    • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ
    • group.idʹΑΔॲཧͷฒྻԽ

    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ

    View Slide

  18. ฐ࣮ࣾྫ

    View Slide

  19. Example

    View Slide

  20. Example: Produce
    Ϣʔβʔͷ࣌ܥྻߦಈϩά͕
    QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ

    View Slide

  21. Example: Kafka
    "QBDIF,BGLB
    "QBDIF;PPLFFQFS
    QBSUJUJPO
    ϲ݄͘Β͍Ͱԯઍສ͘Β͍

    ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

    View Slide

  22. Example: Consume
    σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ
    QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴
    ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

    View Slide

  23. Example: Cassandra
    $BTTBOESB$MVTUFS
    ͪ͜Β΋ԯͪΐͬͱ͘Β͍
    ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ
    τϥϒϧͳ͠

    View Slide

  24. ൃల
    • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋

    • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ

    RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋

    Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ

    Event Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ

    View Slide

  25. ൃల
    • kafka + Spark Streaming͕ڧྗ͗͢Δ

    • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ

    ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍

    • ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ

    View Slide

  26. ͦͷޙ

    View Slide

  27. ·ͱΊ
    • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋
    • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ
    • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋

    • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏

    • Need KSQL REST API Document!

    View Slide