Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Kafka with PHP App

Apache Kafka with PHP App

PHPの話

yuuki takezawa

April 24, 2018
Tweet

More Decks by yuuki takezawa

Other Decks in Programming

Transcript

  1. Apache Kafka With PHP App
    yuuki takezawa

    View full-size slide

  2. profile
    • ஛ᖒ ༗و / ytake
    • גࣜձࣾΞΠελΠϧ
    • PHP, Hack, Go, Scala
    • Hadoop, Spark, HBase
    • twitter https://twitter.com/ex_takezawa
    • facebook https://www.facebook.com/yuuki.takezawa
    • github https://github.com/ytake

    View full-size slide

  3. PHPͰ΋ར༻Ͱ͖ΔͷΛ
    ͝ଘ஌Ͱ͔͢ʁ

    View full-size slide

  4. Rdkafka
    https://github.com/arnaud-lb/
    php-rdkafka

    View full-size slide

  5. Rdkafka
    • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠

    ΤΫεςϯγϣϯ
    • High Level API & Row Level API
    • PHP5/7ରԠ
    • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ

    View full-size slide

  6. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );

    View full-size slide

  7. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ

    View full-size slide

  8. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ
    5PQJD໊
    1BSUJUJPOΛࢦఆ

    View full-size slide

  9. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');

    View full-size slide

  10. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏

    View full-size slide

  11. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ
    ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏
    $POTVNFSઃఆ஋Λࢦఆ

    View full-size slide

  12. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }

    View full-size slide

  13. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔

    View full-size slide

  14. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    ਖ਼ৗʹड৴ͨ͠৔߹ʹ

    ೚ҙͷॲཧΛ࣮ߦ

    View full-size slide

  15. Rdkafka Consumerͷઃఆ
    • heartbeatͱλΠϜΞ΢τ

    heartbeat.interval.ms͸

    session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘

    • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ
    • group.idʹΑΔॲཧͷฒྻԽ

    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ

    View full-size slide

  16. ฐ࣮ࣾྫ

    View full-size slide

  17. Example: Produce
    Ϣʔβʔͷ࣌ܥྻߦಈϩά͕
    QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ

    View full-size slide

  18. Example: Kafka
    "QBDIF,BGLB
    "QBDIF;PPLFFQFS
    QBSUJUJPO
    ϲ݄͘Β͍Ͱԯઍສ͘Β͍

    ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

    View full-size slide

  19. Example: Consume
    σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ
    QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴
    ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

    View full-size slide

  20. Example: Cassandra
    $BTTBOESB$MVTUFS
    ͪ͜Β΋ԯͪΐͬͱ͘Β͍
    ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ
    τϥϒϧͳ͠

    View full-size slide

  21. ൃల
    • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋

    • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ

    RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋

    Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ

    Event Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ

    View full-size slide

  22. ൃల
    • kafka + Spark Streaming͕ڧྗ͗͢Δ

    • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ

    ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍

    • ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ

    View full-size slide

  23. ·ͱΊ
    • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋
    • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ
    • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋

    • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏

    • Need KSQL REST API Document!

    View full-size slide