Tweet
Tweet

Slide 1

Slide 1 text

Apache Kafka With PHP App yuuki takezawa

Slide 2

Slide 2 text

profile • ஛ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ • PHP, Hack, Go, Scala • Hadoop, Spark, HBase • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake

Slide 3

Slide 3 text

author

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

PHPͰ΋ར༻Ͱ͖ΔͷΛ ͝ଘ஌Ͱ͔͢ʁ

Slide 6

Slide 6 text

Rdkafka https://github.com/arnaud-lb/ php-rdkafka

Slide 7

Slide 7 text

Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ

Slide 8

Slide 8 text

Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );

Slide 9

Slide 9 text

Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ

Slide 10

Slide 10 text

Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ

Slide 11

Slide 11 text

Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');

Slide 12

Slide 12 text

Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏

Slide 13

Slide 13 text

Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); 3PVOE3PCJO 1BSUJUJPO෼ׂ͢Δ ͳΒHSPVQJEΛࢦఆ͠·͠ΐ͏ $POTVNFSઃఆ஋Λࢦఆ

Slide 14

Slide 14 text

Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }

Slide 15

Slide 15 text

Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔

Slide 16

Slide 16 text

Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ

Slide 17

Slide 17 text

Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ
 ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ

Slide 18

Slide 18 text

ฐ࣮ࣾྫ

Slide 19

Slide 19 text

Example

Slide 20

Slide 20 text

Example: Produce Ϣʔβʔͷ࣌ܥྻߦಈϩά͕ QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ

Slide 21

Slide 21 text

Example: Kafka "QBDIF,BGLB "QBDIF;PPLFFQFS QBSUJUJPO ϲ݄͘Β͍Ͱԯઍສ͘Β͍
 ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

Slide 22

Slide 22 text

Example: Consume σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴ ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

Slide 23

Slide 23 text

Example: Cassandra $BTTBOESB$MVTUFS ͪ͜Β΋ԯͪΐͬͱ͘Β͍ ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ τϥϒϧͳ͠

Slide 24

Slide 24 text

ൃల • ࠓ͸Kafka StreamsͱKafka Connectͷ૊Έ߹Θͤ΋
 • RDBMSʹॻ͖ࠐΈͭͭElasticsearchʹసૹ
 RDBMSͱશจݕࡧΛ૊Έ͋ΘͤΔ஥հ໾ͱͯ͠΋
 Kafkaͷ࢓૊ΈΛ஌Ε͹ɺ
 Event Sourcing+CQRSରԠͷΞΠσΟΞ͕޿͕Δʂ

Slide 25

Slide 25 text

ൃల • kafka + Spark Streaming͕ڧྗ͗͢Δ
 • ؆୯ͳσʔλ෼ੳͰ͋Ε͹prestoͱ૊Έ߹Θͤͯ
 ଞͷσʔλϕʔεͱ݁߹Ͱ͖Δͷ΋ӡ༻ʹྑ͍
 • ෼ࢄγεςϜɾϚΠΫϩαʔϏεʹඞཁෆՄܽͳଘࡏʹ

Slide 26

Slide 26 text

ͦͷޙ

Slide 27

Slide 27 text

·ͱΊ • ϩάऩूҎ֎ʹ΋ϚΠΫϩαʔϏεΞʔΩςΫνϟʹ΋ • Kafka Streams, Kafka ConnectͰ޿͕Δબ୒ࢶ • PipelinedbɺKSQLͰσʔλΛఏڙ͢Δ͜ͱ΋
 • Lambda/Kappa ArchitectureʹνϟϨϯδ͠·͠ΐ͏
 • Need KSQL REST API Document!