Slide 1
Slide 1 text
Apache KafkaʹΑΔ
εέʔϥϒϧ
ΞϓϦέʔγϣϯ։ൃ
yuuki takezawa
PHP Conference 2017
Slide 2
Slide 2 text
͡Ίʹ
• نಁաੑ
ෛՙͷߴʹ߹ΘͤͯϦ
ιʔεɾϓʔϧΛ֦େɾॖখͰ͖Δ͜ͱ
• Ґஔಁաੑ
ϢʔβʔϦιʔε͕ͲΕ͚ͩΕ͍ͯΔ͔ҙࣝͤͣʹɺ
มΘΒͳ͍͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ
• ҟछಁաੑ
γεςϜΛߏ͢ΔػثιϑτΣΞ͕ҟͳ͍ͬͯΔ͜
ͱΛҙࣝͤͣʹཧɾར༻Ͱ͖Δ͜ͱ
Slide 3
Slide 3 text
Agenda
• Message Queue
• What is Apache Kafka?
• For PHP
Slide 4
Slide 4 text
Message Queue
Slide 5
Slide 5 text
ίϝϯτߘαʔϏε
• ϩάΠϯϢʔβʔίϝϯτ͕ߘͰ͖Δ
• ະϩάΠϯϢʔβʔӾཡͷΈ͕Ͱ͖Δ
• ͓ؾʹೖΓͨ͠ϢʔβʔͷίϝϯτΛӾཡͰ͖Δ
• ࣌ܥྻͰίϝϯτΛӾཡͰ͖Δ
• ίϝϯτͷআ͕Ͱ͖Δ
• ίϝϯτͷฤूͰ͖ͳ͍ ͳͲ
Slide 6
Slide 6 text
ΞϓϦέʔγϣϯ࣮
• ೝূػೳͷ࣮
• ίϝϯτߘɺฤूɺӾཡػೳ࣮
• ݕࡧػೳͷ࣮
• ਓͷ։ൃऀ
Slide 7
Slide 7 text
γϯϓϧͳΞϓϦέʔγϣϯߏ
Slide 8
Slide 8 text
γϯϓϧͳΞϓϦέʔγϣϯߏ
public function __invoke(Comment $comment)
{
$query = $comment->query();
$query->insert(['title', 'body'])
->values([
'title' => 'First post',
'body' => 'Some body text',
'id' => $this->auth->getId(),
])
->execute();
}
Slide 9
Slide 9 text
ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
• ػೳՃ
• ίϝϯτใΛଞͷػೳͰར༻͍ͨ͠
• ݅ʹΑΔҰཡใͷιʔτมߋ
ϢʔβʔใଐੑʹΑΔιʔτॱมߋ
Slide 10
Slide 10 text
ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
Slide 11
Slide 11 text
ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
public function __invoke(Comment $comment, Movie $movie)
{
$query = $comment->query();
$commentId = $query->insert(['title', 'body', 'comment_id'])
->values([
'title' => 'First post',
'body' => 'Some body text',
'comment_id' => $this->auth->getId(),
])
->execute();
$movieQuery = $movie->query();
$movieQuery->insert(['movie_id', 'comment_id'])
->values([
'movie_id' => $movie->getId(),
'comment_id' => $commentId,
])
->execute();
}
Slide 12
Slide 12 text
Ϗδωεͷͱͱʹ
ංେԽ͢ΔΞϓϦέʔγϣϯ
Slide 13
Slide 13 text
ංେԽ͢ΔΞϓϦέʔγϣϯ
• ऩӹԽͷͨΊͷػೳ
• རศੑͷͨΊͷػೳ
• طଘػೳͷ֦ு
Slide 14
Slide 14 text
ංେԽ͢ΔΞϓϦέʔγϣϯ
Slide 15
Slide 15 text
ංେԽ͢ΔΞϓϦέʔγϣϯ
public function __invoke(
Comment $comment,
Movie $movie,
Follow $follow,
Company $company
) {
$movieCommentId = false;
if ($follow->isFollow()) {
$query = $comment->query();
$commentId = $query->insert(['title', 'body', 'comment_id'])
->values([
'title' => 'First post',
'body' => 'Some body text',
'comment_id' => $this->auth->getId(),
])
->execute();
$movieQuery = $movie->query();
$movieCommentId = $movieQuery->insert(['movie_id', 'comment_id'])
->values([
'movie_id' => $movie->getId(),
'comment_id' => $commentId,
])
->execute();
}
if ($movie->isCompany() && $movieCommentId) {
$company->appendMovie($movie->getId());
// ϝʔϧૹ৴
}
}
Slide 16
Slide 16 text
ΞϓϦέʔγϣϯͷΫϥεͦͷͷΛ
ίϯϙʔωϯτԽ
ґଘੑΛऔΓআ͘
Slide 17
Slide 17 text
ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά
• ݁߹ͨ͠ػೳΛίϯϙʔωϯτఏڙ
• APIʹΑΔσʔλૢ࡞ͷநԽ
• σʔλϕʔεύϑΥʔϚϯεվળ
Slide 18
Slide 18 text
ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά
Slide 19
Slide 19 text
ڊେͳΞϓϦέʔγϣϯ
• αʔϏε֦େʹͱͳ͏ΞϓϦέʔγϣϯՃ
• ґଘίϯϙʔωϯτ૿Ճ
-> ༷มߋٵऩ
• ૿Ճ͢ΔAPIར༻
-> ᷓΕग़͢ଞαʔϏεͷϏδωεϩδοΫ
• σʔλϕʔε
-> ElasticsearchɺSolrͳͲͷಋೖ
Slide 20
Slide 20 text
ڊେͳΞϓϦέʔγϣϯ
Slide 21
Slide 21 text
ਂࠁͳ
• ༷มߋʹ͏ίϯϙʔωϯτमਖ਼ͱϦϦʔεෳࡶԽ
ϥΠϒϥϦͷόʔδϣϯ͋͛ͳ͖Όɾɾʂ
͋ͷػೳΛऔΓࠐΉʹ͋Εͱ͜Εͱɾɾɾ
• ΞϓϦέʔγϣϯؒ࿈ܞͷෳࡶԽ
APIमਖ਼ʹ͏Өڹൣғௐࠪ
վमͯͬͯ͘͠Εͳ͍
Slide 22
Slide 22 text
ਂࠁͳ
• σʔλಉظͷෳࡶԽ
ϦΞϧλΠϜͰσʔλ͕ཉ͍͠
ΠϯσοΫεߋ৽͕ఆظ࣮ߦ
• ͦͷଞ
Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒোʹ
• ෳࡶԽ͢Δ࣮ίʔυ
Slide 23
Slide 23 text
͍ͯ͘͠ࣄۀʹ߹Θͤͯɺ
ΞϓϦέʔγϣϯΛ
Ͳ͏εέʔϧ͍͔ͤͯ͘͞
Slide 24
Slide 24 text
มΘΓΏ͘ڥ
• σʔλΛऔಘ͢Δʹʁ
APIίʔϧʹΑΔ֬ೝʁ
ΩϟογϡԿʁ
• ͦΕͧΕͷαʔϏεͰͦΕͧΕͷݴޠ
͋ΔαʔϏεPHPɺ͋ΔαʔϏεͰGo ScalaͳͲ
ϥΠϒϥϦ͕ෛͷҨ࢈Խ
Slide 25
Slide 25 text
σʔλॲཧ
Ϣʔβʔใ͍߹Θͤ
ใΛߋ৽ͨ͠߹Ͱ
ґଘαʔϏε͕ͳ͍ͨΊͳ͠
Slide 26
Slide 26 text
σʔλॲཧͷෳࡶԽ
Ϣʔβʔใมߋ Ϣʔβʔใআ
Ϣʔβʔใ͕ݟ͔ͭΒͳ͍ͨΊ
σʔλॲཧ͝ͱʹ"1*ΞΫηε
͋ͦ͜ͱͦ͜Ͱใ͕ҧ͏ʂ
Ͳͷσʔλ͕ຊʁ
શͯͷΞϓϦέʔγϣϯ͔Β
Ωϟογϡͳ͠ͰΞΫηε
ৗʹߴෛՙঢ়ଶʹ
Slide 27
Slide 27 text
Command and Query
Responsibility Segregation (CQRS)
Slide 28
Slide 28 text
CQRS
• ॻ͖ࠐΈʢCommandʣͱಡग़͠ʢQueryʣͷ
Λ͢Δઃܭ
• ͋ΒΏΔϝιουɺΞΫγϣϯΛ࣮ߦ͢ΔίϚϯυ͔ɺݺͼ
ग़͠ݩʹσʔλΛฦ͢ΫΤϦ͔ͷ͍ͣΕ͔Ͱ͋ͬͯɺ྆ํΛ
ߦͬͯͳΒͳ͍ɻ͜Εɺ࣭Λ͢Δ͜ͱͰճΛมԽͤ͞
ͯͳΒͳ͍ͱ͍͏͜ͱͩɻ
"Greg YoungྲྀCQRS - Mark Nijhof". Digital Romanticism.
http://d.hatena.ne.jp/digitalsoul/20100712/1278886009 ࢀর
Slide 29
Slide 29 text
CQRS
"A few myths about CQRS". Ouarzy's Blog.
http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/ ࢀর
Slide 30
Slide 30 text
CQRS
• σʔλϕʔεૢ࡞ΛಡΈࠐΈɺॻ͖ࠐΈͱ
• RDBMSΛ׆͔͢
ਖ਼نԽɾτϥϯβΫγϣϯ
• αʔϏεʹ߹ΘͤͨϞσϦϯά
RDBMS·ͨElasticsearch/SolrͰฦ٫ͳͲ
• Event SourcingͱΈ߹ΘͤΔ
Slide 31
Slide 31 text
CQRSͱEvent Sourcing
σʔλϕʔεߋ৽
$PNNBOE
σʔλϕʔεߋ৽ྃΠϕϯτ
&WFOU4PVSDJOH
σʔλϕʔεߋ৽ྃݕ
&WFOU-JTUFOFS
ΩϟογϡআͳͲ
Slide 32
Slide 32 text
ϑϨʔϜϫʔΫʹΈΔQueue
• LaravelͰ͕͔͔࣌ؒΔॲཧඇಉظॲཧΉ͚ʹɺ
QueueίϯϙʔωϯτΛఏڙ
• QueueΛར༻͢ΔͨΊʹRedisRDBMSɺ
·ͨυϥΠό֦ுͰҙͷϛυϧΣΞ͕ར༻Մೳ
• ղܾʁ
LaravelͷQueueΛଞݴޠؒɾαʔϏεؒͰ͏͜ͱ
͍͠(php serializeͳͲ͕͞Ε͍ͯΔ)
Slide 33
Slide 33 text
What is Apache Kafka?
Slide 34
Slide 34 text
No content
Slide 35
Slide 35 text
ϝοηʔδϯάγεςϜ
Slide 36
Slide 36 text
ଟ༷ͳࢄܕϝοηʔδϯάϛυϧΣΞ
• Redis (PubSub)
• ZeroMQ
• RabbitMQ
• Apache Kafka
Slide 37
Slide 37 text
Redis
• KVSͷҰػೳɺPubSub(SUBSCRIBE, PUBLISHίϚϯυ)
• Redis SentinelΛར༻ͯ͠ϑΣΠϧΦʔόʔΛߦ͏
• LPUSHͳͲΛར༻ͯ͠ϝοηʔδΛอ ͳͲ
• ϝοηʔδ࠶औಘτϥϯβΫγϣϯػೳͳ͍
• ϨϓϦέʔγϣϯՄ
Slide 38
Slide 38 text
ZeroMQ
• ϛυϧΣΞͰͳ͘ɺϥΠϒϥϦ
• ΞϓϦέʔγϣϯ͕ؒ௨৴Λߦ͍ɺϝοηʔδॲཧ
• ϛυϧΣΞΛར༻͠ͳ͍ͨΊߴ
• ϝοηʔδͷӬଓԽػೳແ͍ҝɺ
ಠࣗͰ࣮͢Δඞཁ͕͋Δ
Slide 39
Slide 39 text
RabbitMQ
• Advanced Message Queuing Protocol(AMQP)
• ΤϯλʔϓϥΠζΞϓϦέʔγϣϯͰଟར༻࣮
• ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯՄೳ
• τϥϯβΫγϣϯΛར༻࣮ͯ֬͠ʹॲཧΛߦ͏
• v3.6.0ͰՃ͞ΕͨLazy QueuesͰେ༰ྔͷσʔλੵՄ
• ϑΣΠϧΦʔόʔ
Slide 40
Slide 40 text
Apache Kafka
• ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ
• ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ
• ϏοάσʔλରԠ
• ϑΝΠϧγεςϜར༻Ͱɺ
γʔέϯγϟϧΞΫηεʹΑΔߴԽ
• ετϦʔϜରԠͷϝοηʔδϯάϛυϧΣΞ
• Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍ੑ
(Amazon kinesisͱ΄΅ಉ͡)
Slide 41
Slide 41 text
Apache Kafka֓ཁ
• Producer
ϝοηʔδ৴Λߦ͏
֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻
• Consumer
ϝοηʔδߪಡΛߦ͏
ফඅ͞Εͨϝοηʔδഁغ͞Εͣɺ
Ұఆظؒอ͞ΕΔ
• Broker
KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
Slide 42
Slide 42 text
Apache ZooKeeper֓ཁ
• ઃఆཧɺ໊લղܾͳͲΛߦ͏
ࢄίʔσΟωʔγϣϯΤϯδϯ
• ෳͰߏ͢ΔZooKeeperͷ͏ͪ1͕ɺ
ͯ͢ͷॻ͖ࠐΈΛௐఀɾडཧ͢ΔϦʔμʔ
• Ϋϥελͷ͏ͪա͕ಈ࡞͢Δঢ়ଶʹ͢Δ͜ͱ͕ඞ
ཁͰ͋ΔͨΊɺحͰߏ͢Δ͜ͱ͕ඞཁ
Slide 43
Slide 43 text
Apache Kafka + ZooKeeper Architecture
Slide 44
Slide 44 text
Apache Kafka֓ཁ
• Topic
Producer͔Βͷϝοηʔδ͜ͷTopicʹ֨ೲ͞ΕΔ
ϝοηʔδҰҙʹཧɺFIFO(ޙड़partition)Ͱॲཧ
• Partition
ෛՙࢄ༻్ʹར༻
ෳͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ
ͦΕͧΕ͕ॲཧΛߦ͏
ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
Slide 45
Slide 45 text
Example Partition
Slide 46
Slide 46 text
Kafka Connect
• Kafka Connectͱɺ
पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ
σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ
• Amazon SQSMongoDBͷσʔλΛKafkaͰऔࠐΉɺ
ϝοηʔδΛͦͷ··ElasticsearchRDBMSʹ֨ೲɺ
͕ߦ͑Δ
• Connectࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮Մೳ
(java, Scala)
Slide 47
Slide 47 text
Example Kafka Connect
&WFOU4PVSDJOH
ଞΞϓϦέʔγϣϯ
$POTVNFS
ૹ৴༰Λͦͷ··
&MBTUJDTFBSDI
Slide 48
Slide 48 text
Kafka Streams
• KafkaͰετϦʔϜॲཧΛ࣮͢Δػೳ
• ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ
Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ
• Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ
• ۙ30ʹ͓͚ΔPVϥϯΩϯάɺ
ϝοηʔδͷจࣈྻมߋɺՃͳͲ
Slide 49
Slide 49 text
ϝοηʔδͷܽଛ
ૹ৴ࣦഊ
BDLड৴ࣦഊ
#SPLFSμϯ
ϝοηʔδड৴ࣦഊ
Slide 50
Slide 50 text
Exactly Once
• 0.11͔ΒՃ
• Producer, BrokerؒͷτϥϯβΫγϣϯ
Broker, ConsumerؒͷτϥϯβΫγϣϯ
• ਖ਼֬ʹҰ͚ͩૹ৴
Slide 51
Slide 51 text
ϝοηʔδୡͷอূ
• At least once semantics
ॏෳΛڐՄ
• At most once semantics
ܽଛΛڐՄ
• Exactly once semantics
ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍
"Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA.
https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
Slide 52
Slide 52 text
KSQL
• Kafkaʹอ͞Ε͍ͯΔϝοηʔδʹ
SQLΠϯλʔϑΣʔεͰΞΫηε͢Δͷ
• ݱࡏDeveloper Preview
• RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢
Slide 53
Slide 53 text
KappaΞʔΩςΫνϟ
Slide 54
Slide 54 text
KappaΞʔΩςΫνϟ
Slide 55
Slide 55 text
Apache Kafka࠾༻
• ੳॲཧΛ࣮͢Δʹ͋ͨΓɺStreamରԠ͕༰қͳࣄ
• Kafka ConnectʹΑΔσʔλετϨʔδͱͷ࿈ܞ
• Facebook Prestoͱͷ࿈ܞ͕؆୯
(RDBMSͱ݁߹ͯ͠σʔλఏڙ͕Մೳ)
• ґଘղͷͨΊʹεέʔϧ͢Δ
ϝοηʔδϯάγεςϜߏங͕༰қ
• ϝοηʔδ࠶औಘͱোରԠ͕༰қͰ͋Δࣄ
• PHPΤΫεςϯγϣϯ͕͋Δ(rdkafka)
Slide 56
Slide 56 text
Architecture
Slide 57
Slide 57 text
Kafka client for PHP
Slide 58
Slide 58 text
Rdkafka
https://github.com/arnaud-lb/
php-rdkafka
Slide 59
Slide 59 text
Rdkafka
• librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
ΤΫεςϯγϣϯ
• High Level API & Row Level API
• PHP5/7ରԠ
• librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
Slide 60
Slide 60 text
Rdkafka Producer࣮
$rk = new RdKafka\Producer();
$rk->setLogLevel(LOG_DEBUG);
$rk->addBrokers(“127.0.0.1");
$topic = $rk->newTopic("test");
$topic->produce(
RD_KAFKA_PARTITION_UA,
0,
json_encode(['message' => ‘phpcon'])
);
Slide 61
Slide 61 text
Rdkafka Producer࣮
$rk = new RdKafka\Producer();
$rk->setLogLevel(LOG_DEBUG);
$rk->addBrokers(“127.0.0.1");
$topic = $rk->newTopic("test");
$topic->produce(
RD_KAFKA_PARTITION_UA,
0,
json_encode(['message' => ‘phpcon'])
);
$MVTUFSͷ߹ɺ
ΧϯϚ۠ΓͰࢦఆ
Slide 62
Slide 62 text
Rdkafka Producer࣮
$rk = new RdKafka\Producer();
$rk->setLogLevel(LOG_DEBUG);
$rk->addBrokers(“127.0.0.1");
$topic = $rk->newTopic("test");
$topic->produce(
RD_KAFKA_PARTITION_UA,
0,
json_encode(['message' => ‘phpcon'])
);
$MVTUFSͷ߹ɺ
ΧϯϚ۠ΓͰࢦఆ
5PQJD໊
1BSUJUJPOΛࢦఆ
Slide 63
Slide 63 text
Rdkafka Consumer࣮
$conf = new RdKafka\Conf();
$conf->set('group.id', 'myConsumerGroup');
$rk = new RdKafka\Consumer($conf);
$rk->addBrokers("127.0.0.1");
$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 100);
$topicConf->set('offset.store.method', 'file');
$topicConf->set(
'offset.store.path',
sys_get_temp_dir()
);
$topicConf->set('auto.offset.reset', 'smallest');
Slide 64
Slide 64 text
Rdkafka Consumer࣮
$conf = new RdKafka\Conf();
$conf->set('group.id', 'myConsumerGroup');
$rk = new RdKafka\Consumer($conf);
$rk->addBrokers("127.0.0.1");
$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 100);
$topicConf->set('offset.store.method', 'file');
$topicConf->set(
'offset.store.path',
sys_get_temp_dir()
);
$topicConf->set('auto.offset.reset', 'smallest');
ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
ॲཧ͢Δ߹ಉ໊͡લʹ
Slide 65
Slide 65 text
Rdkafka Consumer࣮
$conf = new RdKafka\Conf();
$conf->set('group.id', 'myConsumerGroup');
$rk = new RdKafka\Consumer($conf);
$rk->addBrokers("127.0.0.1");
$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 100);
$topicConf->set('offset.store.method', 'file');
$topicConf->set(
'offset.store.path',
sys_get_temp_dir()
);
$topicConf->set('auto.offset.reset', 'smallest');
ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
ॲཧ͢Δ߹ಉ໊͡લʹ
$POTVNFSઃఆΛࢦఆ
Slide 66
Slide 66 text
Rdkafka Consumer࣮
$topic = $rk->newTopic("test", $topicConf);
$topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
while (true) {
$message = $topic->consume(0, 120*10000);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
var_dump($message);
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "No more messages; will wait for more\n";
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Timed out\n";
break;
default:
throw new \Exception($message->errstr(), $message->err);
break;
}
}
Slide 67
Slide 67 text
Rdkafka Consumer࣮
$topic = $rk->newTopic("test", $topicConf);
$topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
while (true) {
$message = $topic->consume(0, 120*10000);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
var_dump($message);
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "No more messages; will wait for more\n";
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Timed out\n";
break;
default:
throw new \Exception($message->errstr(), $message->err);
break;
}
}
Ͳͷ1BSUJUJPOΛར༻͢Δ͔
औಘҐஔͲ͔͜Β͔
Slide 68
Slide 68 text
Rdkafka Consumer࣮
$topic = $rk->newTopic("test", $topicConf);
$topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
while (true) {
$message = $topic->consume(0, 120*10000);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
var_dump($message);
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "No more messages; will wait for more\n";
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Timed out\n";
break;
default:
throw new \Exception($message->errstr(), $message->err);
break;
}
}
Ͳͷ1BSUJUJPOΛར༻͢Δ͔
औಘҐஔͲ͔͜Β͔
ਖ਼ৗʹड৴ͨ͠߹ʹ
ҙͷॲཧΛ࣮ߦ
Slide 69
Slide 69 text
Rdkafka Consumerͷઃఆ
• heartbeatͱλΠϜΞτ
heartbeat.interval.ms
session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
• SupervisordͰϓϩηεࢹඞਢ
• group.idʹΑΔॲཧͷฒྻԽ
ෳConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
Slide 70
Slide 70 text
Notice
• ΫϦʔϯͳ࣮Λ৺͕͚Δ
ConsumerPHPͱݶΒͳ͍
ΫϦʔϯͳϥΠϒϥϦɺ࣮Λ
• RdkafkaͰૹ৴͢Δ͔FluentdͳͲͰૹ৴͢Δ͔
ΞϓϦέʔγϣϯ࣍ୈ
• RdkafkaΛϥΠϒϥϦܦ༝ͳͲͰͳ͘ɺ
ͦͷ··ར༻͢Δ
Slide 71
Slide 71 text
ϝοηʔδͷόϦσʔγϣϯͲ͜Ͱʁ
• ProducerͰόϦσʔγϣϯ
-> ࠷ݶͷܕकΔ
-> ଞαʔϏεͷ͕ࣝඞཁʹͳΔ߹࣮͠ͳ͍
• ConsumerͰόϦσʔγϣϯ
ड৴ͨ͠ϝοηʔδΛௐɺ
ਖ਼͘͠ͳ͍ͷΛεΩοϓ
Slide 72
Slide 72 text
ϝοηʔδͷόϦσʔγϣϯͲ͜Ͱʁ
• Kafka Streamͱ͍͏બࢶ
java·ͨScalaͰ࣮Մೳ
-> ࣮ࣗମͦΜͳʹ͘͠ͳ͍
• αʔϏεͷࣝઐ༻ͷStreamͰߦ͏
-> ϚΠΫϩαʔϏεͰ͋Ε
Slide 73
Slide 73 text
PipelineDBͱ͍͏બࢶ
• ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ
ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠
• ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍
• PostgreSQL Compatible
• PDOͰଓՄೳʂ
Slide 74
Slide 74 text
PipelineDBͱ͍͏બࢶ
CREATE EXTENSION pipeline_kafka;
SELECT pipeline_kafka.add_broker(‘localhost:9092');
CREATE STREAM logs_stream (payload json);
CREATE CONTINUOUS VIEW message_count
AS SELECT COUNT(*) FROM logs_stream;
Slide 75
Slide 75 text
ҰาਐΜͩࢄΞϓϦέʔγϣϯ
• PrestoΛར༻͠ɺ
RDBMSRedisͱΈ߹Θͤͨσʔλऔಘ͕Մೳ
RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ
KafkaΛͬͨۙͷϩάσʔλΛֻ͚߹Θͤͨ
Ϩίϝϯυ
Slide 76
Slide 76 text
Apache Kafkaͷ࢝Ίํ
• Apache Kafkaެࣜϖʔδ͔Βμϯϩʔυ
• Confluent͔Βμϯϩʔυ
• Kafka ConnectΛར༻͢ΔͷͰ͋Εɺ
Confluentར༻͕Φεεϝ
• Hadoop͕ͳͯ͘ར༻ՄೳͰ͢
Slide 77
Slide 77 text
Apache Kafka GUI
• Clusterཧ
https://github.com/yahoo/kafka-manager
• Messageཧ
https://github.com/landoop/kafka-topics-ui
https://github.com/ldaniels528/trifecta
ͳͲ
Slide 78
Slide 78 text
·ͱΊ
• ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾࢄΞϓϦέʔγϣϯ
ͷղܾͷҰͭ
• KafkaҎ֎ͷMessage Queueબࢶʹ
• StreamॲཧͳͲ͕ඞཁͰ͋ΕKafkaΛਪ
• نʹ͋ΘͤͨϛυϧΣΞબఆɾোରԠɾӡ༻Λ