Apache Kafkaによるスケーラブル アプリケーション開発

PHPカンファレンス2017で利用したスライド

Transcript

1. Apache KafkaʹΑΔ εέʔϥϒϧ ΞϓϦέʔγϣϯ։ൃ yuuki takezawa <ytake>
 PHP Conference 2017
2. ͸͡Ίʹ • ن໛ಁաੑ
 ෛՙͷߴ௿ʹ߹ΘͤͯϦ
 ιʔεɾϓʔϧΛ֦େɾॖখͰ͖Δ͜ͱ • Ґஔಁաੑ
 Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ • ҟछಁաੑ
 γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜ ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ
3. Agenda • Message Queue • What is Apache Kafka? • For PHP
4. Message Queue
5. ίϝϯτ౤ߘαʔϏε • ϩάΠϯϢʔβʔ͸ίϝϯτ͕౤ߘͰ͖Δ • ະϩάΠϯϢʔβʔ͸ӾཡͷΈ͕Ͱ͖Δ • ͓ؾʹೖΓͨ͠ϢʔβʔͷίϝϯτΛӾཡͰ͖Δ • ࣌ܥྻͰίϝϯτΛӾཡͰ͖Δ • ίϝϯτͷ࡟আ͕Ͱ͖Δ • ίϝϯτͷฤू͸Ͱ͖ͳ͍ ͳͲ
6. ΞϓϦέʔγϣϯ࣮૷ • ೝূػೳͷ࣮૷ • ίϝϯτ౤ߘɺฤूɺӾཡػೳ࣮૷ • ݕࡧػೳͷ࣮૷ • ਺ਓͷ։ൃऀ
7. γϯϓϧͳΞϓϦέʔγϣϯߏ੒
8. γϯϓϧͳΞϓϦέʔγϣϯߏ੒ public function __invoke(Comment $comment) { $query = $comment->query(); $query->insert(['title', 'body']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'id' => $this->auth->getId(), ]) ->execute(); }
9. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ • ػೳ௥Ճ
 • ίϝϯτ৘ใΛଞͷػೳͰར༻͍ͨ͠
 • ৚݅ʹΑΔҰཡ৘ใͷιʔτมߋ
 Ϣʔβʔ৘ใଐੑʹΑΔιʔτॱมߋ
10. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
11. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ public function __invoke(Comment $comment, Movie $movie) { $query = $comment->query(); $commentId = $query->insert(['title', 'body', 'comment_id']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'comment_id' => $this->auth->getId(), ]) ->execute(); $movieQuery = $movie->query(); $movieQuery->insert(['movie_id', 'comment_id']) ->values([ 'movie_id' => $movie->getId(), 'comment_id' => $commentId, ]) ->execute(); }
12. Ϗδωεͷ੒௕ͱͱ΋ʹ ංେԽ͢ΔΞϓϦέʔγϣϯ
13. ංେԽ͢ΔΞϓϦέʔγϣϯ • ऩӹԽͷͨΊͷػೳ
 • རศੑͷͨΊͷػೳ • طଘػೳͷ֦ு
14. ංେԽ͢ΔΞϓϦέʔγϣϯ
15. ංେԽ͢ΔΞϓϦέʔγϣϯ public function __invoke( Comment $comment, Movie $movie, Follow $follow, Company $company ) { $movieCommentId = false; if ($follow->isFollow()) { $query = $comment->query(); $commentId = $query->insert(['title', 'body', 'comment_id']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'comment_id' => $this->auth->getId(), ]) ->execute(); $movieQuery = $movie->query(); $movieCommentId = $movieQuery->insert(['movie_id', 'comment_id']) ->values([ 'movie_id' => $movie->getId(), 'comment_id' => $commentId, ]) ->execute(); } if ($movie->isCompany() && $movieCommentId) { $company->appendMovie($movie->getId()); // ϝʔϧૹ৴ } }
16. ΞϓϦέʔγϣϯͷΫϥεͦͷ΋ͷΛ ίϯϙʔωϯτԽ ґଘੑΛऔΓআ͘
17. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά • ݁߹ͨ͠ػೳΛίϯϙʔωϯτఏڙ
 • APIʹΑΔσʔλૢ࡞ͷந৅Խ
 • σʔλϕʔεύϑΥʔϚϯεվળ

18. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά
19. ڊେͳΞϓϦέʔγϣϯ΁ • αʔϏε֦େʹͱ΋ͳ͏ΞϓϦέʔγϣϯ௥Ճ
 • ґଘίϯϙʔωϯτ૿Ճ
 -> ࢓༷มߋٵऩ • ૿Ճ͢ΔAPIར༻
 -> ᷓΕग़͢ଞαʔϏεͷϏδωεϩδοΫ • σʔλϕʔε໰୊
 -> ElasticsearchɺSolrͳͲͷಋೖ
20. ڊେͳΞϓϦέʔγϣϯ΁
21. ਂࠁͳ໰୊ • ࢓༷มߋʹ൐͏ίϯϙʔωϯτमਖ਼ͱϦϦʔεෳࡶԽ
 ϥΠϒϥϦͷόʔδϣϯ͋͛ͳ͖Όɾɾʂ
 ͋ͷػೳΛऔΓࠐΉʹ͸͋Εͱ͜Εͱɾɾɾ • ΞϓϦέʔγϣϯؒ࿈ܞͷෳࡶԽ
 APIमਖ਼ʹ൐͏Өڹൣғௐࠪ
 վमͯ͠΋࢖ͬͯ͘Εͳ͍໰୊
22. ਂࠁͳ໰୊ • σʔλಉظͷෳࡶԽ
 ϦΞϧλΠϜͰσʔλ͕ཉ͍͠
 ΠϯσοΫεߋ৽͕ఆظ࣮ߦ
 • ͦͷଞ
 Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ
 • ෳࡶԽ͢Δ࣮૷ίʔυ
23. ੒௕͍ͯ͘͠ࣄۀʹ߹Θͤͯɺ ΞϓϦέʔγϣϯΛ Ͳ͏εέʔϧ͍͔ͤͯ͘͞
24. มΘΓΏ͘؀ڥ • σʔλΛऔಘ͢Δʹ͸ʁ
 APIίʔϧʹΑΔ౎౓֬ೝʁ
 Ωϟογϡ͸Կ෼ʁ
 • ͦΕͧΕͷαʔϏεͰͦΕͧΕͷݴޠ΁
 ͋ΔαʔϏε͸PHPɺ͋ΔαʔϏεͰ͸Go ScalaͳͲ
 ϥΠϒϥϦ͕ෛͷҨ࢈Խ

25. σʔλॲཧ Ϣʔβʔ৘ใ͸౎౓໰͍߹Θͤ ৘ใΛߋ৽ͨ͠৔߹Ͱ΋
 ґଘαʔϏε͕ͳ͍ͨΊ໰୊ͳ͠
26. σʔλॲཧͷෳࡶԽ Ϣʔβʔ৘ใมߋ Ϣʔβʔ৘ใ࡟আ Ϣʔβʔ৘ใ͕ݟ͔ͭΒͳ͍ͨΊ
 σʔλॲཧ͝ͱʹ"1*ΞΫηε ͋ͦ͜ͱͦ͜Ͱ৘ใ͕ҧ͏ʂ
 Ͳͷσʔλ͕ຊ෺ʁ શͯͷΞϓϦέʔγϣϯ͔Β
 Ωϟογϡͳ͠ͰΞΫηε
 ৗʹߴෛՙঢ়ଶʹ
27. Command and Query Responsibility Segregation (CQRS)
28. CQRS • ॻ͖ࠐΈʢCommandʣͱಡग़͠ʢQueryʣͷ
 ੹຿Λ෼཭͢Δઃܭ
 • ͋ΒΏΔϝιου͸ɺΞΫγϣϯΛ࣮ߦ͢ΔίϚϯυ͔ɺݺͼ ग़͠ݩʹσʔλΛฦ͢ΫΤϦ͔ͷ͍ͣΕ͔Ͱ͋ͬͯɺ྆ํΛ ߦͬͯ͸ͳΒͳ͍ɻ͜Ε͸ɺ࣭໰Λ͢Δ͜ͱͰճ౴ΛมԽͤ͞ ͯ͸ͳΒͳ͍ͱ͍͏͜ͱͩɻ
 
 "Greg YoungྲྀCQRS - Mark Nijhof". Digital Romanticism. 
 http://d.hatena.ne.jp/digitalsoul/20100712/1278886009 ࢀর 

29. CQRS "A few myths about CQRS". Ouarzy's Blog. 
 http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/ ࢀর
30. CQRS • σʔλϕʔεૢ࡞ΛಡΈࠐΈɺॻ͖ࠐΈͱ෼཭
 • RDBMSΛ׆͔͢
 ਖ਼نԽɾτϥϯβΫγϣϯ
 • αʔϏεʹ߹ΘͤͨϞσϦϯά
 RDBMS·ͨ͸Elasticsearch/SolrͰฦ٫ͳͲ
 • Event Sourcingͱ૊Έ߹ΘͤΔ
31. CQRSͱEvent Sourcing σʔλϕʔεߋ৽
 $PNNBOE σʔλϕʔεߋ৽׬ྃΠϕϯτ
 &WFOU
    4PVSDJOH σʔλϕʔεߋ৽׬ྃݕ஌
 &WFOU
    -JTUFOFS Ωϟογϡ࡟আͳͲ
32. ϑϨʔϜϫʔΫʹΈΔQueue • LaravelͰ͸͕͔͔࣌ؒΔॲཧ΍ඇಉظॲཧΉ͚ʹɺ
 QueueίϯϙʔωϯτΛఏڙ • QueueΛར༻͢ΔͨΊʹRedis΍RDBMSɺ
 ·ͨ͸υϥΠό֦ுͰ೚ҙͷϛυϧ΢ΣΞ͕ར༻Մೳ • ղܾʁ
 LaravelͷQueueΛଞݴޠؒɾαʔϏεؒͰ࢖͏͜ͱ͸ ೉͍͠(php serializeͳͲ͕͞Ε͍ͯΔ)
33. What is Apache Kafka?
35. ϝοηʔδϯάγεςϜ
36. ଟ༷ͳ෼ࢄܕϝοηʔδϯάϛυϧ΢ΣΞ • Redis (PubSub) • ZeroMQ • RabbitMQ • Apache Kafka
37. Redis • KVSͷҰػೳɺPubSub(SUBSCRIBE, PUBLISHίϚϯυ) • Redis SentinelΛར༻ͯ͠ϑΣΠϧΦʔόʔΛߦ͏ • LPUSHͳͲΛར༻ͯ͠ϝοηʔδΛอ؅ ͳͲ • ϝοηʔδ࠶औಘ΍τϥϯβΫγϣϯػೳ͸ͳ͍ • ϨϓϦέʔγϣϯՄ
38. ZeroMQ • ϛυϧ΢ΣΞͰ͸ͳ͘ɺϥΠϒϥϦ • ΞϓϦέʔγϣϯ͕ؒ௨৴Λߦ͍ɺϝοηʔδॲཧ • ϛυϧ΢ΣΞΛར༻͠ͳ͍ͨΊߴ଎ • ϝοηʔδͷӬଓԽػೳ͸ແ͍ҝɺ
 ಠࣗͰ࣮૷͢Δඞཁ͕͋Δ
39. RabbitMQ • Advanced Message Queuing Protocol(AMQP) • ΤϯλʔϓϥΠζΞϓϦέʔγϣϯͰଟ਺ར༻࣮੷ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯՄೳ • τϥϯβΫγϣϯΛར༻࣮ͯ֬͠ʹॲཧΛߦ͏ • v3.6.0Ͱ௥Ճ͞ΕͨLazy QueuesͰେ༰ྔͷσʔλ஝ੵՄ • ϑΣΠϧΦʔόʔ
40. Apache Kafka • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ • ϏοάσʔλରԠ • ϑΝΠϧγεςϜར༻Ͱɺ
 γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ
 (Amazon kinesisͱ΄΅ಉ͡)
41. Apache Kafka֓ཁ • Producer 
 ϝοηʔδ഑৴Λߦ͏
 ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻ • Consumer 
 ϝοηʔδߪಡΛߦ͏
 ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ
 Ұఆظؒอ؅͞ΕΔ • Broker
 KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
42. Apache ZooKeeper֓ཁ • ઃఆ؅ཧ΍ɺ໊લղܾͳͲΛߦ͏
 ෼ࢄίʔσΟωʔγϣϯΤϯδϯ • ෳ਺୆Ͱߏ੒͢ΔZooKeeperͷ͏ͪ1୆͕ɺ
 ͢΂ͯͷॻ͖ࠐΈΛௐఀɾडཧ͢ΔϦʔμʔ • Ϋϥελͷ͏ͪա൒਺͕ಈ࡞͢Δঢ়ଶʹ͢Δ͜ͱ͕ඞ ཁͰ͋ΔͨΊɺح਺୆Ͱߏ੒͢Δ͜ͱ͕ඞཁ
43. Apache Kafka + ZooKeeper Architecture
44. Apache Kafka֓ཁ • Topic 
 Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ
 ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ • Partition
 ෛՙ෼ࢄ༻్ʹར༻
 ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ
 ͦΕͧΕ͕ॲཧΛߦ͏
 ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
45. Example Partition
46. Kafka Connect • Kafka Connectͱ͸ɺ
 पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ
 σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ
 ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ
 ͕ߦ͑Δ • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ
 (java, Scala)
47. Example Kafka Connect &WFOU
    4PVSDJOH ଞΞϓϦέʔγϣϯ
 $POTVNFS ૹ৴಺༰Λͦͷ··
 &MBTUJDTFBSDI
    ΁
48. Kafka Streams • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ
 Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ • ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ
 ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ
49. ϝοηʔδͷܽଛ ૹ৴ࣦഊ BDLड৴ࣦഊ #SPLFSμ΢ϯ ϝοηʔδड৴ࣦഊ
50. Exactly Once • 0.11͔Β௥Ճ • Producer, BrokerؒͷτϥϯβΫγϣϯ
 Broker, ConsumerؒͷτϥϯβΫγϣϯ • ਖ਼֬ʹҰ౓͚ͩૹ৴
51. ϝοηʔδ఻ୡͷอূ • At least once semantics
 ॏෳΛڐՄ • At most once semantics
 ܽଛΛڐՄ • Exactly once semantics
 ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍
 
 "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 
 https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
52. KSQL • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ
 SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ • ݱࡏDeveloper Preview • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢
53. KappaΞʔΩςΫνϟ
54. KappaΞʔΩςΫνϟ
55. Apache Kafka࠾༻ • ෼ੳॲཧΛ࣮૷͢Δʹ͋ͨΓɺStreamରԠ͕༰қͳࣄ • Kafka ConnectʹΑΔσʔλετϨʔδͱͷ࿈ܞ • Facebook Prestoͱͷ࿈ܞ͕؆୯
 (RDBMS౳ͱ݁߹ͯ͠σʔλఏڙ͕Մೳ) • ґଘ෼ղͷͨΊʹεέʔϧ͢Δ
 ϝοηʔδϯάγεςϜߏங͕༰қ • ϝοηʔδ࠶औಘͱো֐ରԠ͕༰қͰ͋Δࣄ • PHPΤΫεςϯγϣϯ͕͋Δ(rdkafka)
56. Architecture
57. Kafka client for PHP
58. Rdkafka https://github.com/arnaud-lb/ php-rdkafka
59. Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
60. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );
61. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ
62. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ
63. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');
64. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
    ॲཧ͢Δ৔߹͸ಉ໊͡લʹ
65. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
    ॲཧ͢Δ৔߹͸ಉ໊͡લʹ $POTVNFSઃఆ஋Λࢦఆ
66. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }
67. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔
68. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ
69. Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ
 ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
70. Notice • ΫϦʔϯͳ࣮૷Λ৺͕͚Δ
 Consumer͸PHPͱ͸ݶΒͳ͍
 ΫϦʔϯͳϥΠϒϥϦɺ࣮૷Λ • RdkafkaͰૹ৴͢Δ͔FluentdͳͲͰૹ৴͢Δ͔͸
 ΞϓϦέʔγϣϯ࣍ୈ • RdkafkaΛϥΠϒϥϦܦ༝ͳͲͰ͸ͳ͘ɺ
 ͦͷ··ར༻͢Δ
71. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • ProducerͰόϦσʔγϣϯ
 -> ࠷௿ݶͷܕकΔ
 -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍
 • ConsumerͰόϦσʔγϣϯ
 ड৴ͨ͠ϝοηʔδΛௐ΂ɺ
 ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ
72. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • Kafka Streamͱ͍͏બ୒ࢶ
 java·ͨ͸ScalaͰ࣮૷Մೳ
 -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍
 • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏
 -> ϚΠΫϩαʔϏεͰ͋Ε͹
73. PipelineDBͱ͍͏બ୒ࢶ • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ
 ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠
 • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍
 • PostgreSQL Compatible
 • PDOͰ઀ଓՄೳʂ
74. PipelineDBͱ͍͏બ୒ࢶ CREATE EXTENSION pipeline_kafka; SELECT pipeline_kafka.add_broker(‘localhost:9092'); CREATE STREAM logs_stream (payload json); CREATE CONTINUOUS VIEW message_count AS SELECT COUNT(*) FROM logs_stream;
75. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ • PrestoΛར༻͠ɺ
 RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ
 RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ
 KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλΛֻ͚߹Θͤͨ
 Ϩίϝϯυ
76. Apache Kafkaͷ࢝Ίํ • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ • Confluent͔Βμ΢ϯϩʔυ • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ
 Confluentར༻͕Φεεϝ • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢
77. Apache Kafka GUI • Cluster؅ཧ 
 https://github.com/yahoo/kafka-manager • Message؅ཧ 
 https://github.com/landoop/kafka-topics-ui
 https://github.com/ldaniels528/trifecta
 ͳͲ
78. ·ͱΊ • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ ͷ໰୊ղܾͷҰͭ • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑ • ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ