Apache Kafkaによるスケーラブル アプリケーション開発

Apache Kafkaによるスケーラブル アプリケーション開発

PHPカンファレンス2017で利用したスライド

17d4ef53b432ebf7c566fd6a11345570?s=128

yuuki takezawa

October 08, 2017
Tweet

Transcript

  1. Apache KafkaʹΑΔ εέʔϥϒϧ ΞϓϦέʔγϣϯ։ൃ yuuki takezawa <ytake>
 PHP Conference 2017

  2. ͸͡Ίʹ • ن໛ಁաੑ
 ෛՙͷߴ௿ʹ߹ΘͤͯϦ
 ιʔεɾϓʔϧΛ֦େɾॖখͰ͖Δ͜ͱ • Ґஔಁաੑ
 Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ •

    ҟछಁաੑ
 γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜ ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ
  3. Agenda • Message Queue • What is Apache Kafka? •

    For PHP
  4. Message Queue

  5. ίϝϯτ౤ߘαʔϏε • ϩάΠϯϢʔβʔ͸ίϝϯτ͕౤ߘͰ͖Δ • ະϩάΠϯϢʔβʔ͸ӾཡͷΈ͕Ͱ͖Δ • ͓ؾʹೖΓͨ͠ϢʔβʔͷίϝϯτΛӾཡͰ͖Δ • ࣌ܥྻͰίϝϯτΛӾཡͰ͖Δ •

    ίϝϯτͷ࡟আ͕Ͱ͖Δ • ίϝϯτͷฤू͸Ͱ͖ͳ͍ ͳͲ
  6. ΞϓϦέʔγϣϯ࣮૷ • ೝূػೳͷ࣮૷ • ίϝϯτ౤ߘɺฤूɺӾཡػೳ࣮૷ • ݕࡧػೳͷ࣮૷ • ਺ਓͷ։ൃऀ

  7. γϯϓϧͳΞϓϦέʔγϣϯߏ੒

  8. γϯϓϧͳΞϓϦέʔγϣϯߏ੒ public function __invoke(Comment $comment) { $query = $comment->query(); $query->insert(['title',

    'body']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'id' => $this->auth->getId(), ]) ->execute(); }
  9. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ • ػೳ௥Ճ
 • ίϝϯτ৘ใΛଞͷػೳͰར༻͍ͨ͠
 • ৚݅ʹΑΔҰཡ৘ใͷιʔτมߋ
 Ϣʔβʔ৘ใଐੑʹΑΔιʔτॱมߋ

  10. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ

  11. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ public function __invoke(Comment $comment, Movie $movie) { $query =

    $comment->query(); $commentId = $query->insert(['title', 'body', 'comment_id']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'comment_id' => $this->auth->getId(), ]) ->execute(); $movieQuery = $movie->query(); $movieQuery->insert(['movie_id', 'comment_id']) ->values([ 'movie_id' => $movie->getId(), 'comment_id' => $commentId, ]) ->execute(); }
  12. Ϗδωεͷ੒௕ͱͱ΋ʹ ංେԽ͢ΔΞϓϦέʔγϣϯ

  13. ංେԽ͢ΔΞϓϦέʔγϣϯ • ऩӹԽͷͨΊͷػೳ
 • རศੑͷͨΊͷػೳ • طଘػೳͷ֦ு

  14. ංେԽ͢ΔΞϓϦέʔγϣϯ

  15. ංେԽ͢ΔΞϓϦέʔγϣϯ public function __invoke( Comment $comment, Movie $movie, Follow $follow,

    Company $company ) { $movieCommentId = false; if ($follow->isFollow()) { $query = $comment->query(); $commentId = $query->insert(['title', 'body', 'comment_id']) ->values([ 'title' => 'First post', 'body' => 'Some body text', 'comment_id' => $this->auth->getId(), ]) ->execute(); $movieQuery = $movie->query(); $movieCommentId = $movieQuery->insert(['movie_id', 'comment_id']) ->values([ 'movie_id' => $movie->getId(), 'comment_id' => $commentId, ]) ->execute(); } if ($movie->isCompany() && $movieCommentId) { $company->appendMovie($movie->getId()); // ϝʔϧૹ৴ } }
  16. ΞϓϦέʔγϣϯͷΫϥεͦͷ΋ͷΛ ίϯϙʔωϯτԽ ґଘੑΛऔΓআ͘

  17. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά • ݁߹ͨ͠ػೳΛίϯϙʔωϯτఏڙ
 • APIʹΑΔσʔλૢ࡞ͷந৅Խ
 • σʔλϕʔεύϑΥʔϚϯεվળ


  18. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά

  19. ڊେͳΞϓϦέʔγϣϯ΁ • αʔϏε֦େʹͱ΋ͳ͏ΞϓϦέʔγϣϯ௥Ճ
 • ґଘίϯϙʔωϯτ૿Ճ
 -> ࢓༷มߋٵऩ • ૿Ճ͢ΔAPIར༻
 ->

    ᷓΕग़͢ଞαʔϏεͷϏδωεϩδοΫ • σʔλϕʔε໰୊
 -> ElasticsearchɺSolrͳͲͷಋೖ
  20. ڊେͳΞϓϦέʔγϣϯ΁

  21. ਂࠁͳ໰୊ • ࢓༷มߋʹ൐͏ίϯϙʔωϯτमਖ਼ͱϦϦʔεෳࡶԽ
 ϥΠϒϥϦͷόʔδϣϯ͋͛ͳ͖Όɾɾʂ
 ͋ͷػೳΛऔΓࠐΉʹ͸͋Εͱ͜Εͱɾɾɾ • ΞϓϦέʔγϣϯؒ࿈ܞͷෳࡶԽ
 APIमਖ਼ʹ൐͏Өڹൣғௐࠪ
 վमͯ͠΋࢖ͬͯ͘Εͳ͍໰୊

  22. ਂࠁͳ໰୊ • σʔλಉظͷෳࡶԽ
 ϦΞϧλΠϜͰσʔλ͕ཉ͍͠
 ΠϯσοΫεߋ৽͕ఆظ࣮ߦ
 • ͦͷଞ
 Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ
 • ෳࡶԽ͢Δ࣮૷ίʔυ

  23. ੒௕͍ͯ͘͠ࣄۀʹ߹Θͤͯɺ ΞϓϦέʔγϣϯΛ Ͳ͏εέʔϧ͍͔ͤͯ͘͞

  24. มΘΓΏ͘؀ڥ • σʔλΛऔಘ͢Δʹ͸ʁ
 APIίʔϧʹΑΔ౎౓֬ೝʁ
 Ωϟογϡ͸Կ෼ʁ
 • ͦΕͧΕͷαʔϏεͰͦΕͧΕͷݴޠ΁
 ͋ΔαʔϏε͸PHPɺ͋ΔαʔϏεͰ͸Go ScalaͳͲ
 ϥΠϒϥϦ͕ෛͷҨ࢈Խ


  25. σʔλॲཧ Ϣʔβʔ৘ใ͸౎౓໰͍߹Θͤ ৘ใΛߋ৽ͨ͠৔߹Ͱ΋
 ґଘαʔϏε͕ͳ͍ͨΊ໰୊ͳ͠

  26. σʔλॲཧͷෳࡶԽ Ϣʔβʔ৘ใมߋ Ϣʔβʔ৘ใ࡟আ Ϣʔβʔ৘ใ͕ݟ͔ͭΒͳ͍ͨΊ
 σʔλॲཧ͝ͱʹ"1*ΞΫηε ͋ͦ͜ͱͦ͜Ͱ৘ใ͕ҧ͏ʂ
 Ͳͷσʔλ͕ຊ෺ʁ શͯͷΞϓϦέʔγϣϯ͔Β
 Ωϟογϡͳ͠ͰΞΫηε
 ৗʹߴෛՙঢ়ଶʹ

  27. Command and Query Responsibility Segregation (CQRS)

  28. CQRS • ॻ͖ࠐΈʢCommandʣͱಡग़͠ʢQueryʣͷ
 ੹຿Λ෼཭͢Δઃܭ
 • ͋ΒΏΔϝιου͸ɺΞΫγϣϯΛ࣮ߦ͢ΔίϚϯυ͔ɺݺͼ ग़͠ݩʹσʔλΛฦ͢ΫΤϦ͔ͷ͍ͣΕ͔Ͱ͋ͬͯɺ྆ํΛ ߦͬͯ͸ͳΒͳ͍ɻ͜Ε͸ɺ࣭໰Λ͢Δ͜ͱͰճ౴ΛมԽͤ͞ ͯ͸ͳΒͳ͍ͱ͍͏͜ͱͩɻ
 


    "Greg YoungྲྀCQRS - Mark Nijhof". Digital Romanticism. 
 http://d.hatena.ne.jp/digitalsoul/20100712/1278886009 ࢀর 

  29. CQRS "A few myths about CQRS". Ouarzy's Blog. 
 http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/

    ࢀর
  30. CQRS • σʔλϕʔεૢ࡞ΛಡΈࠐΈɺॻ͖ࠐΈͱ෼཭
 • RDBMSΛ׆͔͢
 ਖ਼نԽɾτϥϯβΫγϣϯ
 • αʔϏεʹ߹ΘͤͨϞσϦϯά
 RDBMS·ͨ͸Elasticsearch/SolrͰฦ٫ͳͲ
 •

    Event Sourcingͱ૊Έ߹ΘͤΔ
  31. CQRSͱEvent Sourcing σʔλϕʔεߋ৽
 $PNNBOE σʔλϕʔεߋ৽׬ྃΠϕϯτ
 &WFOU4PVSDJOH σʔλϕʔεߋ৽׬ྃݕ஌
 &WFOU-JTUFOFS Ωϟογϡ࡟আͳͲ

  32. ϑϨʔϜϫʔΫʹΈΔQueue • LaravelͰ͸͕͔͔࣌ؒΔॲཧ΍ඇಉظॲཧΉ͚ʹɺ
 QueueίϯϙʔωϯτΛఏڙ • QueueΛར༻͢ΔͨΊʹRedis΍RDBMSɺ
 ·ͨ͸υϥΠό֦ுͰ೚ҙͷϛυϧ΢ΣΞ͕ར༻Մೳ • ղܾʁ
 LaravelͷQueueΛଞݴޠؒɾαʔϏεؒͰ࢖͏͜ͱ͸

    ೉͍͠(php serializeͳͲ͕͞Ε͍ͯΔ)
  33. What is Apache Kafka?

  34. None
  35. ϝοηʔδϯάγεςϜ

  36. ଟ༷ͳ෼ࢄܕϝοηʔδϯάϛυϧ΢ΣΞ • Redis (PubSub) • ZeroMQ • RabbitMQ • Apache

    Kafka
  37. Redis • KVSͷҰػೳɺPubSub(SUBSCRIBE, PUBLISHίϚϯυ) • Redis SentinelΛར༻ͯ͠ϑΣΠϧΦʔόʔΛߦ͏ • LPUSHͳͲΛར༻ͯ͠ϝοηʔδΛอ؅ ͳͲ

    • ϝοηʔδ࠶औಘ΍τϥϯβΫγϣϯػೳ͸ͳ͍ • ϨϓϦέʔγϣϯՄ
  38. ZeroMQ • ϛυϧ΢ΣΞͰ͸ͳ͘ɺϥΠϒϥϦ • ΞϓϦέʔγϣϯ͕ؒ௨৴Λߦ͍ɺϝοηʔδॲཧ • ϛυϧ΢ΣΞΛར༻͠ͳ͍ͨΊߴ଎ • ϝοηʔδͷӬଓԽػೳ͸ແ͍ҝɺ
 ಠࣗͰ࣮૷͢Δඞཁ͕͋Δ

  39. RabbitMQ • Advanced Message Queuing Protocol(AMQP) • ΤϯλʔϓϥΠζΞϓϦέʔγϣϯͰଟ਺ར༻࣮੷ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯՄೳ

    • τϥϯβΫγϣϯΛར༻࣮ͯ֬͠ʹॲཧΛߦ͏ • v3.6.0Ͱ௥Ճ͞ΕͨLazy QueuesͰେ༰ྔͷσʔλ஝ੵՄ • ϑΣΠϧΦʔόʔ
  40. Apache Kafka • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ • ϏοάσʔλରԠ • ϑΝΠϧγεςϜར༻Ͱɺ


    γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ
 (Amazon kinesisͱ΄΅ಉ͡)
  41. Apache Kafka֓ཁ • Producer 
 ϝοηʔδ഑৴Λߦ͏
 ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻ • Consumer 


    ϝοηʔδߪಡΛߦ͏
 ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ
 Ұఆظؒอ؅͞ΕΔ • Broker
 KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
  42. Apache ZooKeeper֓ཁ • ઃఆ؅ཧ΍ɺ໊લղܾͳͲΛߦ͏
 ෼ࢄίʔσΟωʔγϣϯΤϯδϯ • ෳ਺୆Ͱߏ੒͢ΔZooKeeperͷ͏ͪ1୆͕ɺ
 ͢΂ͯͷॻ͖ࠐΈΛௐఀɾडཧ͢ΔϦʔμʔ • Ϋϥελͷ͏ͪա൒਺͕ಈ࡞͢Δঢ়ଶʹ͢Δ͜ͱ͕ඞ

    ཁͰ͋ΔͨΊɺح਺୆Ͱߏ੒͢Δ͜ͱ͕ඞཁ
  43. Apache Kafka + ZooKeeper Architecture

  44. Apache Kafka֓ཁ • Topic 
 Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ
 ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ • Partition
 ෛՙ෼ࢄ༻్ʹར༻


    ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ
 ͦΕͧΕ͕ॲཧΛߦ͏
 ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
  45. Example Partition

  46. Kafka Connect • Kafka Connectͱ͸ɺ
 पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ
 σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ


    ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ
 ͕ߦ͑Δ • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ
 (java, Scala)
  47. Example Kafka Connect &WFOU4PVSDJOH ଞΞϓϦέʔγϣϯ
 $POTVNFS ૹ৴಺༰Λͦͷ··
 &MBTUJDTFBSDI΁

  48. Kafka Streams • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ
 Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ •

    ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ
 ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ
  49. ϝοηʔδͷܽଛ ૹ৴ࣦഊ BDLड৴ࣦഊ #SPLFSμ΢ϯ ϝοηʔδड৴ࣦഊ

  50. Exactly Once • 0.11͔Β௥Ճ • Producer, BrokerؒͷτϥϯβΫγϣϯ
 Broker, ConsumerؒͷτϥϯβΫγϣϯ •

    ਖ਼֬ʹҰ౓͚ͩૹ৴
  51. ϝοηʔδ఻ୡͷอূ • At least once semantics
 ॏෳΛڐՄ • At most

    once semantics
 ܽଛΛڐՄ • Exactly once semantics
 ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍
 
 "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 
 https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
  52. KSQL • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ
 SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ • ݱࡏDeveloper Preview • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢

  53. KappaΞʔΩςΫνϟ

  54. KappaΞʔΩςΫνϟ

  55. Apache Kafka࠾༻ • ෼ੳॲཧΛ࣮૷͢Δʹ͋ͨΓɺStreamରԠ͕༰қͳࣄ • Kafka ConnectʹΑΔσʔλετϨʔδͱͷ࿈ܞ • Facebook Prestoͱͷ࿈ܞ͕؆୯


    (RDBMS౳ͱ݁߹ͯ͠σʔλఏڙ͕Մೳ) • ґଘ෼ղͷͨΊʹεέʔϧ͢Δ
 ϝοηʔδϯάγεςϜߏங͕༰қ • ϝοηʔδ࠶औಘͱো֐ରԠ͕༰қͰ͋Δࣄ • PHPΤΫεςϯγϣϯ͕͋Δ(rdkafka)
  56. Architecture

  57. Kafka client for PHP

  58. Rdkafka https://github.com/arnaud-lb/ php-rdkafka

  59. Rdkafka • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠
 ΤΫεςϯγϣϯ • High Level API & Row

    Level API • PHP5/7ରԠ • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
  60. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) );
  61. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ
  62. Rdkafka Producer࣮૷ $rk = new RdKafka\Producer(); $rk->setLogLevel(LOG_DEBUG); $rk->addBrokers(“127.0.0.1"); $topic =

    $rk->newTopic("test"); $topic->produce( RD_KAFKA_PARTITION_UA, 0, json_encode(['message' => ‘phpcon']) ); $MVTUFSͷ৔߹͸ɺ
 ΧϯϚ۠੾ΓͰࢦఆ 5PQJD໊ 1BSUJUJPOΛࢦఆ
  63. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest');
  64. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); ฒྻͰಉ͡UPQJDΛಉ࣌ʹ ॲཧ͢Δ৔߹͸ಉ໊͡લʹ
  65. Rdkafka Consumer࣮૷ $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topicConf->set('offset.store.method', 'file'); $topicConf->set( 'offset.store.path', sys_get_temp_dir() ); $topicConf->set('auto.offset.reset', 'smallest'); ฒྻͰಉ͡UPQJDΛಉ࣌ʹ ॲཧ͢Δ৔߹͸ಉ໊͡લʹ $POTVNFSઃఆ஋Λࢦఆ
  66. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } }
  67. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔
  68. Rdkafka Consumer࣮૷ $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true)

    { $message = $topic->consume(0, 120*10000); switch ($message->err) { case RD_KAFKA_RESP_ERR_NO_ERROR: var_dump($message); break; case RD_KAFKA_RESP_ERR__PARTITION_EOF: echo "No more messages; will wait for more\n"; break; case RD_KAFKA_RESP_ERR__TIMED_OUT: echo "Timed out\n"; break; default: throw new \Exception($message->errstr(), $message->err); break; } } Ͳͷ1BSUJUJPOΛར༻͢Δ͔
 औಘҐஔ͸Ͳ͔͜Β͔ ਖ਼ৗʹड৴ͨ͠৔߹ʹ
 ೚ҙͷॲཧΛ࣮ߦ
  69. Rdkafka Consumerͷઃఆ • heartbeatͱλΠϜΞ΢τ
 heartbeat.interval.ms͸
 session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘
 • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ • group.idʹΑΔॲཧͷฒྻԽ


    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
  70. Notice • ΫϦʔϯͳ࣮૷Λ৺͕͚Δ
 Consumer͸PHPͱ͸ݶΒͳ͍
 ΫϦʔϯͳϥΠϒϥϦɺ࣮૷Λ • RdkafkaͰૹ৴͢Δ͔FluentdͳͲͰૹ৴͢Δ͔͸
 ΞϓϦέʔγϣϯ࣍ୈ • RdkafkaΛϥΠϒϥϦܦ༝ͳͲͰ͸ͳ͘ɺ


    ͦͷ··ར༻͢Δ
  71. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • ProducerͰόϦσʔγϣϯ
 -> ࠷௿ݶͷܕकΔ
 -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍
 • ConsumerͰόϦσʔγϣϯ
 ड৴ͨ͠ϝοηʔδΛௐ΂ɺ


    ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ
  72. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ • Kafka Streamͱ͍͏બ୒ࢶ
 java·ͨ͸ScalaͰ࣮૷Մೳ
 -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍
 • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏
 ->

    ϚΠΫϩαʔϏεͰ͋Ε͹
  73. PipelineDBͱ͍͏બ୒ࢶ • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ
 ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠
 • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍
 • PostgreSQL Compatible


    • PDOͰ઀ଓՄೳʂ
  74. PipelineDBͱ͍͏બ୒ࢶ CREATE EXTENSION pipeline_kafka; SELECT pipeline_kafka.add_broker(‘localhost:9092'); CREATE STREAM logs_stream (payload

    json); CREATE CONTINUOUS VIEW message_count AS SELECT COUNT(*) FROM logs_stream;
  75. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ • PrestoΛར༻͠ɺ
 RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ
 RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ
 KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλΛֻ͚߹Θͤͨ
 Ϩίϝϯυ

  76. Apache Kafkaͷ࢝Ίํ • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ • Confluent͔Βμ΢ϯϩʔυ • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ


    Confluentར༻͕Φεεϝ • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢
  77. Apache Kafka GUI • Cluster؅ཧ 
 https://github.com/yahoo/kafka-manager • Message؅ཧ 


    https://github.com/landoop/kafka-topics-ui
 https://github.com/ldaniels528/trifecta
 ͳͲ
  78. ·ͱΊ • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ ͷ໰୊ղܾͷҰͭ • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑ •

    ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ