Apache Kafkaによるスケーラブル アプリケーション開発

PHPカンファレンス2017で利用したスライド

Transcript

1. Apache KafkaʹΑΔ
   εέʔϥϒϧ
   ΞϓϦέʔγϣϯ։ൃ
   yuuki takezawa 

   PHP Conference 2017
   

   View Slide

2. ͸͡Ίʹ
   • ن໛ಁաੑ

   ෛՙͷߴ௿ʹ߹ΘͤͯϦ

   ιʔεɾϓʔϧΛ֦େɾॖখͰ͖Δ͜ͱ
   • Ґஔಁաੑ

   Ϣʔβʔ΍Ϧιʔε͕ͲΕ͚ͩ཭Ε͍ͯΔ͔ҙࣝͤͣʹɺ
   มΘΒͳ͍࢖͍উखͰγεςϜ͕ར༻Ͱ͖Δ͜ͱ
   • ҟछಁաੑ

   γεςϜΛߏ੒͢Δػث΍ιϑτ΢ΣΞ͕ҟͳ͍ͬͯΔ͜
   ͱΛҙࣝͤͣʹ؅ཧɾར༻Ͱ͖Δ͜ͱ
   

   View Slide

3. Agenda
   • Message Queue
   • What is Apache Kafka?
   • For PHP
   

   View Slide

4. Message Queue
   

   View Slide

5. ίϝϯτ౤ߘαʔϏε
   • ϩάΠϯϢʔβʔ͸ίϝϯτ͕౤ߘͰ͖Δ
   • ະϩάΠϯϢʔβʔ͸ӾཡͷΈ͕Ͱ͖Δ
   • ͓ؾʹೖΓͨ͠ϢʔβʔͷίϝϯτΛӾཡͰ͖Δ
   • ࣌ܥྻͰίϝϯτΛӾཡͰ͖Δ
   • ίϝϯτͷ࡟আ͕Ͱ͖Δ
   • ίϝϯτͷฤू͸Ͱ͖ͳ͍ ͳͲ
   

   View Slide

6. ΞϓϦέʔγϣϯ࣮૷
   • ೝূػೳͷ࣮૷
   • ίϝϯτ౤ߘɺฤूɺӾཡػೳ࣮૷
   • ݕࡧػೳͷ࣮૷
   • ਺ਓͷ։ൃऀ
   

   View Slide

7. γϯϓϧͳΞϓϦέʔγϣϯߏ੒
   

   View Slide

8. γϯϓϧͳΞϓϦέʔγϣϯߏ੒
   public function __invoke(Comment $comment)
   {
   $query = $comment->query();
   $query->insert(['title', 'body'])
   ->values([
   'title' => 'First post',
   'body' => 'Some body text',
   'id' => $this->auth->getId(),
   ])
   ->execute();
   }
   

   View Slide

9. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
   • ػೳ௥Ճ

   • ίϝϯτ৘ใΛଞͷػೳͰར༻͍ͨ͠

   • ৚݅ʹΑΔҰཡ৘ใͷιʔτมߋ

   Ϣʔβʔ৘ใଐੑʹΑΔιʔτॱมߋ
   

   View Slide

10. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
    

    View Slide

11. ෳࡶԽ͢ΔΞϓϦέʔγϣϯ
    public function __invoke(Comment $comment, Movie $movie)
    {
    $query = $comment->query();
    $commentId = $query->insert(['title', 'body', 'comment_id'])
    ->values([
    'title' => 'First post',
    'body' => 'Some body text',
    'comment_id' => $this->auth->getId(),
    ])
    ->execute();
    $movieQuery = $movie->query();
    $movieQuery->insert(['movie_id', 'comment_id'])
    ->values([
    'movie_id' => $movie->getId(),
    'comment_id' => $commentId,
    ])
    ->execute();
    }
    

    View Slide

12. Ϗδωεͷ੒௕ͱͱ΋ʹ
    ංେԽ͢ΔΞϓϦέʔγϣϯ
    

    View Slide

13. ංେԽ͢ΔΞϓϦέʔγϣϯ
    • ऩӹԽͷͨΊͷػೳ

    • རศੑͷͨΊͷػೳ
    • طଘػೳͷ֦ு
    

    View Slide

14. ංେԽ͢ΔΞϓϦέʔγϣϯ
    

    View Slide

15. ංେԽ͢ΔΞϓϦέʔγϣϯ
    public function __invoke(
    Comment $comment,
    Movie $movie,
    Follow $follow,
    Company $company
    ) {
    $movieCommentId = false;
    if ($follow->isFollow()) {
    $query = $comment->query();
    $commentId = $query->insert(['title', 'body', 'comment_id'])
    ->values([
    'title' => 'First post',
    'body' => 'Some body text',
    'comment_id' => $this->auth->getId(),
    ])
    ->execute();
    $movieQuery = $movie->query();
    $movieCommentId = $movieQuery->insert(['movie_id', 'comment_id'])
    ->values([
    'movie_id' => $movie->getId(),
    'comment_id' => $commentId,
    ])
    ->execute();
    }
    if ($movie->isCompany() && $movieCommentId) {
    $company->appendMovie($movie->getId());
    // ϝʔϧૹ৴
    }
    }
    

    View Slide

16. ΞϓϦέʔγϣϯͷΫϥεͦͷ΋ͷΛ
    ίϯϙʔωϯτԽ
    ґଘੑΛऔΓআ͘
    

    View Slide

17. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά
    • ݁߹ͨ͠ػೳΛίϯϙʔωϯτఏڙ

    • APIʹΑΔσʔλૢ࡞ͷந৅Խ

    • σʔλϕʔεύϑΥʔϚϯεվળ

    

    View Slide

18. ΞϓϦέʔγϣϯɾσʔλϕʔεϦϑΝΫλϦϯά
    

    View Slide

19. ڊେͳΞϓϦέʔγϣϯ΁
    • αʔϏε֦େʹͱ΋ͳ͏ΞϓϦέʔγϣϯ௥Ճ

    • ґଘίϯϙʔωϯτ૿Ճ

    -> ࢓༷มߋٵऩ
    • ૿Ճ͢ΔAPIར༻

    -> ᷓΕग़͢ଞαʔϏεͷϏδωεϩδοΫ
    • σʔλϕʔε໰୊

    -> ElasticsearchɺSolrͳͲͷಋೖ
    

    View Slide

20. ڊେͳΞϓϦέʔγϣϯ΁
    

    View Slide

21. ਂࠁͳ໰୊
    • ࢓༷มߋʹ൐͏ίϯϙʔωϯτमਖ਼ͱϦϦʔεෳࡶԽ

    ϥΠϒϥϦͷόʔδϣϯ͋͛ͳ͖Όɾɾʂ

    ͋ͷػೳΛऔΓࠐΉʹ͸͋Εͱ͜Εͱɾɾɾ
    • ΞϓϦέʔγϣϯؒ࿈ܞͷෳࡶԽ

    APIमਖ਼ʹ൐͏Өڹൣғௐࠪ

    վमͯ͠΋࢖ͬͯ͘Εͳ͍໰୊
    

    View Slide

22. ਂࠁͳ໰୊
    • σʔλಉظͷෳࡶԽ

    ϦΞϧλΠϜͰσʔλ͕ཉ͍͠

    ΠϯσοΫεߋ৽͕ఆظ࣮ߦ

    • ͦͷଞ

    Ͳ͔͜ͷαʔϏε͕मਖ਼͞ΕͨΒো֐ʹ

    • ෳࡶԽ͢Δ࣮૷ίʔυ
    

    View Slide

23. ੒௕͍ͯ͘͠ࣄۀʹ߹Θͤͯɺ
    ΞϓϦέʔγϣϯΛ
    Ͳ͏εέʔϧ͍͔ͤͯ͘͞
    

    View Slide

24. มΘΓΏ͘؀ڥ
    • σʔλΛऔಘ͢Δʹ͸ʁ

    APIίʔϧʹΑΔ౎౓֬ೝʁ

    Ωϟογϡ͸Կ෼ʁ

    • ͦΕͧΕͷαʔϏεͰͦΕͧΕͷݴޠ΁

    ͋ΔαʔϏε͸PHPɺ͋ΔαʔϏεͰ͸Go ScalaͳͲ

    ϥΠϒϥϦ͕ෛͷҨ࢈Խ

    

    View Slide

25. σʔλॲཧ
    Ϣʔβʔ৘ใ͸౎౓໰͍߹Θͤ
    ৘ใΛߋ৽ͨ͠৔߹Ͱ΋

    ґଘαʔϏε͕ͳ͍ͨΊ໰୊ͳ͠
    

    View Slide

26. σʔλॲཧͷෳࡶԽ
    Ϣʔβʔ৘ใมߋ Ϣʔβʔ৘ใ࡟আ
    Ϣʔβʔ৘ใ͕ݟ͔ͭΒͳ͍ͨΊ

    σʔλॲཧ͝ͱʹ"1*ΞΫηε
    ͋ͦ͜ͱͦ͜Ͱ৘ใ͕ҧ͏ʂ

    Ͳͷσʔλ͕ຊ෺ʁ
    શͯͷΞϓϦέʔγϣϯ͔Β

    Ωϟογϡͳ͠ͰΞΫηε

    ৗʹߴෛՙঢ়ଶʹ
    

    View Slide

27. Command and Query
    Responsibility Segregation (CQRS)
    

    View Slide

28. CQRS
    • ॻ͖ࠐΈʢCommandʣͱಡग़͠ʢQueryʣͷ

    ੹຿Λ෼཭͢Δઃܭ

    • ͋ΒΏΔϝιου͸ɺΞΫγϣϯΛ࣮ߦ͢ΔίϚϯυ͔ɺݺͼ
    ग़͠ݩʹσʔλΛฦ͢ΫΤϦ͔ͷ͍ͣΕ͔Ͱ͋ͬͯɺ྆ํΛ
    ߦͬͯ͸ͳΒͳ͍ɻ͜Ε͸ɺ࣭໰Λ͢Δ͜ͱͰճ౴ΛมԽͤ͞
    ͯ͸ͳΒͳ͍ͱ͍͏͜ͱͩɻ

    

    "Greg YoungྲྀCQRS - Mark Nijhof". Digital Romanticism. 

    http://d.hatena.ne.jp/digitalsoul/20100712/1278886009 ࢀর 

    

    View Slide

29. CQRS
    "A few myths about CQRS". Ouarzy's Blog. 

    http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/ ࢀর
    

    View Slide

30. CQRS
    • σʔλϕʔεૢ࡞ΛಡΈࠐΈɺॻ͖ࠐΈͱ෼཭

    • RDBMSΛ׆͔͢

    ਖ਼نԽɾτϥϯβΫγϣϯ

    • αʔϏεʹ߹ΘͤͨϞσϦϯά

    RDBMS·ͨ͸Elasticsearch/SolrͰฦ٫ͳͲ

    • Event Sourcingͱ૊Έ߹ΘͤΔ
    

    View Slide

31. CQRSͱEvent Sourcing
    σʔλϕʔεߋ৽

    $PNNBOE
    σʔλϕʔεߋ৽׬ྃΠϕϯτ

    &WFOU4PVSDJOH
    
    σʔλϕʔεߋ৽׬ྃݕ஌

    &WFOU-JTUFOFS
    
    Ωϟογϡ࡟আͳͲ
    

    View Slide

32. ϑϨʔϜϫʔΫʹΈΔQueue
    • LaravelͰ͸͕͔͔࣌ؒΔॲཧ΍ඇಉظॲཧΉ͚ʹɺ

    QueueίϯϙʔωϯτΛఏڙ
    • QueueΛར༻͢ΔͨΊʹRedis΍RDBMSɺ

    ·ͨ͸υϥΠό֦ுͰ೚ҙͷϛυϧ΢ΣΞ͕ར༻Մೳ
    • ղܾʁ

    LaravelͷQueueΛଞݴޠؒɾαʔϏεؒͰ࢖͏͜ͱ͸
    ೉͍͠(php serializeͳͲ͕͞Ε͍ͯΔ)
    

    View Slide

33. What is Apache Kafka?
    

    View Slide

34. View Slide

35. ϝοηʔδϯάγεςϜ
    

    View Slide

36. ଟ༷ͳ෼ࢄܕϝοηʔδϯάϛυϧ΢ΣΞ
    • Redis (PubSub)
    • ZeroMQ
    • RabbitMQ
    • Apache Kafka
    

    View Slide

37. Redis
    • KVSͷҰػೳɺPubSub(SUBSCRIBE, PUBLISHίϚϯυ)
    • Redis SentinelΛར༻ͯ͠ϑΣΠϧΦʔόʔΛߦ͏
    • LPUSHͳͲΛར༻ͯ͠ϝοηʔδΛอ؅ ͳͲ
    • ϝοηʔδ࠶औಘ΍τϥϯβΫγϣϯػೳ͸ͳ͍
    • ϨϓϦέʔγϣϯՄ
    

    View Slide

38. ZeroMQ
    • ϛυϧ΢ΣΞͰ͸ͳ͘ɺϥΠϒϥϦ
    • ΞϓϦέʔγϣϯ͕ؒ௨৴Λߦ͍ɺϝοηʔδॲཧ
    • ϛυϧ΢ΣΞΛར༻͠ͳ͍ͨΊߴ଎
    • ϝοηʔδͷӬଓԽػೳ͸ແ͍ҝɺ

    ಠࣗͰ࣮૷͢Δඞཁ͕͋Δ
    

    View Slide

39. RabbitMQ
    • Advanced Message Queuing Protocol(AMQP)
    • ΤϯλʔϓϥΠζΞϓϦέʔγϣϯͰଟ਺ར༻࣮੷
    • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯՄೳ
    • τϥϯβΫγϣϯΛར༻࣮ͯ֬͠ʹॲཧΛߦ͏
    • v3.6.0Ͱ௥Ճ͞ΕͨLazy QueuesͰେ༰ྔͷσʔλ஝ੵՄ
    • ϑΣΠϧΦʔόʔ
    

    View Slide

40. Apache Kafka
    • ZookeeperΛར༻ͨ͠ΫϥελϦϯάʹΑΔߴՄ༻ੑ
    • ϝοηʔδͷӬଓԽɺϨϓϦέʔγϣϯɺ࠶औಘՄ
    • ϏοάσʔλରԠ
    • ϑΝΠϧγεςϜར༻Ͱɺ

    γʔέϯγϟϧΞΫηεʹΑΔߴ଎Խ
    • ετϦʔϜରԠͷϝοηʔδϯάϛυϧ΢ΣΞ
    • Kafka ConnectʹΑΔपลγεςϜͱͷߴ͍਌࿨ੑ

    (Amazon kinesisͱ΄΅ಉ͡)
    

    View Slide

41. Apache Kafka֓ཁ
    • Producer 

    ϝοηʔδ഑৴Λߦ͏

    ֤ݴޠͷΫϥΠΞϯτϥΠϒϥϦΛར༻
    • Consumer 

    ϝοηʔδߪಡΛߦ͏

    ফඅ͞Εͨϝοηʔδ͸ഁغ͞Εͣɺ

    Ұఆظؒอ؅͞ΕΔ
    • Broker

    KafkaຊମͰɺProducerɺConsumerؒͷΩϡʔ
    

    View Slide

42. Apache ZooKeeper֓ཁ
    • ઃఆ؅ཧ΍ɺ໊લղܾͳͲΛߦ͏

    ෼ࢄίʔσΟωʔγϣϯΤϯδϯ
    • ෳ਺୆Ͱߏ੒͢ΔZooKeeperͷ͏ͪ1୆͕ɺ

    ͢΂ͯͷॻ͖ࠐΈΛௐఀɾडཧ͢ΔϦʔμʔ
    • Ϋϥελͷ͏ͪա൒਺͕ಈ࡞͢Δঢ়ଶʹ͢Δ͜ͱ͕ඞ
    ཁͰ͋ΔͨΊɺح਺୆Ͱߏ੒͢Δ͜ͱ͕ඞཁ
    

    View Slide

43. Apache Kafka + ZooKeeper Architecture
    

    View Slide

44. Apache Kafka֓ཁ
    • Topic 

    Producer͔Βͷϝοηʔδ͸͜ͷTopicʹ֨ೲ͞ΕΔ

    ϝοηʔδ͸Ұҙʹ؅ཧɺFIFO(ޙड़partition)Ͱॲཧ
    • Partition

    ෛՙ෼ࢄ༻్ʹར༻

    ෳ਺ͷConsumer͕ͦΕͧΕͷPartitionΛࢀর͠ɺ

    ͦΕͧΕ͕ॲཧΛߦ͏

    ॲཧϑϩʔͷσβΠϯʹΑͬͯଟ༷ͳར༻ํ๏
    

    View Slide

45. Example Partition
    

    View Slide

46. Kafka Connect
    • Kafka Connectͱ͸ɺ

    पลγεςϜ͔ΒͷσʔλΛऔΓࠐΈ(Source)ɺ

    σʔλૹ৴(Sink)ͷೋछྨΛαϙʔτ͢Δػೳ
    • Amazon SQS΍MongoDBͷσʔλΛKafkaͰऔࠐΉɺ

    ϝοηʔδΛͦͷ··Elasticsearch΍RDBMSʹ֨ೲɺ

    ͕ߦ͑Δ
    • Connect͸ࣗ༝ʹ֦ுͯ͠ಠࣗConnectΛ࣮૷Մೳ

    (java, Scala)
    

    View Slide

47. Example Kafka Connect
    &WFOU4PVSDJOH
    ଞΞϓϦέʔγϣϯ

    $POTVNFS
    ૹ৴಺༰Λͦͷ··

    &MBTUJDTFBSDI΁
    

    View Slide

48. Kafka Streams
    • KafkaͰετϦʔϜॲཧΛ࣮૷͢Δػೳ
    • ͋ΔτϐοΫʹσʔλ͕֨ೲ͞ΕΔ࣌ʹ

    Կ͔ॲཧΛ࣮ߦ͠ଞͷτϐοΫʹ֨ೲ͢Δ
    • Consumerͳ͠Ͱ্هͷॲཧΛ࣮ݱ
    • ௚ۙ30෼ʹ͓͚ΔPVϥϯΩϯάɺ

    ϝοηʔδ಺ͷจࣈྻมߋɺ௥ՃͳͲ
    

    View Slide

49. ϝοηʔδͷܽଛ
    ૹ৴ࣦഊ
    BDLड৴ࣦഊ
    #SPLFSμ΢ϯ
    ϝοηʔδड৴ࣦഊ
    

    View Slide

50. Exactly Once
    • 0.11͔Β௥Ճ
    • Producer, BrokerؒͷτϥϯβΫγϣϯ

    Broker, ConsumerؒͷτϥϯβΫγϣϯ
    • ਖ਼֬ʹҰ౓͚ͩૹ৴
    

    View Slide

51. ϝοηʔδ఻ୡͷอূ
    • At least once semantics

    ॏෳΛڐՄ
    • At most once semantics

    ܽଛΛڐՄ
    • Exactly once semantics

    ॏෳɺ͓ΑͼܽଛΛڐՄ͠ͳ͍

    

    "Exactly-once Semantics are Possible: Here’s How Kafka Does it". Confluent APACHE KAFKA. 

    https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ ࢀর
    

    View Slide

52. KSQL
    • Kafkaʹอ؅͞Ε͍ͯΔϝοηʔδʹ

    SQLΠϯλʔϑΣʔεͰΞΫηε͢Δ΋ͷ
    • ݱࡏDeveloper Preview
    • RESTͰΞΫηε͢Δ͜ͱ͕Ͱ͖·͢
    

    View Slide

53. KappaΞʔΩςΫνϟ
    

    View Slide

54. KappaΞʔΩςΫνϟ
    

    View Slide

55. Apache Kafka࠾༻
    • ෼ੳॲཧΛ࣮૷͢Δʹ͋ͨΓɺStreamରԠ͕༰қͳࣄ
    • Kafka ConnectʹΑΔσʔλετϨʔδͱͷ࿈ܞ
    • Facebook Prestoͱͷ࿈ܞ͕؆୯

    (RDBMS౳ͱ݁߹ͯ͠σʔλఏڙ͕Մೳ)
    • ґଘ෼ղͷͨΊʹεέʔϧ͢Δ

    ϝοηʔδϯάγεςϜߏங͕༰қ
    • ϝοηʔδ࠶औಘͱো֐ରԠ͕༰қͰ͋Δࣄ
    • PHPΤΫεςϯγϣϯ͕͋Δ(rdkafka)
    

    View Slide

56. Architecture
    

    View Slide

57. Kafka client for PHP
    

    View Slide

58. Rdkafka
    https://github.com/arnaud-lb/
    php-rdkafka
    

    View Slide

59. Rdkafka
    • librdkafka(C/C++)ΛPHPͰར༻Մೳʹͨ͠

    ΤΫεςϯγϣϯ
    • High Level API & Row Level API
    • PHP5/7ରԠ
    • librdkafkaͱkafkaͷઃఆΛཧղ͢Δ͜ͱ
    

    View Slide

60. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    

    View Slide

61. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ
    

    View Slide

62. Rdkafka Producer࣮૷
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers(“127.0.0.1");
    $topic = $rk->newTopic("test");
    $topic->produce(
    RD_KAFKA_PARTITION_UA,
    0,
    json_encode(['message' => ‘phpcon'])
    );
    $MVTUFSͷ৔߹͸ɺ

    ΧϯϚ۠੾ΓͰࢦఆ
    5PQJD໊
    1BSUJUJPOΛࢦఆ
    

    View Slide

63. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    

    View Slide

64. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
    ॲཧ͢Δ৔߹͸ಉ໊͡લʹ
    

    View Slide

65. Rdkafka Consumer࣮૷
    $conf = new RdKafka\Conf();
    $conf->set('group.id', 'myConsumerGroup');
    $rk = new RdKafka\Consumer($conf);
    $rk->addBrokers("127.0.0.1");
    $topicConf = new RdKafka\TopicConf();
    $topicConf->set('auto.commit.interval.ms', 100);
    $topicConf->set('offset.store.method', 'file');
    $topicConf->set(
    'offset.store.path',
    sys_get_temp_dir()
    );
    $topicConf->set('auto.offset.reset', 'smallest');
    ฒྻͰಉ͡UPQJDΛಉ࣌ʹ
    ॲཧ͢Δ৔߹͸ಉ໊͡લʹ
    $POTVNFSઃఆ஋Λࢦఆ
    

    View Slide

66. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    

    View Slide

67. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    

    View Slide

68. Rdkafka Consumer࣮૷
    $topic = $rk->newTopic("test", $topicConf);
    $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
    while (true) {
    $message = $topic->consume(0, 120*10000);
    switch ($message->err) {
    case RD_KAFKA_RESP_ERR_NO_ERROR:
    var_dump($message);
    break;
    case RD_KAFKA_RESP_ERR__PARTITION_EOF:
    echo "No more messages; will wait for more\n";
    break;
    case RD_KAFKA_RESP_ERR__TIMED_OUT:
    echo "Timed out\n";
    break;
    default:
    throw new \Exception($message->errstr(), $message->err);
    break;
    }
    }
    Ͳͷ1BSUJUJPOΛར༻͢Δ͔

    औಘҐஔ͸Ͳ͔͜Β͔
    ਖ਼ৗʹड৴ͨ͠৔߹ʹ

    ೚ҙͷॲཧΛ࣮ߦ
    

    View Slide

69. Rdkafka Consumerͷઃఆ
    • heartbeatͱλΠϜΞ΢τ

    heartbeat.interval.ms͸

    session.timeout.msΑΓখ͘͞ɺ1/3ΑΓେ͖͘

    • Supervisord౳Ͱϓϩηε؂ࢹ͸ඞਢ
    • group.idʹΑΔॲཧͷฒྻԽ

    ෳ਺ConsumerͰಉ͡idΛࢦఆ͢Δͱฒߦʹ
    

    View Slide

70. Notice
    • ΫϦʔϯͳ࣮૷Λ৺͕͚Δ

    Consumer͸PHPͱ͸ݶΒͳ͍

    ΫϦʔϯͳϥΠϒϥϦɺ࣮૷Λ
    • RdkafkaͰૹ৴͢Δ͔FluentdͳͲͰૹ৴͢Δ͔͸

    ΞϓϦέʔγϣϯ࣍ୈ
    • RdkafkaΛϥΠϒϥϦܦ༝ͳͲͰ͸ͳ͘ɺ

    ͦͷ··ར༻͢Δ
    

    View Slide

71. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • ProducerͰόϦσʔγϣϯ

    -> ࠷௿ݶͷܕकΔ

    -> ଞαʔϏεͷ஌͕ࣝඞཁʹͳΔ৔߹͸࣮૷͠ͳ͍

    • ConsumerͰόϦσʔγϣϯ

    ड৴ͨ͠ϝοηʔδΛௐ΂ɺ

    ਖ਼͘͠ͳ͍΋ͷΛεΩοϓ
    

    View Slide

72. ϝοηʔδͷόϦσʔγϣϯ͸Ͳ͜Ͱʁ
    • Kafka Streamͱ͍͏બ୒ࢶ

    java·ͨ͸ScalaͰ࣮૷Մೳ

    -> ࣮૷ࣗମ͸ͦΜͳʹ೉͘͠ͳ͍

    • αʔϏεͷ஌ࣝ͸ઐ༻ͷStreamͰߦ͏

    -> ϚΠΫϩαʔϏεͰ͋Ε͹
    

    View Slide

73. PipelineDBͱ͍͏બ୒ࢶ
    • ྲྀΕͯ͘Δϝοηʔδʹରͯ͠ɺ

    ϦΞϧλΠϜʹɺSQL Likeʹऔಘ͍ͨ͠

    • ͔ͭετϨʔδΛѹഭͤͨ͘͞ͳ͍

    • PostgreSQL Compatible

    • PDOͰ઀ଓՄೳʂ
    

    View Slide

74. PipelineDBͱ͍͏બ୒ࢶ
    CREATE EXTENSION pipeline_kafka;
    SELECT pipeline_kafka.add_broker(‘localhost:9092');
    CREATE STREAM logs_stream (payload json);
    CREATE CONTINUOUS VIEW message_count
    AS SELECT COUNT(*) FROM logs_stream;
    

    View Slide

75. ҰาਐΜͩ෼ࢄΞϓϦέʔγϣϯ
    • PrestoΛར༻͠ɺ

    RDBMS΍Redisͱ૊Έ߹Θͤͨσʔλऔಘ͕Մೳ

    RedisͷϥϯΩϯάͱRDBMSͷਖ਼نԽ͞Εͨσʔλɺ

    KafkaΛ࢖ͬͨ௚ۙͷϩάσʔλΛֻ͚߹Θͤͨ

    Ϩίϝϯυ
    

    View Slide

76. Apache Kafkaͷ࢝Ίํ
    • Apache Kafkaެࣜϖʔδ͔Βμ΢ϯϩʔυ
    • Confluent͔Βμ΢ϯϩʔυ
    • Kafka ConnectΛར༻͢ΔͷͰ͋Ε͹ɺ

    Confluentར༻͕Φεεϝ
    • Hadoop౳͕ͳͯ͘΋ར༻ՄೳͰ͢
    

    View Slide

77. Apache Kafka GUI
    • Cluster؅ཧ 

    https://github.com/yahoo/kafka-manager
    • Message؅ཧ 

    https://github.com/landoop/kafka-topics-ui

    https://github.com/ldaniels528/trifecta

    ͳͲ
    

    View Slide

78. ·ͱΊ
    • ෳࡶԽͨ͠ΞϓϦέʔγϣϯɾ෼ࢄΞϓϦέʔγϣϯ
    ͷ໰୊ղܾͷҰͭ
    • KafkaҎ֎ͷMessage Queue΋બ୒ࢶʹ
    • StreamॲཧͳͲ͕ඞཁͰ͋Ε͹KafkaΛਪ঑
    • ن໛ʹ͋Θͤͨϛυϧ΢ΣΞબఆɾো֐ରԠɾӡ༻Λ
    

    View Slide