PHPで支える大規模・ラムダアーキテクチャ

Transcript

1. PHPͰࢧ͑Δ
   େن໛ϥϜμΞʔΩςΫνϟ
   takezawa yuuki
   PHP conference kansai 2017
   

   View Slide

2. Lamda Architecture
   

   View Slide

3. ҰൠతͳwebαʔϏε
   • PHP
   • MySQL, PostgreSQL, Oracle, SQL Server
   • Apache, Nginx etc
   

   View Slide

4. େ͖͘ͳ͖ͬͯͨwebαʔϏε
   • DatabaseʹՃ͑ɺElasitcsearch, SolrͳͲ
   • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛ௥Ճ͠

   RDBMSͷ೉͍͠ͱ͜ΖΛิ͏
   • ͍͔ͭ͘ͷindexੜ੒ύλʔϯ
   

   View Slide

5. Application
   Database Elasticsearch
   

   View Slide

6. ύλʔϯ1
   • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ

   ElasticseachͳͲʹૠೖ͢Δύλʔϯ
   • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ

   ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ
   

   View Slide

7. Application
   Database Elasticsearch
   Batch
   

   View Slide

8. ύλʔϯ2
   • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ

   ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ
   • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ
   • batchͰͲ͜·Ͱ࡞੒͔ͨ֬͠ೝ͠ͳ͕Βɺ

   ະ࡞੒ͷ΋ͷͷΈ࡞੒͢Δ
   • ͨͩ͠ϦΞϧλΠϜͰ͸ͳ͍
   

   View Slide

9. Application
   Database
   Elasticsearch
   Message Queue
   Consumer
   Producer
   

   View Slide

10. ύλʔϯ3
    • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ

    ProducerΛհͯ͠Message Queue΁ૠೖ
    • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ
    • Consumer͕൓Ԡ͠ɺElasticsearchͷindexΛ࡞੒
    • Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ
    

    View Slide

11. ΑΓେ͖ͳΞϓϦέʔγϣϯ΁
    

    View Slide

12. ࣄۀ੒௕ʹΑΔେ͖ͳΞϓϦέʔγϣϯ΁
    • ϢʔβʔͷߦಈΛ෼ੳ͍ͨ͠
    • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ
    ʹར༻͍ͨ͠
    • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞
    • ෼ࢄͨ͠αʔϏεͷσʔλΛू໿ͯ͠৽͍͠ίϯςϯ
    πΛఏڙ͍ͨ͠
    • BigData΁
    

    View Slide

13. Big Data + Fast Data
    

    View Slide

14. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊
    • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ
    ཧ͕ऴΘΒͳ͍
    • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ

    σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠
    • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠
    • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠
    • ਖ਼نԽͨ͠σʔλ͸ઈରʹRDBMS
    • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔
    

    View Slide

15. View Slide

16. BigData΁ͷΞϓϩʔν
    • σʔλͦͷ΋ͷͷू໿
    • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻
    ҙ͢Δ
    • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ

    Messageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ
    • લड़ͷdatabase, elasticsearchซ༻ͷऔΓ૊ΈΛɺ

    ΑΓେ͖ͳεέʔϧͰߏங͢Δ
    

    View Slide

17. ετϦʔϜॲཧ
    • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ

    ετϦʔϜσʔλॲཧͷ໨త
    • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν
    • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ
    • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ
    • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
    

    View Slide

18. ϥϜμΞʔΩςΫνϟ
    • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒
    • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳͲΛ୲౰
    ͢Δ -> Hadoop(MapReduce), Spark
    • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ

    Hive, HBase, ElephantDB, Splout SQL, pipelineDB…
    • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚

    Spark, Storm, Kafka, Cassandra etc..
    • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫

    ೉қ౓͸ߴ͍ɾɾɾ

    -> KafkaͳͲʹू໿ͤͨ͞Kappa Architecure΁
    

    View Slide

19. KappaΞʔΩςΫνϟ
    

    View Slide

20. KappaΞʔΩςΫνϟ
    

    View Slide

21. OSSͰߏங
    

    View Slide

22. PHPϝΠϯͰ࡞Δ͜ͱ͸೉͍͠ɾɾ
    

    View Slide

23. Apache Cassandra
    

    View Slide

24. Apache Cassandra
    • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ

    ෼ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ

    εέʔϧ͕༰қͱ͍͏఺Ͱ࠾༻
    • PHP͔Βར༻Մೳ(ext-cassandra)
    • େྔσʔλͷॻ͖ࠐΈʹରԠ
    • ؆୯ͳτϥϯβΫγϣϯαϙʔτ
    • σʔληϯλʔލ͗ͷΫϥελʔߏங
    • Availability ͱ Partition Tolerance
    • SQLΠϯλʔϑΣʔε
    • ୯Ұো֐఺ͳ͠
    

    View Slide

25. Apache Cassandra Architecture
    

    View Slide

26. ؾΛ͚ͭΔ఺
    • RDBMSײ֮Ͱ͸͏·͘ར༻Ͱ͖·ͤΜ
    • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ
    • ৚݅ʹΑΔΦʔμʔ͸ࢦఆͰ͖ͳ͍
    • ϚςϦΞϥΠζυϏϡʔซ༻͢΂͠
    • ো֐࣌ͷϩά෼ੳ͸ͨͩ͘͠
    • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ)
    • ݕࡧͰҾ͔͔ͬΔهࣄ͸େମچόʔδϣϯͰɺ

    ݱߦͱશ͘ผ෺
    

    View Slide

27. ςʔϒϧઃܭ
    • Primary Key͸ࣝผΩʔͰ͋Γͳ͕Βɺ

    Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ
    • ҟͳΔnodeʹ͋Δ΋ͷͷݕࡧ͸޲͔ͳ͍

    ඞཁͳέʔε͕ੜͨ͡৔߹͸ςʔϒϧઃܭΛݟ௚͢
    • ߋ৽࣌ɺ࡟আ࣌ʹؚΊͳ͚Ε͹ͳΒͳ͍
    • ར༻Մೳͳ΋ͷ͸ηΧϯμϦΠϯσοΫε·Ͱ
    • JOIN΍LIKE͸ଘࡏ͠ͳ͍ͨΊɺෳࡶͳ΋ͷ͸SparkͰ
    

    View Slide

28. ςʔϒϧઃܭ
    CREATE TABLE timeline.user_timeline (
    uuid uuid,
    user_id int,
    reference map,
    body text,
    is_read tinyint,
    published_at timestamp,
    PRIMARY KEY (user_id)
    );
    

    View Slide

29. Ϩίʔυͷॱ൪Λܾఆ͢Δ
    CREATE TABLE timeline.user_timeline (
    uuid uuid,
    user_id int,
    reference map,
    body text,
    is_read tinyint,
    published_at timestamp,
    PRIMARY KEY (user_id)
    )
    WITH CLUSTERING ORDER BY (published_at DESC);
    

    View Slide

30. MATERIALIZED VIEW
    CREATE MATERIALIZED VIEW timeline.desc_user_timeline
    AS SELECT uuid, user_id, published_at, reference, body
    FROM timeline.user_timeline
    WHERE user_id IS NOT NULL
    AND published_at IS NOT NULL
    AND uuid IS NOT NULL
    PRIMARY KEY (user_id, published_at, uuid)
    WITH CLUSTERING ORDER BY (published_at DESC);
    

    View Slide

31. From PHP
    $cluster = Cassandra::cluster()
    ->withContactPoints('10.0.1.24', ‘localhost')
    ->withPort(9042)
    ->build();
    $statement = $session->prepare(
    "UPDATE users SET age = ? WHERE user_name = ?”
    );
    $futures = array();
    // execute all statements in background
    foreach ($data as $arguments) {
    $futures[] = $session->executeAsync(
    $statement,
    [
    ‘arguments' => $arguments
    ];
    }
    

    View Slide

32. PHP extension
    • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement
    • ฒྻར༻Մೳ
    • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻)
    • Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ

    Java͔Βར༻ͤͣͱ΋े෼׆༻Ͱ͖Δ
    

    View Slide

33. Apache Kafka
    

    View Slide

34. Apache Kafka
    • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰ͸ඞཁෆՄܽ)
    • ΫϥελϦϯά͕ࣗ༝ࣗࡏ
    • Zookeeperͱ࿈ܞͨ͠෼ࢄγεςϜ
    • ো֐ʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ
    • SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻
    • ϝοηʔδૹ৴ޙͰ΋ࢦఆظؒอ࣋͠ɺ

    ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ಺༰औಘՄೳ
    • PHP͔Βར༻Մೳ(rdkafka)
    

    View Slide

35. Message QueueͰൃੜ͢Δ໰୊
    • Producer͔ΒBroker΁ૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ
    • Broker͕ड৴Λࣦഊ͢Δέʔε
    • Brokerͷૹ৴͕ࣦഊ͢Δέʔε
    • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε
    

    View Slide

36. 0.11
    • Exactly-once delivery and transactional messaging
    • ਖ਼֬ʹҰ౓͚ͩɺ࣮֬ʹಧ͚Δ
    • ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ
    • ΑΓڧݻʹ
    

    View Slide

37. Partition
    • ฒྻ΍෼ࢄॲཧ͕ઃܭ
    • topicΛPartitionͰ෼ׂ͠ɺProducer, Consumer͕೚ҙ
    ͷPartitionʹΞΫηε
    • ࡉ෼Խͱޮ཰Խ͕ࣗ༝ʹ
    

    View Slide

38. Partition
    

    View Slide

39. BigDataͷ࢝·Γ͸PHP͔Β
    

    View Slide

40. View Slide

41. webΞϓϦέʔγϣϯ͔Β
    BigData·Ͱࢧ͑ΔPHP
    

    View Slide

42. ·ͱΊ
    • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ

    ՝୊ղܾ͸େ͖͘ͳΔ୉ޣຯ
    • PHPͰϏδωεΛαϙʔτ͢Δ؅ཧπʔϧ
    • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ
    • PHPͰ΋େ͖͘ߩݙ
    

    View Slide