PHPで支える大規模・ラムダアーキテクチャ

Transcript

1. PHPͰࢧ͑Δ େن໛ϥϜμΞʔΩςΫνϟ takezawa yuuki <ytake> PHP conference kansai 2017

2. Lamda Architecture

3. ҰൠతͳwebαʔϏε • PHP • MySQL, PostgreSQL, Oracle, SQL Server •

   Apache, Nginx etc

4. େ͖͘ͳ͖ͬͯͨwebαʔϏε • DatabaseʹՃ͑ɺElasitcsearch, SolrͳͲ • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛ௥Ճ͠
 RDBMSͷ೉͍͠ͱ͜ΖΛิ͏ • ͍͔ͭ͘ͷindexੜ੒ύλʔϯ

5. Application Database Elasticsearch

6. ύλʔϯ1 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ElasticseachͳͲʹૠೖ͢Δύλʔϯ • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ
 ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ

7. Application Database Elasticsearch Batch

8. ύλʔϯ2 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • batchͰͲ͜·Ͱ࡞੒͔ͨ֬͠ೝ͠ͳ͕Βɺ
 ະ࡞੒ͷ΋ͷͷΈ࡞੒͢Δ •

   ͨͩ͠ϦΞϧλΠϜͰ͸ͳ͍

9. Application Database Elasticsearch Message Queue Consumer Producer

10. ύλʔϯ3 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ProducerΛհͯ͠Message Queue΁ૠೖ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • Consumer͕൓Ԡ͠ɺElasticsearchͷindexΛ࡞੒ •

    Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ

11. ΑΓେ͖ͳΞϓϦέʔγϣϯ΁

12. ࣄۀ੒௕ʹΑΔେ͖ͳΞϓϦέʔγϣϯ΁ • ϢʔβʔͷߦಈΛ෼ੳ͍ͨ͠ • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ ʹར༻͍ͨ͠ • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞ • ෼ࢄͨ͠αʔϏεͷσʔλΛू໿ͯ͠৽͍͠ίϯςϯ

    πΛఏڙ͍ͨ͠ • BigData΁

13. Big Data + Fast Data

14. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ
 σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠ • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠ •

    ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠ • ਖ਼نԽͨ͠σʔλ͸ઈରʹRDBMS • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔
15. None

16. BigData΁ͷΞϓϩʔν • σʔλͦͷ΋ͷͷू໿ • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ
 Messageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ •

    લड़ͷdatabase, elasticsearchซ༻ͷऔΓ૊ΈΛɺ
 ΑΓେ͖ͳεέʔϧͰߏங͢Δ

17. ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ
 ετϦʔϜσʔλॲཧͷ໨త • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ

    • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ

18. ϥϜμΞʔΩςΫνϟ • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒ • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳͲΛ୲౰ ͢Δ -> Hadoop(MapReduce), Spark •

    αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ
 Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚
 Spark, Storm, Kafka, Cassandra etc.. • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫
 ೉қ౓͸ߴ͍ɾɾɾ
 -> KafkaͳͲʹू໿ͤͨ͞Kappa Architecure΁

19. KappaΞʔΩςΫνϟ

20. KappaΞʔΩςΫνϟ

21. OSSͰߏங

22. PHPϝΠϯͰ࡞Δ͜ͱ͸೉͍͠ɾɾ

23. Apache Cassandra

24. Apache Cassandra • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ
 ෼ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ
 εέʔϧ͕༰қͱ͍͏఺Ͱ࠾༻ • PHP͔Βར༻Մೳ(ext-cassandra) • େྔσʔλͷॻ͖ࠐΈʹରԠ

    • ؆୯ͳτϥϯβΫγϣϯαϙʔτ • σʔληϯλʔލ͗ͷΫϥελʔߏங • Availability ͱ Partition Tolerance • SQLΠϯλʔϑΣʔε • ୯Ұো֐఺ͳ͠

25. Apache Cassandra Architecture

26. ؾΛ͚ͭΔ఺ • RDBMSײ֮Ͱ͸͏·͘ར༻Ͱ͖·ͤΜ • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ • ৚݅ʹΑΔΦʔμʔ͸ࢦఆͰ͖ͳ͍ • ϚςϦΞϥΠζυϏϡʔซ༻͢΂͠ •

    ো֐࣌ͷϩά෼ੳ͸ͨͩ͘͠ • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ) • ݕࡧͰҾ͔͔ͬΔهࣄ͸େମچόʔδϣϯͰɺ
 ݱߦͱશ͘ผ෺

27. ςʔϒϧઃܭ • Primary Key͸ࣝผΩʔͰ͋Γͳ͕Βɺ
 Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ • ҟͳΔnodeʹ͋Δ΋ͷͷݕࡧ͸޲͔ͳ͍
 ඞཁͳέʔε͕ੜͨ͡৔߹͸ςʔϒϧઃܭΛݟ௚͢ • ߋ৽࣌ɺ࡟আ࣌ʹؚΊͳ͚Ε͹ͳΒͳ͍

    • ར༻Մೳͳ΋ͷ͸ηΧϯμϦΠϯσοΫε·Ͱ • JOIN΍LIKE͸ଘࡏ͠ͳ͍ͨΊɺෳࡶͳ΋ͷ͸SparkͰ

28. ςʔϒϧઃܭ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference

    map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) );

29. Ϩίʔυͷॱ൪Λܾఆ͢Δ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference

    map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) ) WITH CLUSTERING ORDER BY (published_at DESC);

30. MATERIALIZED VIEW CREATE MATERIALIZED VIEW timeline.desc_user_timeline AS SELECT uuid, user_id,

    published_at, reference, body FROM timeline.user_timeline WHERE user_id IS NOT NULL AND published_at IS NOT NULL AND uuid IS NOT NULL PRIMARY KEY (user_id, published_at, uuid) WITH CLUSTERING ORDER BY (published_at DESC);

31. From PHP $cluster = Cassandra::cluster() ->withContactPoints('10.0.1.24', ‘localhost') ->withPort(9042) ->build(); $statement

    = $session->prepare( "UPDATE users SET age = ? WHERE user_name = ?” ); $futures = array(); // execute all statements in background foreach ($data as $arguments) { $futures[] = $session->executeAsync( $statement, [ ‘arguments' => $arguments ]; }

32. PHP extension • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement • ฒྻར༻Մೳ • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻)

    • Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ
 Java͔Βར༻ͤͣͱ΋े෼׆༻Ͱ͖Δ

33. Apache Kafka

34. Apache Kafka • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰ͸ඞཁෆՄܽ) • ΫϥελϦϯά͕ࣗ༝ࣗࡏ • Zookeeperͱ࿈ܞͨ͠෼ࢄγεςϜ • ো֐ʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ

    • SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻ • ϝοηʔδૹ৴ޙͰ΋ࢦఆظؒอ࣋͠ɺ
 ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ಺༰औಘՄೳ • PHP͔Βར༻Մೳ(rdkafka)

35. Message QueueͰൃੜ͢Δ໰୊ • Producer͔ΒBroker΁ૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ • Broker͕ड৴Λࣦഊ͢Δέʔε • Brokerͷૹ৴͕ࣦഊ͢Δέʔε • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε

36. 0.11 • Exactly-once delivery and transactional messaging • ਖ਼֬ʹҰ౓͚ͩɺ࣮֬ʹಧ͚Δ •

    ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ • ΑΓڧݻʹ

37. Partition • ฒྻ΍෼ࢄॲཧ͕ઃܭ • topicΛPartitionͰ෼ׂ͠ɺProducer, Consumer͕೚ҙ ͷPartitionʹΞΫηε • ࡉ෼Խͱޮ཰Խ͕ࣗ༝ʹ

38. Partition

39. BigDataͷ࢝·Γ͸PHP͔Β

40. None

41. webΞϓϦέʔγϣϯ͔Β BigData·Ͱࢧ͑ΔPHP

42. ·ͱΊ • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ
 ՝୊ղܾ͸େ͖͘ͳΔ୉ޣຯ • PHPͰϏδωεΛαϙʔτ͢Δ؅ཧπʔϧ • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ

    • PHPͰ΋େ͖͘ߩݙ