PHPで支える大規模・ラムダアーキテクチャ

Transcript

1. PHPͰࢧ͑Δ େن໛ϥϜμΞʔΩςΫνϟ takezawa yuuki <ytake> PHP conference kansai 2017
2. Lamda Architecture
3. ҰൠతͳwebαʔϏε • PHP • MySQL, PostgreSQL, Oracle, SQL Server • Apache, Nginx etc
4. େ͖͘ͳ͖ͬͯͨwebαʔϏε • DatabaseʹՃ͑ɺElasitcsearch, SolrͳͲ • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛ௥Ճ͠
 RDBMSͷ೉͍͠ͱ͜ΖΛิ͏ • ͍͔ͭ͘ͷindexੜ੒ύλʔϯ
5. Application Database Elasticsearch
6. ύλʔϯ1 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ElasticseachͳͲʹૠೖ͢Δύλʔϯ • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ
 ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ
7. Application Database Elasticsearch Batch
8. ύλʔϯ2 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • batchͰͲ͜·Ͱ࡞੒͔ͨ֬͠ೝ͠ͳ͕Βɺ
 ະ࡞੒ͷ΋ͷͷΈ࡞੒͢Δ • ͨͩ͠ϦΞϧλΠϜͰ͸ͳ͍
9. Application Database Elasticsearch Message Queue Consumer Producer
10. ύλʔϯ3 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ
 ProducerΛհͯ͠Message Queue΁ૠೖ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • Consumer͕൓Ԡ͠ɺElasticsearchͷindexΛ࡞੒ • Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ
11. ΑΓେ͖ͳΞϓϦέʔγϣϯ΁
12. ࣄۀ੒௕ʹΑΔେ͖ͳΞϓϦέʔγϣϯ΁ • ϢʔβʔͷߦಈΛ෼ੳ͍ͨ͠ • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ ʹར༻͍ͨ͠ • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞ • ෼ࢄͨ͠αʔϏεͷσʔλΛू໿ͯ͠৽͍͠ίϯςϯ πΛఏڙ͍ͨ͠ • BigData΁
13. Big Data + Fast Data
14. BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ
 σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠ • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠ • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠ • ਖ਼نԽͨ͠σʔλ͸ઈରʹRDBMS • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔
16. BigData΁ͷΞϓϩʔν • σʔλͦͷ΋ͷͷू໿ • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ
 Messageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ • લड़ͷdatabase, elasticsearchซ༻ͷऔΓ૊ΈΛɺ
 ΑΓେ͖ͳεέʔϧͰߏங͢Δ
17. ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ
 ετϦʔϜσʔλॲཧͷ໨త • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
18. ϥϜμΞʔΩςΫνϟ • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒ • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳͲΛ୲౰ ͢Δ -> Hadoop(MapReduce), Spark • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ
 Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚
 Spark, Storm, Kafka, Cassandra etc.. • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫
 ೉қ౓͸ߴ͍ɾɾɾ
 -> KafkaͳͲʹू໿ͤͨ͞Kappa Architecure΁
19. KappaΞʔΩςΫνϟ
20. KappaΞʔΩςΫνϟ
21. OSSͰߏங
22. PHPϝΠϯͰ࡞Δ͜ͱ͸೉͍͠ɾɾ
23. Apache Cassandra
24. Apache Cassandra • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ
 ෼ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ
 εέʔϧ͕༰қͱ͍͏఺Ͱ࠾༻ • PHP͔Βར༻Մೳ(ext-cassandra) • େྔσʔλͷॻ͖ࠐΈʹରԠ • ؆୯ͳτϥϯβΫγϣϯαϙʔτ • σʔληϯλʔލ͗ͷΫϥελʔߏங • Availability ͱ Partition Tolerance • SQLΠϯλʔϑΣʔε • ୯Ұো֐఺ͳ͠
25. Apache Cassandra Architecture
26. ؾΛ͚ͭΔ఺ • RDBMSײ֮Ͱ͸͏·͘ར༻Ͱ͖·ͤΜ • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ • ৚݅ʹΑΔΦʔμʔ͸ࢦఆͰ͖ͳ͍ • ϚςϦΞϥΠζυϏϡʔซ༻͢΂͠ • ো֐࣌ͷϩά෼ੳ͸ͨͩ͘͠ • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ) • ݕࡧͰҾ͔͔ͬΔهࣄ͸େମچόʔδϣϯͰɺ
 ݱߦͱશ͘ผ෺
27. ςʔϒϧઃܭ • Primary Key͸ࣝผΩʔͰ͋Γͳ͕Βɺ
 Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ • ҟͳΔnodeʹ͋Δ΋ͷͷݕࡧ͸޲͔ͳ͍
 ඞཁͳέʔε͕ੜͨ͡৔߹͸ςʔϒϧઃܭΛݟ௚͢ • ߋ৽࣌ɺ࡟আ࣌ʹؚΊͳ͚Ε͹ͳΒͳ͍ • ར༻Մೳͳ΋ͷ͸ηΧϯμϦΠϯσοΫε·Ͱ • JOIN΍LIKE͸ଘࡏ͠ͳ͍ͨΊɺෳࡶͳ΋ͷ͸SparkͰ
28. ςʔϒϧઃܭ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) );
29. Ϩίʔυͷॱ൪Λܾఆ͢Δ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) ) WITH CLUSTERING ORDER BY (published_at DESC);
30. MATERIALIZED VIEW CREATE MATERIALIZED VIEW timeline.desc_user_timeline AS SELECT uuid, user_id, published_at, reference, body FROM timeline.user_timeline WHERE user_id IS NOT NULL AND published_at IS NOT NULL AND uuid IS NOT NULL PRIMARY KEY (user_id, published_at, uuid) WITH CLUSTERING ORDER BY (published_at DESC);
31. From PHP $cluster = Cassandra::cluster() ->withContactPoints('10.0.1.24', ‘localhost') ->withPort(9042) ->build(); $statement = $session->prepare( "UPDATE users SET age = ? WHERE user_name = ?” ); $futures = array(); // execute all statements in background foreach ($data as $arguments) { $futures[] = $session->executeAsync( $statement, [ ‘arguments' => $arguments ]; }
32. PHP extension • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement • ฒྻར༻Մೳ • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻) • Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ
 Java͔Βར༻ͤͣͱ΋े෼׆༻Ͱ͖Δ
33. Apache Kafka
34. Apache Kafka • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰ͸ඞཁෆՄܽ) • ΫϥελϦϯά͕ࣗ༝ࣗࡏ • Zookeeperͱ࿈ܞͨ͠෼ࢄγεςϜ • ো֐ʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ • SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻ • ϝοηʔδૹ৴ޙͰ΋ࢦఆظؒอ࣋͠ɺ
 ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ಺༰औಘՄೳ • PHP͔Βར༻Մೳ(rdkafka)
35. Message QueueͰൃੜ͢Δ໰୊ • Producer͔ΒBroker΁ૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ • Broker͕ड৴Λࣦഊ͢Δέʔε • Brokerͷૹ৴͕ࣦഊ͢Δέʔε • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε
36. 0.11 • Exactly-once delivery and transactional messaging • ਖ਼֬ʹҰ౓͚ͩɺ࣮֬ʹಧ͚Δ • ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ • ΑΓڧݻʹ
37. Partition • ฒྻ΍෼ࢄॲཧ͕ઃܭ • topicΛPartitionͰ෼ׂ͠ɺProducer, Consumer͕೚ҙ ͷPartitionʹΞΫηε • ࡉ෼Խͱޮ཰Խ͕ࣗ༝ʹ
38. Partition
39. BigDataͷ࢝·Γ͸PHP͔Β
41. webΞϓϦέʔγϣϯ͔Β BigData·Ͱࢧ͑ΔPHP
42. ·ͱΊ • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ
 ՝୊ղܾ͸େ͖͘ͳΔ୉ޣຯ • PHPͰϏδωεΛαϙʔτ͢Δ؅ཧπʔϧ • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ • PHPͰ΋େ͖͘ߩݙ