PHPで支える大規模・ラムダアーキテクチャ

Slide 1

Slide 1 text

PHPͰࢧ͑Δ େن໛ϥϜμΞʔΩςΫνϟ takezawa yuuki PHP conference kansai 2017

Slide 2

Slide 2 text

Lamda Architecture

Slide 3

Slide 3 text

ҰൠతͳwebαʔϏε • PHP • MySQL, PostgreSQL, Oracle, SQL Server • Apache, Nginx etc

Slide 4

Slide 4 text

େ͖͘ͳ͖ͬͯͨwebαʔϏε • DatabaseʹՃ͑ɺElasitcsearch, SolrͳͲ • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛ௥Ճ͠  RDBMSͷ೉͍͠ͱ͜ΖΛิ͏ • ͍͔ͭ͘ͷindexੜ੒ύλʔϯ

Slide 5

Slide 5 text

Application Database Elasticsearch

Slide 6

Slide 6 text

ύλʔϯ1 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ  ElasticseachͳͲʹૠೖ͢Δύλʔϯ • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ  ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ

Slide 7

Slide 7 text

Application Database Elasticsearch Batch

Slide 8

Slide 8 text

ύλʔϯ2 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ  ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • batchͰͲ͜·Ͱ࡞੒͔ͨ֬͠ೝ͠ͳ͕Βɺ  ະ࡞੒ͷ΋ͷͷΈ࡞੒͢Δ • ͨͩ͠ϦΞϧλΠϜͰ͸ͳ͍

Slide 9

Slide 9 text

Application Database Elasticsearch Message Queue Consumer Producer

Slide 10

Slide 10 text

ύλʔϯ3 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ  ProducerΛհͯ͠Message Queue΁ૠೖ • webΞϓϦέʔγϣϯଆͰ͸σʔλϕʔεʹૠೖͷΈ • Consumer͕൓Ԡ͠ɺElasticsearchͷindexΛ࡞੒ • Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ

Slide 11

Slide 11 text

ΑΓେ͖ͳΞϓϦέʔγϣϯ΁

Slide 12

Slide 12 text

ࣄۀ੒௕ʹΑΔେ͖ͳΞϓϦέʔγϣϯ΁ • ϢʔβʔͷߦಈΛ෼ੳ͍ͨ͠ • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ ʹར༻͍ͨ͠ • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞ • ෼ࢄͨ͠αʔϏεͷσʔλΛू໿ͯ͠৽͍͠ίϯςϯ πΛఏڙ͍ͨ͠ • BigData΁

Slide 13

Slide 13 text

Big Data + Fast Data

Slide 14

Slide 14 text

BigDataʹ൐͏ΞϓϦέʔγϣϯͷ՝୊ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ਺΋୹͍ظؒͰ਺ԯͱ๲େʹͳΓɺ  σʔλϕʔεͷindexΑΓ΋I/O͕ݫ͍͠ • ϨϓϦέʔγϣϯ஗Ԇ୲อ͕೉͍͠ • ਺ઍສϢʔβʔͷϦΞϧλΠϜͷ෼ੳΛ͢Δʹ͸ݫ͍͠ • ਖ਼نԽͨ͠σʔλ͸ઈରʹRDBMS • ෼ࢄͨ͠σʔλϕʔεʹͲ͏ཱͪ޲͔͏͔

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

BigData΁ͷΞϓϩʔν • σʔλͦͷ΋ͷͷू໿ • લ೔·Ͱʹूܭ͓͚ͯ͠͹ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ  Messageॲཧͱɺ෼ࢄՄೳͳσʔλετϨʔδ • લड़ͷdatabase, elasticsearchซ༻ͷऔΓ૊ΈΛɺ  ΑΓେ͖ͳεέʔϧͰߏங͢Δ

Slide 17

Slide 17 text

ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ  ετϦʔϜσʔλॲཧͷ໨త • ऴΘΓ͕ͳ͘ɺແݶʹ΍ͬͯ͘Δ΋ͷ΁ͷΞϓϩʔν • ϝϞϦ಺Ͱॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ؂ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔ΋ͷ • ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ

Slide 18

Slide 18 text

ϥϜμΞʔΩςΫνϟ • όον૚ɺαʔϏε૚ɺεϐʔυ૚Ͱߏ੒ • όον૚͸ɺେ͖ͳσʔλͷूܭ΍ɺେྔσʔλͷ෼ੳͳͲΛ୲౰ ͢Δ -> Hadoop(MapReduce), Spark • αʔϏε૚͸όον૚ͷू໿݁ՌΛఏڙ͢Δ  Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυ૚͸ϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ૚  Spark, Storm, Kafka, Cassandra etc.. • αʔϏε૚ͱεϐʔυ૚ͷ྆ํͷ஋ΛϚʔδͯ͠ฦ٫  ೉қ౓͸ߴ͍ɾɾɾ  -> KafkaͳͲʹू໿ͤͨ͞Kappa Architecure΁

Slide 19

Slide 19 text

KappaΞʔΩςΫνϟ

Slide 20

Slide 20 text

KappaΞʔΩςΫνϟ

Slide 21

Slide 21 text

OSSͰߏங

Slide 22

Slide 22 text

PHPϝΠϯͰ࡞Δ͜ͱ͸೉͍͠ɾɾ

Slide 23

Slide 23 text

Apache Cassandra

Slide 24

Slide 24 text

Apache Cassandra • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ  ෼ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ  εέʔϧ͕༰қͱ͍͏఺Ͱ࠾༻ • PHP͔Βར༻Մೳ(ext-cassandra) • େྔσʔλͷॻ͖ࠐΈʹରԠ • ؆୯ͳτϥϯβΫγϣϯαϙʔτ • σʔληϯλʔލ͗ͷΫϥελʔߏங • Availability ͱ Partition Tolerance • SQLΠϯλʔϑΣʔε • ୯Ұো֐఺ͳ͠

Slide 25

Slide 25 text

Apache Cassandra Architecture

Slide 26

Slide 26 text

ؾΛ͚ͭΔ఺ • RDBMSײ֮Ͱ͸͏·͘ར༻Ͱ͖·ͤΜ • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ • ৚݅ʹΑΔΦʔμʔ͸ࢦఆͰ͖ͳ͍ • ϚςϦΞϥΠζυϏϡʔซ༻͢΂͠ • ো֐࣌ͷϩά෼ੳ͸ͨͩ͘͠ • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ) • ݕࡧͰҾ͔͔ͬΔهࣄ͸େମچόʔδϣϯͰɺ  ݱߦͱશ͘ผ෺

Slide 27

Slide 27 text

ςʔϒϧઃܭ • Primary Key͸ࣝผΩʔͰ͋Γͳ͕Βɺ  Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ • ҟͳΔnodeʹ͋Δ΋ͷͷݕࡧ͸޲͔ͳ͍  ඞཁͳέʔε͕ੜͨ͡৔߹͸ςʔϒϧઃܭΛݟ௚͢ • ߋ৽࣌ɺ࡟আ࣌ʹؚΊͳ͚Ε͹ͳΒͳ͍ • ར༻Մೳͳ΋ͷ͸ηΧϯμϦΠϯσοΫε·Ͱ • JOIN΍LIKE͸ଘࡏ͠ͳ͍ͨΊɺෳࡶͳ΋ͷ͸SparkͰ

Slide 28

Slide 28 text

ςʔϒϧઃܭ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference map, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) );

Slide 29

Slide 29 text

Ϩίʔυͷॱ൪Λܾఆ͢Δ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference map, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) ) WITH CLUSTERING ORDER BY (published_at DESC);

Slide 30

Slide 30 text

MATERIALIZED VIEW CREATE MATERIALIZED VIEW timeline.desc_user_timeline AS SELECT uuid, user_id, published_at, reference, body FROM timeline.user_timeline WHERE user_id IS NOT NULL AND published_at IS NOT NULL AND uuid IS NOT NULL PRIMARY KEY (user_id, published_at, uuid) WITH CLUSTERING ORDER BY (published_at DESC);

Slide 31

Slide 31 text

From PHP $cluster = Cassandra::cluster() ->withContactPoints('10.0.1.24', ‘localhost') ->withPort(9042) ->build(); $statement = $session->prepare( "UPDATE users SET age = ? WHERE user_name = ?” ); $futures = array(); // execute all statements in background foreach ($data as $arguments) { $futures[] = $session->executeAsync( $statement, [ ‘arguments' => $arguments ]; }

Slide 32

Slide 32 text

PHP extension • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement • ฒྻར༻Մೳ • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻) • Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ  Java͔Βར༻ͤͣͱ΋े෼׆༻Ͱ͖Δ

Slide 33

Slide 33 text

Apache Kafka

Slide 34

Slide 34 text

Apache Kafka • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰ͸ඞཁෆՄܽ) • ΫϥελϦϯά͕ࣗ༝ࣗࡏ • Zookeeperͱ࿈ܞͨ͠෼ࢄγεςϜ • ো֐ʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ • SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻ • ϝοηʔδૹ৴ޙͰ΋ࢦఆظؒอ࣋͠ɺ  ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ಺༰औಘՄೳ • PHP͔Βར༻Մೳ(rdkafka)

Slide 35

Slide 35 text

Message QueueͰൃੜ͢Δ໰୊ • Producer͔ΒBroker΁ૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ • Broker͕ड৴Λࣦഊ͢Δέʔε • Brokerͷૹ৴͕ࣦഊ͢Δέʔε • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε

Slide 36

Slide 36 text

0.11 • Exactly-once delivery and transactional messaging • ਖ਼֬ʹҰ౓͚ͩɺ࣮֬ʹಧ͚Δ • ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ • ΑΓڧݻʹ

Slide 37

Slide 37 text

Partition • ฒྻ΍෼ࢄॲཧ͕ઃܭ • topicΛPartitionͰ෼ׂ͠ɺProducer, Consumer͕೚ҙ ͷPartitionʹΞΫηε • ࡉ෼Խͱޮ཰Խ͕ࣗ༝ʹ

Slide 38

Slide 38 text

Partition

Slide 39

Slide 39 text

BigDataͷ࢝·Γ͸PHP͔Β

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

webΞϓϦέʔγϣϯ͔Β BigData·Ͱࢧ͑ΔPHP

Slide 42

Slide 42 text

·ͱΊ • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ  ՝୊ղܾ͸େ͖͘ͳΔ୉ޣຯ • PHPͰϏδωεΛαϙʔτ͢Δ؅ཧπʔϧ • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ • PHPͰ΋େ͖͘ߩݙ