Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
PHPで支える大規模・ラムダアーキテクチャ
yuuki takezawa
July 15, 2017
Technology
0
1k
PHPで支える大規模・ラムダアーキテクチャ
yuuki takezawa
July 15, 2017
Tweet
Share
More Decks by yuuki takezawa
See All by yuuki takezawa
入門 境界づけられたコンテキスト
ytake
2
800
時間軸とドメインイベントとデータ処理
ytake
1
740
事業のスケールアウトを支える PHPで作る分散アーキテクチャ
ytake
4
3.4k
Hack HTTP Request and Response Interfaces
ytake
0
5.9k
Hackで作る堅実な アプリケーションアーキテクチャ / Hack-application-architecture
ytake
3
7.2k
PHPとApache Sparkで始めるデータ解析処理 / php-with-apache-spark
ytake
2
3k
Event Sourcing+CQRS を活用するスケーラブルアプリケーション開発 / event-sourcing-cqrs-v2
ytake
7
11k
大きなデータと戦う技術 / fighting-large-data
ytake
3
370
Laravel JP Conference2019
ytake
1
1.7k
Other Decks in Technology
See All in Technology
サイバー攻撃を想定したクラウドネイティブセキュリティガイドラインとCNAPP及びSecurity Observabilityの未来
sakon310
4
450
セキュキャンを卒業してその後
kurochan
0
460
Cloud Foundryの移行先はどこか? オープンソースPaaS探し
kolinz
0
350
Azure DevOps Online Vol.6 - 業務で必要なCIをみんなで考えよう
kkamegawa
0
240
ソフトウェアアーキテクチャの基礎: Software Architecture in a Nutshell
snoozer05
28
8.5k
2022 COSCUP - GKE Backend Cluster 除雷分享
brentchang
0
120
Trusted Web プロトタイプ
finengine
0
320
Power Automate for desktopで 配信環境を改善してみた話
akiika
0
230
ログ集約基盤をCloudWatchからOpenSearchに変えてみた
yuhta28
0
130
CloudWatchアラームによるサービス継続のための監視入門 / Introduction to Monitoring for Service Continuity with CloudWatch Alarms
inomasosan
1
410
AWS CLI でやってみる ~ AWS Hands-on for Beginners ECS ハンズオン ~
kentosuzuki
1
380
JAWS-UG 朝会 #36 登壇資料
takakuni
1
540
Featured
See All Featured
Facilitating Awesome Meetings
lara
29
4.1k
Design by the Numbers
sachag
271
17k
The Invisible Customer
myddelton
110
11k
Gamification - CAS2011
davidbonilla
75
3.9k
Building Your Own Lightsaber
phodgson
95
4.7k
Building an army of robots
kneath
298
40k
Reflections from 52 weeks, 52 projects
jeffersonlam
337
17k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
29
4.4k
Building Applications with DynamoDB
mza
84
4.8k
Designing for humans not robots
tammielis
241
24k
The World Runs on Bad Software
bkeepers
PRO
57
5.4k
Bootstrapping a Software Product
garrettdimon
296
110k
Transcript
PHPͰࢧ͑Δ େنϥϜμΞʔΩςΫνϟ takezawa yuuki <ytake> PHP conference kansai 2017
Lamda Architecture
ҰൠతͳwebαʔϏε • PHP • MySQL, PostgreSQL, Oracle, SQL Server •
Apache, Nginx etc
େ͖͘ͳ͖ͬͯͨwebαʔϏε • DatabaseʹՃ͑ɺElasitcsearch, SolrͳͲ • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛՃ͠ RDBMSͷ͍͠ͱ͜ΖΛิ͏ • ͍͔ͭ͘ͷindexੜύλʔϯ
Application Database Elasticsearch
ύλʔϯ1 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ElasticseachͳͲʹૠೖ͢Δύλʔϯ • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ
Application Database Elasticsearch Batch
ύλʔϯ2 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ • webΞϓϦέʔγϣϯଆͰσʔλϕʔεʹૠೖͷΈ • batchͰͲ͜·Ͱ࡞͔ͨ֬͠ೝ͠ͳ͕Βɺ ະ࡞ͷͷͷΈ࡞͢Δ •
ͨͩ͠ϦΞϧλΠϜͰͳ͍
Application Database Elasticsearch Message Queue Consumer Producer
ύλʔϯ3 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ProducerΛհͯ͠Message Queueૠೖ • webΞϓϦέʔγϣϯଆͰσʔλϕʔεʹૠೖͷΈ • Consumer͕Ԡ͠ɺElasticsearchͷindexΛ࡞ •
Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ
ΑΓେ͖ͳΞϓϦέʔγϣϯ
ࣄۀʹΑΔେ͖ͳΞϓϦέʔγϣϯ • ϢʔβʔͷߦಈΛੳ͍ͨ͠ • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ ʹར༻͍ͨ͠ • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞ • ࢄͨ͠αʔϏεͷσʔλΛूͯ͠৽͍͠ίϯςϯ
πΛఏڙ͍ͨ͠ • BigData
Big Data + Fast Data
BigDataʹ͏ΞϓϦέʔγϣϯͷ՝ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ͍ظؒͰԯͱେʹͳΓɺ σʔλϕʔεͷindexΑΓI/O͕ݫ͍͠ • ϨϓϦέʔγϣϯԆ୲อ͕͍͠ •
ઍສϢʔβʔͷϦΞϧλΠϜͷੳΛ͢Δʹݫ͍͠ • ਖ਼نԽͨ͠σʔλઈରʹRDBMS • ࢄͨ͠σʔλϕʔεʹͲ͏ཱ͔ͪ͏͔
None
BigDataͷΞϓϩʔν • σʔλͦͷͷͷू • લ·Ͱʹूܭ͓͚ͯ͠ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ MessageॲཧͱɺࢄՄೳͳσʔλετϨʔδ •
લड़ͷdatabase, elasticsearchซ༻ͷऔΓΈΛɺ ΑΓେ͖ͳεέʔϧͰߏங͢Δ
ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ ετϦʔϜσʔλॲཧͷత • ऴΘΓ͕ͳ͘ɺແݶʹͬͯ͘ΔͷͷΞϓϩʔν • ϝϞϦͰॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔͷ
• ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
ϥϜμΞʔΩςΫνϟ • όονɺαʔϏεɺεϐʔυͰߏ • όονɺେ͖ͳσʔλͷूܭɺେྔσʔλͷੳͳͲΛ୲ ͢Δ -> Hadoop(MapReduce), Spark •
αʔϏεόονͷू݁ՌΛఏڙ͢Δ Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ Spark, Storm, Kafka, Cassandra etc.. • αʔϏεͱεϐʔυͷ྆ํͷΛϚʔδͯ͠ฦ٫ қߴ͍ɾɾɾ -> KafkaͳͲʹूͤͨ͞Kappa Architecure
KappaΞʔΩςΫνϟ
KappaΞʔΩςΫνϟ
OSSͰߏங
PHPϝΠϯͰ࡞Δ͜ͱ͍͠ɾɾ
Apache Cassandra
Apache Cassandra • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ εέʔϧ͕༰қͱ͍͏Ͱ࠾༻ • PHP͔Βར༻Մೳ(ext-cassandra) • େྔσʔλͷॻ͖ࠐΈʹରԠ
• ؆୯ͳτϥϯβΫγϣϯαϙʔτ • σʔληϯλʔލ͗ͷΫϥελʔߏங • Availability ͱ Partition Tolerance • SQLΠϯλʔϑΣʔε • ୯Ұোͳ͠
Apache Cassandra Architecture
ؾΛ͚ͭΔ • RDBMSײ֮Ͱ͏·͘ར༻Ͱ͖·ͤΜ • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ • ݅ʹΑΔΦʔμʔࢦఆͰ͖ͳ͍ • ϚςϦΞϥΠζυϏϡʔซ༻͢͠ •
ো࣌ͷϩάੳͨͩ͘͠ • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ) • ݕࡧͰҾ͔͔ͬΔهࣄେମچόʔδϣϯͰɺ ݱߦͱશ͘ผ
ςʔϒϧઃܭ • Primary KeyࣝผΩʔͰ͋Γͳ͕Βɺ Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ • ҟͳΔnodeʹ͋Δͷͷݕࡧ͔ͳ͍ ඞཁͳέʔε͕ੜͨ͡߹ςʔϒϧઃܭΛݟ͢ • ߋ৽࣌ɺআ࣌ʹؚΊͳ͚ΕͳΒͳ͍
• ར༻ՄೳͳͷηΧϯμϦΠϯσοΫε·Ͱ • JOINLIKEଘࡏ͠ͳ͍ͨΊɺෳࡶͳͷSparkͰ
ςʔϒϧઃܭ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference
map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) );
Ϩίʔυͷॱ൪Λܾఆ͢Δ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference
map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) ) WITH CLUSTERING ORDER BY (published_at DESC);
MATERIALIZED VIEW CREATE MATERIALIZED VIEW timeline.desc_user_timeline AS SELECT uuid, user_id,
published_at, reference, body FROM timeline.user_timeline WHERE user_id IS NOT NULL AND published_at IS NOT NULL AND uuid IS NOT NULL PRIMARY KEY (user_id, published_at, uuid) WITH CLUSTERING ORDER BY (published_at DESC);
From PHP $cluster = Cassandra::cluster() ->withContactPoints('10.0.1.24', ‘localhost') ->withPort(9042) ->build(); $statement
= $session->prepare( "UPDATE users SET age = ? WHERE user_name = ?” ); $futures = array(); // execute all statements in background foreach ($data as $arguments) { $futures[] = $session->executeAsync( $statement, [ ‘arguments' => $arguments ]; }
PHP extension • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement • ฒྻར༻Մೳ • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻)
• Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ Java͔Βར༻ͤͣͱे׆༻Ͱ͖Δ
Apache Kafka
Apache Kafka • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰඞཁෆՄܽ) • ΫϥελϦϯά͕ࣗ༝ࣗࡏ • Zookeeperͱ࿈ܞͨ͠ࢄγεςϜ • োʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ
• SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻ • ϝοηʔδૹ৴ޙͰࢦఆظؒอ࣋͠ɺ ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ༰औಘՄೳ • PHP͔Βར༻Մೳ(rdkafka)
Message QueueͰൃੜ͢Δ • Producer͔ΒBrokerૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ • Broker͕ड৴Λࣦഊ͢Δέʔε • Brokerͷૹ৴͕ࣦഊ͢Δέʔε • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε
0.11 • Exactly-once delivery and transactional messaging • ਖ਼֬ʹҰ͚ͩɺ࣮֬ʹಧ͚Δ •
ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ • ΑΓڧݻʹ
Partition • ฒྻࢄॲཧ͕ઃܭ • topicΛPartitionͰׂ͠ɺProducer, Consumer͕ҙ ͷPartitionʹΞΫηε • ࡉԽͱޮԽ͕ࣗ༝ʹ
Partition
BigDataͷ࢝·ΓPHP͔Β
None
webΞϓϦέʔγϣϯ͔Β BigData·Ͱࢧ͑ΔPHP
·ͱΊ • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ ՝ղܾେ͖͘ͳΔޣຯ • PHPͰϏδωεΛαϙʔτ͢Δཧπʔϧ • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ
• PHPͰେ͖͘ߩݙ