Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
PHPで支える大規模アーキテクチャ
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
yuuki takezawa
August 05, 2017
Technology
6.8k
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
PHPで支える大規模アーキテクチャ
このスライドはPHPカンファレンス関西2017でおこなったものに、presto追加とシンプルなデモを追加したものです
yuuki takezawa
August 05, 2017
Transcript
59 slides
PHPͰࢧ͑Δ େنΞʔΩςΫνϟ ver1.1 takezawa yuuki <ytake> builderscon tokyo 2017
Notice ͜Ε͔Β͓͢͠Δ༰ɺ ॴҦϏοάσʔλʹରͯ͠ͷ ΞϓϩʔνͰ ҰൠతͳΞϓϦέʔγϣϯʹ ͯ·Γ·ͤΜ
Lambda/Kappa Architecture
ҰൠతͳwebαʔϏε • PHP • MySQL, PostgreSQL, Oracle, SQL Server •
Apache, Nginx etc
͓ʁݕࡧ͕͘ͳ͖ͬͯͨͧ Index͖ͪΜͱ͋ΔΜ͚ͩͲɾ
େ͖͘ͳ͖ͬͯͨwebαʔϏε • ઍສϨίʔυʹରͯ͠ϑϩϯτ͔ΒLIKE۟ • ύϑΥʔϚϯεվળͷͨΊʹશจݕࡧͳͲΛՃ͠ RDBMSͷ͍͠ͱ͜ΖΛิ͏ • PVूܭͷͨΊʹຖඦߦ͕ॻ͖ࠐ·ΕΔϩά • খ͍͞αʔϏεͰʹͳΒͳ͔ͬͨͷ͕ݟ͑࢝
ΊΔ
ύλʔϯ1 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ElasticseachͳͲʹૠೖ͢Δύλʔϯ • ΞϓϦέʔγϣϯଆͰίϯτϩʔϧͰ͖Δ͕ɺ ΞϓϦέʔγϣϯͷίʔυ͕ංେԽ
Application Database Elasticsearch
/** * @Transactional * * @param ProductEntity $entity */ public
function register(ProductEntity $entity) { $this->productRepository->insert($entity); $this->elasticRepository->index([$entity]); }
ͦͷ2
ύλʔϯ2 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ఆظ࣮ߦ͞ΕΔόονͰૠೖ͢Δύλʔϯ • webΞϓϦέʔγϣϯଆͰσʔλϕʔεʹૠೖͷΈ • batchͰͲ͜·Ͱ࡞͔ͨ֬͠ೝ͠ͳ͕Βɺ ະ࡞ͷͷͷΈ࡞͢Δ •
ͨͩ͠ϦΞϧλΠϜͰͳ͍
Application Database Elasticsearch Batch
ൃੜ͢Δ • େྔͷϨίʔυΛऔಘ͢ΔͱςʔϒϧϩοΫ • ϞϊϦγοΫͳγεςϜͰɺಛఆͷσʔλϕʔεʹूத ͨ͠߹ʹɺҶͮΔࣜʹো • େྔϨίʔυҰׅೖͰϨϓϦέʔγϣϯԆͰো • σΟεΫᷓΕͰো
etc…
ύλʔϯ3 • webΞϓϦέʔγϣϯଆͰɺσʔλϕʔεʹૠೖޙɺ ProducerΛհͯ͠Message Queueૠೖ • webΞϓϦέʔγϣϯଆͰσʔλϕʔεʹૠೖͷΈ • Consumer͕Ԡ͠ɺElasticsearchͷindexΛ࡞ •
Message͕ফࣦ͠ͳ͍ݶΓϦΞϧλΠϜʹ͍ۙ
Application Database Elasticsearch Message Queue Consumer Producer
ൃੜͨ͠ • ಛఆͷαʔϏε͕ඞཁͱ͢ΔΛDefinitionʹೖΕͯ͠ ·͍ɺσʔλෆͰQueue٧·Γ • ConsumerͰϝϞϦϦʔΫ • σʔλϕʔείωΫγϣϯΫϩʔζͤͣʹ connection is
gone
ΑΓେ͖ͳΞϓϦέʔγϣϯ
ࣄۀʹΑΔେ͖ͳΞϓϦέʔγϣϯ • ϢʔβʔͷߦಈΛੳ͍ͨ͠ • ଟ͘ͷϢʔβʔʹར༻͞Ε͍ͯΔݕࡧจࣈΛαδΣετ ʹར༻͍ͨ͠ • Ϣʔβʔͷߦಈʹج͍ͮͨίϯςϯπΛද͍ࣔͤͨ͞ • ࢄͨ͠αʔϏεͷσʔλΛूͯ͠৽͍͠ίϯςϯ
πΛఏڙ͍ͨ͠ • BigData
Big Data + Fast Data
BigDataʹ͏ΞϓϦέʔγϣϯͷ՝ • ͦΕͧΕͷΞϓϦέʔγϣϯͰ࣮ߦ͍ͯͨ͠όονॲ ཧ͕ऴΘΒͳ͍ • Ϩίʔυ͍ظؒͰԯͱେʹͳΓɺ σʔλϕʔεͷindexΑΓI/O͕ݫ͍͠ • ϨϓϦέʔγϣϯԆ୲อ͕͍͠ •
ઍສϢʔβʔͷϦΞϧλΠϜͷੳΛ͢Δʹݫ͍͠ • ਖ਼نԽͨ͠σʔλઈରʹRDBMS • ࢄͨ͠σʔλϕʔεʹͲ͏ཱ͔ͪ͏͔
BigDataͷΞϓϩʔν • σʔλͦͷͷͷू • લ·Ͱʹूܭ͓͚ͯ͠ྑ͍σʔλΛ͋Β͔͡Ί༻ ҙ͢Δ • ϦΞϧλΠϜʹೖྗ͞ΕΔσʔλʹରͯ͠ͷ MessageॲཧͱɺࢄՄೳͳσʔλετϨʔδ •
લड़ͷdatabase, elasticsearchซ༻ͷऔΓΈΛɺ ΑΓେ͖ͳεέʔϧͰߏங͢Δ
ࢄετϨʔδ࠾༻ • MongoDBCouchbaseݕ౼ υϥΠόʔपΓͷෆ҆ఆ͞ͳͲͰݟૹΓ(ݱࡏར༻த) • Hadoop ࢄϑΝΠϧγεςϜͷHDFS ࢄॲཧͷͨΊͷMapReduce ेރΕ͍ͯΔɺ࠾༻ࣄྫे
None
ϥϜμΞʔΩςΫνϟ • όονɺαʔϏεɺεϐʔυͰߏ • όονɺେ͖ͳσʔλͷूܭɺେྔσʔλͷੳͳͲΛ୲ ͢Δ -> Hadoop(MapReduce), Spark •
αʔϏεόονͷू݁ՌΛఏڙ͢Δ Hive, HBase, ElephantDB, Splout SQL, pipelineDB… • εϐʔυϦΞϧλΠϜॲཧͷ݁ՌΛఏڙ͢Δ Spark, Storm, Kafka, Cassandra etc.. • αʔϏεͱεϐʔυͷ྆ํͷΛϚʔδͯ͠ฦ٫ қߴ͍ɾɾɾ -> KafkaͳͲʹूͤͨ͞Kappa Architecure
ετϦʔϜॲཧ • େྔͷσʔλΛϦΞϧλΠϜͰॲཧ͢Δͷ͕ɺ ετϦʔϜσʔλॲཧͷత • ऴΘΓ͕ͳ͘ɺແݶʹͬͯ͘ΔͷͷΞϓϩʔν • ϝϞϦͰॲཧ͞Εɺͦͷޙഁغ͞ΕΔ • ࢹܥͷॲཧΑ͘ར༻͞Ε͍ͯΔͷ
• ηϯαʔΛར༻ͨ͠ΞϓϦέʔγϣϯͳͲ
Spark • ࢄॲཧϑϨʔϜϫʔΫͷҰͭ • RDDͱݺΕΔΠϛϡʔλϒϧͳίϨΫγϣϯΛѻ͏ • Spark SQL • Spark
Streaming • Spark MLlib
KappaΞʔΩςΫνϟ
KappaΞʔΩςΫνϟ
OSSͰߏங
PHPϝΠϯͰ࡞Δ͜ͱ͍͠ɾɾ
σϞ https://github.com/ytake/ builderscon-example
Kappa Architecture(small) PHP ConsoleApp Kafka Spark Streaming PHP Consumer Cassandra
PHP WebApp
Apache Cassandra
Apache Cassandra • Ϩίʔυ͕େྔʹ૿͑Δ͜ͱ͕Θ͔͓ͬͯΓɺ ੳʹར༻͢Δ༧ఆͰ͋ͬͨͨΊɺ εέʔϧ͕༰қͱ͍͏Ͱ࠾༻ • PHP͔Βར༻Մೳ(ext-cassandra) • େྔσʔλͷॻ͖ࠐΈʹରԠ
• ؆୯ͳτϥϯβΫγϣϯαϙʔτ • σʔληϯλʔލ͗ͷΫϥελʔߏங • Availability ͱ Partition Tolerance • SQLΠϯλʔϑΣʔε • ୯Ұোͳ͠???????
Apache Cassandra Architecture
ؾΛ͚ͭΔ • RDBMSײ֮Ͱ͏·͘ར༻Ͱ͖·ͤΜ • ύʔςΟγϣϯΩʔͰ͏·͘ઃܭ͢Δ • ݅ʹΑΔΦʔμʔࢦఆͰ͖ͳ͍ • ϚςϦΞϥΠζυϏϡʔซ༻͢͠ •
ো࣌ͷϩάੳͨͩ͘͠ • ίϯύΫγϣϯͱઓ͏(࣮ࡍʹར༻͢Δ༰ྔ*2Ͱܭࢉ) • ݕࡧͰҾ͔͔ͬΔهࣄେମچόʔδϣϯͰɺ ݱߦͱશ͘ผ
ςʔϒϧઃܭ • Primary KeyࣝผΩʔͰ͋Γͳ͕Βɺ Ͳͷnodeʹ֨ೲ͢Δ͔Λܾఆ͢ΔύʔςΟγϣϯΩʔ • ҟͳΔnodeʹ͋Δͷͷݕࡧ͔ͳ͍ ඞཁͳέʔε͕ੜͨ͡߹ςʔϒϧઃܭΛݟ͢ • ߋ৽࣌ɺআ࣌ʹؚΊͳ͚ΕͳΒͳ͍
• ར༻ՄೳͳͷηΧϯμϦΠϯσοΫε·Ͱ • JOINLIKEଘࡏ͠ͳ͍ͨΊɺෳࡶͳͷSparkͰ
ςʔϒϧઃܭ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference
map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) );
Ϩίʔυͷॱ൪Λܾఆ͢Δ CREATE TABLE timeline.user_timeline ( uuid uuid, user_id int, reference
map<text, text>, body text, is_read tinyint, published_at timestamp, PRIMARY KEY (user_id) ) WITH CLUSTERING ORDER BY (published_at DESC);
MATERIALIZED VIEW CREATE MATERIALIZED VIEW timeline.desc_user_timeline AS SELECT uuid, user_id,
published_at, reference, body FROM timeline.user_timeline WHERE user_id IS NOT NULL AND published_at IS NOT NULL AND uuid IS NOT NULL PRIMARY KEY (user_id, published_at, uuid) WITH CLUSTERING ORDER BY (published_at DESC);
From PHP $cluster = Cassandra::cluster() ->withContactPoints('10.0.1.24', ‘localhost') ->withPort(9042) ->build(); $statement
= $session->prepare( "UPDATE users SET age = ? WHERE user_name = ?” ); $futures = array(); // execute all statements in background foreach ($data as $arguments) { $futures[] = $session->executeAsync( $statement, [ ‘arguments' => $arguments ]; }
PHP extension • Batchʹ࠷దԽ͞ΕͨI/F Batch Statement • ฒྻར༻Մೳ • Pagination͕༻ҙ͞Ε͍ͯΔ(Generatorར༻)
• Cassandraͷ΄ͱΜͲͷػೳ͕ར༻Ͱ͖ΔͷͰɺ Java͔Βར༻ͤͣͱे׆༻Ͱ͖Δ
Apache Kafka
Apache Kafka • Streamαϙʔτ(ϥϜμΞʔΩςΫνϟͰඞཁෆՄܽ) • ΫϥελϦϯά͕ࣗ༝ࣗࡏ • Zookeeperͱ࿈ܞͨ͠ࢄγεςϜ • োʹڧ͘ɺϝοηʔδͷ࠶औಘ͕Մೳ
• SparkͱStormͱ༰қʹ࿈ܞͰ͖Δ͜ͱ͔Β࠾༻ • ϝοηʔδૹ৴ޙͰࢦఆظؒอ࣋͠ɺ ଞͷΫΤϦΤϯδϯ͔Βϝοηʔδ༰औಘՄೳ • PHP͔Βར༻Մೳ(rdkafka)
Message QueueͰൃੜ͢Δ • Producer͔ΒBrokerૹ৴࣌ʹܽଛ͢Δ͜ͱ͕͋Δ • Broker͕ड৴Λࣦഊ͢Δέʔε • Brokerͷૹ৴͕ࣦഊ͢Δέʔε • ॏෳͯ͠ड৴ͯ͠͠·͏έʔε
0.11 • Exactly-once delivery and transactional messaging • ਖ਼֬ʹҰ͚ͩɺ࣮֬ʹಧ͚Δ •
ϝοηʔδૹ৴ͱड৴ʹτϥϯβΫγϣϯʂ • ΑΓڧݻʹ
Partition • ฒྻࢄॲཧ͕ઃܭ • topicΛPartitionͰׂ͠ɺProducer, Consumer͕ҙ ͷPartitionʹΞΫηε • ࡉԽͱޮԽ͕ࣗ༝ʹ
Partition
BigDataͷ࢝·ΓPHP͔Β
Presto
େنʹΑΔσʔλநग़ • hdfsʹ֨ೲ͞ΕͨσʔλΛΈ͍ͨ • RDBMSʹ֨ೲ͞Εͨσʔλͱ݁߹ͯ͠΄͍͠ • Ϗδωε؍Ͱͷσʔλूܭநग़Λͯ͠΄͍͠ • σʔλϕʔεࢄͰқ͕ߴ͍ •
खܰʹੳʹར༻͍ͨ͠ -> ແཧ
σʔλϚʔτ • ඞཁͳσʔλΛूΊͯσʔλϕʔεʹू͢Δ • όονॲཧͰ࣮ߦ͢ΔͨΊɺଈ࣌ʹσʔλΛऔಘ͢Δ ͜ͱ͍͠ • σʔλϚʔτࣗମͷอकӡ༻͕ඞཁͱͳΔ (ͦΕ͕ۀͰ͋ΕՄೳ) •
σʔλϚʔτΛઃܭ͢Δͷ͘͠ɺϋʔυϧ͕ߴ͍
Prestoͱ • facebookͰେنͳσʔλʹରͯ͠ɺ ΠϯλϥΫςΟϒʹσʔλऔಘͰ͖ΔΫΤϦΤϯδϯ • ϑϩϯτΞϓϦέʔγϣϯ͔Βhdfsʹଓ͠ɺ σʔλΛଈ࠲ʹՄࢹԽͤ͞Δͷ͍͠ • Hiveόονॲཧ༻్ͷͨΊɺඵͰฦ٫ෆՄೳ (MapReduce)
• RDBMSʹଓ͍ͨ͠ʂͳͲΛղܾ
Prestoͱ • SQLΠϯλʔϑΣʔεΛఏڙ • Cassandra, Hive, Kafka • MongoDB, MySQL,
PostgreSQL, SQLServer • Redis, Thrift • ରԠ͍ͯ͠ͳ͍ͷͰjavaͰυϥΠόΛ࣮͢Δ͚ͩ Ͱ͋Δఔ֦ுՄೳ • SELECTҎ֎ʹINSERTͳͲʹରԠ͓ͯ͠Γɺ σʔλϕʔεҠߦɺࢄΞʔΩςΫνϟͷΧόʔͳͲ ʹ
Prestoͱ
Prestoͱ • jdbcରԠ • PHP͔Βɺ xtendsys-labs/php-presto-client ytake/php-presto-client
None
·ͱΊ • ෳࡶԽ͢ΔΞϓϦέʔγϣϯɺ ՝ղܾେ͖͘ͳΔޣຯ • PHPͰϏδωεΛαϙʔτ͢Δཧπʔϧ • PHP͔Β࢝·ΔBigData + FastDataΞʔΩςΫνϟ
• PHPͰେ͖͘ߩݙ
webΞϓϦέʔγϣϯ͔Β BigData·Ͱࢧ͑ΔPHP
More Decks by yuuki takezawa
See All by yuuki takezawa
なぜAI時代に 「イベント」を中心に考えるのか? / Why focus on "events" in the age of AI?
ytake
4
2.3k
PHPでアクターモデルを活用したSagaパターンの実践法 / php-saga-pattern-with-actor-model
ytake
0
3.2k
PHP ステートレス VS ステートフル 状態管理と並行性 / php-stateless-stateful
ytake
0
410
PHPでアクターモデルを理解・体験しよう / Understand and experience the actor model in PHP
ytake
2
1.1k
再考 アクターモデル/ reconsider actor model
ytake
0
1.7k
GoとアクターモデルでES+CQRSを実践! / proto_actor_es_cqrs
ytake
1
750
Phluxorでアクターモデルを 理解・体験しよう / toolkit-for-flexible-actor-models-in-php-phluxor
ytake
1
410
オブジェクトのおしゃべり大失敗 メッセージングアンチパターン集 / messaging anti-pattern collection
ytake
2
1.4k
DRE/SREのプラクティス融合によるクラウドネイティブなデータ基盤作り / dre_sre
ytake
0
1.1k
Other Decks in Technology
See All in Technology
個人ASを取得して、自宅Kubernetes上のVMをグローバルIPで公開した話
munenick
0
210
マネージャーをエスパーにしない マネジメントされる側から始める、働きやすい現場づくり
comi190327
1
830
攻撃と守りは何が変わり、何が変わらないのか~最近の不正アクセスから考える、リスク評価と標準の使いどころ
oidfj
2
260
Re:ゼロから始めるシェアハウスのペットな鯖
silmin_
0
260
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
130
ノードが消える前提で設計する!Karpenterで金融システムを止めないためのノウハウ大全
daitak
1
160
閉域網でのAPI公開の選択肢全部比較!
takano0131
4
620
エンタープライズ事業未経験のデータ屋、 AI時代にN回撃沈する.pdf
ryoskdara_
0
170
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
230
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
860
認知の質も文化醸成も。数年取り組んで見えてきた技術広報のKPI設計 / Tech PR KPIs: Lessons on Awareness Quality and Culture After Several Years
bitkey
PRO
0
140
異夢同船〜異なる夢でも同じ船でなんとかする〜
fortegp05
1
560
Featured
See All Featured
SEO Brein meetup: CTRL+C is not how to scale international SEO
lindahogenes
2
3k
The SEO Collaboration Effect
kristinabergwall1
1
620
Large-scale JavaScript Application Architecture
addyosmani
516
110k
Producing Creativity
orderedlist
PRO
348
41k
It's Worth the Effort
3n
188
29k
Conquering PDFs: document understanding beyond plain text
inesmontani
PRO
4
3.3k
Navigating Team Friction
lara
192
16k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
livdayseo
1
250
Kristin Tynski - Automating Marketing Tasks With AI
techseoconnect
PRO
0
600
How To Speak Unicorn (iThemes Webinar)
marktimemedia
1
640
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
508
140k
Amusing Abliteration
ianozsvald
1
400