PHP projects beyond the LAMP stack - PHP Limburg BE 2016

Slides for my PHP projects beyond the LAMP stack talk at the 1 year anniversary of the PHP Limburg BE user group.

Transcript

1. projects Beyond the LAMP stack By Thijs Feryn PHP
2. Hi, I’m Thijs
3. I’m @ThijsFeryn on Twitter
4. I’m an Evangelist At
5. I’m an at Evangelist
6. I’m a at board member
7. LAMP Stack?
8. Linux Apache MySQL PHP
10. ✓ ~81% of the web ✓ Easy to learn ✓ Mature (PHP renaissance) ✓ Frameworks & CMS’s ✓ Lots of online resources ✓ THE COMMUNITY
12. ➡ Still considered a scripting language (by some) ➡ Slow(-ish) ➡ Internal variable structure causes overhead ➡ Everyone can program in PHP, unfortunately everyone does ➡ Doesn’t scale that well 
 (without the tricks)
13. CPU memory I/O PHP consumes lots of
14. It gets worse “at scale”
15. Faster PHP
19. ✓ Fast ✓ Just In Time compiler ✓ FastCGI support ✓ Drop-in replacement for PHP-FPM ✓ Hack language HHVM
20. HACK language <?hh class MyClass { const int MyConst = 0; private string $x = ''; public function increment(int $x): int { $y = $x + 1; return $y; } } Type annotations
21. HACK language <?hh function foo(): (function(string): string) { $x = 'bar'; return $y ==> $x . $y; } function test(): void { $fn = foo(); echo $fn('baz'); // barbaz } Lambdas != closures
22. HACK language <?hh namespace Hack\UserDocumentation\Async\Intro\Examples\Curl; async function curl_A(): Awaitable<string> { $x = await \HH\Asio\curl_exec("http://example.com/"); return $x; } async function curl_B(): Awaitable<string> { $y = await \HH\Asio\curl_exec("http://example.net/"); return $y; } async function async_curl(): Awaitable<void> { $start = microtime(true); list($a, $b) = await \HH\Asio\v(array(curl_A(), curl_B())); $end = microtime(true); echo "Total time taken: " . strval($end - $start) . " seconds" . PHP_EOL; } \HH\Asio\join(async_curl()); Async
23. What kind of PHP code do we run? Why does it slow down?
24. Waiting for DB or API
25. HHVM won’t really speed up that process.
26. Different approach
27. Different tools
28. Caching
29. Optimize database Optimize runtime A void A void
30. Don’t recompute if the data hasn’t changed
31. Varnish
32. Normally User Server
33. Reverse proxy User Varnish Server Proxy in the data center
34. ✓ Varnish Configuration Language ✓ Edge Side Include support ✓ Gzip compression/decompression ✓ Cache purging ✓ HTTP streaming ✓ Grace mode ✓ Configure backends ✓ Backend loadbalancing ✓ ACL protection ✓ VMODs in C Varnish
35. On the request side ✓ Only GET & HEAD ✓ No cookies ✓ No auth headers On the response side ✓ No “no-cache, no store” ✓ TTL > 0 ✓ No set-cookies When does Varnish cache?
36. Extend the default behavior in VCL
37. vcl 4.0; sub vcl_recv { if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.method != "GET" && req.method != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (hash); } Incoming request
38. sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (lookup); } sub vcl_purge { return (synth(200, "Purged")); } Compose hash key Evict cache keys
39. sub vcl_hit { if (obj.ttl >= 0s) { // A pure unadultered hit, deliver it return (deliver); } if (obj.ttl + obj.grace > 0s) { // Object is in grace, deliver it // Automatically triggers a background fetch return (deliver); } // fetch & deliver once we get the result return (fetch); } sub vcl_miss { return (fetch); } sub vcl_deliver { return (deliver); } Deliver output to client Fetch data from backend Or fetch if it’s stale Deliver stored object
40. sub vcl_backend_response { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Surrogate-control ~ "no-store" || (!beresp.http.Surrogate-Control && beresp.http.Cache-Control ~ "no-cache|no-store| private") || beresp.http.Vary == "*") { /* * Mark as "Hit-For-Pass" for the next 2 minutes */ set beresp.ttl = 120s; set beresp.uncacheable = true; } return (deliver); } Response from the backend
41. What to extend?
42. ✓Strip tracking cookies (Google Analytics, …) ✓Sanitize URL ✓URL whitelist/blacklist ✓PURGE ACLs ✓Edge Side Include rules ✓Alway cache static files ✓Extend hash keys ✓Override TTL ✓Define grace mode What to extend?
43. Caching in your architecture
44. Respect HTTP
45. Cache-control: public, max- age=3600, s-maxage=7200 Cache-control: no-cache, no- store VS
46. Cookies?
47. The example
48. Don’t cache header
49. Cache products for 1 hour
50. Cache menus for 1 day
51. Edge Side Includes
52. <!DOCTYPE html> <html> <head> <title>The Demo</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" href="/css/bootstrap.min.css"> <link rel="stylesheet" href="/css/bootstrap-theme.min.css"> <script src=“/js/jquery-2.1.4.min.js"></script> <script src="/js/bootstrap.min.js"></script> </head> <body> <nav class="navbar navbar-inverse navbar-fixed-top"> <esi:include src=“http://mysite.dev/nav" /> </nav> <div class="jumbotron"> <esi:include src="http://mysite.dev/jumbotron" /> </div> <div class="container"> {% block content %}{% endblock %} <hr> <footer> <esi:include src="http://mysite.dev/footer" /> </footer> </div> </body> </html> ESI tags
53. <!DOCTYPE html> <html> <head> <title>The Demo</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" href="/css/bootstrap.min.css"> <link rel="stylesheet" href="/css/bootstrap-theme.min.css"> <script src=“/js/jquery-2.1.4.min.js"></script> <script src="/js/bootstrap.min.js"></script> </head> <body> <nav class="navbar navbar-inverse navbar-fixed-top"> {{ render_esi(url('nav')) }} </nav> <div class="jumbotron"> {{ render_esi(url('jumbotron')) }} </div> <div class="container"> {% block content %}{% endblock %} <hr> <footer> {{ render_esi(url('footer')) }} </footer> </div> </body> </html> Twig template in Silex ESI or internal subrequest Uses HttpFragmen tServiceProv ider
54. Or just use AJAX Async Graceful degradition
55. Minimal VCL Let the application handle it
56. Where does it fit in?
57. ✓Cache pages and static assets ✓Fastest way ✓Hit rate may vary ✓Chop your content up in pieces ✓Use ESI or AJAX ✓Gateway to your application Where does Varnish fit in?
59. ✓Cache all images, js, css, wof, … ✓Cache product pages ✓Cache category pages ✓Cache parts of the layout (via ESI) ✓Cache CMS-ish pages Where does Varnish fit in?
60. You can also host your static files on a separate set of Nginx servers
61. Content Delivery Network
62. CDNs are nothing but a bunch of reverse caching proxies
63. Put the content where your user is
64. A voids network saturation
66. User specific content
67. Keeping track of state
68. Cookies
69. Stateful caching vs high hit rate
70. Not all pages can be cached
71. Make the data retrieval faster
72. Typical MySQL issues
73. Data is stored for flexibility, not for performance SQL (joins) allow different compositions of the same data
74. Offload the database
76. ✓ Key-value store ✓ Fast ✓ Lightweight ✓ Data stored in RAM ✓ ~Memcached ✓ Data types ✓ Data persistance ✓ Replication ✓ Clustering Redis
77. Redis $ redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> set mykey somevalue OK 127.0.0.1:6379> get mykey "somevalue
78. ✓ Strings ✓ Hashes ✓ Lists ✓ Sets ✓ Sorted sets ✓ Geo ✓ … Redis data types
79. Redis $ redis-cli 127.0.0.1:6379> hset customer_1234 id 1234 (integer) 1 127.0.0.1:6379> hset customer_1234 items_in_cart 2 (integer) 1 127.0.0.1:6379> hmset customer_1234 firstname Thijs lastname Feryn OK 127.0.0.1:6379> hgetall customer_1234 1) "id" 2) "1234" 3) "items_in_cart" 4) "2" 5) "firstname" 6) "Thijs" 7) "lastname" 8) "Feryn" 127.0.0.1:6379>
80. $ redis-cli 127.0.0.1:6379> lpush products_for_customer_1234 5 (integer) 1 127.0.0.1:6379> lpush products_for_customer_1234 345 (integer) 2 127.0.0.1:6379> lpush products_for_customer_1234 78 12 345 (integer) 5 127.0.0.1:6379> llen products_for_customer_1234 (integer) 5 127.0.0.1:6379> lindex products_for_customer_1234 1 "12" 127.0.0.1:6379> lindex products_for_customer_1234 2 "78" 127.0.0.1:6379> rpop products_for_customer_1234 "5" 127.0.0.1:6379> rpop products_for_customer_1234 "345" 127.0.0.1:6379> rpop products_for_customer_1234 "78" 127.0.0.1:6379> rpop products_for_customer_1234 "12" 127.0.0.1:6379> rpop products_for_customer_1234 "345" 127.0.0.1:6379> rpop products_for_customer_1234 (nil) 127.0.0.1:6379> Redis
81. Predis https://github.com/nrk/predis
82. Clustering
83. Where does it fit in?
84. ✓ Database/API cache ✓ PHP session storage ✓ Message queue (lists) ✓ NoSQL database ✓ Real-time data retrieval Where does Redis fit in?
86. ✓ Stock quantities ✓ Variable pricing information ✓ Shopping cart ✓ User profile information Where does Redis fit in?
87. Basically: Real-time & volatile data
88. BUT: MySQL can still remain “source of truth” database
90. ✓Full-text search engine ✓Analytics engine ✓NoSQL database ✓Lucene based ✓Built-in clustering, replication, sharding ✓RESTful interface ✓Schemaless ElasticSearch
91. { "name" : "Hijacker", "cluster_name" : "elasticsearch", "version" : { "number" : "2.1.0", "build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87", "build_timestamp" : "2015-11-18T22:40:03Z", "build_snapshot" : false, "lucene_version" : "5.3.1" }, "tagline" : "You Know, for Search" } http://localhost: 9200
92. POST /my-index {"acknowledged":true} POST/my-index/my-type { "key" : "value", "date" : "2015-05-10", "counter" : 1, "tags" : ["tag1","tag2","tag3"] } { "_index": "my-index", "_type": "my-type", "_id": "AU089olr9oI99a_rK9fi", "_version": 1, "created": true } Confirmation
93. GET/my-index/my-type/AU089olr9oI99a_rK9fi?pretty { "_index": "my-index", "_type": "my-type", "_id": "AU089olr9oI99a_rK9fi", "_version": 1, "found": true, "_source": { "key": "value", "date": "2015-05-10", "counter": 1, "tags": [ "tag1", "tag2", "tag3" ] } } Retrieve document by id Document & meta data
94. GET /my-index/_mapping?pretty { "my-index": { "mappings": { "my-type": { "properties": { "counter": { "type": "long" }, "date": { "type": "date", "format": "dateOptionalTime" }, "key": { "type": "string" }, "tags": { "type": "string" } } } } } } Schemaless? Not really … “Guesses” mapping on insert
95. POST /products { "mappings": { "product" : { "_id" : { "path" : "entity_id" }, "properties" : { "entity_id" : {"type" : "integer"}, "name" : { "type" : "string", "index" : "not_analyzed", "fields" : { "raw" : { "type" : "string", "analyzer": "english" } } }, "description" : { "type" : "string", "index" : "not_analyzed", "fields" : { "raw" : { "type" : "string", "analyzer": "english" } } }, "price" : {"type" : "double"}, "sku" : {"type" : "string", "index" : "not_analyzed"}, "created_at" : {"type" : "date", "format" : "YYYY-MM-dd HH:mm:ss"}, "updated_at" : {"type" : "date", "format" : "YYYY-MM-dd HH:mm:ss"} , "category" : { "type" : "string", "index" : "not_analyzed" } } } } } Explicit mapping at index creation time
96. Analyzed vs non-analyzed Full-text vs exact value Filter vs Query
97. Search
98. POST /products/product/_search?pretty { "query": { "match": { "name.raw": "Linen Blazer" } } } POST /products/product/_search?pretty { "query": { "filtered": { "query": { "match_all": {} }, "filter": { "term": { "name": "Linen Blazer" } } } } } Matches 2 products Matches 1 product
99. POST /products/product/_search?pretty { "query": { "filtered": { "filter": { "bool": { "must": [ { "range": { "price": { "gte": 100, "lte": 400 } } } ], "must_not": [ { "term": { "name": "Convertible Dress" } } ], "should": [ { "term": { "category": "Women" } }, { "term": { "category": "New Arrivals" } } ] } } } } }
100. Aggregations
101. Group by on steroids
102. POST /products/product/_search?pretty { "fields": ["category","price","name"], "query": { "match": { "name.raw": "blazer" } }, "aggs": { "avg_price": { "avg": { "field": "price" } }, "min_price" : { "min": { "field": "price" } }, "max_price" : { "max": { "field": "price" } }, "number_of_products_per_category" : { "terms": { "field": "category", "size": 10 } } } } Multi-group by & query
103. "aggregations": { "min_price": { "value": 455 }, "number_of_products_per_category": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "Blazers", "doc_count": 2 }, { "key": "Default Category", "doc_count": 2 }, { "key": "Men", "doc_count": 2 } ] }, "max_price": { "value": 490 }, "avg_price": { "value": 472.5 } } Aggregation output
104. Clustering
105. Where does it fit in?
106. ✓ Full-text search engine with drill-down search ✓ NoSQL database ✓ Big data analytics tool using Kibana Where does ElasticSearch fit in?
109. ✓ All product information ✓ All categories & attributes ✓ Log archive ✓ Both NoSQL DB & search engine Where does ElasticSearch fit in?
110. CPU memory I/O PHP consumes lots of
111. Depends on what you do with it
112. ✓ Image resizing ✓ Logging/reporting ✓ PDF generation ✓ Check-out on super busy event sites ✓ …
113. Offload the webserver
114. Worker scripts
115. ✓ Uses PHP-CLI ✓ Runs continuously ✓ Process forking ✓ Pthreads ✓ Run worker scripts in parallel ✓ Managed by supervisord Worker scripts
116. ✓ Sync MySQL & Redis ✓ Resize images ✓ Async logging & metrics ✓ Update quantities & prices Worker scripts
117. Message queues
119. ✓ Pub/sub ✓ Speaks AMQP protocol ✓ Supported by Pivotal ✓ Channels/Exchanges/ Queues ✓ Built-in clustering ✓ Reliable messaging RabbitMQ
121. https://github.com/ videlalvaro/php-amqplib
122. <?php require_once __DIR__ . '/vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; $connection = new AMQPConnection('127.0.0.1', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->queue_declare('hello', false, false, false, false); $msg = new AMQPMessage('Hello World!'); $channel->basic_publish($msg, '', 'hello'); echo " [x] Sent 'Hello World!'\n"; $channel->close(); $connection->close(); Send to queue
123. <?php require_once __DIR__ . '/vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPConnection; $callback = function($msg) { echo " [x] Received ", $msg->body, "\n"; }; $connection = new AMQPConnection('127.0.0.1', 5672, 'guest', 'guest'); $channel = $connection->channel(); echo ' [*] Waiting for messages. To exit press CTRL+C', "\n"; $channel->basic_consume('hello', '', false, true, false, false, $callback); while(count($channel->callbacks)) { $channel->wait(); } $channel->close(); $connection->close(); Receive from queue
124. Where does it fit in?
125. ✓ Take load away from user process ✓ Free up resources on frontend servers ✓ Elaborate messaging strategies ✓ Async event-based actions Where do RabbitMQ/workers fit in?
127. ✓ Synchronize stock and price changes between Redis & MySQL ✓ Synchronize product changes between ElasticSearch & MysQL ✓ Stock/price/sales notifications ✓ Async checkout for busy event sites Where do RabbitMQ/workers fit in?
128. But with most of these tools we still go through the PHP runtime …
129. Other backend tools to handle stateful data
130. SOA Microservices AJAX Websockets
132. ✓ Javascript runtime ✓ Async ✓ Event-driven ✓ Non-blocking I/O ✓ Callbacks ✓ Lightweight ✓ NPM packages ✓ Backend-code in Javascript NodeJS
133. const http = require('http'); const hostname = '127.0.0.1'; const port = 1337; http.createServer((req, res) => { res.writeHead(200, { 'Content-Type': 'text/plain' }); res.end('Hello World\n'); }).listen(port, hostname, () => { console.log(`Server running at http://${hostname}:${port}/`); });
134. ExpressJS framework for Node
135. var express = require('express'); var app = express(); app.get('/', function (req, res) { res.send('Hello World!'); }); app.post('/', function (req, res) { res.send('Got a POST request'); }); app.put('/user', function (req, res) { res.send('Got a PUT request at /user'); }); app.delete('/user', function (req, res) { res.send('Got a DELETE request at /user'); }); var server = app.listen(3000, function () { var host = server.address().address; var port = server.address().port; console.log('Example app listening at http://%s:%s', host, port); });
136. Gateway to ElasticSearch, RabbitMQ & Redis
137. var elasticsearch = require('elasticsearch');
 var express = require('express');
 var bodyParser = require('body-parser')
 var redis = require("redis");
 var amqp = require('amqplib/callback_api');
 
 var elasticsearchClient = new elasticsearch.Client({
 host: 'localhost:9200',
 log: 'error'
 });
 
 var redisClient = redis.createClient();
 redisClient.on("error", function (err) {
 console.log("Error " + err);
 });
 
 var app = express();
 app.use(bodyParser.urlencoded({ extended: false }))
 app.use(bodyParser.json()) Initialize
138. app.get('/products', function (req, res) {
 elasticsearchClient.search({
 index: 'thedemo',
 type: 'product',
 body: {
 query: {
 match_all: {}
 }
 }
 }).then(function (resp) {
 res.json(resp.hits.hits)
 }, function (err) {
 console.trace(err.message);
 });
 }); app.get('/products/:id([0-9]+)/stock', function (req, res) {
 redisClient.get(req.params.id+':stock', function(err, reply) {
 res.json(parseInt(reply));
 });
 }); Get products from ES Get stock from Redis
139. app.put('/products/:id([0-9]+)/stock', function (req, res) {
 
 var stock = req.body.stock;
 var action = req.body.action;
 
 if(action == 'increment') {
 redisClient.incrby(req.params.id+':stock',stock, function(err, reply) {
 amqp.connect('amqp://localhost', function(err, conn) {
 conn.createChannel(function(err, ch) {
 ch.assertExchange('stock', 'direct', {durable: false});
 ch.publish('stock', 'info', new Buffer(JSON.stringify({id: req.params.id, stock: parseInt(reply)})));
 res.json('Stock for product '+req.params.id+' is now '+parseInt(reply));
 });
 
 });
 });
 } else {
 redisClient.decrby(req.params.id+':stock',stock, function(err, reply) {
 amqp.connect('amqp://localhost', function(err, conn) {
 conn.createChannel(function(err, ch) {
 ch.assertExchange('stock', 'direct', {durable: false});
 ch.publish('stock', 'info', new Buffer(JSON.stringify({id: req.params.id, stock: parseInt(reply)})));
 res.json('Stock for product '+req.params.id+' is now '+parseInt(reply));
 });
 
 });
 });
 }
 }); Update stock in Redis Send message to queue
140. ➜ ~ node node.js Example app listening at http://:::3000 Running node.js
142. ✓ Compiled language for the web ✓ Invented by Google ✓ Strictly typed ✓ Feels like your average interpreted language ✓ Async features ✓ Built for systems programming ✓ REALLY fast ✓ Not object oriented Go(lang)
143. Where does it fit in?
144. Really fast workers Really fast APIs Could replace PHP workers Could replace NodeJS
145. Microservices
146. package main import ( "fmt" "log" "github.com/streadway/amqp" ) func failOnError(err error, msg string) { if err != nil { log.Fatalf("%s: %s", msg, err) panic(fmt.Sprintf("%s: %s", msg, err)) } } Initialize
147. func main() { conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/") failOnError(err, "Failed to connect to RabbitMQ") defer conn.Close() ch, err := conn.Channel() failOnError(err, "Failed to open a channel") defer ch.Close() err = ch.ExchangeDeclare( "stock", // name "direct", // type false, // durable false, // auto-deleted false, // internal false, // no-wait nil, // arguments ) failOnError(err, "Failed to declare an exchange") q, err := ch.QueueDeclare( "", // name false, // durable false, // delete when usused true, // exclusive false, // no-wait Runs from main function
148. conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/") failOnError(err, "Failed to connect to RabbitMQ") defer conn.Close() ch, err := conn.Channel() failOnError(err, "Failed to open a channel") defer ch.Close() err = ch.ExchangeDeclare( "stock", // name "direct", // type false, // durable false, // auto-deleted false, // internal false, // no-wait nil, // arguments ) failOnError(err, "Failed to declare an exchange") Initialize connection Initialize exchange
149. q, err := ch.QueueDeclare( "", // name false, // durable false, // delete when usused true, // exclusive false, // no-wait nil, // arguments ) failOnError(err, "Failed to declare a queue") err = ch.QueueBind( q.Name, // queue name "info", // routing key "stock", // exchange false, nil) failOnError(err, "Failed to bind a queue") Declare queue Bind to queue
150. msgs, err := ch.Consume( q.Name, // queue "", // consumer true, // auto-ack false, // exclusive false, // no-local false, // no-wait nil, // args ) failOnError(err, "Failed to register a consumer") forever := make(chan bool) go func() { for d := range msgs { log.Printf(" [x] %s", d.Body) } }() log.Printf(" [*] Waiting for messages. To exit press CTRL+C") <-forever } Consume messages Async processing
151. The end game
152. ✓ Cache pages (Varnish) ✓ Assemble content via ESI or AJAX ✓ Static assets on Nginx or CDN ✓ Business logic in lightweight API calls (NodeJS, Go) ✓ Key-value stores for volatile & real-time data in Redis ✓ ElasticSearch as a NoSQL database ✓ RabbitMQ for async communication ✓ Worker processes read from message queue End game
153. Use the right tool for the job
154. Use the tools you like