High Availability PHP (Nomad PHP January 2018)

Transcript

1. High Availability PHP Josh Butts Nomad PHP - January 2018
2. About Me • SVP of Engineering,
 Ziff Davis • Austin PHP Organizer • github.com/jimbojsb • @jimbojsb 2
3. Agenda • What can we consider highly available? • How do we mitigate risk? • Why are containers well suited for HA? • Recommendations • Lessons learned the hard way 3
4. Opinion vs Fact • This talk is based on my opinions • There are many different ways to do things • If I trash your favorite, we can still be friends • Why am I even qualified to talk about this? 4
5. This is not a tutorial • There’s no way I can show you enough in an hour to build all this from scratch • See what ideas might apply to your systems • Commit to incremental improvements 5
6. What is high availability? • Your stuff just doesn’t go down • Like ever • This is not just a happy coincidence 6
7. How often are you down • 99% Uptime = Down 7h/month • 99.9% Uptime = Down 45m/month • 99.99% Uptime = Down <5m/month • 99.999% Uptime = Down <30s/month 7
8. What should we shoot for? • Minimum of “4 9’s” • “5 9’s” is totally doable • HA costs real money • Balance potential loss against costs 8
9. How to calculate your risk tolerance • Log in to your AWS account • Hand me your laptop • I will terminate one EC2 instance of my choosing • How long will you let me sit there? 9
10. But seriously… • Risk mitigation costs money • Consider battery backups as an example • Asking “how much reliability do you want” is a silly question • Make these decisions with hard numbers, not feelings 10
11. Obligatory Metaphors • Until the late 2000’s, we treated servers like pets • Then with Chef, Puppet, Ansible, etc we treated them like cattle • Now we can treat them like ants! 11
12. Example App Ecosystem 12 PHP Web App API Scheduled Jobs Queue Workers Database Cache Job Queue Uploaded Files
13. Lets start with hardware • All these tactics work great with cloud providers (doesn’t have to be AWS) • You need at least 2 of everything • You need a plan for how to fail • You need a replacement plan 13
14. Self-healing systems • If a server ceases to exist, it should be replaced without human interaction’ • AWS Cloud Formation and Elastic Beanstalk are good options • Terraform for non-AWSers 14
15. What about my devops tools? • I’ve already got all this ____ stuff set up • Docker can obviate all of that • You can still use these things if you must • Who runs the scripts? • Is the ____ server highly available? 15
16. Why Docker? • Immutable, disposable infrastructure • Requires no bootstrapping if using a Docker-friendly OS • Don’t have to care about what is running where, just that you have enough hardware 16
17. Containerize All The Things! • This isn’t just about containers for the sake of containers • The container way of thinking leads you down the right path 17
18. docker run Is Not Sufficient • Just like with building apps, you’re going to want a framework • API-based deployment and scheduling of containers • Something to wrangle hardware 18
19. Don’t worry, this is a solved problem • Kubernetes • Mesosphere / DCOS • Docker Swarm • Rancher 19
20. Containers and Schedulers • Common to run multiple containers on one piece of hardware • What if that hardware goes down? • What if US-East-1D goes down? 20
21. Example 21
22. I’m so tired of hearing people talk about Docker • This is not a Docker talk, I promise • Containers breed immutable, repeatable infrastructure • Immutable infrastructure is disposable and replaceable • Containers breed 12-factor apps • 12-factor apps are modular enough to facilitate true HA 22
23. Database • What does your I/O load look like • Split reads and writes • AWS Aurora if applicable • You really need at least 2 of your biggest server • Maintenance windows? 23
24. Disk Storage • Try to avoid local disk storage of anything • Put PHP sessions in a memory cache • Upload files directly to S3 • FlySystem is your friend, especially for development 24
25. Cache • How important is your cache? • Does your app work if the cache disappears? • Make sure it’s not the source of truth • Sharding vs Replication for scale 25
26. The Basics 26
27. Getting Started 27
28. Now IP addresses are broken 28 $clientIp = $_SERVER['REMOTE_ADDR']; // 1.2.3.4:80 - address of load balancer
29. Lets fix IPs 29 $clientIp = 0; if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) { $possibleIp = $_SERVER['HTTP_X_FORWARDED_FOR']; if (strpos($possibleIp, ',') !== false) { $ipList = explode(',', $possibleIp); foreach ($ipList as $ip) { $ip = trim($ip); if (filter_var($ip, FILTER_VALIDATE_IP)) { $clientIp = $ip; break; } } } else { $clientIp = $possibleIp; } } else { $clientIp = $_SERVER['REMOTE_ADDR'] ?? null; }
30. Now lets fix the databases 30
31. App Considerations 31 $dbs = ["db1.site.com", "db2.site.com", "db3.site.com"]; $slaveNum = mt_rand(0, count($dbs) - 1); $pdo = new \PDO($dbs[$slaveNum]);
32. Better Version 32 $dbs = ["db1.site.com", "db2.site.com", "db3.site.com"]; $slaveNum = mt_rand(0, count($dbs) - 1); try { $pdo = new \PDO($dbs[$slaveNum]); } catch (\PDOExcetion $e) { for ($i = 0; $i < count($dbs); $i++) { if ($i != $slaveNum) { try { $pdo = new \PDO($dbs[$slaveNum]); break; } catch (\PDOException $e) { } } } if (!$pdo) { throw new \RuntimeException("all out of DBs"); } }
33. Now we need more load balancers 33
34. Lets not forget caching! 34
35. App Considerations 35 require_once __DIR__ . '/vendor/autoload.php'; $client = new \Predis\Client(); $value = $client->get("myvalue"); if (!$value) { $value = reallyExpensiveFunction(); $client->set("myvalue", $value); }
36. Better Version 36 class Cache { private $redis; public function get($key) { try { return $this->redis->get($key); } catch (Exception $e) { return null; } } public function set($key, $value) { try { return $this->redis->set($key, $value); } catch (Exception $e) { } } }
37. • Consider a circuit breaker pattern for other external services • https://github.com/ offers/rho 37 Circuit Breaker Pattern
38. Service Discovery Overview • Distributed data stores that are a registry of what servers are where • Your code connects to these instead of using a config file • Even if you had to update it manually, it’d be faster than deploying 38
39. Service Discovery • Etcd • Consul • Zookeeper • Oh by the way, these all need their own cluster of at least 3 servers 39
40. Quick Service Discovery Example 40 $etcd = new LinkOrb\Component\Etcd\Client($etcdClusterHostname); $dbs = $etcClient->get("/database/slaves"); $slaveNum = mt_rand(0, count($dbs) - 1); try { $pdo = new \PDO($dbs[$slaveNum]); } catch (\PDOExcetion $e) { //... }
41. The problem with service discovery • Latency • Each lookup takes approximately 10ms • If you have to look up DB, Cache, ElasticSearch, SMTP, etc, it adds up • Try to organize services by logical application, so you can query for a whole namespace at once 41
42. Updated Discovery Example 42 $etcd = new LinkOrb\Component\Etcd\Client($etcdClusterHostname); $config = = $etcClient->get("/services/myapp"); $dbs = $config["databases"]["slaves"] $slaveNum = mt_rand(0, count($dbs) - 1); try { $pdo = new \PDO($dbs[$slaveNum]); } catch (\PDOExcetion $e) { //... }
43. Questions?