Our Journey to High Availability and Fast Load

When we started, our app was deployed in a single datacenter, causing outages over which we had no control, providing unoptimized experience for our non-local users, and other developer pain points. Join us as we go over the (mis)steps we took to resolve this by deploying to datacenters across multiple continents and configuring a global load balancer to provide the high availability and improved performance we sought. The talk will also include DNS configuration and testing, certificate management, and availability testing.

Transcript

1. Our Journey to High Availability and Fast Load Times via Global Load Balancing Michal Broz Kin Ueng
2. Presenters • Michal Broz • Kin Ueng
3. What we inherited • openliberty.io site • Source code (development) • CI/CD toolchain (operations)
4. • https://github.com/OpenLiberty/openliberty.io compromised of: • Mostly frontend (html, js, adoc) • Some Java backend (REST APIs, Filters, etc) • Some backend infrastructure enablement (server.xml) • Some cloud infrastructure (IBM Cloud stuff) • Controls the CI/CD • Mapping of app to address (manifest.yaml) • Build & Deploy (various bash scripts) Source Code (public GitHub repo)
5. CI/CD • Deployment on every commit • Build WAR • clones github repos on every commit • Deployed as a CloundFoundry (cf) app on a Liberty for Java applications buildpack • Blue/Green deployment (no downtime, minimal redundancy)
6. What Our World Looked Like Dallas Datacenter github.com/OpenLiberty/openliberty.io CI/CD Pipeline Blue/Green Deployment
7. OH NO: SITE OUTAGE! • Site outage due to network gateway going down • DNS: Akamai • Pointing to a static IP in IBM Cloud (Dallas datacenter) • Registrar: via AT&T • Quick Fix: Update static IP to new Static IP • Site used A Record (IP Address) • Proper Fix: Switch to CNAME (domain)
8. Our World Expands Dallas Datacenter github.com/OpenLiberty/openliberty.io CI/CD Pipeline AT&T Akamai New
9. OH NO: SITE OUTAGE!! • CloudFoundry in Dallas Datacenter goes down • Unable to use site during conference • Dallas had more incidents than other datacenters • Move Datacenter to Washington • Change pipeline to also deploy to Washington Datacenter • Pipeline remains in Dallas • Change CNAME to point at Washington • Too bad we only initially only moved www.openliberty.io (openliberty.io, still pointed at Dallas, but we fixed that soon thereafter)
10. Our World Expands Dallas Datacenter github.com/OpenLiberty/openliberty.io Washington Datacenter Build Deploy Akamai AT&T New
11. OH NO: We’re at 2 strikes, 3rd one and you’re… • What could cause our 3rd outage: • Long turnaround to make DNS changes via Akamai • Single Datacenter means single point of failure • Degrading performance as you get further away from Washington • Lots of European/west coast conferences • Hard deadline for making ‘risky’ environment changes • Huge conference coming up • ‘Conference season’ starting
12. Deploy to all datacenters • Screenshot of pipeline
13. Our World Expands Further Sydney Dallas Washington UK Germany Dallas Datacenter Deploy Build New Deploy Deploy Deploy Deploy Akamai AT&T
14. How to set up Global Load Balancing (GLB)
15. Testing GLB without any public DNS changes GLB openliberty.io environment Laptop Testing Akamai AT&T
16. How to test load balancing the wrong way • What we didn’t know • How to organize the Cloud Foundry routes • Under the impression openliberty.io subdomain was unavailable for testing • The wrong way • Performing tests with a mocked up DNS environment • The test setup • One global load balancer instance with hostname glb.openliberty.io • Leave production untouched • Hardcode DNS servers on laptop to do testing
17. What is glb.openliberty.io? glb.openliberty.io
18. Example of a pool used by glb.openliberty.io
19. Detailed view of glb.openliberty.io Laptop
20. dig command Old DNS servers
21. glb.openliberty.io CIS DNS assignment ns016.name.cloud.ibm.com ns017.name.cloud.ibm.com
22. Test routing to US East through the GLB dig glb.openliberty.io ns017.name.cloud.ibm.com
23. Browser Testing https://glb.openliberty.io ns016.name.cloud.ibm.com ns017.name.cloud.ibm.com
24. Regional Browser Testing with VPN. FAIL! VPN’s DNS server overriding my laptop’s DNS server settings!
25. How we successfully tested the GLB • Get AT&T to create test.openliberty.io • Update Akamai to route test.openliberty.io on IBM Cloud • Tell Akamai to use Cloudflare to resolve test.openliberty.io • Real-world testing
26. Migrate DNS zone file from Akamai to CIS
27. test.openliberty.io environment Dallas Datacenter github.com/OpenLiberty/openliberty.io CI/CD Pipeine openliberty.io environment Akamai AT&T AT&T Cloudflare
28. Multiple Certificate Issues (test.openliberty.io) • Incorrect certificates being sent to users • Domain mismatch (syd.ol.io) • GLB Health check fails for all pool due to TLS security • Custom certificate not valid for *.test.openliberty.io subdomains
29. Our Word, Untrusted Dallas Datacenter Build CIS GLB Region Route Sydney Datacenter Dallas Datacenter Washington Datacenter UK Datacenter Germany Datacenter Deploy Deploy Deploy Deploy Deploy Region Route Region Route Region Route Region Route Incorrect Certificate (Proxy Off) Pools Cloudflare
30. test.openliberty.io – GLB proxy off mode GLB Proxy Off Incorrect Certificates: *.mybluemix.net Correct Certificates: *.openliberty.io *.mybluemix.net *.mybluemix.net Dallas Datacenter
31. test.openliberty.io – proxy on (routing directly to app) CIS GLB Incorrect Certificates Correct Certificates 404 Dallas Datacenter
32. test.openliberty.io – proxy on (routing to correct datacenter endpoint) Incorrect Certificates Correct Certificates 404 us-south.test.openlibery.io != *.openliberty.io CIS GLB Dallas Datacenter
33. TLS Validation failed between GLB and origin server
34. GLB on hold; Migrate openliberty.io to CIS! • We know migrating domain from Akamai to CIS works from our experience with test.openliberty.io • Get rid of using *.test.openliberty.io domains • Keep openliberty.io hardcoded to point at a single datacenter • Validate the GLB on the side before routing openliberty.io to the GLB
35. us-south.openliberty.io – secure endpoints Incorrect Certificates Correct Certificates us-south.openlibery.io == *.openliberty.io CIS GLB Dallas Datacenter
36. Our World Now Dallas Datacenter Build CIS GLB Region Route Sydney Datacenter Dallas Datacenter Washington Datacenter UK Datacenter Germany Datacenter Deploy Deploy Deploy Deploy Deploy Region Route Region Route Region Route Region Route Trusted Certificates (Proxy On) Pools Cloudflare
37. Monitoring: Health Check • Supports 15 regions • Response & Load Times • Response Codes for resources • Alerts • Tip: Append a query parameter to URL for easy analytics filtering
38. Monitoring: Health Check
39. TODOs: ❑Caching • Enabled itself automatically when we turned on GLB ❑DDoS (IP whitelist) ❑Replicate pipeline on other datacenters
40. Thank You!