Making a Sports Car From a Horse-Drawn Carriage

How a three-man team moved a profitable company’s primary (but legacy) application from on-premises to the cloud, jumping PHP versions and hurtling through seven Laravel versions all in 1 year. What we did right, what we did wrong, and what we learned along the way.

Transcript

1. Making a Sports Car From a Horse-Drawn Carriage Patrick Guevara
2. A Horse-Drawn Carriage The app did everything: CRM, sales pipeline & scripting, purchase orders, IT assets, knowledge base, marketing & transactional emails, call- coaching, reporting, compensation, buying leads…
3. A Careening Carriage We were pushing that carriage as fast as it could go, sometimes restarting the database multiple times a day. But the company was profitable and wanted to grow.
4. Legacy Apps Make Money
5. $Legacy Legacy apps more than likely mean financial stability But they can become bottlenecks hindering future endeavors
6. $Stability We had two directives: 1. Make the app stable 2. Upgrade it But how do you make it stable?
7. To the cloud!
8. Disclaimer
9. Some context… PHP 5.5.9 Laravel 3 Ubuntu 14.04 MySQL 5.6
10. Critical Chokepoints Storage Drive (external NAS, mounted) Database locks
11. AWS At least PHP 7.0 Ubuntu 16.04 Aurora (MySQL 5.6 compatible)
12. Ok, Now to the cloud! 3 months to prepare Keep feature parity on two branches Luckily, PHP 5.5 to PHP 7.0 is fairly straightforward Need to use ENV variables
13. Test Environment Relative vs Absolute Paths
14. Test Environment RPC Function referencing localhost
15. Push it Live! Came in on a Sunday to reroute traffic Everything looked great
16. Hiccup #1 Everything looked great until 8am on Monday Could not handle the load at all Scaled up to r3.8xlarge; still not enough
17. Hiccup #1 - Fix PHP worker processes vastly under-scoped On-premises had 500 or so PHP children Cloud only had about 5
18. Hiccups #2 to #1000 Spent about 3 weeks chasing down issues This was painful for everyone
19. Curated Hiccups
20. Sweeping Sessions
21. Portable Document Failure
22. Slack Overload Every single exception, log, and debug went to #log channel in Slack Wildly unhelpful
23. The Busy Season™ Annual Enrollment Period (AEP) ~3 months during October - December
24. Directives Check-in 1. Make the app stable 2. Upgrade it
25. The Busy Season Promise™ 1. Stable (minimal downtime) 2. High Performance (fast, reliable) 3. Halt development (reduce entropy) 4. Upgrade core, relaunch in January
26. Turns Out…
27. Update on Carriage Robust infrastructure Uptime in weeks not hours Productive (happy) salesforce
28. Mind Map
29. Enter the Shift
30. Laravel Shift Path from Laravel 3 to Laravel 5.5 Set a schedule to volley back and forth
31. Laravel 3 to 4.2 New Laravel 4.2 app within Laravel 3 app Scripted move from 3 to 4.2 Super simple smoke test
32. Laravel 4.2 to 5.1 Jmac ran Shift from 4.2 to 5.1 This one was a doozy. Namespaces, Composer, autoloading, authentication, routing, views, models, Artisan, etc.
33. Laravel 5.1 Test A ton changed going to 5.1 But more changes were coming in 5.2, 5.3, and 5.4
34. Laravel 5.1 to 5.5 A no-brainer at ~$50 Wrote 300 acceptance tests over a few weeks
36. Factory
37. LDAP Auth From roll-your-own to open-source
41. The Next Hiccup Also switched from PHP 7.0 to PHP 7.1 Need to use OpenSSL instead of mcrypt Decrypt using Laravel 3 (PHP 7.0) mcrypt Encrypt using Laravel 5.5 (PHP 7.1) OpenSSL
42. We ready…
43. Deploy #1 Came in early on Tuesday to re-route traffic Almost immediately met with errors
44. Deploy #2 Came in again on Tuesday Quickly met with errors and had to rollback
45. Deploy #3, #4, #5… Came in early on Tuesday Took notes Created tickets Rolled back
46. Deploy Steps Backfill features from Prod to Alpha Git pull on Alpha Composer update Decrypt sensitive info (Laravel 3, Prod) Encrypt sensitive (Laravel 5, Alpha) Set ENV variables
47. Deploy Steps 2 Verify EFS mount points Verify Supervisor config Verify Laravel Horizon config Point LB target groups to Alpha Point cron jobs from Prod to Alpha Reset app configs stored in database Electric Boogaloo
48. Deploy #6 It happened!
49. Psych!
50. Deploy #7 It’s been 2 months…
51. The Greatest Compliment
52. – Agent in passing “Glad y’all rolled back because it’s been working a lot better!”
53. Kinda Anti-Climactic The actual deployment was anti-climactic We were cautiously optimistic the rest of the week
54. What Went Well? We got really good at identifying dark corners and blind spots We learned more by source-diving and upgrading than by just maintaining it
55. What Didn’t Go Well? Too many interruptions to production Data loss through deploy/rollback downtime and re-encrypt Not perfect yet (still fixing bugs)
56. What Did We Get? A more robust web app built on a modern framework A community of developers Modern tooling for performance and development Attractive code base for recruiting talent Lower maintenance cost from following convention over configuration
57. What I Learned Any kind of upgrade takes buy-in It’s not pretty In fact, it’s pretty ugly
58. Plan, Over-plan, & Commit
59. Find The Path Source-dive as deep as you can Make a mind map Identify bottlenecks
60. Chew On It Explore different strategies Define contingency plans Research other options: Whole Upgrade Partial Upgrade Partial Upgrade with Eventual Migration Off-The-Shelf Solution
61. Be thorough! -vvv
62. Win Buy-In Make the case for the upgrade (Increase velocity? Relieve maintenance burden?) Count job requisitions for current stack vs. upgraded stack Develop timeline with all the gotchas you can think of
63. Inherit and Improve
64. Questions? Comments? Please leave feedback! https://joind.in/talk/2d5b8 Say Hi on Twitter! @psamg