Exploring the History of a 12-year-old Rails Application

From RailsConf 2017, Phoenix, AZ.

Transcript

1. EXPLORING THE HISTORY OF A 12-YEAR-OLD RAILS APPLICATION Nathan L. Walls RailsConf 2017 — Phoenix, AZ
2. A SMALL SLIVER OF OUR STORY
3. …WELL, A SLIVER AT SOME DISTANCE
4. A PROLOGUE
5. OCT. 2015 An upgrade from Rails 2.3 and Ruby 1.8.7 was in progress I was on the project, gaining experience with the app suite One of the pieces we ended up needing to upgrade was lots of JavaScript
6. UPGRADING JAVASCRIPT Prototype was going away in favor of jQuery We had a lot of link_to_remote and link_to_remote_js calls In some places, what we had wasn't really JavaScript (Hi, RJS!)
7. UPGRADING TO NON-RUBY- FLAVORED JAVASCRIPT I'm working on upgrading a substantial amount of JavaScript as part of this Rails upgrade. And I come across some early, early code. I run git blame…
8. … LEADS TO DISCOVERY I see that some of the code I'm working on is from the first committer on the project.
9. … AND DISCUSSION
10. I GOT CURIOUS ABOUT HISTORY Looked at mix of code across several of our connected codebases Interviewed members of the development team, past and present Interviewed some members of the business team
11. WHAT WE'RE GOING TO TOUCH ON How the team has grown and changed How the code has grown How the code has changed
12. WHAT WE'RE GOING TO TOUCH ON Changes in how developers and the business interact Team changes Reflections and some available lessons
13. QUOTES "In the absence of logic, there's history" "It seemed like a good idea at the time"
14. ACT I: A FRAMEWORK APPEARS
15. OVER AT VITALSOURCE TECHNOLOGIES… Had a business process of building/assembling ebooks. There was Java in place doing the work now But, there was a desire to use a visual process
16. THIS RAILS THING… Dan Benjamin had been experimenting with Rails He liked it more than a PHP/Java combo So, Dan and Damon Clinksales started building
17. GETTING GOING… This got started at an Apple World Wide Developer Conference Dan and Damon, skipped most of the sessions to write this new Rails application Dan focused on the front-end application Damon was focused on back-end pieces
18. A LOT WAS BRAND NEW OR UNAVAILABLE AJAX was new Prototype and jQuery didn't yet exist REST conventions were not yet part of Rails API pieces were still using XML-RPC
19. MORE NEW STUFF Deployments didn't have the well-worn path of Capistrano (yet) A lot of what we have common expectations around in Rails didn't yet exist
20. BUT, THIS RAILS APPLICATION LAUNCHES
21. ONE CONTROLLER
22. THE NEXT APPLICATION P2Services, a services layer, came along in Jan 2006.
23. OTHER MILESTONES Rails hits 1.0, Dec. 2005 Rails 1.1 (RJS), March 26, 2006 Rails 1.2 (REST!), Jan. 2007 Rails 2.0, Dec. 2007 Rails 2.3, March 2009
24. ACT II Growing the business, the code, and frustrations
25. RAILS MARCHING FORWARD Heroku founded GitHub founded Passenger is out Rails is riding high
26. THE BUSINESS WAS GROWING …and starting to outgrow how the development organization worked Tension of building out features vs. interrupt- driven research Teams had managers, but work dictated directly by the business Developer interaction with direct manager was minimal
27. AS THE APPLICATIONS GREW… Developers had a lot of area/task specialization
28. THE NUMBER OF APPS GREW Phoenix: August, 2005 vst-models:, Jan., 2006 p2services: Jan., 2006 goose: Feb., 2007 reporter: Aug., 2007 uber-reporter: Sept., 2009 jiffy: April, 2010
29. THERE WAS ALSO A LIMITATION These, and other applications, would shift versions of Ruby and Rails in lockstep The idea, was to keep machines these apps were deployed to all the same.
30. MEANWHILE… Rails hits 3.0 (Aug. 2010) Ruby hits 1.9.3 (Oct. 2011) Rails 3.2 (Jan. 2012) Ruby 2.0 (Feb. 2013) Rails 4.0 (June 2013) Rails 4.2 (Dec. 2014)
31. TECHNICAL MANAGERS ARRIVE New CTO (2013) New development director
32. PROCESS + ORGANIZATIONAL CHANGES Starting to get out of the mindset of "ship it as soon as possible" Building things to be (more) testable Lifting some of the business pressure Agile practices introduced Product manager role brought on
33. WORK CONTINUED, BUT UPGRADES DID NOT The lockstep upgrade requirement was lifted. But, as consensus was building for an upgrade, the company bought its primary competitor (Q1, 2014).
34. ABSORBING COURSESMART The teams had to spend roughly the next 18 months (2014 to mid 2015) to: Migrate users Merge functionality from CourseSmart into VST applications Begin work on a new, combined, storefront
35. AND SO, RAILS AND RUBY KEPT MOVING And these now 5-10 year old applications were kept at Ruby 1.8.7 and Rails 2.3
36. ACT III The Big Upgrade
37. FALL 2015 Most of the work integrating CourseSmart was complete A plan was made to get the applications upgraded How? CTO made it clear that security was at issue Most support mechanisms for Ruby 1.8.7 and Rails 2.3 had ceased
38. HOW THE UPGRADE HAPPENED Mostly parallel effort across each of the applications Stepwise through Rails 3.0, 3.1, 3.2, 4.0, 4.1, 4.2 Similarly stepwise through Ruby
39. UPGRADE CHALLENGES Mixed urgent work and upgrade work Keeping diverging codebases shippable Coordinating cross-application dependencies
40. NOTABLE CHALLENGES Addressing items deprecated from earlier versions of Rails getting to Rails 3.0 Moving off of RJS templates to native JavaScript Transitioning fully to ERb Incorporating the Asset Pipeline Restructuring ActiveRecord queries
41. TEAMS ALSO SHIFTED An API/Platform team had been started, had a changeover in personnel, then grew Additional were developers hired for Connect applications Role of product management grew
42. NEW PRODUCT AND TEAM As the ugprade project wound down in early 2016, a new product started taking shape. It was not as strongly coupled to the existing database, instead using the API-layer where needed instead Rails on the server-side, React on the client-side
43. TECHNICAL CULTURE SHIFTS The beginnings of increased code review culture and more collaborative work.
44. ACT IV Growing the code with a guided technical approach
45. REVIEW CULTURE GETS STRONGER, EARLIER Increased adoption of code review and pull requests vs. merge and pray Moved essential reviews from before production deployment to before QA deployment Moved reviews away from a single lead engineer to a team-based approach
46. API V4 STARTED JSON as first-preference, XML supported for external APIs Return meaningful error messages Designed from experience with earlier versions and integrations
47. MORE OPPORTUNISTIC REFACTORING Lots of model/logic extractions A lot less controller code for new work Newer apps making use of service objects, use cases
48. MOVING TOWARD API- BASED CORE DATA Move away from shared models in time Move away from shared migrations in time Use an API for new functionality where records are kept in the Phoenix DB
49. GROWING APIS AND INTEGRATIONS AS A BUSINESS Improving documentation and performance for internal and external integrations.
50. STAY CURRENT Teams can make their own best decisions on upgrading to new versions of Rails and Ruby Updates to Ruby 2.4 and Rails 5.0 pending
51. UX REFRESH We'll see an example later, but we should give these critical applications the benefit of guidance from UX professionals.
52. INCREASING AUTOMATED TEST COVERAGE Adding missing automated test coverage around critical business functionality Developing a technical culture of keeping the build green Identifying and rectifying obstacles to keeping builds green (like coupling) Writing code to be easier to test Leading with test automation for new feature work
53. REFINING PROCESSES Teams transitioning from Scrum to Kanban and iterating on their own practices Balancing new business features with refactoring, technical stability
54. MOVING SOME APPLICATIONS TO RETIREMENT p2services retired for more up-to-date APIs Reporter and Uber Reporter being replaced by a new, more capable reporting application under development
55. TOOLS AND METHODS TO DIG FURTHER INTO HISTORY Identifying where it's interesting to look, and following an available path.
56. CODE CLIMATE CHURN/COMPLEXITY
57. GITHUB'S CODE FREQUENCY GRAPHS
58. ANOTHER CODE FREQUENCY GRAPH
59. GITHUB'S HISTORY FOR INDIVIDUAL FILES
60. PROJECT RELEASE HISTORY Rails Your gems and other external dependencies Your own project release history
61. GIT LOG An incredibly rich tool for exploration with filtering A great way to export commit data for further analysis and visualization
62. VISUALIZING GIT COMMITS
63. EXPORTING DATA git log --after='2005-07-01' --pretty=format:'%h|%an|%ae|%ai|%s|' \ --shortstat --no-merges > ~/Desktop/phoenix-data-log.txt ./scripts/git_log_cleaner.rb ~/Desktop/phoenix-data-log.txt \ > ~/Desktop/phoenix-data-log-cleaned.txt 23b08fa|Graham Turner|turnertgraham@example.com|2017-04-11 15:52:15 -0400|up a91ced7|Graham Turner|turnertgraham@gmail.com|2017-04-11 15:50:48 -0400|crea 1e3cf79|Erin Szarpa|erin.szarpa@example.com|2017-04-11 11:47:10 -0400|CON-31 0ef7afc|Todd Edwards|todd.edwards@example.com|2017-04-11 10:17:39 -0400|Upda 4c1ef08|Erin Szarpa|erin.szarpa@example.com|2017-04-10 17:19:35 -0400|CON-31 6e198a4|Todd Edwards|todd.edwards@example.com|2017-04-10 17:01:35 -0400|Upda
64. GIT ITSELF Git itself can act as a time machine, so if you're really interested, you can get a past version of your app up and running.
65. TALKING Finding out who was involved in your application over time, past and present. Developers QA Business owners
66. TALKING If something seems strange or interesting, ask about it Ask with kindness Be understanding about evolving knowledge Be understanding about past business and technical circumstances
67. LESSONS AVAILABLE
68. LESSONS AVAILABLE Hitting MVP was critical because the business was overworked and needed more automation and better tooling Switching from internal access to allowing customers into the tool was a big shift that, years later, we are still reacting to Waiting so long to upgrade was somewhat painful, stalling innovation
69. LESSONS AVAILABLE There is more we can, and should do to refactor business logic, reducing coupling and making better use of Single Responsibility.
70. LESSONS AVAILABLE Shared migrations and models have not served us particularly well How the apps are coupled together at the database layer and with shared code vs. a true API has slowed progress
71. LESSONS AVAILABLE There's joy to be found in writing Rails vs. other software development
72. LESSONS AVAILABLE If we had understood the scale we'd be at, we'd have modeled some data differently and structured the application differently.
73. LESSONS AVAILABLE Make API errors clear and descriptive Don't use HTTP status 200 for all responses ;)
74. LESSONS AVAILABLE Expect that your prior colleagues did the best work they could, given their knowledge, understanding and circumstances.
75. LESSONS AVAILABLE For these particular applications, Rails generally serves us very well
76. LESSONS AVAILABLE The overall consistency and stability of these applications have been a surprise.
77. LESSONS AVAILABLE Change can come: Old
78. LESSONS AVAILABLE Change can come: New
79. LESSONS AVAILABLE The business took a risk Rails wasn't at 1.0 There was a lot of trust to build a business critical application in something very new
80. LESSONS AVAILABLE There's a mix of astonishment at how long this has lasted… …and some sense that it has more potential to fulfill.
81. LESSONS AVAILABLE Be curious, ask questions, listen.
82. INTERVIEWS AND RESEARCH ASSISTANCE Willie Abrams Chris Amidon Dan Benjamin Kendall Bridges William Chesser Damon Clinksales Rick Johnson Manton Reece Ryan Schwartz Tommy Stokes Erin Szarpa
83. SOFTWARE Rails 5.0 Ruby 2.4 High Voltage Reveal.js Chartkick
84. NATHAN L. WALLS Twitter: @base10 Site: wallscorp.us Slides (soon): wallscorp.us/presentations
85. THANK YOU! I appreciate you all being here today. Enjoy the rest of your conference.