Exploring the History of a 12-year-old Rails Application

Exploring the History of a 12-year-old Rails Application

From RailsConf 2017, Phoenix, AZ.

1932c0ac21b6792f2572d8e56f84a1c2?s=128

Nathan L. Walls

April 25, 2017
Tweet

Transcript

  1. EXPLORING THE HISTORY OF A 12-YEAR-OLD RAILS APPLICATION Nathan L.

    Walls RailsConf 2017 — Phoenix, AZ
  2. A SMALL SLIVER OF OUR STORY

  3. …WELL, A SLIVER AT SOME DISTANCE

  4. A PROLOGUE

  5. OCT. 2015 An upgrade from Rails 2.3 and Ruby 1.8.7

    was in progress I was on the project, gaining experience with the app suite One of the pieces we ended up needing to upgrade was lots of JavaScript
  6. UPGRADING JAVASCRIPT Prototype was going away in favor of jQuery

    We had a lot of link_to_remote and link_to_remote_js calls In some places, what we had wasn't really JavaScript (Hi, RJS!)
  7. UPGRADING TO NON-RUBY- FLAVORED JAVASCRIPT I'm working on upgrading a

    substantial amount of JavaScript as part of this Rails upgrade. And I come across some early, early code. I run git blame…
  8. … LEADS TO DISCOVERY I see that some of the

    code I'm working on is from the first committer on the project.
  9. … AND DISCUSSION

  10. I GOT CURIOUS ABOUT HISTORY Looked at mix of code

    across several of our connected codebases Interviewed members of the development team, past and present Interviewed some members of the business team
  11. WHAT WE'RE GOING TO TOUCH ON How the team has

    grown and changed How the code has grown How the code has changed
  12. WHAT WE'RE GOING TO TOUCH ON Changes in how developers

    and the business interact Team changes Reflections and some available lessons
  13. QUOTES "In the absence of logic, there's history" "It seemed

    like a good idea at the time"
  14. ACT I: A FRAMEWORK APPEARS

  15. OVER AT VITALSOURCE TECHNOLOGIES… Had a business process of building/assembling

    ebooks. There was Java in place doing the work now But, there was a desire to use a visual process
  16. THIS RAILS THING… Dan Benjamin had been experimenting with Rails

    He liked it more than a PHP/Java combo So, Dan and Damon Clinksales started building
  17. GETTING GOING… This got started at an Apple World Wide

    Developer Conference Dan and Damon, skipped most of the sessions to write this new Rails application Dan focused on the front-end application Damon was focused on back-end pieces
  18. A LOT WAS BRAND NEW OR UNAVAILABLE AJAX was new

    Prototype and jQuery didn't yet exist REST conventions were not yet part of Rails API pieces were still using XML-RPC
  19. MORE NEW STUFF Deployments didn't have the well-worn path of

    Capistrano (yet) A lot of what we have common expectations around in Rails didn't yet exist
  20. BUT, THIS RAILS APPLICATION LAUNCHES

  21. ONE CONTROLLER

  22. THE NEXT APPLICATION P2Services, a services layer, came along in

    Jan 2006.
  23. OTHER MILESTONES Rails hits 1.0, Dec. 2005 Rails 1.1 (RJS),

    March 26, 2006 Rails 1.2 (REST!), Jan. 2007 Rails 2.0, Dec. 2007 Rails 2.3, March 2009
  24. ACT II Growing the business, the code, and frustrations

  25. RAILS MARCHING FORWARD Heroku founded GitHub founded Passenger is out

    Rails is riding high
  26. THE BUSINESS WAS GROWING …and starting to outgrow how the

    development organization worked Tension of building out features vs. interrupt- driven research Teams had managers, but work dictated directly by the business Developer interaction with direct manager was minimal
  27. AS THE APPLICATIONS GREW… Developers had a lot of area/task

    specialization
  28. THE NUMBER OF APPS GREW Phoenix: August, 2005 vst-models:, Jan.,

    2006 p2services: Jan., 2006 goose: Feb., 2007 reporter: Aug., 2007 uber-reporter: Sept., 2009 jiffy: April, 2010
  29. THERE WAS ALSO A LIMITATION These, and other applications, would

    shift versions of Ruby and Rails in lockstep The idea, was to keep machines these apps were deployed to all the same.
  30. MEANWHILE… Rails hits 3.0 (Aug. 2010) Ruby hits 1.9.3 (Oct.

    2011) Rails 3.2 (Jan. 2012) Ruby 2.0 (Feb. 2013) Rails 4.0 (June 2013) Rails 4.2 (Dec. 2014)
  31. TECHNICAL MANAGERS ARRIVE New CTO (2013) New development director

  32. PROCESS + ORGANIZATIONAL CHANGES Starting to get out of the

    mindset of "ship it as soon as possible" Building things to be (more) testable Lifting some of the business pressure Agile practices introduced Product manager role brought on
  33. WORK CONTINUED, BUT UPGRADES DID NOT The lockstep upgrade requirement

    was lifted. But, as consensus was building for an upgrade, the company bought its primary competitor (Q1, 2014).
  34. ABSORBING COURSESMART The teams had to spend roughly the next

    18 months (2014 to mid 2015) to: Migrate users Merge functionality from CourseSmart into VST applications Begin work on a new, combined, storefront
  35. AND SO, RAILS AND RUBY KEPT MOVING And these now

    5-10 year old applications were kept at Ruby 1.8.7 and Rails 2.3
  36. ACT III The Big Upgrade

  37. FALL 2015 Most of the work integrating CourseSmart was complete

    A plan was made to get the applications upgraded How? CTO made it clear that security was at issue Most support mechanisms for Ruby 1.8.7 and Rails 2.3 had ceased
  38. HOW THE UPGRADE HAPPENED Mostly parallel effort across each of

    the applications Stepwise through Rails 3.0, 3.1, 3.2, 4.0, 4.1, 4.2 Similarly stepwise through Ruby
  39. UPGRADE CHALLENGES Mixed urgent work and upgrade work Keeping diverging

    codebases shippable Coordinating cross-application dependencies
  40. NOTABLE CHALLENGES Addressing items deprecated from earlier versions of Rails

    getting to Rails 3.0 Moving off of RJS templates to native JavaScript Transitioning fully to ERb Incorporating the Asset Pipeline Restructuring ActiveRecord queries
  41. TEAMS ALSO SHIFTED An API/Platform team had been started, had

    a changeover in personnel, then grew Additional were developers hired for Connect applications Role of product management grew
  42. NEW PRODUCT AND TEAM As the ugprade project wound down

    in early 2016, a new product started taking shape. It was not as strongly coupled to the existing database, instead using the API-layer where needed instead Rails on the server-side, React on the client-side
  43. TECHNICAL CULTURE SHIFTS The beginnings of increased code review culture

    and more collaborative work.
  44. ACT IV Growing the code with a guided technical approach

  45. REVIEW CULTURE GETS STRONGER, EARLIER Increased adoption of code review

    and pull requests vs. merge and pray Moved essential reviews from before production deployment to before QA deployment Moved reviews away from a single lead engineer to a team-based approach
  46. API V4 STARTED JSON as first-preference, XML supported for external

    APIs Return meaningful error messages Designed from experience with earlier versions and integrations
  47. MORE OPPORTUNISTIC REFACTORING Lots of model/logic extractions A lot less

    controller code for new work Newer apps making use of service objects, use cases
  48. MOVING TOWARD API- BASED CORE DATA Move away from shared

    models in time Move away from shared migrations in time Use an API for new functionality where records are kept in the Phoenix DB
  49. GROWING APIS AND INTEGRATIONS AS A BUSINESS Improving documentation and

    performance for internal and external integrations.
  50. STAY CURRENT Teams can make their own best decisions on

    upgrading to new versions of Rails and Ruby Updates to Ruby 2.4 and Rails 5.0 pending
  51. UX REFRESH We'll see an example later, but we should

    give these critical applications the benefit of guidance from UX professionals.
  52. INCREASING AUTOMATED TEST COVERAGE Adding missing automated test coverage around

    critical business functionality Developing a technical culture of keeping the build green Identifying and rectifying obstacles to keeping builds green (like coupling) Writing code to be easier to test Leading with test automation for new feature work
  53. REFINING PROCESSES Teams transitioning from Scrum to Kanban and iterating

    on their own practices Balancing new business features with refactoring, technical stability
  54. MOVING SOME APPLICATIONS TO RETIREMENT p2services retired for more up-to-date

    APIs Reporter and Uber Reporter being replaced by a new, more capable reporting application under development
  55. TOOLS AND METHODS TO DIG FURTHER INTO HISTORY Identifying where

    it's interesting to look, and following an available path.
  56. CODE CLIMATE CHURN/COMPLEXITY

  57. GITHUB'S CODE FREQUENCY GRAPHS

  58. ANOTHER CODE FREQUENCY GRAPH

  59. GITHUB'S HISTORY FOR INDIVIDUAL FILES

  60. PROJECT RELEASE HISTORY Rails Your gems and other external dependencies

    Your own project release history
  61. GIT LOG An incredibly rich tool for exploration with filtering

    A great way to export commit data for further analysis and visualization
  62. VISUALIZING GIT COMMITS

  63. EXPORTING DATA git log --after='2005-07-01' --pretty=format:'%h|%an|%ae|%ai|%s|' \ --shortstat --no-merges >

    ~/Desktop/phoenix-data-log.txt ./scripts/git_log_cleaner.rb ~/Desktop/phoenix-data-log.txt \ > ~/Desktop/phoenix-data-log-cleaned.txt 23b08fa|Graham Turner|turnertgraham@example.com|2017-04-11 15:52:15 -0400|up a91ced7|Graham Turner|turnertgraham@gmail.com|2017-04-11 15:50:48 -0400|crea 1e3cf79|Erin Szarpa|erin.szarpa@example.com|2017-04-11 11:47:10 -0400|CON-31 0ef7afc|Todd Edwards|todd.edwards@example.com|2017-04-11 10:17:39 -0400|Upda 4c1ef08|Erin Szarpa|erin.szarpa@example.com|2017-04-10 17:19:35 -0400|CON-31 6e198a4|Todd Edwards|todd.edwards@example.com|2017-04-10 17:01:35 -0400|Upda
  64. GIT ITSELF Git itself can act as a time machine,

    so if you're really interested, you can get a past version of your app up and running.
  65. TALKING Finding out who was involved in your application over

    time, past and present. Developers QA Business owners
  66. TALKING If something seems strange or interesting, ask about it

    Ask with kindness Be understanding about evolving knowledge Be understanding about past business and technical circumstances
  67. LESSONS AVAILABLE

  68. LESSONS AVAILABLE Hitting MVP was critical because the business was

    overworked and needed more automation and better tooling Switching from internal access to allowing customers into the tool was a big shift that, years later, we are still reacting to Waiting so long to upgrade was somewhat painful, stalling innovation
  69. LESSONS AVAILABLE There is more we can, and should do

    to refactor business logic, reducing coupling and making better use of Single Responsibility.
  70. LESSONS AVAILABLE Shared migrations and models have not served us

    particularly well How the apps are coupled together at the database layer and with shared code vs. a true API has slowed progress
  71. LESSONS AVAILABLE There's joy to be found in writing Rails

    vs. other software development
  72. LESSONS AVAILABLE If we had understood the scale we'd be

    at, we'd have modeled some data differently and structured the application differently.
  73. LESSONS AVAILABLE Make API errors clear and descriptive Don't use

    HTTP status 200 for all responses ;)
  74. LESSONS AVAILABLE Expect that your prior colleagues did the best

    work they could, given their knowledge, understanding and circumstances.
  75. LESSONS AVAILABLE For these particular applications, Rails generally serves us

    very well
  76. LESSONS AVAILABLE The overall consistency and stability of these applications

    have been a surprise.
  77. LESSONS AVAILABLE Change can come: Old

  78. LESSONS AVAILABLE Change can come: New

  79. LESSONS AVAILABLE The business took a risk Rails wasn't at

    1.0 There was a lot of trust to build a business critical application in something very new
  80. LESSONS AVAILABLE There's a mix of astonishment at how long

    this has lasted… …and some sense that it has more potential to fulfill.
  81. LESSONS AVAILABLE Be curious, ask questions, listen.

  82. INTERVIEWS AND RESEARCH ASSISTANCE Willie Abrams Chris Amidon Dan Benjamin

    Kendall Bridges William Chesser Damon Clinksales Rick Johnson Manton Reece Ryan Schwartz Tommy Stokes Erin Szarpa
  83. SOFTWARE Rails 5.0 Ruby 2.4 High Voltage Reveal.js Chartkick

  84. NATHAN L. WALLS Twitter: @base10 Site: wallscorp.us Slides (soon): wallscorp.us/presentations

  85. THANK YOU! I appreciate you all being here today. Enjoy

    the rest of your conference.