Exploring the History of a 12-year-old Rails Application

Slide 1

Slide 1 text

EXPLORING THE HISTORY OF A 12-YEAR-OLD RAILS APPLICATION Nathan L. Walls RailsConf 2017 — Phoenix, AZ

Slide 2

Slide 2 text

A SMALL SLIVER OF OUR STORY

Slide 3

Slide 3 text

…WELL, A SLIVER AT SOME DISTANCE

Slide 4

Slide 4 text

A PROLOGUE

Slide 5

Slide 5 text

OCT. 2015 An upgrade from Rails 2.3 and Ruby 1.8.7 was in progress I was on the project, gaining experience with the app suite One of the pieces we ended up needing to upgrade was lots of JavaScript

Slide 6

Slide 6 text

UPGRADING JAVASCRIPT Prototype was going away in favor of jQuery We had a lot of link_to_remote and link_to_remote_js calls In some places, what we had wasn't really JavaScript (Hi, RJS!)

Slide 7

Slide 7 text

UPGRADING TO NON-RUBY- FLAVORED JAVASCRIPT I'm working on upgrading a substantial amount of JavaScript as part of this Rails upgrade. And I come across some early, early code. I run git blame…

Slide 8

Slide 8 text

… LEADS TO DISCOVERY I see that some of the code I'm working on is from the ﬁrst committer on the project.

Slide 9

Slide 9 text

… AND DISCUSSION

Slide 10

Slide 10 text

I GOT CURIOUS ABOUT HISTORY Looked at mix of code across several of our connected codebases Interviewed members of the development team, past and present Interviewed some members of the business team

Slide 11

Slide 11 text

WHAT WE'RE GOING TO TOUCH ON How the team has grown and changed How the code has grown How the code has changed

Slide 12

Slide 12 text

WHAT WE'RE GOING TO TOUCH ON Changes in how developers and the business interact Team changes Reﬂections and some available lessons

Slide 13

Slide 13 text

QUOTES "In the absence of logic, there's history" "It seemed like a good idea at the time"

Slide 14

Slide 14 text

ACT I: A FRAMEWORK APPEARS

Slide 15

Slide 15 text

OVER AT VITALSOURCE TECHNOLOGIES… Had a business process of building/assembling ebooks. There was Java in place doing the work now But, there was a desire to use a visual process

Slide 16

Slide 16 text

THIS RAILS THING… Dan Benjamin had been experimenting with Rails He liked it more than a PHP/Java combo So, Dan and Damon Clinksales started building

Slide 17

Slide 17 text

GETTING GOING… This got started at an Apple World Wide Developer Conference Dan and Damon, skipped most of the sessions to write this new Rails application Dan focused on the front-end application Damon was focused on back-end pieces

Slide 18

Slide 18 text

A LOT WAS BRAND NEW OR UNAVAILABLE AJAX was new Prototype and jQuery didn't yet exist REST conventions were not yet part of Rails API pieces were still using XML-RPC

Slide 19

Slide 19 text

MORE NEW STUFF Deployments didn't have the well-worn path of Capistrano (yet) A lot of what we have common expectations around in Rails didn't yet exist

Slide 20

Slide 20 text

BUT, THIS RAILS APPLICATION LAUNCHES

Slide 21

Slide 21 text

ONE CONTROLLER

Slide 22

Slide 22 text

THE NEXT APPLICATION P2Services, a services layer, came along in Jan 2006.

Slide 23

Slide 23 text

OTHER MILESTONES Rails hits 1.0, Dec. 2005 Rails 1.1 (RJS), March 26, 2006 Rails 1.2 (REST!), Jan. 2007 Rails 2.0, Dec. 2007 Rails 2.3, March 2009

Slide 24

Slide 24 text

ACT II Growing the business, the code, and frustrations

Slide 25

Slide 25 text

RAILS MARCHING FORWARD Heroku founded GitHub founded Passenger is out Rails is riding high

Slide 26

Slide 26 text

THE BUSINESS WAS GROWING …and starting to outgrow how the development organization worked Tension of building out features vs. interrupt- driven research Teams had managers, but work dictated directly by the business Developer interaction with direct manager was minimal

Slide 27

Slide 27 text

AS THE APPLICATIONS GREW… Developers had a lot of area/task specialization

Slide 28

Slide 28 text

THE NUMBER OF APPS GREW Phoenix: August, 2005 vst-models:, Jan., 2006 p2services: Jan., 2006 goose: Feb., 2007 reporter: Aug., 2007 uber-reporter: Sept., 2009 jiffy: April, 2010

Slide 29

Slide 29 text

THERE WAS ALSO A LIMITATION These, and other applications, would shift versions of Ruby and Rails in lockstep The idea, was to keep machines these apps were deployed to all the same.

Slide 30

Slide 30 text

MEANWHILE… Rails hits 3.0 (Aug. 2010) Ruby hits 1.9.3 (Oct. 2011) Rails 3.2 (Jan. 2012) Ruby 2.0 (Feb. 2013) Rails 4.0 (June 2013) Rails 4.2 (Dec. 2014)

Slide 31

Slide 31 text

TECHNICAL MANAGERS ARRIVE New CTO (2013) New development director

Slide 32

Slide 32 text

PROCESS + ORGANIZATIONAL CHANGES Starting to get out of the mindset of "ship it as soon as possible" Building things to be (more) testable Lifting some of the business pressure Agile practices introduced Product manager role brought on

Slide 33

Slide 33 text

WORK CONTINUED, BUT UPGRADES DID NOT The lockstep upgrade requirement was lifted. But, as consensus was building for an upgrade, the company bought its primary competitor (Q1, 2014).

Slide 34

Slide 34 text

ABSORBING COURSESMART The teams had to spend roughly the next 18 months (2014 to mid 2015) to: Migrate users Merge functionality from CourseSmart into VST applications Begin work on a new, combined, storefront

Slide 35

Slide 35 text

AND SO, RAILS AND RUBY KEPT MOVING And these now 5-10 year old applications were kept at Ruby 1.8.7 and Rails 2.3

Slide 36

Slide 36 text

ACT III The Big Upgrade

Slide 37

Slide 37 text

FALL 2015 Most of the work integrating CourseSmart was complete A plan was made to get the applications upgraded How? CTO made it clear that security was at issue Most support mechanisms for Ruby 1.8.7 and Rails 2.3 had ceased

Slide 38

Slide 38 text

HOW THE UPGRADE HAPPENED Mostly parallel effort across each of the applications Stepwise through Rails 3.0, 3.1, 3.2, 4.0, 4.1, 4.2 Similarly stepwise through Ruby

Slide 39

Slide 39 text

UPGRADE CHALLENGES Mixed urgent work and upgrade work Keeping diverging codebases shippable Coordinating cross-application dependencies

Slide 40

Slide 40 text

NOTABLE CHALLENGES Addressing items deprecated from earlier versions of Rails getting to Rails 3.0 Moving off of RJS templates to native JavaScript Transitioning fully to ERb Incorporating the Asset Pipeline Restructuring ActiveRecord queries

Slide 41

Slide 41 text

TEAMS ALSO SHIFTED An API/Platform team had been started, had a changeover in personnel, then grew Additional were developers hired for Connect applications Role of product management grew

Slide 42

Slide 42 text

NEW PRODUCT AND TEAM As the ugprade project wound down in early 2016, a new product started taking shape. It was not as strongly coupled to the existing database, instead using the API-layer where needed instead Rails on the server-side, React on the client-side

Slide 43

Slide 43 text

TECHNICAL CULTURE SHIFTS The beginnings of increased code review culture and more collaborative work.

Slide 44

Slide 44 text

ACT IV Growing the code with a guided technical approach

Slide 45

Slide 45 text

REVIEW CULTURE GETS STRONGER, EARLIER Increased adoption of code review and pull requests vs. merge and pray Moved essential reviews from before production deployment to before QA deployment Moved reviews away from a single lead engineer to a team-based approach

Slide 46

Slide 46 text

API V4 STARTED JSON as ﬁrst-preference, XML supported for external APIs Return meaningful error messages Designed from experience with earlier versions and integrations

Slide 47

Slide 47 text

MORE OPPORTUNISTIC REFACTORING Lots of model/logic extractions A lot less controller code for new work Newer apps making use of service objects, use cases

Slide 48

Slide 48 text

MOVING TOWARD API- BASED CORE DATA Move away from shared models in time Move away from shared migrations in time Use an API for new functionality where records are kept in the Phoenix DB

Slide 49

Slide 49 text

GROWING APIS AND INTEGRATIONS AS A BUSINESS Improving documentation and performance for internal and external integrations.

Slide 50

Slide 50 text

STAY CURRENT Teams can make their own best decisions on upgrading to new versions of Rails and Ruby Updates to Ruby 2.4 and Rails 5.0 pending

Slide 51

Slide 51 text

UX REFRESH We'll see an example later, but we should give these critical applications the beneﬁt of guidance from UX professionals.

Slide 52

Slide 52 text

INCREASING AUTOMATED TEST COVERAGE Adding missing automated test coverage around critical business functionality Developing a technical culture of keeping the build green Identifying and rectifying obstacles to keeping builds green (like coupling) Writing code to be easier to test Leading with test automation for new feature work

Slide 53

Slide 53 text

REFINING PROCESSES Teams transitioning from Scrum to Kanban and iterating on their own practices Balancing new business features with refactoring, technical stability

Slide 54

Slide 54 text

MOVING SOME APPLICATIONS TO RETIREMENT p2services retired for more up-to-date APIs Reporter and Uber Reporter being replaced by a new, more capable reporting application under development

Slide 55

Slide 55 text

TOOLS AND METHODS TO DIG FURTHER INTO HISTORY Identifying where it's interesting to look, and following an available path.

Slide 56

Slide 56 text

CODE CLIMATE CHURN/COMPLEXITY

Slide 57

Slide 57 text

GITHUB'S CODE FREQUENCY GRAPHS

Slide 58

Slide 58 text

ANOTHER CODE FREQUENCY GRAPH

Slide 59

Slide 59 text

GITHUB'S HISTORY FOR INDIVIDUAL FILES

Slide 60

Slide 60 text

PROJECT RELEASE HISTORY Rails Your gems and other external dependencies Your own project release history

Slide 61

Slide 61 text

GIT LOG An incredibly rich tool for exploration with ﬁltering A great way to export commit data for further analysis and visualization

Slide 62

Slide 62 text

VISUALIZING GIT COMMITS

Slide 63

Slide 63 text

Slide 64

Slide 64 text

GIT ITSELF Git itself can act as a time machine, so if you're really interested, you can get a past version of your app up and running.

Slide 65

Slide 65 text

TALKING Finding out who was involved in your application over time, past and present. Developers QA Business owners

Slide 66

Slide 66 text

TALKING If something seems strange or interesting, ask about it Ask with kindness Be understanding about evolving knowledge Be understanding about past business and technical circumstances

Slide 67

Slide 67 text

LESSONS AVAILABLE

Slide 68

Slide 68 text

LESSONS AVAILABLE Hitting MVP was critical because the business was overworked and needed more automation and better tooling Switching from internal access to allowing customers into the tool was a big shift that, years later, we are still reacting to Waiting so long to upgrade was somewhat painful, stalling innovation

Slide 69

Slide 69 text

LESSONS AVAILABLE There is more we can, and should do to refactor business logic, reducing coupling and making better use of Single Responsibility.

Slide 70

Slide 70 text

LESSONS AVAILABLE Shared migrations and models have not served us particularly well How the apps are coupled together at the database layer and with shared code vs. a true API has slowed progress

Slide 71

Slide 71 text

LESSONS AVAILABLE There's joy to be found in writing Rails vs. other software development

Slide 72

Slide 72 text

LESSONS AVAILABLE If we had understood the scale we'd be at, we'd have modeled some data differently and structured the application differently.

Slide 73

Slide 73 text

LESSONS AVAILABLE Make API errors clear and descriptive Don't use HTTP status 200 for all responses ;)

Slide 74

Slide 74 text

LESSONS AVAILABLE Expect that your prior colleagues did the best work they could, given their knowledge, understanding and circumstances.

Slide 75

Slide 75 text

LESSONS AVAILABLE For these particular applications, Rails generally serves us very well

Slide 76

Slide 76 text

LESSONS AVAILABLE The overall consistency and stability of these applications have been a surprise.

Slide 77

Slide 77 text

LESSONS AVAILABLE Change can come: Old

Slide 78

Slide 78 text

LESSONS AVAILABLE Change can come: New

Slide 79

Slide 79 text

LESSONS AVAILABLE The business took a risk Rails wasn't at 1.0 There was a lot of trust to build a business critical application in something very new

Slide 80

Slide 80 text

LESSONS AVAILABLE There's a mix of astonishment at how long this has lasted… …and some sense that it has more potential to fulﬁll.

Slide 81

Slide 81 text

LESSONS AVAILABLE Be curious, ask questions, listen.

Slide 82

Slide 82 text

INTERVIEWS AND RESEARCH ASSISTANCE Willie Abrams Chris Amidon Dan Benjamin Kendall Bridges William Chesser Damon Clinksales Rick Johnson Manton Reece Ryan Schwartz Tommy Stokes Erin Szarpa

Slide 83

Slide 83 text

SOFTWARE Rails 5.0 Ruby 2.4 High Voltage Reveal.js Chartkick

Slide 84

Slide 84 text

NATHAN L. WALLS Twitter: @base10 Site: wallscorp.us Slides (soon): wallscorp.us/presentations

Slide 85

Slide 85 text

THANK YOU! I appreciate you all being here today. Enjoy the rest of your conference.