Escape Velocity - A Full Stack Migration

Tech debt is a serious issue with regard to the health of an organization's software development lifecycle. The longer one holds out on paying off debt, the more expensive the cost is - eventually. In this talk I will recap the strategies Sauce Labs has applied, and the lessons we have learned along the way, to successfully move beyond our legacy web stack. I will also show how we achieved this with zero downtime and a large number of innovative technologies built on top of the battle-proven Django framework.

Transcript

1. © Sauce Labs, Inc. Presented by Sebastian Tiedtke, Director of Engineering, Web February 25, 2015 Wednesday, February 25, 2015 6:00 PM to 8:30 PM wifi: SauceDjango / pwd: unchained Welcome to the Django Meetup
2. © Sauce Labs, Inc. Imagine you were a pro pitcher... How fast do you have to throw a baseball to launch it into outer space?
3. © Sauce Labs, Inc. 11,180 m/s 40,248 km/h 25,009 mi/h How fast do you have to throw a baseball to launch it into space?
4. © Sauce Labs, Inc. Whoa. That’s hella fast!
5. © Sauce Labs, Inc. Ever dealt with a large pile of tech debt? The baseball that keeps falling back to the ground no matter how hard you throw it to the sky.
6. © Sauce Labs, Inc. Sebastian Tiedtke Director of Engineering, Web https://twitter.com/sourishkrout Sauce Labs 539 Bryant Street, Ste. 303 San Francisco, CA 94102 Escape Velocity A Full Stack Migration
7. © Sauce Labs, Inc. © Sauce Labs, Inc. What is tech debt? “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.” - Ward Cunningham, 1992 Break free from high interest tech debt.
8. © Sauce Labs, Inc. © Sauce Labs, Inc. Common sources for tech debt Short term gain over long term investment We all cut corners occasionally. Mistakes People, process, product and technology leave a lot of room for mistakes. Finite resources Priorities and deadline pressure seemingly did not allow the time investment when code was shipped.
9. © Sauce Labs, Inc. © Sauce Labs, Inc. Web Stack Pylons powered web stack. First deployed back in 2008. Refactored into deprecation (Pyramid) circa November 2010 Tightly coupled, messy code base Tech debt at Sauce Labs: Web Stack Sauce Labs celebrated its 6th birthday in 2014
10. © Sauce Labs, Inc. Productivity impacts Monolithic architecture Potential for failures across the distributed app introduced brittleness and test flakes (CI). Development workflow Slowed down deploys due to lengthy continuous integration runs. Complexity Onboarding, development, deploy, troubleshooting and debugging was hard due to unbound complexity.
11. © Sauce Labs, Inc. What do we do? The big question.
12. © Sauce Labs, Inc. Pylons The full stack migration project Refactored into deprecation (Pyramid) circa November 2010 Tightly coupled, messy code base Django Active, growing community since 2005, more than MVC
13. © Sauce Labs, Inc. Before The full stack migration project After db sauce ondemand website + rest api saucelabs.com db sauce ondemand website saucelabs.com rest api REST Calls
14. © Sauce Labs, Inc. © Sauce Labs, Inc. What did we set out for? Never change a running system, right? Not quite! Agile development Maintain quick feature turnaround to continuously improve our product leveraging automation everywhere. Innovation and scale Serve more users while allowing contributions from a larger team. Zero downtime Leverage battle proven frameworks, toolchains and monitoring tools.
15. © Sauce Labs, Inc. Let’s talk about gravitational forces. Wait a minute...
16. © Sauce Labs, Inc. © Sauce Labs, Inc. Management buy-in. Gravitational forces
17. © Sauce Labs, Inc. © Sauce Labs, Inc. Learning curves. Gravitational forces
18. © Sauce Labs, Inc. © Sauce Labs, Inc. Focus. Gravitational forces
19. © Sauce Labs, Inc. © Sauce Labs, Inc. Creature of habit. Gravitational forces
20. © Sauce Labs, Inc.
21. © Sauce Labs, Inc. The Migration What? How?
22. © Sauce Labs, Inc. Embrace the hybrid stack: Pylons + Django /account /tests /tests/<job-id> /users/<username>/tests ... # anything else /*
23. © Sauce Labs, Inc. De-centralized user identity JSON Web Token Claim based authentication. ITEF Standard. Travels securely via https cookie (“jwt-login”). Identity detached from session Local sessions in Django and Pylons. Cross-stack, cross-domain. JWT everywhere Django or Pylons can initiate session as long as they posses shared secret. REST API server auth accepts token too. Source: http://jwt.io/ “JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).” - IETF
24. © Sauce Labs, Inc. JWT Authentication Middleware and Backend Validates Identity Verifies JWT using shared secret without DB query or session access. Identity detached from session Local sessions in Django and Pylons. Cross-stack, cross-domain. Authentication backend Verifies username and password against REST API on login. JwtUser + User Only creates persistent “shadow” User object for admins to leverage Django permissions. JWTAuthenticationMiddleware Django view
25. © Sauce Labs, Inc. import jwt import time from django.conf import settings from django.contrib.auth.models import User class JwtUser(User): managed = False ... Shadow users in Django db Sauce users Any user not identified as an admin by the REST API are represented as a transient JwtUser object. Sauce admins Users identified as Sauce admins are represented as User objects. Django local shadow users match Sauce REST users by ID. Permissions and auth Shadow user object allows Django to leverage its permission framework. Authentication works exclusively through the REST API.
26. © Sauce Labs, Inc. Pylons Shim Middleware Session sync Initiates session in Pylons app and propagates cookie back to user thru Django app. Flash messages Pings Pylons app for session stored flash messages thru private REST API endpoint. CSRF token sync Enables form cross posts. Keeps CSRF token in sync between Django and Pylons apps. PylonsShimMiddleware
27. © Sauce Labs, Inc. Look familiar? Role of data models and ORM Sauce REST as primary API Typical Django Project Django ORM Sauce Website REST API Client Django ORM db db Sauce OnDemand
28. © Sauce Labs, Inc. REST API Client to consume the Sauce from encore.core.utils import Kitchen def user_details(request): ... saucer = Kitchen(request.user).private.users. get( request.user.username, full=True) ... def change_access_key(request): ... Kitchen(request.user).users.change_access_key() ... def saml_settings(request): ... Kitchen(request.user).saml.update(...) ... Simple API - Kitchen Resource orientated based on JSON. Convenience methods for get and update operations. Error and Exception handling. Authentication and signing Picks up on auth context through request object. Private API calls get signed with shared secret. Public and private endpoints Public REST API as per docs. Private endpoints require elevated permissions and signed requests.
29. © Sauce Labs, Inc. Full Stack: Django at the foundation Development workflow SASS compilation, sprite sheets, minification, CDN upload. Fully automated using Grunt and Django’s manage.py. Dependency management Bower for frontend libs. WebPack for CommonJS module resolution in Javascript code (bundles). Content delivery network Website assets delivered through edge locations. Styleguide & App
30. © Sauce Labs, Inc. Why? Clear goal: Port UX which served production at the time. Proof dev workflow and stages are ready for primetime. Operational insights Made sure stack and operations are ready for drastic changes in UX / UI. A/B Testable (Experiments) Allow soft launch of new UX / UI. Leverage existing experiments framework. Objective: Port core user experience
31. © Sauce Labs, Inc. © Sauce Labs, Inc. Highway to the Danger Zone! Deployage well past midnight - Oct 1 2014
32. © Sauce Labs, Inc. Smarter testing Selenium test suite Re-used existing integration tests to verify migrated functionality. Staged continuous integration Isolated, fast website build. Green builds go into Sauce build for legacy integration testing. 5min builds and deploys. Heavy utilization of unit tests Django level unit tests as well as javascript unit tests. New functionality has integration smoke tests. Easy to execute with nose and karma.js on Sauce Cloud.
33. © Sauce Labs, Inc. Smarter testing Selenium test suite Re-used existing integration tests to verify migrated functionality. Staged continuous integration Isolated, fast website build. Green builds go into Sauce build for legacy integration testing. 5min builds and deploys. Heavy utilization of unit tests Django level unit tests as well as javascript unit tests. New functionality has integration smoke tests. Easy to execute with nose and karma.js on Sauce Cloud.
34. © Sauce Labs, Inc. Django unit tests Fake REST Context Mgmr Patches requests library to intercept HTTP calls. Fake data based on routes Jinja2 templates being used to return with faked JSON data. Verifies contract Validates HTTP method, status codes, auth tokens, HMAC signatures, params, and other elements of REST contract. ROUTES = ( ... ([GET], '/rest/v1/users/{username}', dict( private_api=False, response_template='users/details.jinja2' )), ... ) from encore.test.utils import fake_resto def test_dashboard_get(self): with fake_resto(): request = make_request( url_for('admin_dashboard_index'), is_loggedin=True, permissions=['admin.dashboard.index.view']) response = index(request) self.assertEqual(response.status_code, 200) self.assertTrue("<span>Dashboard</span>" in response)
35. © Sauce Labs, Inc. Biggest Wins Decoupled web stack. Code base is neatly broken out into its own repo. REST API sole interface between website and Sauce platform. Faster turnarounds. Cheap builds in continuous integration and fast website deploys allow rapid product iterations. Smarter test coverage. Extensive layers of test case at the unit level in their native runtimes. Integration tests leverage Sauce scale. More robust REST API. Website’s sole interface to the Sauce platform hardens public and private endpoints. Dogfooding. Real time monitoring. Monitoring tools such as Nagios, Kibana and Grafana provide real time insights to help identify bottlenecks and operational issues. Easier onboarding. Web stack is composed of battle proven frameworks / libraries such as Django. Greatly simplifies onboarding of new team members.
36. © Sauce Labs, Inc.
37. © Sauce Labs, Inc. © Sauce Labs, Inc. Thank you Questions? @sourishkrout https://saucelabs.com/careers careers@saucelabs.com Photos by Flickr (Creative Commons). Animated GIFs by giphy.com. full timers and interns