2016 - Trisha Kothari - Data in a dynamic system: Strategies for backwards compatibility

Description
There are several unanswered questions in deploying huge schema or logic changes: How do you modify systems with zero downtime or service interruption? How do you optimize online data migrations to allow for fallbacks? Any changes in schema or code in dynamic systems may cause existing users to experience downtime. The talk focuses on strategies to ensure backwards compatibility and prevent breaking data integrity.
​
Abstract
In an ideal scenario, feature development is easy. Just replace the old code with new code, and you’re done. This is, in fact, true for a system in state of inertia. However, in a dynamic system, with constantly moving pieces of business logic, this presents a hard problem. There are several unanswered questions while deploying huge schema or logic changes: How do you make code and schema changes with zero downtime or service interruption? How do you optimize online migrations of data to allow for fallbacks? Any changes in schema or code in dynamic systems may cause existing users to experience downtime. The talk focuses on strategies to ensure backwards compatibility and prevent breaking data integrity.

Bio
Trisha works as a Software Engineer at Affirm, a take on modern banking started by Max Levchin. At Affirm, Trisha has worked on several projects including the creation of the underlying financial system, architecture of systems for underwriting data processing, and several other product features. She graduated from the University of Pennsylvania studying Computer Science.

Transcript

1. Data in a dynamic system Strategies for backwards compatibility
9. Breaking user experience is not a new problem
10. Who am I?
15. What is a loan? Movement of money in a ledger Double entry accounting system id balance amount Mate id 1 cash 100 2 2 principal -100 1
16. Bank v1 Bank v2
18. Why is data hard in a dynamic system?
20. Dynamic systems • Service level changes • Changes to data at rest DATA INTEGRITY IS OF PARAMOUNT IMPORTANCE!
21. Deployment strategies must ensure backwards compatibility
22. Changes to service level code • Conditionals ◦ if <Condition1>: Treatment1() else: Treatment2() ◦ Messy code :( • API versioning • Deploying new dependencies first ◦ Optional arguments ◦ Make sure results can be consumed by the caller
23. Why is dealing with data at rest hard? Data is big! Data is dumb! How do you get backwards compatibility?
24. Data versioning
26. Write-both migrations Writer Old data Reader
27. Write-both migrations Writer Old data Reader New data
28. Write-both migrations Writer Old data Reader New data Migration!! #hearandnow @trisha_kothari
29. Write-both migrations Writer New data Reader
30. Complex Data migrations
33. luigi Open sourced in late 2012 Awesome for batch jobs Not for replacing Hive or Pig Spotify, Foursquare, Stripe, Affirm, hotels.com, etc Why Luigi for data migration?
37. Quick note on schema migrations
38. Alembic Database migration tool for SQLAlchemy Autogenerate: • Easy! • Gotcha: Renaming a column ⇒ Removal and addition of new column • Another gotcha: “Multiple heads not supported”
41. Paying it forward: lessons learned
46. Evolution of database normalizations
47. Thank you! @trishak42 trisha@affirm.com