• Ruby 1.9.3, Rails 3.2 • No tests… at all… for anything • Custom ORM into DynamoDB • Metaprogramming everywhere • No logical separation of concerns (authorization, persistence, domain logic) • No metrics or visibility into performance API ISSUES PRE MIGRATION
• Everything is a string … even nulls. • Foreign keys stored as a JSON encoded list of strings on the model. No atomic updates = lost data. • DynamoDB cost $$$ to support our workload • No ability to paginate anything = up to 45s response times and large payloads (> 1mb of JSON) DATABASE ISSUES PRE-MIGRATION
• Easy-ish ramp up for our existing Ruby / Python developers • Highly concurrent: use resources much more efficiently • Building on a mature VM (BEAM) and established language (Erlang & OTP) • Language attributes promote explicitness: immutability, pattern matching, multi function heads. WHY ELIXIR?
API (Ruby on Rails) Websocket connection Real-time Service (Node.JS) Support Tool (Node.js) Push Notifications SES Email Digest Service (Node.js) Client (iOS / Web / Adobe) DynamoDB LEGACY ARCHITECTURE
API (Ruby on Rails) Websocket connection Real-time Service (Node.JS) Support Tool (Node.js) Push Notifications SES Email Digest Service (Node.js) Client (iOS / Web / Adobe) DynamoDB LEGACY ARCHITECTURE: WHAT WE REPLACED
• Elixir powered API, notifications system, real-time service, and support tool • Migrated all of our data to Postgres from DynamoDB • Dockerized all of the above and rebuilt our tooling and deploy process from the ground up WHAT WE SHIPPED
V2 API (Phoenix) Websocket connection Real-time Service (Elixir / Phoenix) Support Tool (Phoenix) Push Notifications SES Email Service Client (iOS / Web / Adobe) Postgres UPDATED ARCHITECTURE Munger API (Phoenix) Core Business Logic Memcached Umbrella App
• ~40 EC2 Instances to ~5 (running on ECS) • API 95th Percentile: ~30ms @ ~120rps • Database cost 91% reduced • Full visibility into all parts of the system (via statsd & datadog) • Modular, documented, maintainable codebase WHAT WE SHIPPED: RESULTS
• Consumes new API, spits out old schemas and maintains legacy contract (by stringifying everything) • Allowed us to ship our new stack sooner with fewer implications for our different clients • Complexity is high, but designed to be thrown away (~6 months time) THE INTERMEDIARY API
• Apps built and deployed as separate Docker containers in CircleCI via Distillery • Each build & deploy takes ~5 minutes (run in parallel) • Blue / green deploys via ECS • All auto-scaled via CPU / Memory threshold alarms UMBRELLA APP STRUCTURE
• Core houses all of our business logic, services, Ecto schemas, access policies, deferred logic, and more • Broken into two contexts: Accounts & Projects • API & Support Tool use the Core to fetch data and execute requests – they are effectively dumb HTTP wrappers UMBRELLA APP STRUCTURE: CORE
• All changes through our system broadcasted through a single, local event bus • Provides a powerful hook to build deferred functionality on-top of (like notifications, analytics tracking etc) • Implemented using GenStage and Protocols EVENT SYSTEM: WHAT IS IT?
• Moved all our records from DynamoDB into Postgres • Migrated through Flow tasks that streamed data from our tables and converted into the appropriate Postgres schemas • Largest table size was ~9m records, each record > 100kb (lots of JSON) MOVING MILLIONS OF RECORDS
MOVING MILLIONS OF RECORDS: HOW IT WORKS • Each table migration job runs in its own isolated Docker container using the ECS run task • We monitored errors in our jobs and constantly refined and tweaked the parallelism for each job • Ran weekly migrations and manually checked the migrated data in our QA environment
• Bugs and replicating old bugs • Team ramp up: 3 new developers learning Elixir trying to ship a thing is hard. Protip: establish patterns. • Understanding the performance characteristics of a new system and new database. • Estimation of complexity: went 6 weeks over our planned delivery date. CHALLENGES DURING THE MIGRATION
cjbell88
ramimac
oracle4engineer
senkin13
doradora09
ritou
hk_shino
th0x0472
tomzoh
nktamago
k_koheyi
nagiss
saboyutaka
paulrobertlloyd
lara
morganepeng
gr2m
malarkey
3n
productmarketing
jrom
jlugia
myddelton
mclloyd