DB Migrations equal Pain

Transcript

1. DB Migrations = Pain
2. Context • Look is an application for live video streaming • Backend, iOS and Android client, Admin page, frontend for customers • Good management • Good architecture
3. Context • 3 environments: develop, qa, production (and local) • 3 core services: ◦ web (aka api) ◦ rtmp (video streaming) ◦ cent (realtime messaging)
4. Context • There are 2 backend developers • We think about code quality: ◦ very strict linter ◦ tests: unit and behave ◦ deploy in 1 command
5. Story • Deployment after 3 monthes of development • DB redesign: changed one of the core models to fit business logic ◦ Schema migration ◦ Data migration • Statistics on the admin page • Successfully deployed to dev and qa
6. Story • Data migrations was running during 40 minutes: ◦ I was ready for it • Production was down during 5 hours ◦ Kernel Panic! • I deployed the previous version and restore DB from snapshot – lost last 3 hours of data
7. Plan Analyze Fix Learn the lesson
8. What was the symptoms? • Django was not responding to request at all • Memory usage was fine • CPU was fine • Network was fine • Actually, Django was responding with HUGE latency ◦ the best case was 5 minutes, to the simplest request!
9. How did we investigate? • Find bottlenecks: ◦ analyze latencies locally – django-silk is the best • Fix them one by one • Test the fixes on the develop environment
10. How did we fix it? • Speed up data migrations: 40 minutes → 7 minutes ◦ select_related • Move all long running tasks to celery tasks • To prevent race between celery and django we run them on separate instances
11. How did we fix it? • Simplify admin page ◦ Calculate metrics in periodic celery task ▪ each 10 minutes, with timeout 1 hour ◦ Keep in DB ◦ Join with the metric table
12. What do we need to do? Zero downtime deployment aka Continuous Deployment
13. Continuous Deployment Blue Green Deployment
14. Our way • Use 2 web instances: ◦ Current ◦ Staging • Use 2 DB instances: ◦ Current ◦ Staging
15. Our way • Deployment steps: ◦ Deploy to staging ◦ Run migrations ◦ Wait ◦ Swap the DNS
16. The fixes deployment • Production was down during 4 hour ◦ Panic! • The same symptoms!
17. The guess • Look at whole stack: ◦ DB flood the disk space ◦ The free disk space metric has reverse sawtooth form • Super hot fix: turn off metric task ◦ The free disk space metric have the same period as the periodic task for calculating metrics
18. Investigation • Use the production DB clone • Run the raw query that collects metrics ◦ It was running 1 hour! • This is the reason!
19. How did we fix it? • The raw query looks like: − SELECT DISTINCT − 8 LEFT OUTER JOINs − 5 COUNTs − 3 CASEs − GROUP BY user.id • Use EXPLAIN
20. How did we fix it? • We were not trying to use the raw query in django ◦ There is no reasons to do so • Attempts: ◦ Remove metrics that requires CASEs ◦ Reduce amount of COUNTs and JOINs ◦ Remove DISTINCT – Fetch row by row ◦ Use one query for each metric
21. How did we fix it? • The fix is: ◦ Use one query for each metric ▪ The best performance in the production case
22. Did it help? Yes
23. The lesson • Good management and good architecture are matter • Deploy more frequently • Do not use data migrations as is – Use commands • Django admin is not efficient for aggregation queries • Analyze and synthesize are matter
24. A proof • I have refactored another core model: ◦ A schema migration ◦ A command for data migration • I have deployed it without downtime • Look production environment is still alive
25. Summary • Analyze • Fix • Learn the lesson
26. References • https://crystalnix.com/works/look/ • http://martinfowler.com/bliki/BlueGreenDeployment .html • https://gist.github.com/EvgeneOskin/99880b7b7e0 cd2d0115f87b7eeb5ae57