ISTA 2019 - Migrating data-intensive microservices from Python to Go

In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.

Transcript

2. Migrating data-intensive microservices from Python to Go Nikolay Stoitsev Engineering Manager @ Uber
3. Early years Dispatch API Storage
4. Early years Dispatch API Storage Python Node.js
5. Invoice Generation Service
6. Background Legal document
7. Background Legal document Vary by country
8. Background Legal document Vary by country Vary by business line
9. Background Legal document Vary by country Vary by business line Triggered after every trip or food delivery
10. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store
11. More than 30 upstream systems Large Scale
12. More than 30 upstream systems More than 100 TBs of data stored Large Scale
13. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Large Scale
14. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years Large Scale
15. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Large Scale
16. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Implemented in Python Large Scale
17. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store Web API Hive
18. Building blocks http://flask.pocoo.org
19. Flask Example
20. Flask Usage
21. MVCS
22. MVCS Controller Mapper Service Entities External Services Database
23. Building blocks https://uwsgi-docs.readthedocs.io/
24. uWSGI uwsgi python python python
25. Building blocks http://www.celeryproject.org/
26. Celery celery-worker celery-worker celery-worker kafka consumer Redis
27. “Use the right tool for the job”
28. It hurts velocity at some point
29. What we need for each language? Training / Best practices / Documentation / Experts
30. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping
31. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration
32. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers
33. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers
34. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers Building, Packaging, Deploying
35. We picked Go and Java
36. Why Go?
37. Broad applicability
38. High performance
39. Static typing
40. Has momentum
41. From Python to Go
42. EAFP versus LBYL
43. EAFP versus LBYL
44. Dependency injection https://github.com/uber-go/fx
45. Cadence instead of Celery https://github.com/uber/cadence
46. Cadence Cadence DB queue Timers invoice service worker worker worker worker
47. MVCS translates nicely
48. How to migrate? Money System Invoice Generation Storage Python
49. Option #1 - Big Bang Rewrite Money System Invoice Generation Storage Python Invoice Generation Go
50. Option #1 - Big Bang Rewrite Money System Storage Invoice Generation Go
51. No visibility on regressions
52. No visibility on performance degradation
53. No visibility on feature parity
54. Option #2 - Do it iteratively
55. Invoice Generation Storage Kafka
56. Storage Kafka Preprocess Render
57. Storage Kafka Preprocess Render Preprocess Go
58. Storage Kafka Preprocess Render Preprocess Go Compare
59. Storage Kafka Preprocess Render Preprocess Go Compare Toggle
60. Volume?
61. m3 DB https://www.m3db.io
62. Tally - stats collection in Go https://github.com/uber-go/tally
63. Tally - stats collection in Go
64. Measure processing time p95, p99
65. Storage Kafka Preprocess Render Preprocess Go Compare m3 Grafana
66. Correctness?
67. Storage Kafka Preprocess Render Preprocess Go Compare Kafka ELK
69. Structured logging
70. Structured logging
71. Structured logging
72. Structured logging
73. Zap https://github.com/uber-go/zap
74. Zap
75. ELK
76. Benefits of iterative approach Verify regressions Verify performance problems Verify feature parity
77. Lessons learned
78. Spend time to learn the new language
79. Spend time to read code in the new language
80. Do a rollout plan and stick to it
81. Python can scale and is reliable
82. Q&A Thank you! Nikolay Stoitsev, stoitsev@uber.com