ISTA 2019 - Migrating data-intensive microservices from Python to Go

ISTA 2019 - Migrating data-intensive microservices from Python to Go

In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.

Eb44761e0fb3a5ec8e23ec28048dd7a5?s=128

Nikolay Stoitsev

November 15, 2019
Tweet

Transcript

  1. None
  2. Migrating data-intensive microservices from Python to Go Nikolay Stoitsev Engineering

    Manager @ Uber
  3. Early years Dispatch API Storage

  4. Early years Dispatch API Storage Python Node.js

  5. Invoice Generation Service

  6. Background Legal document

  7. Background Legal document Vary by country

  8. Background Legal document Vary by country Vary by business line

  9. Background Legal document Vary by country Vary by business line

    Triggered after every trip or food delivery
  10. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer

    Object Store
  11. More than 30 upstream systems Large Scale

  12. More than 30 upstream systems More than 100 TBs of

    data stored Large Scale
  13. More than 30 upstream systems More than 100 TBs of

    data stored Running on 400 containers in multiple DCs Large Scale
  14. More than 30 upstream systems More than 100 TBs of

    data stored Running on 400 containers in multiple DCs Running for 5 years Large Scale
  15. More than 30 upstream systems More than 100 TBs of

    data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Large Scale
  16. More than 30 upstream systems More than 100 TBs of

    data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Implemented in Python Large Scale
  17. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer

    Object Store Web API Hive
  18. Building blocks http://flask.pocoo.org

  19. Flask Example

  20. Flask Usage

  21. MVCS

  22. MVCS Controller Mapper Service Entities External Services Database

  23. Building blocks https://uwsgi-docs.readthedocs.io/

  24. uWSGI uwsgi python python python

  25. Building blocks http://www.celeryproject.org/

  26. Celery celery-worker celery-worker celery-worker kafka consumer Redis

  27. “Use the right tool for the job”

  28. It hurts velocity at some point

  29. What we need for each language? Training / Best practices

    / Documentation / Experts
  30. What we need for each language? Training / Best practices

    / Documentation / Experts Project template / Bootstrapping
  31. What we need for each language? Training / Best practices

    / Documentation / Experts Project template / Bootstrapping Configuration
  32. What we need for each language? Training / Best practices

    / Documentation / Experts Project template / Bootstrapping Configuration Debuggers
  33. What we need for each language? Training / Best practices

    / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers
  34. What we need for each language? Training / Best practices

    / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers Building, Packaging, Deploying
  35. We picked Go and Java

  36. Why Go?

  37. Broad applicability

  38. High performance

  39. Static typing

  40. Has momentum

  41. From Python to Go

  42. EAFP versus LBYL

  43. EAFP versus LBYL

  44. Dependency injection https://github.com/uber-go/fx

  45. Cadence instead of Celery https://github.com/uber/cadence

  46. Cadence Cadence DB queue Timers invoice service worker worker worker

    worker
  47. MVCS translates nicely

  48. How to migrate? Money System Invoice Generation Storage Python

  49. Option #1 - Big Bang Rewrite Money System Invoice Generation

    Storage Python Invoice Generation Go
  50. Option #1 - Big Bang Rewrite Money System Storage Invoice

    Generation Go
  51. No visibility on regressions

  52. No visibility on performance degradation

  53. No visibility on feature parity

  54. Option #2 - Do it iteratively

  55. Invoice Generation Storage Kafka

  56. Storage Kafka Preprocess Render

  57. Storage Kafka Preprocess Render Preprocess Go

  58. Storage Kafka Preprocess Render Preprocess Go Compare

  59. Storage Kafka Preprocess Render Preprocess Go Compare Toggle

  60. Volume?

  61. m3 DB https://www.m3db.io

  62. Tally - stats collection in Go https://github.com/uber-go/tally

  63. Tally - stats collection in Go

  64. Measure processing time p95, p99

  65. Storage Kafka Preprocess Render Preprocess Go Compare m3 Grafana

  66. Correctness?

  67. Storage Kafka Preprocess Render Preprocess Go Compare Kafka ELK

  68. None
  69. Structured logging

  70. Structured logging

  71. Structured logging

  72. Structured logging

  73. Zap https://github.com/uber-go/zap

  74. Zap

  75. ELK

  76. Benefits of iterative approach Verify regressions Verify performance problems Verify

    feature parity
  77. Lessons learned

  78. Spend time to learn the new language

  79. Spend time to read code in the new language

  80. Do a rollout plan and stick to it

  81. Python can scale and is reliable

  82. Q&A Thank you! Nikolay Stoitsev, stoitsev@uber.com

  83. None