Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ISTA 2019 - Migrating data-intensive microservices from Python to Go

ISTA 2019 - Migrating data-intensive microservices from Python to Go

In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.

Nikolay Stoitsev

November 15, 2019
Tweet

More Decks by Nikolay Stoitsev

Other Decks in Technology

Transcript

  1. Migrating data-intensive
    microservices from Python to Go
    Nikolay Stoitsev
    Engineering Manager @ Uber

    View full-size slide

  2. Early years
    Dispatch API Storage

    View full-size slide

  3. Early years
    Dispatch API Storage
    Python
    Node.js

    View full-size slide

  4. Invoice
    Generation
    Service

    View full-size slide

  5. Background
    Legal document

    View full-size slide

  6. Background
    Legal document
    Vary by country

    View full-size slide

  7. Background
    Legal document
    Vary by country
    Vary by business line

    View full-size slide

  8. Background
    Legal document
    Vary by country
    Vary by business line
    Triggered after every trip or food
    delivery

    View full-size slide

  9. Sample architecture
    Money
    System
    Cassandra
    Kafka Preprocess Render
    Kafka
    Consumer
    Object Store

    View full-size slide

  10. More than 30 upstream systems
    Large Scale

    View full-size slide

  11. More than 30 upstream systems
    More than 100 TBs of data stored
    Large Scale

    View full-size slide

  12. More than 30 upstream systems
    More than 100 TBs of data stored
    Running on 400 containers
    in multiple DCs
    Large Scale

    View full-size slide

  13. More than 30 upstream systems
    More than 100 TBs of data stored
    Running on 400 containers
    in multiple DCs
    Running for 5 years
    Large Scale

    View full-size slide

  14. More than 30 upstream systems
    More than 100 TBs of data stored
    Running on 400 containers
    in multiple DCs
    Running for 5 years
    99.999% availability for last 6 months
    Large Scale

    View full-size slide

  15. More than 30 upstream systems
    More than 100 TBs of data stored
    Running on 400 containers
    in multiple DCs
    Running for 5 years
    99.999% availability for last 6 months
    Implemented in Python
    Large Scale

    View full-size slide

  16. Sample architecture
    Money
    System
    Cassandra
    Kafka Preprocess Render
    Kafka
    Consumer
    Object Store
    Web API
    Hive

    View full-size slide

  17. Building blocks
    http://flask.pocoo.org

    View full-size slide

  18. Flask Example

    View full-size slide

  19. MVCS
    Controller
    Mapper
    Service Entities
    External
    Services
    Database

    View full-size slide

  20. Building blocks
    https://uwsgi-docs.readthedocs.io/

    View full-size slide

  21. uWSGI
    uwsgi
    python
    python
    python

    View full-size slide

  22. Building blocks
    http://www.celeryproject.org/

    View full-size slide

  23. Celery
    celery-worker
    celery-worker
    celery-worker
    kafka
    consumer Redis

    View full-size slide

  24. “Use the right tool
    for the job”

    View full-size slide

  25. It hurts velocity at some
    point

    View full-size slide

  26. What we need for each language?
    Training / Best practices /
    Documentation / Experts

    View full-size slide

  27. What we need for each language?
    Training / Best practices /
    Documentation / Experts
    Project template /
    Bootstrapping

    View full-size slide

  28. What we need for each language?
    Training / Best practices /
    Documentation / Experts
    Project template /
    Bootstrapping
    Configuration

    View full-size slide

  29. What we need for each language?
    Training / Best practices /
    Documentation / Experts
    Project template /
    Bootstrapping
    Configuration
    Debuggers

    View full-size slide

  30. What we need for each language?
    Training / Best practices /
    Documentation / Experts
    Project template /
    Bootstrapping
    Configuration
    Debuggers
    Profilers

    View full-size slide

  31. What we need for each language?
    Training / Best practices /
    Documentation / Experts
    Project template /
    Bootstrapping
    Configuration
    Debuggers
    Profilers
    Building, Packaging,
    Deploying

    View full-size slide

  32. We picked Go and Java

    View full-size slide

  33. Broad applicability

    View full-size slide

  34. High performance

    View full-size slide

  35. Static typing

    View full-size slide

  36. Has momentum

    View full-size slide

  37. From Python to Go

    View full-size slide

  38. EAFP versus LBYL

    View full-size slide

  39. EAFP versus LBYL

    View full-size slide

  40. Dependency injection
    https://github.com/uber-go/fx

    View full-size slide

  41. Cadence instead of Celery
    https://github.com/uber/cadence

    View full-size slide

  42. Cadence
    Cadence
    DB queue
    Timers
    invoice
    service
    worker
    worker
    worker
    worker

    View full-size slide

  43. MVCS translates nicely

    View full-size slide

  44. How to migrate?
    Money
    System
    Invoice
    Generation
    Storage
    Python

    View full-size slide

  45. Option #1 - Big Bang Rewrite
    Money
    System
    Invoice
    Generation
    Storage
    Python
    Invoice
    Generation
    Go

    View full-size slide

  46. Option #1 - Big Bang Rewrite
    Money
    System
    Storage
    Invoice
    Generation
    Go

    View full-size slide

  47. No visibility on regressions

    View full-size slide

  48. No visibility on performance
    degradation

    View full-size slide

  49. No visibility on feature
    parity

    View full-size slide

  50. Option #2 - Do it iteratively

    View full-size slide

  51. Invoice
    Generation
    Storage
    Kafka

    View full-size slide

  52. Storage
    Kafka Preprocess Render

    View full-size slide

  53. Storage
    Kafka Preprocess Render
    Preprocess
    Go

    View full-size slide

  54. Storage
    Kafka Preprocess Render
    Preprocess
    Go
    Compare

    View full-size slide

  55. Storage
    Kafka Preprocess Render
    Preprocess
    Go
    Compare
    Toggle

    View full-size slide

  56. m3 DB
    https://www.m3db.io

    View full-size slide

  57. Tally - stats collection in Go
    https://github.com/uber-go/tally

    View full-size slide

  58. Tally - stats collection in Go

    View full-size slide

  59. Measure processing time
    p95, p99

    View full-size slide

  60. Storage
    Kafka Preprocess Render
    Preprocess
    Go
    Compare
    m3 Grafana

    View full-size slide

  61. Correctness?

    View full-size slide

  62. Storage
    Kafka Preprocess Render
    Preprocess
    Go
    Compare
    Kafka ELK

    View full-size slide

  63. Structured logging

    View full-size slide

  64. Structured logging

    View full-size slide

  65. Structured logging

    View full-size slide

  66. Structured logging

    View full-size slide

  67. Zap
    https://github.com/uber-go/zap

    View full-size slide

  68. Benefits of iterative approach
    Verify regressions
    Verify performance problems
    Verify feature parity

    View full-size slide

  69. Lessons learned

    View full-size slide

  70. Spend time to learn the new
    language

    View full-size slide

  71. Spend time to read code in
    the new language

    View full-size slide

  72. Do a rollout plan and stick
    to it

    View full-size slide

  73. Python can scale and is
    reliable

    View full-size slide

  74. Q&A
    Thank you!
    Nikolay Stoitsev, [email protected]

    View full-size slide