Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving big microservice architectures

Evolving big microservice architectures

Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.

Nikolay Stoitsev

November 02, 2019
Tweet

More Decks by Nikolay Stoitsev

Other Decks in Technology

Transcript

  1. Evolving big
    microservice
    architectures
    Nikolay Stoitsev, Engineering Manager @ Uber Engineering

    View full-size slide

  2. Dispatch API

    View full-size slide

  3. A monolith?
    Code was
    consistently good
    Easy to reuse code
    Well tested
    Hard to deploy

    View full-size slide

  4. Pull Request
    Pull Request
    Pull Request
    Pull Request
    Pull Request
    ?
    ?
    ?
    ?
    ?

    View full-size slide

  5. Pull Request
    Pull Request
    Pull Request
    Pull Request
    Pull Request
    ?
    ?
    ?
    ?
    ?
    Unknown risk
    Unknown priority

    View full-size slide

  6. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build

    View full-size slide

  7. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build
    ?

    View full-size slide

  8. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build

    View full-size slide

  9. Low Risk / P0
    High Risk / P1
    High Risk / P0
    Med Risk / P3
    High Risk / P0
    Build

    View full-size slide

  10. Low Risk / P0
    High Risk / P1
    High Risk / P0
    Med Risk / P3
    High Risk / P0
    Build CI / CD

    View full-size slide

  11. Gradual rollout is important

    View full-size slide

  12. Lesson learned:
    A monolith can scale a lot, but...

    View full-size slide

  13. Dispatch API

    View full-size slide

  14. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View full-size slide

  15. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View full-size slide

  16. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View full-size slide

  17. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View full-size slide

  18. 4000+ microservices
    1K+ avg deploys a day

    View full-size slide

  19. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka
    Sample architecture

    View full-size slide

  20. We want to introduce Uber Elevate

    View full-size slide

  21. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  22. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  23. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  24. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  25. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  26. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  27. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  28. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  29. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Task #6
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  30. How do we evolve
    big microservice
    architectures?

    View full-size slide

  31. You can’t change what you can’t test

    View full-size slide

  32. Staging environment?

    View full-size slide

  33. Integration tests?

    View full-size slide

  34. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  35. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    MOCK
    MOCK
    MOCK MOCK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  36. Multitenancy

    View full-size slide

  37. Tenancy - set of data that is isolated
    from the other sets of data

    View full-size slide

  38. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  39. Testing tenancy vs. Production tenancy

    View full-size slide

  40. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Test
    Data
    Test
    Config
    Test
    Config
    Test
    Data
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  41. Audit tenancy

    View full-size slide

  42. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Audit
    tenancy
    traffic
    Check
    Test
    Config
    Test
    Config
    Test
    Data
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka
    Check Check
    Check

    View full-size slide

  43. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  44. Services are too concrete so it’s hard to
    support new use cases

    View full-size slide

  45. Services need to support abstraction

    View full-size slide

  46. AWS is a good example

    View full-size slide

  47. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  48. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  49. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  50. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  51. Hard shift in mentality in the way you
    design your services

    View full-size slide

  52. Can you sell it outside of the company?

    View full-size slide

  53. Tax Calc Payments Pricing

    View full-size slide

  54. Tax Calc Payments Pricing
    3.14159

    View full-size slide

  55. Tax Calc Payments Pricing
    3.14159 “3.14”

    View full-size slide

  56. Tax Calc Payments Pricing
    3.14159 “3.14” 314159

    View full-size slide

  57. Common data
    model

    View full-size slide

  58. Data extensions = schema + payload

    View full-size slide

  59. How to do it?
    Schema service
    Repo
    Library
    ...

    View full-size slide

  60. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  61. … but there is still
    business specific logic.

    View full-size slide

  62. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Where to put it?
    Business
    logic
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View full-size slide

  63. No common understanding
    of what should go where

    View full-size slide

  64. How do we call code with business
    logic, presentation logic, and SQL
    queries all over the place?

    View full-size slide

  65. We fix spaghetti code
    with separation of concerns

    View full-size slide

  66. Presentation
    Logic
    Data

    View full-size slide

  67. Layered
    architecture

    View full-size slide

  68. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend

    View full-size slide

  69. Common language

    View full-size slide

  70. Clear separation of concerns

    View full-size slide

  71. Consistent patterns

    View full-size slide

  72. API and functionality discoverability

    View full-size slide

  73. Engineering velocity

    View full-size slide

  74. Users Vehicles
    Order
    Platform
    Pricing
    Tax
    Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Generic
    Specific

    View full-size slide

  75. Users Vehicles
    Order
    Platform
    Pricing
    Tax
    Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Breaks
    everything
    Breaks
    specific thing

    View full-size slide

  76. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Presentation
    LoB
    Biz Infra
    Core

    View full-size slide

  77. https://philcalcado.com/2018/09/24/services_layers.html
    Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Clay
    Rocks

    View full-size slide

  78. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Presentation

    View full-size slide

  79. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    LoB

    View full-size slide

  80. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Biz Infra

    View full-size slide

  81. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Core

    View full-size slide

  82. protobuf
    Apache Thrift
    GraphQL

    View full-size slide

  83. End result: Layered multitenant
    architecture built with abstract
    platforms and a common data model

    View full-size slide

  84. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides
    Web
    Services
    Mobile
    Backend

    View full-size slide

  85. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend

    View full-size slide

  86. Q&A
    Feedback:

    View full-size slide

  87. Proprietary © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any
    means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in
    writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is
    privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the
    information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any
    way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for
    consultations with authorized personnel of Uber.

    View full-size slide