Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving big microservice architectures

Evolving big microservice architectures

Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.

Nikolay Stoitsev

November 02, 2019
Tweet

More Decks by Nikolay Stoitsev

Other Decks in Technology

Transcript

  1. Evolving big
    microservice
    architectures
    Nikolay Stoitsev, Engineering Manager @ Uber Engineering

    View Slide

  2. View Slide

  3. View Slide

  4. Dispatch API

    View Slide

  5. A monolith?
    Code was
    consistently good
    Easy to reuse code
    Well tested
    Hard to deploy

    View Slide

  6. Pull Request
    Pull Request
    Pull Request
    Pull Request
    Pull Request
    ?
    ?
    ?
    ?
    ?

    View Slide

  7. Pull Request
    Pull Request
    Pull Request
    Pull Request
    Pull Request
    ?
    ?
    ?
    ?
    ?
    Unknown risk
    Unknown priority

    View Slide

  8. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build

    View Slide

  9. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build
    ?

    View Slide

  10. Low Risk
    High Risk
    High Risk
    Med Risk
    High Risk
    Build

    View Slide

  11. Low Risk / P0
    High Risk / P1
    High Risk / P0
    Med Risk / P3
    High Risk / P0
    Build

    View Slide

  12. Low Risk / P0
    High Risk / P1
    High Risk / P0
    Med Risk / P3
    High Risk / P0
    Build CI / CD

    View Slide

  13. Gradual rollout is important

    View Slide

  14. Lesson learned:
    A monolith can scale a lot, but...

    View Slide

  15. Dispatch API

    View Slide

  16. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View Slide

  17. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View Slide

  18. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View Slide

  19. Dispatch API
    Users
    Trips
    Receipts
    Payments

    View Slide

  20. View Slide

  21. 4000+ microservices
    1K+ avg deploys a day

    View Slide

  22. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka
    Sample architecture

    View Slide

  23. We want to introduce Uber Elevate

    View Slide

  24. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  25. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  26. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  27. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  28. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  29. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  30. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  31. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  32. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Task #6
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  33. How do we evolve
    big microservice
    architectures?

    View Slide

  34. You can’t change what you can’t test

    View Slide

  35. Staging environment?

    View Slide

  36. Integration tests?

    View Slide

  37. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  38. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    MOCK
    MOCK
    MOCK MOCK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  39. Multitenancy

    View Slide

  40. Tenancy - set of data that is isolated
    from the other sets of data

    View Slide

  41. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  42. Testing tenancy vs. Production tenancy

    View Slide

  43. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Change
    Test
    Data
    Test
    Config
    Test
    Config
    Test
    Data
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  44. Audit tenancy

    View Slide

  45. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Audit
    tenancy
    traffic
    Check
    Test
    Config
    Test
    Config
    Test
    Data
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka
    Check Check
    Check

    View Slide

  46. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Task #1
    OK
    Task #2
    Task #3 Task #4
    Task #5 OK
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  47. Platforms

    View Slide

  48. Services are too concrete so it’s hard to
    support new use cases

    View Slide

  49. Services need to support abstraction

    View Slide

  50. AWS is a good example

    View Slide

  51. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Trips
    Services
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  52. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  53. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  54. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  55. Hard shift in mentality in the way you
    design your services

    View Slide

  56. Can you sell it outside of the company?

    View Slide

  57. Tax Calc Payments Pricing

    View Slide

  58. Tax Calc Payments Pricing
    3.14159

    View Slide

  59. Tax Calc Payments Pricing
    3.14159 “3.14”

    View Slide

  60. Tax Calc Payments Pricing
    3.14159 “3.14” 314159

    View Slide

  61. Common data
    model

    View Slide

  62. View Slide

  63. View Slide

  64. Data extensions = schema + payload

    View Slide

  65. How to do it?
    Schema service
    Repo
    Library
    ...

    View Slide

  66. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  67. … but there is still
    business specific logic.

    View Slide

  68. Dispatch
    Pricing
    Services
    Tax
    Calc
    Vehicles
    Orders
    Platform
    Users
    Payments
    Services
    Receipts
    Where to put it?
    Business
    logic
    Apache
    Kafka
    Apache
    Kafka
    Apache
    Kafka

    View Slide

  69. No common understanding
    of what should go where

    View Slide

  70. How do we call code with business
    logic, presentation logic, and SQL
    queries all over the place?

    View Slide

  71. We fix spaghetti code
    with separation of concerns

    View Slide

  72. Presentation
    Logic
    Data

    View Slide

  73. Layered
    architecture

    View Slide

  74. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend

    View Slide

  75. Why?

    View Slide

  76. Common language

    View Slide

  77. Clear separation of concerns

    View Slide

  78. Consistent patterns

    View Slide

  79. API and functionality discoverability

    View Slide

  80. Engineering velocity

    View Slide

  81. Users Vehicles
    Order
    Platform
    Pricing
    Tax
    Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Generic
    Specific

    View Slide

  82. Users Vehicles
    Order
    Platform
    Pricing
    Tax
    Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Breaks
    everything
    Breaks
    specific thing

    View Slide

  83. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Presentation
    LoB
    Biz Infra
    Core

    View Slide

  84. https://philcalcado.com/2018/09/24/services_layers.html
    Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Clay
    Rocks

    View Slide

  85. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Presentation

    View Slide

  86. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    LoB

    View Slide

  87. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Biz Infra

    View Slide

  88. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend
    Core

    View Slide

  89. Clear API

    View Slide

  90. protobuf
    Apache Thrift
    GraphQL

    View Slide

  91. End result: Layered multitenant
    architecture built with abstract
    platforms and a common data model

    View Slide

  92. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides
    Web
    Services
    Mobile
    Backend

    View Slide

  93. Users Vehicles
    Order
    Platform
    Pricing Tax Calc Payments Receipts
    Rides Elevate
    Web
    Services
    Mobile
    Backend

    View Slide

  94. Q&A
    Feedback:

    View Slide

  95. Proprietary © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any
    means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in
    writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is
    privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the
    information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any
    way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for
    consultations with authorized personnel of Uber.

    View Slide