Evolving big microservice architectures

Evolving big microservice architectures

Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.

Eb44761e0fb3a5ec8e23ec28048dd7a5?s=128

Nikolay Stoitsev

November 02, 2019
Tweet

Transcript

  1. Evolving big microservice architectures Nikolay Stoitsev, Engineering Manager @ Uber

    Engineering
  2. None
  3. None
  4. Dispatch API

  5. A monolith? Code was consistently good Easy to reuse code

    Well tested Hard to deploy
  6. Pull Request Pull Request Pull Request Pull Request Pull Request

    ? ? ? ? ?
  7. Pull Request Pull Request Pull Request Pull Request Pull Request

    ? ? ? ? ? Unknown risk Unknown priority
  8. Low Risk High Risk High Risk Med Risk High Risk

    Build
  9. Low Risk High Risk High Risk Med Risk High Risk

    Build ?
  10. Low Risk High Risk High Risk Med Risk High Risk

    Build
  11. Low Risk / P0 High Risk / P1 High Risk

    / P0 Med Risk / P3 High Risk / P0 Build
  12. Low Risk / P0 High Risk / P1 High Risk

    / P0 Med Risk / P3 High Risk / P0 Build CI / CD
  13. Gradual rollout is important

  14. Lesson learned: A monolith can scale a lot, but...

  15. Dispatch API

  16. Dispatch API Users Trips Receipts Payments

  17. Dispatch API Users Trips Receipts Payments

  18. Dispatch API Users Trips Receipts Payments

  19. Dispatch API Users Trips Receipts Payments

  20. None
  21. 4000+ microservices 1K+ avg deploys a day

  22. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka Sample architecture
  23. We want to introduce Uber Elevate

  24. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  25. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 Apache Kafka Apache Kafka Apache Kafka
  26. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Apache Kafka Apache Kafka Apache Kafka
  27. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Apache Kafka Apache Kafka Apache Kafka
  28. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Apache Kafka Apache Kafka Apache Kafka
  29. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Task #4 Apache Kafka Apache Kafka Apache Kafka
  30. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 Apache Kafka Apache Kafka Apache Kafka
  31. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Apache Kafka Apache Kafka Apache Kafka
  32. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Task #6 Apache Kafka Apache Kafka Apache Kafka
  33. How do we evolve big microservice architectures?

  34. You can’t change what you can’t test

  35. Staging environment?

  36. Integration tests?

  37. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Change Apache Kafka Apache Kafka Apache Kafka
  38. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Change MOCK MOCK MOCK MOCK Apache Kafka Apache Kafka Apache Kafka
  39. Multitenancy

  40. Tenancy - set of data that is isolated from the

    other sets of data
  41. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Change Apache Kafka Apache Kafka Apache Kafka
  42. Testing tenancy vs. Production tenancy

  43. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Change Test Data Test Config Test Config Test Data Apache Kafka Apache Kafka Apache Kafka
  44. Audit tenancy

  45. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Audit tenancy traffic Check Test Config Test Config Test Data Apache Kafka Apache Kafka Apache Kafka Check Check Check
  46. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Apache Kafka Apache Kafka Apache Kafka
  47. Platforms

  48. Services are too concrete so it’s hard to support new

    use cases
  49. Services need to support abstraction

  50. AWS is a good example

  51. Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  52. Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  53. Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  54. Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  55. Hard shift in mentality in the way you design your

    services
  56. Can you sell it outside of the company?

  57. Tax Calc Payments Pricing

  58. Tax Calc Payments Pricing 3.14159

  59. Tax Calc Payments Pricing 3.14159 “3.14”

  60. Tax Calc Payments Pricing 3.14159 “3.14” 314159

  61. Common data model

  62. None
  63. None
  64. Data extensions = schema + payload

  65. How to do it? Schema service Repo Library ...

  66. Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments

    Services Receipts Apache Kafka Apache Kafka Apache Kafka
  67. … but there is still business specific logic.

  68. Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments

    Services Receipts Where to put it? Business logic Apache Kafka Apache Kafka Apache Kafka
  69. No common understanding of what should go where

  70. How do we call code with business logic, presentation logic,

    and SQL queries all over the place?
  71. We fix spaghetti code with separation of concerns

  72. Presentation Logic Data

  73. Layered architecture

  74. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend
  75. Why?

  76. Common language

  77. Clear separation of concerns

  78. Consistent patterns

  79. API and functionality discoverability

  80. Engineering velocity

  81. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Generic Specific
  82. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Breaks everything Breaks specific thing
  83. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Presentation LoB Biz Infra Core
  84. https://philcalcado.com/2018/09/24/services_layers.html Users Vehicles Order Platform Pricing Tax Calc Payments Receipts

    Rides Elevate Web Services Mobile Backend Clay Rocks
  85. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Presentation
  86. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend LoB
  87. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Biz Infra
  88. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend Core
  89. Clear API

  90. protobuf Apache Thrift GraphQL

  91. End result: Layered multitenant architecture built with abstract platforms and

    a common data model
  92. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Web Services Mobile Backend
  93. Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides

    Elevate Web Services Mobile Backend
  94. Q&A Feedback:

  95. Proprietary © 2019 Uber Technologies, Inc. All rights reserved. No

    part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.