MTC2018 - Mercari API: from Monolithic to Microservices

92cdcff298e89e2fcd2fb705155c2d4b?s=47 mercari
October 04, 2018

MTC2018 - Mercari API: from Monolithic to Microservices

Speaker: Tonghui Li

Mercari has evolved tremendously in the last 5 years, and so is our code base. As of 2018, the backend API is a monolithic repository of half-million lines of code, serving all the functions of the Mercari web and mobile applications. It is becoming a bottle neck of our growth, in terms of both the system and the organization. This talk will focus on how we migrate the monolithic backend API to a microservice architecture, the methodology we choose and the lessons we learned.

92cdcff298e89e2fcd2fb705155c2d4b?s=128

mercari

October 04, 2018
Tweet

Transcript

  1. Mercari API: from Monolithic to Microservices Tonghui Li Tech Lead

    (Backend)
  2. Tech Lead Backend Tonghui Li

  3. Mercari API: from Monolithic to Microservices

  4. Background

  5. Mercari API

  6. Mercari API iOS Android web

  7. Mercari API iOS Android web

  8. Mercari API iOS Android web admin tools

  9. Mercari API iOS Android web admin tools

  10. • 500 KLOC • 100+ contributors • 100 PR /

    week Mercari API iOS Android web admin tools
  11. • 500 KLOC • 100+ contributors • 100 PR /

    week • And growing! Mercari API iOS Android web admin tools
  12. How to migrate to microservices

  13. How to migrate to microservices • Implement new features in

    microservices • Break down existing API into multiple services
  14. How to implement new features

  15. client Mercari API

  16. client Service X Service Y Mercari API

  17. client Service X Service Y Mercari API

  18. API Gateway Service X Service Y Mercari API client

  19. API Gateway Service X Service Y Mercari API client SSL

    termination DDos protection Authn and Authz Traffic routing Logging and tracing
  20. Technical Stack

  21. API Gateway Service X Service Y Mercari API client Sakura

  22. API Gateway Service X Service Y Mercari API client Sakura

    GCP
  23. API Gateway Service X Service Y Mercari API client Sakura

    kubernetes GCP
  24. API Gateway Service X Service Y Mercari API client Sakura

    kubernetes GCP
  25. API Gateway Service X Service Y Mercari API client Sakura

    kubernetes GCP
  26. API Gateway Service X Service Y Mercari API client Sakura

    kubernetes GCP
  27. How to break down existing API

  28. API Gateway Service X Service Y Mercari API client

  29. API Gateway Service X Service Y Mercari API client

  30. Start from UX features

  31. Mercari Login Buy Sell Search Shipping ...

  32. Analyze fundamental components

  33. Mercari Login Buy Sell Search Shipping ... User Item Photo

    User Item Address
  34. Split them into microservices

  35. Gateway Login Buy Sell Search Shipping ... User Item Photo

    Notification Address ...
  36. Gateway Login Buy Sell Search Shipping ... User Item Photo

    Notification Address ...
  37. Gateway Login Buy Sell Search Shipping ... User Item Photo

    Notification Address ... BFF Layer DAO Layer notf db user db item db s3
  38. In reality...

  39. None
  40. How to migrate to microservices • Implement new features in

    microservices ◦ API Gateway ◦ Technical stack: Kubernetes, Go, gRPC, etc. • Break down existing API into multiple services ◦ Start from UX features ◦ Analyze fundamental components ◦ Split them into microservices
  41. Challenges

  42. Challenges: Testing With so many services, how do we test

    them?
  43. Challenges: Testing With so many services, how do we test

    them? • Enforce unit tests coverage • Automated integration tests • Reduce manual QA process
  44. Challenges: DB DB is in Sakura but microservices are in

    GCP.
  45. Challenges: DB DB is in Sakura but microservices are in

    GCP. • Run DAO layer services in Sakura • Migrate DB from Sakura to GCP • Create Tokyo DC and host DB / services in it
  46. Challenges: Transactions and Locking

  47. Challenges: Transactions and Locking     Distributed Locking

  48. Challenges: Transactions and Locking     Refactor with PubSub

  49. Challenges: Transactions and Locking Eventual Consistency!

  50. Thank you!