Share
Embed
Microservice Conversion at Demae-can, Food Delivery for 20+ Years
Shunsuke Urushihara (Demae-can / Product Division / Platform Department / Platform Development Group 1 / Engineering Manager)
https://tech-verse.me/ja/sessions/121
https://tech-verse.me/en/sessions/121
https://tech-verse.me/ko/sessions/121
Tech-Verse2022
November 18, 2022
Transcript
48 slides
-
About me - Shunsuke Urushihara - Engineering Manager - Server
Side - LINE Corporation (2017~) - Demae-can (2021~) -
About this session - A Case study of Micrcoservice migration
of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation. -
About Demae-can - A delivery site where you can immediately
order from restaurants and delivery stores. - Main users - Orderers - Restaurants/Stores - Drivers -
Demae-can : Business Growth (From : https://corporate.demae-can.com/ir_information/kpi.html) - Increased demand
for Food Delivery - The number of orders has also increased sharply - Increased competition in the Food Delivery industry -
Technical issues due to historical background - The now 20-year-old
food delivery service Demae-can has faced multiple technical problems caused by its long years of usage. - Bad user experience by site stop - Low development productivity makes it difficult to provide new value -
- Drastic improvement is needed - Decided to make an
Incremental migration from a legacy system to new system - There was a proposal to do a full replacement at the beginning of 2021. However, we abandoned this proposal because we needed to grow the product in parallel. Technical issues due to historical background -
Why Microservices? - Independent development and deployment - Highly scalable,
maintainable and testable - Service per team - Diversity of Technology Stack -
Disadvantages of Microservices - Difficulty of distributed systems - Eventual
Consistency - Manage large number of services, Crosscutting Concern - Costly implementation -
Trade-off in Microservices - Microservices have various tradeoffs - However,
we thought the advantages outweighed the Demae-can issues -
Architecture in 2021 API1 Frontend & API API2 Delivery System
Database Admin System Batch Merchant System -
Architecture in 2021 API1 Frontend & API API2 Delivery System
Database Admin System Batch Merchant System -
Architecture in 2021 API1 Frontend & API API2 Delivery System
Database Admin System Batch Merchant System -
Architecture in 2021 API1 Frontend & API API2 Delivery System
Database Admin System Batch Merchant System -
Architecture in 2021 API1 Frontend & API API2 Database Admin
System Batch Delivery System Merchant System -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Microservices for Order Linkage - Orchestration of order linkage, one
of Demaecan's core functions, is migrated to a microservice as an "Order Service”. - References to Order-related tables were particularly numerous, and were referenced by almost all functions. - Order linkage had unorganized concepts and many unknown errors. -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link
Delivery Service Database Database Admin Database Delivery Engine Order Service -
Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF
2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack -
Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF
2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack -
Issue 1: Complex specifications due to historical background - 20
years of complex specifications and patterns exist - Examples of terms: Own delivery, Sharing Delivery, Regular order, Off-hour order, Reserved order, Guest order, ASP, Takeout ASP etc… - Cognitive load is very high - Also, some specifications are unrecognizable -
Issue 1: Complex specifications due to historical background - New
team assigned to this project - Only two members had been with Demae-can for a long time, but the others had no domain knowledge - Members have different skills and backgrounds - Facilitated remotely. -
Solution 1-1: Learning domain knowledge - Without business domain knowledge,
it is difficult to effectively migrate a legacy system - Reading and understand legacy code -
Solution 1-1: Learning domain knowledge - Redefinition of specifications -
Create a common language - Data Modeling - Simplifying, Deprecation - We use Miro, Confluence and Zoom etc… -
Solution 1-2: Testing in production - Dealing with unrecognized problems
- Validate in production while avoiding degrading the user experience - Take risks step-by-step and then learn as a team. -
Solution 1-2: Testing in production (example) - Progressive Delivery (
Blue-Green Deployment ) - Run the old application and the new application in parallel to verify - Incrementally switch to a new application for each specific key or percentage -
Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF
2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack -
Issue 2 : SPoF (Single Point of Failure) - Cause
of deadlock and high load - Table refactoring not possible - In particular, order-related tables are read from almost all applications Database App1 App2 App3 App4 -
App4 Issue 2 : SPoF (Single Point of Failure) -
Site down due to high load caused by some applications. - Deadlock doesn't know what triggered it. - For large releases, multiple teams work on the release together early in the morning or late at night Database App1 App2 App3 -
Solution 2-1: Database per service - Basic Patterns of Microservice
Architecture - Responsible for tables that fit microservice responsibilities. - “Order Service” has ownership of order-related tables - DB refactoring at a time when it is sufficiently loosely coupled App1 App2 App3 Database Database App4 -
Solution 2-2: Asynchronous messaging - Asynchronous communication between microservices -
Reduce cascading failures - We use Apache Kafka - Orchestration, Choreography App1 App2 App3 Database Database App4 -
Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF
2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack -
Issue 3 : Legacy Technology stack - Old technology stack
was affecting development productivity. - Example: Java6, Seasar2, Oracle 11g etc… - CI/CD isn’t adopted - Limitations of Scalability - Unknown error and Alert Fatigue -
Solution 3 : Update Technology stack Renewed at the same
time as migration to microservices - Kotlin - Spring Boot - Apache Kafka - AWS - Terraform - Docker - GitHub Actions - NewRelic etc… -
Solution 3 : Update Technology stack - Improved Developer experience
and productivity - Achieve scalability and portability through containerization - Reduce fault resolution time by optimizing alerting -
Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF
2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack -
Architecture in 2021 Merchant System API1 Frontend & API API2
Delivery System Database Admin System Batch -
Architecture in 2022 Merchant System Order Service Frontend ɾ BFFɾCart
Merchant Link Delivery System Database Database Admin Database Delivery Engine API 2 Batch -
Architecture in 2023 Merchant System Order Service Frontend ɾ BFFɾCart
Merchant Link Delivery Service Database Database Admin Database Delivery Engine -
Effects of microservices migration progress - Unknown errors are reduced
: 100 -> 0~2 (almost none) / month - Reduced downtime : few hours -> few minutes or none / month - Number of releases : 50 -> 120 / month -
Next Actions - Migration to microservices still underway - Continuous
improvement continues after migration to microservices - The goal is to create a foundation that facilitates better user experience and value delivery -
About this session (Conclusion) - A Case study of Micrcoservice
migration of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation