Microservice Conversion at Demae-can, Food Delivery for 20+ Years

Shunsuke Urushihara (Demae-can / Product Division / Platform Department / Platform Development Group 1 / Engineering Manager)

https://tech-verse.me/ja/sessions/121
https://tech-verse.me/en/sessions/121
https://tech-verse.me/ko/sessions/121

Transcript

2. About me - Shunsuke Urushihara - Engineering Manager - Server Side - LINE Corporation (2017~) - Demae-can (2021~)
3. About this session - A Case study of Micrcoservice migration of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation.
4. About Demae-can - A delivery site where you can immediately order from restaurants and delivery stores. - Main users - Orderers - Restaurants/Stores - Drivers
5. Demae-can : Business Growth (From : https://corporate.demae-can.com/ir_information/kpi.html) - Increased demand for Food Delivery - The number of orders has also increased sharply - Increased competition in the Food Delivery industry
6. Technical issues due to historical background - The now 20-year-old food delivery service Demae-can has faced multiple technical problems caused by its long years of usage. - Bad user experience by site stop - Low development productivity makes it difficult to provide new value
7. - Drastic improvement is needed - Decided to make an Incremental migration from a legacy system to new system - There was a proposal to do a full replacement at the beginning of 2021. However, we abandoned this proposal because we needed to grow the product in parallel. Technical issues due to historical background
8. Why Microservices? - Independent development and deployment - Highly scalable, maintainable and testable - Service per team - Diversity of Technology Stack
9. Disadvantages of Microservices - Difficulty of distributed systems - Eventual Consistency - Manage large number of services, Crosscutting Concern - Costly implementation
10. Trade-off in Microservices - Microservices have various tradeoffs - However, we thought the advantages outweighed the Demae-can issues
11. Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System
12. Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System
13. Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System
14. Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System
15. Architecture in 2021 API1 Frontend & API API2 Database Admin System Batch Delivery System Merchant System
16. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
17. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
18. Microservices for Order Linkage - Orchestration of order linkage, one of Demaecan's core functions, is migrated to a microservice as an "Order Service”. - References to Order-related tables were particularly numerous, and were referenced by almost all functions. - Order linkage had unorganized concepts and many unknown errors.
19. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
20. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
21. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
22. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
23. Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service
24. Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack
25. Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack
26. Issue 1: Complex specifications due to historical background - 20 years of complex specifications and patterns exist - Examples of terms: Own delivery, Sharing Delivery, Regular order, Off-hour order, Reserved order, Guest order, ASP, Takeout ASP etc… - Cognitive load is very high - Also, some specifications are unrecognizable
27. Issue 1: Complex specifications due to historical background - New team assigned to this project - Only two members had been with Demae-can for a long time, but the others had no domain knowledge - Members have different skills and backgrounds - Facilitated remotely.
28. Solution 1-1: Learning domain knowledge - Without business domain knowledge, it is difficult to effectively migrate a legacy system - Reading and understand legacy code
29. Solution 1-1: Learning domain knowledge - Redefinition of specifications - Create a common language - Data Modeling - Simplifying, Deprecation - We use Miro, Confluence and Zoom etc…
30. Solution 1-2: Testing in production - Dealing with unrecognized problems - Validate in production while avoiding degrading the user experience - Take risks step-by-step and then learn as a team.
31. Solution 1-2: Testing in production (example) - Progressive Delivery ( Blue-Green Deployment ) - Run the old application and the new application in parallel to verify - Incrementally switch to a new application for each specific key or percentage
32. Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack
33. Issue 2 : SPoF (Single Point of Failure) - Cause of deadlock and high load - Table refactoring not possible - In particular, order-related tables are read from almost all applications Database App1 App2 App3 App4
34. App4 Issue 2 : SPoF (Single Point of Failure) - Site down due to high load caused by some applications. - Deadlock doesn't know what triggered it. - For large releases, multiple teams work on the release together early in the morning or late at night Database App1 App2 App3
35. Solution 2-1: Database per service - Basic Patterns of Microservice Architecture - Responsible for tables that fit microservice responsibilities. - “Order Service” has ownership of order-related tables - DB refactoring at a time when it is sufficiently loosely coupled App1 App2 App3 Database Database App4
36. Solution 2-2: Asynchronous messaging - Asynchronous communication between microservices - Reduce cascading failures - We use Apache Kafka - Orchestration, Choreography App1 App2 App3 Database Database App4
37. Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack
38. Issue 3 : Legacy Technology stack - Old technology stack was affecting development productivity. - Example: Java6, Seasar2, Oracle 11g etc… - CI/CD isn’t adopted - Limitations of Scalability - Unknown error and Alert Fatigue
39. Solution 3 : Update Technology stack Renewed at the same time as migration to microservices - Kotlin - Spring Boot - Apache Kafka - AWS - Terraform - Docker - GitHub Actions - NewRelic etc…
40. Solution 3 : Update Technology stack - Improved Developer experience and productivity - Achieve scalability and portability through containerization - Reduce fault resolution time by optimizing alerting
41. Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack
42. Architecture in 2021 Merchant System API1 Frontend & API API2 Delivery System Database Admin System Batch
43. Architecture in 2022 Merchant System Order Service Frontend ɾ BFFɾCart Merchant Link Delivery System Database Database Admin Database Delivery Engine API 2 Batch
44. Architecture in 2023 Merchant System Order Service Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine
45. Effects of microservices migration progress - Unknown errors are reduced : 100 -> 0~2 (almost none) / month - Reduced downtime : few hours -> few minutes or none / month - Number of releases : 50 -> 120 / month
46. Next Actions - Migration to microservices still underway - Continuous improvement continues after migration to microservices - The goal is to create a foundation that facilitates better user experience and value delivery
47. About this session (Conclusion) - A Case study of Micrcoservice migration of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation
48. Thank you