Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

About me - Shunsuke Urushihara - Engineering Manager - Server Side - LINE Corporation (2017~) - Demae-can (2021~)

Slide 3

Slide 3 text

About this session - A Case study of Micrcoservice migration of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation.

Slide 4

Slide 4 text

About Demae-can - A delivery site where you can immediately order from restaurants and delivery stores. - Main users - Orderers - Restaurants/Stores - Drivers

Slide 5

Slide 5 text

Demae-can : Business Growth (From : https://corporate.demae-can.com/ir_information/kpi.html) - Increased demand for Food Delivery - The number of orders has also increased sharply - Increased competition in the Food Delivery industry

Slide 6

Slide 6 text

Technical issues due to historical background - The now 20-year-old food delivery service Demae-can has faced multiple technical problems caused by its long years of usage. - Bad user experience by site stop - Low development productivity makes it difficult to provide new value

Slide 7

Slide 7 text

- Drastic improvement is needed - Decided to make an Incremental migration from a legacy system to new system - There was a proposal to do a full replacement at the beginning of 2021. However, we abandoned this proposal because we needed to grow the product in parallel. Technical issues due to historical background

Slide 8

Slide 8 text

Why Microservices? - Independent development and deployment - Highly scalable, maintainable and testable - Service per team - Diversity of Technology Stack

Slide 9

Slide 9 text

Disadvantages of Microservices - Difficulty of distributed systems - Eventual Consistency - Manage large number of services, Crosscutting Concern - Costly implementation

Slide 10

Slide 10 text

Trade-off in Microservices - Microservices have various tradeoffs - However, we thought the advantages outweighed the Demae-can issues

Slide 11

Slide 11 text

Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System

Slide 12

Slide 12 text

Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System

Slide 13

Slide 13 text

Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System

Slide 14

Slide 14 text

Architecture in 2021 API1 Frontend & API API2 Delivery System Database Admin System Batch Merchant System

Slide 15

Slide 15 text

Architecture in 2021 API1 Frontend & API API2 Database Admin System Batch Delivery System Merchant System

Slide 16

Slide 16 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 17

Slide 17 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 18

Slide 18 text

Microservices for Order Linkage - Orchestration of order linkage, one of Demaecan's core functions, is migrated to a microservice as an "Order Service”. - References to Order-related tables were particularly numerous, and were referenced by almost all functions. - Order linkage had unorganized concepts and many unknown errors.

Slide 19

Slide 19 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 20

Slide 20 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 21

Slide 21 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 22

Slide 22 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 23

Slide 23 text

Architecture in 2023 Merchant System Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine Order Service

Slide 24

Slide 24 text

Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack

Slide 25

Slide 25 text

Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack

Slide 26

Slide 26 text

Issue 1: Complex specifications due to historical background - 20 years of complex specifications and patterns exist - Examples of terms: Own delivery, Sharing Delivery, Regular order, Off-hour order, Reserved order, Guest order, ASP, Takeout ASP etc… - Cognitive load is very high - Also, some specifications are unrecognizable

Slide 27

Slide 27 text

Issue 1: Complex specifications due to historical background - New team assigned to this project - Only two members had been with Demae-can for a long time, but the others had no domain knowledge - Members have different skills and backgrounds - Facilitated remotely.

Slide 28

Slide 28 text

Solution 1-1: Learning domain knowledge - Without business domain knowledge, it is difficult to effectively migrate a legacy system - Reading and understand legacy code

Slide 29

Slide 29 text

Solution 1-1: Learning domain knowledge - Redefinition of specifications - Create a common language - Data Modeling - Simplifying, Deprecation - We use Miro, Confluence and Zoom etc…

Slide 30

Slide 30 text

Solution 1-2: Testing in production - Dealing with unrecognized problems - Validate in production while avoiding degrading the user experience - Take risks step-by-step and then learn as a team.

Slide 31

Slide 31 text

Solution 1-2: Testing in production (example) - Progressive Delivery ( Blue-Green Deployment ) - Run the old application and the new application in parallel to verify - Incrementally switch to a new application for each specific key or percentage

Slide 32

Slide 32 text

Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack

Slide 33

Slide 33 text

Issue 2 : SPoF (Single Point of Failure) - Cause of deadlock and high load - Table refactoring not possible - In particular, order-related tables are read from almost all applications Database App1 App2 App3 App4

Slide 34

Slide 34 text

App4 Issue 2 : SPoF (Single Point of Failure) - Site down due to high load caused by some applications. - Deadlock doesn't know what triggered it. - For large releases, multiple teams work on the release together early in the morning or late at night Database App1 App2 App3

Slide 35

Slide 35 text

Solution 2-1: Database per service - Basic Patterns of Microservice Architecture - Responsible for tables that fit microservice responsibilities. - “Order Service” has ownership of order-related tables - DB refactoring at a time when it is sufficiently loosely coupled App1 App2 App3 Database Database App4

Slide 36

Slide 36 text

Solution 2-2: Asynchronous messaging - Asynchronous communication between microservices - Reduce cascading failures - We use Apache Kafka - Orchestration, Choreography App1 App2 App3 Database Database App4

Slide 37

Slide 37 text

Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack

Slide 38

Slide 38 text

Issue 3 : Legacy Technology stack - Old technology stack was affecting development productivity. - Example: Java6, Seasar2, Oracle 11g etc… - CI/CD isn’t adopted - Limitations of Scalability - Unknown error and Alert Fatigue

Slide 39

Slide 39 text

Solution 3 : Update Technology stack Renewed at the same time as migration to microservices - Kotlin - Spring Boot - Apache Kafka - AWS - Terraform - Docker - GitHub Actions - NewRelic etc…

Slide 40

Slide 40 text

Solution 3 : Update Technology stack - Improved Developer experience and productivity - Achieve scalability and portability through containerization - Reduce fault resolution time by optimizing alerting

Slide 41

Slide 41 text

Challenges and Solutions 1-1 :Learning domain knowledge Challenge 2: SPoF 2-1: Database per service Challenge 1: Complex specifications due to historical background 1-2 : Testing in production Challenge 3: Legacy Technology 2-2: Asynchronous messaging 3: Update Technology Stack

Slide 42

Slide 42 text

Architecture in 2021 Merchant System API1 Frontend & API API2 Delivery System Database Admin System Batch

Slide 43

Slide 43 text

Architecture in 2022 Merchant System Order Service Frontend ɾ BFFɾCart Merchant Link Delivery System Database Database Admin Database Delivery Engine API 2 Batch

Slide 44

Slide 44 text

Architecture in 2023 Merchant System Order Service Frontend ɾ BFFɾCart Merchant Link Delivery Service Database Database Admin Database Delivery Engine

Slide 45

Slide 45 text

Effects of microservices migration progress - Unknown errors are reduced : 100 -> 0~2 (almost none) / month - Reduced downtime : few hours -> few minutes or none / month - Number of releases : 50 -> 120 / month

Slide 46

Slide 46 text

Next Actions - Migration to microservices still underway - Continuous improvement continues after migration to microservices - The goal is to create a foundation that facilitates better user experience and value delivery

Slide 47

Slide 47 text

About this session (Conclusion) - A Case study of Micrcoservice migration of Demae-can - Microservices are effective even for products with a long history - The project of migration should not only aim to introduce specific practices or modern technologies, but also to analyze and propose issues according to the situation

Slide 48

Slide 48 text

Thank you