Background of Development of Next-Generation Marketing Data Utilization Infrastructure "Business Manager"

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Agenda - What is Business Manager? - About the system - Findings from development - Summary

Slide 3

Slide 3 text

What is Business Manager?

Slide 4

Slide 4 text

LINE Ads Account LINE Official Account Business Manager Delivery LINE Ads LINE Official Account Other Service Account1 Other Service Account2 Business Manager etc

Slide 5

Slide 5 text

LINE B2B Products

Slide 6

Slide 6 text

LINE B2B Products LINE Official Account LINE Ads LINE POINT etc.

Slide 7

Slide 7 text

LINE Ads Account LINE Official Account Business Manager Other Service Account1 Other Service Account2 Business Manager

Slide 8

Slide 8 text

LINE Ads Account LINE Official Account Business Manager Other Service Account1 Other Service Account2 Business Manager Delivery LINE Ads LINE Official Account etc

Slide 9

Slide 9 text

LINE Ads Account LINE Official Account Business Manager Delivery Other Service Account1 Other Service Account2 Analytics Business Manager LINE Ads LINE Official Account etc

Slide 10

Slide 10 text

What makes Business Manager “Next-Generation”

Slide 11

Slide 11 text

Cross targeting (As Is) LINE Ads LINE Official Account Brand A Company LINE Ads LINE Official Account Brand B

Slide 12

Slide 12 text

Business Manager (To Be) LINE Ads LINE Official Account Brand A Company LINE Ads LINE Official Account Brand B

Slide 13

Slide 13 text

Business Manager features

Slide 14

Slide 14 text

Manage an organization Organization A Data Data Data Data Data Data Data Data LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1

Slide 15

Slide 15 text

Manage an organization Organization A LINE Tag 1 Data LINE Tag 2 Data LINE Tag 3 Data LINE Tag 4 Data LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1

Slide 16

Slide 16 text

Manage an organization Organization A LINE Tag 1 Audience 1 LINE Tag 2 Audience 2 LINE Tag 3 Audience 3 LINE Tag 4 Audience 4 LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1

Slide 17

Slide 17 text

Verification LINE Ads 2 LINE Ads 1 LINE Official Account 2 LINE Official Account 1 Organization A Audience 1 Audience 2 Audience 3 Audience 4

Slide 18

Slide 18 text

Verification LINE Ads 2 LINE Ads 1 LINE Official Account 2 LINE Official Account 1 Organization A Audience 1 Audience 2 Audience 3 Audience 4 WFSJGJFE WFSJGJFE WFSJGJFE WFSJGJFE

Slide 19

Slide 19 text

Change permissions LINE Ads 1 LINE Official Account 1 Organization A Audience 1 Audience 2

Slide 20

Slide 20 text

Create LINE tags and audience data LINE Official Account 1 LINE Ads 1 LINE Ads 2 Organization A LINE Tag 1 Audience 1 LINE Tag 2 Audience 2 LINE Tag 3 Audience 3

Slide 21

Slide 21 text

Create LINE tags and custom audiences LINE Official Account 1 LINE Ads 1 LINE Ads 2 Organization A LINE Tag 1 Audience 1 LINE Tag 2 Audience 2 LINE Tag 3 Audience 3 Audience 4 LINE Tag 4

Slide 22

Slide 22 text

About the system

Slide 23

Slide 23 text

Before that

Slide 24

Slide 24 text

LINE DMP “LINE DMP: Connect the All B2B Products with Over 100M Users” https://linedevday.linecorp.com/2020/ja/sessions/6695 LINE Services LINE DMP LINE Platform Client Data LINE Ads LINE Official Account ・・・

Slide 25

Slide 25 text

Cross Targeting (As Is) API LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 26

Slide 26 text

Business Manager (To Be) Management System Business Manager API API Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 27

Slide 27 text

System collaboration patterns Management System Business Manager API API Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 28

Slide 28 text

REST API Management System Business Manager API API Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 29

Slide 29 text

Messaging Management System Business Manager API API Kafka Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions

Slide 30

Slide 30 text

Messaging Management System Business Manager API API Kafka Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions

Slide 31

Slide 31 text

Messaging Management System Business Manager API API Kafka Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions

Slide 32

Slide 32 text

Technology stack - MySQL, Redis, Kafka - Verda (LINE private cloud) - Kotlin, Spring Boot

Slide 33

Slide 33 text

Technology stack - MySQL, Redis, Kafka - Verda (LINE private cloud) - Kotlin, Spring Boot

Slide 34

Slide 34 text

Technology stack - MySQL, Redis, Kafka - Verda (LINE private cloud) - Kotlin, Spring Boot

Slide 35

Slide 35 text

Technology stack - MySQL, Redis, Kafka - Verda (LINE private cloud) - Kotlin, Spring Boot

Slide 36

Slide 36 text

Findings from development

Slide 37

Slide 37 text

System collaboration

Slide 38

Slide 38 text

System overview Management System Business Manager API API Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 39

Slide 39 text

Synchronize transactions between DB and Kafka “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/

Slide 40

Slide 40 text

Sync Business Manager changes to LINE DMP - Synchronize transactions between DB and Kafka - Users can change data permissions on Business Manager - Changes need to sync to LINE DMP at least once and in the correct order

Slide 41

Slide 41 text

Three ideas “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/

Slide 42

Slide 42 text

1. API Callback API Kafka Permissions Consumer transaction 1. insert/update/delete 2. publish Business Manager 3. check

Slide 43

Slide 43 text

1. API Callback Pros - Simple - Proven method in other teams Cons - API performance issue

Slide 44

Slide 44 text

1. API Callback Pros - Simple - Proven method in other teams Cons - API performance issue

Slide 45

Slide 45 text

2. CDC(Change Data Capture) Permissions MySQL insert/update/ delete binlog Processor (with Debezium) Kafka publish

Slide 46

Slide 46 text

2. CDC(Change Data Capture) Pros - Real-time - Proven method in other teams Cons - Consumer design is difficult - Needed to build a dedicated infrastructure - API Callbacks are also needed

Slide 47

Slide 47 text

2. CDC(Change Data Capture) Pros - Real-time - Proven method in other teams Cons - Consumer design is difficult - Needed to build a dedicated infrastructure - API Callbacks are also needed

Slide 48

Slide 48 text

3. Polling publisher Permissions MySQL Event queue Batch Kafka transaction insert/update/ delete insert publish polling

Slide 49

Slide 49 text

3. Polling publisher Pros - Simple - Ensure correct order Cons - Not in real-time - It does not scale if many changes occurred

Slide 50

Slide 50 text

3. Polling publisher Pros - Simple - Ensure correct order Cons - Not in real-time - It does not scale if many changes occurred

Slide 51

Slide 51 text

Final architecture Permissions MySQL Event queue Batch 1 (polling) Kafka transaction update Permissions (resolved) MySQL Batch 2 (polling)

Slide 52

Slide 52 text

Development by multiple teams

Slide 53

Slide 53 text

REST APIs Management System Business Manager API API Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data

Slide 54

Slide 54 text

Management System API API Business Manager Management System Frontend (SPA) backend me frontend

Slide 55

Slide 55 text

Internal API Business Manager Internal API API LINE DMP backend me API LINE Ads API LINE Official Account LINE DMP LINE Ads LINE Official Account

Slide 56

Slide 56 text

Development teams Business Manager frontend LINE DMP backend LINE Ads LINE Official Account me

Slide 57

Slide 57 text

Cost of maintaining API docs “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/

Slide 58

Slide 58 text

Cost of maintaining API docs - Every time change code, modify API docs as well - Complicated specifications, tight schedule - API docs in internal-wiki

Slide 59

Slide 59 text

springdoc-openapi “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/

Slide 60

Slide 60 text

springdoc-openapi - Swagger UI, Kotlin support - Can customize programmatically - Helps automating the generation of API docs using Spring Boot projects https://github.com/springdoc/springdoc-openapi

Slide 61

Slide 61 text

Controller(Spring Web MVC)

Slide 62

Slide 62 text

Swagger UI

Slide 63

Slide 63 text

Generate OpenAPI v3 YAML

Slide 64

Slide 64 text

API doc review - Can review on GitHub Pull-Request - API docs are auto-generated by Jenkins - ReDoc (https://github.com/Redocly/redoc)

Slide 65

Slide 65 text

Frontend integration > npx openapi-typescript api-spec.yml --version 3 --output generated.ts - Convert OpenAPI v3 YAML to TypeScript interface - openapi-typescript (https://github.com/drwpow/openapi- typescript)

Slide 66

Slide 66 text

Customization

Slide 67

Slide 67 text

Customization

Slide 68

Slide 68 text

Customization

Slide 69

Slide 69 text

Programming approach

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Many IDs - More account types will be added - More data types will be added - There are more than 20 types of IDs in Business Manager

Slide 72

Slide 72 text

ID as a Value Object “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/

Slide 73

Slide 73 text

Value Object DDD(Domain Drive Design) - Just a data class as a simple wrapper for primitive values - Value Equality, Immutability

Slide 74

Slide 74 text

Use Value Object ID from Controller to DAO - Avoid mistakes

Slide 75

Slide 75 text

Value Object with springdoc-openapi

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Null safety

Slide 78

Slide 78 text

Sealed classes/interfaces, Enum + when

Slide 79

Slide 79 text

Inline classes(not used yet)

Slide 80

Slide 80 text

New team, Covid-19, etc.

Slide 81

Slide 81 text

Mob programming - Pair programming with all team members - Screen sharing via Zoom - Working on the same git branch - Infrastructure task is also done by “mob”

Slide 82

Slide 82 text

Always on Zoom - Always stay connected to Zoom - Turn off the mic and camera - Talk to team members at any time

Slide 83

Slide 83 text

Daily huddle - A 15-30 minutes daily team meeting - Start at 11:00 a.m. - Discuss system design - Share even the most trivial things - Manage tasks

Slide 84

Slide 84 text

Stream all changes to Slack - Trello, GitHub pull-request, CI Results, etc. - Releases are done by bot (ChatOps)

Slide 85

Slide 85 text

Summary

Slide 86

Slide 86 text

Summary - There are many challenges in the development of Business Manager - We are hiring - Business Manager is a next-generation data utilization platform

Slide 87

Slide 87 text

Thank you