$30 off During Our Annual Pro Sale. View Details »

Background of Development of Next-Generation Marketing Data Utilization Infrastructure "Business Manager"

LINE DEVDAY 2021
PRO
November 11, 2021
88

Background of Development of Next-Generation Marketing Data Utilization Infrastructure "Business Manager"

LINE DEVDAY 2021
PRO

November 11, 2021
Tweet

Transcript

  1. None
  2. Agenda - What is Business Manager? - About the system

    - Findings from development - Summary
  3. What is Business Manager?

  4. LINE Ads Account LINE Official Account Business Manager Delivery LINE

    Ads LINE Official Account Other Service Account1 Other Service Account2 Business Manager etc
  5. LINE B2B Products

  6. LINE B2B Products LINE Official Account LINE Ads LINE POINT

    etc.
  7. LINE Ads Account LINE Official Account Business Manager Other Service

    Account1 Other Service Account2 Business Manager
  8. LINE Ads Account LINE Official Account Business Manager Other Service

    Account1 Other Service Account2 Business Manager Delivery LINE Ads LINE Official Account etc
  9. LINE Ads Account LINE Official Account Business Manager Delivery Other

    Service Account1 Other Service Account2 Analytics Business Manager LINE Ads LINE Official Account etc
  10. What makes Business Manager “Next-Generation”

  11. Cross targeting (As Is) LINE Ads LINE Official Account Brand

    A Company LINE Ads LINE Official Account Brand B
  12. Business Manager (To Be) LINE Ads LINE Official Account Brand

    A Company LINE Ads LINE Official Account Brand B
  13. Business Manager features

  14. Manage an organization Organization A Data Data Data Data Data

    Data Data Data LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1
  15. Manage an organization Organization A LINE Tag 1 Data LINE

    Tag 2 Data LINE Tag 3 Data LINE Tag 4 Data LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1
  16. Manage an organization Organization A LINE Tag 1 Audience 1

    LINE Tag 2 Audience 2 LINE Tag 3 Audience 3 LINE Tag 4 Audience 4 LINE Official Account 2 LINE Ads 2 LINE Official Account 1 LINE Ads 1
  17. Verification LINE Ads 2 LINE Ads 1 LINE Official Account

    2 LINE Official Account 1 Organization A Audience 1 Audience 2 Audience 3 Audience 4
  18. Verification LINE Ads 2 LINE Ads 1 LINE Official Account

    2 LINE Official Account 1 Organization A Audience 1 Audience 2 Audience 3 Audience 4 WFSJGJFE WFSJGJFE WFSJGJFE WFSJGJFE
  19. Change permissions LINE Ads 1 LINE Official Account 1 Organization

    A Audience 1 Audience 2
  20. Create LINE tags and audience data LINE Official Account 1

    LINE Ads 1 LINE Ads 2 Organization A LINE Tag 1 Audience 1 LINE Tag 2 Audience 2 LINE Tag 3 Audience 3
  21. Create LINE tags and custom audiences LINE Official Account 1

    LINE Ads 1 LINE Ads 2 Organization A LINE Tag 1 Audience 1 LINE Tag 2 Audience 2 LINE Tag 3 Audience 3 Audience 4 LINE Tag 4
  22. About the system

  23. Before that

  24. LINE DMP “LINE DMP: Connect the All B2B Products with

    Over 100M Users” https://linedevday.linecorp.com/2020/ja/sessions/6695 LINE Services LINE DMP LINE Platform Client Data LINE Ads LINE Official Account ・・・
  25. Cross Targeting (As Is) API LINE DMP API Other services

    LINE Ads LINE Official Account etc Audience data
  26. Business Manager (To Be) Management System Business Manager API API

    Kafka Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data
  27. System collaboration patterns Management System Business Manager API API Kafka

    Batch Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data
  28. REST API Management System Business Manager API API Kafka Batch

    Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data
  29. Messaging Management System Business Manager API API Kafka Consumer LINE

    DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions
  30. Messaging Management System Business Manager API API Kafka Consumer LINE

    DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions
  31. Messaging Management System Business Manager API API Kafka Consumer LINE

    DMP API Other services LINE Ads LINE Official Account etc Audience data Batch Permissions
  32. Technology stack - MySQL, Redis, Kafka - Verda (LINE private

    cloud) - Kotlin, Spring Boot
  33. Technology stack - MySQL, Redis, Kafka - Verda (LINE private

    cloud) - Kotlin, Spring Boot
  34. Technology stack - MySQL, Redis, Kafka - Verda (LINE private

    cloud) - Kotlin, Spring Boot
  35. Technology stack - MySQL, Redis, Kafka - Verda (LINE private

    cloud) - Kotlin, Spring Boot
  36. Findings from development

  37. System collaboration

  38. System overview Management System Business Manager API API Kafka Batch

    Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data
  39. Synchronize transactions between DB and Kafka “Twemoji” ©Twitter, Inc and

    other contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/
  40. Sync Business Manager changes to LINE DMP - Synchronize transactions

    between DB and Kafka - Users can change data permissions on Business Manager - Changes need to sync to LINE DMP at least once and in the correct order
  41. Three ideas “Twemoji” ©Twitter, Inc and other contributors (Licensed under

    CC-BY 4.0) https://twemoji.twitter.com/
  42. 1. API Callback API Kafka Permissions Consumer transaction 1. insert/update/delete

    2. publish Business Manager 3. check
  43. 1. API Callback Pros - Simple - Proven method in

    other teams Cons - API performance issue
  44. 1. API Callback Pros - Simple - Proven method in

    other teams Cons - API performance issue
  45. 2. CDC(Change Data Capture) Permissions MySQL insert/update/ delete binlog Processor

    (with Debezium) Kafka publish
  46. 2. CDC(Change Data Capture) Pros - Real-time - Proven method

    in other teams Cons - Consumer design is difficult - Needed to build a dedicated infrastructure - API Callbacks are also needed
  47. 2. CDC(Change Data Capture) Pros - Real-time - Proven method

    in other teams Cons - Consumer design is difficult - Needed to build a dedicated infrastructure - API Callbacks are also needed
  48. 3. Polling publisher Permissions MySQL Event queue Batch Kafka transaction

    insert/update/ delete insert publish polling
  49. 3. Polling publisher Pros - Simple - Ensure correct order

    Cons - Not in real-time - It does not scale if many changes occurred
  50. 3. Polling publisher Pros - Simple - Ensure correct order

    Cons - Not in real-time - It does not scale if many changes occurred
  51. Final architecture Permissions MySQL Event queue Batch 1 (polling) Kafka

    transaction update Permissions (resolved) MySQL Batch 2 (polling)
  52. Development by multiple teams

  53. REST APIs Management System Business Manager API API Kafka Batch

    Permissions Consumer LINE DMP API Other services LINE Ads LINE Official Account etc Audience data
  54. Management System API API Business Manager Management System Frontend (SPA)

    backend me frontend
  55. Internal API Business Manager Internal API API LINE DMP backend

    me API LINE Ads API LINE Official Account LINE DMP LINE Ads LINE Official Account
  56. Development teams Business Manager frontend LINE DMP backend LINE Ads

    LINE Official Account me
  57. Cost of maintaining API docs “Twemoji” ©Twitter, Inc and other

    contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/
  58. Cost of maintaining API docs - Every time change code,

    modify API docs as well - Complicated specifications, tight schedule - API docs in internal-wiki
  59. springdoc-openapi “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY

    4.0) https://twemoji.twitter.com/
  60. springdoc-openapi - Swagger UI, Kotlin support - Can customize programmatically

    - Helps automating the generation of API docs using Spring Boot projects https://github.com/springdoc/springdoc-openapi
  61. Controller(Spring Web MVC)

  62. Swagger UI

  63. Generate OpenAPI v3 YAML

  64. API doc review - Can review on GitHub Pull-Request -

    API docs are auto-generated by Jenkins - ReDoc (https://github.com/Redocly/redoc)
  65. Frontend integration > npx openapi-typescript api-spec.yml --version 3 --output generated.ts

    - Convert OpenAPI v3 YAML to TypeScript interface - openapi-typescript (https://github.com/drwpow/openapi- typescript)
  66. Customization

  67. Customization

  68. Customization

  69. Programming approach

  70. Many IDs “Twemoji” ©Twitter, Inc and other contributors (Licensed under

    CC-BY 4.0) https://twemoji.twitter.com/
  71. Many IDs - More account types will be added -

    More data types will be added - There are more than 20 types of IDs in Business Manager
  72. ID as a Value Object “Twemoji” ©Twitter, Inc and other

    contributors (Licensed under CC-BY 4.0) https://twemoji.twitter.com/
  73. Value Object DDD(Domain Drive Design) - Just a data class

    as a simple wrapper for primitive values - Value Equality, Immutability
  74. Use Value Object ID from Controller to DAO - Avoid

    mistakes
  75. Value Object with springdoc-openapi

  76. Kotlin “Twemoji” ©Twitter, Inc and other contributors (Licensed under CC-BY

    4.0) https://twemoji.twitter.com/
  77. Null safety

  78. Sealed classes/interfaces, Enum + when

  79. Inline classes(not used yet)

  80. New team, Covid-19, etc.

  81. Mob programming - Pair programming with all team members -

    Screen sharing via Zoom - Working on the same git branch - Infrastructure task is also done by “mob”
  82. Always on Zoom - Always stay connected to Zoom -

    Turn off the mic and camera - Talk to team members at any time
  83. Daily huddle - A 15-30 minutes daily team meeting -

    Start at 11:00 a.m. - Discuss system design - Share even the most trivial things - Manage tasks
  84. Stream all changes to Slack - Trello, GitHub pull-request, CI

    Results, etc. - Releases are done by bot (ChatOps)
  85. Summary

  86. Summary - There are many challenges in the development of

    Business Manager - We are hiring - Business Manager is a next-generation data utilization platform
  87. Thank you