Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integration and Modernization of the Ad Serving System in Yahoo! JAPAN Ads - Get rid of 10 Years of Legacy -

Integration and Modernization of the Ad Serving System in Yahoo! JAPAN Ads - Get rid of 10 Years of Legacy -

Sho Kawasaki (Yahoo! JAPAN / Ad Development Division, Marketing Platform Group, MS Group / Engineer)

https://tech-verse.me/ja/sessions/127
https://tech-verse.me/en/sessions/127
https://tech-verse.me/ko/sessions/127

Tech-Verse2022
PRO

November 18, 2022
Tweet

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. None
  2. Target audience - People who develop and operate complex backend

    systems - People who are planning a large-scale system integration - Engineering Managers © 2022 Yahoo Japan Corporation
  3. Self-introduction - Sho Kawasaki - Joined as a new graduate

    in 2016 - Develop and operate ad serving systems - Raise a newborn son © 2022 Yahoo Japan Corporation
  4. We develop and operate the ad serving system Yahoo! JAPAN

    Ads Display Ads © 2022 Yahoo Japan Corporation Requirement: • High performance • Many functions
  5. Issues of the ad serving system Technical debt has accumulated

    for 10+ years © 2022 Yahoo Japan Corporation アフロ
  6. Integration and Modernization of the Ad Serving System Optimize communication

    process Integration of multiple components Reduce operating costs Migration to a CaaS environment Modernize development language Rewriting to Go from C++98 © 2022 Yahoo Japan Corporation
  7. Integration of multiple components © 2022 Yahoo Japan Corporation Ad

    Server Ad Optimization Server A B C Ad Optimization Server A B C Integrated component can handle communication wait times in parallel. Rethinking the roles of components and integrate them into one component.
  8. Results of the system renewal 85% Deployment time faster 10%

    Average latency faster © 2022 Yahoo Japan Corporation
  9. Devised points of the system renewal © 2022 Yahoo Japan

    Corporation
  10. Devised points of the system renewal Preparation Testing Development ©

    2022 Yahoo Japan Corporation
  11. Devised points of the system renewal Preparation Testing Development ©

    2022 Yahoo Japan Corporation
  12. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
  13. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
  14. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
  15. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
  16. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
  17. Create code templates Study session for components Preparation © 2022

    Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people Tidy functions Necessary Unnecessary A B C D E Too complex functions Share vision Too many people related System renewal ! Lower development cost Efficient parallel development Understand each other about the system Sharing goals with stakeholders System and teams are separated
  18. Devised points of the system renewal Preparation Testing Development ©

    2022 Yahoo Japan Corporation
  19. Knowledge sharing Development The number of developers increased. → Share

    at meetings and add to guidelines © 2022 Yahoo Japan Corporation Keep code quality if develop member increased. Naming Rules External Libraries Type Constraints %FQFOEFODZ*OKFDUJPO XJUIJOUFSGBDF /FX" #VJME# (FOFSBUF$ VJOU JOU For Example…
  20. Efficient development © 2022 Yahoo Japan Corporation Reusable common library

    Development required or optional? presence or absence? JSON tag extension with reflect For Example… JSON → Go struct Go struct → JSON Speed-up mock generation 1BDLBHF 1BDLBHF 1BDLBHF NPDL NPDL QBDLBHF QBDLBHF JOUFSGBDF JOUFSGBDF JOUFSGBDF HPHFOFSBUF HPHFOFSBUF Faster mock test Duplicate implementation in parallel development Execution time of test is slower
  21. Devised points of the system renewal Preparation Testing Development ©

    2022 Yahoo Japan Corporation
  22. Testing of the system renewal Testing © 2022 Yahoo Japan

    Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
  23. Testing of the system renewal Testing © 2022 Yahoo Japan

    Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
  24. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server

    Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
  25. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server

    Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
  26. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server

    Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
  27. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server

    Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock Check Check Check
  28. Testing of the system renewal Testing © 2022 Yahoo Japan

    Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
  29. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new)

    Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
  30. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new)

    Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
  31. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new)

    Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
  32. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new)

    Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock Diff Check
  33. Testing of the system renewal Testing © 2022 Yahoo Japan

    Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
  34. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server

    (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
  35. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server

    (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
  36. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server

    (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
  37. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server

    (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
  38. Performance Test Testing - gRPC server communication latency is slow.

    → Improved by using connection pool. - Large memory usage of function that generate structures. (by pprof) → Improved by returning a pointer instead of a value copy. © 2022 Yahoo Japan Corporation Improved performance by making it easier to survey bottlenecks For Example…
  39. Devised points of the system renewal © 2022 Yahoo Japan

    Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test Please try to reference that this devised points.
  40. Devised points of the system renewal © 2022 Yahoo Japan

    Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to proceed with a large-scale system renewal project Please try to reference that this devised points.
  41. Devised points of the system renewal © 2022 Yahoo Japan

    Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to parallel development with increasing development members Please try to reference that this devised points.
  42. Devised points of the system renewal © 2022 Yahoo Japan

    Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to test the system requiring many functions and high performance Please try to reference that this devised points.
  43. Summary Testing Migration to a CaaS environment Development Preparation Rewriting

    to Go from C++98 Integration of multiple components Integration and Modernization of the Ad Serving System Devised points of the system renewal Deployment time 85% improvement Average latency 10% improvement Results of the system renewal © 2022 Yahoo Japan Corporation
  44. Thank you! © 2022 Yahoo Japan Corporation