Integration and Modernization of the Ad Serving System in Yahoo! JAPAN Ads - Get rid of 10 Years of Legacy -

Sho Kawasaki (Yahoo! JAPAN / Ad Development Division, Marketing Platform Group, MS Group / Engineer)

https://tech-verse.me/ja/sessions/127
https://tech-verse.me/en/sessions/127
https://tech-verse.me/ko/sessions/127

Transcript

2. Target audience - People who develop and operate complex backend systems - People who are planning a large-scale system integration - Engineering Managers © 2022 Yahoo Japan Corporation
3. Self-introduction - Sho Kawasaki - Joined as a new graduate in 2016 - Develop and operate ad serving systems - Raise a newborn son © 2022 Yahoo Japan Corporation
4. We develop and operate the ad serving system Yahoo! JAPAN Ads Display Ads © 2022 Yahoo Japan Corporation Requirement: • High performance • Many functions
5. Issues of the ad serving system Technical debt has accumulated for 10+ years © 2022 Yahoo Japan Corporation アフロ
6. Integration and Modernization of the Ad Serving System Optimize communication process Integration of multiple components Reduce operating costs Migration to a CaaS environment Modernize development language Rewriting to Go from C++98 © 2022 Yahoo Japan Corporation
7. Integration of multiple components © 2022 Yahoo Japan Corporation Ad Server Ad Optimization Server A B C Ad Optimization Server A B C Integrated component can handle communication wait times in parallel. Rethinking the roles of components and integrate them into one component.
8. Results of the system renewal 85% Deployment time faster 10% Average latency faster © 2022 Yahoo Japan Corporation
9. Devised points of the system renewal © 2022 Yahoo Japan Corporation
10. Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation
11. Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation
12. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
13. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
14. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
15. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
16. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !
17. Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people Tidy functions Necessary Unnecessary A B C D E Too complex functions Share vision Too many people related System renewal ! Lower development cost Efficient parallel development Understand each other about the system Sharing goals with stakeholders System and teams are separated
18. Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation
19. Knowledge sharing Development The number of developers increased. → Share at meetings and add to guidelines © 2022 Yahoo Japan Corporation Keep code quality if develop member increased. Naming Rules External Libraries Type Constraints %FQFOEFODZ
    *OKFDUJPO XJUI
    JOUFSGBDF /FX" #VJME# (FOFSBUF$ VJOU JOU For Example…
20. Efficient development © 2022 Yahoo Japan Corporation Reusable common library Development required or optional? presence or absence? JSON tag extension with reflect For Example… JSON → Go struct Go struct → JSON Speed-up mock generation 1BDLBHF 1BDLBHF 1BDLBHF NPDL NPDL QBDLBHF QBDLBHF JOUFSGBDF JOUFSGBDF JOUFSGBDF HP
    HFOFSBUF HP
    HFOFSBUF Faster mock test Duplicate implementation in parallel development Execution time of test is slower
21. Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation
22. Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
23. Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
24. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
25. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
26. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock
27. Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock Check Check Check
28. Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
29. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
30. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
31. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock
32. Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock Diff Check
33. Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?
34. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
35. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
36. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
37. Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers
38. Performance Test Testing - gRPC server communication latency is slow. → Improved by using connection pool. - Large memory usage of function that generate structures. (by pprof) → Improved by returning a pointer instead of a value copy. © 2022 Yahoo Japan Corporation Improved performance by making it easier to survey bottlenecks For Example…
39. Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test Please try to reference that this devised points.
40. Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to proceed with a large-scale system renewal project Please try to reference that this devised points.
41. Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to parallel development with increasing development members Please try to reference that this devised points.
42. Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to test the system requiring many functions and high performance Please try to reference that this devised points.
43. Summary Testing Migration to a CaaS environment Development Preparation Rewriting to Go from C++98 Integration of multiple components Integration and Modernization of the Ad Serving System Devised points of the system renewal Deployment time 85% improvement Average latency 10% improvement Results of the system renewal © 2022 Yahoo Japan Corporation
44. Thank you! © 2022 Yahoo Japan Corporation