Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Target audience - People who develop and operate complex backend systems - People who are planning a large-scale system integration - Engineering Managers © 2022 Yahoo Japan Corporation

Slide 3

Slide 3 text

Self-introduction - Sho Kawasaki - Joined as a new graduate in 2016 - Develop and operate ad serving systems - Raise a newborn son © 2022 Yahoo Japan Corporation

Slide 4

Slide 4 text

We develop and operate the ad serving system Yahoo! JAPAN Ads Display Ads © 2022 Yahoo Japan Corporation Requirement: • High performance • Many functions

Slide 5

Slide 5 text

Issues of the ad serving system Technical debt has accumulated for 10+ years © 2022 Yahoo Japan Corporation アフロ

Slide 6

Slide 6 text

Integration and Modernization of the Ad Serving System Optimize communication process Integration of multiple components Reduce operating costs Migration to a CaaS environment Modernize development language Rewriting to Go from C++98 © 2022 Yahoo Japan Corporation

Slide 7

Slide 7 text

Integration of multiple components © 2022 Yahoo Japan Corporation Ad Server Ad Optimization Server A B C Ad Optimization Server A B C Integrated component can handle communication wait times in parallel. Rethinking the roles of components and integrate them into one component.

Slide 8

Slide 8 text

Results of the system renewal 85% Deployment time faster 10% Average latency faster © 2022 Yahoo Japan Corporation

Slide 9

Slide 9 text

Devised points of the system renewal © 2022 Yahoo Japan Corporation

Slide 10

Slide 10 text

Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation

Slide 11

Slide 11 text

Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation

Slide 12

Slide 12 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !

Slide 13

Slide 13 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !

Slide 14

Slide 14 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !

Slide 15

Slide 15 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !

Slide 16

Slide 16 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people System and teams are separated Tidy functions Necessary Unnecessary A B C D E Too many complex functions Share vision Too many people related System renewal !

Slide 17

Slide 17 text

Create code templates Study session for components Preparation © 2022 Yahoo Japan Corporation Ad serving system is … func getAd(request, user) Ad { // TODO } type Ad strunct { // TODO } type Boundary { getAd(request, user) Ad } Development by multiple people Tidy functions Necessary Unnecessary A B C D E Too complex functions Share vision Too many people related System renewal ! Lower development cost Efficient parallel development Understand each other about the system Sharing goals with stakeholders System and teams are separated

Slide 18

Slide 18 text

Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation

Slide 19

Slide 19 text

Knowledge sharing Development The number of developers increased. → Share at meetings and add to guidelines © 2022 Yahoo Japan Corporation Keep code quality if develop member increased. Naming Rules External Libraries Type Constraints %FQFOEFODZ*OKFDUJPO XJUIJOUFSGBDF /FX" #VJME# (FOFSBUF$ VJOU JOU For Example…

Slide 20

Slide 20 text

Efficient development © 2022 Yahoo Japan Corporation Reusable common library Development required or optional? presence or absence? JSON tag extension with reflect For Example… JSON → Go struct Go struct → JSON Speed-up mock generation 1BDLBHF 1BDLBHF 1BDLBHF NPDL NPDL QBDLBHF QBDLBHF JOUFSGBDF JOUFSGBDF JOUFSGBDF HPHFOFSBUF HPHFOFSBUF Faster mock test Duplicate implementation in parallel development Execution time of test is slower

Slide 21

Slide 21 text

Devised points of the system renewal Preparation Testing Development © 2022 Yahoo Japan Corporation

Slide 22

Slide 22 text

Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?

Slide 23

Slide 23 text

Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?

Slide 24

Slide 24 text

Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock

Slide 25

Slide 25 text

Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock

Slide 26

Slide 26 text

Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock

Slide 27

Slide 27 text

Smoke Test Testing © 2022 Yahoo Japan Corporation Ad Server Smoke Kafka Automation on CI/CD 1. Register mock responses 2. Request to the Ad Server 3. Check Ad Server response, access logs and mock requests access logs HTTP / gRPC Mock Servers Does large-scale application behave expected I/F? → Automation check expected I/F with mock servers Efficient testing with mock Check Check Check

Slide 28

Slide 28 text

Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?

Slide 29

Slide 29 text

Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock

Slide 30

Slide 30 text

Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock

Slide 31

Slide 31 text

Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock

Slide 32

Slide 32 text

Functional Test Testing © 2022 Yahoo Japan Corporation Ad Server(new) Functional Test Tool 1. Prepare for test cases 2. Request to the Ad Server with tool 3. Check diff for Ad Server response and access log Backend servers (not mock server) Ad Server(old) result (old) result (new) Smoke test cannot cover whether the expected behavior with actual backend servers. → Semi-automation test with actual servers Efficient testing without mock Diff Check

Slide 33

Slide 33 text

Testing of the system renewal Testing © 2022 Yahoo Japan Corporation Functional Test Does application meet the functional requirements? Performance Test Does the new system perform worse than the old system? Smoke Test Does application behave expected I/F?

Slide 34

Slide 34 text

Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers

Slide 35

Slide 35 text

Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers

Slide 36

Slide 36 text

Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers

Slide 37

Slide 37 text

Performance Test Testing © 2022 Yahoo Japan Corporation Ad Server (New) perf job Automation on CI/CD Ad Server (Old) on production Prometheus Automation on CI/CD aggregate cron script Ad Server (Old) perf result viewer Prometheus Grafana Grafana HDFS HDFS 1. Aggregate production queries 2. Request to the Ad Servers and post perf results 3. Check performance results Does the new system perform worse than the old system? → Send a lot of queries to the old and new Ad Servers

Slide 38

Slide 38 text

Performance Test Testing - gRPC server communication latency is slow. → Improved by using connection pool. - Large memory usage of function that generate structures. (by pprof) → Improved by returning a pointer instead of a value copy. © 2022 Yahoo Japan Corporation Improved performance by making it easier to survey bottlenecks For Example…

Slide 39

Slide 39 text

Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test Please try to reference that this devised points.

Slide 40

Slide 40 text

Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to proceed with a large-scale system renewal project Please try to reference that this devised points.

Slide 41

Slide 41 text

Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to parallel development with increasing development members Please try to reference that this devised points.

Slide 42

Slide 42 text

Devised points of the system renewal © 2022 Yahoo Japan Corporation Tidy functions Code template Study session Share vision Preparation Development Testing Knowledge sharing Efficient development Smoke Test Functional Test Performance Test How to test the system requiring many functions and high performance Please try to reference that this devised points.

Slide 43

Slide 43 text

Summary Testing Migration to a CaaS environment Development Preparation Rewriting to Go from C++98 Integration of multiple components Integration and Modernization of the Ad Serving System Devised points of the system renewal Deployment time 85% improvement Average latency 10% improvement Results of the system renewal © 2022 Yahoo Japan Corporation

Slide 44

Slide 44 text

Thank you! © 2022 Yahoo Japan Corporation