About the Creative Tester, an AI Examination Service for the Yahoo! JAPAN Advertising

Yuki Kai (Yahoo! JAPAN / Trust & Safety Division, Marketing Platform Group, MS Group / Product Manager)

https://tech-verse.me/ja/sessions/157
https://tech-verse.me/en/sessions/157
https://tech-verse.me/ko/sessions/157

Transcript

1. About the Creative Tester, an AI Examination Service for the Yahoo! JAPAN Advertising Yuki Kai / Yahoo! JAPAN
2. Speaker introduc/on Product manager working to improve ad quality Joined Yahoo! JAPAN in 2018. Leading the system planning and business design aspects to solve the problem of ad review, which used to be done by manpower. In my private time, I enjoys car driving and playing poker. Yuki Kai
3. Business innovation takes place when people and AI work together.
4. Agenda - About Yahoo! JAPAN Ads and ad review - Yahoo! JAPAN Ads review a few years ago - What is CreativeTester? - Barriers to AI implementation - People and AI work together
5. About Yahoo! JAPAN Ads and ad review Delivering advertisements in Japan's leading media Search Ads Display Ads (Guaranteed) Display Ads (Auction)
6. About Yahoo! JAPAN Ads and ad review Attracting highly interested prospects Search Ads show your ad on the search result page. Reach people who search "now“
7. About Yahoo! JAPAN Ads and ad review Branding using diverse and rich advertising expressions It delivers high visible image and video ads on Yahoo! JAPAN's homepage and our pages, including Yahoo! JAPAN News. Strongly promote products and services to users with significant exposure
8. About Yahoo! JAPAN Ads and ad review Approaching potential customers You can reach out to potential customers who browse the content pages of Yahoo! JAPAN and its partner sites and who may be interested in your products and services
9. About Yahoo! JAPAN Ads and ad review Ad review is based on three guidelines Ad Insertion Guidelines Ad Delivery Guidelines Ad Traffic Quality Guidelines
10. About Yahoo! JAPAN Ads and ad review For example, we reject such offending ads Drinking only diet Apply and remove stains
11. About Yahoo! JAPAN Ads and ad review Ad review is available at all times AI and System Review Manual Review Before submission At submission Before publication During publication After publication is complete
12. About Yahoo! JAPAN Ads and ad review Ad review operations are conducted by hundreds of people AOMORI TOKYO FUKUOKA
13. 40,000,000
14. Yahoo! JAPAN Ads review a few years ago 40 million ads are reviewed every day 40,000,000/day
15. Yahoo! JAPAN Ads review a few years ago In four years, the number of reviews has increased 2.6 times 0 500 1000 1500 2000 2500 3000 3500 4000 4500 2018 2022
16. Yahoo! JAPAN Ads review a few years ago Several system reviews and manual review are conducted Regular expression Ads Title Description Landing Page (Web Crawl) KW Approve Manual Review Image/Video
17. Yahoo! JAPAN Ads review a few years ago Over millions of instances of phrase maintenance were performed weekly Regular expression Result Data Phrase editing No phrase change Phrase delete
18. Yahoo! JAPAN Ads review a few years ago Ads have increased and manual review had reached its limit Regular expression Approve Manual Review Ad Delivery Post-exam Increase Decrease
19. Yahoo! JAPAN Ads review a few years ago The limitation of manual reviews was not good for users and advertisers Users see ads that violate guidelines Unsubscribe after ad serving
20. Limitations of ad review should not have created business limitations
21. Yahoo! JAPAN Ads review a few years ago We needed a system to break out of this situation
22. Creative Tester
23. What is Crea/veTester? Scalable review system that does not become a business stopper Testing to improve accuracy Loosely coupled configuration Ability to handle sudden increase in ads High-maintenance environment Judgment by similarity Ad review by machine-learning
24. What is Crea/veTester? To solve the problem, we built Ad review system on Google Cloud
25. CreativeTester What is Crea/veTester? Converting Ads to a system-judgable format Ads Title/Description Landing Page (Web Crawl) KW Image Video Image Analysis &OCR Voice Analysis OCR regular expression ML Review
26. CreativeTester What is Crea/veTester? Determine the target of manual review from multiple ML model results Regular expression [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester Ads Tag + Score Tag + Score Tag + Score
27. What is Crea/veTester? Promoted MLOps and streamlined relearning, evaluation, and deployment ML Model Evaluation Data Re-learning Evaluation Deploy Comparison of old and new Win Lose
28. What is Crea/veTester? Need to verify which combination of models will produce the best results B Model A Model C Model 60% correct 75% correct How do A and B combine? How do A and C combine? How do A and B and C combine?
29. What is Crea/veTester? Need to easily verify optimal combination B Model A Model C Model α Combination β Combination Γ Combination A Model B Model C Model D Model Best solution
30. What is Crea/veTester? Environment where non-engineers can improve accuracy Evaluation Data Manual Rule A Model Regular expression α Combination β Combination Γ Combination Judges
31. What is Crea/veTester? “CreativeTesterPlayGround” system for continuous improvement of accuracy Evaluation Data Judges New Tester Deploy Re-create Current Data
32. What is Crea/veTester? Selection of ads to be manual reviewed Obviously high risk Obviously low risk AI determines AI determines Manual review enlargement enlargement
33. We want to shift human resources to more creative work
34. Will AI take my job?
35. Barriers to AI implementa/on AI does not take jobs, but people teach AI to do jobs The judges really told me in 2018. Multiple trips were made to Aomori and Fukuoka to explain the design concept. Face-to-face contact is important.
36. Barriers to AI implementa/on What if the AI makes a bad decision? There was strong concern from service and sales personnel. ・Set accuracy targets for release decisions. ・Set up a flow for error determination.
37. Barriers to AI implementa/on Many ads still need to be manually reviewed to make a decision Examples of easily determinable offending ads Drinking only diet Apply and remove stains It is difficult to replace AI with ad screening, which is difficult even for a human to judge.
38. People and AI work together It is important to develop an environment where correct data can be gathered New product New Words Trendy Items Legal Interpretation Past Review Results Review Criteria
39. Business innovation takes place when people and AI work together.
40. People and AI work together Contributing to the evolving advertising market through “CreativeTester” We want to deliver ads that are useful to someone, not ads that are disliked. To achieve this, we need to continue to improve the quality of our advertisements.
41. Thank you