Speaker introduc/on Product manager working to improve ad quality Joined Yahoo! JAPAN in 2018. Leading the system planning and business design aspects to solve the problem of ad review, which used to be done by manpower. In my private time, I enjoys car driving and playing poker. Yuki Kai
Agenda - About Yahoo! JAPAN Ads and ad review - Yahoo! JAPAN Ads review a few years ago - What is CreativeTester? - Barriers to AI implementation - People and AI work together
About Yahoo! JAPAN Ads and ad review Attracting highly interested prospects Search Ads show your ad on the search result page. Reach people who search "now“
About Yahoo! JAPAN Ads and ad review Branding using diverse and rich advertising expressions It delivers high visible image and video ads on Yahoo! JAPAN's homepage and our pages, including Yahoo! JAPAN News. Strongly promote products and services to users with significant exposure
About Yahoo! JAPAN Ads and ad review Approaching potential customers You can reach out to potential customers who browse the content pages of Yahoo! JAPAN and its partner sites and who may be interested in your products and services
About Yahoo! JAPAN Ads and ad review Ad review is based on three guidelines Ad Insertion Guidelines Ad Delivery Guidelines Ad Traffic Quality Guidelines
About Yahoo! JAPAN Ads and ad review Ad review is available at all times AI and System Review Manual Review Before submission At submission Before publication During publication After publication is complete
Yahoo! JAPAN Ads review a few years ago In four years, the number of reviews has increased 2.6 times 0 500 1000 1500 2000 2500 3000 3500 4000 4500 2018 2022
Yahoo! JAPAN Ads review a few years ago Several system reviews and manual review are conducted Regular expression Ads Title Description Landing Page (Web Crawl) KW Approve Manual Review Image/Video
Yahoo! JAPAN Ads review a few years ago Over millions of instances of phrase maintenance were performed weekly Regular expression Result Data Phrase editing No phrase change Phrase delete
Yahoo! JAPAN Ads review a few years ago Ads have increased and manual review had reached its limit Regular expression Approve Manual Review Ad Delivery Post-exam Increase Decrease
Yahoo! JAPAN Ads review a few years ago The limitation of manual reviews was not good for users and advertisers Users see ads that violate guidelines Unsubscribe after ad serving
What is Crea/veTester? Scalable review system that does not become a business stopper Testing to improve accuracy Loosely coupled configuration Ability to handle sudden increase in ads High-maintenance environment Judgment by similarity Ad review by machine-learning
CreativeTester What is Crea/veTester? Converting Ads to a system-judgable format Ads Title/Description Landing Page (Web Crawl) KW Image Video Image Analysis &OCR Voice Analysis OCR regular expression ML Review
CreativeTester What is Crea/veTester? Determine the target of manual review from multiple ML model results Regular expression [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester Ads Tag + Score Tag + Score Tag + Score
What is Crea/veTester? Promoted MLOps and streamlined relearning, evaluation, and deployment ML Model Evaluation Data Re-learning Evaluation Deploy Comparison of old and new Win Lose
What is Crea/veTester? Need to verify which combination of models will produce the best results B Model A Model C Model 60% correct 75% correct How do A and B combine? How do A and C combine? How do A and B and C combine?
What is Crea/veTester? Need to easily verify optimal combination B Model A Model C Model α Combination β Combination Γ Combination A Model B Model C Model D Model Best solution
What is Crea/veTester? Environment where non-engineers can improve accuracy Evaluation Data Manual Rule A Model Regular expression α Combination β Combination Γ Combination Judges
What is Crea/veTester? “CreativeTesterPlayGround” system for continuous improvement of accuracy Evaluation Data Judges New Tester Deploy Re-create Current Data
What is Crea/veTester? Selection of ads to be manual reviewed Obviously high risk Obviously low risk AI determines AI determines Manual review enlargement enlargement
Barriers to AI implementa/on AI does not take jobs, but people teach AI to do jobs The judges really told me in 2018. Multiple trips were made to Aomori and Fukuoka to explain the design concept. Face-to-face contact is important.
Barriers to AI implementa/on What if the AI makes a bad decision? There was strong concern from service and sales personnel. ・Set accuracy targets for release decisions. ・Set up a flow for error determination.
Barriers to AI implementa/on Many ads still need to be manually reviewed to make a decision Examples of easily determinable offending ads Drinking only diet Apply and remove stains It is difficult to replace AI with ad screening, which is difficult even for a human to judge.
People and AI work together It is important to develop an environment where correct data can be gathered New product New Words Trendy Items Legal Interpretation Past Review Results Review Criteria
People and AI work together Contributing to the evolving advertising market through “CreativeTester” We want to deliver ads that are useful to someone, not ads that are disliked. To achieve this, we need to continue to improve the quality of our advertisements.
techverse_2022
toshi0383
smasato
lwug
otanet
takasehideki
pospome
ufried
yokawasa
kddisnowvillage
akiroom
peisuke
mosa_siru
sachag
eileencodes
jennybc
ufuk
maltzj
chriscoyier
iamctodd
productmarketing
gr2m
addyosmani
soudai