$30 off During Our Annual Pro Sale. View Details »

About the Creative Tester, an AI Examination Service for the Yahoo! JAPAN Advertising

About the Creative Tester, an AI Examination Service for the Yahoo! JAPAN Advertising

Yuki Kai (Yahoo! JAPAN / Trust & Safety Division, Marketing Platform Group, MS Group / Product Manager)

https://tech-verse.me/ja/sessions/157
https://tech-verse.me/en/sessions/157
https://tech-verse.me/ko/sessions/157

Tech-Verse2022
PRO

November 17, 2022
Tweet

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. About the Creative Tester, an AI Examination Service for the

    Yahoo! JAPAN Advertising Yuki Kai / Yahoo! JAPAN
  2. Speaker introduc/on Product manager working to improve ad quality Joined

    Yahoo! JAPAN in 2018. Leading the system planning and business design aspects to solve the problem of ad review, which used to be done by manpower. In my private time, I enjoys car driving and playing poker. Yuki Kai
  3. Business innovation takes place when people and AI work together.

  4. Agenda - About Yahoo! JAPAN Ads and ad review -

    Yahoo! JAPAN Ads review a few years ago - What is CreativeTester? - Barriers to AI implementation - People and AI work together
  5. About Yahoo! JAPAN Ads and ad review Delivering advertisements in

    Japan's leading media Search Ads Display Ads (Guaranteed) Display Ads (Auction)
  6. About Yahoo! JAPAN Ads and ad review Attracting highly interested

    prospects Search Ads show your ad on the search result page. Reach people who search "now“
  7. About Yahoo! JAPAN Ads and ad review Branding using diverse

    and rich advertising expressions It delivers high visible image and video ads on Yahoo! JAPAN's homepage and our pages, including Yahoo! JAPAN News. Strongly promote products and services to users with significant exposure
  8. About Yahoo! JAPAN Ads and ad review Approaching potential customers

    You can reach out to potential customers who browse the content pages of Yahoo! JAPAN and its partner sites and who may be interested in your products and services
  9. About Yahoo! JAPAN Ads and ad review Ad review is

    based on three guidelines Ad Insertion Guidelines Ad Delivery Guidelines Ad Traffic Quality Guidelines
  10. About Yahoo! JAPAN Ads and ad review For example, we

    reject such offending ads Drinking only diet Apply and remove stains
  11. About Yahoo! JAPAN Ads and ad review Ad review is

    available at all times AI and System Review Manual Review Before submission At submission Before publication During publication After publication is complete
  12. About Yahoo! JAPAN Ads and ad review Ad review operations

    are conducted by hundreds of people AOMORI TOKYO FUKUOKA
  13. 40,000,000

  14. Yahoo! JAPAN Ads review a few years ago 40 million

    ads are reviewed every day 40,000,000/day
  15. Yahoo! JAPAN Ads review a few years ago In four

    years, the number of reviews has increased 2.6 times 0 500 1000 1500 2000 2500 3000 3500 4000 4500 2018 2022
  16. Yahoo! JAPAN Ads review a few years ago Several system

    reviews and manual review are conducted Regular expression Ads Title Description Landing Page (Web Crawl) KW Approve Manual Review Image/Video
  17. Yahoo! JAPAN Ads review a few years ago Over millions

    of instances of phrase maintenance were performed weekly Regular expression Result Data Phrase editing No phrase change Phrase delete
  18. Yahoo! JAPAN Ads review a few years ago Ads have

    increased and manual review had reached its limit Regular expression Approve Manual Review Ad Delivery Post-exam Increase Decrease
  19. Yahoo! JAPAN Ads review a few years ago The limitation

    of manual reviews was not good for users and advertisers Users see ads that violate guidelines Unsubscribe after ad serving
  20. Limitations of ad review should not have created business limitations

  21. Yahoo! JAPAN Ads review a few years ago We needed

    a system to break out of this situation
  22. Creative Tester

  23. What is Crea/veTester? Scalable review system that does not become

    a business stopper Testing to improve accuracy Loosely coupled configuration Ability to handle sudden increase in ads High-maintenance environment Judgment by similarity Ad review by machine-learning
  24. What is Crea/veTester? To solve the problem, we built Ad

    review system on Google Cloud
  25. CreativeTester What is Crea/veTester? Converting Ads to a system-judgable format

    Ads Title/Description Landing Page (Web Crawl) KW Image Video Image Analysis &OCR Voice Analysis OCR regular expression ML Review
  26. CreativeTester What is Crea/veTester? Determine the target of manual review

    from multiple ML model results Regular expression [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester [ML] xxx Tester Ads Tag + Score Tag + Score Tag + Score
  27. What is Crea/veTester? Promoted MLOps and streamlined relearning, evaluation, and

    deployment ML Model Evaluation Data Re-learning Evaluation Deploy Comparison of old and new Win Lose
  28. What is Crea/veTester? Need to verify which combination of models

    will produce the best results B Model A Model C Model 60% correct 75% correct How do A and B combine? How do A and C combine? How do A and B and C combine?
  29. What is Crea/veTester? Need to easily verify optimal combination B

    Model A Model C Model α Combination β Combination Γ Combination A Model B Model C Model D Model Best solution
  30. What is Crea/veTester? Environment where non-engineers can improve accuracy Evaluation

    Data Manual Rule A Model Regular expression α Combination β Combination Γ Combination Judges
  31. What is Crea/veTester? “CreativeTesterPlayGround” system for continuous improvement of accuracy

    Evaluation Data Judges New Tester Deploy Re-create Current Data
  32. What is Crea/veTester? Selection of ads to be manual reviewed

    Obviously high risk Obviously low risk AI determines AI determines Manual review enlargement enlargement
  33. We want to shift human resources to more creative work

  34. Will AI take my job?

  35. Barriers to AI implementa/on AI does not take jobs, but

    people teach AI to do jobs The judges really told me in 2018. Multiple trips were made to Aomori and Fukuoka to explain the design concept. Face-to-face contact is important.
  36. Barriers to AI implementa/on What if the AI makes a

    bad decision? There was strong concern from service and sales personnel. ・Set accuracy targets for release decisions. ・Set up a flow for error determination.
  37. Barriers to AI implementa/on Many ads still need to be

    manually reviewed to make a decision Examples of easily determinable offending ads Drinking only diet Apply and remove stains It is difficult to replace AI with ad screening, which is difficult even for a human to judge.
  38. People and AI work together It is important to develop

    an environment where correct data can be gathered New product New Words Trendy Items Legal Interpretation Past Review Results Review Criteria
  39. Business innovation takes place when people and AI work together.

  40. People and AI work together Contributing to the evolving advertising

    market through “CreativeTester” We want to deliver ads that are useful to someone, not ads that are disliked. To achieve this, we need to continue to improve the quality of our advertisements.
  41. Thank you