Why Doesn’t AI Reduce the Workload as Much as Expected? : The Limits of Human-in-the-Loop and AI-Ready Workflow Design

Although AI was supposed to make the work easier, the workload has not decreased as much as expected.

In this session, we will share our work on an AI workflow that combines classification models and LLMs, using product patrol operations for Yahoo! JAPAN Auction and Yahoo! JAPAN Flea Market as a case study. We will also discuss the limits of Human-in-the-loop that became clear through this initiative.

By using classification models to narrow down review targets and LLMs to provide reasoning, AI-assisted review has improved in both accuracy and explainability. At the same time, we found that as long as the workflow continues to rely on human confirmation, there remains a gap between improving quality and actually reducing operational workload.

Instead of simply applying AI to existing operations as they are, we have been rethinking workflows, rules, and operations with AI as a premise. We will share our hands-on trial and error around designing rules that are easier for AI to judge and easier for operations teams to use, defining the roles of classification models and LLMs, and building workflows that can be improved through repeated testing in the field.

What does it take to move AI-driven process improvement beyond PoC and make it work in real operations? This session will offer practical perspectives for engineers, as well as those involved in operations, PM, planning, CS, and business process improvement, on how to make AI useful in day-to-day work.

Transcript

1. 1 2026.06.29 LY Corporation Tomohiro Nishimura | LY Corporation Kanako Onishi | LY Corporation Why Doesnʼt AI Reduce the Workload as Much as Expected? The Limits of Human-in-the-Loop and AI-Ready Workflow Design
2. Speakers Tomohiro Nishimura Security Platform Content Abuse Prevention Product Owner, AutoLearning No-code AutoML and AI workflow tooling Kanako Onishi CX Promotion Monitoring Planning & Development CS AI Transformation Driving AI adoption for CS workflow automation Co-creating AI solutions across engineering and CS automation
3. Agenda Session Roadmap Part 1 Improving Work with AI • About Listing Patrol • How AI Changed Listing Patrol • The Limits of Human-in-the-Loop Part 2 Redesigning Work for AI • AI-Only decision domains • AI-Ready Workflows and Prompts • New Human Roles
4. Listing Patrol Listing patrol has two hard jobs: Find risky listings, and Judge guideline violations. Find  Millions of listings / day  Risk patterns keep changing + Judge  300+ review criteria  Context-dependent decisions
5. Before AI: Query-Based Patrol Search Queries AI Filtering AI Pre-Review All Listings 1000+ Search Queries Human Review Search logic and review focus depended on reviewer expertise.
6. AI Filtering: Risk Scoring + Review Focus Search Queries AI Filtering AI Pre-Review All Listings Classifier Filters by Score Suggests Top 3 Risks Human Review AI improved Find and Focus. But explaining violations still relied on reviewers.
7. AI Pre-Review: Making Risk Easier to Review Search Queries AI Pre-Review AI Filtering Classifier Narrows the Target AI Pre-Review Explains Why and Where to Review Human Review All Listings
8. What Improved ̶ and What Didnʼt Review Efficiency Index Human Review Time Technical efficiency improved dramatically. But operational workload did not drop enough. 1× 50 × 26× Search Queries AI Filtering AI Pre-Review Search Queries AI Filtering AI Pre-Review Almost unchanged
9. Why Workload Didnʼt Drop Enough Find Less Noise Harder Cases AI filtered out easy cases. The rest became harder. Judge Clearer Reasons Same Decision Structure AI clarified the reasons. Final decisions still went through human review. The bottleneck was not AI quality. It was the Human-in-the-Loop structure.
10. Redesign Workflows Around AI, Not Add AI to Existing Processes Add AI tools to existing workflows AI-native workflow redesign
11. Clearly define areas that can be delegated to AI Expand AI-only decision domains Expand AI-Only Decision Domains Human-only AI-only Human-in-the-loop Human-in-the-loop Human-only • AI-ready workflows maximize AI-only domains. • Expanding AI-only domains requires redefining ownership.
12. • The ceiling of efficiency gains • Comparison of AI and human decision quality • Detailed review of workflow documentation • Issue identification Visualize the limitations of human intervention Evidence that current workflows are not AI-ready Toward AI-Native Workflow Redesign
13. Why Are Human-Oriented Workflows Not AI-Ready? Instructions written for humans Undocumented operational rules Human: People can resolve uncertainties through communication Human: Experience compensates for ambiguity Context that humans infer implicitly must be documented and structured for AI.
14. "Unrelated" lacks an explicit definition, making consistent decision-making difficult. Rules that evolved through operational practice remain undocumented. Overly complex processes negatively impact operations. Ambiguous descriptions Decision process complexity Tacit knowledge and local rules Issues in Human-Oriented Work Instructions
15. Clear objectives and goals Explicit decision criteria Structured information Defined exceptions and boundaries Simple workflow design 1 3 2 4 5 What Is AI-Ready Workflow Design?
16. Policies Policy-specific guidance Determining What Information AI Needs and How to Provide It  Instructions written for humans Extract only the information required by AI
17. Policies Policy-specific guidance Maintainable Prompt for Long-Term Use  Common rules Specific rules  Even when multiple people contribute, the final prompt must be coherent and consistent.
18. From Product Monitoring to AI Operations Before Humans monitor products AI supports humans After Humans operate and improve AI AI monitors products People can shift their focus to higher-value work that only humans can do.
19. Summary • Human-assisted AI has efficiency limits • Create AI-only decision domains • Define clear decision ownership for AI • Shift human roles toward operating and improving AI Unlock AIʼs full potential through workflow redesign