LLM Agent - Part 2

In real-world applications such as recommendation systems, an important aspect of LLM-driven agents is their ability to take actions that adapt to users' preferences. In this talk, we will briefly introduce the basic concepts of reinforcement learning (RL) and two widely used policy optimization algorithms, PPO and DPO. Finally, through a demonstration of a recommendation agent, we will show how RL can enable agents to provide more user-adaptive responses.

Transcript

1. LLM Agent - Part 2 Couger AI
2. Previously ...
3. Reinforcement Learning (RL) • An optimal control concerned with how an intelligent agent ought to take actions in a dynamic environment in order to maximize the cumulative reward.
4. LLM and RL
5. LLM and RL • InstructGPT
6. Policy in RL • Policy is the mapping between observations and actions. • Policy directly affects the behaviors of your model. • The purpose of RL is to optimize the policy with rewards on “correct behaviors”.
7. Proximal Policy Optimization (PPO) • Find the best path to climb a mountain. Line Search Trust Region
8. And for Implementation, you only need to ....
9. Direct Preference Optimization (DPO) • Directly optimize the model over preference
10. RL and LLM Agent • RL is originally from agent system, to optimize action selection. • RL is perfect for building “customization” and “personalization”. • Use cases • Personalized tutorial (each student’s learning style) • Customer support (tones, feedback) • Game AI (diverse NPC behaviors) • Recommendation system (user preferences)
11. A recommendation sample • Use Amazon review dataset to construct the environment • Build recommendation agent with certain actions • Demonstrate the strength of such llm agents • RL to be implemented... • Causal to be implemented...
12. A recommendation sample
13. Identify shopping patterns • Categories: What kind of items is the user interested in? • Rating trends: Why does the user give a low rating? • Price: How does the user feel about price and quality trade-off? ...
15. Analysing user information • From the reviews left by the user, we can have some hints on • personality: practical? clear? mocking? • empathy: does the user by a gift? experienced loss? • attitude: straightforward? detail-oriented? • frequency: why does the user by certain items frequently?
17. Analysing recommendation features • Get a report of user preferences. • price • rating • categories • ...
18. Give recommendations • category based • give a reson • estimate the expected rating
19. Implementations • Example prompts for reasoning, planning and actions.