.NET Day 2025: Enhancing Legal Document Analysis with Reflection Agents, Semantic Kernel, and Azure AI Search

Transcript

1. ENHANCING LEGAL DOCUMENT ANALYSIS WITH REFLECTION AGENTS, SEMANTIC KERNEL, AND AZURE AI SEARCH
2. CÉDRIC MENDELIN Software Developer - isolutions AG
3. AGENDA LLM Basics Customer Project - FTA The Microsoft Way - Azure AI Services Step-by-Step Improvements - FTA Conclusion
4. LLM BASICS
7. PROMPT History Parameters (Top-P, Temperature) System Message (Persona) Input
8. What is the current date?
11. RAG – RETRIEVAL-AUGMENTED GENERATION Search Engine Data Model User Application
12. RETRIEVAL STEP Is the document relevant for the query? + =
13. EMBEDDINGS Source: https://weaviate.io/blog/how-to-choose-an-embedding-model
14. VECTOR SEARCH VECTORIZE DATA VECTORIZE QUERY VECTOR SEARCH (COSINE SIMILARITY)
15. FEDERAL TAX ADMINISTRATION
16. Federal Tax Administration 4 k
17. LAWS AND ORDINANCES (XML)
18. COURT RULING – FTA PUBLICATIONS (PDF)
19. CHALLENGES Wording/Style of texts Citation requirement Hierarchy in data Amount of data
20. AZURE AI SERVICES
21. AZURE AI SERVICES OpenAI Vision Speech Language Content Safety Face Document Intelligence Azure AI services AI Search AI Agent Service AI Model Inference AI Foundry
23. SEMANTIC KERNEL • Open-Source SDK • Middleware • Abstraction over different Models
25. RAG – Azure OpenAI on your data Azure OpenAI Azure AI Search
29. What is the current date?
32. FUNCTION CALLING
33. STEP-BY-STEP IMPROVEMENTS FTA
34. 1ST ITERATION
35. 1ST ITERATION Focus on 50 documents Index Laws per article Index PDFs per page Azure Open AI – on your data Vector Search
36. AZURE OPENAI – ON YOUR DATA
37. 1ST ITERATION – FEEDBACK Relevant documents not found Hallucination Poor Answer Quality Other reasons
38. 2ND ITERATION
39. 2ND ITERATION Hybrid Search Prompting Writing Profiles (Persona) Chat Settings
40. HYBRID SEARCH PIPELINE Vector Search Text Search N Search Results Derive Vector Query Derive Text Query
41. HYBRID SEARCH CODE
42. 2ND ITERATION - FEEDBACK Relevant documents not found Hallucination Poor Answer Quality Other reasons
44. We need more than a gut feeling
45. LLM EVALUATION
46. EVALUATION TYPES LLM evaluation How good the foundation models performs on a certain task. LLM system evaluation How good the LLM performs in your specific use case, on your data, in your domain.
47. EVALUATION PIPELINE LLM System Evaluation Dataset Evaluators Score Input Reference Answer Expected Doc Answer
48. LLM SYSTEM EVALUATION - METRICS Reference-based Reference-free LLM based
49. MEAI.EVALUATION OVERVIEW • Open-source • Predefined LLM-based evaluators • Interface for custom-evaluators • Local and Azure Storage Account • In Preview
51. MEAI.EVALUATION.CONSOLE
53. EVALUATION QUESTIONS Question Reference Answer Expected Doc(s) Category
55. EVALUATION RESULTS 0 10 20 30 40 50 60 70 80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search
56. EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.7 0.75 0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search
57. 3RD ITERATION
58. 3RD ITERATION – RETRIEVAL OPTIMIZATION AI Enrichment Semantic Reranking
59. AI ENRICHMENT
60. SEMANTIC RERANKING Vector Search Text Search N Search Results Derive Vector Query Derive Text Query Reranking
61. SEMANTIC RERANKING
62. 3RD ITERATION - FEEDBACK Retrieval improved Sometimes poor Answer Quality Other reasons
63. EVALUATION RESULTS 0 10 20 30 40 50 60 70 80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
64. EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.86 0.945 0.917 0.9 0.988 0.927 0.7 0.75 0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
65. 4TH ITERATION
66. 4TH ITERATION – ANSWER GENERATION OPTIMIZATION Reflection Agent
67. REFLECTION AGENT Writer Agent Critic Agent N Fact Checker Style Checker Citation Checker
68. IMPLEMENTATION • Not supported by Azure OpenAI - On your data • Derive Search Query • Using Azure AI Search SDK + Autogen
71. 4TH ITERATION - FEEDBACK Other Reason
72. EVALUATION – EXECUTION TIME 2889 1999 2966 0 500 1000 1500 2000 2500 3000 3500 Azure OYOD - Hybrid Custom - Hybrid Multiagent (with Reranking) ms LLM system evaluation – Mean Execution Time
73. EVALUATION RESULTS 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1 Relevance Groundedness Cos Sim LLM System evaluation – MultiAgent Single Agent MultiAgent
74. LAST ITERATION
75. CONCLUSION
76. SURVEY RESULTS 86% OF ANSWERS RATED POSITIVELY 89% OF THE USERS WANT TO USE THE SYSTEM PRODUCTIVELY
77. CONCLUSION • SK is your SDK of choice • Azure AI Search for unstructured data • Use advanced capabilities • Start Evaluating early • What is your Use case • Business Value & Innovation The dotnet Stack is ready for productive AI Applications