Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
.NET Day 2025: Enhancing Legal Document Analysi...
Search
.NET Day
August 29, 2025
Technology
0
11
.NET Day 2025: Enhancing Legal Document Analysis with Reflection Agents, Semantic Kernel, and Azure AI Search
.NET Day
August 29, 2025
Tweet
Share
More Decks by .NET Day
See All by .NET Day
.NET Day 2025: How to Lie with AI: Understanding Bias, Ethics, and the Hidden Risks in Machine Learning
dotnetday
0
10
.NET Day 2025: Most Expensive Design Mistakes (Ever) and how to avoid them
dotnetday
0
11
.NET Day 2025: Turbocharged: Writing High-Performance C# and .NET Code
dotnetday
0
18
.NET Day 2025: Developing ASP.NET Core Microservices with Dapr: A practical guide
dotnetday
0
10
.NET Day 2025: Future-Proof Your Blazor Apps with bUnit
dotnetday
0
6
.NET Day 2025: .NET Core Testing: pushing the limits
dotnetday
0
10
.NET Day 2025: The best ways to use the latest OpenAPI features in .NET 9!
dotnetday
0
10
.NET Day 2025: Supercharged Search with Semantic Search and Vector Embeddings
dotnetday
0
7
.NET Day 2025: Tickets to Ride: Conquering Booking Chaos with Resilient .NET Architecture
dotnetday
0
12
Other Decks in Technology
See All in Technology
GopherCon Tour 概略
logica0419
2
120
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
20250929_QaaS_vol20
mura_shin
0
100
Why React!?? Next.jsそしてReactを改めてイチから選ぶ
ypresto
7
2.8k
日経が挑戦するデータ民主化 ~ セルフサービス基盤がもたらす利点と苦悩~/nikkei-tech-talk-37
nikkei_engineer_recruiting
0
200
非エンジニアのあなたもできる&もうやってる!コンテキストエンジニアリング
findy_eventslides
3
770
Beyond Multiprocessing: A Real-World ML Workload Speedup with Python 3.13+ Free-Threading
kitsuya0828
0
300
あなたのWebサービスはAIに自動テストしてもらえる?アクセシビリティツリーで読み解く、AIの『視点』
yusukeiwaki
1
3.1k
VCC 2025 Write-up
bata_24
0
100
DEFCON CHV CTF 2025 Write-up
bata_24
0
180
GC25 Recap+: Advancing Go Garbage Collection with Green Tea
logica0419
1
230
AI Agentと MCP Serverで実現する iOSアプリの 自動テスト作成の効率化
spiderplus_cb
0
180
Featured
See All Featured
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.1k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Embracing the Ebb and Flow
colly
87
4.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
BBQ
matthewcrist
89
9.8k
Done Done
chrislema
185
16k
Producing Creativity
orderedlist
PRO
347
40k
4 Signs Your Business is Dying
shpigford
185
22k
Become a Pro
speakerdeck
PRO
29
5.5k
How to Ace a Technical Interview
jacobian
280
23k
Transcript
ENHANCING LEGAL DOCUMENT ANALYSIS WITH REFLECTION AGENTS, SEMANTIC KERNEL, AND
AZURE AI SEARCH
CÉDRIC MENDELIN Software Developer - isolutions AG
AGENDA LLM Basics Customer Project - FTA The Microsoft Way
- Azure AI Services Step-by-Step Improvements - FTA Conclusion
LLM BASICS
None
None
PROMPT History Parameters (Top-P, Temperature) System Message (Persona) Input
What is the current date?
None
None
RAG – RETRIEVAL-AUGMENTED GENERATION Search Engine Data Model User Application
RETRIEVAL STEP Is the document relevant for the query? +
=
EMBEDDINGS Source: https://weaviate.io/blog/how-to-choose-an-embedding-model
VECTOR SEARCH VECTORIZE DATA VECTORIZE QUERY VECTOR SEARCH (COSINE SIMILARITY)
FEDERAL TAX ADMINISTRATION
Federal Tax Administration 4 k
LAWS AND ORDINANCES (XML)
COURT RULING – FTA PUBLICATIONS (PDF)
CHALLENGES Wording/Style of texts Citation requirement Hierarchy in data Amount
of data
AZURE AI SERVICES
AZURE AI SERVICES OpenAI Vision Speech Language Content Safety Face
Document Intelligence Azure AI services AI Search AI Agent Service AI Model Inference AI Foundry
None
SEMANTIC KERNEL • Open-Source SDK • Middleware • Abstraction over
different Models
None
RAG – Azure OpenAI on your data Azure OpenAI Azure
AI Search
None
None
None
What is the current date?
None
None
FUNCTION CALLING
STEP-BY-STEP IMPROVEMENTS FTA
1ST ITERATION
1ST ITERATION Focus on 50 documents Index Laws per article
Index PDFs per page Azure Open AI – on your data Vector Search
AZURE OPENAI – ON YOUR DATA
1ST ITERATION – FEEDBACK Relevant documents not found Hallucination Poor
Answer Quality Other reasons
2ND ITERATION
2ND ITERATION Hybrid Search Prompting Writing Profiles (Persona) Chat Settings
HYBRID SEARCH PIPELINE Vector Search Text Search N Search Results
Derive Vector Query Derive Text Query
HYBRID SEARCH CODE
2ND ITERATION - FEEDBACK Relevant documents not found Hallucination Poor
Answer Quality Other reasons
None
We need more than a gut feeling
LLM EVALUATION
EVALUATION TYPES LLM evaluation How good the foundation models performs
on a certain task. LLM system evaluation How good the LLM performs in your specific use case, on your data, in your domain.
EVALUATION PIPELINE LLM System Evaluation Dataset Evaluators Score Input Reference
Answer Expected Doc Answer
LLM SYSTEM EVALUATION - METRICS Reference-based Reference-free LLM based
MEAI.EVALUATION OVERVIEW • Open-source • Predefined LLM-based evaluators • Interface
for custom-evaluators • Local and Azure Storage Account • In Preview
None
MEAI.EVALUATION.CONSOLE
None
EVALUATION QUESTIONS Question Reference Answer Expected Doc(s) Category
None
EVALUATION RESULTS 0 10 20 30 40 50 60 70
80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search
EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.7 0.75
0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search
3RD ITERATION
3RD ITERATION – RETRIEVAL OPTIMIZATION AI Enrichment Semantic Reranking
AI ENRICHMENT
SEMANTIC RERANKING Vector Search Text Search N Search Results Derive
Vector Query Derive Text Query Reranking
SEMANTIC RERANKING
3RD ITERATION - FEEDBACK Retrieval improved Sometimes poor Answer Quality
Other reasons
EVALUATION RESULTS 0 10 20 30 40 50 60 70
80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.86 0.945
0.917 0.9 0.988 0.927 0.7 0.75 0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
4TH ITERATION
4TH ITERATION – ANSWER GENERATION OPTIMIZATION Reflection Agent
REFLECTION AGENT Writer Agent Critic Agent N Fact Checker Style
Checker Citation Checker
IMPLEMENTATION • Not supported by Azure OpenAI - On your
data • Derive Search Query • Using Azure AI Search SDK + Autogen
None
None
4TH ITERATION - FEEDBACK Other Reason
EVALUATION – EXECUTION TIME 2889 1999 2966 0 500 1000
1500 2000 2500 3000 3500 Azure OYOD - Hybrid Custom - Hybrid Multiagent (with Reranking) ms LLM system evaluation – Mean Execution Time
EVALUATION RESULTS 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1
Relevance Groundedness Cos Sim LLM System evaluation – MultiAgent Single Agent MultiAgent
LAST ITERATION
CONCLUSION
SURVEY RESULTS 86% OF ANSWERS RATED POSITIVELY 89% OF THE
USERS WANT TO USE THE SYSTEM PRODUCTIVELY
CONCLUSION • SK is your SDK of choice • Azure
AI Search for unstructured data • Use advanced capabilities • Start Evaluating early • What is your Use case • Business Value & Innovation The dotnet Stack is ready for productive AI Applications