Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
.NET Day 2025: Enhancing Legal Document Analysi...
Search
.NET Day
August 29, 2025
Technology
0
12
.NET Day 2025: Enhancing Legal Document Analysis with Reflection Agents, Semantic Kernel, and Azure AI Search
.NET Day
August 29, 2025
Tweet
Share
More Decks by .NET Day
See All by .NET Day
.NET Day 2025: How to Lie with AI: Understanding Bias, Ethics, and the Hidden Risks in Machine Learning
dotnetday
0
10
.NET Day 2025: Most Expensive Design Mistakes (Ever) and how to avoid them
dotnetday
0
12
.NET Day 2025: Turbocharged: Writing High-Performance C# and .NET Code
dotnetday
0
19
.NET Day 2025: Developing ASP.NET Core Microservices with Dapr: A practical guide
dotnetday
0
11
.NET Day 2025: Future-Proof Your Blazor Apps with bUnit
dotnetday
0
6
.NET Day 2025: .NET Core Testing: pushing the limits
dotnetday
0
16
.NET Day 2025: The best ways to use the latest OpenAPI features in .NET 9!
dotnetday
0
10
.NET Day 2025: Supercharged Search with Semantic Search and Vector Embeddings
dotnetday
0
7
.NET Day 2025: Tickets to Ride: Conquering Booking Chaos with Resilient .NET Architecture
dotnetday
0
17
Other Decks in Technology
See All in Technology
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
lehupa
0
190
Data Hubグループ 紹介資料
sansan33
PRO
0
2.2k
GoでもGUIアプリを作りたい!
kworkdev
PRO
0
150
初めてのDatabricks Apps開発
taka_aki
1
180
現場データから見える、開発生産性の変化コード生成AI導入・運用のリアル〜 / Changes in Development Productivity and Operational Challenges Following the Introduction of Code Generation AI
nttcom
0
310
難しいセキュリティ用語をわかりやすくしてみた
yuta3110
0
320
AWS Top Engineer、浮いてませんか? / As an AWS Top Engineer, Are You Out of Place?
yuj1osm
2
220
Wasmの気になる最新情報
askua
0
140
Copilot Studio ハンズオン - 生成オーケストレーションモード
tomoyasasakimskk
0
130
HR Force における DWH の併用事例 ~ サービス基盤としての BigQuery / 分析基盤としての Snowflake ~@Cross Data Platforms Meetup #2「BigQueryと愉快な仲間たち」
ryo_suzuki
0
240
プレーリーカードを活用しよう❗❗デジタル名刺交換からはじまるイベント会場交流のススメ
tsukaman
0
180
CoRL 2025 Survey
harukiabe
1
220
Featured
See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
GraphQLとの向き合い方2022年版
quramy
49
14k
A designer walks into a library…
pauljervisheath
209
24k
Building a Modern Day E-commerce SEO Strategy
aleyda
44
7.8k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
980
Leading Effective Engineering Teams in the AI Era
addyosmani
7
490
Building Adaptive Systems
keathley
44
2.8k
Producing Creativity
orderedlist
PRO
347
40k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
115
20k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
Transcript
ENHANCING LEGAL DOCUMENT ANALYSIS WITH REFLECTION AGENTS, SEMANTIC KERNEL, AND
AZURE AI SEARCH
CÉDRIC MENDELIN Software Developer - isolutions AG
AGENDA LLM Basics Customer Project - FTA The Microsoft Way
- Azure AI Services Step-by-Step Improvements - FTA Conclusion
LLM BASICS
None
None
PROMPT History Parameters (Top-P, Temperature) System Message (Persona) Input
What is the current date?
None
None
RAG – RETRIEVAL-AUGMENTED GENERATION Search Engine Data Model User Application
RETRIEVAL STEP Is the document relevant for the query? +
=
EMBEDDINGS Source: https://weaviate.io/blog/how-to-choose-an-embedding-model
VECTOR SEARCH VECTORIZE DATA VECTORIZE QUERY VECTOR SEARCH (COSINE SIMILARITY)
FEDERAL TAX ADMINISTRATION
Federal Tax Administration 4 k
LAWS AND ORDINANCES (XML)
COURT RULING – FTA PUBLICATIONS (PDF)
CHALLENGES Wording/Style of texts Citation requirement Hierarchy in data Amount
of data
AZURE AI SERVICES
AZURE AI SERVICES OpenAI Vision Speech Language Content Safety Face
Document Intelligence Azure AI services AI Search AI Agent Service AI Model Inference AI Foundry
None
SEMANTIC KERNEL • Open-Source SDK • Middleware • Abstraction over
different Models
None
RAG – Azure OpenAI on your data Azure OpenAI Azure
AI Search
None
None
None
What is the current date?
None
None
FUNCTION CALLING
STEP-BY-STEP IMPROVEMENTS FTA
1ST ITERATION
1ST ITERATION Focus on 50 documents Index Laws per article
Index PDFs per page Azure Open AI – on your data Vector Search
AZURE OPENAI – ON YOUR DATA
1ST ITERATION – FEEDBACK Relevant documents not found Hallucination Poor
Answer Quality Other reasons
2ND ITERATION
2ND ITERATION Hybrid Search Prompting Writing Profiles (Persona) Chat Settings
HYBRID SEARCH PIPELINE Vector Search Text Search N Search Results
Derive Vector Query Derive Text Query
HYBRID SEARCH CODE
2ND ITERATION - FEEDBACK Relevant documents not found Hallucination Poor
Answer Quality Other reasons
None
We need more than a gut feeling
LLM EVALUATION
EVALUATION TYPES LLM evaluation How good the foundation models performs
on a certain task. LLM system evaluation How good the LLM performs in your specific use case, on your data, in your domain.
EVALUATION PIPELINE LLM System Evaluation Dataset Evaluators Score Input Reference
Answer Expected Doc Answer
LLM SYSTEM EVALUATION - METRICS Reference-based Reference-free LLM based
MEAI.EVALUATION OVERVIEW • Open-source • Predefined LLM-based evaluators • Interface
for custom-evaluators • Local and Azure Storage Account • In Preview
None
MEAI.EVALUATION.CONSOLE
None
EVALUATION QUESTIONS Question Reference Answer Expected Doc(s) Category
None
EVALUATION RESULTS 0 10 20 30 40 50 60 70
80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search
EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.7 0.75
0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search
3RD ITERATION
3RD ITERATION – RETRIEVAL OPTIMIZATION AI Enrichment Semantic Reranking
AI ENRICHMENT
SEMANTIC RERANKING Vector Search Text Search N Search Results Derive
Vector Query Derive Text Query Reranking
SEMANTIC RERANKING
3RD ITERATION - FEEDBACK Retrieval improved Sometimes poor Answer Quality
Other reasons
EVALUATION RESULTS 0 10 20 30 40 50 60 70
80 90 100 Provided source Applied source Retrieval Step - % of documents provided and applied Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
EVALUATION RESULTS 0.8 0.948 0.914 0.85 0.948 0.917 0.86 0.945
0.917 0.9 0.988 0.927 0.7 0.75 0.8 0.85 0.9 0.95 1 Relevance Groundedness Cosin Sim Answer Generation – Quality Metrics Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking
4TH ITERATION
4TH ITERATION – ANSWER GENERATION OPTIMIZATION Reflection Agent
REFLECTION AGENT Writer Agent Critic Agent N Fact Checker Style
Checker Citation Checker
IMPLEMENTATION • Not supported by Azure OpenAI - On your
data • Derive Search Query • Using Azure AI Search SDK + Autogen
None
None
4TH ITERATION - FEEDBACK Other Reason
EVALUATION – EXECUTION TIME 2889 1999 2966 0 500 1000
1500 2000 2500 3000 3500 Azure OYOD - Hybrid Custom - Hybrid Multiagent (with Reranking) ms LLM system evaluation – Mean Execution Time
EVALUATION RESULTS 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1
Relevance Groundedness Cos Sim LLM System evaluation – MultiAgent Single Agent MultiAgent
LAST ITERATION
CONCLUSION
SURVEY RESULTS 86% OF ANSWERS RATED POSITIVELY 89% OF THE
USERS WANT TO USE THE SYSTEM PRODUCTIVELY
CONCLUSION • SK is your SDK of choice • Azure
AI Search for unstructured data • Use advanced capabilities • Start Evaluating early • What is your Use case • Business Value & Innovation The dotnet Stack is ready for productive AI Applications