Evaluation of ranking model Petrie Wong

Who am I - Machine Learning Engineer for Search and GenAI at LegalOn Technologies - Our product: Help users review their contract and identify potential risks - My job: Help our users retrieve contracts or articles - Build better ranking models for legal documents

Process of building ranking model - Ideas (read papers) - Build a PoC model - Benchmark x 100 - On a set of static datasets - Offline evaluation x 5 - Summit queries on production dataset - Ask domain experts which model is better (baseline model or new model) - Implementation - Online evaluation - A/B testing - Multi-arm bandit, or - Reinforcement learning - …

Cost of evaluation - Benchmark - CICD or MLOps - Few minutes to 1 hour - Offline evaluation - Web interface or Spreadsheet - Few hours to days (even a week) - Online evaluation - A/B testing - Two weeks to a month - Multi-arm bandit, or - Few weeks for exploration - Reinforcement learning - Few weeks for warming up

Cost of evaluation - Benchmark - CICD or MLOps - Few minutes to 1 hour - Offline evaluation - Web interface or Spreadsheet - Few hours to days (even a week) - Online evaluation - A/B testing - Two weeks to a month - Multi-arm bandit, or - Few weeks for exploration - Reinforcement learning - Few weeks for warming up

Offline evaluation - For n queries - Top k results of model A - Top k results of model B - Your colleague: - Score of model A: 5 - Score of model B: 10 - Aggregate and calculate the win rate - Model A: 50% - Tie: 10% - Model B: 40% - Model B is the new model - Go back and change the parameters

Solution?

Ask your colleague work harder!

import openai

No content

LLM evaluation - Top k results of model A - Top k results of model B - LLM: - Score of model A: 5 - Score of model B: 10 - Calculate the win rate - Next step: - Integrate it into your MLOps - Repeat the evaluation 100 time with different hyper-parameters

Pay the bill

We're hiring! Join our Team! https://recruit.legalontech.jp/engineer/