Predictive Coding (DataPhilly)

E-discovery and predictive coding. Presented at DataPhilly on December 16, 2014.

Transcript

1. Predictive Coding Bill Dimm DataPhilly December 16, 2014
2. PredictiveCodingBook.com
3. What is E-Discovery? • Organization is required to produce any documents it has relevant to some matter – Litigation – Mergers
4. The Problem • 1 TB = 1.8 million documents – 1.1 million emails – 460,000 scanned documents – 230,000 loose files • $50 for hard drive to store • $10 million to review • Cheap storage → junk & duplicates
5. Document Deletion • Document Retention Policy – Policy of routinely deleting old documents is OK if done in good faith • Litigation Hold – Documents must be preserved if litigation is anticipated
6. Technological Solutions • Keyword Search • Unsupervised Machine Learning – Organizing / analyzing without knowing what you are looking for – Clustering • Supervised Machine Learning – Learning by example – “Predictive Coding”
7. Keyword Search • Study by Blair and Maron in 1985 – 2 lawyers with paralegals and search engine – Aim for at least 75% recall – Got 75% recall once out of 40 tasks – Averaged 20% recall • Queries often fail to recognize all of the ways to say the same thing
8. Keyword Search Sorting • Strategy: Very broad query + hope relevant docs tend to be at top of sort • Term frequency-inverse document frequency – Sorting is based on a heuristic rule – Not specific to the problem at hand
9. Keyword Search Sorting
10. Keyword Search Sorting
11. Relevance Score Sorting
12. Precision & Recall • Recall – % of relevant docs found – High recall required for defensibility • Precision – % of docs predicted to be relevant that actually are – High precision desired to reduce cost
13. Training – Word Exposure
14. Training – Feature Prevalence
15. Training – Feature Prevalence
16. Training – The Big Debate • Random Sampling • Judgmental Sampling / Active Learning
17. PredictiveCodingBook.com