Automating recommendation

Replacing a hand-built decision tree with a robust machine learner for a website backend

Transcript

1. Joshua Dempster Automating Recommendation consulting project
2. Company runs web-app recommender for various products/services Consider a fictional health care recommender: Medicare supplement silver plan
3. Company runs web-app recommender for various products/services Consider a fictional health care recommender: Currently uses hand-built rules Medicare supplement silver plan
4. Company runs web-app recommender for various products/services Consider a fictional health care recommender: Currently uses hand-built rules
5. Company runs web-app recommender for various products/services Consider a fictional health care recommender: HypochondriacTM Unlimited Checkups Blue Cross Blue Shield Currently uses hand-built rules
6. Company runs web-app recommender for various products/services Consider a fictional health care recommender: HypochondriacTM Unlimited Checkups Blue Cross Blue Shield NOT SCALABLE Currently uses hand-built rules
7. HypochondriacTM Unlimited Checkups Blue Cross Blue Shield MLP (6 layers, 57 node default) Solution: self-maintaining machine learner
8. No data available on what plan previous users chose Training
9. No data available on what plan previous users chose Solution: use old recommender to supervise new recommender Training
10. No data available on what plan previous users chose Solution: use old recommender to supervise new recommender New Recommender: how likely is it that this plan is ranked first by old recommender? old recommender rank 1-N new recommender score 0-1 Training
11. Not all plans scored by old recommender: Need to generate missing plans Training
12. Pipeline Design {f1: ‘like’, f2: 6, f3: ‘blue’, lines: [{l1: ‘big’, l2: 12, l3: False} {l1: ‘small’, l2: 1, l3: True} {l1: ‘small’, l2: 0, l3: False} ] } 0.5 0.3 -.5 0 0.12 1.0 -0.4 translate to ML vector
13. Pipeline Design User request (website) request vector translate to ML vector
14. Pipeline Design User request (website) request vector translate to ML vector request vector plan1 vector request vector plan2 vector request vector plan3 vector SQL plan data
15. Pipeline Design User request (website) request vector translate to ML vector request vector plan1 vector request vector plan2 vector request vector plan3 vector SQL plan data plan1 score plan2 score plan3 score Score each request-plan line with machine learner Return results sorted by score
16. Pipeline Design User request (website) request vector translate to ML vector request vector plan1 vector request vector plan2 vector request vector plan3 vector SQL plan data plan1 score plan2 score plan3 score Score each request-plan line with machine learner Return results sorted by score Retrain on user selection
17. Performance:
18. Performance: Reliably predicts if a plan would be ranked first by old recommender ROC
19. Performance: Old rank vs new score (10 request sample):
20. Performance: Old rank vs new score (10 request sample): High-scoring false positives still plausible options
21. Performance: Old rank vs new score (10 request sample):
22. Performance: Old rank vs new score (10 request sample): bug in old recommender
23. Performance: Old rank vs new score (10 request sample): bug in old recommender
24. Delivered: 1. Self-contained, scalable Python module to replace recommender 2. Built-in method for changing feature space 3. Retraining method to learn from user feedback
25. Josh Dempster Computational physics Northwestern University PhD candidate Monte Carlo algorithms to design new active materials and nano- devices