Metrics Analysis 12. Ablation and Error Analysis 13. Output Postprocessing 14. Model Explanation 15. A/B testing 16. Reproducible Research and Models 17. Model Deployment and Retraining Table of Contents
data generating process, user journey and business process (BP) in a company - You can create your ML services by automating any process in BP. - Veriﬁcation process - Identiﬁcation process - Scoring/estimation process - etc.
about top N recommendation - Change objective metrics to precision-recall as an information retrieval problem - Change objective metrics to top@K precision-recall - Decision-support metrics: - ROC AUC, Breese score, later precision/recall - Error meets decision-support/user experience: - “Reversals” - User-centered metrics: - Coverage, user retention, recommendation uptake, satisfaction
afraid to invest in ﬁntech because they never had any experience in crises. - Bank Perkreditan Rakyat have data before and after the ﬁnancial crises. - We need to build a Credit Scoring with 96-99 data to simulate a ﬁnancial crises.
- We need to ﬁnd patterns which might affect our model/objective metrics. - We need to ﬁnd anomalies which will decrease our model accuracy. - In the next section, we can generate new variables from this understanding to ease our model to learn and ﬁt.
more likely to post when they are Accepted. • They are likely to post results for multiple applications. • More successful candidates are likely to post their personal numbers. • Ostensibly, the quality of applicants who engage heavily with an online forum are far more serious about their application than the entirety of the test-taking pool, leading to better results/numbers. Sources: https://debarghyadas.com/writes/the-grad-school-statistics-we-never-had/
case: - We will focus on Diversity Metrics vs Accuracy Metrics. - More diverse recommendation will increase Netﬂix CTR - More accurate recommendation will increase Netﬂix CTR - Diversity and Accuracy are negatively correlated.