Ensemble Learning to Predict Television Program Rating

Transcript

1. Ensemble Learning for Predicting Television Program Rating International Conference on

   Mathematics, Statistics, and Data Science 2020 Iqbal Hanif 1, Regita Fachri Septiani 2 1 Digital Next Business Department, Telkom Indonesia 2 School of Computer Science, Binus University

2. OUTLINE Research background Introduction Rating, ensemble learning, random forest, extreme

   gradient boosting Literarture Review Data preparation, feature engineering, modelling, evaluation Data and Methodology 01 02 03 Data preparation, feature engineering, modelling, evaluation Results To summarize all results and what next? Conclusion 04 05

3. INTRODUCTION International Conference on Mathematics, Statistics, and Data Science 2020

   Source: Katadata 23.7 23.46 18.94 17.66 13.11 14.92 84.94 85.86 90.27 91.68 91.47 93.02 50.29 40.26 23.5 18.57 7.54 13.31 0 10 20 30 40 50 60 70 80 90 100 2003 2006 2009 2012 2015 2018 Percentage Percentage of People Attention (with Age more than 10 years) to the Media (Radio, Newspaper, Television) in year 2003-2018 Source : Indonesia Statistics Bureau, 2019 Read Magazine/Newspaper Watch Television Listen to Radio 674 national and local TV channels 60 channels are owned by the 12 largest media group Rp 145.5 trillion total advertisement expenditure in Indonesia media (2017) 80% of it was occupied by TV broadcasters Television Business Overview

4. INTRODUCTION International Conference on Mathematics, Statistics, and Data Science 2020

   • TV Rating Prediction A service that analytically determines when, where and how many commercial ads of a product/service will be run for achieving optimum result. Provide by media research company or institution. • Media Planning Very useful for future advertisement budget, because the media planner will calculate how long the ads will be run to achieve Gross Rating Points (GRP) during the campaign period based on the prediction result This research is aimed to develop a prediction model for TV program rating scores for InRate by using TV program rating data from UseeTV and implementing ensemble learning methods InRate Media Research Media research product from Telkom Indonesia, providing TV Audience measurement using UseeTV population data with detail audience profile. Extreme Gradient Boosting Random Forest Neural Network Penalized Regression Other Algorithms Linear Regression Multiple Adaptive Regression Splines Support Vector Machine Source: Sereday and Cuy (2017)

5. OUTLINE Research background Introduction Rating, ensemble learning, random forest, extreme

   gradient boosting Literature Review Data preparation, feature engineering, modelling, evaluation Data and Methodology 01 02 03 Data preparation, feature engineering, modelling, evaluation Results To summarize all results and what next? Conclusion 04 05

6. LITERATURE REVIEW: Rating Rating is a percentage of total spectators

   (a person or a household) achieved by a TV program or TV channel divided by TV audience population International Conference on Mathematics, Statistics, and Data Science 2020 = With x is a TV program or TV channel broadcasted at a specific time

7. LITERATURE REVIEW: Ensemble Learning An ensemble contains several base learners

   which are also called weak learners. It has ability to improve prediction accuracy by generalizing results from weak learners (which slightly better than random guess) to be a strong learner with significantly higher prediction accuracy. International Conference on Mathematics, Statistics, and Data Science 2020 Bagging Boosting Stacking Train base learners based on a random sample of the training data set Increasing the weight of misclassified classes for each base learner training iteration Train several base learners simultaneously, results are combined by the second level learner (meta learner)

8. LITERATURE REVIEW: Random Forest • Developed from the bagging method

   by Breiman (2001). • Randomly picks several features (by generating random vector ) to train a base learner for minimizing the correlation between base learners. • From p features, Random Forest Method usually pick √p features for every base learner. • Evaluation is conducted by measuring mean-squared generalization error for each numerical predictor h(x) International Conference on Mathematics, Statistics, and Data Science 2020 PE∗ = ൯ , ( − ℎ( , ) 2 With , ′ is mutually independent and assumed that the training set was drawn randomly from the distribution of random vector Y, X

9. LITERATURE REVIEW: XGBoost • Developed from Tree Boosting by Chen

   and Guestrin (2016) • Each boosting iteration adds weight for miss-classified error sample and subtract weight for correct-classified sample • Solves the limitation of common tree boosting algorithms that could not develop a good model prediction in large scale datasets by improving the scalability of tree boosting algorithms. • Each iteration in XGBoost will produce a tree for optimizing an objective function. International Conference on Mathematics, Statistics, and Data Science 2020 ℒ ϕ = ෍ ෝ , + ෍ Ω where Ω = + 1 2 2 with l is a differentiable convex loss function that measures the difference between the prediction ෝ and the target , and Ω penalizes the complexity of the model.

10. OUTLINE Research background Introduction Rating, ensemble learning, random forest, extreme

    gradient boosting Literature Review Data preparation, feature engineering, modelling, evaluation Data and Methodology 01 02 03 Data preparation, feature engineering, modelling, evaluation Results To summarize all results and what next? Conclusion 04 05

11. DATA: Data Preparation Phase International Conference on Mathematics, Statistics, and

    Data Science 2020 VARIABLE NAME DESCRIPTION DATE Date when tv program was shown CHANNEL Channel where tv program was shown PROGRAM Program Name GENRE Program Genre SUB_GENRE Program Sub-Genre TVR Mean of TV Rating (TVR) per day when the TV program was shown • This research utilized TV program rating data gathered from UseeTV. • Collected data was aggregated data by calculating daily average rating from 20 top programs (18 daily programs and 2 weekly programs), determined by rating score and program consistency • Date range: January 1st, 2018 to September 30th, 2019. • For privacy reasons, program names and channels were masked with other names

12. METHODS: Data Preparation Phase Population Prepared Dataset Sampling based on

    data range Split X (feature) and Y (rating score) Check data incompletion Delete incomplete rows • Check for incomplete, noisy, and inconsistent, which can disguise useful patterns. • Generate a dataset smaller than the original one, which can significantly improve the efficiency of data mining. • Generate quality data based on checking/observation result, which leads to quality patterns. International Conference on Mathematics, Statistics, and Data Science 2020 3 Main aspects of Data Preparation Source: Zhang et. al (2003)

13. METHODS: Feature Engineering Phase Time Series Feature Engineering Categorical Feature

    Engineering International Conference on Mathematics, Statistics, and Data Science 2020 Time Series Feature Engineering, by developing new numerical features from date, such as hour, day of the week, quarter, month, year, day of the year, day of the month, and week of the year Categorical Feature Engineering, by doing a one-hot-encoding process from categorical data (string format), producing new features based on several categories DATE DAY OF WEEK QUARTER MONTH YEAR DAY OF YEAR DAY OF MONTH WEEK OF YEAR 03/09/ 2019 1 3 9 2019 246 3 36 04/09/ 2019 2 3 9 2019 247 4 36 05/09/ 2019 3 3 9 2019 248 5 36 PROGRAM GENRE GENRE _News GENRE _Series GENRE_ Children PROGRAM 6 SERIES 0 1 0 PROGRAM 10 CHILDRE N 0 0 1 PROGRAM 13 NEWS 1 0 0 Source: Sisiaridis and Markowitch (2017) and Saradhi and Nelaturi (2018)

14. METHODS: Modelling Phase Split Training and Testing Dataset Defining Hyperparameters

    Cross Validation with Training Dataset Predicting Testing Dataset Best Model with Best Hyperparameters Evaluation Evaluation XGBoost Randomized Search Cross Validation with fold=10 and n_iter=10 International Conference on Mathematics, Statistics, and Data Science 2020 Hyperparameter Value max_depth [3, 5, 7, 9] min_child_weight [1, 3, 5] Gamma [0.0, 0.33333, 0.25, 0.5, 0.66667, 0.75] reg_alpha [1e-5, 1e-2, 0.1, 1, 100] n_estimators [100, 200, 300, 500, 750, 1000] learning_rate [0.01, 0.015, 0.02, 0.05, 0.08, 0.1] Hyperparameter Value Bootstrap [True, False] max_depth [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None] max_features ['auto', 'sqrt'] min_samples_leaf [1, 2, 4] min_samples_split [2, 5, 10] n_estimators [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000] Random Forest

15. METHODS: Evaluation Phase RMSE International Conference on Mathematics, Statistics, and

    Data Science 2020 = 1 ෍ =1 MAPE Value Interpretation = σ =1 2 Τ 1 2 = 100 ෍ =1 MAE MAPE Root Mean Square Error, obtained by calculating the sum square of deviation between prediction value and real value divided by the number of observations (like calculating average), and then root square it back. Mean Absolute Error, obtained by calculating the sum of absolute deviation between prediction value and real value and then divided by the number of observations Mean Absolute Percentage Error, obtained by calculating the sum of the absolute deviation of deviation between prediction value and real value divided by real value. After that, the sum result was multiplied by 100 before divided by the number of observations MAPE Interpretation <10 Highly Accurate Forecasting 10-20 Good Forecasting 20-50 Reasonable Forecasting >50 Inaccurate Forecasting Source: Moreno et. al (2013)

16. OUTLINE Research Background Introduction Rating, ensemble learning, random forest, extreme

    gradient boosting Literature Review Data preparation, feature engineering, modelling, evaluation Data and Methodology 01 02 03 Data preparation, feature engineering, modelling, evaluation Results To summarize all results and what next? Conclusion 04 05

17. RESULT: Data Preparation Phase International Conference on Mathematics, Statistics, and

    Data Science 2020 Total Rows Total Columns Removed Missing Value 9.396 6 27

18. RESULT: Feature Engineering Phase International Conference on Mathematics, Statistics, and

    Data Science 2020 Dataset Total Rows Total Columns Training 8.949 49 Testing 447 49

19. RESULT: Modelling Phase (Hyperparameters) Random Forest XGBoost International Conference on

    Mathematics, Statistics, and Data Science 2020 Hyperparameter Value n_estimators 200 min_samples_split 10 min_samples_leaf 2 max_features auto max_depth 70 Bootstrap True Mean Square Error Score CV -0.000001025 Hyperparameter Value reg_alpha 0.01 n_estimators 500 min_child_weight 1 max_depth 5 learning_rate 0.015 Gamma 0 Mean Square Error Score CV -0.000002804

20. RESULT: Modelling Phase (Feature Importance) International Conference on Mathematics, Statistics,

    and Data Science 2020 Random Forest XGBoost

21. RESULT: Evaluation Phase (Training) International Conference on Mathematics, Statistics, and

    Data Science 2020 Program Algorithm RMSE MAE MAPE PROGRAM 1 XGBoost 0.001079 0.000880 1169.376591 Random Forest 0.000635 0.000456 780.146159 PROGRAM 2 XGBoost 0.001727 0.001271 957.977207 Random Forest 0.001114 0.000745 708.413365 PROGRAM 3 XGBoost 0.003241 0.002631 86.958874 Random Forest 0.002706 0.002197 65.715629 PROGRAM 4 XGBoost 0.001949 0.001493 1356.841966 Random Forest 0.001191 0.000871 621.195855 PROGRAM 5 XGBoost 0.001557 0.001162 716.017067 Random Forest 0.000954 0.000680 353.662242 PROGRAM 6 XGBoost 0.001800 0.001359 948.559290 Random Forest 0.000976 0.000632 556.757284 PROGRAM 7 XGBoost 0.001334 0.001038 2199.488160 Random Forest 0.000864 0.000598 1145.764204 PROGRAM 8 XGBoost 0.001956 0.001543 31.532514 Random Forest 0.001073 0.000782 13.723757 PROGRAM 9 XGBoost 0.001523 0.001274 97.534369 Random Forest 0.000926 0.000684 51.839412 PROGRAM 10 XGBoost 0.001324 0.001063 1317.207181 Random Forest 0.000663 0.000477 578.403591 Program Algorithm RMSE MAE MAPE PROGRAM 11 XGBoost 0.001780 0.001278 189.019988 Random Forest 0.001116 0.000770 104.725419 PROGRAM 12 XGBoost 0.001849 0.001377 94.266201 Random Forest 0.001165 0.000805 49.290944 PROGRAM 13 XGBoost 0.002005 0.001514 454.319257 Random Forest 0.001152 0.000764 171.798316 PROGRAM 14 XGBoost 0.001731 0.001275 19.978834 Random Forest 0.001148 0.000782 11.724369 PROGRAM 15 XGBoost 0.001647 0.001202 814.267250 Random Forest 0.000918 0.000648 438.282048 PROGRAM 16 XGBoost 0.001318 0.001052 1318.655935 Random Forest 0.000702 0.000485 684.929290 PROGRAM 17 XGBoost 0.001541 0.001209 34.410839 Random Forest 0.000858 0.000573 17.593960 PROGRAM 18 XGBoost 0.001653 0.001222 1888.771860 Random Forest 0.000725 0.000707 869.335587 PROGRAM 19 XGBoost 0.001329 0.001047 22.886654 Random Forest 0.000725 0.000520 10.849950 PROGRAM 20 XGBoost 0.001721 0.001240 107.311019 Random Forest 0.001038 0.000725 49.864228 • Random Forest : 6 programs under reasonable forecasting quality threshold (4 of them are good quality forecasting). • XGBoost : 4 programs under reasonable forecasting quality threshold (1 of them is good quality forecasting)

22. RESULT: Evaluation Phase (Training) International Conference on Mathematics, Statistics, and

    Data Science 2020 Random Forest XGBoost Daily Program Example (PROGRAM 8) Weekly Program Example (PROGRAM 9)

23. RESULT: Evaluation Phase (Testing) International Conference on Mathematics, Statistics, and

    Data Science 2020 Program Algorithm RMSE MAE MAPE PROGRAM 1 XGBoost 0.001225 0.001136 50.068081 Random Forest 0.000805 0.000653 25.648749 PROGRAM 2 XGBoost 0.001603 0.001486 26.563746 Random Forest 0.000884 0.000621 10.148792 PROGRAM 3 XGBoost 0.004312 0.003441 33.118269 Random Forest 0.005805 0.004863 41.280830 PROGRAM 4 XGBoost 0.002130 0.001911 35.355186 Random Forest 0.002009 0.001670 30.117729 PROGRAM 5 XGBoost 0.001611 0.001396 16.167560 Random Forest 0.001012 0.000796 9.865887 PROGRAM 6 XGBoost 0.001678 0.001559 31.007823 Random Forest 0.000988 0.000689 11.970533 PROGRAM 7 XGBoost 0.001624 0.001503 56.546293 Random Forest 0.000732 0.000552 19.036968 PROGRAM 8 XGBoost 0.002852 0.002643 22.357429 Random Forest 0.000805 0.000632 5.790603 PROGRAM 9 XGBoost 0.001227 0.001105 51.979243 Random Forest 0.000566 0.000464 20.161858 PROGRAM 10 XGBoost 0.000587 0.000490 7.570485 Random Forest 0.000640 0.000509 7.439328 Program Algorithm RMSE MAE MAPE PROGRAM 11 XGBoost 0.001220 0.000938 8.619393 Random Forest 0.001258 0.000919 8.587563 PROGRAM 12 XGBoost 0.001287 0.001017 9.027126 Random Forest 0.001488 0.001117 10.146653 PROGRAM 13 XGBoost 0.002801 0.002324 57.208752 Random Forest 0.002400 0.001306 27.783101 PROGRAM 14 XGBoost 0.001418 0.001149 10.533514 Random Forest 0.001138 0.000772 6.957159 PROGRAM 15 XGBoost 0.000734 0.000579 6.988784 Random Forest 0.000530 0.000449 5.228035 PROGRAM 16 XGBoost 0.000638 0.000552 8.739498 Random Forest 0.000596 0.000367 5.514766 PROGRAM 17 XGBoost 0.001112 0.000994 13.860131 Random Forest 0.000620 0.000452 6.461361 PROGRAM 18 XGBoost 0.000979 0.000802 8.637751 Random Forest 0.000932 0.000705 7.943226 PROGRAM 19 XGBoost 0.000679 0.000518 6.415885 Random Forest 0.000383 0.000299 3.915973 PROGRAM 20 XGBoost 0.001867 0.001637 17.024738 Random Forest 0.001049 0.000763 8.912322 • Random Forest : 20 programs under reasonable forecasting quality threshold (16 of them are good quality forecasting). • XGBoost : 16 programs under reasonable forecasting quality threshold (12 of them are good quality forecasting)

24. RESULT: Evaluation Phase (Testing) International Conference on Mathematics, Statistics, and

    Data Science 2020 Daily Program Example (PROGRAM 8) Weekly Program Example (PROGRAM 9) Random Forest XGBoost

25. RESULT: Evaluation Phase (Combination) International Conference on Mathematics, Statistics, and

    Data Science 2020 Random Forest XGBoost

26. OUTLINE Research background Introduction Rating, ensemble learning, random forest, extreme

    gradient boosting Literature Review Data preparation, feature engineering, modelling, evaluation Data and Methodology 01 02 03 Data preparation, feature engineering, modelling, evaluation Result To summarize all results and what next? Conclusion 04 05

27. CONCLUSION • Regarding the evaluation result, the Random Forest method

    produced lower RMSE and MAE scores in the majority of programs both in testing and training datasets compared to the XGBoost method. This fact indicates that the Random Forest method produces a rating prediction score closer to the actual rating score (lower error) than the XGBoost method. • According to the MAPE score, the RF method produced more prediction scores with good forecasting quality in both training and testing datasets compared to the XGB method. • Another finding is that prediction quality in the testing dataset (with a shorter period) gives a better forecasting quality compared in the training dataset (with a longer period); this indicates that prediction model capability is decreased due to a longer period, which presumably produces a larger error than a short period. International Conference on Mathematics, Statistics, and Data Science 2020

28. WHAT’s NEXT? International Conference on Mathematics, Statistics, and Data Science

    2020 This is a preliminary research for building a prediction model with high accuracy (low error). We have never done this before, and we also surprised that ensemble learning can give a good result for time series prediction (we had tried ARIMA at first). But, there is still room of improvement, so we have suggestions for next research: • Gather more data (more program, more variation (daily/weekly), and longer period of time). • Try to predict rating program with longer testing period of time. If the model capability is decreased, find other comparing algorithm that could handle that problem. • If the model give satisfactory results for the stakeholder, deploy the model.

29. THANK YOU International Conference on Mathematics, Statistics, and Data Science

    2020