Predicting irregularities in public bidding: an application of neural networks

Transcript

1. Predicting irregularities in public bidding: an application of neural networks Observatory of Public Spending
2. Government contractor doesn’t pay employees Default epidemy in the federal government: 4 companies went bankrupt Construction company abandons 3 projects Observatory of Public Spending
3. Observatory of Public Spending what if we could predict which contractors will become headaches?
4. Observatory of Public Spending
5. Observatory of Public Spending impossible to do manually ~25k new contracts every year
6. Observatory of Public Spending
7. Observatory of Public Spending data + neural networks = predictions
8. Observatory of Public Spending data: - n = 10186 - 9442 (~93%) not problem - 744 (~ 7%) problem - 2011-2016
9. Observatory of Public Spending data: - Y: has the company been punished before?
10. Observatory of Public Spending data: - X: a total of 183 attributes, like: - # of employees - average salary of employees - # of auctions it participated - donated $ to politicians? - …
11. Observatory of Public Spending neural networks: - two approaches: - (“traditional”) neural network - deep neural network
12. Observatory of Public Spending TNN: - 2 hidden layers - can’t handle 183 attributes - hence must use PCA first
13. Observatory of Public Spending TNN: - PCA - selected 24 continuous variables based on covariance matrix - PCA reduced 24 variables to 9 components (~70% of variance; all components w/ eigenvalue > 1)
14. Observatory of Public Spending TNN: - 9 components + 21 binary vars. - 80% training - w/ oversampling - 20% testing - boosting (10 models)
15. Observatory of Public Spending DNN: - 3 hidden layers - hundreds of neurons - can handle all 183 variables - can handle complex relationships between the variables
16. Observatory of Public Spending DNN: - all 183 variables (no PCA) - no oversampling - 80% training - 20% testing - 5-fold cross-validation
17. Observatory of Public Spending
18. Observatory of Public Spending how can we evaluate performance? - accuracy (% of correct predictions overall) - recall (% of problems predicted to be problems) - precision (% of predicted problems that are problems)
19. Observatory of Public Spending how can we evaluate performance? - accuracy (% of correct predictions overall) - recall (% of problems predicted to be problems) - precision (% of predicted problems that are problems)
20. Observatory of Public Spending results: - TNN precision: 0.24 - DNN precision: 0.79 - huge difference! extra computational cost of DNN is worth it
21. Observatory of Public Spending to do: - improve recall - 0.58 w/ TNN - 0.26 w/ DNN - change the law - must allow gov not to contract w/ high risk companies
22. Observatory of Public Spending Ting Sun tsun9920@gmail.com Leonardo Sales leonado.sales@cgu.gov.br
23. Observatory of Public Spending @tmarzagao thiagomarzagao.com