Reproducibility in Regional Science Robin Lovelace 21/08/2014 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 1 of 32 21/08/14 10:42 

What I'm going to talk about The motivations for reproducibility How reproducible is Regional Science? Preliminary results Measures to improve reproducibility · · · · 2/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 2 of 32 21/08/14 10:42 

The motivations for reproducibility Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 3 of 32 21/08/14 10:42 

Why reproducibility? Many powerful arguments put forward in medicine, physical sciences and health But how do these relate to the discipline of Regional Science? Do we even need reprodicibility? We'll consider 6 reasons · · · · 4/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 4 of 32 21/08/14 10:42 

Why reproducibility I: Education 5/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 5 of 32 21/08/14 10:42 

II: Error detection "The intellectual edifice of austerity economics rests largely on two academic papers that were seized on by policy makers, without ever having been properly vetted" (Krugman) In April 2013, a mistake in an Excel spreadsheet was found. This mistake meant that an influential paper (Reinhart and Rogof, 2010) contained faulty results · · Reinhart and Rogof deserve credit for publishing methods and admitting mistake. · 6/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 6 of 32 21/08/14 10:42 

III: Encouragement of collaboration 7/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 7 of 32 21/08/14 10:42 

IV: Reproducibility is empowering, democratic Millions of the world's poorest people lack access to data Decisions usually made behind closed doors using unknown methods The 'black box' economic evaluation of new roads is a good example Open source software and data remove barriers to entry Bridges digitial divide and prevents academic elitism · · · · · 8/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 8 of 32 21/08/14 10:42 

V: Falsifiability is the foundation of science Karl Popper, 1902-1994 Can an idea be disproven? This determines whether it can be classified as 'science' or not. Hypothesis testing and refutation of existing theory: integral to the physical sciences (Popper, 1959) · · 9/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 9 of 32 21/08/14 10:42 

VI: Reproducibility is being scrutinised in other disciplines The replication of important findings by multiple independent investigators is fundamental to the accumulation of scientific evidence (Peng et al., 2006) Reproducibility taken seriously in the physical sciences Recent debate in Psychology sparked by a Special Issue (Pashler et al., 2012) asking if there is a 'crisis of confidence' in the field Reproducibility especially critical in areas with human consequences e.g. epidemiology: · · · 10/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 10 of 32 21/08/14 10:42 

How reproducible is Regional Science? Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 11 of 32 21/08/14 10:42 

Regional science and reproducibility Reproducibility is a good thing Reproducibility -> falsifiability -> scientific credebility H0: Most Regional Science research papers contain reproducible results. H1: Most Regional Science research lacks reproducibility Regional Scientists pride their 'scientific' credentials Method based on premises: · · The hypotheses: · 12/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 12 of 32 21/08/14 10:42 

Methodology Different ways to test reproducibility Most ambitious: re-do experiment and analysis (Open Science Collaboration, 2012) Use of established criteria for reproducibility (Peng et al., 2006) Select sample of papers representative of Regional Science and test Peng's criteria · · · · 13/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 13 of 32 21/08/14 10:42 

Criteria for reproducibility Inadequate description of data and methods Basic description of datasets and outline of methodology Detailed description of data and methodology Provision of sample data and code/procedure enabling reproduction of results Peng et al. (2006) provided simple and clear criteria in epidemiology Minimum standards for data, methods, documentation and distribution These standards were used as the basis to assess the reproducibility of papers on a scale of 1 to 4: · · · Related issue of accessibility (available/unavailable) to public scrutiny also recorded · 14/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 14 of 32 21/08/14 10:42 

Selection of papers to assess Papers in Regional Science Regional Science and Urban Economics Journal of Regional Science Regional Studies Only empirical (based on analysis of data) papers published in leading regional science journals selected: Google Scholar search term: "regional science" data between 2000 and 2010 in the following journals: · · Most cited articles analysed (most influential) Plans to expand sampling (ideas?) · · 15/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 15 of 32 21/08/14 10:42 

Results ## Loading required package: XML Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 16 of 32 21/08/14 10:42 

The characteristics of the sample papers Paper Rating (1:4) Availability Citations Software 2 Esteban (2010) 2 2 249 0 3 Mohl (2010) 3 2 74 0 4 Duranton (2008) 2 2 130 0 5 Yu (2008) 3 2 42 1 6 Espoti and Bussoletti (2008) 2 2 71 0 7 Fingleton (2005) 3 3 29 0 8 Frenken et al. (2007) 2 2 676 0 9 Van Stel et al. (2004) 2 1 74 0 10 Elhorst (2003) 1 1 656 0 11 Braunerhjelm and Borgman (2004) 2 1 122 0 17/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 17 of 32 21/08/14 10:42 

Description of data and methods 18/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 18 of 32 21/08/14 10:42 

Mentions of software 19/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 19 of 32 21/08/14 10:42 

Accessibility 20/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 20 of 32 21/08/14 10:42 

Overall score A weighted total was applied to the data scores for each paper: Total <- Rating + Availability + 2 * Software 21/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 21 of 32 21/08/14 10:42 

Lessons from the winning paper Yu and Wei (2008) describe software and specific packages used Cited for methods, not just findings (benefit of reproducibility) Still far from fully reproducible: no code or data Full version available online · · · · 22/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 22 of 32 21/08/14 10:42 

Measures to improve reproducibility Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 23 of 32 21/08/14 10:42 

Overview of steps towards reproducibility Individual authors Academia Civil society and funding councils Steps needed to improve reproducibility are simple and relatively easy to implement Reproducibility is being advocated on 3 levels: · · 24/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 24 of 32 21/08/14 10:42 

Steps individual authors can take 3 levels recommended for Regional Scientists: Minimum standard: clear method used including software and links to data source Sample procedure and data: sample datasets and 'code snippets' provided Completely reproducible: e.g. everything needed to reproduce results available in a documented zip file · · · 25/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 25 of 32 21/08/14 10:42 

Data, methods, documentation, distribution Lovelace and Ballas (2013) 26/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 26 of 32 21/08/14 10:42 

Measures in academia "PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception." (plosone.org) Koole and Lakens (2012) recommend parallel publication of 'replications' to incentivise reproducibility and testing findings PLOSONE and some other journals insist on publication of code and data source Move towards interactive graphics encourages reproducibility · · · 27/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 27 of 32 21/08/14 10:42 

Civil society and funding councils "Data are the main assets of economic and social research. We recognise publicly- funded research data as valuable, long-term resources that, where practical, must be made available for secondary scientific research." (ESRC.ac.uk) Rapid uptake of open source software Open data movements making it easier to share datasets Funding councils increasingly demand data outputs Want methods to be of use to others · · · · 28/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 28 of 32 21/08/14 10:42 

Conclusions Reproducibility is important for Regional Science Regional Scientists tend to describe data and methods well (by social science standards) but much room for improvement The steps needed to make your work more reproducible are easy to implement and may make your work more citeable and useful to others This will help Regional Science grow as a trustworthy, robust and scientific discipline · · · · 29/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 29 of 32 21/08/14 10:42 

Key References Koole, S. L., & Lakens, D. (2012). Rewarding Replications: A Sure and Simple Way to Improve Psychological Science. Perspectives on Psychological Science, 7(6), 608–614. Lovelace, R., & Ballas, D. (2013). “Truncate, replicate, sample”: A method for creating integer weights for spatial microsimulation. Computers, Environment and Urban Systems, 41, 1–11. Open Science Collaboration. (2012). An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science. Perspectives on Psychological Science, 7(6), 657–660. Pashler, H., & Wagenmakers, E.–J. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science A Crisis of Confidence? Perspectives on Psychological Science, 7(6), 528–530. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–9. doi:10.1093/aje/kwj093 Popper, K. R. (1959). The Logic of scientific discovery: Karl R. Popper (p. 480). Hutchinson. Rogoff, K., & Reinhart, C. (2010). Growth in a Time of Debt. American Economic Review, 100(2), 573–578. 30/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 30 of 32 21/08/14 10:42 

Papers assessed I Braunerhjelm, P., & Borgman, B. (2004). Geographical Concentration, Entrepreneurship and Regional Growth: Evidence from Regional Data in Sweden, 1975-99. Regional Studies, 38(8), 929–947. Duranton, G., & Overman, H. (2008). Exploring the detailed location patterns of UK manufacturing industries using microgeographic data. Journal of Regional Science, (756). Elhorst, J. P. (2003). Specification and Estimation of Spatial Panel Data Models. International Regional Science Review, 26(3), 244–268. Esposti, R., & Bussoletti, S. (2008). The impact of Objective 1 funds on regional growth convergence in the EU. A panel-data approach. Regional Studies, 02, 159–173. Esteban, J. (2000). Regional convergence in Europe and the industry mix: a shift-share analysis. Regional Science and Urban Economics, 30(3), 353–364. 31/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 31 of 32 21/08/14 10:42 

Papers assessed II Fingleton, B. (2005). Beyond neoclassical orthodoxy: a view based on the new economic geography and UK regional wage data. Papers in Regional Science, 84.3, 351–375. Frenken, K., Oort, F. Van, & Verburg, T. (2007). Related variety, unrelated variety and regional economic growth. Regional Studies, 05, 685–697. Mohl, P., & Hagen, T. (2010). Do EU structural funds promote regional growth? New evidence from various panel data approaches. Regional Science and Urban Economics, 40(5). Van Stel, A. J., & Nieuwenhuijsen, H. R. (2004). Knowledge Spillovers and Economic Growth: An Analysis Using Data of Dutch Regions in the Period 1987–1995. Regional Studies, 38(4), 393–407. Yu, D., & Wei, Y. D. (2008). Spatial data analysis of regional development in Greater Beijing, China, in a GIS environment. Papers in Regional Science, 87(1), 97–117. 32/32 Reproducibility in Regional Science file:///home/robin/Dropbox/Conferences/rsai-aberystwyth-20-21-august/rsaiPr... 32 of 32 21/08/14 10:42