DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility DMDX and R Rob Davies Department of Psychology, Oxford Brookes University 23 April 2013 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Outline I Introduction DMDX Motivations R Motivations R-DMDX and reproducibility 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility DMDX and R benefits power reproducibility free 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility What is DMDX? freely available written by psychologists used by psychologists (e.g. MACCS, CBU, UM-Wisconsin) active help-list supported Figure : Forster & Forster (2003), DMDX, Behavior Research Methods 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility What is DMDX? DMDX runs on Win32 machines Runs things - stimulus display - through DirectX User writes an .rtf item file DMDX parses the script and instructs DirectX Figure : Example script drawn from Matt Davis (MRC CBU) DMDX tutorial 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility DMDX vs. Superlab in 2000 Things DMDX could do that Superlab could not . . . Figure : CC flickr hasfurrychildren 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) In 2000, I was using Superlab and trying to conduct picture-word interference experiments ‘tree’ ‘shirt’ 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) Picture-word interference effects were/are thought to obtain in narrow SOA windows semantic effect at -150ms phonological effect at +150ms Figure : Schriefers, Meyer, & Levelt (1990), JML 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) DMDX allowed me to get precise SOAs DMDX: event timing is coordinated with respect to a PC’s screen refresh cycle ticks via DirectX Figure : drawn from Matt Davis (MRC CBU) DMDX tutorial 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Things DMDX continues to do that Superlab or other applications cannot . . . It is now no longer acceptable to extract vocal response reaction time using voicekeys Figure : Roelofs (2006) 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Kessler & Treiman (JML): Phonetic biases in voicekey response time measurements Words with different initial phonemes have significantly different RTs Voiceless, posterior, and obstruent consonants detected later than others Phoneme-bases biases are large, persistant and pervasive, confounding RT interpretation Figure : Screen shot of spectrogram analysis window for stimulus “ask” naming response, CheckVocal (Protopapas, 2007, BRM) 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Kessler & Treiman (JML): Phonetic biases in voicekey response time measurements Words with different initial phonemes have significantly different RTs Voiceless, posterior, and obstruent consonants detected later than others Phoneme-bases biases are large, persistant and pervasive, confounding RT interpretation 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility The DMDX-checkvocal-phoneme-coding combination The ‘gold standard’ Present stimuli and record spoken responses to hard disk (DMDX) Analyze spectrograms (Checkvocal) Code phonetic features and incorporate as covariates in regression analyses 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility DMDX vs. proprietary software DMDX advantages Active user community help: http://www.u.arizona.edu/ kforster/dmdx/listserv.htm Reactive creators: if enough people require a feature, it will be built in (within reason) Did I mention it is ‘free as in beer’? Reproducible research: where do your data come from? Figure : Example script drawn from Matt Davis (MRC CBU) DMDX tutorial 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility What is R? first released by Robert and Ross Inhaka in 1996 Figure : NYT, 2009 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility What is R? a system for statistical computation and graphics a language plus a run-time environment with graphics, a debugger and ability to run programmes stored in script files 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility What is R? a system for statistical computation and graphics unix, Mac OS and Windows read any data do any analysis 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R is used by a fast growing dynamic community “lingua franca” of professional statistics (Revolution-R) “second language” for post grads (Pfizer) “hard to overvalue” (google) Figure : Muenchen (website rforsasandspssusers.com): impact of software on publications/hits on google scholar 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R is used by a fast growing dynamic community SAS has “customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet” (NYT, 2009) 2009: SAS announced plans to offer R integration Figure : Muenchen (website rforsasandspssusers.com): impact of software on publications/hits on google scholar 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R is used by a fast growing dynamic community . . . because of rapid growth in capacity capabilities extended by user generated packages developed primarily in R (also sometimes Java, C and Fortran) alow specialized techniques: 4300 in March 2011 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R vs. SPSS Everybody loves SPSS because it is easy . . . 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R vs. SPSS in 2007 easy for what? for whom? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R better that SPSS Linear mixed-effects analyses I started using R in 2007 because SPSS took 12+ hours to complete an analysis completed by R in < 12 minutes 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R better that SPSS Superficial differences R console: stripped down and intimidating but do you want pretty windows or effective functions? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R better that SPSS Superficial differences SPSS has an R plugin 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R better that SPSS Profound differences R analysis is reproducible analysis 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R better that SPSS Profound differences R analysis is reproducible analysis: reproducibility is important and using SPSS makes it hard 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R advantages R graphics capacities are supreme: How Michael Jackson Billboard Rankings Compare With Other Notable Artists (NYT) Figure : NYT, 2009 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R advantages R graphics capacities are supreme: detecting flu using search data (Ginsberg et al., 2009, google) 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility R advantages Active user community help: R help list, stack overflow, nabble, specialist user list servs . . . Reactive creators: if you want to do it, someone else has done it before and put code online Did I mention it is ‘free as in beer’? Reproducible research: where do your data come from? Figure : CC flickr 1875Brian 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? 1. how do researchers report (advertise) observations? 2. Do study – stimulus presentation parameters, programme version? 3. Collate data – who did what and under what circumstances? 4. Collate data – munge and save ‘final version’ for analysis, which version? 5. Perform analysis and choose outputs from one – which analysis, what analysis choices? 6. Write observations and copy plots into Word – which plots, which version analysis? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility SPSS plus Word Irreproducible research How is this acceptable? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? What did I do? What did I do it to? Which version did I use for my report? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Figure : CC flickr 1875Brian 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle My work flow development goal: in 10 years, can I reproduce my results? 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility DMDX plus R plus latex 1. how do researchers report (advertise) observations? 2. Do study – stimulus presentation parameters self-report in DMDX item file 3. Collate data – collate, treat, combine self-report in .R analysis script 4. Collate data – data analyzed for report listed, all treatment noted, on-the-fly 5. Perform analysis and choose outputs from one – analysis choices commented 6. Write observations and copy plot names into latex – transparent link to analysis, data, source 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Stimulus presentation parameters self-report in DMDX item file 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Collate, treat, combine self-report in .R analysis script 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Write observations and copy plot names into latex trial order RT 300 400 500 600 700 800 900 1000 5 10 15 20 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Perform analysis and choose outputs from one – analysis choices commented Figure : R plot code Figure : latex report code 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility Reproducibility: a scientific principle Figure : CC flickr 1875Brian 

DMDX and R Rob Davies Introduction DMDX Motivations R Motivations R-DMDX and reproducibility