R-DMDX and reproducible science

Transcript

1. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility DMDX and R Rob Davies Department of Psychology, Oxford Brookes University 23 April 2013

2. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility Outline I Introduction DMDX Motivations R Motivations R-DMDX and reproducibility

3. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility DMDX and R benefits power reproducibility free

4. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility What is DMDX? freely available written by psychologists used by psychologists (e.g. MACCS, CBU, UM-Wisconsin) active help-list supported Figure : Forster & Forster (2003), DMDX, Behavior Research Methods

5. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility What is DMDX? DMDX runs on Win32 machines Runs things - stimulus display - through DirectX User writes an .rtf item file DMDX parses the script and instructs DirectX Figure : Example script drawn from Matt Davis (MRC CBU) DMDX tutorial

6. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility DMDX vs. Superlab in 2000 Things DMDX could do that Superlab could not . . . Figure : CC flickr hasfurrychildren

7. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) In 2000, I was using Superlab and trying to conduct picture-word interference experiments ‘tree’ ‘shirt’

8. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) Picture-word interference effects were/are thought to obtain in narrow SOA windows semantic effect at -150ms phonological effect at +150ms Figure : Schriefers, Meyer, & Levelt (1990), JML

9. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

   R-DMDX and reproducibility Picture-word interference experiments with precise stimulus onset asynchronies (SOAs) DMDX allowed me to get precise SOAs DMDX: event timing is coordinated with respect to a PC’s screen refresh cycle ticks via DirectX Figure : drawn from Matt Davis (MRC CBU) DMDX tutorial

10. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Things DMDX continues to do that Superlab or other applications cannot . . . It is now no longer acceptable to extract vocal response reaction time using voicekeys Figure : Roelofs (2006)

11. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Kessler & Treiman (JML): Phonetic biases in voicekey response time measurements Words with different initial phonemes have significantly different RTs Voiceless, posterior, and obstruent consonants detected later than others Phoneme-bases biases are large, persistant and pervasive, confounding RT interpretation Figure : Screen shot of spectrogram analysis window for stimulus “ask” naming response, CheckVocal (Protopapas, 2007, BRM)

12. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Kessler & Treiman (JML): Phonetic biases in voicekey response time measurements Words with different initial phonemes have significantly different RTs Voiceless, posterior, and obstruent consonants detected later than others Phoneme-bases biases are large, persistant and pervasive, confounding RT interpretation

13. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility The DMDX-checkvocal-phoneme-coding combination The ‘gold standard’ Present stimuli and record spoken responses to hard disk (DMDX) Analyze spectrograms (Checkvocal) Code phonetic features and incorporate as covariates in regression analyses

14. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility DMDX vs. proprietary software DMDX advantages Active user community help: http://www.u.arizona.edu/ kforster/dmdx/listserv.htm Reactive creators: if enough people require a feature, it will be built in (within reason) Did I mention it is ‘free as in beer’? Reproducible research: where do your data come from? Figure : Example script drawn from Matt Davis (MRC CBU) DMDX tutorial

15. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility

16. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility What is R? first released by Robert and Ross Inhaka in 1996 Figure : NYT, 2009

17. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility What is R? a system for statistical computation and graphics a language plus a run-time environment with graphics, a debugger and ability to run programmes stored in script files

18. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility What is R? a system for statistical computation and graphics unix, Mac OS and Windows read any data do any analysis

19. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R is used by a fast growing dynamic community “lingua franca” of professional statistics (Revolution-R) “second language” for post grads (Pfizer) “hard to overvalue” (google) Figure : Muenchen (website rforsasandspssusers.com): impact of software on publications/hits on google scholar

20. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R is used by a fast growing dynamic community SAS has “customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet” (NYT, 2009) 2009: SAS announced plans to offer R integration Figure : Muenchen (website rforsasandspssusers.com): impact of software on publications/hits on google scholar

21. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R is used by a fast growing dynamic community . . . because of rapid growth in capacity capabilities extended by user generated packages developed primarily in R (also sometimes Java, C and Fortran) alow specialized techniques: 4300 in March 2011

22. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R vs. SPSS Everybody loves SPSS because it is easy . . .

23. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R vs. SPSS in 2007 easy for what? for whom?

24. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R better that SPSS Linear mixed-effects analyses I started using R in 2007 because SPSS took 12+ hours to complete an analysis completed by R in < 12 minutes

25. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R better that SPSS Superficial differences R console: stripped down and intimidating but do you want pretty windows or effective functions?

26. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R better that SPSS Superficial differences SPSS has an R plugin

27. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R better that SPSS Profound differences R analysis is reproducible analysis

28. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R better that SPSS Profound differences R analysis is reproducible analysis: reproducibility is important and using SPSS makes it hard

29. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R advantages R graphics capacities are supreme: How Michael Jackson Billboard Rankings Compare With Other Notable Artists (NYT) Figure : NYT, 2009

30. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R advantages R graphics capacities are supreme: detecting flu using search data (Ginsberg et al., 2009, google)

31. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility R advantages Active user community help: R help list, stack overflow, nabble, specialist user list servs . . . Reactive creators: if you want to do it, someone else has done it before and put code online Did I mention it is ‘free as in beer’? Reproducible research: where do your data come from? Figure : CC flickr 1875Brian

32. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility

33. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility SPSS plus Word Where does this stuff come from?

34. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? 1. how do researchers report (advertise) observations? 2. Do study – stimulus presentation parameters, programme version? 3. Collate data – who did what and under what circumstances? 4. Collate data – munge and save ‘final version’ for analysis, which version? 5. Perform analysis and choose outputs from one – which analysis, what analysis choices? 6. Write observations and copy plots into Word – which plots, which version analysis?

35. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility SPSS plus Word Irreproducible research How is this acceptable?

36. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility SPSS plus Word Where does this stuff come from?

37. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility SPSS plus Word Where does this stuff come from? What did I do? What did I do it to? Which version did I use for my report?

38. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Figure : CC flickr 1875Brian

39. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle My work flow development goal: in 10 years, can I reproduce my results?

40. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility DMDX plus R plus latex 1. how do researchers report (advertise) observations? 2. Do study – stimulus presentation parameters self-report in DMDX item file 3. Collate data – collate, treat, combine self-report in .R analysis script 4. Collate data – data analyzed for report listed, all treatment noted, on-the-fly 5. Perform analysis and choose outputs from one – analysis choices commented 6. Write observations and copy plot names into latex – transparent link to analysis, data, source

41. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Stimulus presentation parameters self-report in DMDX item file

42. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Collate, treat, combine self-report in .R analysis script

43. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Write observations and copy plot names into latex trial order RT 300 400 500 600 700 800 900 1000 5 10 15 20

44. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Perform analysis and choose outputs from one – analysis choices commented Figure : R plot code Figure : latex report code

45. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility Reproducibility: a scientific principle Figure : CC flickr 1875Brian

46. DMDX and R Rob Davies Introduction DMDX Motivations R Motivations

    R-DMDX and reproducibility