Data Preparation for Digital Humanities Research

Introduction to structuring and cleaning data. Paired with workshop site: http://thomaspadilla.org/dataprep

Transcript

1. Data Preparation for Digital Humanities Research Thomas Padilla Digital Scholarship Librarian @thomasgpadilla
2. It is often said that 80% of data analysis is spent on the process of cleaning and preparing the data (Dasu and Johnson 2003). Data preparation is not just a first step, but must be repeated many times over the course of analysis as new problems come to light or new data is collected. Hadley Wickham, Tidy Data
4. app.raw.densitydesign.org/
6. Many types of data
7. Today: mostly tabular data
8. Structuring Data
9. Cleaning Data
11. openrefine.org
12. 1. Secret(s) 2. Preparation Principles ◦ structure 3. Cleaning 4.
14. Secret 1: The DH Workshop Experience
15. method(s) + tool(s) + data = something cool
18. If you’ve put together a workshop, you know how much work it is. You have to get the dataset, clean the dataset, put it somewhere people can find it, troubleshoot the software, plan out the steps, etcetera, etcetera. You have to do this if you want to get through all the material in an hour. But guess what? THAT PREP WORK IS WHAT DOING DH IS. All that garbage prep work is what we spend most of our time doing. This seamless processing of data is a fantasy world! Miriam Posner, Here and There: Creating DH Community
19. Secret 2: Getting Started
21. Miriam Posner, How Did They Make That?
22. Its complicated
23. kindredbritain.stanford.edu
24. kindredbritain.stanford.edu
26. Kindred Britain presented a challenge that is growing ever more common in digital humanities scholarship: create a publication that ties together traditionally distinct categories of data in an accessible and scholarly manner. In this case, it meant bringing together network data, geographic data biographical data, and chronological data, while presenting that in tandem with traditional linear narratives. Elijah Meeks and Karl Grossner, Developing Kindred Britain
27. It is often said that 80% of data analysis is spent on the process of cleaning and preparing the data (Dasu and Johnson 2003). Data preparation is not just a first step, but must be repeated many times over the course of analysis as new problems come to light or new data is collected. Hadley Wickham, Tidy Data
29. openrefine.org
30. data tidying structuring datasets to facilitate analysis Hadley Wickham, Tidy Data
32. app.raw.densitydesign.org/
33. dataset a collection of values . . . every value belongs to a variable and an observation. Hadley Wickham, Tidy Data
34. tidy data • Each variable forms a column • Each observation forms a row • Each type of observational unit forms a table Hadley Wickham, Tidy Data
35. Author Book Subject Thomas Padilla Magic is amazing Magic Kristen Mapes CSAs are great Food Bobby Smiley Planes! Planes
36. common messy dataset issues • column headers are values not variable names • multiple variable stored in one column • variable are stored in both rows and columns Hadley Wickham, Tidy Data
37. column headers are values not variable names MovieGenre <$1,000,000 $1,000,000 - $4,000,000 $4,000,000-$10,000,000 Action 10 20 40 Mystery 5 12 6 Drama 20 9 10
38. MovieGenre Revenue Frequency Action >$1,000,000 10 Action $1,000,000 - $4,000,000 20 Action $4,000,000-$10,000,000 40 Mystery >$1,000,000 5 Mystery $1,000,000 - $4,000,000 12 Mystery $4,000,000-$10,000,000 6 solution: turn columns into rows
39. multiple variables stored in one column museum year_acquired p050 p50100 s040 SFMOMA 2002 4 1 1 MOMA 2002 8 8 5 Walters 2002 2 1 3
40. museum year_acquired acquisition_type price frequency SFMOMA 2002 painting $0-$50,000 4 SFMOMA 2002 painting $50,000-$100,000 1 SFMOMA 2002 sculpture $0-$40,000 1 solution: (1) split columns (2) convert to rows museum year_acquired p050 p50100 s040 SFMOMA 2002 4 1 1 MOMA 2002 8 8 5 Walters 2002 2 1 3
41. variables are stored in both rows and columns caller year month place d1 d2 Brandon 2002 7 work 10 3 Brandon 2002 7 home 5 10 Brandon 2002 7 home 7 12
42. solution:(1) condense variables (2) reposition nested variables caller date home work Brandon 2002-07-01 10 3 Brandon 2002-07-01 5 10 Brandon 200-07-01 7 12 caller year month place d1 d2 Brandon 2002 7 work 10 3 Brandon 2002 7 home 5 10 Brandon 2002 7 home 7 12
43. tidy data = maintaining sense of childlike wonder happiness
44. Other Data...
46. Data Cleaning remove data format inconsistencies (e.g. Lansing, MI | Lansing Michigan) split multivariable fields (e.g. Lansing, MI, USA) fix inconsistent terminology (e.g. lansing, Lansing)
47. openrefine.org
48. Faceting Filtering Clustering Transforming
49. bl.uk/bibliographic/datasamples.html
50. Getting Started with Open Refine thomaspadilla.org/dataprep