SOC 4650/5650 - Lecture-07 - Data Cleaning

Transcript

1. DATA CLEANING INTRO TO GISC CHRISTOPHER PRENER, PH.D. SPRING 2020 WEEK 07 LECTURE 07
2. AGENDA INTRO TO GISC / WEEK 07 / LECTURE 07 1. Front Matter 2. Types of Data 3. Data Wrangling 4. Back Matter
3. 1 FRONT 
 MATTER
4. Clarifying PS-02 expectations - two conflicting copies were available through the course website. Please submit what you have on Opportunity Zones by Thursday at 5pm. 1. FRONT MATTER ANNOUNCEMENTS Next Week: Lab-05 (from last week), Lab-06 (from today), final project check-in Two Weeks: PS-03 (from today), Lab-07 (from next week), video lectures over spring break
5. TYPES OF DATA 2
6. 2. TYPES OF DATA WHAT ARE SPATIAL DATA? Tabular Geometric
7. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous Tabular
8. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous crime “ucr crime type”
 1. murder 2. rape 3. robbery 4. aggravated assault
9. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous crimeStr “ucr crime type, string”
 murder rape robbery aggravated assault
10. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous newCrime “new incident”
 1. no 2. yes 0. 1.
11. 2. TYPES OF DATA TABULAR DATA qualitative ordinal continuous newCrime “new incident”
 1. no 2. yes 0. 1.
12. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous leadLevels “rate of exposure”
 1. very low 2. low 3. high 4. very high
13. 2. TYPES OF DATA TABULAR DATA categorical ordinal continuous leadCount “number of high tests”
 1 2 … 85
14. 2. TYPES OF DATA GEOMETRIC DATA Tabular Geometric
15. DATA WRANGLING 3
16. 4. DATA WRANGLING HIGH LEVEL WORKFLOW 1. Plan 2. Organize 3. Document 4. Execute FOR EACH
 STEP:
17. IT IS OFTEN SAID THAT 80% OF DATA ANALYSIS IS SPENT ON THE PROCESS OF CLEANING AND PREPARING THE DATA Hadley Wickham “Tidy Data”
 (2014)
18. HAPPY FAMILIES ARE ALL ALIKE; EVERY UNHAPPY FAMILY IS UNHAPPY IN ITS OWN WAY Leo Tolstoy Anna Karenina
 (1878)
19. HAPPY FAMILIES ARE ALL ALIKE; EVERY UNHAPPY FAMILY IS UNHAPPY IN ITS OWN WAY Leo Tolstoy Anna Karenina
 (1878)
20. LIKE FAMILIES, TIDY DATASETS ARE ALL ALIKE BUT EVERY MESSY DATASET IS MESSY IN ITS OWN WAY. Hadley Wickham “Tidy Data”
 (2014)
21. TIDY DATASETS PROVIDE A STANDARDIZED WAY TO LINK THE STRUCTURE OF A DATASET (ITS PHYSICAL LAYOUT) WITH ITS SEMANTICS (ITS MEANING). Hadley Wickham “Tidy Data”
 (2014)
22. 4. DATA WRANGLING TIDY DATA Each variable should be saved in its own column. A B C D
23. ▸ Are variables named consistently? ▸ Are variable names clear? ▸ Are variables stored in the format that makes the most sense for their data? ▸ Do variables represent one and only one construct? ▸ Is there a unique identification variable? ▸ Is there missing or incomplete data? 4. DATA WRANGLING A B VARIABLES
24. 4. DATA WRANGLING TIDY DATA Each observation should be saved in its own row. A B C D
25. ▸ What is the observational unit? 4. DATA WRANGLING A B OBSERVATIONS
26. 4. DATA WRANGLING TIDY DATA Each table (i.e. each file or data frame) should contain a single observational unit. A B C D Car Dealer Brand Cars Dealers Brands
27. ▸ What is the observational unit? • Do the data need to be subset into tables with different observational units? ▸ Are there duplicate observations? ▸ Are there “near” duplicate observations? 4. DATA WRANGLING A B OBSERVATIONS
28. ▸ readr for reading tabular data ▸ janitor for its data wrangling functions ▸ dplyr for data wrangling functions ▸ naniar for missing data analyses ▸ stringr for detecting patterns in string data 4. DATA WRANGLING PACKAGES
29. 3. DATA WRANGLING VERBS FOR DATA CLEANING janitor::clean_names() is used for renaming all columns id Bad name Very long name Meh name id bad_name very_long_name meh_name
30. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::rename() is used for renaming some columns id a b c id new_name b c
31. 3. DATA WRANGLING VERBS FOR DATA CLEANING naniar::miss_var_summary() is used identifying missing data id a b c NA NA NA miss_var_summary(data) #> # A tibble: 4 x 3 #> variable n_miss pct_miss #> <chr> <int> <dbl> #> 1 c 2 50.0 #> 2 d 1 24.0 #> 3 id 0 0 #> 4 a 0 0
32. 3. DATA WRANGLING VERBS FOR DATA CLEANING janitor::get_dupes() is used identifying duplicates id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE > get_dupes(data) No variable names specified - using all columns. # A tibble: 2 x 5 id a b c dupe_count <dbl> <chr> <dbl> <lgl> <int> 1 1 high 24 TRUE 2 2 1 high 24 TRUE 2
33. 3. DATA WRANGLING VERBS FOR DATA CLEANING janitor::get_dupes() is used identifying duplicates id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE > get_dupes(data, id) # A tibble: 2 x 5 id dupe_count a b c <dbl> <int> <chr> <dbl> <lgl> 1 1 2 high 24 TRUE 2 1 2 high 24 TRUE
34. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::distinct() is used for removing duplicates id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE id a b c 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE
35. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::select() is used for subsetting columns id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE id a b 1 high 24 1 high 24 2 low 67 3 low 89
36. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::select() is also used for reordering columns id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE id c a b 1 TRUE high 24 1 TRUE high 24 2 FALSE low 67 3 TRUE low 89
37. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::filter() is used for subsetting observations id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE id a b c 2 low 67 FALSE 3 low 89 TRUE
38. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::mutate() is used for creating new variables id a b c 1 high 24 TRUE 1 high 24 TRUE 2 low 67 FALSE 3 low 89 TRUE id a b c d 1 high 24 TRUE TRUE 1 high 24 TRUE TRUE 2 low 67 FALSE FALSE 3 low 89 TRUE FALSE
39. 3. DATA WRANGLING VERBS FOR DATA CLEANING stringr::str_detect() is used with mutate() to search within strings id name b c 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE 2 Lost River 67 FALSE 3 Highland River 89 TRUE id name b c d 1 Highland Creek 24 TRUE TRUE 1 Highland Creek 24 TRUE TRUE 2 Lost River 67 FALSE FALSE 3 Highland River 89 TRUE TRUE
40. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::arrange() is used to re-order observations id name b c 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE 2 Lost River 67 FALSE 3 Highland River 89 TRUE id name b c 3 Highland River 89 TRUE 2 Lost River 67 FALSE 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE
41. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::group() is used to create groups that can be summarized id name b c 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE 2 Lost River 67 FALSE 3 Highland Creek 89 TRUE id name b c 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE 3 Highland Creek 89 TRUE id name b c 2 Lost River 67 FALSE
42. 3. DATA WRANGLING VERBS FOR DATA CLEANING dplyr::summarize() is used to then summarize grouped data id name b c 1 Highland Creek 24 TRUE 1 Highland Creek 24 TRUE 2 Lost River 67 FALSE 3 Highland Creek 89 TRUE name n mean_b Highland Creek 3 77.6 Lost River 1 67
43. 4 BACK 
 MATTER
44. AGENDA REVIEW 4. BACK MATTER 2. Types of Data 3. Data Wrangling
45. REMINDERS 4. BACK MATTER Final project data posted; additional vignettes forthcoming Next Week: PS-02 (from last few weeks), Lab-06 (from today) Two Weeks: PS-03 (from today), Lab-07 (from next week), final project check-in, video lectures over spring break