Data Wrangling: Getting Started Working with Data for Visualization

These are the slides from Rachel Shadoan's OSBridge talk on June 26, 2014.

Good data visualization allows us to leverage the incredible pattern-recognition abilities of the human brain to answer questions we care about. But how do you make a good visualization? Here's a crash course.

Transcript

1. DATA WRANGLING
2. Rachel Shadoan @rachelshadoan @akashiclabs
3. What do you need for a data visualization?
4. Data
5. A Spectrum of Data Quantitative Qualitative Nominal •City •Gender Ordinal •Months •Seasons •Agreement Interval •Latitude •Longitude •Dates Ratio •Amount •Age •Height
6. Another Spectrum of Data Structured Unstructured •Books •Blogs •Video •Images •Music •Databases •Spreadsheets •Etc
7. What is structure? Unstructured Structured
8. Unstructured Data Examples: blog posts books, images, video Not easily searchable Has no consistent underlying organization
9. Structured Data Searchable Has a consistent underlying organization Examples: spreadsheets, .csv files, xml files, databases
10. What Can Be Visualized? Quantitative Qualitative Unstructured Structured
11. What Can Be Visualized? Quantitative Qualitative Unstructured Structured
12. What Can Be Visualized? Quantitative Qualitative Unstructured Structured
13. What Can Be Visualized? Quantitative Qualitative Unstructured Structured
14. Unstructured data can be transformed into structured data
15. What Can Be Visualized? Quantitative Qualitative Unstructured Structured
16. ANY DATA CAN BE STRUCTURED.
17. ANY DATA CAN BE VISUALIZED.
18. Structuring Data: Abstract Data Models This is a person. This is person data model, which is an abstraction of a person. o Person ID o Name o Age o Height o Profession o Hobbies o Nationality o Native Language Person
19. An OSBridge Attendee Data Model • Name • Email address • Home address • Company/organization • Twitter/identi.ca • Website • Age • Gender • Years in open source • Food preference • Favorite language • Current projects • Favorite color
20. One Rule for Transforming Unstructured Data into Structured Data CONSISTENCY. CONSISTENCY. CONSISTENCY.
21. What you need for data visualization: Structured Data
22. But what do you need for a good data visualization?
23. The Dimensionality Problem
24. A dimension is a variable in the data • Name • Email address • Home address • Company/organization • Twitter/identi.ca • Website • Age • Gender • Years in open source • Food preference • Favorite language • Current projects • Favorite color These are all dimensions of the data
25. Dimensionality Increases Quickly Email Home address Company/Org Twitter Website Age Gender Years in OS Food preference Favorite language Current projects Favorite color 78 pairwise relationships!
26. Questions Reduce Dimensionality Email Home address Company/Org Twitter Website Age Gender Years in OS Food preference Favorite language Current projects Favorite color How are food preferences distributed among languages and projects?
27. Questions Reduce Dimensionality Email Home address Company/Org Twitter Website Age Gender Years in OS Food preference Favorite language Current projects Favorite color How are languages and projects distributed geographically?
28. Multiple comparison problem The more relationships you look at, the more likely you are to find a pattern that only exists because of random chance
29. The View: The Smallest Unit of Visualization
30. The View: The Smallest Unit of Visualization
31. The View: The Smallest Unit of Visualization
32. The View: The Smallest Unit of Visualization
33. Why is reducing dimensionality especially important for visualization?
34. Dimension count: 1 Date dimension encoded as position along the horizontal axis
35. Dimension count: 2 Time dimension encoded as position along the vertical axis
36. Dimension count: 3 Type of training dimension encoded as color
37. Dimension count: 4 Number of stat points gained from training dimension encoded as bubble size
38. So what do you need for a good data visualization?
39. What you need: Structured Data Questions
40. Choosing the right views for your data
41. Quantitative Data
42. Spatial (Location) Data
43. Temporal (Time) Data
44. Relational (Network) Data
45. What if one view isn’t enough?
46. Multiple Coordinated Views to the rescue!
47. Filtering, Brushing + Linking
48. Focus + Context This view shows the context, or overview of the data
49. Focus + Context This view shows the details of the data selected in the context view Selection
50. Tools
51. D3.js Limited support for multiple coordinated views, large data sets Beautiful browser- based visualizations JavaScript visualization library
52. Improvise Few resources for learning Java-based desktop design environment Powerful, flexible tool for creating multiple-coordinated view visualizations
53. Python Great for data wrangling and rapid prototyping Check out MatPlotLib and ggplot visualization libaries
54. Go forth and visualize!
55. Credits Liz Mc, CC 2.0, via Flickr Kent K. Barns, CC 2.0, kentkb.com Jerome Collins, CC 2.0, via Flickr Cristinacosta, CC 2.0, via Flickr Thinking Machine 4, Mid- game, by Martin Wattenberg and Marek Walczak