IRE 2012 Unstructured Data Talk

Transcript

1. Analyzing Unstructured Data for Stories eugenewu@mit.edu
2. What Am I Talking About? •  Example •  Structured Data 101 •  Structured Data Continuum •  More Examples
3. http://projects.propublica.org/drywall/
4. http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
5. 1056. Plaintiffs - Intervenors, Robert and Tasha Lambert are citizens of Alabama and together own real property located at 541 Lynn Hurst Court, Montgomery, Alabama 36117. Plaintiffs are participating as class representatives in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a citizen of Alabama and owns real property located at 2105 Lane Avenue, Birmingham, Alabama 35217. Plaintiff is participating as a class representative in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1058. Plaintiffs-Intervenors, Daniel and Nicole Smith are citizens of Alabama and together own real property located at 766 Tabernacle Road, Monroeville, Alabama http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
6. 1056. Plaintiffs - Intervenors, Robert and Tasha Lambert are citizens of Alabama and together own real property located at 541 Lynn Hurst Court, Montgomery, Alabama 36117. Plaintiffs are participating as class representatives in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a citizen of Alabama and owns real property located at 2105 Lane Avenue, Birmingham, Alabama 35217. Plaintiff is participating as a class representative in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1058. Plaintiffs-Intervenors, Daniel and Nicole Smith are citizens of Alabama and together own real property located at 766 Tabernacle Road, Monroeville, Alabama http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
7. 541 Lynn Hurst Court, Montgomery, Alabama 36117 37.0625, -95.677068
8. 541 Lynn Hurst Court, Montgomery, Alabama 36117 37.0625, -95.677068 Jefferson County
9. http://projects.propublica.org/drywall/
10. http://projects.propublica.org/drywall/
11. Scanned Documents Addresses Google Maps
12. Unstructured Information Structured Data Visualization Scanned Documents Addresses Google Maps
13. Unstructured Information Structured Data Visualization Scanned Documents Addresses Google Maps
14. Unstructured Information Structured Data Visualization Scanned Documents Addresses Google Maps
15. Unstructured Information Structured Data Visualization Scanned Documents Addresses Google Maps Who cares? What is it?
16. Who Cares? Software Visualization Mashups Store Databases PANDA Analyze Fusion tables Excel Databases R/Python/Ruby
17. Who Cares? Software Visualization Mashups
18. Who Cares? Software Visualization Mashups Tainted House Data + Economic Data + Health Stats + Crime Stats + Corruption Data
19. Structured Data
20. Structured Data Attribute Name Data type Consistent
21. Structured Data Attribute Name Data type Consistent
22. Structured Data Attribute Name Data type Consistent
23. Structured Data Attribute Name Data type Consistent Florida’s Lee County has 1518 addresses
24. Structured Data Attribute Name Data type Consistent
25. Structured Data Attribute Name Data type Consistent
26. Structured Data Attribute Name Data type Consistent Numeric (integers, dollars,…) Date/Time Lat, Lon
27. Structured Data Attribute Name Data type Consistent Numeric (integers, dollars,…) Date/Time Lat, Lon Structured strings (Florida)
28. Structured Data Attribute Name Data type Consistent FLORIDA FL Flroida FloridaState Florida’s
29. Structured Data Attribute Name Data type Consistent FLORIDA FL Flroida FloridaState Florida’s 5 10 1 1 1
30. Structured Data Attribute Name Data type Consistent
31. What Am I Talking About? •  Structured Data 101 •  Structured Data Continuum •  More Examples
32. structured unstructured Continuum
33. structured unstructured Images Images http://www.whatisstephenharperreading.ca/2010/03/01/book-number-76-one-day-in-the-life-of-ivan-denisovich-by-alexander-solzhenitsyn/
34. structured unstructured Images Images Text Blob 1056. Plaintiffs - Intervenors, Robert and Tasha Lambert are citizens of Alabama and together own real property located at 541 Lynn Hurst Court, Montgomery, Alabama 36117. Plaintiffs are participating as class representatives in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a citizen of Alabama and owns real property located at 2105 Lane Avenue, Birmingham, Alabama 35217. Plaintiff is participating as a class representative in the
35. structured unstructured Images Images Text Blob Email
36. structured unstructured Images Images Text Blob Email Re: IRE conference in Boston June 1, 3:08PM jaimi@ire.org Subject Date From
37. structured unstructured Images Images Text Blob Email Excel
38. structured unstructured Images Text Blob Email Excel
39. “It’s sunny in texas” structured unstructured Images Text Blob Email Excel
40. “It’s sunny in texas” Tweet Weather Location It’s sunny in texas Sunny Texas structured unstructured Images Text Blob Email Excel
41. “It’s sunny in texas” Tweet Weather Location It’s sunny in texas Sunny (37.06, -95.67) structured unstructured Images Text Blob Email Excel
42. You have unstructured data What structure do I need? Attributes with simple types When Ask Find
43. What Am I Talking About? •  Structured Data 101 •  Structured data continuum •  More Examples
44. http://www.boston.com/news/politics/specials/obama_state_of_the_union_word_cloud/ 2011 State of the Union
45. Type/Meaning String Name Word
46. Mr. Speaker, Mr. Vice President, members of Congress, distinguished guests, and fellow Americans: Tonight I want to begin by congratulating the men and women of the 112th Congress, as well as your new Speaker, John Boehner. And as we mark this occasion, we're also mindful of the empty chair in this chamber, and we pray for the health of our colleague -- and our friend -- Gabby Giffords. It's no secret that those of us here tonight have had our differences over the last two years. The debates have been contentious; we have fought
47. Mr. Speaker, Mr. Vice President, members of Congress, distinguished guests, and fellow Americans: Tonight I want to begin by congratulating the men and women of the 112th Congress, as well as your new Speaker, John Boehner. And as we mark this occasion, we're also mindful of the empty chair in this chamber, and we pray for the health of our colleague -- and our friend -- Gabby Giffords. It's no secret that those of us here tonight have had our differences over the last two years. The debates have been contentious; we have fought Word Mr Speaker Vice President Members Congress Distinguished Guests Americans People Jobs New years
48. Bin Laden Tweets/Sec http://www.flickr.com/photos/twitteroffice/5681263084/
49. Type/Meaning Time Name Time
50. Deadly Day in Baghdad http://www.nytimes.com/interactive/2010/10/24/world/1024-surge-graphic.html?pagewanted=all
51. Type/Meaning Lat, Lon Number Name Location Body Count
52. http://www.nytimes.com/interactive/world/iraq-war-logs.html?pagewanted=all
53. http://www.nytimes.com/interactive/world/iraq-war-logs.html?pagewanted=all 14, 12 Killed in Action Lat Lon
54. Sentiment of NZ Earthquake http://twitinfo.csail.mit.edu/detail/4/
55. Type/Meaning -1 to 1 Name Happiness
56. !
57. Pattern Matching Great, 7AM meeting 7:00AM
58. Interpret Meaning Great, 7AM meeting Not Happy
59. Interpret Meaning It’s still new Great, 7AM meeting Happy!
60. Interpret Meaning It’s still new
61. Interpret Meaning It’s still new Lack of context Earthquakes
62. Extracting meaning is by far the most difficult
63. What if it’s just unstructured?
64. CrowdSourcing Lots of humans do tasks computers suck at Training Quality Issues
65. Dealing with Forms
66. Dealing with Forms
67. Entity Information
68. Pattern Matching •  Regex – Describe and find patterns – Killed in action (?P<n>\d{1,3})(\s[A-Z]{1,3})?\sKIA
69. DBTruck demo?
70. Structure = Super Valuable
71. You have unstructured data What structure do I need? Attributes with simple types When Ask Find Structure = Super Valuable
72. You have unstructured data What structure do I need? Attributes with simple types When Ask Find tinyurl.com/iredatatipsheet eugenewu@mit.edu @sirrice Structure = Super Valuable