News Applications at ProPublica - GDA/CAF Seminar, Quito, 2013

Transcript

1. News Applications at ProPublica Al Shaw al.shaw@propublica.org @A_L http://j.mp/shaw-gda-2013
2. ProPublica
3. Born in 2008 2 Pulitzer Prizes
4. Tools & Data
5. News Applications Desk http://propublica.org/nerds
6. Editor Deputy Editor Developers & Fellows
7. Editor Deputy Editor Developers Designers Reporters
8. Data Stories Graphics News Apps
9. Data Stories Graphics News Apps Gov’t Web Users
10. Gov’t Web Users The Opportunity Gap http://projects.propublica.org/schools
11. Gov’t US Department of Education 60,000+ schools 6,000+ districts districts self-report data
12. Gov’t Get data Clean data Analyze data Present data (news app!)
13. Look for “preposterousness” Counts & Totals Limits of Excel & MySQL Absurd max/mins Blanks vs. nulls Misspellings Data types (ask for a record layout) Bad geocoding, duplicate city names Check against reports & hard copy Call, don’t assume Do random spot checks
14. Data Bulletproofing: Jen LaFleur http://j.mp/data-bulletproofing Schools Methodology http://j.mp/schools-methodology
15. News App By Jennifer LaFleur, Al Shaw, Sharona Coutts and Jeff Larson, ProPublica, Updated January 24, 2013 This database includes all public schools in districts with more than 3,000 students from the 2009-2010 school year -- about three-qu country. Use it to find out how well your state provides poor and wealthier schools equal access to advanced classes that researchers s Our latest data also includes AP pass rates and sports participation | Related: About the Data and Our Analysis ». ...at providing students these programs across all income levels... SPORTS AP (PASSING) AP CLASSES GIFTED/TALENTED ADVANCED MATH PHYSICS CHEMISTRY Ala. Alaska Ariz. Ark. Calif. Colo. Conn. D.C. N/A Del. Fla. Ga. Hawaii Idaho Ill. Ind. Iowa Kan. Ky. La. Maine Mass. Md. Mich. ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT How States Compare Connect schools y get stats Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL Find a School Latest Local Stories Based o Univision: Problemas Escolares: Escue Reprueban Virginian-Pilot: In Norfolk, newest tea Boise State Public Radio: Students Wh Do Well In Idaho Faribault Daily News: Faribault School AP opportunities Shawnee Dispatch: As student bodies d have trouble finding minority candidat California Watch: Low-income student NBC New York: AP Opportunity Gap: N in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are In Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area distric state averages ProPublica intern Sergio Hernandez c project. Source: U.S. Department of Education The Opportunity Gap Is Your State Providing Equal Access to Education? RESET Home Our Investigations MuckReads Get Involved About Us Search ProPubl Tools & Data Don't Miss: Fracking Dollars for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assis Journalism in the Public Interest Journalism in the Public Interest Email address Receive Telling a story with software instead of words and pictures
16. Lede By Jennifer LaFleur, Al Shaw, Sharona Coutts and Jeff Larson, ProPublica, Updated January 24, 2013 This database includes all public schools in districts with more than 3,000 students from the 2009-2010 school year -- about three-qu country. Use it to find out how well your state provides poor and wealthier schools equal access to advanced classes that researchers s Our latest data also includes AP pass rates and sports participation | Related: About the Data and Our Analysis ». ...at providing students these programs across all income levels... SPORTS AP (PASSING) AP CLASSES GIFTED/TALENTED ADVANCED MATH PHYSICS CHEMISTRY Ala. Alaska Ariz. Ark. Calif. Colo. Conn. D.C. N/A Del. Fla. Ga. Hawaii Idaho Ill. Ind. Iowa Kan. Ky. La. Maine Mass. Md. Mich. ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT How States Compare Connect schools y get stats Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL Find a School Latest Local Stories Based o Univision: Problemas Escolares: Escue Reprueban Virginian-Pilot: In Norfolk, newest tea Boise State Public Radio: Students Wh Do Well In Idaho Faribault Daily News: Faribault School AP opportunities Shawnee Dispatch: As student bodies d have trouble finding minority candidat California Watch: Low-income student NBC New York: AP Opportunity Gap: N in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are In Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area distric state averages ProPublica intern Sergio Hernandez c project. Source: U.S. Department of Education The Opportunity Gap Is Your State Providing Equal Access to Education? RESET Home Our Investigations MuckReads Get Involved About Us Search ProPubl Tools & Data Don't Miss: Fracking Dollars for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assis Journalism in the Public Interest Journalism in the Public Interest Email address Receive Nut Far
17. na Coutts and Jeff Larson, ProPublica, Updated January 24, 2013 hools in districts with more than 3,000 students from the 2009-2010 school year -- about three-quarters of all such students in the your state provides poor and wealthier schools equal access to advanced classes that researchers say will help them later in life. | ss rates and sports participation | Related: About the Data and Our Analysis ». N/A ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT re Connect your Foursquare account to find schools you've checked into, and instantly get stats about schools when you check in. Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL or 77054 or Stuyvesant High Find a School Latest Local Stories Based on This Project Univision: Problemas Escolares: Escuelas en Philadelphia Reprueban Virginian-Pilot: In Norfolk, newest teachers face tougher tasks Boise State Public Radio: Students Who Take Advanced Courses Do Well In Idaho Faribault Daily News: Faribault School District needs to improve AP opportunities Shawnee Dispatch: As student bodies diversify, school districts have trouble finding minority candidates California Watch: Low-income students score lower on AP tests NBC New York: AP Opportunity Gap: NY's Poor Students Enroll in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are Investigating the Toledo Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area districts stack up well against state averages ProPublica intern Sergio Hernandez contributed research to this project. Source: U.S. Department of Education Office for Civil Rights nity Gap qual Access to Education? Tweet Tweet 128 RESET Submit Submit MuckReads Get Involved About Us Search ProPublica ools & Data for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assisted Living Journalism in the Public Interest Journalism in the Public Interest Email address SUBSCRIBE Receive our top stories daily DONATE “Do something” box Near
18. News apps generate stories
19. News apps generate stories
21. Gov’t Web Users Dollars for Docs http://projects.propublica.org/docdollars
22. Web 15 Pharmaceutical Companies 2 million doctor payments Scraped HTML, XML, SWF , PDF , other formats
23. Web Scraping “There is no data on the Internet that is actually impossible to download” — Dan Nguyen http://j.mp/coders-cause
24. Web Scraping “There is no data on the Internet that is actually impossible to download” — Dan Nguyen http://j.mp/coders-cause
25. Web Scraping “The trouble is, PDF was not designed as a data format.” — Jeremy Merrill http://j.mp/d4d-2013
26. Web Scraping “The trouble is, PDF was not designed as a data format.” — Jeremy Merrill http://j.mp/d4d-2013
27. Code! Upton Easy web scraping Tabula Turn PDFs into data http://github.com/propublica/upton http://github.com/jazzido/tabula
28. Gov’t Web Users Free the Files http://projects.propublica.org/free-the-files
29. Users 43,000 filings 17,000 “verified” through crowdsourcing
31. 4 simple questions that users agree on
32. Build a map of political TV spending
33. Users How to get them interested? “Casino Driven Design” http://j.mp/casino-driven-design
34. “Casino Driven Design” http://j.mp/casino-driven-design
35. Users “Casino Driven Design” http://j.mp/casino-driven-design
37. Code! Transcribable Make any Rails app a crowdsourced app http://github.com/propublica/transcribable
38. Casino Included! Transcribable
39. Data Stories Graphics News Apps Gov’t Web Users
40. Stories Graphics News Apps
44. Thank you! Al Shaw al.shaw@propublica.org @A_L