Upgrade to Pro — share decks privately, control downloads, hide ads and more …

News Applications at ProPublica - GDA/CAF Seminar, Quito, 2013

4fc2d89264f76f56dc7d63d5f3ff1cbc?s=47 Al Shaw
September 19, 2013
220

News Applications at ProPublica - GDA/CAF Seminar, Quito, 2013

4fc2d89264f76f56dc7d63d5f3ff1cbc?s=128

Al Shaw

September 19, 2013
Tweet

Transcript

  1. News Applications at ProPublica Al Shaw al.shaw@propublica.org @A_L http://j.mp/shaw-gda-2013

  2. ProPublica

  3. Born in 2008 2 Pulitzer Prizes

  4. Tools & Data

  5. News Applications Desk http://propublica.org/nerds

  6. Editor Deputy Editor Developers & Fellows

  7. Editor Deputy Editor Developers Designers Reporters

  8. Data Stories Graphics News Apps

  9. Data Stories Graphics News Apps Gov’t Web Users

  10. Gov’t Web Users The Opportunity Gap http://projects.propublica.org/schools

  11. Gov’t US Department of Education 60,000+ schools 6,000+ districts districts

    self-report data
  12. Gov’t Get data Clean data Analyze data Present data (news

    app!)
  13. Look for “preposterousness” Counts & Totals Limits of Excel &

    MySQL Absurd max/mins Blanks vs. nulls Misspellings Data types (ask for a record layout) Bad geocoding, duplicate city names Check against reports & hard copy Call, don’t assume Do random spot checks
  14. Data Bulletproofing: Jen LaFleur http://j.mp/data-bulletproofing Schools Methodology http://j.mp/schools-methodology

  15. News App By Jennifer LaFleur, Al Shaw, Sharona Coutts and

    Jeff Larson, ProPublica, Updated January 24, 2013 This database includes all public schools in districts with more than 3,000 students from the 2009-2010 school year -- about three-qu country. Use it to find out how well your state provides poor and wealthier schools equal access to advanced classes that researchers s Our latest data also includes AP pass rates and sports participation | Related: About the Data and Our Analysis ». ...at providing students these programs across all income levels... SPORTS AP (PASSING) AP CLASSES GIFTED/TALENTED ADVANCED MATH PHYSICS CHEMISTRY Ala. Alaska Ariz. Ark. Calif. Colo. Conn. D.C. N/A Del. Fla. Ga. Hawaii Idaho Ill. Ind. Iowa Kan. Ky. La. Maine Mass. Md. Mich. ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT How States Compare Connect schools y get stats Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL Find a School Latest Local Stories Based o Univision: Problemas Escolares: Escue Reprueban Virginian-Pilot: In Norfolk, newest tea Boise State Public Radio: Students Wh Do Well In Idaho Faribault Daily News: Faribault School AP opportunities Shawnee Dispatch: As student bodies d have trouble finding minority candidat California Watch: Low-income student NBC New York: AP Opportunity Gap: N in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are In Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area distric state averages ProPublica intern Sergio Hernandez c project. Source: U.S. Department of Education The Opportunity Gap Is Your State Providing Equal Access to Education? RESET Home Our Investigations MuckReads Get Involved About Us Search ProPubl Tools & Data Don't Miss: Fracking Dollars for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assis Journalism in the Public Interest Journalism in the Public Interest Email address Receive Telling a story with software instead of words and pictures
  16. Lede By Jennifer LaFleur, Al Shaw, Sharona Coutts and Jeff

    Larson, ProPublica, Updated January 24, 2013 This database includes all public schools in districts with more than 3,000 students from the 2009-2010 school year -- about three-qu country. Use it to find out how well your state provides poor and wealthier schools equal access to advanced classes that researchers s Our latest data also includes AP pass rates and sports participation | Related: About the Data and Our Analysis ». ...at providing students these programs across all income levels... SPORTS AP (PASSING) AP CLASSES GIFTED/TALENTED ADVANCED MATH PHYSICS CHEMISTRY Ala. Alaska Ariz. Ark. Calif. Colo. Conn. D.C. N/A Del. Fla. Ga. Hawaii Idaho Ill. Ind. Iowa Kan. Ky. La. Maine Mass. Md. Mich. ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT How States Compare Connect schools y get stats Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL Find a School Latest Local Stories Based o Univision: Problemas Escolares: Escue Reprueban Virginian-Pilot: In Norfolk, newest tea Boise State Public Radio: Students Wh Do Well In Idaho Faribault Daily News: Faribault School AP opportunities Shawnee Dispatch: As student bodies d have trouble finding minority candidat California Watch: Low-income student NBC New York: AP Opportunity Gap: N in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are In Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area distric state averages ProPublica intern Sergio Hernandez c project. Source: U.S. Department of Education The Opportunity Gap Is Your State Providing Equal Access to Education? RESET Home Our Investigations MuckReads Get Involved About Us Search ProPubl Tools & Data Don't Miss: Fracking Dollars for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assis Journalism in the Public Interest Journalism in the Public Interest Email address Receive Nut Far
  17. na Coutts and Jeff Larson, ProPublica, Updated January 24, 2013

    hools in districts with more than 3,000 students from the 2009-2010 school year -- about three-quarters of all such students in the your state provides poor and wealthier schools equal access to advanced classes that researchers say will help them later in life. | ss rates and sports participation | Related: About the Data and Our Analysis ». N/A ...at enrolling and passing students in Advanced Placement classes... AP PASS RATE AP ENROLLMENT re Connect your Foursquare account to find schools you've checked into, and instantly get stats about schools when you check in. Your Address, ZIP, or school name For example, 1605 E. 55th St. Chicago, IL or 77054 or Stuyvesant High Find a School Latest Local Stories Based on This Project Univision: Problemas Escolares: Escuelas en Philadelphia Reprueban Virginian-Pilot: In Norfolk, newest teachers face tougher tasks Boise State Public Radio: Students Who Take Advanced Courses Do Well In Idaho Faribault Daily News: Faribault School District needs to improve AP opportunities Shawnee Dispatch: As student bodies diversify, school districts have trouble finding minority candidates California Watch: Low-income students score lower on AP tests NBC New York: AP Opportunity Gap: NY's Poor Students Enroll in Fewer College-Prep Courses StateImpact Ohio: Why the Feds are Investigating the Toledo Public Schools C-Ville Weekly: Feed your head The Reporter: Study shows area districts stack up well against state averages ProPublica intern Sergio Hernandez contributed research to this project. Source: U.S. Department of Education Office for Civil Rights nity Gap qual Access to Education? Tweet Tweet 128 RESET Submit Submit MuckReads Get Involved About Us Search ProPublica ools & Data for Docs Surveillance Patient Safety Prescriber Checkup Debt Inc. 990s Assisted Living Journalism in the Public Interest Journalism in the Public Interest Email address SUBSCRIBE Receive our top stories daily DONATE “Do something” box Near
  18. News apps generate stories

  19. News apps generate stories

  20. None
  21. Gov’t Web Users Dollars for Docs http://projects.propublica.org/docdollars

  22. Web 15 Pharmaceutical Companies 2 million doctor payments Scraped HTML,

    XML, SWF , PDF , other formats
  23. Web Scraping “There is no data on the Internet that

    is actually impossible to download” — Dan Nguyen http://j.mp/coders-cause
  24. Web Scraping “There is no data on the Internet that

    is actually impossible to download” — Dan Nguyen http://j.mp/coders-cause
  25. Web Scraping “The trouble is, PDF was not designed as

    a data format.” — Jeremy Merrill http://j.mp/d4d-2013
  26. Web Scraping “The trouble is, PDF was not designed as

    a data format.” — Jeremy Merrill http://j.mp/d4d-2013
  27. Code! Upton Easy web scraping Tabula Turn PDFs into data

    http://github.com/propublica/upton http://github.com/jazzido/tabula
  28. Gov’t Web Users Free the Files http://projects.propublica.org/free-the-files

  29. Users 43,000 filings 17,000 “verified” through crowdsourcing

  30. None
  31. 4 simple questions that users agree on

  32. Build a map of political TV spending

  33. Users How to get them interested? “Casino Driven Design” http://j.mp/casino-driven-design

  34. “Casino Driven Design” http://j.mp/casino-driven-design

  35. Users “Casino Driven Design” http://j.mp/casino-driven-design

  36. None
  37. Code! Transcribable Make any Rails app a crowdsourced app http://github.com/propublica/transcribable

  38. Casino Included! Transcribable

  39. Data Stories Graphics News Apps Gov’t Web Users

  40. Stories Graphics News Apps

  41. None
  42. None
  43. None
  44. Thank you! Al Shaw al.shaw@propublica.org @A_L