Open Data Quality Dashboard – NHTG14

Presentation of our (@ElliotJH, @danpalmer) hack at National Hack the Government 2014.

Transcript

1. Government Open Data Quality Dashboard National Hack the Government 2014
2. Elliot Hughes @ElliotJH Dan Palmer @danpalmer “That Southampton Lot” You may remember us from such hacks as Greedy MPs, Hillsborough Unlocked,! Rate Your Member, Medical Now,! Insulate Me, One Nation Under CCTV
3. Why do we need to do this?
4. Climate_change_and_ transport_choices.sav
5. Climate_change_and_ transport_choices.sav
6. Climate_change_and_ transport_choices.wtf
7. Climate_change_and_ transport_choices.wtf “SPSS is used for statistical analysis, initially released in 1968”
8. Climate_change_and_ transport_choices.wtf “SPSS is used for statistical analysis, initially released in 1968” “…can only be used on the platform that created the file…”
9. viewfile.ashx
10. CloudStore - May 2012 cat export (comma delimited - text string comma escaped with backslash - three header rows).csv
11. coreaccessindicators2008.6
12. Possible Uses
13. Possible Uses • Government File Name or Geocities website game
14. Possible Uses • Government File Name or Geocities website game • Flashcard or Top-Trumps “my data is better than your data” game
15. Possible Uses • Government File Name or Geocities website game • Flashcard or Top-Trumps “my data is better than your data” game • This has probably already been done.
16. Possible Uses • Government File Name or Geocities website game • Flashcard or Top-Trumps “my data is better than your data” game • This has probably already been done. • Twitter bot
17. Possible Uses • Government File Name or Geocities website game • Flashcard or Top-Trumps “my data is better than your data” game • This has probably already been done. • Twitter bot • Open Data Quality Metrics
18. 1. Get all the data from data.gov.uk for all Ministerial Departments.
19. 2. Get all the data from gov.uk for all Ministerial Departments.
20. 2. Get all the data from gov.uk for all Ministerial Departments. They don’t have an API for that.
21. 3. Validate the data
22. 4. LEADERBOARDS!
24. 35%
25. *.csv 35%
27. 40%
28. https 40%
30. 9%
31. Not hosted on *.gov.uk 9%
33. 16%
34. index.htm index.php http://somewhere.gov.uk/ 16%
36. 8.5%
37. Unreachable 8.5%
38. Total indirectly linked or unreachable 25%
39. test_results_2010.txt.gz
40. test_results_2010.txt.gz “Anonymised MOT tests and results”
41. test_results_2010.txt.gz “Anonymised MOT tests and results” 750MB
42. test_results_2010.txt.gz “Anonymised MOT tests and results” 750MB 3.3GB uncompressed
43. 1044521|362808|2010-07-12|2|N|P|34414|TN|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1200|1948-01-01! 1044532|362815|2010-05-15|4|N|P|61618|LE|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|0|1998-08-01! 1044546|362824|2010-09-03|4|PL|P|54070|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044547|362824|2010-09-01|4|N|F|54065|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044580|362842|2010-01-13|4|N|F|91834|SA|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2380|1998-11-30! 1044591|362851|2010-02-27|1|N|F|0|PE|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|125|1970-01-01! 1044592|362852|2010-09-03|4|N|PRS|53897|SL|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1597|1998-03-01! 1044595|362853|2010-09-03|2|N|F|7764|ST|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1200|2005-09-04! 1044620|362872|2010-09-08|4|N|F|82997|GL|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|1275|1997-01-01! 1044653|297603|2010-03-10|1|N|PRS|779|WF|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|150|1962-01-01! 1044684|362918|2010-03-15|4|N|PRS|34169|CV|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1248|2004-07-02! 1044692|362923|2010-09-06|4|N|F|70286|S|UNCLASSIFIED|UNCLASSIFIED|GREY|D|2490|1997-04-01! 1044770|362974|2010-10-14|2|N|PRS|18440|LA|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|350|1996-11-28! 1044797|287632|2010-06-29|1|N|F|23426|BH|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|149|1958-01-01! 1044819|363009|2010-03-04|4|N|F|145955|RH|UNCLASSIFIED|UNCLASSIFIED|RED|P|1215|1998-10-27! 1044822|363011|2010-08-05|4|N|F|123795|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1987-06-18! 1044828|363015|2010-04-15|4|N|P|119891|LE|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2000|1996-05-01! 1044897|363063|2010-12-16|4|N|F|97105|RG|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|1998|1988-01-01! 1044911|363072|2010-05-12|4|N|F|111694|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2500|1999-01-01! 1044937|362747|2010-01-21|5|N|F|84748|DE|UNCLASSIFIED|UNCLASSIFIED|WHITE|D|2402|2001-09-01! 1044951|363101|2010-03-30|4|N|F|98311|NP|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|2000|1999-11-30! 1044954|363103|2010-05-01|2|N|F|0|BH|UNCLASSIFIED|UNCLASSIFIED|RED|P|25|1999-11-11! 1044955|363104|2010-03-02|4|N|P|185807|N|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1596|2004-04-30! 1044966|281511|2010-02-06|1|N|F|194|LL|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|125|1959-06-01! 1045025|363159|2010-07-20|4|N|F|61718|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|3200|1999-01-01! 1045042|363170|2010-03-04|4|N|F|90210|BS|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|0|1991-07-01! 1045053|363179|2010-07-30|1|N|P|1|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|125|1964-01-01! 1045113|363222|2010-02-06|1|N|P|8079|S|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|150|1968-01-01! 1045122|363228|2010-06-07|4|N|ABR|0|GU|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1995|2006-05-29! 1045172|363259|2010-09-27|4|N|PRS|77777|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|2495|1996-04-04! 1045197|363277|2010-04-19|4|F|P|178744|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045198|363277|2010-03-22|4|N|F|178741|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045253|363319|2010-06-03|4|N|F|1365|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1986-06-26! 1045281|363339|2010-08-27|4|N|F|26477|NG|UNCLASSIFIED|UNCLASSIFIED|RED|P|4600|2006-03-01! 1045315|363367|2010-02-23|4|N|F|56355|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|2200|1997-08-01!
44. 1044521|362808|2010-07-12|2|N|P|34414|TN|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1200|1948-01-01! 1044532|362815|2010-05-15|4|N|P|61618|LE|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|0|1998-08-01! 1044546|362824|2010-09-03|4|PL|P|54070|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044547|362824|2010-09-01|4|N|F|54065|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044580|362842|2010-01-13|4|N|F|91834|SA|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2380|1998-11-30! 1044591|362851|2010-02-27|1|N|F|0|PE|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|125|1970-01-01! 1044592|362852|2010-09-03|4|N|PRS|53897|SL|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1597|1998-03-01! 1044595|362853|2010-09-03|2|N|F|7764|ST|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1200|2005-09-04! 1044620|362872|2010-09-08|4|N|F|82997|GL|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|1275|1997-01-01! 1044653|297603|2010-03-10|1|N|PRS|779|WF|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|150|1962-01-01! 1044684|362918|2010-03-15|4|N|PRS|34169|CV|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1248|2004-07-02! 1044692|362923|2010-09-06|4|N|F|70286|S|UNCLASSIFIED|UNCLASSIFIED|GREY|D|2490|1997-04-01! 1044770|362974|2010-10-14|2|N|PRS|18440|LA|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|350|1996-11-28! 1044797|287632|2010-06-29|1|N|F|23426|BH|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|149|1958-01-01! 1044819|363009|2010-03-04|4|N|F|145955|RH|UNCLASSIFIED|UNCLASSIFIED|RED|P|1215|1998-10-27! 1044822|363011|2010-08-05|4|N|F|123795|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1987-06-18! 1044828|363015|2010-04-15|4|N|P|119891|LE|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2000|1996-05-01! 1044897|363063|2010-12-16|4|N|F|97105|RG|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|1998|1988-01-01! 1044911|363072|2010-05-12|4|N|F|111694|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2500|1999-01-01! 1044937|362747|2010-01-21|5|N|F|84748|DE|UNCLASSIFIED|UNCLASSIFIED|WHITE|D|2402|2001-09-01! 1044951|363101|2010-03-30|4|N|F|98311|NP|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|2000|1999-11-30! 1044954|363103|2010-05-01|2|N|F|0|BH|UNCLASSIFIED|UNCLASSIFIED|RED|P|25|1999-11-11! 1044955|363104|2010-03-02|4|N|P|185807|N|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1596|2004-04-30! 1044966|281511|2010-02-06|1|N|F|194|LL|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|125|1959-06-01! 1045025|363159|2010-07-20|4|N|F|61718|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|3200|1999-01-01! 1045042|363170|2010-03-04|4|N|F|90210|BS|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|0|1991-07-01! 1045053|363179|2010-07-30|1|N|P|1|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|125|1964-01-01! 1045113|363222|2010-02-06|1|N|P|8079|S|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|150|1968-01-01! 1045122|363228|2010-06-07|4|N|ABR|0|GU|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1995|2006-05-29! 1045172|363259|2010-09-27|4|N|PRS|77777|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|2495|1996-04-04! 1045197|363277|2010-04-19|4|F|P|178744|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045198|363277|2010-03-22|4|N|F|178741|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045253|363319|2010-06-03|4|N|F|1365|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1986-06-26! 1045281|363339|2010-08-27|4|N|F|26477|NG|UNCLASSIFIED|UNCLASSIFIED|RED|P|4600|2006-03-01! 1045315|363367|2010-02-23|4|N|F|56355|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|2200|1997-08-01! WHAT FORMAT IS THIS?
45. 1044521|362808|2010-07-12|2|N|P|34414|TN|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1200|1948-01-01! 1044532|362815|2010-05-15|4|N|P|61618|LE|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|0|1998-08-01! 1044546|362824|2010-09-03|4|PL|P|54070|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044547|362824|2010-09-01|4|N|F|54065|SO|UNCLASSIFIED|UNCLASSIFIED|RED|D|1870|1997-04-01! 1044580|362842|2010-01-13|4|N|F|91834|SA|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2380|1998-11-30! 1044591|362851|2010-02-27|1|N|F|0|PE|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|125|1970-01-01! 1044592|362852|2010-09-03|4|N|PRS|53897|SL|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1597|1998-03-01! 1044595|362853|2010-09-03|2|N|F|7764|ST|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1200|2005-09-04! 1044620|362872|2010-09-08|4|N|F|82997|GL|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|1275|1997-01-01! 1044653|297603|2010-03-10|1|N|PRS|779|WF|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|150|1962-01-01! 1044684|362918|2010-03-15|4|N|PRS|34169|CV|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1248|2004-07-02! 1044692|362923|2010-09-06|4|N|F|70286|S|UNCLASSIFIED|UNCLASSIFIED|GREY|D|2490|1997-04-01! 1044770|362974|2010-10-14|2|N|PRS|18440|LA|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|350|1996-11-28! 1044797|287632|2010-06-29|1|N|F|23426|BH|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|149|1958-01-01! 1044819|363009|2010-03-04|4|N|F|145955|RH|UNCLASSIFIED|UNCLASSIFIED|RED|P|1215|1998-10-27! 1044822|363011|2010-08-05|4|N|F|123795|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1987-06-18! 1044828|363015|2010-04-15|4|N|P|119891|LE|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2000|1996-05-01! 1044897|363063|2010-12-16|4|N|F|97105|RG|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|1998|1988-01-01! 1044911|363072|2010-05-12|4|N|F|111694|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|2500|1999-01-01! 1044937|362747|2010-01-21|5|N|F|84748|DE|UNCLASSIFIED|UNCLASSIFIED|WHITE|D|2402|2001-09-01! 1044951|363101|2010-03-30|4|N|F|98311|NP|UNCLASSIFIED|UNCLASSIFIED|SILVER|P|2000|1999-11-30! 1044954|363103|2010-05-01|2|N|F|0|BH|UNCLASSIFIED|UNCLASSIFIED|RED|P|25|1999-11-11! 1044955|363104|2010-03-02|4|N|P|185807|N|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|1596|2004-04-30! 1044966|281511|2010-02-06|1|N|F|194|LL|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|125|1959-06-01! 1045025|363159|2010-07-20|4|N|F|61718|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|3200|1999-01-01! 1045042|363170|2010-03-04|4|N|F|90210|BS|UNCLASSIFIED|UNCLASSIFIED|GREEN|P|0|1991-07-01! 1045053|363179|2010-07-30|1|N|P|1|CT|UNCLASSIFIED|UNCLASSIFIED|WHITE|P|125|1964-01-01! 1045113|363222|2010-02-06|1|N|P|8079|S|UNCLASSIFIED|UNCLASSIFIED|CREAM|P|150|1968-01-01! 1045122|363228|2010-06-07|4|N|ABR|0|GU|UNCLASSIFIED|UNCLASSIFIED|BLUE|D|1995|2006-05-29! 1045172|363259|2010-09-27|4|N|PRS|77777|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|2495|1996-04-04! 1045197|363277|2010-04-19|4|F|P|178744|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045198|363277|2010-03-22|4|N|F|178741|NW|UNCLASSIFIED|UNCLASSIFIED|BLACK|P|1799|1999-02-12! 1045253|363319|2010-06-03|4|N|F|1365|NG|UNCLASSIFIED|UNCLASSIFIED|GREEN|D|0|1986-06-26! 1045281|363339|2010-08-27|4|N|F|26477|NG|UNCLASSIFIED|UNCLASSIFIED|RED|P|4600|2006-03-01! 1045315|363367|2010-02-23|4|N|F|56355|WA|UNCLASSIFIED|UNCLASSIFIED|SILVER|D|2200|1997-08-01! WHAT FORMAT IS THIS? wtf? wtf? wtf? wtf? wtf? wtf?
46. HEAD /file.csv
47. HEAD /file.csv 405 Method Not Supported 405 Method Not Supported 405 Method Not Supported 500 Internal Server Error 500 Internal Server Error
48. How do we want data? • Atom feeds - 1 • CSV - 35% • JSON - 1 • RDF - 34 • Valid • UTF-8 or ASCII
49. Demo
51. Better than it’s ever been.
52. Better than it’s ever been.
53. Better than it’s ever been. Still a long way to go…