Open States TCamp11 Presentation

slides on OpenStates from TCamp 2011

Transcript

1. http://openstates.sunlightlabs.com these slides: http://goo.gl/l2IOk
2. & lots more... Open Gov Legislative Ecosystem
3. State Open Gov Legislative Ecosystem
7. PyCon 2009 OpenGov Hackathon
8. The "govtrack" role • Collect • Augment • Publish • Also working with Participatory Politics Foundation to make state-level OpenCongress-like site
9. Brute Force Solution • Have to scrape in most states • Large volunteer effort ◦ 3500 commits ◦ 35 contributors ◦ 20,000 LoC ▪ (Python)
10. Technical Approach: Scraping • main tools: python, lxml • derived libraries: ◦ scrapelib - intelligently fetch webpages ◦ validictory - validation of arbitrary data structures ◦ jellyfish - string metrics (fuzzy matching) • we provide an API for scrapers so they can write code like Bill('upper', 'HB 1', '2010 Budgetary Proposal') bill.add_action('lower', 'Introduced', '2010-01-20') • when run locally writes JSON to disk, when we run it can be imported to mongodb
11. Technical Approach: Backend • all scrapers are run via a Hudson instance ◦ can use EC2 instances on-demand • data goes into MongoDB • exposed via simple API and we push dumps to S3
12. Project Status • ~6 months ago public beta of our API ◦ http://openstates.sunlightlabs.com/api/ • 13 states "ready" and 11 "experimental" ◦ AK, CA, MD, MN, NJ, NC, OH, PA, TX, UT, VT, WI ◦ AZ, CT, FL, IN, MI, MS, NV, SD, VA, WA, DC • Lots of data ◦ 4012 legislators ◦ 123,239 bills ◦ 1,001,473 actions ◦ 96383 votes • All 50 by early 2012
13. photo credit http://www.flickr.com/photos/gottgraphicsdesign/4543701893/ CC-BY
14. Use the data • Get Sunlight API Key @ http://services.sunlightlabs.com/ • API Docs: http://openstates.sunlightlabs.com/api/ • RESTful JSON-based API ◦ state metadata ◦ bills, legislators, committees ▪ search by attribute or lookup by ID ◦ legislator lookup by lat+long • govkit ruby gem http://github.com/opengovernment/govkit • python-openstates http://github.com/sunlightlabs/python-openstates openstates.Legislator.search(state='ca', first_name='Mike') openstates.Bill.search('agriculture', state='vt')
15. Adopt a State • If you have python/scraping skills or want a good project to learn on ◦ Contributor Guide http://goo.gl/CceA ◦ Join the Google Group: http://goo.gl/M3An ◦ http://github.com/sunlightlabs/openstates
16. Thanks! http://openstates.sunlightlabs.com/ Additional Questions? Google Group: http://goo.gl/M3An jturk@sunlightfoundation.com @openstates @jamesturk this presentation: http://goo.gl/l2IOk