Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Opening Government with Python

Avatar for jamesturk jamesturk
September 26, 2011
Programming
4
200

Opening Government with Python

OpenStates technical presentation

Avatar for jamesturk

jamesturk

September 26, 2011
Tweet

More Decks by jamesturk

See All by jamesturk
Open States TCamp11 Presentation
Avatar for jamesturk jamesturk
1
56

Other Decks in Programming

See All in Programming
検索機能リプレイスを4ヶ月→2ヶ月に！ AI Agentで実現した2倍速リプレイス
Avatar for Maple fuuki12
3
590
Testing Trophyは叫ばない
Avatar for toms toms74209200
0
910
GoogleI/O2025 LT報告会資料
Avatar for shinsuke-fujita shinsukefujita1126
0
150
HTMLの品質ってなんだっけ？ “HTMLクライテリア”の設計と実践
Avatar for una unachang113
5
3.1k
麻雀点数計算問題生成タスクから学ぶ Single Agentの限界と Agentic Workflowの底力
Avatar for po3rin po3rin
4
1.3k
開発者への寄付をアプリ内課金として実装する時の気の使いどころ
Avatar for Yunosuke Sakai ski
0
110
プロポーザル駆動学習 / Proposal-Driven Learning
Avatar for mackey0225 mackey0225
2
1.4k
請來的 AI Agent 同事們在寫程式時，怎麼用 pytest 去除各種幻想與盲點
Avatar for Keith Yang keitheis
0
140
チームのテスト力を鍛える
Avatar for Hiroki Iseri goyoki
3
1.1k
Android端末で実現するオンデバイスLLM 2025
Avatar for Masayuki Suda masayukisuda
1
210
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
Avatar for Taito Tanaka taitotnk
0
890
Deep Dive into Kotlin Flow
Avatar for Matsuda Jumpei jmatsu
1
400

Featured

See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
7
850
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
51k
KATA
Avatar for Megan Lloyd mclloyd
32
14k
Done Done
Avatar for chrislema chrislema
185
16k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
368
19k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
3
83
It's Worth the Effort
Avatar for 3n 3n
187
28k
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
110k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
237
140k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
8
550

Transcript

  1. http://openstates.sunlightlabs.com these slides: http://goo.gl/c7U94

  2. & lots more... Federal Open Gov Ecosystem

  3. State Open Gov Ecosystem

  4. PyCon 2009 OpenGov Hackathon

  5. Scraping the 50 States • Large volunteer effort ◦ 3500

    commits ◦ 34 contributors ◦ 20,000 LoC ▪ (Python)

  6. Start to Finish • Page Retrieval: scrapelib • Parsing: lxml.html

    • Validation: validictory • Storage: pymongo (mongoDB) • Clean up: name_tools, jellyfish • API: Django, django-piston, boundaryservice

  7. Scraping: scrapelib • retrieving HTML from pages # instantiate a

    scraper object, objects keep their own cache/etc. import scrapelib s = scrapelib.Scraper(requests_per_minute=10,allow_cookies=True, follow_robots=True, retry_attempts=2, retry_wait_seconds=5) # use urlopen as you would use urllib2's # useable as context manager to deal with scrape errors with s.urlopen('http://example.com') as html: # result object is string-like html[0:6] == '<html>' # also has a response attribute with more info html.response.code == 200 html.response.headers['server'] == 'Apache 2.0.63' https://github.com/sunlightlabs/scrapelib

  8. Parsing: lxml.html • parsing and extracting information from HTML #

    parse HTML into a lxml tree >>> doc = lxml.html.fromstring(html) # possible to use css selectors (all <a> children of <p> tags) >>> links = doc.cssselect('p > a') >>> links[0].get('href') 'http://example.com/link/0' # can also use xpath to dive deep doc.xpath('//h4[text()="TEST"]/following-sibling::ul[1]/li/a[1]') much less code compared to fragile BeautifulSoup quite a bit faster in most cases http://lxml.de/

  9. Validation: validictory • validation of python data structures ◦ based

    on http://json-schema.org spec schema = {"type": "object", "properties": { "state": {"type": "string", "minLength": 2, "maxLength": 2}, "session": {"type": "string"}, "chamber": {"type": "string", "enum": ["upper", "lower"]}, "bill_id": {"type": "string"}, "type": {"type": "array", "items": {"type": "string"}} }} bill = {'state':'nc', 'session': '2010', 'chamber': 'senate', 'bill_id': 4, 'type': ['bill']} validictory.validate(bill, schema) # would raise exception http://github.com/sunlightlabs/validictory

  10. Name Matching: name_tools # split names into (title, first, last,

    suffix) >>> name_tools.split("President Barack Hussein Obama II") ('President', 'Barack Hussein', 'Obama', 'II') >>> name_tools.split("Obama, President Barack H., II") ('President', 'Barack H', 'Obama', 'II') >>> name_tools.split("Fleet Admiral William F. Halsey, Jr., USN") ('Fleet Admiral', 'William F', 'Halsey', 'Jr., USN') # score name similarity >>> name_tools.match('Eric Schmidt', 'Eric Schmidt') 1.0 >>> name_tools.match('Bob Dole', 'Dole, Bob') 0.97999999999999998 >>> name_tools.match('Jeff Tweedy', 'J Tweedy') 0.90000000000000002 >>> name_tools.match('Ferris Bueller', 'Bueller') 0.80000000000000002 http://github.com/mikejs/name_tools

  11. Name Matching: jellyfish • string comparison ◦ Levenshtein & Damerau-Levenshtein

    distance ◦ Jaro & Jaro-Winkler distance ◦ Match Ratching Approach ◦ Hamming Distance • Phonetic encoding ◦ American soundex ◦ Metaphone ◦ NYSIIS ◦ Match Rating Codex

  12. Name Matching: jellyfish # string comparison >>> jellyfish.jaro_distance('barak', 'barack') 0.94444444

    >>> jellyfish.levenshtein_distance('barak', 'barack') 1 # soundalikes >>> jellyfish.soundex('payne') P500 >>> jellyfish.soundex('pain') P500 >>> jellyfish.metaphone('jellyfish') JLFX >>> jellyfish.metaphone('gellyfish') JLFX http://github.com/sunlightlabs/jellyfish

  13. Project Status • Public beta of our API ◦ http://openstates.sunlightlabs.com/api/

    • 10 states "ready" and 11 "experimental" ◦ MD, LA, CA, TX, WI, NJ, VT, MN, UT, NC ◦ NV, SD, WA, AK, AZ, MS, FL, VA, PA, OH, DC • Lots of data ◦ 3,300 legislators ◦ 110,000 bills ◦ 900,000 actions ◦ 91,000 votes • new states every month

  14. photo http://www.flickr.com/photos/gottgraphicsdesign/4543701893/ CC- BY

  15. Use the data • Get Sunlight API Key @ http://services.sunlightlabs.com/

    • API Docs: http://openstates.sunlightlabs.com/api/ • RESTful JSON-based API ◦ state metadata ◦ bills, legislators, committees, events ▪ search by attribute or lookup by ID ◦ legislator lookup by lat+long • python-openstates http://github.com/sunlightlabs/python-openstates openstates.Legislator.search(state='ca', first_name='Mike') openstates.Bill.search('agriculture', state='vt')

  16. Adopt a State ◦ Contributor Guide ▪ http://openstates.sunlightlabs.com/contributing/ ◦ Join

    the Google Group ▪ http://goo.gl/M3An ◦ Code on GitHub: ▪ http://github.com/sunlightlabs/openstates We're a great project to get started with if you want to brush up on web scraping.

  17. Bonus Content: Other Sunlight APIs • Sunlight Labs APIs: http://services.sunlightlabs.com/

    ◦ Sunlight Congress API - info on all members of congress ◦ TransparencyData API - campaign finance information ◦ Real Time Congress API - bills, real time updates, etc. ◦ and of course, Open States API • python libraries for all of these except RTC • same key works across all APIs • virtually no limits on usage ("be reasonable") • API users mailing list: http://goo.gl/KYlKy

  18. Thanks! http://openstates.sunlightlabs.com/ Additional Questions? Google Group: http://goo.gl/M3An [email protected] @jamesturk this

    presentation: http://goo.gl/c7U94