Scraping Data That Doesn't Want to be Scraped

Talk for BarCampBrighton7. Code on github:hrickards.

Transcript

1. Exams Data Harry Rickards (@harryrickards) Or: Scraping Data That Doesn't Want to be Scraped
2. 'Open' Data "Idea that certain data should be freely available to everyone" (Wikipedia)
3. 'Open' Data
4. Not-so-open Data DfE Examinations Timetable Scraping --- "a computer software technique of extracting information from websites" (Wikipedia)
5. Two main parts: • Finding information • Extracting information But, both can be done in similar ways (in this case at least).
6. Initial Problems Only get 10 results at a time. Isn't a problem - we can just scrape all of the thousands of pages.
7. Flow • Start at the entry page • Accept T&Cs • On the season filter form, iterate through all • Click on link to see all subjects • Iterate through every subject • Iterate through every page and scrape it
8. Mechanize: • Can pretend we're a browser • Multi-language (Ruby, Perl, Python) • Really simple to use Mechanize
9. browser = Mechanize.new { |b| b.user_agent_alias = "Linux Firefox" } page = browser.get BASE_TIMETABLE_URL + "/Default.aspx"
10. ASP.Net Buttons • Form with an input in • JS detects when user clicks input • Finds mouse coordinates and posts these
11. page.form_with(:name => 'aspnetForm') do |f| f['ctl00$mainContent$accept.x'] = 50 f['ctl00$mainContent$accept.y'] = 19 scrape_all_exam_sessions f.submit() end This is okay. Can inspect POST headers, and copy values from there.
12. Iterating through Sessions Iterating through all the sessions (exam months) is fairly simple. Get values from select box: page.form_with(:name => 'search'). field_with(:name => 'UCBasicSearch$ddSession'). options
13. sessions.each_with_index do |session, index| page.form_with(:name => 'search') do |f| f['UCBasicSearch$ddSession'] = session f['UCBasicSearch$showAllSub.x'] = 150 f['UCBasicSearch$showAllSub.y'] = 24 scrape_exam_subjects f.submit(), index end end e.g. session = 'ac6a501b-2f3b-4638-af29-48c8075a5fb9'
14. Click on Link to See All Subjects f['UCBasicSearch2$UCSearchByLetter$btnAll'] = 'All'
15. Each link is just an input, so it's a little bit harder to get a list of them. Introducing... XPath Iterating through Subjects
16. XPath 101 XPath - query language for XML documents (like HTML). Only identifier is name. e.g. UCBasicSearch2$UCPostBackSubjectList$subjectsRepeater$ctl 00$SubjectList$ctl01$subjectName
17. //input - all input elements //input [@name='UCBasicSearch2$UCPostBackSubjectList$subj ectsRepeater$ctl00$SubjectList$ctl01$subjectName'] - all input elements with name UCBa...
18. Turns out all have a name of the format UCBasicSearch2$UCPostBackSubjectList$subjectsRepeater $<stuff>$SubjectList$<stuff>$subjectName contains(@attr, 'value') rather than @attr='value' //input[ contains(@name, "UCBasicSearch2$UCPostBackSubjectList$subjectsRepeater") and contains(@name, "SubjectList") and contains(@name, "subjectName") ]
19. Running XPaths $x("EXPR") in Firebug/Chrome Inspector Can then run this in Mechanize with just subjects = page.parser.xpath(LONG_XPATH_HERE)
20. eg. UCBasicSearch2$UCPostBackSubjectList$subjectsRepeater$ ctl00$SubjectList$ctl01$subjectName' => 'Accounting' subjects.each do |subject| page.form_with(:name => 'search') do |f| f[subject['name']] = subject['value'] page = f.submit() scrape_results_page page, subject['value'] end end
21. next_button = page.parser.xpath( "//input[ @name='UCResultsTable$topNavigation$btnNext' ]" ).first if next_button page.form_with(:name => 'search') do |f| f['UCResultsTable$topNavigation$btnNext'] = 'Next >' scrape_results_page_link f.submit(), subject_name end end
22. results = page.parser.xpath("//table[@class='results']").first results = results.xpath( '//table[@id="UCResultsTable_resultsTbl"] //tr[td/@class = "results noprint"]' ) Parsing the Data
23. results.map! { |row| row.xpath("td[not(input)]") results.map! { |row| row.map { |cell| cell.text.strip.lstrip } ["January 2009 [Final]", "OCR", "GCE", "Accounting: Management Accounting:Written Paper", "F003/01", "1h 30m", "16 January 2009", "Friday pm"]
24. ASP.Net POSTS a variable called __VIEWSTATE, containing some sort of view state string (hence the name). /wEPDwUJMzQxMjMzMDAyD2QWAgIHD2QWBAIBD2QWBgIBDxAPFgYeDkRhdGFWYWx1ZUZpZWxkBQJJZB4NRGF0YVRleHRGaWVsZAULU2Vzc2lvbk5hbWUeC1 8hRGF0YUJvdW5kZ2QQFRMMSmFudWFyeSAyMDA5Ck1hcmNoIDIwMDkLU3VtbWVyIDIwMDkNTm92ZW1iZXIgMjAwOQxKYW51YXJ5IDIwMTAKTWFyY2ggMjAx MAtTdW1tZXIgMjAxMA1Ob3ZlbWJlciAyMDEwDEphbnVhcnkgMjAxMQpNYXJjaCAyMDExC1N1bW1lciAyMDExDU5vdmVtYmVyIDIwMTEMSmFudWFyeSAyMDEyCk 1hcmNoIDIwMTILU3VtbWVyIDIwMTINTm92ZW1iZXIgMjAxMgxKYW51YXJ5IDIwMTMKTWFyY2ggMjAxMwtTdW1tZXIgMjAxMxUTJDg1YTczYzU2LWExM2YtNDEz Zi05YTQ2LTg0ZTUxYjNmNWNkOCRhYzZhNTAxYi0yZjNiLTQ2MzgtYWYyOS00OGM4MDc1YTVmYjkkODllOWU3NGQtYjZlOC00ZWY0LWFkZTMtOGM1YWVlNmQ0Y jJkJDQ0YzIwNDI1LTQ1YWEtNDZhMS1iNmVhLTZjNjkzMjE2N2FmMCQwZDkxYjczYS0wODY3LTRhYzctYTY2Yy0zYTBjNmU5M2IyMDQkZjUxNTI5ODAtNmNlNC00 M2U3LThiNjEtYzljYTA3YmZkYWFlJDI1MDVmNWM5LWEyZjMtNGQyZi04MzhjLTI2NjczMzQxMmE1ZiRiY2NlMTAyMC1hZTQ0LTQzOTItYWM0Yi00ZjgwNDFjODZjO DIkZTFiNDRlYjgtZGIzYi00MzE2LWFlNTQtMWY4ZmMzYzA1YWRlJDcxYTUwNjUxLWIxYjQtNDdmYy05OGFlLWUxNjMzNTE5Mjg1NyQ5ODI2MDVkMi0wMjg2LTQwO DgtYmEwOC0xNTk0Yjc4NWYyMDIkMWE0ZjkxOTMtZjNiZC00ZTViLWFhOWQtNWY4ZGMzNGJkOTkzJDgyMGRlYmI1LTA4MGUtNDJmYy05YTUyLTc4M2ZiMmU5N mE0MyQ5MGIzNjI4Zi1jNDAwLTQ1NWYtYmE5Yy04ODI5OTlmNDliNzgkMzM0NjljZDItZTU4MC00ZWNlLTk0NmYtYTA4ZDMyYmMzNzYzJGQ4ZjcyZDE5LWU3MGM tNDMyYy1iOTAzLTUxNzVkZDU0NDUyZiQ2ZWIxNjlmNC0zMjk4LTRlYWEtOWU2NS1iZTc0NzZjNmYxZjEkMGM0ZDljNGUtMmY5OC00NDg2LWIzZTItMWE1YzYyN GI5YmI3JDc0NTNjNmRjLWE5ZDgtNGI0Yi1iNGRhLWM0NDJkZjI1ZDExYhQrAxNnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZGQCAw8QDxYGHwAFD1F1YWxpZmljYXRpb25 JZB8BBRFRdWFsaWZpY2F0aW9uTmFtZR8CZ2QQFQkDQWxsA0FFQQNFTEMERlNNUQNHQ0UER0NTRQRHTlZRBFNURVADVkNFFQkDQWxsJDc4ODE3ZTI2LWFlY WYtNDFlMy1hMmQ5LTMyOTcxYjY2N2Q2ZCQ0M2M4ZmU4Ni1jMjkyLTQ4N2EtYjA0MS1mYzE3MDlmNjQ2OTIkNDgxNTkzNWYtNzBlMy00OTAwLWI4ZmEtNDI4ZD IxMGMwNjY4JDM5OTczZTRmLTdlYzUtNGVkNS1iOWE0LTU4NmEyZmQzMzU1ZCQxNTVmNjljYS05NzMzLTQ5MjAtOWNjZC1mYmJhNDEwNTkyOTEkYjNkZjRjZmI tYTRjMS00NWQ5LTk4NzEtMmZlYjQwYWY4NThiJDM2MWViMTQ0LTk2M2MtNDdlMS1hOWFmLTY5MGQzMWJmOGUzOCQ4Y2MxNGVlMS0xNjQ5LTQwNDEtOGEw NC1lMWYxNjA3ZDkzY2YUKwMJZ2dnZ2dnZ2dnZGQCBQ8QDxYGHwAFDkF3YXJkaW5nQm9keUlkHwEFEEF3YXJkaW5nQm9keU5hbWUfAmdkEBUGA0FsbANBUU EEQ0NFQQdFZGV4Y2VsA09DUgRXSkVDFQYDQWxsJGI4ZWMzNDFmLTc2YWUtNGZmNi04MzhkLTgwNjk3MGRhM2E5YiRhNDYxYTJkMC0zZGE2LTQwMDEtYTViM C1jMjY4NDUzMjYzZDQkMDllZmM2NzAtODI1NC00OWU5LTg2MzgtMGM5NmFhMjQ4ZGVhJDZhZTYzNDg1LTdkMzgtNDI1Yy1hNWE2LTc4MjE4NTVmZGJjNCQwM TRkNjk5Ny1lNzM0LTQ2NjItOWVhZS01Y2QxOTEwOGY0OTcUKwMGZ2dnZ2dnZGQCBQ9kFgYCAw8WAh4HVmlzaWJsZWhkAgcPFgIfA2hkAhMPZBYGAgEPZBYCZ g9kFgRmD2QWAgIBDxBkZBYBZmQCAQ9kFgICAQ9kFgJmD2QWFgIBDw8WBB4PQ29tbWFuZEFyZ3VtZW50BQItMR8DaGRkAgMPDxYEHwQFAi0xHwNoZGQCBQ 8PFgIfA2hkZAIHDw8WBh4IQ3NzQ2xhc3MFDGxpbmtpbmFjdGl2ZR4EXyFTQgICHwNoZGQCCQ8PFgYfBQUEbGluax8GAgIfA2hkZAILDw8WBh8FBQRsaW5rHwYC Ah8DaGRkAg0PDxYGHwUFBGxpbmsfBgICHwNoZGQCDw8PFgYfBQUEbGluax8GAgIfA2hkZAIRDw8WBB8EBQExHwNoZGQCEw8PFgQfBAUBMR8DaGRkAhUPDxY CHwNoZGQCBw88KwALAGQCDQ9kFgJmD2QWFgIBDw8WBB8EBQItMR8DaGRkAgMPDxYEHwQFAi0xHwNoZGQCBQ8PFgIfA2hkZAIHDw8WBh8FBQxsaW5raW5 hY3RpdmUfBgICHwNoZGQCCQ8PFgYfBQUEbGluax8GAgIfA2hkZAILDw8WBh8FBQRsaW5rHwYCAh8DaGRkAg0PDxYGHwUFBGxpbmsfBgICHwNoZGQCDw8PFg YfBQUEbGluax8GAgIfA2hkZAIRDw8WBB8EBQExHwNoZGQCEw8PFgQfBAUBMR8DaGRkAhUPDxYCHwNoZGQYAQUeX19Db250cm9sc1JlcXVpcmVQb3N0QmFja 0tleV9fFgIFGFVDQmFzaWNTZWFyY2gkc2hvd0FsbFN1YgUaVUNCYXNpY1NlYXJjaCRzZWFyY2hieUNvZGUb0RAeB276TxZcEFhfGWejGPUrFw==
25. • Web-based platform for scraping • Ruby/Python/more • Documentation + library for data storage • Query language for accessing scraped data ScraperWiki
26. Rewired State