Improving Data Gathering And Research

How to improve data gathering using web scraping methodologies.

Transcript

1. RESEARCH IMPROVING DATA GATHERING & Luca Matteis
2. What is Research?
3. "In the broadest sense of the word, the definition of research includes any gathering of data, information and facts for the advancement of knowledge."
4. "Research is a process of steps used to collect and analyze information to increase our understanding of a topic or issue"
5. Data is essential for research
6. Where do we get data from? Einstein got his data from his own experiments and from other peoples experiments Information exchange took weeks if not months
7. Today we have the internet! Information exchange takes milliseconds Works much better than anything Einstein had
8. BUT THERE’S STILL ISSUES
9. DATA IS SCATTERED ALL OVER THE WEB
10. http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/ themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
11. Information that can be extremely valuable, lives somewhere online and we don’t know it because we can’t find it
12. EVEN WITH GOOGLE, IT’S STILL HARD TO FIND WHAT WE NEED
13. Scientific data searching is facilitated if there is a central repository or data bank
14. http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/ themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
15. When our information is centralized by context, we can more easily find what we’re looking for
16. We already have websites that centralize this information
17. And allow us to find data that Google couldn’t
18. BUT THERE’S ROOM FOR IMPROVEMENT
19. How is this data currently being centralized?
20. Each center sends us their data in the form of Excel or Access files, through FTP or Email
22. THIS IS AN ENTIRELY MANUAL PROCESS
23. Is this sustainable?
24. Is this sustainable? This process needs to be automated
25. • no human interference • less communication hassles • less human errors • more accurate data • more data What are the advantages of automating the data exchange process?
26. How do we automate? Centers no longer have to send us anything. We get it directly from their website
27. There’s no secret. Google, hotel sites, flight search engines and many others do this It is called web scraping
28. How does it work
29. We automatically navigate to the centers websites and fetch the information that we need
30. We automatically navigate to the centers websites and fetch the information that we need This is done by little scripts called spiders or web crawlers
31. What? Spiders?
32. “A Web crawler (or spider) is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.”
34. This process allows us to reach more centers and gather more data
35. For each center to have a website that displays their information The main requirement Without a website we wouldn’t be able to automate this exchange
36. Working prototype http://seeds.iriscouch.com/
37. Working prototype http://seeds.iriscouch.com/ PASSPORT DATA
38. Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION
39. Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION OTHER...
40. RECAP
41. RECAP Automation of the data exchange process is the only sustainable solution
42. RECAP Automation of the data exchange process is the only sustainable solution With new technologies, web scraping has become a very reliable system
43. RECAP Automation of the data exchange process is the only sustainable solution With new technologies, web scraping has become a very reliable system The process is modular and will allow us to plug-in systems such as GRIN-Global
44. THANK YOU