Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Improving Data Gathering And Research
Search
Luca Matteis
November 26, 2011
Programming
160
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Improving Data Gathering And Research
How to improve data gathering using web scraping methodologies.
Luca Matteis
November 26, 2011
Transcript
44 slides
RESEARCH IMPROVING DATA GATHERING & Luca Matteis
What is Research?
"In the broadest sense of the word, the definition of
research includes any gathering of data, information and facts for the advancement of knowledge."
"Research is a process of steps used to collect and
analyze information to increase our understanding of a topic or issue"
Data is essential for research
Where do we get data from? Einstein got his data
from his own experiments and from other peoples experiments Information exchange took weeks if not months
Today we have the internet! Information exchange takes milliseconds Works
much better than anything Einstein had
BUT THERE’S STILL ISSUES
DATA IS SCATTERED ALL OVER THE WEB
http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/
themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
Information that can be extremely valuable, lives somewhere online and
we don’t know it because we can’t find it
EVEN WITH GOOGLE, IT’S STILL HARD TO FIND WHAT WE
NEED
Scientific data searching is facilitated if there is a central
repository or data bank
http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/
themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
When our information is centralized by context, we can more
easily find what we’re looking for
We already have websites that centralize this information
And allow us to find data that Google couldn’t
BUT THERE’S ROOM FOR IMPROVEMENT
How is this data currently being centralized?
Each center sends us their data in the form of
Excel or Access files, through FTP or Email
None
THIS IS AN ENTIRELY MANUAL PROCESS
Is this sustainable?
Is this sustainable? This process needs to be automated
• no human interference • less communication hassles • less
human errors • more accurate data • more data What are the advantages of automating the data exchange process?
How do we automate? Centers no longer have to send
us anything. We get it directly from their website
There’s no secret. Google, hotel sites, flight search engines and
many others do this It is called web scraping
How does it work
We automatically navigate to the centers websites and fetch the
information that we need
We automatically navigate to the centers websites and fetch the
information that we need This is done by little scripts called spiders or web crawlers
What? Spiders?
“A Web crawler (or spider) is a computer program that
browses the World Wide Web in a methodical, automated manner or in an orderly fashion.”
None
This process allows us to reach more centers and gather
more data
For each center to have a website that displays their
information The main requirement Without a website we wouldn’t be able to automate this exchange
Working prototype http://seeds.iriscouch.com/
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION OTHER...
RECAP
RECAP Automation of the data exchange process is the only
sustainable solution
RECAP Automation of the data exchange process is the only
sustainable solution With new technologies, web scraping has become a very reliable system
RECAP Automation of the data exchange process is the only
sustainable solution With new technologies, web scraping has become a very reliable system The process is modular and will allow us to plug-in systems such as GRIN-Global
THANK YOU
More Decks by Luca Matteis
See All by Luca Matteis
Linked Open Data
lmatteis
1
150
What I do
lmatteis
1
93
Crop Ontology
lmatteis
1
110
Why NPM rocks!
lmatteis
2
310
Informatics Development Tools
lmatteis
0
120
Other Decks in Programming
See All in Programming
React Native経験者が Flutterを30⽇間触って驚いたこと
3saito
0
190
GraphQL の onError と Nullability のこれから
quramy
0
220
baseline으로 마이그레이션, 리팩토링 안전하게 맡기기
fornewid
1
990
コード生成時代のAPI定義を支えるRequest Spec移行
pvcresin
0
180
まずはコードより業務と向き合う:テーブル解体のためにPdMを呼んだ話
sakawe_ee
0
240
Seven New Principles of Software Engineering
yegor256
0
500
AIがコードを書いてくれる時代、我々は何を確かめるのか
yosuke_furukawa
PRO
13
6k
止まらない比較を止める — 余帰納法で無限ストリームを比べる
tarao
0
180
あなたはMCPの機能を、まだ知らない。
4geru
1
350
社内AIエージェント「Askワンバン」における コストチューニングの取り組み
t0yohei
0
170
Будущее локальных LLM – Алексей Голобурдин, PythoNN
sobolevn
0
120
Spec-Driven Development - Why Vaadin Is the Perfect Fit
simas
PRO
0
250
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
23k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.9k
More Than Pixels: Becoming A User Experience Designer
marktimemedia
3
620
Designing for humans not robots
tammielis
254
26k
Efficient Content Optimization with Google Search Console & Apps Script
katarinadahlin
PRO
1
990
My Coaching Mixtape
mlcsv
0
390
Collaborative Software Design: How to facilitate domain modelling decisions
baasie
1
380
The browser strikes back
jonoalderson
0
1.9k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
uxyall
2
530
How GitHub (no longer) Works
holman
316
150k
What Being in a Rock Band Can Teach Us About Real World SEO
427marketing
0
1.1k
Bootstrapping a Software Product
garrettdimon
PRO
306
120k