Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Improving Data Gathering And Research
Search
Luca Matteis
November 26, 2011
Programming
2
130
Improving Data Gathering And Research
How to improve data gathering using web scraping methodologies.
Luca Matteis
November 26, 2011
Tweet
Share
More Decks by Luca Matteis
See All by Luca Matteis
Linked Open Data
lmatteis
1
110
What I do
lmatteis
1
68
Crop Ontology
lmatteis
1
71
Why NPM rocks!
lmatteis
2
280
Informatics Development Tools
lmatteis
0
94
Other Decks in Programming
See All in Programming
The Niche of CDK Grant オブジェクトって何者?/the-niche-of-cdk-what-isgrant-object
hassaku63
1
700
Understanding Kotlin Multiplatform
l2hyunwoo
0
150
バイブコーディング超えてバイブデプロイ〜CloudflareMCPで実現する、未来のアプリケーションデリバリー〜
azukiazusa1
2
710
202507_ADKで始めるエージェント開発の基本 〜デモを通じて紹介〜(奥田りさ)
risatube
PRO
5
860
AI コーディングエージェントの時代へ:JetBrains が描く開発の未来
masaruhr
2
210
チームのテスト力を総合的に鍛えて品質、スピード、レジリエンスを共立させる/Testing approach that improves quality, speed, and resilience
goyoki
6
1.2k
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
1
220
階層化自動テストで開発に機動力を
ickx
1
420
The Evolution of Enterprise Java with Jakarta EE 11 and Beyond
ivargrimstad
0
470
ソフトウェア設計とAI技術の活用
masuda220
PRO
25
6.7k
MySQL9でベクトルカラム登場!PHP×AWSでのAI/類似検索はこう変わる
suguruooki
1
220
リッチエディターを安全に開発・運用するために
unachang113
1
240
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
109
19k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Why Our Code Smells
bkeepers
PRO
337
57k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
Documentation Writing (for coders)
carmenintech
72
4.9k
The Cost Of JavaScript in 2023
addyosmani
51
8.6k
Git: the NoSQL Database
bkeepers
PRO
431
65k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Gamification - CAS2011
davidbonilla
81
5.4k
We Have a Design System, Now What?
morganepeng
53
7.7k
Transcript
RESEARCH IMPROVING DATA GATHERING & Luca Matteis
What is Research?
"In the broadest sense of the word, the definition of
research includes any gathering of data, information and facts for the advancement of knowledge."
"Research is a process of steps used to collect and
analyze information to increase our understanding of a topic or issue"
Data is essential for research
Where do we get data from? Einstein got his data
from his own experiments and from other peoples experiments Information exchange took weeks if not months
Today we have the internet! Information exchange takes milliseconds Works
much better than anything Einstein had
BUT THERE’S STILL ISSUES
DATA IS SCATTERED ALL OVER THE WEB
http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/
themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
Information that can be extremely valuable, lives somewhere online and
we don’t know it because we can’t find it
EVEN WITH GOOGLE, IT’S STILL HARD TO FIND WHAT WE
NEED
Scientific data searching is facilitated if there is a central
repository or data bank
http://science.com/paper.... http://newton.com/research... http://national.com/ goo... http://biology.com/ science... http://newscientist.com/ neutrinodiscovery... http:// astronomynow.com/
themoon http://space.com/ november2001 http://science.com/ paper.... http://newton.com/research... http://science.com/paper.... http://space.com/astro... http://space.com/astro... http://space.com/astro... http://science.com/paper....
When our information is centralized by context, we can more
easily find what we’re looking for
We already have websites that centralize this information
And allow us to find data that Google couldn’t
BUT THERE’S ROOM FOR IMPROVEMENT
How is this data currently being centralized?
Each center sends us their data in the form of
Excel or Access files, through FTP or Email
None
THIS IS AN ENTIRELY MANUAL PROCESS
Is this sustainable?
Is this sustainable? This process needs to be automated
• no human interference • less communication hassles • less
human errors • more accurate data • more data What are the advantages of automating the data exchange process?
How do we automate? Centers no longer have to send
us anything. We get it directly from their website
There’s no secret. Google, hotel sites, flight search engines and
many others do this It is called web scraping
How does it work
We automatically navigate to the centers websites and fetch the
information that we need
We automatically navigate to the centers websites and fetch the
information that we need This is done by little scripts called spiders or web crawlers
What? Spiders?
“A Web crawler (or spider) is a computer program that
browses the World Wide Web in a methodical, automated manner or in an orderly fashion.”
None
This process allows us to reach more centers and gather
more data
For each center to have a website that displays their
information The main requirement Without a website we wouldn’t be able to automate this exchange
Working prototype http://seeds.iriscouch.com/
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION
Working prototype http://seeds.iriscouch.com/ PASSPORT DATA CHARACTERIZATION OTHER...
RECAP
RECAP Automation of the data exchange process is the only
sustainable solution
RECAP Automation of the data exchange process is the only
sustainable solution With new technologies, web scraping has become a very reliable system
RECAP Automation of the data exchange process is the only
sustainable solution With new technologies, web scraping has become a very reliable system The process is modular and will allow us to plug-in systems such as GRIN-Global
THANK YOU