Crowdsourcing Historical Research

Presented at Drupal Downunder 2012.

Transcript

1. Crowdsourcing Historical Research Claudine Chionh Drupal Downunder 2012
2. Founders and Survivors • Study of the 73,000 convicts transported to Van Diemen's Land (Tasmania) between 1803 and 1853 • Records from the convict system and elsewhere • Health, environment, lifestyle, wellbeing • Effects on health and resilience of descendants http://foundersandsurvivors.org/
3. Goals of the project • Compile (health and demographic) data about this population from a range of sources • Enable other researchers to use this data • Explore quantitative and geographic tools and analyses that are not commonly used in historical research • Combine professional expertise with the enthusiasm of volunteers
4. Some research projects • Morbidity and mortality on the voyage to Australia • Crime and convicts in Tasmania, 1853-1900 • Fertility decline in late C19 Tasmania • Prostitution and female convicts • Tracing convicts' descendants who served in WWI http://foundersandsurvivors.org/research
5. Project staff • Historians • Demographers • Epidemiologists • Two part-time developers
6. Who are our users? • Research team • Other interested researchers • Genealogists/family historians • Local historians
7. Data sources • Conduct records • Surgeons' journals • Newspaper reports • Births, deaths, marriages • Parish records • Family histories, memories, legends
8. Official/formal sources Records from the convict system • Trial, conviction documents • Conduct records • Ship surgeons' journals • Permissions to marry • Ticket of leave Outside the convict system • Births, deaths, marriages • Later convictions
9. Paper databases Broader historical context: • Mass transportation • Modern record-keeping and statistics
10. Informal sources • Newspaper reports • Family history: primary sources, compiled genealogies, anecdote and legend
11. Our volunteers • Amateur historians, genealogists • Librarians • IT specialists
12. How volunteers can contribute • Individual convict biographies • Tracing batches of convicts in ships
13. Solutions • XML database • Drupal • Google Docs
14. The Founders and Survivors database • XML (based on Text Encoding Initiative http://www.tei-c.org/) • BaseX XML database engine http://basex.org/
15. Experimenting with Drupal • Used an older version of Migrate to import some tabular data as nodes • Problem of scale: 73,000 convicts • XML approach proved to be more efficient
16. Getting data into our system Formal sources • Collected by archives and individual researchers • CSV, Excel, Filemaker, Access ... • Incorporated into BaseX database with Perl scripts Informal sources • Individual convicts' life histories are captured in a Drupal content type ('Community contributed content') • Some sub-projects also capture summary data in Google spreadsheets
17. Viewing data • Master database in BaseX: presented in XSLT, different views for logged in researchers and others • Community contributed content (CCC): Drupal • Two-way link between master database and CCC • Google spreadsheets prepopulated with links to corresponding records in master database
18. Data capture • Convict biographies captured in Drupal – Community Contributed Content (CCC) • Linked to entry in XML database • Perl scripts to incorporate CCC records into master database
19. XML entry for an individual convict
20. Prepopulated Drupal form
21. Community contributed content
22. Ships (batches of data) • Tracing all convicts on a ship • Summary data in Google Spreadsheets • Spreadsheets are prepopulated from the master database
23. Ship summary data in Google Spreadsheets
24. Drupal can't do everything • Scale • Complexity • Expertise
25. Where Drupal is appropriate for our project • Web frontend • Data capture • Collaboration, forums
26. Summary • Massive XML database with complex relations • Drupal for capturing slightly complex data and facilitating collaboration • Google Spreadsheets for capturing tabular data
27. Questions? Founders and Survivors http://foundersandsurvivors.org/ inquiries@foundersandsurvivors.org Claudine Chionh http://www.onefewercar.net/ claudine@chionh.org