Managing Research Data with Fedora 4

Given at eResearch Australasia, October 30, 2014.

Transcript

1. Managing Research Data with Fedora 4 David Wilcox Fedora Product Manager DuraSpace
2. What is Fedora? • Flexible Extensible Durable Object Repository Architecture • Open Source digital repository software • Community-driven design and implementation • Fedora 4 is the new, revitalized version
3. “DuraSpace is an independent 501(c)(3) not-for-profit organization providing leadership and innovation for open technologies that promote durable, persistent access to digital data. We collaborate with academic, scientific, cultural, and technology communities by supporting projects and creating services to help ensure that current and future generations have access to our collective digital heritage.”
4. How is DuraSpace organized? Education Services Projects Funded through community sponsorship & RSP program Funded through service revenue & grants
5. Funding Sources
6. DuraSpace and Fedora • Leadership and planning • Budgeting - planning, support, administration • Staffing - hiring, oversight, coordination • Outreach, marketing, advocacy
7. 2014 Fedora Members (61) • Arizona State University Libraries • Brown University Library • Case Western Reserve University Libraries • Charles Darwin University • Colorado Alliance of Research Libraries (CARL) • Columbia University Library • Cornell University • Docuteam GmbH • Durham University • FIZ Karlsruhe • George Washington University • Ghent University Library • Gothenburg University Library • Indiana University • ICPSR • Johns Hopkins University Libraries • La Trobe University • London School of Economics & Political Science • LYRASIS • Macquarie University • National Library of Medicine • National Library of Wales / Llyfrgell Genedlaethol Cymru • National Research Council of Canada • Northeastern University Libraries • Northwestern University Libraries • Ohio State • Oregon State • Pennsylvania State University • Princeton University • Rutgers University Libraries • Smithsonian Institution, Office of Research Infomation Services • Stanford University • State and University Library of Denmark • Technical University of Denmark • The Art Institute of Chicago • Tufts University • University of Alberta • University of California, Los Angeles • University of California, Santa Barbara • University of Cincinnati • University of Connecticut Libraries • University of Hull • University of Lausanne • University of Manitoba • University of Massachusetts Amherst Libraries • University of New South Wales • University of Notre Dame • University of North Carolina • University of Oklahoma • University of Pittsburgh • University of Oxford • University of Prince Edward Island • University of Rochester Libraries • University of Texas Libraries Austin • University of Toronto • University of Virginia • University of Wisconsin • University of York • Uppsala University • Yale University • York University
8. Membership is on the rise! $0 $125,000 $250,000 $375,000 $500,000 2010 2011 2012 2013 2014
9. Building sustainability • We’re broadening the funding base • More members at lower funding amounts • Raising the overall level of funding • We’ve established a governance model • Fedora is governed by a Leadership group and a Steering group
11. Governance benefits Level of contribution Benefit $2,500 Contributors can nominate and elect one contributor at the $2,500 tier to join the Leadership Group $5,000 Contributors can nominate and elect two contributors at the $5,000 tier to join the Leadership Group $10,000 Contributors can nominate and elect four contributors at the $10,000 tier to join the Leadership Group $20,000 Contributors guaranteed one seat on the Leadership Group 0.5 FTE Developer Contributors guaranteed one seat on the Leadership Group
12. Fedora 4 project goals • Improved performance • Flexible storage options • Research data management • Linked open data support • Improved platform for developers
13. Research data requirements • Diverse formats • Complex relationships • Large file sizes • Many files • Long-term preservation and access • Others…
14. Fedora 4.0 feature set • Content modelling • Authorization • Durable Storage • Versioning • Scale (large files and many files) • Linked data / RDF (and external triplestore) • External search • Transactions • Performance • Clustering
15. Hydra and Islandora • Popular Fedora front-end applications • Both will support Fedora 4 • PSU and UCSD Beta pilots are Hydra-based • Currently organizing code sprints for Islandora integration
16. Content Modelling • Research data can take many forms • Fedora is flexible enough to store and preserve any file type • Your data may also be inter-related in complex ways • Fedora provides native RDF support • Complies with Linked Data Platform specifications
17. Linked Open Data
18. Enhancing repository data • Linked Open Data can make your repository better! • Link to authority records • Publish content to other systems • Pull in new content
19. RDF metadata • Adopt an RDF metadata schema • Link to authorities • Link to other related objects • Integrate multiple RDF ontologies • Make your data reusable!
20. LOD Browsers
21. Open Annotations
22. Flexible storage • Project Fedora over an external file system • Use Fedora’s management and preservation features • No need to ingest large datasets • Pluggable support for other back-end systems • Support for asynchronous storage is on the roadmap
23. Fixity and versioning • Verify and maintain the fixity of repository objects • Checksums can be calculated and verified on ingest • Checksums can be recalculated at any time • Support for flexible versioning • Turn on versioning for the entire repository • Or selectively create new versions using a REST API call
24. Backup and long-term access • The repository can be backed up and restored at any time • Particular objects or groups of objects can be exported and imported • The repository can also be exported in RDF/XML format for long-term transparency
25. Transactions • Writing data to disk is slow • Transactions bundle multiple actions together • Results: 30-60% performance increase • Transactions can be rolled back
26. Clustering • Fedora 4.0 supports clustering for high- availability • Multiple Fedora instances in replication mode • Load balancer distributes server requests evenly • Failover if one instance goes down
27. Scalability • A number of scalability tests have been run: • Uploaded a 1 TB file via REST API • 16 million objects via federation • 10 million objects via REST API
28. Fedora 4 roadmap • Complete Beta pilots by end of October • Release Fedora 4.0-Production in 2014 • Fedora 4.1 focus: • Support Fedora 3.x to 4.x migrations • Polish Fedora 4.0 edges
29. Resources • Fedora 4 wiki • https://wiki.duraspace.org/display/FF/ • Fedora 4.0 Features • https://wiki.duraspace.org/display/FF/Fedora +4.0+Feature+Set • Mailing lists • https://wiki.duraspace.org/display/FF/Mailing +Lists+etc
30. Questions?