Evaluation Insights to Key Processes of Digital Repositories

A presentation by Konstantinos Koumoutsos, Angelos Mitrelis and Giannis Tsakonas in Libraries In the Digital Age 2010 in Zadar, Croatia.

Digital repositories are considered essential information tools for scholarly communication. Their acceptability and extensive use by communities and institutions, as well as the users’ commitment in self-archiving, highlight the need for developing alternative channels of communication to expose scholarly productivity. Furthermore, the digital repositories community is interested into transforming them into viable, reliable and useful systems. This interest is primarily expressed by intense research activity, including - among the others - the evaluation and the usability of the technological solutions that support these services. On an institutional level, digital repositories are systems supported by physical organizations, such as libraries, which undertake many tasks in order to enable a variety of processes associated with these systems, such as submission, editing and access. In this paper, we present a multifaceted evaluation initiative that aimed at the redesign of University of Patras’ institutional repository, namely ‘Nemertes’. ‘Nemertes’ is operating on a DSpace installation and the ‘Theses and Dissertations’ collection was placed at the center of evaluation as the most important collection accommodated in the service. Emphasis was given to key processes held inside the repository by conducting surveys and interviews with typical classes of users. In order to collect data from these sources three different studies were held. First the quality of Submission process inside the physical and the digital space was evaluated through a questionnaire survey, which was addressed to people who had earlier submitted in the ‘Theses and Dissertation’ collection. Secondly, the information retrieval processes and the interface were evaluated by Human-Computer Interaction savvy students using the usability heuristics principles. Finally, the Editing processes and the quality of the delivery of services were assessed through interviews with the librarians that support the service. The findings of these studies point to areas that the system can be improved and help to eliminate the barriers that prohibit the service to be upgraded and host new collections. The areas identified concern both the way of delivering the service and the operation of the system. While the contextual parameters make the generalization of the findings about the service more ambiguous, the findings concerning the system performance and the interface intuitiveness validate the results of previous studies, such as the case of terminology, the affordances and the effectiveness of search interfaces. It is anticipated that the findings of the study can be further exploited by organizations with similar repository services and technological infrastructures.

Transcript

1. Evaluation Insights to Key Processes of Digital Repositories Konstantinos Koumoutsos, Angelos Mitrelis  Giannis Tsakonas Library  Information Center, University of Patras, Greece. Libraries In the Digital Age 2010, May 24-28, 2010, Zadar, Croatia
2. Digital Repositories • A familiar stranger: – Important channels for disseminating scientific/academic productivity. – Important means for preserving scientific/academic outputs. – Representing different types of communities (institutional/ thematic). – Wide-spread systems, but with high degree of uniqueness. – Reflecting different modes of policies and interactions. 2
3. Digital Repositories’ Evaluation • Institutional Repositories: – Social aspects • impact of scientific documents, motives of self-archiving activity, relationships with other types of digital repositories, etc. – Technical aspects • effectiveness/efficiency measures of submission, interfaces, etc. • DSpace* in particular: – Comparative evaluation of system features – Evaluation of end user performance – Administration/configuration easiness * Used in this study 3
4. is Research • Part of a wider evaluation initiative for the ‘Nemertes’ IR. • Emphasizes in key processes held inside the repository: – Submission (users as self-submitters). – Retrieval (users as searchers). – Editing (librarians, moderators, administrators). 4
5. Context • ‘Nemertes’, the University of Patras’ IR. – Developed by Library and Information Center. – Since 2006 on a DSpace installation (v 1.4). – Link: http://nemertes.lis.upatras.gr • Collections: – Self-submission. – ‘eses and Dissertations of University of Patras’ (primary collection, also on OPAC, printed/digital format) – ‘Technical Reports’ – ‘Journals/Proceedings Publications of the LIC personnel’ – ‘Faculty Members Publications’ (pre/post-prints of the faculty members of the University) 5
6. Methodology ~ Overview Submission process: a questionnaire survey addressed to people who had earlier submitted in the IR. Retrieval process / Interfaces: a HCI-based user study. Editing processes: interviews with librarians. 6
7. Methodology ~ Some Details • uestionnaire survey: – Previously submitted in the IR. – From 1129 registered, 126 participated. – Online, 25 questions, scale from 1 (-) to 5 (+). • HCI-based user study: – 24 graduate students, Department of Electrical Engineering, HCI background. – Based on usability heuristic evaluation principles (10 principles, 4 levels of problem significance). • Interviews: – 5 librarians supporting the service. – Semi-structured, approx. 19 minutes duration. 7
8. Insights to Submission • Reported: Overall satisfaction with the process. • Fairly satisfied with help facilities and submission wizard. • Satisfied with the support of the librarians. • Very satisfied with the self-submission and the bilingualism of the system. 8
9. Further Insights to Submission • Satisfied with copyright (now: institutional license/plus an option for a Creative Commons one). – Yet 15.1 % do not know or do not answer. • Satisfied with exclusion period (publication embargo) – Alternatives: most: up to 2 years / a few: up to 5 years. • Fairly satisfied with access policy (now: free access to all) – Alternatives: most: controlled access to everyone / a few: closed to everyone. 9
10. Insights to Retrieval  Interfaces Principle Unique problems Number of Reports (by Severity) Number of Reports (by Severity) Number of Reports (by Severity) Minor Important Serious Visibility of system status 4 2 4 3 Match between system and the real world 17 10 29 20 User control and freedom 6 3 9 3 Consistency and standards 12 5 24 8 Error prevention 4 1 14 2 Recognition rather than recall 3 1 11 4 Flexibility and efficiency of use 9 9 13 5 Aesthetic and minimalist design 20 0 74 18 Help users recognize, diagnose, & recover from errors 4 0 11 7 Help and documentation 5 5 6 2 36 (11.88%) 195 (64.36%) 72 (23.76) 10
11. Further Insights to Interfaces 11 x16 x6 x9 x8 x3
12. Further Insights to Retrieval 12 x16 x1
13. Insights to Editing • e editing process is long and time consuming. – Yet minimizes the possibility of errors. • Retrospective editing of metadata and corrections to files is considered essential. • Submitters’ keywords are considered invaluable. • Reporting problems in users’ navigation in the physical space to accomplish a submission. • Receiving complaints for the completion of the license. 13
14. Further Insights to Editing • Usual problems: – e absence of abstracts and keywords in parallel language. – e errors in bibliographic descriptions (capitals, mixed keyboard languages, ‘greeklish’). – e uploading of damaged or non compliant files. • Auto-suggestion fields are recommended. • Faculties and Departments should assist in promoting ‘Nemertes’. • Library should be proactive. 14
15. A Wide View of Findings • Submitters seem very positive. Librarians and ‘HCI users’ were strict. • Search functionalities need revision. • Interface features need clarity and consistency. • Editing needs revision to enhance efficiency. 15
16. Response • Migration to new DSpace version. • Resolve problems in information retrieval. • Apply add-ons that assist submitters and librarians. – Secure correct entry of metadata. • Careful consideration of labels and terms. • Exploit submitters’ keywords. • Design a new theme and graphical elements. 16
17. Conclusions • An obvious conclusion: more research. – More views are needed. – Each collection poses new challenges. • e three processes are linked. – Each process has an effect on the other. – e physical aspect of the service affects the digital. 17
18. e End • ank you for your attention. • uestions? Addendum: • ‘Nemertes’: http://nemertes.lis.upatras.gr/dspace/handle/123456789/3069 • ‘E-LIS’: http://eprints.rclis.org/18502/ • Contact: Giannis Tsakonas, john@lis.upatras.gr 18