200324_EvaluationOfSoftwareImpact

Slides used for presenting the paper "Evaluation of software impact designed for biomedical research: Are we meassuring what's meaningful?" DOI: https://doi.org/10.48550/arXiv.2306.03255 at LIBD Leo Collado's team journal club.

Transcript

1. MELISSA MAYÉN QUIROZ 20 MARCH 2024
2. BIOMEDICAL SOFTWARE IS CRITICAL FOR BIOMEDICAL RESEARCH INITIALLY DEVELOPED FOR PERSONAL USE SHARING BENEFITS THE COMUNITY SOFTWARE EVALUATION CRUCIAL TO IMPROVE ITS USE AND IMPACT SECURE FUNDING FOR MAINTENANCE AND NEW DEVELOPMENT COMMON METRICS NEW USERS RETURNING USERS TOTAL DOWNLOADS DOWNLOAD COUNT BY VERSION
4. TWO ANALYSES WERE PERFORMED TO UNDERSTAND SOFTWARE EVALUATION . SURVEY TO UNDERSTAND DEVELOPER ATTITUDES AND PRACTICES. ASSESSMENT OF IMPLEMENTED INFRASTRUCTURE AND ITS IMPACT ON SOFTWARE EVALUATION. SURVEY ANALYSIS SURVEYED 48 PARTICIPANTS FROM THE ITCR COMMUNITY. IDENTIFIED MAJOR BARRIERS TO PERFORMING SOFTWARE IMPACT EVALUATIONS: LIMITED TIME (68% OF RESPONDENTS) FUNDING CONSTRAINTS (57%) PRIVACY CONCERNS (38%) TECHNICAL ISSUES (32%) LACK OF KNOWLEDGE ABOUT EVALUATION METHODS (27%)
5. FUNDING SUPPORT CURRENT FUNDING MECHANISMS LIKE ITCR PROGRAM AND CHAN ZUCKERBERG INITIATIVE ARE INSUFFICIENT. NEED FOR MORE FUNDING TO SUPPORT SOFTWARE SUSTAINABILITY. BENEFITS OF SOFTWARE EVALUATION DESPITE CHALLENGES, EVALUATIONS HAVE INFORMED: NEW DEVELOPMENT IDEAS (73% OF RESPONDENTS) DOCUMENTATION (60%) FUNDING JUSTIFICATION (54%) CHALLENGES AND GOALS OPEN-ENDED QUESTION RESPONSES HIGHLIGHTED CHALLENGES: COLLABORATION TRACKING COMMERCIAL APPLICATION USAGE USER BASE FRACTION DOWNSTREAM ACTIVITY USER FRUSTRATION
6. INFRASTRUCTURE ANALYSIS MANUALLY INSPECTED 44 SOFTWARE TOOLS FUNDED BY ITCR AND CTD² NETWORK. INSPECTED FOR INFRASTRUCTURE RELATED TO: KNOWING ABOUT THE TOOL AND HOW TO USE IT SOFTWARE HEALTH METRICS ASSOCIATED ASPECTS WITH INCREASED USAGE: PRESENCE ON SOCIAL MEDIA (E.G., TWITTER) BADGES INDICATING SOFTWARE HEALTH METRICS EXTENSIVE DOCUMENTATION AND FEEDBACK MECHANISMS USAGE EVALUATION USED SOFTWAREKG-PMC DATABASE FOR USAGE EVALUATION TIME SINCE TOOL RELEASE IS THE LARGEST CONTRIBUTOR TO VARIATION IN THE NUMBER OF PAPERS DESCRIBING USAGE. VARIOUS ASPECTS OF INFRASTRUCTURE ARE ASSOCIATED WITH INCREASED USAGE RATES.
8. 3.1 SUCCESSFUL EVALUATIONS ARE ANCHORED BY AN UNDERSTANDING OF THE INTENDED USE OF THE SOFTWARE 3.2 METRIC SELECTION SHOULD BE HYPOTHESIS DRIVEN 3.3 INTENTIONS FOR EVALUATION CAN ALSO INFORM DESIGN CHOICE 3.4 METRICS CAN ACHIEVE DIFFERENT GOALS FOR DIFFERENT AUDIENCES 3.5 NO SINGLE EVALUATION METHOD WORKS FOR EVERY TYPE OF SOFTWARE 3.6 METRICS SHOULD BE INTERPRETED 3.7 SOFTWARE INFRASTRUCTURE ENABLES IMPACT EVALUATION -> 3.8 SOFTWARE PROJECT HEALTH METRICS CAN REASSURE USERS AND FUNDERS -> 3.9 METRICS RELATED TO SOFTWARE QUALITY AND RE-USIBILITY COULD REASSURE USERS AND FUNDERS
10. Table 3: Software infrastructure enables the capture of valuable metrics for evaluating engagement and impact. Note that there are other helpful tools to enable metric collection. These are simply examples.
12. 4.1 CITATION CHALLENGES 4.2 LIMITATIONS OF TRACKING SYSTEMS 4.3 DISTORTED METRICS -> 4.4 CLINICAL DATA CHALLENGES 4.5 GOODHART’S LAW 4.6 SECURITY, LEGAL AND ETHICAL CONSIDERATIONS
14. SOFTWARE IS VITAL FOR THE ADVANCEMENT OF BIOLOGY AND MEDICINE. ANALYSIS OF USAGE AND IMPACT METRICS HELPS DETERMINE USER AND COMMUNITY ENGAGEMENT. METRICS JUSTIFY ADDITIONAL FUNDING, ENCOURAGE MORE USE, AND IDENTIFY UNANTICIPATED USE CASES. CHALLENGES INCLUDE DISTORTED, EXAGGERATED, OR UNDERSTATED METRICS, AS WELL AS ETHICAL AND SECURITY CONCERNS. MORE ATTENTION TO NUANCES AND CHALLENGES IS NEEDED FOR CAPTURING IMPACT ACROSS DIVERSE BIOLOGICAL SOFTWARE. NOT A SINGLE PERFECT METRIC OR APPROACH EXISTS; IT DEPENDS ON EACH TOOL'S UNIQUE ASPECTS AND USAGE. PROPOSED GUIDELINES AND STRATEGIES FOR SOFTWARE EVALUATION AND RESOURCES. FINDINGS FROM A SURVEY OF SCIENTIFIC SOFTWARE PROJECTS FOR THE ITCR PROGRAM FUNDED BY THE NCI. DEVELOPERS STRUGGLE TO FIND TIME OR FUNDING FOR SOFTWARE IMPACT OR USAGE ANALYSES. INFRASTRUCTURE SUCH AS SOCIAL MEDIA PRESENCE AND CLEAR CONTACT INFORMATION SEEM TO INCREASE SOFTWARE USAGE RATES.
15. AFIAZ, A., IVANOV, A. A., CHAMBERLIN, J., HANAUER, D., SAVONEN, C. L., GOLDMAN, M. J., ... & WRIGHT, C. (2023). EVALUATION OF SOFTWARE IMPACT DESIGNED FOR BIOMEDICAL RESEARCH: ARE WE MEASURING WHAT’S MEANINGFUL?. ARXIV.