Deciphering the Monolith with DATIM 9000

DHIS2 Expert Academy

Abstract
Each year, The US President's Emergency Plan for AIDS Relief (PEPFAR)
undertakes a target-setting process for all standard process operating units
which the program supports (23 this year). The result of this process are country
operational plans (COPs) which provide detailed information on budgets, program
activities, and M&E targets.

In previous years, country teams have utilized Excel-based tools to develop higher
level targets as well as their own processes for establishing targets down to the
site level before having to manually enter them into the a highly customized
instance of DHIS2 known as DATIM. DATIM is used for the global collection and
analysis of PEPFAR’s programmatic results and targets data across all operating
units. Due to the effort and time required to enter hundreds of thousands of
site-level targets, an automated process was developed to allow country teams to
use historical data and business-rule algorithms to distribute targets from district
to site level, and then import targets directly into DATIM in bulk from Excel
workbooks. This consisted of a rigorous schema-based import process that
validated, transformed and imported Excel-based worksheets using R, an
open-source statistical software. Robust end-user support processes, process
documentation, and Agile software development approaches were utilized to
meet the requirements to import data of all operating units according to a rigid
and compressed timeline

Transcript

1. Deciphering the Monolith with DATIM 9000 A.Chafetz, S.Jackson, and J.P.Pickering DHIS2 Experts Academy Oslo | June 2018 The contents in this presentation are those of the presenters and do not necessarily reflect the view of the U.S. President's Emergency Plan for AIDS Relief, the U.S. Agency for International Development or the U.S. Government.
2. OUTLINE 1. PEPFAR 2. Targeting 3. File Parsing 4. Lessons Learned 90% 90% 90%
3. PEPFAR
4. Working in over 50 countries, the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) has saved and improved millions of lives, prevented millions of HIV infections, and changed the course of the global HIV/AIDS epidemic.
5. diagnosed 90% on treatment 90% virally suppressed 90%
6. Through PEPFAR, the U.S. Government has invested $70 billion dollars in bilateral HIV/AIDS and tuberculosis programs
8. TARGETING
9. Population-based Historic results Numerous stakeholders ? ? ? Assumptions & inputs Complications around targeting Resource & time constraints Geography Total Age/Sex <15/Unknown 15+/Male 15+/Female Age/Sex/Modality Disaggregation
10. Due to the complexity, country teams have been targeting using standardized Excel-based tools.
11. Previously country teams or their partners manually entered targets back into DATIM by indicator and site.
12. Target Entry by: Partner X Site X Indicator
13. PEPFAR 2018 8+ US Government departments/agencies 50+ PEPFAR supported countries 30+ PEPFAR indicators 1,100+ PEPFAR indicator disaggregates 60,000+ PEPFAR supported sites (active)
14. How do we reduce the thousands of man hours needed to enter site targets back into DATIM (in a short time frame)? PROBLEM
15. FILE PARSING
16. Ideal state = closed loop where all targeting occurs within the system
17. Old status quo
18. New implementation A B
19. DataPackR Library
20. • Written in R • Parse Excel files - readxl • Validate Excel files - jsonlite • Transform and reshape data - tidyr, plyr • Load support files - as Rds files • Perform distribution - dplyr • Validate against DHIS2 rules - datim-validation • Create site-level tools for end-users - openxlsx KEY FEATURES
21. Testability Reproducibility Extensibility
22. • Define each sheet’s geometry • Validate column position • Validate cell contents • Write unit tests to identify possible defects • Use continuous integration for automated testing Testability
23. • Use GitHub to distribute R scripts • Hash and verify support files • Create an R package • Use Vagrant for a reproducible environment • Use continuous integration Reproducibility
24. • Separate parsing, validation, and export • Allow for additional file formats with schema-driven parsing • Allow for a web interface to be integrated Extensibility
25. LESSONS LEARNED
26. Lessons Learned B A Large-scale, distributed import of data to DHIS2 from Excel is possible … under highly controlled conditions Keep Excel file structure simple. Use code to create the Excel files not cut and paste Ensure you have a good support team. 1 2 3 4
27. </> 1 2 3 earlier start on build with requirements clearly articulated up front developer led process template and processing all done by code Future Iterations
28. AARON CHAFETZ Data & Visual Analyst US Agency for International Development SCOTT JACKSON Senior Analytics Solutions Strategist BAO Systems JASON PICKERING Lead Developer, DATIM HISP Nordic AB/University of Oslo Deciphering the Monolith with DATIM 9000
29. Deciphering the Monolith with DATIM 9000 Project Repositories - Data Pack - Data Pack Importer
30. NOTES AND ATTRIBUTION • Front Matter Inspired by mailparser (https://mailparser.io/features) • PEPFAR. PEPFAR Overview 2018 (https://www.pepfar.gov/documents/organization/251737.pdf) • UNAIDS. 90-90-90: An ambitious treatment target to help end the AIDS epidemic (http://www.unaids.org/sites/default/files/media_asset/90-90-90_en.pdf) • PEPFAR. Data for Impact (https://www.pepfar.gov/priorities/data/index.htm) • Icons from the Noun Project: Clinic – Vectors Market; Cog - Yuri Mazursky; Computer – Cassandra Cappello; Kenya – Dolly Holmes, US; Partner - Tomas Knopp; Person – Richa
31. Deciphering the Monolith with DATIM 9000 A.Chafetz, S.Jackson, and J.Pickering DHIS2 Experts Academy Oslo | June 2018 The contents in this presentation are those of the presenters and do not necessarily reflect the view of the U.S. President's Emergency Plan for AIDS Relief, the U.S. Agency for International Development or the U.S. Government.