Whip: Communicate and test what to expect from data

Talk at the TDWG 2018 annual conference in Dunedin, New Zealand - August 28, 2018.

Abstract: https://doi.org/10.3897/biss.2.25317

Transcript

1. Whip Communicate and test what to expect from data Stijn Van Hoey & Peter Desmet
2. Expectations Data Users
3. Expectations Data Users Fit for my research? Fit for specific user community?
4. We are a data publisher
5. We care
6. What to expect Data Publisher
7. What to expect Data Publisher Data quality Standardization Community recommendations Dataset characteristics
8. Expectations / What to expect Data Publisher Users Expectations What to expect
9. How to communicate expectations? Data Publisher Users Expectations What to expect
10. How to test expectations? Data Publisher Users Expectations What to expect
11. Whip
12. Whip syntax
13. Whip syntax
14. Whip syntax Field
15. Whip syntax Field Specification
16. Whip syntax Comment Field Specification
17. Whip syntax Comment Field Specification
18. Whip specifications allowed minlength / maxlength stringformat regex min / max numberformat mindate / maxdate dateformat
19. Whip scope specifications empty delimitedvalues if
20. Using whip to document
21. Pywhip: a whip implementation
22. import whip_csv from pywhip # load specifications with open("my_specifications.yml") as spec_file: specifications = yaml.load(spec_file) # test specifications test = whip_csv("my_data.csv", specifications) # get report test.get_report("html") Pywhip Or “json”
23. Pywhip
24. Pywhip
25. Pywhip
26. Conclusion Human and machine-readable syntax to express specifications for data Not specific to Darwin Core (but we plan to use it for that) Can be adopted by users (expectations) and publishers (what to expect) Can be included with dataset as testable metadata Pywhip: first implementation for testing whip specifications
27. github.com/inbo/whip github.com/inbo/pywhip bit.ly/pywhip_binder Thank you! Data Specifications