Camtrap DP: A frictionless data exchange format for camera trapping data

Talk at the TDWG 2021 virtual conference - October 22, 2021.

Abstract: https://doi.org/10.3897/biss.5.73188

Transcript

1. Camtrap DP A frictionless data exchange format for camera trapping data Peter Desmet Ben Norton Jakub Bubnicki TDWG 22 October 2021 0000-0002-8442-8025 0000-0002-5819-9134 0000-0002-2064-3113
2. - Wildlife monitoring technique - Non-invasive - Well-established - Enables study of animal abundance, distribution, behaviour - Data-intensive: lots of images or videos Camera trapping
3. - Data management platforms - Upload and manage data - Annotate with species identifications (often using image recognition) - Limited data exchange between platforms - Limited data publication from platforms Data are well-managed, not shared Agouti Wildlife Insights TRAP PER eMam mal
4. - Does not capture full scope - Project setup - Camera setup - Blank, vehicle, unknown sequences of images - Star schema too limited to capture all relationships - Camera trap researchers do not recognize data model Darwin Core (Archive)? Agouti Wildlife Insights TRAP PER eMam mal Darwin Core (Archive)?
5. - “Camera Trap Data Package” - Designed to capture all essential data and metadata of a single camera trap study - Model to exchange camera trapping data - Format to exchange camera trapping data Camtrap DP
6. - Metadata about project Camtrap DP model Project / Study img 1 grey heron img 2 grey heron img 3 blank seq 1 moorhen seq 1 coot gbif.org/occurrence/3045046810 gbif.org/occurrence/3045043163 - Deployments: start/end date, location, camera info - Media: file path/url, timestamp, sequence - Observations: blank, or animal of certain species, count, sex, ...
7. - Metadata as datapackage.json - Project metadata - Package structure - Deployments as csv - Media as csv - Observations as csv Camtrap DP format datapackage .json media.csv observations .csv deployments .csv sequenceID mediaID deploymentID deploymentID
8. - Developed by Frictionless Data - Set of open specifications (JSON schemas) that can be combined - Data Package for datasets - Data Resource for data files - Table Schema for table fields - Simple, machine-usable & extensible Using Frictionless Standards specs.frictionlessdata.io
9. - Open, versioned, collaborative - Camtrap DP profile (extends Data Package) - Project metadata - Spatial, temporal, taxonomic scope - 3 Table Schemas - Fields, definitions, data types, controlled vocabularies - Relationships Camtrap DP development github.com/tdwg/camtrap-dp
10. { "id": "https://doi.org/10.5281/zenodo.4893244", "profile": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/ camtrap-dp-profile.json", "contributors": {}, "project": {}, "spatial": {}, "temporal": {}, "taxonomic": [], "resources": [ { "name": "deployments", "path": "deployments.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/ deployments-table-schema.json" }, { "name": "media", "path": "media.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/ media-table-schema.json" }, { "name": "observations", "path": "observations.csv", "profile": "tabular-data-resource", "schema": "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/ observations-table-schema.json" } ] } doi.org/10.5281/zenodo.4893244
11. - Python: validate with frictionless-py - Metadata - Structure - Fields - Controlled vocabularies - Relationships - R: read with datapackage - Data management platforms: export format # devtools::install_github("inbo/datapackage") library(datapackage) pkg <- read_package( "https://zenodo.org/record/4893244/files/datapackage.json" ) #> Please make sure you have the right to access data from this Data Package for your proposed use. #> Follow applicable norms or requirements to credit the dataset and its authors. #> For more information, see https://doi.org/10.5281/zenodo.4893244 pkg$resource_names #> [1] "deployments" "multimedia" "observations" read_resource(pkg, "deployments") #> # A tibble: 505 × 16 #> deploymentID locationID locationName #> <chr> <chr> <chr> #> 1 005eaf17-3197-425a-b… 81e247d0-edc9-452… B_ML_val 04_Roes… #> 2 00a2c20d-f038-490c-9… e254a13c-26e8-483… B_HS_val 2_proce… #> 3 00b0ecf3-a098-4e91-9… 9541cd66-93ee-42e… B_DM_val 2_Aloam #> 4 00ce371b-a2b5-4712-b… a934bb70-90d5-440… B_HS_val 6_keers… #> 5 0162ecfb-dc2a-4bc3-a… 91d9abdd-da56-49a… B_ML_val 05_mole… #> 6 01d9f82e-b1e4-4d95-8… ce943ced-1bcf-414… B_DM_val 4_'t WAD #> 7 01dd1933-9738-4859-9… 9541cd66-93ee-42e… B_DM_val 2_Aloam #> 8 01e48853-9ece-4d19-a… 2b477cf0-513d-4bb… B_ML_val 10_Sint… #> 9 01e889b8-a8ae-4f84-8… d2f5034c-3699-4e6… D_MICA 328 #> 10 0432546b-9c28-495f-b… 56128022-6061-4b5… B_ML_val 06_Oost… #> # … with 495 more rows, and 13 more variables: longitude <dbl>, #> # latitude <dbl>, start <dttm>, end <dttm>, setupBy <chr>, #> # cameraID <chr>, cameraModel <chr>, cameraInterval <dbl>, #> # cameraHeight <dbl>, baitUse <fct>, featureType <fct>, #> # tags <chr>, comments <chr> Software github.com/frictionlessdata/frictionless-py github.com/inbo/datapackage
12. - Lossy transformation to Darwin Core possible and documented - Dublin/Darwin Core term equivalents indicated in rdfType - Direct GBIF harvesting in the future? Darwin Core doi.org/10.15468/5tb6ze bit.ly/dwc-for-biologging-camtrap-dp
13. - Capture essential data and metadata of a camera trap study - Data exchange model and format - Uses Frictionless Standards - Data Package (extended) - Data Resource - Table Schema (extended) - Built-in validation - Open, versioned and collaborative Summary tdwg.github.io/camtrap-dp
14. Thank you tdwg.github.io/camtrap-dp Desmet, Norton & Bubnicki (2021) Camtrap DP: A frictionless data exchange format for camera trapping data. Presentation. https://bit.ly/camtrap-dp-tdwg-2021