Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Camtrap DP: A frictionless data exchange format for camera trapping data

Peter Desmet
October 22, 2021

Camtrap DP: A frictionless data exchange format for camera trapping data

Talk at the TDWG 2021 virtual conference - October 22, 2021.

Abstract: https://doi.org/10.3897/biss.5.73188

Peter Desmet

October 22, 2021
Tweet

More Decks by Peter Desmet

Other Decks in Science

Transcript

  1. Camtrap DP
    A frictionless data exchange format
    for camera trapping data
    Peter Desmet
    Ben Norton
    Jakub Bubnicki
    TDWG
    22 October 2021
    0000-0002-8442-8025
    0000-0002-5819-9134
    0000-0002-2064-3113

    View full-size slide

  2. - Wildlife monitoring technique
    - Non-invasive
    - Well-established
    - Enables study of animal
    abundance, distribution,
    behaviour
    - Data-intensive: lots of images or
    videos
    Camera trapping

    View full-size slide

  3. - Data management platforms
    - Upload and manage data
    - Annotate with species
    identifications (often using
    image recognition)
    - Limited data exchange between
    platforms
    - Limited data publication from
    platforms
    Data are well-managed,
    not shared
    Agouti Wildlife
    Insights
    TRAP
    PER
    eMam
    mal

    View full-size slide

  4. - Does not capture full scope
    - Project setup
    - Camera setup
    - Blank, vehicle, unknown
    sequences of images
    - Star schema too limited to
    capture all relationships
    - Camera trap researchers do not
    recognize data model
    Darwin Core (Archive)?
    Agouti Wildlife
    Insights
    TRAP
    PER
    eMam
    mal
    Darwin Core (Archive)?

    View full-size slide

  5. - “Camera Trap Data Package”
    - Designed to capture all essential
    data and metadata of a single
    camera trap study
    - Model to exchange camera
    trapping data
    - Format to exchange camera
    trapping data
    Camtrap DP

    View full-size slide

  6. - Metadata about project
    Camtrap DP model Project / Study
    img 1 grey heron
    img 2 grey heron
    img 3 blank
    seq 1 moorhen
    seq 1 coot
    gbif.org/occurrence/3045046810
    gbif.org/occurrence/3045043163
    - Deployments: start/end date,
    location, camera info
    - Media: file path/url, timestamp,
    sequence
    - Observations: blank, or animal
    of certain species, count, sex, ...

    View full-size slide

  7. - Metadata as datapackage.json
    - Project metadata
    - Package structure
    - Deployments as csv
    - Media as csv
    - Observations as csv
    Camtrap DP format datapackage
    .json
    media.csv observations
    .csv
    deployments
    .csv
    sequenceID
    mediaID
    deploymentID
    deploymentID

    View full-size slide

  8. - Developed by Frictionless Data
    - Set of open specifications (JSON
    schemas) that can be combined
    - Data Package for datasets
    - Data Resource for data files
    - Table Schema for table fields
    - Simple, machine-usable &
    extensible
    Using Frictionless Standards
    specs.frictionlessdata.io

    View full-size slide

  9. - Open, versioned, collaborative
    - Camtrap DP profile
    (extends Data Package)
    - Project metadata
    - Spatial, temporal, taxonomic scope
    - 3 Table Schemas
    - Fields, definitions, data types,
    controlled vocabularies
    - Relationships
    Camtrap DP development
    github.com/tdwg/camtrap-dp

    View full-size slide

  10. {
    "id": "https://doi.org/10.5281/zenodo.4893244",
    "profile":
    "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/
    camtrap-dp-profile.json",
    "contributors": {},
    "project": {},
    "spatial": {},
    "temporal": {},
    "taxonomic": [],
    "resources": [
    {
    "name": "deployments",
    "path": "deployments.csv",
    "profile": "tabular-data-resource",
    "schema":
    "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/
    deployments-table-schema.json"
    },
    {
    "name": "media",
    "path": "media.csv",
    "profile": "tabular-data-resource",
    "schema":
    "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/
    media-table-schema.json"
    },
    {
    "name": "observations",
    "path": "observations.csv",
    "profile": "tabular-data-resource",
    "schema":
    "https://raw.githubusercontent.com/tdwg/camtrap-dp/0.1.6/
    observations-table-schema.json"
    }
    ]
    }
    doi.org/10.5281/zenodo.4893244

    View full-size slide

  11. - Python: validate with frictionless-py
    - Metadata
    - Structure
    - Fields
    - Controlled vocabularies
    - Relationships
    - R: read with datapackage
    - Data management platforms:
    export format
    # devtools::install_github("inbo/datapackage")
    library(datapackage)
    pkg <- read_package(
    "https://zenodo.org/record/4893244/files/datapackage.json"
    )
    #> Please make sure you have the right to access data from this Data
    Package for your proposed use.
    #> Follow applicable norms or requirements to credit the dataset and
    its authors.
    #> For more information, see https://doi.org/10.5281/zenodo.4893244
    pkg$resource_names
    #> [1] "deployments" "multimedia" "observations"
    read_resource(pkg, "deployments")
    #> # A tibble: 505 × 16
    #> deploymentID locationID locationName
    #>
    #> 1 005eaf17-3197-425a-b… 81e247d0-edc9-452… B_ML_val 04_Roes…
    #> 2 00a2c20d-f038-490c-9… e254a13c-26e8-483… B_HS_val 2_proce…
    #> 3 00b0ecf3-a098-4e91-9… 9541cd66-93ee-42e… B_DM_val 2_Aloam
    #> 4 00ce371b-a2b5-4712-b… a934bb70-90d5-440… B_HS_val 6_keers…
    #> 5 0162ecfb-dc2a-4bc3-a… 91d9abdd-da56-49a… B_ML_val 05_mole…
    #> 6 01d9f82e-b1e4-4d95-8… ce943ced-1bcf-414… B_DM_val 4_'t WAD
    #> 7 01dd1933-9738-4859-9… 9541cd66-93ee-42e… B_DM_val 2_Aloam
    #> 8 01e48853-9ece-4d19-a… 2b477cf0-513d-4bb… B_ML_val 10_Sint…
    #> 9 01e889b8-a8ae-4f84-8… d2f5034c-3699-4e6… D_MICA 328
    #> 10 0432546b-9c28-495f-b… 56128022-6061-4b5… B_ML_val 06_Oost…
    #> # … with 495 more rows, and 13 more variables: longitude ,
    #> # latitude , start , end , setupBy ,
    #> # cameraID , cameraModel , cameraInterval ,
    #> # cameraHeight , baitUse , featureType ,
    #> # tags , comments
    Software
    github.com/frictionlessdata/frictionless-py
    github.com/inbo/datapackage

    View full-size slide

  12. - Lossy transformation to Darwin
    Core possible and documented
    - Dublin/Darwin Core term
    equivalents indicated in rdfType
    - Direct GBIF harvesting in the
    future?
    Darwin Core
    doi.org/10.15468/5tb6ze
    bit.ly/dwc-for-biologging-camtrap-dp

    View full-size slide

  13. - Capture essential data and
    metadata of a camera trap study
    - Data exchange model and format
    - Uses Frictionless Standards
    - Data Package (extended)
    - Data Resource
    - Table Schema (extended)
    - Built-in validation
    - Open, versioned and collaborative
    Summary
    tdwg.github.io/camtrap-dp

    View full-size slide

  14. Thank you
    tdwg.github.io/camtrap-dp
    Desmet, Norton & Bubnicki (2021) Camtrap
    DP: A frictionless data exchange format for
    camera trapping data. Presentation.
    https://bit.ly/camtrap-dp-tdwg-2021

    View full-size slide