Using volunteered weather observations to explore urban and regional patterns in the Netherlands

Slides used during our presentation at the AGU Fall Meeting 2019 in San Francisco (USA), in the session "IN22A - Making Data Usable and Accessible: Gaining Insight from Citizen Science Applications". You can check the information of that session in the following link: https://agu.confex.com/agu/fm19/meetingapp.cgi/Session/88576

Transcript

1. Using volunteered weather
   observations to explore
   urban and regional
   weather patterns in the
   Netherlands
   Irene Garcia-Marti
   Marijn de Haij
   Jan-Willem Noteboom
   Gerard van der Schrier
   Cees de Valk
   AGU Fall Meeting 2019
   IN22A - Making Data Usable and Accessible:
   Gaining Insight from Citizen Science Applications
   10th December 2019
   

   View Slide

2. › Weather observations are
   crucial!
   › Spatial sparsity is a
   challenge for high-res
   weather forecasts
   › Increasing number of
   weather-related citizen
   science projects
   – WOW, Wunderground,
   Netatmo, Meteoclimatic
   Motivation
   2
   

   View Slide

3. 1st September 2019: 1,400 million observations and 17K stations worldwide
   

   View Slide

4. › 2015: KNMI partner of WOW
   › Contributors: 400+ CWS
   › Data NL+BE: 3.7M obs/month
   › Devices: semi-professional
   – Manufacturers: Davis, Oregon
   scientific, Ventus, Alecto…
   – Expected “reasonable” quality of
   the observations
   WOW-NL
   4
   Province of Utrecht
   94 CWS
   

   View Slide

5. › Quality not only related to
   device:
   – Good with respect to what? What
   variables are (not) properly
   monitored?
   – Local processes: radiation,
   shadowing, siting
   › Classical challenges of citizen
   science data:
   – Gaps in data
   – Noisy observations
   WOW-NL
   5
   

   View Slide

6. WOW-NL
   6
   › Quality not only related to
   device:
   – Good with respect to what? What
   variables are (not) properly
   monitored?
   – Local processes: radiation,
   shadowing, siting
   › Classical challenges of citizen
   science data:
   – Discretization
   – Scale of the phenomena
   

   View Slide

7. 7
   What is the quality of WOW-NL?
   R2 correlation between citizen and official weather stations
   

   View Slide

8. 8
   What is the quality of WOW-NL?
   

   View Slide

9. Preprocessing
   WOW json csv
   11.6M
   observations
   65 features
   SVF
   Feature engineering
   

   View Slide

10. Quality control
    › Based on (Napoly, 2018)
    › Variable: temperature
    – Feasible to implement on
    WOW-NL
    – Levels have been
    compacted
    – Each of the 11.6M
    observations is labeled with
    a quality level
    M0: incorrect metadata
    M1: insufficient Z-score
    (presence outliers)
    M2: insufficient day/mon
    coverage
    M3: insufficient (Pearson)
    correlation
    M4: OK
    (Napoly et al., 2018)
    Development and Application of a Statistically-Based
    Quality Control for Crowdsourced Air Temperature Data
    Frontiers in Earth Science
    

    View Slide

11. Overview of the quality of WOW-NL
    (Each square represents 10K observations)
    M0: incorrect metadata
    (not shown)
    M1: insufficient Z-score
    (presence outliers)
    M2: insufficient day/mon
    coverage
    M3: insufficient (Pearson)
    correlation
    M4: OK
    

    View Slide

12. › Heat wave 27-07-2018
    › Methodology:
    – Kriging interpolation:
    ▪ WOW-NL observations per hour
    ▪ Calibrated for this day
    ▪ No external drift
    – Visual comparison with
    HARMONIE
    ▪ Regional numerical model
    ▪ Provides forecast up to 48h in
    advance
    Exploring regional
    temperature
    12
    

    View Slide

13. › Results:
    – WOW-NL captures the daily
    temperature cycle
    – Spatial patterns are different:
    ▪ Radiation: proximity to buildings
    ▪ Cooling: shadowing of trees
    ▪ Meaning:
    • Predicted weather in the
    column different to what
    happens at ground level
    • More work to reduce this gap
    Exploring regional
    temperature
    13
    

    View Slide

14. › If good enough:
    – Open the door for new research:
    ▪ Fine-grained interpolated layers
    ▪ Nowcasting / hi-res weather
    ▪ Crowdsourced NWP
    – Governance level:
    ▪ Lower cost for administration
    ▪ Better weather forecast for
    underrepresented areas
    ▪ Bottom-up initiatives might work in
    developing regions
    Why the quality of citizen
    science weather data is
    important?
    14
    Imperfect data, but volume is difficult to ignore
    Big data problem!
    We are here
    

    View Slide

15. Questions? ☺
    garciamarti@knmi.nl
    Thanks!
    15
    

    View Slide