Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gridded data with harpIO

harp
October 17, 2019

Gridded data with harpIO

harp

October 17, 2019
Tweet

More Decks by harp

Other Decks in Education

Transcript

  1. Gridded data with harpIO Alex Deckmyn (RMI), Andrew Singleton (MET

    Norway), Christoph Zingerle (ZAMG), Bent Hansen Sass (DMI)
  2. Many grid formats • In spatial verification, we may encounter

    many different file formats: FA, grib[2], netcdf[4], hdf5, BUFR, ... • harpIO includes many “reader” functions for different formats: • If your file format is missing, you can add it as a function read_myformat(file_name, ...). For instance, RMI uses tar files that contain the FA output of a model run. read_grid(file_name, file_format, ...) read_fatar(file_name, parameter, lead_time, ...)
  3. Format specific packages To read these formats, we use several

    packages in the background. Some are standard CRAN packages, others are harp specific and available on github GRIB 1&2: Rgrib2 FA: Rfa (not yet on github) HDF5: h5 (but only ODIM format is fully supported in harpIO) NetCDF: ncdf4
  4. A unified grid format: meteogrid meteogrid (which used to be

    called geogrid for many years), is a package that implements a unified internal representation for gridded data. It specialises in (and optimises for) weather model data: • multi-dimensional arrays (ensemble members, lead times, vertical levels…) • Projections, grid interpolations, wind fields • Visualisation on maps • Land/sea mask • Original emphasis was on interactive use: data exploration
  5. A unified grid format: meteogrid Two main data classes: geodomain

    and geofield A geodomain represents the grid properties. It is a list with • projection: a proj4 string • nx, ny : grid dimensions • dx, dy : grid resolution • clonlat : central longitude & latitude A geofield is a matrix (or array) with attributes • domain : a geodomain • info : name, variable, time, vert. level, origin...
  6. A unified grid format: meteogrid If you have a matrix

    of data, you turn it into a geofield by adding geodomain (and info) attributes. This is effectively how we would go about implementing new formats. All geofields are equal : meteogrid itself does not know anything about the original data format. You can see the internal list structure of a geodomain by running > dump(“domainname”, file=””)
  7. Interactive grid analysis with harp & friends testfile <- system.file(“data/ICMSHtoy25+0015”,

    package=”Rfa”) x <- read_grid(testfile, “fa”, “t2m”) iview(x, legend=TRUE) point.interp(x, lon=0, lat=50, method=”bilin”) Some simple data viewing could look like this: But x is also a simple matrix, so many basic R functions also work: max(x)
  8. Rfa for power users All geofields are equal but …

    Rfa (and, to a point, it’s sibling Rgrib2) has much more to offer than just decoding 2d fields. • Vertical interpolation (model levels to pressure) • Cross sections • Data encoding, field & file manipulation • Command line tools (bash) via Rscript: This can easily be defined as a bash function. > Rscript -e ‘Rfa::FAopen(“myfile”)$list’
  9. FAdec or read_grid(file_format=”fa”,...) ? FAdec(“myfile”, “CLSTEMPERATURE”) vs read_grid(“myfile”,,”t2m”) FAdec() interface

    is almost 20 years old. It will not change. read_grid() is easier and generic, but sometimes the full Rfa toolbox may be required: extracting & interpreting model levels, “frame” details By the way: did you notice the 2 commas in the read_grid command?
  10. What is to come Some more graphical possibilities are in

    the pipeline: the extra dimensions in geofields can represent many things • Ensemble members -> spread, mean… • Lead time -> animations? • Vertical levels -> cross sections, profiles This will probably be in harpVis, not in meteogrid. But maybe both.