OGC API - Environmental Data Retrieval Sprint Engineering Report

(1) Automated aggregation of metadata from collections (3D, 4D, 5D)

This creates collections of parameters (that have common dimensions) from operational data stores, and outputs in JSON which is used to convert the original dataset to zarr. Each zarr data store represents a specific collection. With the parameters grouped by common dimensions, more complex queries than EDR API can be made. Demo at https://github.com/ShaneMill1/edr-automation.

(2) Demonstration of client side APIs (single and multi-domain feature extractions)

Uses the EDR API to access time series at a point for:

Develop a simple EDR server to demonstrate Point, Polygon, and Items queries against the same Collection of data. This is still work in progress, to track and implement the changes to the latest EDI API specification, and also to try out novel ideas. It will also demonstrate an alternative approach to the proposed application item_id as a unique identifier for a location (i.e., a METAR id, GeoHash key, WhatThreeWords, etc.) which has a set of data assigned to it. This addressed Issues #04 and #05: Point, timeseries at a point, vertical profile at a point, and 2D polygon.

Extend the automated aggregation of metadata from Collections (3D, 4D, 5D) by storing the output from a forecast model run as a concatenation of binary GRIB files, then process as xarray dataset with pynio and extract metadata (parameter ID determined by pynio, long name, level type, dimensions) and put in a pandas data frame. The data frame creates a map between dimension names and dimension values. Parameters that have the same dimensions are grouped. The metadata is output in JSON. The original dataset is converted to zarr using the collection JSON. This approach will enable more complex queries than just EDR API. The implementation is: https://data-api.mdl.nws.noaa.gov/EDR-API/groups/US_Models?outputFormat=application%2Fjson based on NAM and GFS GRIB data. The aggregation will be extended to other national weather service data sources, or other file types such as HDF5, NetCDF:

Currently, one can search for a keyword, and Collections, which have parameters with a long name containing that keyword, will be shown as a URI, which will link to the EDR API query endpoint. The links returned need to be ordered in a dictionary. Additional work would be to:

Looking at the documentation, pycsw was incorporated into the EDR-API implementation following this approach: https://docs.pycsw.org/en/latest/api.html

Then, following the steps provided at the link below, a compliant blank SQLite database was created to start from: http://docs.pycsw.org/en/stable/administration.html#metadata-repository-setup

Finally, the beginning of the implementation can be seen at the following endpoint: https://data-api.mdl.nws.noaa.gov/EDR-API/csw?service=CSW&version=2.0.2&request=GetCapabilities

The next steps will be to connect the dots from how metadata is created in the aggregation of collections software and incorporate that metadata into these services. The discussion needs to continue on what/how the formal metadata would be to serve as part of a CSW instance (Dublin Core, ISO, etc.). This will also give us an opportunity to investigate the OGC API - Records work (disclosure: Tom Kralidis is on this SWG and working on an implementation). This is now an issue in the EDR SWG GitHub at https://github.com/opengeospatial/Environmental-Data-Retrieval-API#40.

Outcomes

Observations

Future work

This is the same as Issue #09, Objective: Trajectory, 2D, 3D, or 4D. The trajectory query was successfully implemented in all the 2D, 3D, and 4D cases, using the EDR API specification proposal to use Well Known Text (WKT) LineString formats.

This is ongoing work. There do not seem to be any fundamental architectural problems with possible approaches for the proposed use cases. It is expected that the queries supported would be for points/time series/vertical profiles, and cube/tiles.

The main aim of this work is to show how the EDR "Pattern" maybe "mapped" onto a proprietary RESTful API, such as ESRI’s Arc Image Server. The output from this work will be a document outlining the issues that are brought to light and a very simple prototype consisting of a simple proxy server that uses the EDR API and translates into the Image Server REST API.

There was general support for this topic, in particular, for the returned content of the payload from a specific query, but no work was done.

OGC API Features has a conformance test suite, written to be modular, and presumably, some of this will be shared with an OGC API Common test suite.

Conformance tests for the returned payload is a bit harder.

This is ongoing work that started after the formal Sprint days. STAC provides a common language to describe a range of geospatial information, so it can more easily be indexed and discovered. A “spatiotemporal asset” is any file that represents information about the earth captured in a certain space and time. While STAC work has been primarily focused on space-based earth observation imagery, there is value in investigating STAC capability for real-time Numerical Weather Prediction (NWP) data offerings. This would allow for an arbitrary hierarchy based on NWP workflow (model runs, forecast hours, forecast variables, etc.) as well as simple, file-based discovery/traversal of the same.

Questions

Observations

The EDR API specification uses ISO8601 style of notation to specify dates and times for query parameters. This is very convenient and understandable for users. However, the Trajectory query specifies time as a number of seconds since the UNIX epoch. This is specified in the Well Known Text (WKT) standard for defining lines/trajectories by using LineString. This makes time into a proper coordinate, enabling easier software calculations, though the units are not easily comprehended by users. It was agreed to keep the two different approaches, as both had merits and disadvantages, and elicit wider feed back from the OGC Members, OAB, and the public. A mitigation may be to incorporate an easy to use service to convert between the two different time representations.

Would the current EDR API specification allow simultaneously both a time series and a vertical profile at a point? I.e., A 2D array of values would be returned. At present, the Point query only allows a time series at a point or a vertical profile at a point, but not both. It was agreed to stay with these minimal cases. The general 2D vertical time series is not widely used outside of meteorology, where it is known as a Hovmöller diagram.

Another consequence of these discussions was that Point was re-named Position, and Polygon re-named as Area.

Should bounding boxes for a query, in 4D, be specified by ranges of values or coordinates of the corners? The consensus was that ranges are more natural for time and vertical coordinates. This is then the same as specified 2D Polygons in WKT.

There was no disagreement that there will be no polyhedra or polytopes (complex multi-dimensional polygons, rather than just “extruded” 2D polygons. An extruded 2D polygon can be called a “prism.”

This was the usual semantic question. It was agreed that the initial resource was a persistent, dense data store. The queries against it were Discrete Sampling Geometries sampling the data store, and were transient, but could be made a persistent resource if required by another service. [The Research Data Alliance, RDA, recommends a query store in its Best Practice Recommendations for citing dynamic data.] the returned data payload is also a transient resource, which also could be made persistent. It was agreed to add words to this effect in the EDR API candidate standard.

If the data returned in response to a query is small, there is no need for streaming of the results. It may be necessary for large responses, but that is starting to be outside the scope of the EDR API. It was agreed that the EDR API specification would not mention streaming, and it would be an implementation decision as to whether streaming is supported. This decision would obviously be influenced by the choice of supported media type, which may or may not support streaming.

The EDR API provides no mechanism for the user to discover available location identifiers (such as ICAO ids) in the metadata. The identifiers are available in the query results, but not in any of the available metadata outputs. What would the JSON-Schema be for these items? E.g: collections/metar/raw/items?id=KIAD&parametername=icao_id&time=2020-01-31T00:00:00Z/2020-02-01T04:00:00Z. If consistent with Features, then collections/metar/raw/items returns the list of items (paged, if there are many) and that the query would look like: collections/metar/raw/items/KIAD?parametername=icao_id&datetime=2020-01-31T00:00:00Z/2020-02-01T04:00:00Z.

Currently, the EDR API “Items” specifies CoverageJSON rather than a GeoJSON Feature Collection with valid query parameters for each item in the collection. Can lists of available parameters in GeoJSON be exposed?

This validates against the GeoJSON schema: id is what goes in /collection/{collectionID}/items/{itemID}; datetime follows features; parametername follows the draft spec naming convention; label needed because there has to be a list/link label in the client; uri which could be @id, like JSON-LD and it should really be a linked data feature ID which 303 redirects if a real-world sampling feature. If these extensions are put in the GeoJSON FeatureCollection schema, the software SF/GDAL outputs:

The properties schema is here:

The EDR API specification found a need to have Groups of Collections in the API path. In the wider OGC, there is now a discussion of whether APIs could have Collection of Collections. The Sprint agreed to stay with Groups until the wider issue is resolved.

Corridors were originally envisaged as a volume defined by a surface of constant distance from a line trajectory. The idea of the bottom of a corridor volume being delineated by the earth’s surface (or some other surface) was raised. It was agreed to tackle this interesting, practical, and difficult problem later.

Observers, from outside the Sprint, have raised some substantive questions, including about interpretation and implementation of vertical coordinates in the EDR API. These will be raised and addressed in the EDR API Standard WG.