OGC Testbed-16: Data Access and Processing Engineering Report

The purpose of the EO Exploitation Platform Working Group is to foster interoperability (i.e. the pre-requisites for successful interaction between platform components and across platforms) among Exploitation Platforms and their components. To this end, the working group will act as an open form for discussion and documentation of interoperability requirements for the domain in order to drive OGC standards evolution towards better support for the use cases of the EO Exploitation Platforms.

The increasingly huge amount of new EO satellite data and In-Situ data every day has incentivized the creation of several web accessible platforms that enables scientists and commercial operators to use such data without the need to download content and have in-house Information Technology infrastructure to manage the large volume of data.

EO Exploitation platforms have been independently developed by public organization or commercial companies, but all share a common set of functionalities:

These platforms, however, have many different implementations, with interfaces and data formats which often hampers interoperability among platforms. In fact, the next step in this data exploitation revolution is to link the platforms together creating an ecosystem. This ecosystem is potentially across different administrative domains, in which each platform can contribute with the offered services to the implementation of more complex use cases.

The work in this Testbed 16 thread focused on the data process and analysis aspects of EO exploitation platforms. The particpants believe that the outcomes of this testbed would be of interest to users of EO exploitation platforms. Specifically the work of the testbed is to develop data access and processing interfaces that are simpler to use and thus simpler to learn and more easily integrated into the kinds of tools that EO exploitation platform users might use (e.g. using Jupyter notebooks to perform some analysis).

The purpose of the Environmental Data Retrieval (EDR) API SWG is to standardize several APIs, defined using OpenAPI (Version 3), to retrieve various common data patterns from a relatively persistent data store. The data patterns could include, but are not restricted to, data at a point in space and time, time series at a point, data along a trajectory, which may be 2, 3, or 4 dimensional, and covering a specified polygon or rectangular tile. The APIs will enable service users to retrieve resources over a discrete sampling geometry, created by the service in response to a standardized query pattern.

The work of the Testbed-16 Data Access and Processing task was closely aligned with the stated goals of the SWG. The participants believe that the outcomes of this testbed task could help inform the design of the API components that the SWG is endeavouring to specify.

The purpose of the OGC API - Processes SWG is to design a Web API that enables the execution of computing processes and the retrieval of metadata describing their purpose and functionality. Typically, these processes combine raster, vector, coverage and/or point cloud data with well-defined algorithms to produce new raster, vector, coverage and/or point cloud information.

In the current design of the Processes API, executing a process involves creating a JSON document that contains the process inputs and POSTing that document to the execution endpoint of the process. From an end-user perspective, this workflow is more complicated than may be necessary. Since one of the goals of the testbed task was to design a simpler API for invoking processes, the participants believe that this work could inform the design of an alternate invocation of WPS processes using KVP/GET method rather than the current JSON/POST method.

Furthermore, the work being done in Testbed-16 for binding processes to specific datasets and to advertise which combinations of data and processes are valid would be of interest to the SWG.

There are a large and increasing number of citizen science projects active around the world involving the public in environmental monitoring and other scientific research activities. The OGC Citizen Science DWG is motivated to support citizen science by providing a forum for increasing understanding and demonstration of the benefits brought by the use of open standards and best practices. This DWG will support the development of improved interoperability arrangements for the citizen science community.

The work of the testbed to simplify the interfaces for accessing and processing data from the end-user perspective is directly related to the goal of the DWG to improve interoperability arrangements for the citizen science community.

The volume of Earth Observation data has grown so large that moving such data to a local machine for processing is no longer feasible. Instead of moving the data to local processing, the trend is now to store large repositories of Earth Observation data in the cloud or compute back-ends, and move the processing software to the data. The results of this processing on the cloud can then be browsed remotely or downloaded as desired.

In order to enable this "move the process to the data" concept, the openEO organization developed an API that connects clients such as R, Python and JavaScript to big Earth observation cloud back-ends in a simple and unified way. The main objectives of the project are the following concepts:

GeoTrellis is a geographic data processing engine for high performance applications. It is implemented as a Scala library and framework that uses Apache Spark to work with raster data and supports many Map Algebra operations as well as vector to raster or raster to vector operations.

The OGC GeoAPI Implementation Standard defines the GeoAPI library. GeoAPI provides a set programming interfaces for geospatial applications. In a series of packages or modules, GeoAPI 3.0 defines interfaces for metadata handling and for geodetic referencing (map projections). The GeoAPI interfaces closely follow the abstract models published collaboratively by ISO in its 19100 series of documents and the OGC in its abstract and implementation specifications. GeoAPI provides an interpretation and adaptation of these standards to match the expectations of Java or Python programmers. C

For this use case the typical user is a developer.

The user wants to access geospatial data for a specific area in a simple function call. The function call identifies the data and allows the user to define the discrete sampling geometry. Valid geometries include:

All geometries are provided either in-line or by reference, as illustrated by the following examples:

Specifying sampling geometries by reference also supports the use of an OGC API - Feature endpoint as shown in the example above where the value of the location parameter is an OGC API - Feature invocation to fetch the feature, and thus the geometry, of the city for Frankfurt.

Users need the ability to access the original data. The use of the term "original" is a little ambiguous in this context as data often undergoes some process on its way from original access to the final product. As an example, imagine a digital temperature sensor. The actual reading performed in the sensor is some form of electricity, but the value provided at the sensor interface is 21°Celsius (approximately 69.8°Fahrenheit). Thus, some form of calibration curve has been applied to the original reading, which might not be available at all. In this case, the value 21°Celsius can be considered as “original”. The same principles apply to satellite data. The original raw data readings are often not accessible. Instead, the data undergoes some correction process before being made available. Higher product levels may include orthorectification or re-gridding processes. In any case, data providers need to provide a description of the performed processing together with the actual data. In addition, data should be available as "raw" as possible.

End-users want to retrieve all data that exists within the provided target geometry. In the case of polygon geometries, the end-user would receive all data that intersects that polygon. In the case of point geometries, the end-user would retrieve the value exactly at that point.

In addition, end-users have the option to define the (interpolation) method for value generation. If no option is selected, the Web API indicates how a given value was produced. Testbed-16 participants developed a set of frequently used production options, including for example:

or any combination thereof. This use case differentiates the following data requests:

Testbed-16 participants explored these use-cases in combination with additional processing steps. For example, the end-user requests synoptic, map, or time series data, that is interpolated to a grid.

Testbed-16 participants explored simple data processing functions. These include calculations for:

These values are for any given data retrieval subset as accessible in the Data Retrieval use cases.

This section describes the detailed use cases articulated by the Testbed 16 participants, based on the general use cases, that guided the development of thread clients and end-points.

This use case focus on the DAPA API approach for gridded data covers with Sentinel-5P tropospheric NO2 column density and supports the DAPA API evaluation by several data scientists or earth scientists regarding:

The typical use cases under evaluation are:

These user stories were the starting point for defining the DAPA API methods and are mapped according to the table below:

(*) aggregate functions allows the request of minimum, maximum, and average values of a variable in a collection for any given data retrieval subset

This use case focuses on data from the Copernicus Sentinel-5P satellite and exposes the monthly (from May 2018 to April 2020) averaged nitrogen dioxide concentrations over a portion of Europe. Concentrations of short-lived pollutants, such as nitrogen dioxide, are indicators of changes in economic slowdowns and are comparable to changes in emissions.

This use case describes the data access requirements of a gridded climate dataset. This use case is heavily inspired by the Canadian Climate Data Portal.

The primary function of this notebook is to exercise each processing endpoint offered by a service that implements the DAPA. This is done by crawling the machine-readable API description that each service must offer.

The source code for this notebook is available here: TBD

Internal components of the client attempt to map specific DAPA concepts to more generic ISO, OGC or open-source APIs. To model resources, the Apache SIS API was used which was bound to OGC GeoAPI interfaces to define core concepts such as naming and metadata. In Apache SIS a resource can be either:

A DAPA client is made of custom implementations to ease usage. The client represents an aggregate of collections. Each collection is itself an aggregate composed solely of products. Products are specialization of datasets for DAPA processing/derivation/download endpoints. They are derivable through the specification of a set of query parameters.

Note: Even when trying to create an interoperability layer based on open-source specifications, the current client engine is based upon Geomatys proprietary/closed-source tool Examind Datacube.

The notebook develpoed during the Testbed is still incomplete in the following ways:

GeoSolutions and DLR created a notebook to compare the Terradue, EOX and GeoSolutions DAPA API endpoints. The purpose of this notebook was to compare various aspects for each of the APIs and support the harmonization of the API specification.

Generally, both APIs share the same approach and overall semantics as described in OGC Testbed-16: Data Access and Processing API Engineering Report but differ in some aspects as described in this section.

The source code for this notebook can be found here: https://gitlab.com/ogceo/t16-dapa-dlr/-/blob/master/DAPA_Comparison.ipynb.

A live link of the notebook can be found here: https://mybinder.org/v2/gl/ogceo%2Ft16-dapa-dlr/master?urlpath=lab/tree/DAPA_Comparison.ipynb

The following table compares the URL paths and parameters for each endpoint.

All APIs encode output differently. The specific details can be found in the Jupyter notebook but are summarized here:

The ability to negotiate output formats was not investigated during this testbed and is left as a future work item. Future investigation should start by defining the set of possible output data type structures (e.g. data cube, coverage, time series, multi-dimensional data set, feature or simple value) and then define recommended standard encodings for each (e.g. GeoTIFF, GeoJSON). The result of doing this would be to:

Specifying the output format alone is not sufficient since the structure of the content could be quite different, for example time series, multi-dimensional data set, feature data, or simple values.

The creation of MIME types or MIME type templates that capture the valid combinations of data structure and encoding format would be very useful to facilitate response negotiation.

These ideas were sketched out in the data pipeline illustration and ended up implemented in the EOX API as one of the URL path elements as described in the table above.

The structure of the collection resources is similar. However sub-resources use different names: fields for EOX vs. variables for Terradue and GeoSolutions. The specific resources are:

Collection metadata in all cases contained start/end date, but the temporal resolution is missing thus making it difficult to determine. For example, if the collection is a collection of discrete time instances or daily/monthly periods. The DGGS-based collection metadata additionally publishes the available spatial resolutions of the available grid zones.

The actual API results are not comparable due to differences in the source data. The EOX endpoint uses original S5P L2 scenes. The Terradue endpoint uses gridded S5P monthly aggregates. The GeoSolutions endpoint uses Sentinel-2 scenes over Canberra stored in two different DGGS grid systems (H3 and rHealPix).

EOX fields parameter support simple algebra which is not supported in the Terradue and GeoSolutions endpoint.

Finally, all endpoints provide no capability to specify the temporal resolution of aggregation. For example, the APIs cannot be asked to provide all data between 2000 and 2001 aggregated as monthly means. Specification of spatial resolution is only possible in the GeoSolutions DAPA API through the use of the resolution parameter that selects the zoom level in which the aggregation is executed.

The primary function of this notebook was to compare NO2 tropospheric column between March/April 2019 and March/April 2020 using Sentinel-5P data products.

The source code for this notebook is available here: https://github.com/opengeospatial/t16-dapa-terradue-sentinel-5p-NO2

A live link that uses Binder to execute the notebook can be found here: https://mybinder.org/v2/gl/terradue-ogctb16%2Fdapa%2Fd167-api-endpoint%2Ffastapi-dapa/master?urlpath=/proxy/8001/

The operation of the notebook proceeds as follows:

The first step is to identify the available collections querying the DAPA endpoint.

The response returns a JSON document

The second step is to list the variables available on the selected collection.

The returned JSON object identifies the available variables.

And then the variable name can be retrieved.

With this information the user is able to query using a location name.

The xArray represents the obtained data.

The data can be retrieved to a different object and visualized.

The request can be executed with a aggregation function over space.

And a plot can be made.

And with DAPA the request can also be executed with a aggregation function over space and time.

The DAPA API is implemented by extending the OGC-EDC application. This application (app_) provides a layer of OGC services on top of Euro Data Cube (EDC). The source code is available on GitHub in the Euro Data Cube organization.

The DAPA API itself is available as an app from the EDC marketplace and can easily be deployed in any user’s workspace. The source code is also available on GitHub in the tb16-dapa branch of the OGC-EDC app.

The DAPA API app description on the EDC marketplace provides a detailed step-by-step guide as well as tutorial Jupyter notebooks to set up and use the DAPA service endpoint. In the end it is as simple as deploying the app from the marketplace in the before activated EDC EOxHub Workspace after registration at Euro Data Cube. There is a free three-month promotional plan available to try out the platform.

Once the app is successfully deployed, the generated access URL can be used in the provided notebook tutorials in the JupyterLab environment of the user’s workspace. The tutorials are available either from the EDC marketplace or directly in the JupyterLab environment from where they can with one click be imported in the user’s workspace for interactive usage.

The DAPA API service endpoint is based on the Euro Data Cube environment and thus provides access to a wide range of datasets:

Please note: The API might not work with every dataset. The API has been tested successfully with Sentinel-2, Sentinel-3, and the Digital Elevation Model (DEM).

The online DAPA API app description provides a detailed API documentation which is not copied here. A technical API description using the Swagger UI is included in the app as well.

The deployment as described above in the Usage section relies on the Euro Data Cube (EDC) platform. The platform is deployed in a Kubernetes cluster provided by the Amazon Web Services (AWS).

The EDC EOxHub Workspace is a dedicated namespace in the Kubernetes cluster with reserved resources according to the user subscription. By default, a JupyterLab environment is provided in each user workspace to interactively execute provided notebooks. Available apps can be deployed in the workspace and are available under custom URLs. This way each user has, for example, their private custom DAPA API endpoint running on the reserved resources. In particular this setup ensures that there is no interference with other users using the DAPA API.

The DAPA API is implemented in the Python programming language (3.6) using the Flask web application framework. The DAPA API is built on the OGC API layer of EDC, and inherits all of its capabilities.

All requests to the provided endpoints are internally translated to API calls to the catalogue and processing API of EDC. Only minor processing is actually done in the DAPA App itself. To route these requests, extensive use of the OAuth and “requests” libraries is made.

Global Historical Climatology Network Daily (GHCN-D) is a dataset from NOAA that contains daily observations over global land areas. The dataset contains station-based measurements from land-based stations worldwide, about two thirds of which are for precipitation measurement only. Other meteorological elements include, but are not limited to, daily maximum and minimum temperature, snowfall, and snow depth. The dataset is a composite of climate records from numerous sources that were merged together and subjected to a common suite of quality assurance reviews.

The 2019 data from the GHCN-D dataset was made available via the API. The API provided access to 115,082 weather stations and 34,093,913 observations.

The endpoint provided a Web API consistent with the emerging OGC API family of standards. The API provided access to the weather observations in two ways:

The landing page of the Web API was at https://t16.ldproxy.net/ghcnd.

Note that all readable URIs for this endpoint are not percent-encoded for better readability.

The API was formally specified using OpenAPI, available as JSON, YAML and HTML. The HTML documentation also acted as an interactive client and could be used to play with the API.

Consistent with OGC API practice, the API supported JSON and HTML for most endpoints.

Software clients including scripts in a Jupyter Notebook would typically request responses in JSON - either by including application/json or application/geo+json in the Accept header of the HTTP request or by adding a query parameter f=json to the request.

The HTML responses enabled to explore the API directly in the web browser without the need for a specific client.

The endpoint implements the standard resources for an API implementing OGC API standards sharing geospatial data:

This section lists some examples of how to access the raw observations directly as GeoJSON.

The API endpoint for these requests is:

This request returns 10 observation features and a link to the next "page" of features (the link with rel set to next ). This enables clients to page through the observations. The response is shown below but reduced to two observations.

Each observation has the point geometry of the weather station that recorded the observation and the following properties based on the Observation & Measurement standard:

The response also includes a JSON-LD 1.1 context that relates the JSON values to definitions in the W3C Semantic Sensor Network Ontology and the GeoJSON vocabulary. JSON-LD-aware clients can process the observation data and convert it to RDF.

Query parameters can be used to adjust the request and select specific observations. The following request selects the first 1000 precipitation observations in the border area of Germany, France and Luxembourg during August 2019:

The next request selects the maximum temperature at Berlin-Tegel airport on July 29, 2019, with the geometry in the response in the Web Mercator projection.

In addition to the access to the raw observation data, additional patterns to access and process the observation data are available via the DAPA endpoints. Data retrieval in DAPA is based on the selection of the observations by the spatial sampling geometry (point, area, or grid), the temporal sampling geometry (instant or interval), and the observed properties followed by optional post-processing to aggregate observations by space and/or time.

The endpoint provided by interactive instruments covered two use cases described in Use Cases: Data Retrieval and Data Processing. This endpoint processed the data in two phases, which are described in the next sections.

Variables are the properties that have been observed and are included in this collection of observations.

This DAPA endpoint returns a time series with daily observation values at the selected location (parameter coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

The time series contains values for each selected variable (parameter variables) for which a value can be interpolated at the location.

This DAPA endpoint returns a time series for each station in an area (parameter box, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

Each time series contains daily observation values for each selected variable (parameter variables) for which a value has been observed at the station during the time interval.

This DAPA endpoint retrieves a time series for each station in an area (parameter box, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime). Each time series contains daily observation values for each selected variable (parameter variables) for which a value has been observed at the station during the time interval.

The data is then resampled to a time series for each cell in a 2D spatial grid.

This DAPA endpoint returns observation values at the selected location (parameter coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

All values in the time interval for each requested variable (parameter variables) are aggregated and each of the requested statistical functions (parameter functions) is applied to the aggregated values.

This DAPA endpoint returns observation values for an area (parameter bbox, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

All values for each requested variable (parameter variables) are aggregated for each weather station and each of the requested statistical functions (parameter functions) is applied to the aggregated values.

This DAPA endpoint returns a time series for an area (parameter bbox, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

All values in the area for each requested variable (parameter variables) are aggregated for each time step and each of the requested statistical functions (parameter functions) is applied to the aggregated values.

This DAPA endpoint returns observation values for an area (parameter bbox, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime).

All values for each requested variable (parameter variables) are aggregated and each of the requested statistical functions (parameter functions) is applied to the aggregated values.

This DAPA endpoint retrieves a time series for each station in an area (parameter box, coords or coordsRef) in the selected time interval or at the selected time instant (parameter datetime). Each time series contains daily observation values for each selected variable (parameter variables) for which a value has been observed at the station during the time interval.

The data is then resampled to a time series for each cell in a 2D spatial grid and all values in each time series are aggregated and each of the requested statistical functions (parameter functions) is applied to the values.

Description: this method returns the list of collections exposed by the DAPA endpoint

Method: GET

Parameters: None

Response format:

Description: this method returns the list of variables for the collection with the id {collection_id}

Method: GET

Parameters: collection_id - the collection id

Response format:

Description: this method retrieves data at a location (point sampling) for a time instant, interval or series the collection with the id {collection_id}

Method: GET

Parameters:

Response format: xarray serialized to JSON

Description: this method retrieves observation data at a location (point sampling) for a time interval or series and aggregate over time the collection with the id {collection_id}

Method: GET

Parameters:

Response format: xarray serialized to JSON

*Description: this method retrieves data in an area for a time instant, interval or series the collection with the id {collection_id}

Method: GET

Parameters:

Response format: xarray serialized to JSON

Description: this method retrieves observation data in an area for a time instant, interval or series and aggregate over space the collection with the id {collection_id}

Method: GET

Parameters:

Response format: xarray serialized to JSON

Description: this method retrieves observation data in an area (polygonal sampling) for a time instant, interval or series and aggregates over space and time the collection with the id {collection_id}

Method: GET

Parameters:

Response format:

Description: this method retrieves observation data in an area (polygonal sampling) for a time interval or series and aggregate over time the collection with the id {collection_id}

Method: GET

Parameters:

Response format:

Description: this method takes a location expressed as a longitude & latitude pair, a bounding box, a Well-Known Text or free-text to return a GeoJSON feature.

Method: GET

Parameters:

Response format: GeoJSON feature

In July 2020, the OGC facilitated a five day virtual workshop to provide an opportunity for hands-on evaluation of technologies and draft specifications that facilitate ad hoc processing and analysis of large-scale observational datasets on cloud computing platforms “close to the data”. The workshop was a collaborative effort between OGC members participating in the EOS thread of Testbed-16 and those engaged in the EOApps Pilot activity. Each endpoint was evaluated based on the following high-level criteria:

EOS participants and other expert workshop attendees evaluated the new draft Data Access and Processing API (DAPA API). The API is intended to provide simple access to processed samples of large datasets for subsequent analysis within interactive Jupyter Notebook (JPYNB) environments. They also examined procedures and tools for moving a local DAPA - JPYNB environment to the cloud as a Docker-based application package (AP) for sharing and close-to-data execution in complete and reproducible form.

EOApps Pilot participants then demonstrated and evaluated the maturity of several Earth Observation Applications-to-the-Data specifications such as ADES (Application Deployment and Execution Service) developed over the last two years as part of various OGC Innovation Program (IP) initiatives. ADES-enabled exploitation platforms are designed to convert and deploy an application package such as produced by the Testbed-16 tools for batch execution and further orchestration as an independently invocable processing service.

The following table provides a summary of the agenda.