OGC Testbed-16: Data Access and Processing Engineering Report

The purpose of the EO Exploitation Platform Working Group is to foster interoperability (i.e. the pre-requisites for successful interaction between platform components and across platforms) among Exploitation Platforms and their components. To this end, the working group will act as an open form for discussion and documentation of interoperability requirements for the domain in order to drive OGC standards evolution towards better support for the use cases of the EO Exploitation Platforms.

The increasingly huge amount of new EO satellite data and In-Situ data every day has incentivized the creation of several web accessible platforms that enables scientists and commercial operators to use such data without the need to download content and have in-house Information Technology infrastructure to manage the large volume of data.

EO Exploitation platforms have been independently developed by public organization or commercial companies, but all share a common set of functionalities:

These platforms, however, have many different implementations, with interfaces and data formats which often hampers interoperability among platforms. In fact, the next step in this data exploitation revolution is to link the platforms together creating an ecosystem. This ecosystem is potentially across different administrative domains, in which each platform can contribute with the offered services to the implementation of more complex use cases.

The work in this Testbed 16 thread focused on the data process and analysis aspects of EO exploitation platforms. The particpants believe that the outcomes of this testbed would be of interest to users of EO exploitation platforms. Specifically the work of the testbed is to develop data access and processing interfaces that are simpler to use and thus simpler to learn and more easily integrated into the kinds of tools that EO exploitation platform users might use (e.g. using Jupyter notebooks to perform some analysis).

The purpose of the Environmental Data Retrieval (EDR) API SWG is to standardize several APIs, defined using OpenAPI (Version 3), to retrieve various common data patterns from a relatively persistent data store. The data patterns could include, but are not restricted to, data at a point in space and time, time series at a point, data along a trajectory, which may be 2, 3, or 4 dimensional, and covering a specified polygon or rectangular tile. The APIs will enable service users to retrieve resources over a discrete sampling geometry, created by the service in response to a standardized query pattern.

The work of the Testbed-16 Data Access and Processing task was closely aligned with the stated goals of the SWG. The participants believe that the outcomes of this testbed task could help inform the design of the API components that the SWG is endeavouring to specify.

The purpose of the OGC API - Processes SWG is to design a Web API that enables the execution of computing processes and the retrieval of metadata describing their purpose and functionality. Typically, these processes combine raster, vector, coverage and/or point cloud data with well-defined algorithms to produce new raster, vector, coverage and/or point cloud information.

In the current design of the Processes API, executing a process involves creating a JSON document that contains the process inputs and POSTing that document to the execution endpoint of the process. From an end-user perspective, this workflow is more complicated than may be necessary. Since one of the goals of the testbed task was to design a simpler API for invoking processes, the participants believe that this work could inform the design of an alternate invocation of WPS processes using KVP/GET method rather than the current JSON/POST method.

Furthermore, the work being done in Testbed-16 for binding processes to specific datasets and to advertise which combinations of data and processes are valid would be of interest to the SWG.

There are a large and increasing number of citizen science projects active around the world involving the public in environmental monitoring and other scientific research activities. The OGC Citizen Science DWG is motivated to support citizen science by providing a forum for increasing understanding and demonstration of the benefits brought by the use of open standards and best practices. This DWG will support the development of improved interoperability arrangements for the citizen science community.

The work of the testbed to simplify the interfaces for accessing and processing data from the end-user perspective is directly related to the goal of the DWG to improve interoperability arrangements for the citizen science community.

The volume of Earth Observation data has grown so large that moving such data to a local machine for processing is no longer feasible. Instead of moving the data to local processing, the trend is now to store large repositories of Earth Observation data in the cloud or compute back-ends, and move the processing software to the data. The results of this processing on the cloud can then be browsed remotely or downloaded as desired.

In order to enable this "move the process to the data" concept, the openEO organization developed an API that connects clients such as R, Python and JavaScript to big Earth observation cloud back-ends in a simple and unified way. The main objectives of the project are the following concepts:

GeoTrellis is a geographic data processing engine for high performance applications. It is implemented as a Scala library and framework that uses Apache Spark to work with raster data and supports many Map Algebra operations as well as vector to raster or raster to vector operations.

The OGC GeoAPI Implementation Standard defines the GeoAPI library. GeoAPI provides a set programming interfaces for geospatial applications. In a series of packages or modules, GeoAPI 3.0 defines interfaces for metadata handling and for geodetic referencing (map projections). The GeoAPI interfaces closely follow the abstract models published collaboratively by ISO in its 19100 series of documents and the OGC in its abstract and implementation specifications. GeoAPI provides an interpretation and adaptation of these standards to match the expectations of Java or Python programmers. C

For this use case the typical user is a developer.

The user wants to access geospatial data for a specific area in a simple function call. The function call identifies the data and allows the user to define the discrete sampling geometry. Valid geometries include:

All geometries are provided either in-line or by reference, as illustrated by the following examples:

Specifying sampling geometries by reference also supports the use of an OGC API - Feature endpoint as shown in the example above where the value of the location parameter is an OGC API - Feature invocation to fetch the feature, and thus the geometry, of the city for Frankfurt.

Users need the ability to access the original data. The use of the term "original" is a little ambiguous in this context as data often undergoes some process on its way from original access to the final product. As an example, imagine a digital temperature sensor. The actual reading performed in the sensor is some form of electricity, but the value provided at the sensor interface is 21°Celsius (approximately 69.8°Fahrenheit). Thus, some form of calibration curve has been applied to the original reading, which might not be available at all. In this case, the value 21°Celsius can be considered as “original”. The same principles apply to satellite data. The original raw data readings are often not accessible. Instead, the data undergoes some correction process before being made available. Higher product levels may include orthorectification or re-gridding processes. In any case, data providers need to provide a description of the performed processing together with the actual data. In addition, data should be available as "raw" as possible.

End-users want to retrieve all data that exists within the provided target geometry. In the case of polygon geometries, the end-user would receive all data that intersects that polygon. In the case of point geometries, the end-user would retrieve the value exactly at that point.

In addition, end-users have the option to define the (interpolation) method for value generation. If no option is selected, the Web API indicates how a given value was produced. Testbed-16 participants developed a set of frequently used production options, including for example:

or any combination thereof. This use case differentiates the following data requests:

Testbed-16 participants explored these use-cases in combination with additional processing steps. For example, the end-user requests synoptic, map, or time series data, that is interpolated to a grid.

Testbed-16 participants explored simple data processing functions. These include calculations for:

These values are for any given data retrieval subset as accessible in the Data Retrieval use cases.

This section describes the detailed use cases articulated by the Testbed 16 participants, based on the general use cases, that guided the development of thread clients and end-points.