OGC Earth Observation Applications Pilot: Spacebel Engineering Report

Data Discovery can be performed through either OpenSearch (two-steps) queries or OGC WMS/WCS API.

Despite that the data discovery aspects are not yet properly covered in the Testbed architecture, Testbed-14 already introduced the OpenSearch API for selecting the EO data inputs submitted to the process.

The Spacebel platform already includes an OpenSearch Catalogue, and detailed product metadata can be requested in the OGC O&M EO profile format (OGC 10-157r4). The Web Map Service (WMS) and WCS are not supported.

Data can be retrieved by an internal stage-in (performed by the ADES), through WMS/WCS, or using Object Storage.

The Spacebel platform stage-in requires mounting local data resources to the Kubernetes cluster node. In addition to the nominal stage-in, it is not perfectly clear if the proposed alternatives offer direct (network) access from the application, or if they suggest the support of those protocols for provisioning local files (as for the stage-in).

The Spacebel platform fetches all remote HTTP and FTP URLs and makes them available through the local file system to the processes. As discussed during OGC Testbed-14, the application descriptor should typically describe if the process supports natively HTTP or Object Storage access.

The agreed ADES/EMS interface is based on the draft OGC API – Processes specification, with REST JSON bindings, and Web Processing Service (WPS) standard.

The Spacebel platform required to be aligned with the work performed during OGC Testbed-14. However, the OGC repository hosting the mentioned OpenAPI specification does not exist anymore, so Spacebel shared its own version (OGC-ADES-Hackaton) reviewed during the 2019 OGC API Hackathon that was held in London [1].

The baseline also recommended to explore the handling of asynchronous callback. Because most application providers focus on the platform’s user interface, the priority was not given to this aspect during the project.

The agreed Application Package is the CWL profile defined in Testbed-14. Additionally, bulk (parallel) processing might be investigated.

Based on the Application Providers feedback, Spacebel essentially worked on the parallelization approach, as described further in the document.

Applications developed by application providers are deployed using the Application Deployment and Execution Service (ADES). OGC Web Processing Service 2.0 is the standard interface used to expose all ADES operations (application discovery and execution management). Some operations rely on an OGC WPS extension:

An EMS is defined as the Thematic Exploitation Platform (TEP) processing frontend ahead of multiple platform (Mission Exploitation Platforms).

The end-users of the TEP register their applications through the EMS, and the target platform used to execute a process is selected at runtime based on the location of the data to be processed. The EMS also provides support for processing chains: workflows can be deployed as a CWL (Common Workflow Language) document. The Business Process Model and Notation (BPMN) alternative has also been explored in Testbed-14 [5].

The current EMS interface also implements the WPS standard, and thus the component acts like a proxy aggregator.

The application is registered in a package providing the execution unit and optionally the application descriptor. Testbed architecture currently defines the following Application Package profiles:

Examples are provided in OGC Testbed-14: Application Package Engineering Report [2].

The EO Regions! Platform was originally funded by the Walloon region and is being operated by Spacebel. EO Regions! is the Walloon hub of the European EUGENIUS network and is recognised as Copernicus Relay for Wallonia by the European Commission for the promotion and support to users of Copernicus data. The main user facing Web Portal is available at http://www.eoregions.com/ and provides more background information about the platform and its objectives.

The Walloon regional government has recently confirmed its industrial policy and related strategy for exploiting EO data in its Déclaration de Politique Régionale (DPR) 2019-2024 , and the intention to implement the Walloon part of the Luxembourg/Walloon Collaborative Ground Segment (CGS). EO Regions! and its applications (currently hosted on Google Cloud Platform) will be hosted on the CGS (expected to be located in Transinne-Belgium) when it becomes available.

The EO Regions! platform mostly relies on a set of components known internal as "Geospatial Exploitation Platform" and described in section below.

The main components of the EO Regions! Platform are described below:

The nominal scenario for developing an application that can be registered and executed on an EO platform consists of the following steps:

Docker is a widely used technology that provides OS-level virtualization and has been introduced since Testbed-13. The application can be packaged as standard containers that embed all environment and software dependencies for running applications written in any language, and a number of tools to manage the execution of the containers.

The main complexity of deploying Docker containers is that the runtime environment and dependencies must be prepared by Application Providers.

Additionally to the Docker Application Package, the ADES might support the build of Docker images for a set of formats proposed to novices and non-experts users such Java, Python, Go, etc.

For each language or backend environment, the ADES defines an Application Package profile (e.g. in Java, the library main class typically must copy a given interface). From the registered source package, the ADES then automatically generates the application descriptor, builds the binaries, and then prepares the Docker environment.

As all possible packages are actually packaged as Docker images, the support of additional languages or backends only requires effort for implementing build pipelines of the specific language: the execution remains a container managed process.

The access control and confidentiality of Docker images has been discussed as an important aspect of the containers management. As a matter of fact, the architecture has yet only been prototyped in testbeds with Docker images being publicly available on the Docker Hub. To this end, Spacebel has provided access to a private platform repository during the project.

However, the automation of the Docker build on the ADES side (explained in above section) could also be envisioned to solve the access control aspects of containers: instead of providing the Docker image reference, the application provider would supply the software source and Dockerfile in order to let the ADES build the container and register the image in the platform registry. This would also solve the container access control issues related to federated platforms as the involved ADES would trust and share their mutual Docker registries.

When registering a given Docker container image, the ADES has no control of the remote reference of the Docker image. Concretely, the Application Provider could update the docker image and break a working process by introducing bugs in the updated image.

Still with the same approach of building the image on the ADES side, providing the source and Dockerfile would avoid any uncontrolled changed and enforce the Application Provider to request an update using the dedicated and explicit operation.

Some concerns have been discussed about the security aspects related to Docker containers. Although some isolation can be enforced on containers (e.g. through non-privileged Docker mode, closed Kubernetes environment, advanced write access control), executing the containers as the root user is generally seen as a bad practice.

Again using the same approach of generating the Docker image on the ADES side, the possible security breaches might be detected by inspecting the Dockerfile and fixed before building the container.

The Process Description is also generic as used with any specific Application Packages (e.g Docker, BPEL, Java, etc.). Therefore, defining properties (e.g. hardware requirements) in the Process Description benefits all implementations and projects.

Therefore, Spacebel advises, for any Application Package definition, to allow the Application Provider to supply the Process Description, and thus provide advanced information about the process.

As very often in Processing, the applications generally require a minimal hardware configuration for one of the following resources:

When using the CWL language, the ResourceRequirement property can be used to express such requirements.

However, Spacebel recommends not relying on the specific CWL language, while the hardware requirement is a general concern that should be handled thus by the OGC WPS specification and for any kind of Application Package. In particular, we suggest to define a set of new properties in the Process Description based on the Kubernetes properties requests and limits as documented on Managing resources for Containers page.

Furthermore, the network policies might be restricted and controlled by the ADES component. Therefore, it might also be useful to list the possible external URLs (e.g. remote catalogue or data repository) required by the application in the Process Description. More generally, all properties supported by Kubernetes (resource sharing, health check, affinity, security context, priority, etc.) could be potentially explored if relevant.

As already mentioned, the Process Description is the document that allows a client application to discover and interact with a specific application. The information provided by the Process Description may include details on the supported inputs formats, the constraints, some useful services or endpoints for collecting the inputs, etc.

Inputs & outputs constraints was a major topic of the CFP, and was confirmed as a main concern from the Application Providers. Moreover, the users looked at how they might express some correlations between some inputs, such as '12 days older than input 1', or "same relative orbit".

In the architecture, the EO data products can be searched using the OpenSearch standard extended with the OGC OpenSearch Extension for Earth Observation specification (OGC 13-026r8) and OpenSearch Geo and Time Extensions (OGC 10-032r8). Therefore, the properties defined by the OpenSearch specification qualify perfectly as standard for expressing the inputs constraints in the Process Description: geo:box, geo:geometry, geo:relation, time:start, time:end, time:relation, eo:orbitNumber, eo:acquisitionType, eo:cloudCover, eo:snowCover, etc.

Furthermore, future work could explore how those properties may define possible values, ranges or expression from other inputs. The following example define some input constraints as being 5 days older than input1, between 0 and 60 percent of cloud coverage, and with a geometry intersecting input 1:

Finally, expressing the constraints like this would also clearly facilitate the automation of executions. During our technology integration experiment with the SatCen Coherence application, we missed the means for expressing the discussed constraints (e.g. same relative orbit and previous sensing period): our scheduler component may automatically compute the constraints and enforce the criteria when selecting the products.

This technology integration experiment consisted of the parallel execution of the DLR Urban Indicee process, and the automated triggering of the Temporal Aggregation process from available Urban Indicee products. The source products were Sentinel-2 scenes retrieved from an OpenSearch Query (based on the 28QED tile during the experiment).

The diagram below provides an overview of the integration experiment:

The parallel execution of Urban Indicee is started manually using the Scheduler component of the Spacebel platform. The query includes:

The automated execution of Indicee Aggregator is configured using the Scheduler component of the Spacebel platform. The query includes:

The Indicee executions were started in parallel as shown on the figure below:

The Indicee Aggregator execution was then automatically triggered as soon as Indicee output data and metadata were available and successfully completed. The screen capture below shows the aggregator received 10 source Indicee products:

This experiment essentially showed the power of a scalable cluster (Kubernetes) for processing a set of tasks.

The CWL below essentially provides the Docker image reference (dockerPull), the UNIX command (baseCommand), the various command arguments (describing the command-line format), and the output file pattern (glob).

The SatCen experiment executed, in parallel, a coherence analysis from 3 pairs of Sentinel-1 products. It then automatically triggers the multi-coherence process from the two generated coherence outputs. The diagram below provides an overview of the integration experiment:

The parallel execution of Coherence is started manually using the Scheduler component of the Spacebel platform. The query includes:

The automated execution of multi-coherence is configured using the Scheduler component of the Spacebel platform. The query includes:

The coherence executions are started in parallel as shown on the figure below:

The multi-coherence execution is then automatically triggered as soon as a coherence output is available and successfully completed:

Unfortunately, the second input submitted to the multi-coherence process by the scheduler component is not the previous coherence product. Indeed, no sorting option is supported when configuring the automated execution.

When trying to automate the execution of the coherence process, a gap of current ADES component is clearly highlighted because of the following expectations:

ADES (in particular the Process Description document) does not define any property for expressing such advanced constraints (spatial, temporal, others). As already mentioned, OpenSearch properties would enable declaring allowed values as well as mandatory search fields for a set of inputs (e.g. in this example: the relative orbit and geometry).

Going further, if the EO product metadata (e.g. O&M EOP) was provided as input to the ADES, the value of some inputs (such aoi_wkt) could be expressed and computed at runtime from the input metadata, e.g. : aoi_wkt=input_files.geometry

The CWL below still embeds the same information but points to a different Docker image, and includes 2 inputs (inputfiles and _aoi_wkt).

Multi Coherence CWL:

Pixalytics’s application executes a Mosaic process from EO data downloaded from a remote repository.

The input selection in this experiment is hardcoded in the application container. Therefore, the application is simply executed using the execute button of the platform user interface.

The Mosaic process generates a single GeoTIFF output as illustrated on figure below.

From a provider point of view, the Mosaic process has the particularity to fetch remote data. Pixalytics explored multiple approaches which highlight valuable improvement such as supporting remote storage system support (S3, FTP, etc.) and handling authentication to possibly mount and access remote repositories.

During the project, Pixalytics also expressed the need to register the Docker image in a private Docker registry. To this end, Spacebel provided access to such a Docker registry.

The CWL of the Mosaic process is provided below:

ESA’s application executes a Mosaic process from S3 EO data.

The input selection is performed manually by browsing the Catalogue S3 products. The execution is started using the platform User Interface.

The Mosaic process generates a GeoTIFF image as output, a PNG quicklook and a a property file.

The experiment required a new EO data collection (Sentinel-3) which has been added to the ingestion system.

Moreover, ESA tried to rapidly be able to deploy and execute the process which showed that a simpler deployment profile would be welcome. Indeed, in this experiment, the CWL does not really bring any added-value and is too complex.

The CWL of the Mosaic process is provided below: