OGC Earth Observations Applications Pilot: Terradue Engineering Report

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

Terradue’s Ellip Platform provides an application integration and processing environment with access to EO data to support Earth Sciences. The Ellip User Algorithm Hosting Service gives access to a dedicated application integration environment in the Cloud, with easily exploitable software tools and libraries and access to distributed EO Data repositories to support the services adaptation and customization. The service provides developers with well-defined operational processes and procedures in order to allow for robust packaging of applications, test and validation activities, and application deployment in production. The adaptation and customization of the services is focused on supporting the developer in defining the parallelization strategy, the data requirements needs, tools and libraries necessary to execute the applications and the overall production strategy.

Ultimately, the User Algorithms are exposed through a Web Service endpoint, Web Processing Service (OGC WPS), that allows end-user portals and B2B client applications to pass processing parameters, trigger a data processing or systematic requests and establish the data pipeline to retrieve the information produced.

This section shows how Ellip was adapted and the evolutions implemented to respond to specific challenges addressed during the pilot activities.

Previous OGC Testbeds (13, 14 and 15) initiated the design of an architecture to allow the packaging, deployment and execution of Earth Observation Applications in distributed Cloud Platforms.

These testbed activities built on OGC standards (e.g. WPS, OWS-Context) to describe data processing applications or workflows as Application Packages that can be deployed on and executed within diverse Cloud Platforms.

The WPS service allows end-user portals and B2B client applications to pass processing parameters, trigger on-demand or systematic data processing requests and establish the data pipeline to retrieve the information produced.

The Application Package includes information about the execution unit to be executed or specific workflow script that can be invoked on the processor directly together with the mappings for parameterization or tailoring.

The testbeds defined an architecture where the Application Package is used with an Execution Management Service (EMS) and Application Deployment and Execution Service (ADES) as seen in the figure below. The EMS provides the workflow orchestration service (WPS-T) to deploy and invoke processing services workflows on multiple deployment and execution services. The ADES provides the execution engine service of the application that was previously deployed as a WPS service by the EMS.

The Execution Management Service (EMS) provides the orchestration service to deploy and invoke services of an application package on an ADES of the selected platform. It is responsible for managing the on-demand deployment and execution of the workflow building blocks on the “local” or “remote” platform.

The main responsibilities of the ADES are:

In order to accomplish the execution and monitor steps above, the internal sub-steps also need to be responsible for the operations of:

An Earth Observation Platform provides processing functions, tools and applications invoked individually or utilized in workflows. Users are able to integrate their own processing services into the platform and make them available for exploitation by other users also individually or in their own workflows. To support their integration activities, users are provided with an environment where they can integrate, build, test & debug and deploy.

The platform gives access to a dedicated application integration environment with exploitable software tools and libraries and access to distributed EO Data repositories to support the services adaptation and customization. It allows the design and integration of services as scalable data processing chains, leveraging selected data programming models.

The environment considers two scenarios of EO application integration:

The main objective of the current pilot project is to cover the importing scenario.

The Application Package targets algorithms (developed by a third-party developer) that are to be deployed in a Cloud Platform. The Application Package will contain all the information necessary to allow applications to be specified and deployed in federated resources providers. The Application Package takes into consideration past efforts to integrate OGC services with well-established formats and protocols that are natively ready for interoperability within Cloud solutions.

The main objective of the Application Package description file is to define a data processing application or workflow providing the information about the parameters, software item, executable, metadata and dependencies such that it can be deployed to and executed within an Exploitation Platform. This file must require a simple encoding so that application developers are able to create it, ensure that the application is fully portable among all supporting platforms and supports automatic deployment in a Machine-To-Machine (M2M) scenario. The Application Package information model must also allow the deployment of the application as a WPS service.

The execution block (i.e. Application Artefact) describes the ‘software’ component that represents the execution unit in a specific container (e.g. Docker) to be executed or specific workflow script that can be invoked on the processor directly. Based on the context information provided with the processor, the execution block maps how the container can be parameterized or tailored.

Docker is currently the selected container format and enables applications to be quickly assembled from components and eliminates the friction between development and production environments. As a result, applications can ship faster and run the same process into large multi-tenant data centers in the Cloud. Docker images work as an isolated process in the host operating system, which shares a kernel with other containers. Thereby, still having the benefits of virtualization, it is more portable and more effective by using less disk space. The application packaging allows the developers to remove their focus from the infrastructure details.

Previous work in OGC Testbed-13 (OGC 17-023) defined two alternative encodings for the Application Package. One as an OWS Context Document (OGC 12-084r2) with an XML encoding and an alternative where the information was included in the WPS Process Description in the ows:metadata element.

Subsequent OGC Testbed-14 (OGC 18-049r1) updated this model with an OWS Context JSON encoding (OGC 14-055r2) with the addition of a (embedded) Common Workflow Language (CWL) file (in JSON or YAML) that could be used in the simplest case to describe a single application and how to invoke the command-line within the container.

The CWL is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high-performance computing (HPC) environments. While CWL is not heavily used in the EO domain, it is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.

The CWL contains two main specifications. The Command Line Tool Description Specification that specifies the document schema and execution semantics for wrapping and executing command line tools and the Workflow Description Specification that specifies the document schema and execution semantics for composing workflows from components such as command line tools and other workflows.

To summarize, OGC Testbed-14 proposed the use of CWL initially for the command line invocation, the activities during this pilot showed a simple way of pointing to a Docker container from inside the CWL file included in the Application Packaging. We concluded that the CWL file has the potential to reference the application containers (e.g. Dockers) and also allows the definitions of the application parameters, input/output interface and the overall process offering parameters.

Previous OGC Testbeds (13, 14 and 15) followed an approach where the ADES is responsible for providing the data stage-in of product input references specified on the WPS request and stage-out the results of the application.

Data stage-in is the process to locally retrieve the inputs for the processing. Processing inputs are provided as EO Catalogue references and the ADES is responsible to translate those references into inputs available for the local processing.

Data stage-out is the process to upload the outputs of the processing onto external system(s), and make them available for later usage. ADES retrieves the processing outputs and automatically stores them onto an external persistent storage. Additionally, ADES should publish the metadata of the outputs onto a Catalogue and provide their references as an output.

In Testbed-13 the ADES makes the files available in a specific location defined by an environment variable pointing to the working directory. In Testbed-14, the input files location was provided directly in the CWL file, mentioned in the previous section, by defining one or more input files or folder directories.

However, EO product files come in different formats (e.g. GeoTIFF, HDF5, SAFE) and might have sub-items (e.g. metadata, bands, masks) that can be encoded in the same file or follow a given folder structure.

For example, SENTINEL-2 products are made available to users in SENTINEL-SAFE format, including image data in JPEG2000 format, quality indicators (e.g. defective pixels mask), auxiliary data and metadata. The SAFE format wraps a folder containing image data in a binary data format and product metadata in XML. A SENTINEL-2 product refers to a directory folder that contains a collection of information that can include several files like seen in Figure 5.

Different platforms will catalog and stage the products differently. Some will reproduce the exact folder structure and return the folder root or the main XML manifest file. Other platforms will compress the folder structure and return the compressed file. Other platforms will even convert the file and return a GeoTIFF aggregating all the information.

In general, the approach followed in the previous testbeds assumed that the applications were responsible for navigating the input folder directory and programmatically reacting to how the file was staged-in by the platform. This implies that the application developer needs to consider all possible cases when developing their read routines.

Conversely, the outputs of the application are fully managed by the developer that must place the resulting files in the output directory. The only information the ADES has about the output files is the file media type (formerly known as “MIME-type”).

To summarize, in previous testbeds the ADES had limited knowledge about the input and output files respectively their data contents. Thus, ADES was often missing critical information like spatial footprint, sub-items (e.g. masks, bands) and additional metadata (e.g. ground sample distance, orbit direction). To respond to this issue, the activities during this Pilot activity showed the potential to follow an approach that uses a local catalogue encoded using the SpatioTemporal Asset Catalog (STAC) specification as a data manifest for application input and output [2].

The STAC specification standardizes the way geospatial assets are exposed online and queried. A 'spatiotemporal asset' is any file that represents information about the earth captured in a certain space and time (e.g. satellites, planes, drones, balloons). Most importantly the STAC specification can be implemented in a completely 'static' manner as flat files, enabling data publishers to expose their data by simply publishing one or several JSON files drilling down from collection (e.g. Sentinel-2), item (e.g. Sentinel-2 product) and assets (e.g. JPG2000 band file, auxiliary data, browse) with a relative path (something that was not possible using OpenSearch as defined by OGC 13-026r8, OGC 13-032r8).

The STAC specification defines several objects:

The STAC specification supports the ADES stage-in operation for all the identified application patterns using EO data catalog references as inputs. A local STAC catalog for the fan-in (Figure 6) and pairs (Figure 8) patterns or several local STAC catalogs with a single item for the fan-out pattern (Figure 7).

In each item, there can be multiple assets expressed in relative paths enabling the application to navigate on the mounted volume. The use of STAC in the ADES data stage-in and stage-out follows the following steps:

The Application Package in the Exploitation Platforms context is required to:

The Common Workflow Language (CWL) targets data-intensive processing scenarios and makes these portable and scalable across platforms capable of interpreting and execute the processes by describing:

CWL is used as the application package descriptor. It covers the following elements necessary to describe the application:

The application package is thus composed of a CWL file with the role of the application descriptor. The container reference is included in the CWL using the CWL DockerRequirement.

The CWL has two main classes: Workflow and CommandLineTool.

The Workflow class defines an analysis task represented by a directed graph describing a sequence of operations that transform an input data set to output.

The CommandLineTool class defines a non-interactive executable program that reads some input, performs a computation, and terminates after producing some output.

For the application package, the Workflow class level is the interface used to map the parameters of the exposed WPS service and respective mapping with the exposed service. The CommandLineTool class defines the actual interface of the application.

The CWL Workflow class defines the overall processing service. As seen in Figure 9 there are two main sections: one for Service definition and the other for Service parameters.

The ADES offers an entry point to deploy the application package. Different protocols exist to perform the application deployment such as WPS-T that defines transactions.

Regardless of the deployment mechanism, CWL is used to convey the information to describe the web service inputs/outputs using WPS concepts. The mapping between CWL and WPS follows the rules described in this section.

The Workflow level is the entry point to the overall CWL workflow and thus the interface used to map the inputs/outputs. The inputs/outputs description defined in the command line or workflow steps are not taken into account directly since they are linked between steps or bound at workflow level as foreseen by the CWL specification. Indeed, at the web processing service level, only the Workflow input and output parameters are exposed. This section describes the mapping rules to describe inputs and outputs in the WPS data model from the possible inputs/outputs definition from the CWL document.

The ADES is responsible for the data flow management by using a local catalogue encoded according to the Spatio Temporal Asset Catalog (STAC) specification as a data manifest for application inputs and outputs. The local catalogue provides knowledge about the input and output files data contents like spatial footprint, sub-items (e.g. masks, bands) and additional metadata.

This section describes the strategy to data stage-in, locally retrieving the inputs products for the processing, and data stage-out, making the outputs of the processing available for the subsequent steps (locally or on external systems).

The European Union Satellite Centre (SatCen) brought an application taking as input an interferometric pair of Sentinel-1 Single Look Complex acquisition. The app produces the interferometric coherence between the two acquisitions and the backscatter for each of the acquisitions as results.

Sentinel-1 Level-1 Single Look Complex (SLC) products are images in the slant range by azimuth imaging plane, in the image plane of satellite data acquisition. Each image pixel is represented by a complex (I and Q) magnitude value and therefore contains both amplitude and phase information. The Interferometric Wide swath (IW) SLC product contains one image per sub-swath per polarization channel, for a total of three or six images. Each sub-swath image consists of a series of bursts, where each burst was processed as a separate SLC image.

The coherence is the fixed relationship between waves in a beam of electromagnetic (EM) radiation. Two wave trains of EM radiation are coherent when they are in phase. That is, they vibrate in unison. In terms of the application to things like radar, the term coherence is also used to describe systems that preserve the phase of the received signal. In an interferogram, coherence is a measure of correlation. It ranges from 0, where there is no useful information in the interferogram; to 1, where there is no noise in the interferogram (a perfect interferogram).

The backscatter is the portion of the outgoing radar signal that the target redirects directly back towards the radar antenna. Backscattering is the process by which backscatter is formed. The scattering cross section in the direction toward the radar is called the backscattering cross section; the usual notation is the symbol sigma. It is a measure of the reflective strength of a radar target. The normalized measure of the radar return from a distributed target is called the backscatter coefficient, or sigma nought, and is defined as per unit area on the ground.

SatCen’s application uses ESA’s Sentinel Application Platform toolbox (SNAP) to derive the coherence and the backscatter and the Geospatial Data Abstraction Library (GDAL) to generate the RGB composite combining the outputs produced by SNAP.

This application implements the fan-in pattern where the two Sentinel-1 SLC inputs are processed in a single process and the input data is staged-in. The processing requires a computing environment with at least 14GB of RAM as declared by SatCen.

SatCen provided a simple CWL file with the CommandLineTool class including the DockerRequirement hint providing the Docker image to use.

Terradue updated the provided CWL to include:

The final CWL is listed in the figure below.

The application was then deployed on Ellip Platform and exposed on a Web based graphical interface as shown in the figure below.

The interaction with SatCen allowed us to consolidate the concept of using the CWL language:

Nevertheless, the application defined by SatCen for the EO Apps Pilot is a simplified subset of a real-life scenario.

During the Pilot, we’ve used a single pair of Sentinel-1 SLC. The inputs could be up to four when using Sentinel-1 A and B over some areas of the world to obtain the change detection with 6 days apart instead of 12 days. In this case, the acquisitions of the same day may have to be assembled before they’re processed by the interferometric chain.

During the Pilot, we’ve processed a single swath. The swaths to process may be IW1, IW2 and IW3 (or a combination of these). We’ve also processed a single polarization VV. SatCen uses this polarization so we could skip that as a parameter.

So in terms of an efficient workflow that reduces moving around large sets of data, we would define a workflow running on the ADES with several steps or directed acyclic graph nodes. We would have a first node that generates processing sets split by swath and master and slave combinations to do the pre-processing in a second node. This second node would then pre-process these sets (fan-out) optionally applying the SliceAssembly operator. A third node and finale would then do the TOPSAR-Merge and generate the Coherence/backscatter products.

This approach can be achieved using a CWL that defines these steps in a workflow.

The German space agency (DLR) brought an application taking as input one or more Sentinel-2 Level 1C products to produce a list of vegetation indexes.

SENTINEL-2 is a wide-swath, high-resolution, multi-spectral imaging mission with its Multispectral Instrument (MSI) sampling 13 spectral bands: four bands at 10 meters, six bands at 20 meters and three bands at 60 meters spatial resolution.

The DLR TimeScan Processing Framework utilizes a novel approach to exploit large multi temporal satellite time series for a variety of applications like urban monitoring and land cover classification. DLR’s goal in the OGC EO Apps Pilot is thus to adapt the TimeScan processors to an Application Package that is deployable and executable in an EMS/ADES infrastructure.

The provided application implemented the Sentinel-2 pre-processing module as a processing graph in the ESA SNAP Toolbox including a reader interpreting the multi-resolution input, a resampling operator rescaling all bands to 20m spatial resolution, and the SNAP Sentinel2.Idepix operator generating a flag band with cloud identification, whereas BandMaths operators compute the vegetation indices.

As done by SatCen, the DLR provided a CWL file as the application package. Terradue applied the same enhancements to allow deploying the DLR application on the Ellip Platform as a WPS processing service:

Extending the CWL lessons learned with the SatCen case, the interaction with DLR allowed us to consolidate the concept of using the CWL language to express the fan-out pattern ScatterFeatureRequirement CWL requirement to process inputs in parallel and independently and thus reducing the processing time of large stacks of Sentinel-2 data required by the TimeScan processing scenarios.

The DLR team was provided access to a Web based graphical interface to submit processing jobs as shown in the figure below.

The European Space Agency (ESA), supported by Rhea, created an application taking as input a number of Sentinel-3 SLSTR Level-2 Land products to produce a temporal and spatial aggregation of the Land Surface Temperature (LST).

The principal aim of the SLSTR instrument mission on-board SENTINEL-3 is to maintain continuity with the (A)ATSR series of instruments (on-board the ERS-2 and Envisat satellites). The (A)ATSR provided a reference Sea Surface Temperature dataset for other satellite missions and the SLSTR design is based on the reuse of AATSR concepts with existing and qualified technologies.

The application takes as inputs gridded Land Surface Temperature generated on the wide 1 km measurement grid to aggregate the observations in time and space producing an LST mosaic.

ESA also provided a simple CWL with the CommandLineTool CWL class only. As such, Terradue applied the same changes to the provided CWL to yield the script below.

As for the SatCen and DLR, ESA was provided with the access to a web-based interface to interact with the deployed WPS processing service.

The interaction with ESA’s application allowed us to define how an application provider is asked to test his application before providing it for deployment on an exploitation platform. The method is quite simple: using the cwltool, a CWL execution tool and the set of processing parameters defined in a YAML file, the application provider can run his/her application locally and thus have a reference execution that can be used for validation purposes on the platform in an acceptance phase for example. This testing step is also a valid strategy to identify potential not recommended practices in creating the Docker container (root execution for instance).

This facet of the CWL as a mechanism to describe the application, define how to execute it, describe the inputs/outputs and finally validate the docker execution as if on an exploitation platform provided an additional confirmation that CWL may be the sole required element of an application package to deploy it on an ADES as a WPS service.

Terradue brought the EO Apps Pilot a set of applications derived from the World Food Programme (WFP) Vulnerability Analysis and Mapping (VAM) network that addresses critical aspects of the food security:

The use-cases are focused on seasonal monitoring for tracking growing season and climate & vegetation time series analysis with:

The derived applications are depicted in the figure below.

We describe thoroughly the Sentinel-3 SLSTR pre-processing as the remaining applications implement the same concepts.

As an example of the applications provided, the next paragraphs describe the Sentinel-3 SLSTR reprojection and tiling that takes as input one or more Sentinel-3 SLSTR Level-2 products to:

These tiles are then post-processed in another application that applies the Whitaker smoothing filter to, on the one hand, fill the temporal and spatial gaps and on the other smooth the time-series.

The Sentinel-3 SLSTR reprojection and tiling CWL file is shown in the figure below.

This CWL was extended to declare that the application expects a STAC local catalog with the Sentinel-3 staged data included in a STAC collection named input_reference.

This type of extension in a CWL is foreseen and provides additional degrees of freedom to express concepts such as the expected input format for the application.

During the EO Apps Pilot, we presented STAC as a potential solution to be the input and output manifests between the ADES and the application. STAC provides several advantages to application developers as data is staged and described as a local STAC catalog that describes a catalog, one or more collections, the items in the collections and the assets in the item. The assets describe the content at band level or the metadata and thus avoiding developers to implement find, grep, glob strategies to identify the GeoTIFFs or XML manifests.

The drawback in terms of application interoperability experiment is that the Terradue applications have only been deployed on Ellip as the other platforms did not implement the support for STAC during the Pilot.

The Application Package targets algorithms (developed by a third-party developer) that are to be deployed in a Cloud Platform. During this pilot, we extended the use of CWL to reference the application containers (e.g. Dockers) and also to allow the definitions of the application parameters, input/output interface and the overall process offering parameters.

Providing a single CWL file showed to reduce the complexity and facilitate the communication between the application providers and the platforms. The CWL file provides the information model for the full application definition from input cardinality and constraint to memory, CPU or storage requirements.

From application developers just wrapping their code on a Workflow class, to Expert users that want to fully control the application workflow, CWL showed a great potential for portability and reproducibility. CWL also demonstrated the importance of independent testing and validation when deploying the applications in a heterogeneous environment of platforms.

As CWL is widely used and there are several backends to support the testing of application packages at local or cluster level there is no major obstacle to its adoption. Documentation about the language is freely available, support forums are quite active and the main patterns for EO application design were explored during this pilot activity. The implemented examples provide a basic template for future applications substantially reducing the implementation effort.

In terms of security, the CWL execution engine can detect situations where the containers are executed as a root user and abort the processes. This situation should be avoided and must be part of the application developers guidelines for building docker images to be deployed on platforms.

As hardware requirements fall outside the scope of the OGC WPS specification, the application package needs also to be able to allow developers to describe their application and thus to convey the information about the runtime environment to the ADES. The approach followed during this Pilot demonstrated how CWL allows the definition of the hardware requirements independently of the container-orchestration system (e.g. Kubernetes) and additional software requirements can be specified as the developer raises the complexity of the application.

A major issue discussed, but not fully addressed, on previous OGC Testbeds (13, 14 and 15) relates to the approach by the ADES to provide the data stage-in of product input references specified on the WPS request and stage-out the results of the application.

In previous testbeds, the ADES had limited knowledge about the input and output files data contents missing critical information like spatial footprint, sub-items (e.g. masks, bands) and additional metadata (e.g. ground sample distance, orbit direction).

In this pilot we decided to support the data flow management by using a local catalogue encoded using the SpatioTemporal Asset Catalog (STAC) specification as a data manifest for application inputs and outputs. The local catalogue provides knowledge about the input and output files data contents like spatial footprint, sub-items (e.g. masks, bands) and additional metadata.

The usage of CWL together with STAC to support the data stage-in allows the transparent definition of the parameters that convey resources to be made available to the application.

In previous OGC testbeds, the majority of implementation examples focused on a single node machine and didn’t fully explore the application scalability.

In this pilot, the possibility to directly specify the parallel processing potential of the application at the CWL Workflow class level with the use of CWL requirement ScatterFeatureRequirement allowed the support of the data driven application fan-out patterns.

These applications demonstrated the execution of a data processing function that processes concurrently several products generating independent output for each input. As such, parallelization and scalability of the application can be transparently managed by the ADES cluster (e.g. Kubernetes) taking into consideration the available computing resources not only for multiple requests but also the specific application internal workflows.

To continue the activities explored in this pilot, it is important to support the new Transactions extension of the OGC API - Processes. This extension, currently being addressed by the working group, will define the behavior of an implementation that supports the ability to dynamically add, update or remove processes from a deployed OGC API-Processes instance.

In this ER we demonstrated that the usage of the Common Workflow Language can define the application package and allows the developer to define single or multiple nodes applications without the need to specify new properties. CWL provides a simple way of pointing to an application container (e.g. Docker) together with the definitions of the application parameters, input/output interface and the overall process offering parameters. It is our recommendation that the Transactions extension use CWL to define the application package.

We recommend that a Best Practice should be created to further document the use of Common Workflow Language for application packages. The proposed best practice should extend the application design patterns discussed in this ER and provide guidelines on how to improve the automatic generation of CWL (e.g. from Jupyter Notebooks). The design or creation of specific wizard templates that would facilitate the creation of application packages is also recommended.

We also recommend producing a Best Practice for the data flow management using STAC local catalogues as the input and output manifests between the ADES and the hosted application and between the ADES and the EMS.