OGC Earth Observation Applications Pilot: European Union Satellite Centre Engineering Report

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

The application developed by SatCen and integrated into the different platforms aims to identify changes between two Sentinel-1 (SAR) images acquired at different times but acquired with very similar acquisition conditions (sometimes also called interferometric conditions), by computing the "Amplitude" or calibrated backscatter and the "Coherence" between the two images. The coherence refers to the amplitude of the complex correlation coefficient between two SAR images.

The analysis of the coherence is very useful for change detection. Weather and illumination independence make SAR acquisitions particularly interesting for change detection and monitoring scenarios, especially in certain areas of the world or periods of time where weather conditions prevent the use of optical imagery. Additionally, the coherence maps have been found to be of great utility in detecting changes that occur on the ground and that are not detectable by using only the amplitude or backscatter signal.

The inputs of the application are two Sentinel-1 SLC images that should be acquired in the same conditions (same relative orbit). Also, it is possible to define the area of interest in order to compute the coherence only over the desired area. Some other ancillary data is needed, but it is not directly selected by the user: the application automatically will look for some orbit and DEM (Digital Elevation Model) files to download them.

The application is written in Python and it uses SNAP and GDAL internally. The main process for the coherence calculation is carried out in SNAP, but some functions of GDAL are also used.

The main steps in the processing are:

The output is a GeoTIFF image with three bands:

The output can be the input for other algorithms (e.g. thresholding and algorithms based on Artificial Intelligence, …​) for automatic change detection but is also useful for a visual analysis. A usual Red-Green-Blue (RGB) color composite (R:Image1 backscatter, G:Image2 backscatter, B:Coherence) facilitates the interpretation of the results, since it allows the interpreter to assign the different colors of the pixels to different conceptual categories, for example:

SatCen runs locally the application using Docker but wanted to explore the advantages of moving the application to the data, and the feasibility of creating a single package deployable to many different external platforms.

The main advantages of using a platform are:

From the data processing perspective, the inputs required for the EO Application are two Sentinel-1 SLC images acquired with the same geometry. Sometimes, from the user perspective, it is more convenient to use as inputs the date and the area of interest. With this information it is possible to discover the Sentinel-1 SLC scenes in the catalogue, to identify automatically which are the pairs of images that satisfy the user needs and to download them or access them from the platform data archive. In fact, this was the approach followed by the application when working locally: The input from the user are the AOI and the date and the application itself performed the search for the most suitable S1 images; and download them.

One of the first discussions during the project was about the possible approaches for the final EO Application. Different alternatives were considered:

The last option was chosen but it is important to highlight this item as one of the most important topics to be further analyzed in the future.

On the other hand, during the integration phase, it was needed to adapt the source code of the application for each specific platform: some preprocessing steps were needed because the inputs were provided in different ways. With the evolution carried on within the project in terms of standardization on the platforms, these adaptations are no longer needed, since the files are now mounted directly in the working directory of the docker automatically. This is indeed a very relevant evolution obtained during the project, since the different platforms and application providers converging into a solution that satisfies everyone is essential in the way to define a standard in the future.

However, it is important to note that the convergence imposes several constraints that might have an impact on the applications, for example in terms of protocol (direct file access, which was the initial expectation of SatCen’s application), and file format (since SatCen’s application is able to read from a variety of file formats, this had no impact in the developed application). Generally speaking, inputs from the same collection can have different formats. As an example, Sentinel-1 data is usually offered in zip format or SAFE format. Currently, it is not clear if the application should support any possible format or if it should be possible to define the expected format. Then, it would be up to the Platform to provide the correct format, a process that could introduce transformation issues. Alternatively, the Platform could inform the EO Application about its data offerings, protocols, and file formats.

Finally, another point to mention is the evolution of the capabilities of certain Platforms, that were able to rapidly move from versions imposing some constraints to the supported inputs to versions providing a more flexible approach. As an example, the Spacebel platform, during the first test imposed the limitation to only pass the application inputs and parameters as environment variables, while at the end it was more flexible and easier for the application providers, supporting the standard inputs from the CWL definition.

At the early phases of the project, some constraints were also imposed on the generated outputs. As an example, in one platform, the output name was fixed with an environment variable parameter but with the evolution of the platform the situation improved.

There are open questions, however, about the cataloguing of the outputs. The application needs to generate some metadata in order to provide information about the output to the platform (metadata is discussed later).

This item has a major impact, since the expectation of developers and final users of the applications is to be able to easily discover and access the results of the executions of the applications triggered by them. Also, following the concept of chaining applications, other applications might re-use the outputs. The different platforms provide different tastes on how output results are provided (e.g. via APIs, via Web-UIs).

As a more advanced experiment, a second application whose inputs are the outputs of the coherence application was developed and deployed on Spacebel’s platform. This test was carried out to verify that the data cataloguing is properly done, thus, demonstrating the feasibility of chaining applications. A full end-to-end integration experiment with a set of chained applications was not carried out, though, due to orchestration issues (which application needs to run first?), and lack of an endpoint fully compatible with the CWL specification.

Docker was introduced in Testbed-13 into this EO series of OGC Innovation Program initiatives. By using Docker, the application can be packaged with all its dependencies at the specific versions expected by its developer.

The coherence application was already available in a docker container that was used to run it locally, so this step was not a problem, except for some errors in the work directory in the first version of the docker container generated. It is important to highlight that docker uses the concept of "volumes" to persist data. This aspect requires special considerations when working with large files (as EO imagery). Volumes, though, are provisioned and managed by the Platform and therefore, there is a need to match the docker expectations in terms of size and availability of disk space, and the capabilities of the platform.

Regarding the access to the docker, the coherence application can be uploaded to a public docker repository. If developers need to control access, then a private repository is needed which could be specific for each platform. Further accounting and access control mechanisms are still to be addressed: e.g. how the developer of the application assigns permissions to users to have access to the application.

With respect to security, it is considered a bad practice to execute containers as root user. When moving to a platform, this can cause additional challenges when the local version of the Docker was executed as root.

An interesting topic is how applications interface with Platforms in order to for example report the status of the execution or communicate errors to the Platform. This topic has not been fully investigated during the Pilot and needs further investigation.

The integration experiments consisted of the manual execution of the application using two Sentinel-1 products as input.

It has been demonstrated during the project that it is possible to create a single docker image containing a simple application in order to be integrated into different platforms with no change in the docker whatsoever.

SatCen’s application has been integrated in Spacebel, Terradue and CRIM platforms. At the beginning, in order to start interacting with the platforms, different Docker-based containerization approaches were used, but during the project the different approaches started to converge and finally the main objective of using the same containerization approach in all the platforms was achieved.

The collaboration with partners, multilateral and bilateral meetings and the will to reach agreement allowed the participants to make significant progress during the pilot.

The Pilot is perceived as very useful for advancing and validating certain aspects in real-life scenarios.

The use of docker for packaging the applications with all its dependencies has been demonstrated to be very useful. The status of this technology, its widespread use and the numerous resources and examples makes Docker the recommended solution.

The Docker containing the application is not enough to integrate the application into the platform. What is also needed is a description of the application that tells the platform how to execute it, what are the inputs, the outputs, and additional parameters (e.g. hardware requirements or other specifics of the application).

With respect to provisioning application descriptions to the platform, the use of CWL has been recommended and implemented by all the partners. Though, some differences in the level of CWL support caused some integration issues. CWL is very powerful and although not all features are currently implemented in the platforms, it is perceived as a suitable, powerful, and extensible solution. It has substituted the WPS Process Description, which is a lot more verbose. Though the WPS Process Description has some other advantages, CWL is considered a simpler solution for application developers. Further investigations shall evaluate the differences in more detail regarding definition of hardware requirements and input constraints in particular.

Still, it is necessary to review CWL in contrast to other workflow orchestration engines that exist either platform independent or for specific platforms. Examples include Airflow, Argo, or Pachyderm. The analysis should include compatibility issues and limitations of the various orchestration engines in particular with respect to support cloud environments such as e.g. Kubernetes.

The platform (ADES) shall manage both discovery and retrieval of input data as discussed previously in this report. A deeper analysis is needed in order to address current interoperability challenges:

SatCen opted for an interactive selection of the inputs when running on the various Pilot platforms. Ideally, this approach should be replaced by a two-step process (first discover the data and finally run the algorithm).

Taking a look at the expected advantages of deploying applications on a remote platform compared to in-house deployment and execution, the following aspects shall be named:

A critical review of the achievement agreements has to be done in order to be sure that the proposed solutions, besides meeting the needs of application developers and platforms providers, are the best taking into account current technologies.

There are some points that have not been discussed in detail during the pilot but are interesting for the application developers. More tests and discussions are needed for: