OGC Earth Observation Applications Pilot: Summary Engineering Report

The Earth Observations Applications Pilot explored and evaluated the maturity of the Earth Observation Applications architecture. The architecture has been developed in various OGC Innovation Program initiatives over the last three years. This pilot now explored everything in operational environments.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

OGC activities in Testbed-13, Testbed-14, and Testbed-15 initiated the development of an architecture to allow the ad-hoc deployment and execution of applications close to the physical location of the source data with the goal to minimize data transfer between data repositories and application processes (see References). The various Testbeds produced several draft specifications, which addressed both application description and discovery, APIs for deployment, execution, and result access, billing and quotation models, as well as specifications for service chaining and workflow building.

In summary, the architecture fulfils the following requirements:

The following figure provides a high-level view on the resulting setup. Application developers on the left side develop their application and make it available in the form of a Docker container image. The developers register their application in the platform and thus make it available to all users of that platform. At the same time, application developers can discover other applications and integrate these into workflows, where the output of one application serve as input into the next application.

Platform providers offer APIs that allow both, application developers as well as application consumers to interact with the platform. All interaction, including application registration, deployment, and execution requests, are supported by a Web Processing Service (WPS) interface or an interface conforming to the OGC API - Processes candidate standard. To differentiate the various interactions, two logical services have been defined: The Execution Management Service (EMS), and the Application Development and Execution Service (ADES).

Together, both APIs provide all functionality necessary to handle the different types of interactions between application developers and the platform (mostly registration and update of applications) and application consumers and the platform (mostly application discovery and execution).

The Terradue report provides a good introduction into ADES and EMS, as do the reports by CRIM and Spacebel.

All results have been captured in detailed Engineering Reports. Each participant delivered a report that documents all detail about the various applications and platforms, describes interoperability challenges, and recommended solutions.

In addition to the Engineering Reports, all participants produced promotional videos that are available on the OGC Youtube channel:

The pilot was executed during the transition phase from the Service Oriented Architecture period towards the modern Web APIs. The pilot has demonstrated that the transition is in fact very smooth. Though it is essential to use a limited set of open standards to achieve interoperability, the choice of JSON encodings vs. XML payloads and consistent usage of REST principles vs. SOA-remote procedure calls has effects on implementation costs, but less on achievable interoperability.

On a more detailed level, the pilot has proven that the path towards a harmonized set of Web API building blocks rather than a service-oriented concept with disjunct functionality bundled in a concrete service is very promising. It needs to be implemented carefully with keeping both interface design as well as resources representation and serialization in mind. It needs to be avoided that a variety of resource models will have negative effects on the level of interoperability and thus perceived performance from both application developers' and application consumers' perspectives.

It has been demonstrated that open standards are required for all steps of the process. This is in particular true for the platform-internal processes of data provisioning and result handling and their respective discovery. For application developers, these steps have the highest risk for interoperability issues and thus unsuccessful application execution. Platforms shall provide open standards based discovery and data access mechanisms independently of their internal data storage and management system. Only then application developers can deploy their apps on multiple platforms without adapting these crucial parts of their applications from one platform to the next.

General agreement was achieved on the definitions of data processing design patterns.

The data driven application fan-in patterns refers to the execution of a data processing function that aggregates several input products. The platform application accesses a list of input products, retrieves and proceeds with the stage-in of the products making them available to the application execution block.

The data driven application fan-out patterns refers to the execution of a data processing function that processes concurrently several products generating independent output for each input. The platform application loops from a list of input products, retrieves and proceeds with the stage-in of the individual products making them available to the application execution block. The platform can apply different strategies to parallelize the execution of each individual product.

Parameter-driven data flows permit cyclic, systematic retrieval of selected groups of input products between selected the parameter intervals (e.g. start and end dates). In this scenario, the parameter interval acts as a step function, determining how the next batch of products is to be selected.

The Application Package (AP) contains all information necessary for application platforms to deploy and execute the application correctly. These include information about the application process itself, input and output requirements, dependencies, or hardware requirements. There was common agreement on the items that the AP needs to include, but some discussion about its serialization format. Overall, there was a clear tendency towards the adoption of the Common Workflow Language (CWL) for that purpose. Only Spacebel emphasized the need for complementary approaches. Further work shall identify the essential CWL elements and clarify the use of the CWL specifications Command Line Tool and Workflow Description. See Terradue’s report for a mapping of CWL elements to the WPS data model and CWL element usage in general. EOX on the other hand reported a necessary use of heuristics for properly translating CWL types to the corresponding WPS types and vice versa. There was common agreement to add hardware parameters to the Application Package.

Further on, supporting a well-established declarative format like CWL to define application workflows gives platform providers the possibility to leverage a wide ecosystem of existing tools. These can be used by the platforms themselves as well as offered to application developers (e.g. to compose workflows visually) or to application consumers (e.g. to monitor workflow execution).

CRIM successfully deployed the same core ADES-EMS implementation in three different sites, PBC, EODMS and CRIM and reported only minor adaptation requirements (mostly to deal with different security environments and settings). The use of an application profile to register external WPS proves to be a rather simple, but useful tool to build a federated cloud.

The separation of roles between EMS and ADES is seen as beneficial, the former being closer to thematic platforms with cross-cutting concerns, the latter located near mission data, or more specifically to the ground segment.

Testbed-14 introduced the OpenSearch API for selecting the EO data inputs submitted to the process. Experiences have shown that, despite some inconsistencies among OpenSearch implementations, OpenSearch provides the most promising approach to handle data discovery aspects across platforms. This is particularly important as different platforms use different identifiers for the same product and make the data available in different formats (zipped, unzipped, folder, tar-ball, SAFE etc.).

The data flow management plays a prominent role in this architecture. Most applications do not bring their own data, but require data being made available by the platform. In addition, applications produce a variety of output data as result. Thus, it is essential to describe both the inputs and outputs of an application in full detail. In the pilot, it turned out that data flow management by using local catalogues encoded in compliance with the SpatioTemporal Asset Catalog (STAC) specification as a data manifest for application inputs and outputs is a very promising approach. STAC, a profile of OGC API-Features and thus based on open standards, blends in very well with the set OGC API building blocks and design patterns.

For a detailed discussion of STAC and its integration with CWL see Terradue’s report. Spacebel argued that STAC would introduce additional, unnecessary conventions that that could be avoided by using OGC data access services such as OGC Coverage (WCS) and Feature (WFS) Services or their OGC API counterparts.

There is common agreement that Docker is the state-of-the-art container format to be used in the EO Applications Architecture. Its maturity, widespread use, and the numerous resources and examples make Docker the recommended solution. Experiences differ in terms of optimal Docker image handling, which led Spacebel to run experiments with server-side production of Docker images; mostly triggered by security concerns (see Spacebel report).

CRIM noted that producing accurate quotations for workflows will be a very difficult challenge, while quoting for applications in simple federations is probably feasible.

Experiments with CWL workflows have been successfully conducted.

As has been shown that the availability of GUIs, ideally paired with directly accessible log files, is an essential ease-of-use cornerstone for application developers.

Although Earth Observation platforms are typically installed in a Cloud infrastructure, the implementations in previous OGC Testbeds had limited the execution on a single node machine. This considerably restricts the system scalability and computing possibilities. Kubernetes has become the de facto standard for deploying containerized applications in private and public cloud environments. This pilot experimented with Kubernetes clusters and explored the options to execute atomic tasks in parallel. Experiences have been very promising so far. The automated scalability of the cluster nodes allows to extend the infrastructure by provisioning additional machines when multiple processes are requested or when single requests can be split into parallel tasks. This approach allows to execute a very high number of concurrent jobs and exploits advantages of cloud platforms.

Align the architecture with the emerging OGC API-Processes, which will complement the WPS standard in the near future.

Conformance classes for ADES API shall be standardized.

Define subsets of CWL that shall be supported by all platforms to ensure consistent experiences for application developers. Produce an OGC Best Practice document to document the use of Common Workflow Language for application packages. The proposed Best Practice document should cater for the various application design patterns discussed in this pilot and provide guidelines for automated generation of CWL (e.g. from Jupyter Notebooks).

In addition to CWL, other approaches shall be reviewed again, such as a simple command line syntax to describe the base command and corresponding arguments for Docker container execution.

The usage of different CWL versions may lead to issues in the future and should be addressed.

More tests are required to better support application workflows running in multiple platforms. In order to build a federated cloud, these tests have to run regularly, in a structured and systemic fashion.

Currently, platforms provide individual settings for reporting progress, management of logs, and handling of errors. These should be further harmonized.

Data discovery and access is among the biggest challenges for application developers. All platforms use slightly different approaches. A consistent usage of data access services that are offered by all platforms might facilitate this problem.

OpenSearch has been reported as both being very powerful for data discovery, but still not fully consistent across the various platforms. Here, further tests and derived standards shall clarify both OpenSearch usage and provisioning of results.

Extend the set of provided metadata to allow platforms to describe their hard- and software capabilities. Explore how metadata can be generated more automatically rather than being based on application developer inputs.

From the application perspective, it shall be investigated how the application package can register the supported input/output formats and file access mechanisms (e.g. direct access S3, http(s), GDAL virtual file systems, etc.), so that platforms can choose the optimal match between platform and application and potentially skip staging in data or outright reject the package if no compatible format is available.

The use of services for data access, such as the OGC Web Coverage Service, have not been investigated as most platforms provide their data for processing as files staged into the application environment. Thus, it is still open whether application packages should make use of web services for data access or rely on the data provided by each platform.

Further explore strong semantics and the usage of standard compliant taxonomies for attribute names and values. Explore parameter correlations and constraints can be handled.

Produce an OGC Best Practice document that discusses data flow management using STAC local catalogues as the input and output manifests between the ADES and the hosted application and between ADES and EMS.

Explore modified Docker build pipelines to solve concerns with respect to Docker image access control, security enforcement, and version control. Docker images could be built on the ADES side rather than by the application developer.

In most cases, Docker images have been stored in public repositories. The usage of private repositories shall be standardized.

Tutorials should be developed as lots of development follows examples, in support of standards and specifications.

Further work is required to explore the parallelization and other application performance drivers. These include RAM optimization and GPU processing.

For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard OGC 06-121r9 shall apply. In addition, the following terms and definitions apply.