OGC Testbed-14: ADES & EMS Results and Best Practices Engineering Report

As introduced above, the ADES and EMS are two architectural blocks with different functions within TEPs/MEPs. They are required to support application deployment, execution and chaining, as well as the agreed authentication and authorization mechanisms. All interfaces, including the security ones, have to be standard so as to achieve the Sponsor’s interoperability goal in the heterogeneous Exploitation Platform context.

Given that several options are possible for what concerns the functionality allocation and interface details, it is important to record those that were subject to experiment in Testbed-14, as well as the results of the experiments. Finally, it is important to recommend best practices in doing so.

In Testbed-13, a simpler architecture was considered, with just an Application Management Client (AMC) and an Application Deployment and Execution Service (ADES) involved, i.e. no Execution Management Service (EMS). The interface between the AMC and the ADES was based on WPS, in its traditional XML encoding, with two alternative implementations having been pursued, one relying on standard WPS 2.0 and two dedicated - but regular - WPS processes for deploying and undeploying processes/applications in the ADES; and the other one relying on the so-called transactional extension of WPS, which has not been standardized but is described in a OGC discussion paper [1]. For more details, please refer to [2].

In Testbed-14, WPS is again the key OGC standard involved and there was consensus in using the transactional extension, both for the client<→EMS and for the EMS<→ADES interface. Furthermore, in line with one of the Sponsor’s requirements, the Extensible Markup Language (XML) encoding was dropped in favor of one based on Representational State Transfer (REST)/JavaScript Object Notation (JSON) which is also not yet an OGC standard but a version of which was submitted for public review by the WPS Standards Working Group (SWG) during the timeline of the Testbed. In fact, the WPS 2.0 SWG and the Testbed-14 EOC Participants coordinated to align, to the maximum extent possible, the work and lessons learned in Testbed-14 with the evolving WPS 2.0 REST/JSON specification.

The OGC WPS 2.0 standard was already a key part of the work done in Testbed-13 and continued to be so in Testbed-14. Indeed, given that one of the core aspects of Earth Observation Exploitation Platforms is the data processing, WPS is a natural fit. At the same time, not only in Testbed-13 but in general, variants and possibilities related to WPS, such as the Transactional extension (WPS-T) and the reliance on REST bindings and JSON encoding, are of increasing interest. The growing popularity of WPS-T and REST/JSON is because they bring the OGC, and WPS in particular, closer to the mainstream web and technological realms.

Given that a significant amount of effort in the EOC thread of Testbed-14 has been dedicated to experimenting with these WPS variants and possibilities, this work is expected to be of high interest to the WPS 2.0 SWG. The relevance of the Common Workflow Language (CWL) to the OGC should also be investigated. It could be particularly interesting for the Workflow DWG and could be considered as a potential future Community Standard.

Future work should be dedicated to consolidating the WPS REST API that has been proposed by the WPS SWG and contributed to by Testbed-14 EOC thread Participants, as well as initiating a standardization path for the transactional extension of WPS (WPS-T). A specific issue of interest (but also risk) resides in how to transform a message-based protocol such as WPS (with its traditional GetCapabilities, DescribeProcess, Execute operations) into a resource-based protocol (WPS REST), where resources such as processes and jobs are managed using the basic HTTP operations (GET, POST, PUT, etc.). A straightforward conversion (i.e. simply mapping the XML messages into REST/JSON) may result in APIs which are not intuitive and are semantically unclear. It is therefore important to dedicate time and effort to properly designing a resource-based version of a previously existing XML-based protocol. Another relevant and general issue in this discussion is how to transform/encode XML into JSON content. In coordination with the WPS 2.0 SWG, several smaller or one or two larger Change Requests can be prepared so as to start the standardization path for WPS-T and the WPS REST/JSON work.

Further areas of interest for future work are:

The work described in the ER makes the case for how OGC standards are helpful in such a densely populated ecosystem of very heterogeneous entities, technologies and implementations such as the Exploitation Platforms one. For OGC therefore, the work described in this ER is deemed extremely relevant in satisfying the needs of the European Space Agency as one of its Strategic Partners.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Besides the Tested-14 Participants, the following institutions have supported the activities of the thread:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

The Application Deployment and Execution Service in Testbed-13 was a single server-side component and it was in charge of all aspects pertaining to the deployment and execution of EO applications on different cloud providers. It therefore shielded instances of the Application Management Client from all infrastructure details. In the canonical case, it was exposed as a WPS interface, with two pre-existing processes dedicated to the deployment and undeployment of applications. In the alternative implementation, it exposed a WPS-T interface in-line with [1], which adds two new operations to the basic WPS ones, namely DeployProcess and UndeployProcess.

The ability to register and unregister processes, just as any other WPS server, makes the ADES also a defacto repository and catalogue of processes that can be discovered using a GetCapabilities request and described in detail by issuing a DescribeProcess request for the process of interest. Thus, in Testbed-13, the ADES also performed the role of process/application catalogue, available for query by clients.

More details can be found in [2].

The Application Management Client in Testbed-13 consisted of a WPS client implementing the WPS XML binding with support for the Simple Object Access Protocol (SOAP), and relying on ADES instances for all functionality. Briefly, the client allowed:

More details can be found in [2].

As described above, the OGC WPS protocol was the basis for the interface between the several AMC and ADES instances in Testbed-13. It was used in its SOAP/XML bindings and encoding and, except for the alternative approach pursued by one of the Participants, which relied on the WPS-T extension described in [1], was the standard WPS 2.0 specification.

More details can be found in [2] and [4].

The figure below depicts the architecture adopted by Testbed-14 Participants, with the main building blocks and interfaces, as well as some implementation notes that highlight important assumptions agreed between all stakeholders, including the Sponsor.

Three aspects in particular are worth noting as they represent changes with respects to the Sponsor requirements or issues left open in the CFP that have been herewith addressed:

Particularly the first and second bullets above represent simplifications of the real scenario that would need to be addressed in a more operational context. For what concerns the first bullet, if instead it were the TEP Client or the ADES in charge of performing the search (or if this was a possibility), further work and logic would be required between the elements involved in the flow, to communicate the list of products to be processed and whether such a search had been performed already or not.

This section describes in full detail the steps of the sequence of operations required for deploying and executing both applications and workflows:

Steps 5, 6 and 7 pertain to steps internal to the EMS, required to execute a workflow or application.

Once again, the security, quoting and billing aspects are not covered here as they belong to the scope of [3].

The sequence of operations is shown in the diagram below.

The execution of a simple application (following step 1) is not covered explicitly because the steps (5, 6 and 7 in particular) are similar to the execution of a workflow, except that the EMS in that case is only a broker in front of the ADES.

For a more focused discussion of the Application Package format and its facets, the reader is referred to [6]. That document contains a brief introduction of CWL and an explanation of its role in the Testbed work.

The Spacebel EMS implementation is based on the 52° North WPS 2.0 Java implementation and on the Spacebel WPS transactional extension which provides an XML interface. The interface made use of the OpenAPI 3 Specification (OAS3). The XML interface is brokered by a proxy based on a Java JAX-RS server stub generated from the WPS-T REST/JSON OAS3 specification with Swagger CodeGen.

The proxy does not support the GetJobs operation. That operation was proposed in the WPS SWG REST/JSON draft, but is not compliant with the WPS 2.0 official specification, so it cannot be supported when using a JSON proxy on the frontend.

The execution of the workflow is performed using the CWL-runner tool which is provided as part of the standard CWL toolset.

The CWL workflow steps perform runs of Docker containers while the EMS invokes the appropriate ADES to execute the processes. Instead of tailoring the CWL-runner or developing a custom CWL engine, the Spacebel approach consists of rewriting the workflow steps to match the desired behavior. This approach allows the use of any alternative CWL engine (e.g. Airflow) for running the workflow.

For each workflow step, the appropriate target ADES is computed depending on its inputs, and a command line step is created for starting a WPS commmand-line client which performs the execute request.

The Spacebel EMS has been deployed on the Google Cloud Platform.

Due to lack of time, the quotation and billing operations have not been implemented.

The EMS server is a proxy publishing a WPS 2.0 JSON API. It is based on a Spring boot controller using pojo generated with swagger codegen from the openAPI document.

The aims of the EMS are to:

Test EMS and ADES servers are based on components of the Open Source software framework Birdhouse [8]. The Twitcher component acts as a Policy Enforcement Point (PEP) and a WPS 2.0 JSON proxy to WPS 1.0 endpoints. It has been enhanced to comply to the EMS API which involves adding the dynamic deployment of processes and CWL workflow capabilities. The Magpie component is used as an adapter to manage ACLs of deployed processes and process permissions (WPS requests) for a given user’s credentials.

Although fulfilling different roles, EMS and ADES components use the same underlying CWL engines and require security access verification. Both of them use common operations that are available, such as:

CRIM’s implementation of the EMS differs from the ADES only in the runtime context used by the CWL tool engine. While the ADES uses the default runtime environment which pulls the Docker image and executes it using the CWL parameters, the EMS uses a custom runtime implementation. That implementation instead chooses the right ADES to dispatch the processing based on the input’s data source, makes sure that the process is deployed on the corresponding ADES, then executes and monitors it until it can finally retrieve the outputs on successful completion. This simple approach allows the dispatching of a single process only by invoking the CWL engine as well as executing a full workflow in which case every step is executed by dispatching the processing to a remote ADES and chaining the outputs to the next step.

The principal requests of the EMS involve workflow operations which are DeployProcessWorkflow and ExecuteWorkflow. The workflow itself is presented as CWL Workflow with Workflow parameters. Additional details on the AP described in this section can be found in [6].

The ADES implementation is based on the 52° North WPS 2.0 Java implementation and on the Spacebel WPS transactional extension which provides a XML interface. The XML interface is brokered by a proxy based on Java JAX-RS server stub generated from the WPS-T REST/JSON OAS3 specification with Swagger CodeGen.

The proxy does not support the GetJobs operation. That operation was proposed in the WPS SWG REST/JSON draft, but is not compliant with the WPS 2.0 official specification, so it cannot be supported when using a JSON proxy on frontend.

The execution of the Docker container is performed using CWL-runner which interprets the CWL file provided during the application deployment. The input files are downloaded (and renamed to avoid filename collisions) on the ADES file system.

Spacebel ADES has been deployed on IPT Poland cloud and on PFC boreal cloud.

The Geomatys ADES Server is based on the Examind-server WPS 2.0. It is a Java-based server that uses the Geotoolkit library for data-binding and processing. It handles WPS 1.0/2.0 in XML and JSON. Geomatys added the Transactional and Quotation part during this Testbed.

The ADES features are to:

The OpenSearch specification provide a pagination mechanism for listing search results. For the purpose of the Testbed, some EMS implementations do not support the pagination and retrieve results from the first result page which is limited to 50 entries on the Spacebel Gateway. In operational contexts, the pagination would have to be addressed.

Even though [3] describes these aspects in more detail, the following is a brief rationale for how the Quotation part of the WPS-T REST API was designed:

For further information about the parts of the WPS-T REST specification introduced in Chapter 7 that deal with Quotation and Billing - as well as all related issues of interest - the reader is referred to [3].

As mentioned in Chapter 6, the Sponsor in the CFP included a requirement to “make use of standardized error handling and user messaging”. Even though the Testbed Participants did not dedicate significant or special efforts to this requirement in particular, standardized error handling and user messaging has been elegantly included in the solution by relying on standard HTTP error codes (4xx) in the communication between functional blocks. This can be seen extensively in the OpenAPI 3.0 file with the WPS-T REST specification linked to in Chapter 7. A large part of the file’s content is indeed dedicated to the specification of error responses, codes and messages.

It has been decided that applications should be deployed on the ADES at runtime in order to determine the best suitable ADES based on the data. While this is good behavior it causes an issue when the EMS is required to deploy an application on Bob’s behalf (which should be forbidden because he is not a developer) when this user wants to execute an application for which permission has been granted to him on the EMS. It has been discussed to deploy every application on every ADES when it is first deployed on the EMS, but that defeats the purpose of on-demand deployment. It has also been discussed to add a trusted channel between EMS and ADES such that a deployment can be always authorized if initiated by a known EMS.

Spacebel has submitted a set of corrections to the WPS-T draft implemented in the context of the Testbed-14 NextGen thread. It has been agreed to base the WPS-T JSON API on this specification.

There are still some minor elements that have been noticed as ambiguous, which will be reported to the WPS 2.0 SWG.

Generally speaking, the WPS REST/JSON API draft submitted by the SWG sometimes explores contradictory guidance: from a side, it distances from the XML schemas, losing a strict analogy with the specification. On the other side, the liberties taken for applying the REST principles fail to meet the target.

The following comments should be considered to improve the future versions of the API:

During the project, it has been noticed that some applications require that the input files meet some specific criteria:

The output of an application or script may be a set of files. The WPS specification states the following: “Processes, as well as their inputs and outputs, are elements with identity. A process input may have arbitrarily defined value cardinality, i.e. for a given input, multiple datasets may be submitted for execution. A process output always has a value cardinality of one”. Therefore, the question is how to return an array of files in a WPS response.

The case typically happens when an application receives multiple files as input and applies the same treatment to each of those files. Various approaches have been considered to handle this case:

The Testbed-14 experience has allowed identification of problems affecting application execution, some of them specifically related to CWL. It has also allowed development of workarounds for them. Herewith follows a list:

One of the participants suggested that the processExecuteUrl attribute used in the Process Description should be changed to a links section where hypermedia controls with appropriate rel links could be used. At least one link for executing the process must be in that section, but other links could be allowed too.

Due to lack of time this has not been tried in Testbed-14.

As a clarification, even though participants agreed to use HTTP URLs as inputs for the ADES executions, this is not to be understood as implying a download from a remote server, as this would completely defeat the purpose of running the processing close to the data. Instead, the idea is that data to be processed in a given platform can be hosted on that same platform for example in Object Storage buckets (accessed using HTTP) or in a WCS server (accessed using the WCS HTTP API). Such access is still to be considered "local".