OGC Testbed-14: Application Package Engineering Report

The Application Package in the Exploitation Platforms context is required to:

The work done in Testbed-13 addressed these basic requirements and provided a good starting point for the work in Testbed-14. Indeed, lessons learned and limitations were identified that the work herewith described attempts to address.

The work in the EOC thread of both Testbed-13 and Testbed-14 establishes the existence of a Dockerized application as a pre-requisite for using the specified Application Package formats. This assumption is therefore unchanged after this Testbed. Particularly due to the new requirement in Testbed-14 of supporting application chaining, a workflow description language, CWL has been introduced. This was not needed in Testbed-13 and this part of the work might be of interest to the Workflow DWG more than for the WPS Standards Working Group (SWG).

Catalogue endpoint information that in Testbed-13 was conveyed within an OWS Context document encoded in Extensible Markup Language (XML) is conveyed in Testbed-14 within a WPS-T DeployProcess document encoded in JavaScript Object Notation (JSON). The OpenSearch for EO standard is used for catalogue searches but it was agreed during the Testbed that the TEP (Thematic Exploitation Platform) Client would not perform such searches on behalf of the user and it would be actually the EMS (Execution Management Service) deliverable doing this. Such considerations are therefore not considered of particular interest to the OpenSearch for EO SWG in the context of this ER. More information on this part of the work can be found on the "EMS & ADES Best Practices ER" of the Testbed-14 EOC thread ([1]).

Given that the adopted solution is totally based on WPS and its transactional extension (WPS-T), the work described in this ER, and more broadly the work done in the EOC thread of Testbed-14, is particularly relevant to the WPS SWG. More so because the actual Application Package consists of a WPS-T DeployProcess document. Whilst in Testbed-13 the XML encoding of WPS requests/responses was the basis for the Client-ADES interface, in Testbed-14 a JSON encoding of WPS is used and RESTful bindings of WPS-T constitute the basis of both the Client-EMS and EMS-ADES interfaces. The work done in the EOC thread of Testbed-14 has followed closely the work of the WPS SWG and indeed fed back to it, particularly for what concerns the WPS Representational State Transfer (REST) Application Programming Interface (API) - not the transactional extension.

Given that the Application Package encoding is based on the WPS standard, the work described in this ER is considered relevant to the corresponding SWG activities. In particular, some Change Requests may be identified in Testbed-14 or, at least, the preparation of Best Practices on how to use a WPS-T DeployProcess message for an Exploitation Platform Application Package could be envisaged and suggested.

For OGC, the work described in this Engineering Report is deemed extremely relevant in satisfying the needs of the European Space Agency as one of its Strategic Partners. The ER makes the case for how OGC standards are helpful in such a densely populated ecosystem of very heterogeneous entities, technologies and implementations such as the Exploitation Platforms one.

Future work should be dedicated to consolidating the WPS REST API that has been proposed by the WPS SWG and contributed to by Testbed-14 EOC thread Participants, as well as initiating a standardization path for the transactional extension of WPS (WPS-T). Even though those two topics are more relevant in the context of the separate Testbed-14 ER on the EMS and ADES Best Practices (D009, [1]), given that in the adopted solution the Application Package is itself an element of the WPS protocols, such future work is also relevant for this ER. A specific issue of interest (but also risk) resides in how to transform a message-based protocol such as WPS (with its traditional GetCapabilities, DescribeProcess, Execute operations) into a resource-based protocol (WPS REST), where resources such as processes and jobs are managed using the basic HTTP operations (GET, POST, PUT, etc.). A straightforward conversion (i.e. simply mapping the XML messages into REST/JSON) may result in APIs which are not intuitive and are semantically unclear. It is therefore important to dedicate time and effort to properly designing a resource-based version of a previously existing XML-based protocol. Another relevant and general issue in this discussion is how to transform/encode XML into JSON content.

The relevance of CWL to the OGC should also be investigated. It could be particularly interesting for the Workflow DWG and could be considered as a potential future Community Standard.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

There were a few reasons that justified the choice of OWS Context as a way to convey the Application Package.

One was that OWS Context is already an OGC standard and therefore a new standardization path would not be required before its wide adoption.

Another one was that, particularly in its ATOM encoding, it is a very browser-friendly format, enabling the best of two worlds, i.e. an OGC standard and a general-purpose, well-tolerated format which is accepted by the mainstream community.

Finally, the OWS Context standard was created to convey a "Common Operating Picture" to systems involved in a potentially complex, distributed flow. This makes it particularly adequate for the use case at hand, in which different systems from different vendors need to cooperate to implement such a flow.

As explained at https://www.commonwl.org/index.html (accessed on 30/08/2018), “The Common Workflow Language (CWL) is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry. CWL is developed by a multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows. The CWL project is maintained on Github and follows the Open-Stand.org principles for collaborative open standards development. CWL builds on technologies such as JSON-LD for data modelling and Docker for portable runtime environments.”

There are a few aspects in the description above worth noting. First of all, CWL originates from and is specified by the scientific community, which is also the main user community of interest in the EOC thread. Even though it does not target Earth Observation, Remote Sensing or generically Space scientists, it targets several other scientific communities which are very diverse one from the other. Secondly, it provides mechanisms for the formal description of data analysis workflows in a way that is portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high-performance computing (HPC) environments, also aiming to enable scientists to share data analysis workflows - these are all highly desirable characteristics in the Exploitation Platforms context. Finally, CWL builds on JSON-LD and Docker, which are two technologies that are:

The following snippet shows a very simple example of CWL, wrapping a command-line tool (the echo command in this case) that takes one file as input, reads a message from that file and writes that message to a console.

The second snippet below shows a slightly more complex example of CWL, highlighting how it can be used to describe a simple workflow consisting of two steps, one running after the other, with the first step’s output(s) being the second step’s input(s). Each step of the workflow is itself described in CWL, in its own file (tar-param.cwl and arguments.cwl). In this specific case, the workflow consists of a first step in which a Java source code file is extracted from a TAR file (untarred) and a second step in which the Java code is compiled.

The following aspects are particularly relevant:

Participants did not dedicate substantial effort to investigating and collecting potential disadvantages and limitations of CWL and its associated tool-chain, but the following can be highlighted:

The figure below shows how the Application Package and its two facets fit in the overall architecture, with its three main building blocks: the TEP Client, the EMS and the ADES.

The finally adopted solution for Facet 2 thus consists of a WPS-T DeployProcess document, through which the application’s inputs and outputs are described. The execution unit can be conveyed as a Docker container for simple applications or a CWL file for workflows. Any other additional required information is passed using ows:metadata fields.

With respect to Testbed-13, it continues to be a pre-requisite that the application has been previously Dockerized and is registered in a Docker Hub. CWL offers a simple way of pointing to a Docker container (by adding a hints or requirements section and a DockerRequirement attribute to the CWL file). It is not strictly required for a developer Alice to already have a CWL file prepared as the TEP Client offers a CWL editor which can be used to upload an existing CWL file or to create one from scratch. Facet 1 of the Application Package consists of this. The TEP Client will also assist in generating Facet 2 of the Application Package by building a WPS-T DeployProcess document that includes Facet 1, i.e. the CWL file.

In line with the architectural constraints provided by the sponsor, the WPS-T DeployProcess document is encoded in JSON, not XML, and the interfaces all throughout are based on a RESTful protocol fully described in another Testbed-14 EOC ER, gathering the ADES & EMS Best Practices ([1]).

As mentioned in Chapter 6, the Sponsor requires an AP with two facets, one based on a simple encoding, targeting the application developer, Alice, who is an Earth Observation scientist that is an expert in remote sensing and writes Earth Observation data processing algorithms, often in a programming language convenient for such use (Matlab, Fortran, Python, bash script); and one for the Machine-To-Machine (M2M) exchanges of the AP, therefore needing to contain all the information required by the several architectural building blocks for executing the application.

It should be noted that, as is the case in Testbed-14 with the TEP Client deliverable, clients can exist to greatly simplify and automate the work of putting together an AP and, specifically, the two facets described below. In the particular case of Facet 1, a CWL file, the application developer Alice may have prepared it already (i.e. Facet 1 is pre-existing), or she can use the client and its CWL editor to generate one (i.e. Facet 1 is a client output).

The following snippet provides an example of Facet 2 of the Application Package, a WPS-T DeployProcess document encoded in JSON. This specific example is for a workflow.

As explained above, Facet 1 of the AP - a CWL file - is contained in Facet 2, the WPS-T DeployProcess message. In the example, this can be seen at the end, whereby the deploymentProfileName attribute is set to http://www.opengis.net/profiles/eoc/workflow -, and the executionUnit points (using href) to the CWL workflow file (https://raw.githubusercontent.com/spacebel/testbed14/master/cwl-examples/multiSensorNDVIStacker_2collections/multiSensorNDVIStacker_2collections-v4.cwl).

As explained in [1], the adopted architecture relies on a WPS-T interface using JSON encoding. WPS-T is a transactional extension of WPS, as discussed in [6]. The information model proposed in WPS-T foresees that applications can be written in any one of a number of programming or description languages (e.g. Java, BPEL, etc.) and the so-called "execution unit" allows associating the WPS process to the set of instructions concretely implementing the application. In the Testbed-14 EOC thread, execution units based on CWL are considered. The deploymentProfileName and executionUnit attributes used in the example and solution are features of WPS-T.

Apart from the deploymentProfileName and executionUnit attributes, the third element visible in the example is a WPS Process Description encoded in JSON, which describes the application’s inputs and outputs, still a major feature of any AP format.

The second snippet below provides another example of Facet 2 of the Application Package, in this case for a single application (not a workflow).

Whilst in the first snippet executionUnit points to a CWL file describing a workflow (https://raw.githubusercontent.com/spacebel/testbed14/master/cwl-examples/multiSensorNDVIStacker_2collections/multiSensorNDVIStacker_2collections-v4.cwl) and deploymentProfileName is set to http://www.opengis.net/profiles/eoc/workflow, in the second executionUnit points to a Docker container (images.geomatys.com/ndvis:latest) and deploymentProfileName is set to http://www.opengis.net/profiles/eoc/dockerizedApplication.

This illustrates how the same AP format consisting of a WPS-T DeployProcess message encoded in JSON can be used to request the deployment both of simple applications (second snippet) and workflows (first snippet).

It is finally important to note that, even in the second case, the owsContext element is used to point to a CWL file associated to the process/application. This is because CWL has been used in the EOC Thread of Testbed-14 not only to describe workflows but also to address one of the most important issues left open in Testbed-13 (see [7]), related to how inputs/outputs are made available by the ADES component to the Docker containers for proper execution. See [1] for more information.

For all details pertaining to relevant security aspects - authentication, authorization - as well as quotation and billing, the reader is referred to [8].

It is accepted as a limitation that interactive applications are not supported.

From the three applications below, a workflow can be created to chain the Stacker, then the Feature Extractor and, finally, the Flood Detector.