OGC Testbed-13: Application Deployment and Execution Service ER

This ER will document the automation of the application package deployment taking in consideration an application package defining:

The serialization and deployment of applications behind an OGC Web Processing Service (WPS) service was the subject of several discussions at OGC Standards Working Group (SWG) and Domain Working Group (DWG) in the past. A diverse number of issues were identified but the main barriers were due to the diversity of runtime environment to support and the respective technology maturity hampered its feasibility.

The advance in container technology together with the concept of federation of Clouds presents new opportunities to address this issue. Previously the constraints imposed in describing the runtime environments (e.g. Linux, Java VM, Perl, Python) and the respective application installation would create a considerable amount of information needed for the application package information model outside the OGC aegis. With containers most of these requirements are addressed at their respective level with configuration and software elements packaged into isolated elements.

This ER will capture implementation process experiences for Exploitation Platform (EP) Application Packages and their deployment in multiple clouds.

This ER is relevant to the WPS SWG because it lists the requirements fulfilled by Cloud APIs in order to allow WPS-supported automation of application package deployment and execution.

All questions regarding this document should be directed to the editor or the contributors:

Future OGC testbeds should look to develop draft WPS profiles for Cloud application deployment and execution. It may also be necessary to develop extensions to the WPS standard, for example using interfaces based on Representational State Transfer (REST) to support Cloud application deployment and execution.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

An Area of Interest is a geographic area that is significant to a user.

A Context Document is a document describing the set of services and their configuration, and ancillary information (area of interest etc) that defines the information representation of a common operating picture.

A text document that contains all the commands a user could call on the command line to assemble an image.

A lightweight and configurable virtual machine with a simulated version of an operating system and its hardware, which allow software developers to share their computational environments.

A standardized, highly automated offering, where compute resources, complemented by storage and networking capabilities are owned and hosted by a service provider and offered to customers on-demand. Customers are typically able to self-provision this infrastructure, using a Web-based graphical user interface that serves as an IT operations management console for the overall environment. API access to the infrastructure may also be offered as an option.

A resource is a configured set of information that is uniquely identifiable to a user. Can be realized as in-line content or by one or more configured web services.

A virtual machine image is a template for creating new instances. An image can offer plain operating systems or can have software installed on them, such as databases, application servers, or other applications.

A "Thematic Exploitation Platform" refers to an environment providing a user community interested in a common Theme with a set of services such as very fast access to (i) large volume of data (EO/non-space data), (ii) computing resources (iii) processing software (e.g. toolboxes, RTMs, retrieval baselines, visualization routines), and (iv) general platform capabilities (e.g. user management and access control, accounting, information portal, collaborative tools, social networks etc.). The platforms thus provide a complete work environment for its' users, enabling them to effectively perform data-intensive research and data exploitation by running dedicated processing software close to the data.

The application package will allow the developer (e.g. Markus) to register an application that will be deployed in a given ICT provider. The application is first built and registered in a third-party Application Hub (e.g. DockerHub). The developer will then directly interact with the Application Management Client (AMC) that will register the application on the Application Deployment and Execution Service (ADES).

The sequence of actions is described in the following image.

With the application registered, a user (e.g. Jim) can directly interact with the AMC that will allow the execution of the application on the ADES.

The sequence of actions for the execution is described in the following image.

The overall components and respective flow is presented below.

The Application Package targets applications built by third party developers and contains all the information necessary for their deployment in different infrastructures. It uses an ATOM OWS Context document as the document encoding and WPS as the execution interface. This chapter will address several implementation issues discussed by the EOC thread with particular focus on the approaches to application packaging, application execution and data access.

The application package encoding conveys all the information needed for encapsulating user applications and enabling their automatic management (upload, deploy, run) in diverse platforms. It uses the OWS Context document encoding to convey the necessary information model as an input parameter to a dedicated WPS service that creates and deploys it in a new WPS instance. The requirements and respective encoding are described on the other Testbed-13 EOC ER (OGC 17-023) [2] dedicated to reporting of the Application Package

The ADES provides a framework to deal with the data management tasks regarding the input file access and the registration of the output files. This is provided by two abstract functions that deal with the data staging and stage out that is executed by the application.

For data staging the convention is to map a file location from a URL (e.g. S3, FTP, Atom feed) to a local copy. The actual operation performed depends on the deployed infrastructure and data access mechanisms and it should be transparent for the application. It can consist of a simple string transformation based on previous knowledge about the data storage or to an actual data transfer mechanism that will provide the file data stream to the local processing.

Equivalently, for the data stage out, the infrastructure will make available a specific convention. This operation will map a specific resulting file into the infrastructure publish mechanism guaranteeing the file safeguard and the subsequent access by the WPS client.

The ADES provides a framework to deal with correct assignment of the application parameters. This is provided by one abstract function that provides the application a method for accessing the parameters values. For that the application uses a function for accessing the request parameters that from the command line would behave like:

An alternative solution using the environmental variable approach is also described in the next chapter.

The approach described in this chapter is an alternative to the one described in the previous Implementation chapter. It is based on the use of a Transactional WPS (WPS-T) as described in the WPS 2.0 Transactional Extension [3] discussion paper. This discussion paper was itself derived from the WPS 1.0 based Transactional WPS Extension (WPS-T) Specification [4] and from the experience gained in several ESA projects where the Transactional WPS 1.0 was used to deploy Business Process Execution Language (BPEL) workflows, Grid applications, Oozie workflows, and Java applications.

In this approach, as illustrated in Figure 15, the Application Deployment and Execution Service (ADES) is essentially a set of WPS-T servers, each server encapsulating a particular cloud processing environment (e.g. IPT Poland or AWS) where the TEP applications are deployed and executed. The TEP applications are containerized applications running on virtual machines equipped with a container execution engine (e.g. Docker Engine) and possibly container clustering tools (e.g. Docker Swarm, Mesos, or Kubernetes). The container technology used in this testbed is Docker.

The Application package proposed in this testbed is an OWS Context document as defined in the Exploitation Platform Application Package Engineering Report [2] . This document provides all the information necessary for both the Application Management Client (AMC) and the Application Deployment and Execution Service to perform their work for the corresponding application. This information includes a WPS Process Description that matches the inputs and outputs of the application.

In this approach, instead of embedding (using an OWC Offering) the WPS Process Description inside the OWS Context document, an enhanced WPS Process Description is actually used to embed (using OWS Metadata) the remaining part of the OWS Context. This is done using the ApplicationContext element as illustrated in the XML fragment below. Therefore, no information is lost when using the enhanced WPS Process Description. The advantage of this approach is that the discovery (e.g. by the AMC) of Application Packages coincides with the discovery of WPS Processes.

The source of the Docker image used in this chapter is available on GitHub (https://github.com/spacebel/landcover) and the image is available on Docker Hub (https://hub.docker.com/r/cnlspacebel/landcover/).

The AMC is a generic component that can be used for any TEP Application. It relies entirely on the information provided in the Application Package (i.e. the enhanced WPS Process Description described in the previous section) in order to dynamically build the application specific user interface needed to:

The AMC uses operations provided by the Central Geospatial Resource Catalogue and the ADES as illustrated in Figure 16.

The TEP Application deployment is performed (at Marco’s request) using the WPS-T DeployProcess operation. The typical case is when the container image is available on an external registry. However, if the image is a relatively small layer on top of another image, this image could also be passed by value in the DeployProcess request and stored in a local registry.

The discovery by the AMC of available (i.e. deployed) applications is performed using the WPS GetCapabilities operation followed by a WPS DescribeProcess operation. The WPS DescribeProcess operation returns a process description with all the information specific to the selected application. In particular, the OWS Context contains the entry point in the Central Geospatial Resource Catalogue i.e. the URL to get the OpenSearch Description Document (OSDD) and the reference to the container image in a Docker registry. Of course, the process description also contains all the details regarding the application inputs and outputs.

The execution of the application is performed (at Jim’s request) using WPS Execute operation. To capture the inputs, the AMC uses the process description to build the appropriate user interface. For inputs to be obtained from the Central Geospatial Resource Catalogue, the AMC uses the information (template and parameters) contained in the OSDD to build the appropriate user interface.

To monitor the execution of the applications, the AMC uses the WPS GetStatus operation. Note that TEP Applications have typically long execution times and are, therefore, executed in asynchronous mode.

Finally, to get the results generated by the application, the AMC uses the WPS GetResult operation. Depending on the type of results, a download and/or preview of the results is also provided.

The TEP Application used in Alternative approach is the Land Cover Mapping application described in the F-TEP Application in OGC Testbed 13 [8] document.

The development of the testbed AMC is done using a Web application based on JavaServer Faces (JSF) and deployed in a GlassFish server. The AMC is available at the following location: http://ld-ogc-tb13-tep.spacebel.be/TB13-WebApp/ .

The ADES is a component that implements the WPS-T operations shown on Figure 16. This figure also illustrates the ADES interaction with the cloud environment and with the containerized TEP Application.

The WPS-T implementation is very close to the WPS 2.0 Transactional Extension [3] document with some minor differences as explained below. This document covers the HTTP POST + XML and KVP encodings. REST + JSON encoding is proposed below in a separate section.

As shown on Figure 24, the Capabilities document is unchanged and contains the list of the deployment profiles supported by the server.

The DeployProcess request is modified slightly to include a ProcessOffering (that includes a ProcessDescription) instead of a ProcessDescription and a DeploymentProfile element is added to hold the DeploymentProfileName and the ExecutionUnit. A ProfileExtension is introduced to allow for additional information specific to a particular processing environment. For example, it could possibly be used to pass the ApplicationContext described earlier instead of including it in the OWS Metadata of the Process Description. The ExecutionUnit is made optional (if specified, the corresponding information in the enhanced WPS Process Description is overwritten) and its InstructionPackage is renamed simply as Package. This is illustrated on Figure 25.

As shown on Figure 26, the UndeployProcess operation is unchanged. The KeepExecutionUnit is not used in the testbed as this is handled by the Docker Engine.

As illustrated in Figure 16, the ADES interacts with the Docker Engine and with the TEP Application. The ADES interacts with the Docker Engine in order to prepare the container environment (e.g. mounting shared storage in the container and creating environment variables) and to start the application in the container (resulting in the execution of the shell script defined in the Docker image). The ADES interacts with the TEP Application through environment variables to pass WPS inputs and WPS outputs.

The values of WPS inputs of LiteralData type and the file location of WPS inputs of ComplexData type are assigned (by the ADES) to the corresponding environment variables provided to the TEP Application. The target file location of WPS outputs (of both LiteralData and ComplexData types) are assigned (by the ADES) to the corresponding environment variables provided to the TEP Application. It is the responsibility of the ADES to make the input files available in the container environment by extracting them from the execute request (data transmission by value), or by fetching them from a remote location (data transmission by reference with a protocol such as http://), or by mounting them from a local store (data transmission by reference with a protocol such as file:/// or s3://). It is the responsibility of the TEP Application to create/append results to ouput files.

To ensure interoperability, a convention is used to name the environment variables i.e. by prefixing the WPS input identifier with WPS_INPUT_ (e.g. WPS_INPUT_Image) and the output identifier with WPS_OUTPUT_ (e.g. WPS_OUTPUT_Image). Although not needed by the Land Cover Mapping application used in the testbed, the convention should also cover repeated WPS inputs and WPS outputs (e.g. WPS_INPUT_Mask_x with x=1,2,3…​) as well as nested WPS inputs and WPS outputs (e.g. WPS_INPUT_Band_x_IntensityMin).

The development of the testbed ADES is based on 52°North WPS 2.0 implementation that was used in OGC Testbed 12 (https://github.com/52North/WPS/tree/wps-4.0, Git revision 9aaa70b341) and is available on GitHub (https://github.com/spacebel/WPS). Its Wiki (https://github.com/spacebel/WPS/wiki) provides additional details. The ADES is available at the following location: http://ld-ogc-tb13-tep.spacebel.be:8081/wps/ .

The approach described in this chapter is an alternative to the approach described in the Implementation chapter. This section proposes mechanisms that can be used to allow for interoperability of the above described WPS-T based ADES with other AMC clients and Application Packages.

Some AMC clients are designed to interface with an ADES service that offers a dedicated DeployProcess WPS process that is used for the deployment of the TEP Application. The execution of this process takes the application OWS Context as an input. Therefore, a similar non-transactional DeployProcess WPS process can be added in the WPS-T based ADES. This process can extract from the OWS Context all the information needed to invoke an "internal" DeployProcess operation as illustrated in Figure 27. Once deployed, the execution of the newly deployed process (e.g. LandCover) is identical for all AMC clients. A similar approach can be used for the undeployment of the TEP Application using an UndeployProcess WPS process.

Some ADES servers are designed to interface with an application that uses staging commands. Therefore, to be able to use the same application container image for all ADES servers, staging commands can be written to wrap the interface based on environment variables and volume mounting of shared storage.

The REST interface to the Transactional WPS is built on top of the REST binding defined in chapter 11 of OGC Testbed 12 REST Architecture Engineering Report [5] . No new resources are needed but, as shown on Figure 28, two new HTTP operations must be defined to support the deployment and undeployment of WPS processes.

Note that a new HTTP DELETE operation was also added to support the Dismiss extended WPS operation defined in chapter 12 of the OGC WPS 2.0 Interface Standard [OGC 14-065].

The JSON encoding of the request and responses are derived from XML encoding using the rules defined in chapter 6 of OGC Testbed 11 Implementing JSON/GeoJSON in an OGC Standard Engineering Report [6] .

An example of the JSON encoding of (the body of) the DeployProcess request is shown below.

The JSON encoding of the (body of the) response to the above DeployProcess request example is shown below.

The JSON encoding of (the body of) the UndeployProcess response is shown below. The UndeployProcess request has no body.

The plan was to develop the REST + JSON encodings for the WPS-T ADES using the WPS 2.0 REST façade implementation from 52° North that was also used in OGC Testbed 12 (https://github.com/52North/wps-proxy). However, due to lack of time, this implementation was not done.

As illustrated on Figure 15, the security approach to be followed in the testbed, at least for IPT Poland where an Identity Provider (IdP) and a Security Token Service (STS) are available, is compliant with the User Management Interfaces for Earth Observation Services [7] document. The HTTP bindings described in sections 7.1.2 (STS interface) and 7.2.2 (PEP interface) of that document are used (instead of the SOAP bindings).

Authentication (and Single Sign-On) is performed on the client side (AMC) using a checkpoint (in this case a Shibboleth Service Provider) that intercepts and redirects, if the user is not already signed on, access requests to protected AMC resources to a Web page of the IdP (in this case a Shibboleth IdP) for authentication. Once authenticated, the checkpoint includes IdP provided assertions (e.g. about the username, email, etc.) in the HTTP header of the user request and forwards this request to the AMC.

Authentication and authorization is performed on the server side (ADES) using an STS and a Policy Enforcement Point (PEP) respectively. According to the "Local STS with External IdP" approach described in section 6.4.3.3 of [7] , the AMC client sends a signed Request Security Token (RST) with the username to the STS and gets back a SAML token signed by STS and encrypted with the public key of the PEP. The AMC then inserts this SAML token in the HTTP header of all service requests it sends towards the ADES (a new SAML token must be obtained frequently to avoid token expiration). The PEP then decrypts the SAML token, verifies the token signature, and performs authorization using policies based on the SAML assertions. If authorized, the AMC request is finally forwarded to the ADES (with or without the decrypted SAML token).

The PEP software provided by ESA for the testbed does not actually perform the authorization part, but simply passes the assertions (SAML attributes) to the ADES in the HTTP header.

Although compliant with section 7.2.2 of [7] , the provided PEP is not inline with the spirit of the User Management Interfaces for Earth Observation Services [7] document (see section 6.1) where the PEP was meant to perform authorization in a transparent (non-intrusive) way i.e. without impacting the target service (in this case the ADES). Therefore, and also due to lack of time, the security aspects were not implemented in this testbed.