OGC Testbed-13: EP Application Package ER

This ER will define an application package according to the following requirements:

The serialization and deployment of applications behind an OGC Web Processing Service (WPS) was the subject of several discussions at OGC Standards Working Groups (SWG) and Domain Working Groups (DWG) in the past. A diverse number of issues were identified but the main barriers were due to the diversity of runtime environments to support and the respective technology maturity hampered its feasibility.

The advance in container technology together with the concept of federation of Clouds presents new opportunities to address this issue. Previously the constraints imposed in describing the runtime environments (e.g. Linux, Java VM, Perl, Python) and the respective application installation would create a considerable amount of information needed for the application package information model outside the OGC aegis. With containers, most of these requirements are addressed at their respective level with configuration and software elements packaged into isolated elements.

This ER will define the application package aggregating containers and application information to defines a data model and serialization for EP Application Packages.

The concepts presented in this ER will define operational procedures to allow the automatic packaging of applications and deployment of WPS in different computing infrastructures. The main focus of the work presented is on Earth Observation (EO) data processing services but it can be directly applied in other geospatial services.

All questions regarding this document should be directed to the editor or the contributors:

Future OGC testbeds should look to develop draft OGC Web Services Context (OWS Context) profiles for describing Cloud application packages. The approach adopted for Testbed-13 offers a convenient way of conveying relevant process description information (e.g. inputs/outputs) in an OGC standard way, but future work should consider using this as a process description template, for example using template fields and properties (e.g. $id).

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

An Area of Interest is a geographic area that is significant to a user.

A Context Document is a document describing the set of services and their configuration, and ancillary information (area of interest etc) that defines the information representation of a common operating picture.

A text document that contains all the commands a user could call on the command line to assemble an image

A lightweight and configurable virtual machine with a simulated version of an operating system and its hardware, which allow software developers to share their computational environments.

A standardized, highly automated offering, where compute resources, complemented by storage and networking capabilities are owned and hosted by a service provider and offered to customers on-demand. Customers are typically able to self-provision this infrastructure, using a Web-based graphical user interface that serves as an IT operations management console for the overall environment. Application Programming Interface (API) access to the infrastructure may also be offered as an option.

A resource is a configured set of information that is uniquely identifiable to a user. Can be realized as in-line content or by one or more configured web services.

A virtual machine image is a template for creating new instances. A image can be plain operating systems or can have software installed on them, such as databases, application servers, or other applications.

A "Thematic Exploitation Platform" refers to an environment providing a user community interested in a common Theme with a set of services such as very fast access to (i) large volume of data (EO/non-space data), (ii) computing resources (iii) processing software (e.g. toolboxes, retrieval baselines, visualization routines), and (iv) general platform capabilities (e.g. user management and access control, accounting, information portal, collaborative tools, social networks etc.). The platforms thus provide a complete work environment for its' users, enabling them to effectively perform data-intensive research and data exploitation by running dedicated processing software close to the data.

There are different valid strategies for packaging and deploying an Application in a Cloud environment. Overall, the goal is always to have an application able to run in a predictable manner in a large computing production environment, potentially formed by hundreds of nodes. The simplest approach for packaging is the "snapshot", also known as the “Golden image”. Its lifecycle is quite straightforward: when the development of the application is finished, the application and the underlying operating system are frozen and then moved in production to be executed (i.e., using multiple resources). In this case the dependencies are included in the snapshot that thwarts the maintainability and versioning of the service. There are a few major disadvantages to this approach due to the large manual intervention required to prepare the snapshots, dependencies not being explicitly specified and, most importantly, the precise reproducibility cannot be guaranteed.

The other, preferable approach is to guide the user (in this case a developer or application integrator) in order to build an application compatible with the platform’s applications. Such applications have the main advantage of being self-contained. In other words, a single package (e.g. in RPM Package Manager (RPM) format, formerly Red Hat Package Manager format) contains the application resources and the dependencies specification. To complement the approach, the contextualization process can be optimized with the definition of specialized baselines with pre-installed packages (e.g., Interactive Data Language, Matlab). Following the proposed approach, the application packaging will include dependencies specification that improve:

In both cases, the wide usage of virtualization and the possibility to start virtual environments within Cloud services significantly simplify the creation of environments and the provisioning of resources. However, this still leaves a problem of portability of complete environments where applications are developed unresolved. Previous virtualization solutions are not light and flexible enough partially due to the size of the images. Docker fills this gap by providing a platform designed both for developers and system administrators. Docker is an open platform for developers to build, ship and run distributed applications. Composed of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a Cloud service for sharing light images, Docker enables applications to be quickly assembled from components and eliminates the friction between development and production environments. As a result, user applications can ship faster and run the same process into large multi-tenant data centers in the Cloud. The main difference between a Docker container and a Virtual Machine lies in the fact that a Virtual Machine consists of, besides the application itself, also an operating system with all binary files included. Docker images work as an isolated process in the host operating system, which shares a kernel with other containers. Thereby, still having the benefits of virtualization, it is more portable and more effective – an application uses less disk space.

This solution lets the developers package their applications and all of the dependencies in a Docker container to provide a consistent environment for execution and also provides isolation from other applications or software installed on the host. Processes are therefore running in an isolated processing environment which is exactly the same that the developer built for it and can be deployed in several cloud providers (as shown in the figure below).

From the existing developer environments, the process to build a Docker image is fairly straightforward and well documented and outside of the scope of this ER. Overall, all files and dependencies of the application should be self-contained and thus easily installable on the Docker image baseline, the same on which the user has integrated the application and part of the application integration output set. When an application is deployed on a given environment, it will need the docker engine operation to deploy the image on the processing container.

The Application Package will allow the developer to register an application that can be deployed and executed in multiple ICT providers. The application is first built and registered in a third-party Application Hub (e.g. DockerHub). The developer will then directly interact with the Application Management Client (AMC) that will register the application on the Application Deployment and Execution Service (ADES).

The sequence of actions is described in the following image.

Initially the developer builds its Application Package after the development & integration phase (Step 0). The output of this activity is generally a file, be it a ZIP or a RPM that, once validated, can be packaged in a container to expose the same software baseline as the original developer environment. The package provides both the Application and the 'delta'-environment required for executing the Application itself on the target environment. The ‘delta’-environment can also be defined as the explicit dependencies of the Application. After the authentication on the CP/IdP (Step 1), the developer will then push and register his/her application in the Application Hub (Step 2).

With the registration of the application the developer can then request its deployment in any of the supported infrastructure. For this - even though not strictly required - the developer can request the available application list (Step 3) to the Application Management Client. This component will retrieve the necessary information by executing a "describe deployed processes" operation (Step 4, consisting of a WPS GetCapabilities and multiple WPS DescribeProcess requests) that will collect the list of supported clusters (Step 5) and list of registered applications (Step 6).

With this information, the user is able to request an application deployment (Step 7) to the Application Management Client that will be responsible for submitting the request to the Application Deployment and Execution Service, taking in consideration the requested application and resources (Step 8).

The ADES will read the manifest (Step 9), register the service (Step 10) and request the application container (Step 11) that will be retrieved for use in the ICT infrastructure (Step 12). The resulting response will be the new address of the deployed WPS service instance.

The overall components and respective flow is also presented below.

The Application Package will contain information about the application title, developer, and short abstract available for display to a human.

The Application should be executed as a Docker Container in a Docker environment.

The developer builds a virtual environment containing all required libraries and resources to execute the application processing (including any input data which will not be provided as an input by the front-end).

A Docker Container Image (the portable snapshot of the container) can be built from a Docker build context stored in a repository. A build context is a Dockerfile and any files at a specific location.

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. A Dockerfile begins with defining an image from which the build process starts, followed by various other methods, commands and arguments (or conditions). The Dockerfile also specifies which command should be called when running the container.

With the Dockerfile, the developer creates a container build context defining the OS, libraries and any environment considerations. It also includes the application files (binary, resources, etc.) and indicates the command that should be called when running the container.

A last but essential step remains, the application should be wrapped by a script in order to read and write the data shared with the ADES.

An application execution always implies a set of input parameter(s) and file(s), and generates, as result of the processing, a set of outputs (values or files). The execution of a process behind a WPS relies on the description of the application in an eXtensible Markup Language (XML) document named a WPS Process Description. The WPS specification defines the following terms:

In order to pass simple parameters (WPS Literal Data) to the Application, the ADES can execute the container application passing the parameters by command line parameters (or by environment variables in alternative approaches). The name of the parameters (or environment variable) is specified by the developer in the WPS Process Description as the input Identifier.

When the ADES executes a (WPS) process, the service needs to provide received input files into the (cloud) Docker environment. This folder can be specified by the developer or default to a specific location defined by an environment variable pointing to the working directory

Note that files (inputs and outputs) may be passed by value (file content) or by reference through a Uniform Resource Locator (URL). Passing by reference might impose an additional step of data retrieval by the application. An alternative solution is for the ADES to provide a local reference to the running application (on the platform) if the data is available locally. Constraints or information about this approach should be described in input metadata of the WPS Process Description.

To discover which files are best suited for the required processing, the Application Package must provide the access points of the catalogue(s) where the dataset metadata is stored and where it is possible to perform complex queries. The catalogue should provide a search engine capable to deal with different types of queries (Geographic and non-Geographic) and include the online access points of the files. This catalogue will provide a framework to ease the discovery of the necessary data.

EO products are differentiated by parameters such as the date of acquisition and the image footprint as well as characteristics pertaining to the type of sensor, the type of platform, the applied processing chain, and more. Using the best practices for search services in the EO domain the catalogue should support the OpenSearch specification with Geo, Time (OGC 10-032) and EO extensions. The EO OpenSearch Extension (OGC 13-026) allows also the specializations for specific thematic classes of EO products, such as optical, radar, atmospheric, altimetry, limb-looking and systematic/synthesized EO products and also includes the support for the European Union’s INSPIRE directive.

The application might also need auxiliary layers to guide the user in selecting the processing parameters or to display the processing result. These layers can be in image format (e.g. map with elevation) or vector (e.g. map with volcanoes positioning) and should be included on the Application Package information.

Once the container is running, the shared input files are consumed by the application running in the virtual container. Finally, the application finishes execution and the ADES collects outputs of the process. The Application shall write (or copy) output data files into the location specified by the metadata.

The OWS Context (OWC) document was initially created to aggregate a set of configured information resources as a collection of services to be shared among different applications or domains. An OWS Context supports OGC services, online resources and in-line content as well. The specification document follows the OGC Approach of a conceptual model (OGC 12-080r2) and implements specific encodings for ATOM (OGC 12-084r2) and GeoJSON (14-055r2) with well-defined mappings.

The OWS Context was designed to support use cases such as the distribution of search results, the exchange of a set of resources such as OGC Web Feature Service (WFS), Web Map Service (WMS), Web Map Tile Service (WMTS), Web Coverage Service (WCS) and others in a ‘common operating picture’. Additionally, OWS Context can deliver a set of configured processing services as Web Processing Service (WPS) parameters to allow the processing to be reproduced. For that, the standard establishes requirements classes for a series of offering extensions, relating to services or content. This ER extends that notion to containers and application parameters to support the deployment of applications in multiple cloud providers.

The Application Package information model uses the ows:Context class (OGC 12-084r2) mapped to the atom:feed element as the overall container class for the document. Its properties are documented below.

The XML representation of a Application Package in feed is shown below

The OWS Context document includes specific resources to identify specific application information covering:

Also included on the document is the information about the area of interest of the application package. At feed-level, this optional element expresses a geographic area that is significant to the applications.

The Application Package information model uses the ows:Resource class (OGC 12-084r2) mapped to the atom:entry element to define each of the previous elements. Each individual characteristic is documented below.

The application entry has a profile referenced as "http://www.opengis.net/tb13/eoc/application" represented as

The application container information is mapped to a owc:offering that must have a code attribute equal to "http://www.opengis.net/tb13/eoc/docker" represented as

The application parameters are conveyed by a WPS Description mapped to a owc:offering (OGC 12-084r2) that must have a code attribute equal to "http://www.opengis.net/tb13/eoc/wpsProcessOffering". The offering must have the owc:content (OGC 12-084r2) element with the XML representing the result of a DescribeProcess request. It is important to note that even though this WPS Description could match exactly the corresponding DescribeProcess response after the process has been deployed, it is not conveyed as part of the Application Package with that intention, given that this poses some problems (e.g. the client/user defines the process identifier before the process is deployed). For the purpose of this testbed, it offers a convenient way of conveying relevant process description information (e.g. inputs/outputs) in an OGC standard way, but future work should consider using this as a process description template, for example using dummy id values or template fields (e.g. $id).

Both offerings are part of a single entry as shown below

It is also possible to extend the entry with additional informative data using the atom category element.

The information is application-driven and this ER does not provide or mandate any specific guidance for their interpretation.

An Application Package can contain any given number of entries identified as catalogues. The catalogue entry has a profile referenced as "http://www.opengis.net/tb13/eoc/catalogue"

The application catalogue information is mapped to a owc:offering that must have a code attribute equal to "http://www.opengis.net/spec/owc-atom/1.0/opensearch" (currently the only catalogue specification supported)

The search operation can be refined with the inclusion of a OpenSearch Query element defining a query subset. The following example shows how to define a restriction on the cloud cover query parameter.

A catalogue entry can be represented as

The catalogue entry can include information about the preferred geographical area of the query using the georss:where element.

For the preferred temporal extent the date or range of dates relevant to the catalogue resource can be included inside a dc:date element. The content of this element SHALL conform to the "date-time" production of ISO-8601. An uppercase "T" character SHALL be used to separate date and time, and an uppercase "Z" character SHALL be present in the absence of a numeric time zone offset. To specify a range of dates the "/" character SHALL be used.

If more than one catalogue entry is present the application package should use the OWS Context Active flag value (OGC 12-084r2) indicating to the client if the entry should be used by default.

If needed, the Application Package can use the OWS Context Folder (OGC 12-084r2) to define an organisation of the entries.

Although it is anticipated that only one application catalogue entry would appear in a context feed, it is possible to have multiple catalogue entries. To link the application catalogue to a given WPS parameter it should be used the ows:Metadata element to include the atom:link with the respective search catalogue.

In this example, one catalogue is used to search for data for the "Image" input and another catalogue is used to search for data for the "ReferenceData" input.

It was also the case in the thread that different catalogues needed to be used for different platforms. For example, a different catalogue was used to search for data products on IPT POLAND than was used to search for data products on AWS. This was handled in the testbed using two different process offerings and thus two different applications packages. Another, untested, approach for handling this situation could have been to make use of the "about" attribute on the ows:Metadata element to discriminate one platform from the other.

An Application Package can contain any given number of entries identified as base maps. The base map is defined as an auxiliary layer of information that supports the Application Package user but is not considered essential to its execution. The base map entry has a profile referenced as "http://www.opengis.net/tb13/eoc/basemap"

The base maps can be of any type of OGC service (for example, not just OGC Web Map Service but also OGC Web Feature Service and OGC Web Coverage Service) or non-OGC compliant service. This section shows an example for a WMS and a MapBox Service as base maps.

An Application Package can contain any other number of entries. It is out of the scope of this ER to provide guidance for their interpretation.