OGC Testbed-13: Cloud ER

The following requirements are to be addressed in this ER:

Current cloud computing architectures have advanced from virtual hypervisors and shared compute resources to include containerization using Docker. As cloud computing continues to evolve, the scientific community seeks to achieve a more efficient process for deploying, processing, and retrieving data results. It is true that today, acquiring shared computing resources is easier than ever. However significant effort is still required in order to stage computing resources, determine compute requirements, deploy software libraries, and general administrative tasks more suited to information Technology (IT) administrators before processing of scientific data can occur. Furthermore, efficient use of compute resources when not being utilized needs to be accounted for through scalable solutions.

This engineering report seeks to document the Testbed 13 cloud implementations in which OGC web service specifications are used in conjunction with software containers (i.e. Docker) to establish a process flow and retrieve results for data processing functions. The goal is to reduce the amount of administrative work required to stage compute resources and process data, and simplify data retrieval, specifically for earth observation data which can vary depending on data size or volume.

The Working Group identified for the review of this engineering report is the Big Data Domain Working Group (DWG). The Big Data DWG’s current purpose statement defines their scope of work to include Big Data interoperability and especially analytics. However, key members of the DWG are interested in expanding the scope to include cloud in their discussions. Particularly, at least one member of the DWG is also working on multi-cloud discovery and access to Earth Observation data within cloud compute architectures.

The work of the EOC thread aligns with the Big Data DWG interests due to the use of earth observation data storage and processing of this data using OGC WPS standards in a dynamic cloud deployment for interoperable ease of access for scientific study. The goal of this effort is to improve the way EO data is disseminated, processed, stored, and searched without scientists needing to understand how to administer cloud computing resources. This engineering report describes future architectures and deployment methods for cloud architectures utilizing OGC WPS. The use of these architectures may assist in future developments of data processing including use cases such as big data processing. Other OGC W*S services may be considered as future follow-on effort.

All questions regarding this document should be directed to the editor or the contributors:

The work contained herein may lead toward future interoperability tests across private-public cloud implementations. Future developments in cloud computing technology and container technology may lead to future work in interoperability research. The recommended future work on this topic includes the following:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

A virtualized computing resource which provides functionality needed to execute an operating system. A hypervisor is used for native execution to share and manage hardware, allowing for multiple environments which are isolated from one another, yet exist on the same physical machine.

Two separate implementations have been developed by Computer Research Institute of Montréal (CRIM) and George Mason University (GMU), respectively, to support the testbed experiments. Each implementation has been developed independently using various software tools and specifications to achieve the same implementations goals and capabilities as follows:

Based on the architectures described earlier, the overview architecture can be expanded to describe additional components within each category as shown in Figure 2 below. The WPS Client may also include a graphical user interface and security functions. The WPS Server may include synchronous or asynchronous monitoring such as a polling function to get status of a running process. The VMs may contain Docker (see Section Docker Containers) containers and the RSTB processes to process EO data and produce a result. Additionally, a Docker Registry may be used for distributing the Docker images. Data may be accessed in external clouds, and the resulting data may be shared across shared cloud storage.

In general, the location of the WPS server is agnostic from the cloud environment and can be contained as a separate server or within a cloud server. While the WPS may not be represented within a cloud environment in Figure 1, the actual deployment may be within the same or separate cloud environment. However, it is important to note that the cloud environments are able to retrieve data (i.e., RADARSAT-2 Samples) via an external cloud. In this combined expanded architecture, the WPS client contains multiple functions such as application execution and monitoring functions. The virtual machines deployed in the cloud architectures contain application containers using Docker. Both architecture implementation follow this expanded architecture with various differences in how each configures their Docker deployments and VM provisioning.

The development process for the cloud environments were broken out into a Work Breakdown Structure as shown in Figure 3. This structure was developed and agreed upon by all participants to ensure all implementation aspects of the cloud environment were considered.

The engineering report describes the architecutre, configuration, execution, test, and demonstration of the component implementations as follows:

GMU has established an architecture framework as seen in Figure 4 below. The framework is divided into four layers: Client, OGC Web Services, Cloud environment, and Internet (i.e., external network). The client consists of a browser-based application dashboard containing stored WPS requests, virtual host locations, and additional features such as priority and resource allocation functions.

All the capabilities of the WPS server and cloud environment are performed via the WPS interface. In other words, WPS is the only exposed interface for users to access the cloud and Docker container functionality. The user (i.e., Actor) can perform WPS request functions through the client to an OGC WPS 2.0 implementation which also resides in the cloud. Job requests/responses with the cloud are managed by Secure Shell (SSH) using the Docker command-line interface (CLI). The WPS server RSTB processes requests using the Docker CLI by deploying Docker containers in a Virtual Machine (VM) (i.e., cloud instance). The processes contained within each job request publishes the final products via an OGC Web Coverage Service (WCS) service. The final product is returned through OGC WCS. The WPS client receives reference URL to the WCS containing the results.

Additionally, the WPS and cloud environment can interact over the Amazon Elastic Compute Cloud (EC2) API which can manage the deployment and scalability of VM instances using the Amazon Cloud Management Server. The cloud is capable of communicating with other cloud blocks to exchange data or call geospatial functions from different sources hosted externally to the local cloud environment. It should be noted that while there is a good compatibility guide to the OpenStack implementation of the EC2 API, many commands are not supported [EC2].

The CRIM High Level Architecture contains a WPS built on the open-source PyWPS 4.0.0, an OGC WPS 1.0 specification. Since PyWPS uses Python as its code-base, several software applications and libraries adopted for this implementation are also Python based. The current solution targets Python 2.7, in part because PyWPS 4.0 still exhibits input/output issues with Python 3.6. Python 2.7 is currently being phased out of CRIM’s research infrastructure, in favor of Python 3.6. The cloud environment is developed using OpenStack Juno, a free and open source cloud operating system that controls compute, storage, and networking resources in a data center. Docker 1.10 is used as an application package containing user’s process and pulled on demand from the Docker registry, but also as a way to bundle and deploy all other required components of the system. More details regarding each software package utilized can be found in Appendix C - Software Packages. The CRIM High Level Architecture is represented in Figure 5 below.

In this architecture view, the WPS 1.0 Client/Application Manager interacts with a WPS 1.0 Server. The server contains a job manager that splits jobs into tasks. A request containing parameters for processes, cloud, and application registry can be sent to the WPS server to manage the distribution of tasks across one to many task queues within multiple cloud environments. An elasticity manager monitors the tasks queues and automatically triggers the deployment of new VMs when predefined system load thresholds are met. The elasticity manager has been developed by CRIM and has been in operation for the last 3 years in a software research platform called VESTA (http://vesta.crim.ca/index_en.html). Additionally, the input and output parameters of the Docker application packages, as well as execution status, are passed via the task queue. The data results are stored on a shared fileserver and can be accessed either in their raw form or as WCS 1.0 or Web Map Service (WMS) 1.1.1 layers implemented using GeoServer. Once the results are computed, the WPS receives the output in the form of a reference URL to the WMS Server where the user can retrieve the image.