Publication Date: 2018-12-20

Approval Date: 2018-07-10

Submission Date: 2018-06-12

Reference number of this document: OGC 18-046

Reference URL for this document: http://www.opengis.net/doc/PER/eoep-Hack2018

Category: Public Engineering Report

Editor: Ingo Simonis

Title: OGC Earth Observation Exploitation Platform Hackathon 2018 Engineering Report

group
OGC Engineering Report

COPYRIGHT

Copyright © 2018 Open Geospatial Consortium. To obtain additional rights of use, visit http://www.opengeospatial.org/

WARNING

This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.

LICENSE AGREEMENT

Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.

If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.

THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.

This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.

Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.

This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.

None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.

Table of Contents

1. Summary

The Earth Observation Exploitation Platform Hackathon 2018 was conducted to evaluate the standards based architecture for deploying and executing arbitrary applications close to the physical location of the data in heterogeneous cloud environments. The Hackathon was very successful in demonstrating both efficiency and sustainability of the architecture developed in Testbed-13. Efficient, because it was possible to setup the full execution workflow of 128 Sentinel-1 images within the 1.5 days of the Hackathon in a multi-vendor environment. Sustainable, because the architectural approach provides sufficient flexibility to cater for possible extensions and exchange of cloud & container middleware.

The Hackathon produced a number of suggestions for future work items. These include new tools to facilitate the process of Application Package generation to make it even simpler for scientists to bring their applications to the market; a more detailed specification to further improve the level of interoperability; and a best practice document with lots of examples that illustrate the necessary steps to make applications available.

Hackathon participants highlighted that such a level of robustness, flexibility, and maturity of the application-to-the-cloud architecture has been developed in nine months only during Testbed-13. The participants recommend to continue interlacing major OGC Innovation Program activities, such as testbeds, with short term rapid prototyping initiatives such as hackathons. Almost all participants of the Hackathon had been new to the OGC Innovation Program. These participants emphasized that the Hackathon provided an outstanding opportunity for newcomers to get quickly familiar with the latest standardization efforts and helped tremendously in understanding investments and new market opportunities for applications-in-the-cloud.

1.1. Requirements & Research Motivation

The Hackathon was conducted to stress-test results from Testbed-13 and to evaluate the benefit of having hackathons in between major OGC Innovation Program initiatives such as testbeds. Further on, the goal was to attract organizations that have not participated in the development of the applications-in-the-cloud architecture developments of Testbed-13 for a critical review.

1.2. Prior-After Comparison

Organizationally, it turned out that having hackathons in between Testbeds is extremely valuable. Hackathons attract new organizations, allow exploring alternatives very efficiently, and help shaping upcoming initiatives with experiences and suggestions from the Hackathon discussions.

On the engineering side, Testbed-13 produced a number of alternative options without giving clear guidance on the preferred approach. This was issue was addressed successfully in the Hackathon. It turned out that the general architecture, interface design and Application Package model have been confirmed during the Hackathon. This is an important achievement, as it allows to continue the development of standardized approaches for application provision, ad-hoc deployment and dynamic execution in cloud environments in future OGC Innovation Program initiatives.

Interesting alternatives have been developed that include a WPS instance as essential part of a Docker Image. This approach (which can be considered a micro-service approach, as each instance contains a full service interface) gives more flexibility to the application developer. All mappings (e.g. the mounting of external data to internal paths) can be realized on the programmatic level. This approach requires higher programming skills if no tools are available to configure the embedded WPS service and to package the application on behalf of the developer.

1.3. Recommendations for Future Work

The results of this Engineering Report serve as direct input for the Earth Observation Clouds thread in Testbed-14. It is recommended to address the following aspects:

  • More complex applications that involve data from multiple clouds,

  • Fully secured environments,

  • Develop tools that generate Application Packages or at least support scientists in this task,

  • Develop best practices illustrating service setup and Application Package examples, and

  • Provide more detailed specifications to enhance the level of interoperability.

1.4. Document contributor contact points

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Name Organization

Ingo Simonis (editor)

OGC

Benjamin Pross

52°North

Matthes Rieke

52°North

Christoph Stasch

52°North

Vincent Gaudissart

CS

Christophe Triquet

CS

Armin Costa

Eurac Research

Alexander Jacob

Eurac Research

Paulo Sacramento

Solenix

Daniel Robinson

Solenix

Bernard Valentin

Space Applications Services

Leslie Gale

Space Applications Services

Marian Neagul

UVT

Teodora Selea

UVT

Silviu Panica

IeAT

Jeroen Dries

VITO

David Pérez

Thales Alenia Space

Alejandro Mousist

Thales Alenia Space

Elisa Callejo

Thales Alenia Space

1.5. Cloud Providers

A special thank you to our CloudSigma, CloudFerro and Amazon for making cloud resources available to the Hackathon!

1.6. Foreword

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

2. Terms and definitions

For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard OGC 06-121r9 shall apply.

2.1. Abbreviated terms

ADES

Application Deployment and Execution Service

AP

Application Package

EMS

Exploitation Platform Management Service

EO

Earth Observation

EP

Exploitation Platform

IAAS

Infrastructure as a Service

MEP

Mission Exploitation Platform

PAAS

Platform as a Service

SOA

Service Oriented Architecture

WPD

Web Processing Service ProcessDescriptor

WPS

Web Processing Service

3. Introduction

The Earth Observation Exploitation Platform Hackathon 2018 was organized by OGC at the European Space Agency Operations Center in Darmstadt, Germany, on May 3rd and 4th. Sponsored by ESA and NRCan, the goal of the Hackathon was to evaluate results from the recently concluded Testbed-13 initiative. Testbed-13 produced three Engineering Reports to describe a standards-based approach for deploying arbitrary applications on cloud platforms.

In particular the first two Engineering Reports state several alternatives and provide recommendations on the preferred approach very carefully only. The full architecture is described best in [1] Thus, the Hackathon was organized to experiment with the alternatives and to give room for additional approaches suggested by Hackathon participants.

3.1. The Challenge

The challenge was rather straight forward: Take an arbitrary application, store this application in a Docker container, describe this application with the necessary detail to allow standards-based automated deployment and execution, and then register this application with a Web service that allows executing the workflow publicly. Then, pretend to be another user, execute the application in the cloud and access the results.

3.2. Use Case

The implemented use case is briefly outlined here and described in full detail in chapter Implementation Challenge, section UseCase. The use case describes a burn scare mapping scenario in Canada, where the extent of wild fires is analyzed based on radar data processing. The use case required to process a collection of Sentinel-1 radar data for the Northwest Territories (NWT) at the beginning and the end of the 2017 summer. Large wildfires occurred in 2013, 2014 and 2015 in this region, and the application should allow spatially-extensive, timely and cost effective wildfire mapping and monitoring to better assess post-fire burns and tracking of impacts of disturbances on forestlands.

usecase
Figure 1. The use case required to process radar data over Canada’s Northwest Territories, a land area of approximately 1,144,000 km2 with a population of less than 50,000 people

Canada’s forests cover an area of 348 million hectares, which is 35% of Canada’s land area and 9% of the world’s forested area. Because vast areas are inaccessible, researchers use satellites such as Sentinel-1 to gain valuable insights into Canada’s forest ecosystem. The Hackathon evaluated the extent of wild fires based on Sentinel-1 data for the summer of 2017 over the Northwest Territories, Canada. In total, 128 Sentinel-1 IW images (832 GB) for the two coverages (beginning and end of summer 2017) of the entire NWT had to be processed.

3.3. What was provided

Organizers provided the Sentinel Application Platform (SNAP) Software Toolbox together with a pre-defined workflow packaged in a Docker container. Hackathon participants could use the Docker container "as is", i.e., there was no need to modify the container or the application. Given that the container included both SNAP and an executable SNAP graph, participants could on the hand execute the application independently of the available Docker image to test alternatives to the Testbed-13 solutions.

The SNAP workflow in the Docker container required two types of data: Digital Elevation Model (DEM) data and Sentinel-1 data. Both have been made available to the participants as cloud resources. Natural Resources Canada provided a Canadian Digital Elevation Model (CDEM) for terrain correction due to lack of SRTM over 60° North, ESA provided 128 Sentinel-1 scenes.

3.4. Requested Solution

The requested solution is defined in full detail in chapter Implementation Challenge and only briefly outlined here. The requested solution was based on the Testbed-13 results, which are illustrated in the figure below and briefly described in the following paragraphs. There was no exclusiveness, though. Participants could develop their own solution that deviated from the Testbed-13 base.

t13solution
Figure 2. Testbed-13 Architecture

The application developer in the upper left corner packages the application together with all required libraries into a Docker container (1) and describes it following the Application Package specification (2). The Application Package will then be registered with the Application Deployment and Execution Services, ADES (3). The ADES provides a WPS interface. The ADES registers the new application and makes it available as a new WPS process. On request from an application consumer (4), the ADES provides a description of all parameters required to be provided as part of an execution request. On execution (6), the ADES deploys the application container on a cloud and executes it. Once done, the application consumer is provided with instructions on how to access the results. The discovery process of the new process (5) that allows the discovery of the new process in a catalog service is ignored in the Hackathon.

3.5. Possible Deviation

Participants have been free to deviate from the architecture outlined above as long as the following requirements are met:

  • the application developer can make an application available in a container to the cloud platform;

  • the application can be executed with a simple WPS execute() request, i.e., mounts the input data to the Docker mount points automatically, downloads necessary data, executes all processing steps, stores the results persistently on the cloud, and informs the user upon completion of the process; and

  • any consumer can discover the application and request its deployment and execution in the cloud.

3.6. Hackathon Participants

The following organizations participated in the Hackathon as sponsors, organizers, participants, cloud providers, or observers.

Table 1. Participating organizations. Organizations marked with '*' participated in the Testbed-13 Earth Observation Cloud activities.

ESA*

52north

West University of Timisoara

NRCan*

Eurac Research

VITO

OGC*

Bind to service

Institute e-Austria

Solenix Deutschland GmbH*

CloudSigma

Space Applications Services

C-S

EUMETSAT

Thales Alenia Space

4. Key Findings

This chapter reviews architectural aspects that have been discussed at the Hackathon most intensively. Some of them identify potential alternative approaches to the solutions developed in Testbed-13, others help to answer open questions that had been identified at the end of Testbed-13.

4.1. Existing Architecture

The overall experience with the implementation based on Testbed-13 results are good.

4.1.1. WPS

It was generally agreed that it makes perfect sense to facade any type of application with a WPS. Though, mixed opinions have been stated on the actual location of the WPS, which could be a service offered by the ADES, a service that is available per MEP, or integral part of the Docker container itself (see microservice based approach). The WPS-based solutions all worked well and provided sufficient flexibility.

WPS RESTful

It was emphasized that all participants prefer an OpenAPI 3.0 RESTful interface rather than the Service Oriented Architecture Remote Procedure Call approach described in the current standard WPS 2.0. Participants said that it is much easier and better supported by available tools to interact with a WPS instance that way.

WPS Interpretation of New Process Deployment

It turned out that the WPS mechanisms to deploy or register new processes have been perceived differently by the participants. In principle, two options exist to add a new process to an WPS instance: First, to use a dedicated operation, second, to use the Execute operation with specific parameterization.

WPS operations are standard built-in functions supported at the service level (e.g., GetCapabilities, DescribeProcess, or Execute). In a transactional WPS, these can be extended by additional operations such as Deploy and Undeploy to allow registering new processes. WPS processes are application specific functions only available in certain WPS instances (in their process offerings). Additional processes can also be added by using the Execute operation. Then, the Execute(New Process) operation gets specific parameters that indicate that this operation execution shall add a new process to the WPS instance. Due to timing constraints, the latter approach was used in the Hackathon, though the used approach has no influence on the actual interpretation problem.

As said, it turned out that the WPS Deploy() respectively Execute(NewProcess) operation was perceived differently by the participants. The intention of these calls is to add a new process to a WPS instance. The WPS advertises this new process as part of its capabilities (section ProcessOfferings). Confusion was caused by different interpretations what the WPS is actually doing in the background when adding a new process. In most cases, the WPS might just add this new process to the list of available processes without any further actions. Only when the new process is executed, then the WPS will run the necessary steps to deploy the process on the cloud and starts its execution. Some participants interpreted the word deploy in the sense of actual deployment in the cloud, which would cause a whole lot of additional overhead, such as instance control and monitoring in the cloud. It is emphasized that the behavior of the WPS is totally opaque in that sense. Any WPS instance can do with the new process whatever deemed necessary. It is only required that the new process is offered to WPS users.

4.1.2. Application Package

The Application OWS Context Document (OWC) should have a clear, minimum set of mandatory fields required for an application to be valid. This minimum set should than be validated and accepted on all clusters.

Each process should have a unique process identifier. The process identifier is provided within the Application Package (AP) in the <ows:Identifier> element, so it is not generated by the WPS instance usually. It would be possible to generate unique identifiers on the AP production side by using e.g., the SHA-256 message digest of the OWC Application context document. This would help to have a unique, traceable and interchangeable mapping between the OWC document and the process identifier.

OWS-Context vs. WPD vs. QaDJSON

Testbed-13 discussed three options to encode an implementation package:

  1. Using OWS-Context with embedded Web Processing Service ProcessDescriptor elements;

  2. Using Web Processing Service ProcessDescriptor (WPD) elements exclusively, i.e. avoid OWS-Context; and

  3. Using a Quick and Dirty JSON (QaDJSON) representation, either based on some JSON Schema or following conventions.

Then first approach uses OWS-Context to encode the AP. WPS ProcessDescriptor elements are embedded. This approach has the advantage that:

  • the developer can add any additional information to the AP that can be used by sophisticated clients to visualize e.g., background maps as part of the graphical consumer interaction