OGC Earth Observation Exploitation Platform Hackathon 2018 Engineering Report

It was generally agreed that it makes perfect sense to facade any type of application with a WPS. Though, mixed opinions have been stated on the actual location of the WPS, which could be a service offered by the ADES, a service that is available per MEP, or integral part of the Docker container itself (see microservice based approach). The WPS-based solutions all worked well and provided sufficient flexibility.

The Application OWS Context Document (OWC) should have a clear, minimum set of mandatory fields required for an application to be valid. This minimum set should than be validated and accepted on all clusters.

Each process should have a unique process identifier. The process identifier is provided within the Application Package (AP) in the <ows:Identifier> element, so it is not generated by the WPS instance usually. It would be possible to generate unique identifiers on the AP production side by using e.g., the SHA-256 message digest of the OWC Application context document. This would help to have a unique, traceable and interchangeable mapping between the OWC document and the process identifier.

OWS-Context vs. WPD vs. QaDJSON

Testbed-13 discussed three options to encode an implementation package:

Then first approach uses OWS-Context to encode the AP. WPS ProcessDescriptor elements are embedded. This approach has the advantage that:

On the disadvantage side, one can argue that OWS-Context adds additional complexity to the AP that is not absolutely necessary. It requires the AP developer to read and understand yet another standard without gaining much. After some discussion, this aspect has been solved by agreeing on two aspects. Either tools will be developed that guide the AP developer through the process. Think of a kind of wizard that requests essential elements from the AP developer and produces the AP as a result. In this case, the application developer never sees the actual AP code. Alternatively, application developers will follow examples of existing APs and realize that the OWS-Context add some elements, but do not cause any harm during AP development. In any case, we need to have a feedback channel for AP developers to ensure that the additional flexibility gained with OWS-Context comes at minimum costs for the AP developer.

An interesting alternative that was presented by defining a single Docker image loaded with major image processing software like GDAL, OrfeoToolBox and SNAP, instead of providing a link to a Docker image including a processing command. This image can then be reused for various processes, as described here. The approach would lead to modifications in the Application Package model and may lack from version incompatibilities of the used tools and dependent libraries, but is certainly worth further experimentation.

The currently favored architectural approach uses Application Package for application description and deployment; and WPS for process registration, deployment and execution. It has been developed based on the target to minimize extra work for the application developer. All the application developer needs to do is to run a simple command line command to package the application in a Docker container, to upload this container to a Docker hub and to provide the application package.

A very interesting alternative approach follows a different paradigm. In the microservice approach, the developer is familiar with WPS and provides everything required to execute the application in the cloud programmatically. Using IAAS/PAAS solutions such as Kubernetes or Amazon, the user produces a Docker container with a single process WPS embedded. This solution allows the application developer to take care of all data mountings and mappings following his own preferences. The Docker container could then be deployed on cloud platforms.

The approach was developed by several participants, e.g., by VITO as described in more detail in this chapter, or Space Applications. Potential disadvantages of that approach, including parallelization and resource scheduling issues as well as versioning issues, have been identified by Eurac.

It turned out that many OpenSearch based catalogs use different terminology for both request and response parameters. As an example, some catalogs require the request parameter "start", where others require it to be "startDate". The same applies to the responses, as illustrated in the following examples.

The following queries are requesting the same data. Note the different parameter names: VITO: http://www.vito-eodata.be/openSearch_all/findProducts?collection=urn:eop:VITO:CGS_S1_SLC_L1&start=2018-01-01

CloudFerro: http://finder.eocloud.eu/resto/api/collections/Sentinel1/search.atom?startDate=2018-01-01&_pretty=true

This issue can be mitigated by using URL templates as provided in the OpenSearch Description (OSD) documents. Given that only the name of the placeholders is standardized (such as searchTerms, geo:box, time:start, time:end), not the rest of the URL (which is completely free, including the base path and the name of the parameters), a link to an OSD document could be provided together with the list of key/values pairs for substituting the search placeholders. This link to would be provided instead of providing an OpenSearch request directly.

A major issue are inhomogeneous responses. It is currently impossible for a developer to produce a response module that could process responses from arbitrary catalogs. The response structure is always different, the response terminology is non-homogeneous, and the actual paths to the data differ from one cloud platform to the other. OWS-Context provides some mitigation options, as at least terminology mappings can be described, but standardization should address this issue in a more robust way by either developing a set of conventions, or even better to include commonly used terms in the specification directly. It cannot be expected that mediation and mapping technologies coming from the Semantic Web can solve this issue within the foreseeable future.

Network connectivity (firewalls, blocked ports) and CORS (Cross-Site Scripting protection) issues took the first half day to be resolved. In particular CORS needs to be resolved on a general level, as it affects all secured interactions between clients and servers and can be considered a general issue of Spatial Data Infrastructures. It is not specific to the technology or setup used in the Hackathon.

In terms of general security (which was not in focus of the Hackathon), initial tests have shown that it requires solid best practices to setup a new secure WPS instance as part of an application-in-the-cloud-deployment scenario. Currently, security settings and requirements are very heterogeneous, thus making it hard to implement secure solutions.

There is lots of dynamic in the cloud platform market at the moment. Tools that have been state of the art just a little while ago a now superseded by tools that are even simpler to use, more user friendly, better scalable, cheaper, more robust, or better integrated with other services. At the Hackathon, several of these technologies have been discussed. The following table gives a brief overview of these technologies. Discussing and comparing these technologies is not simple. There are lots of articles or blog posts and an endless amount of social chatter available describing and comparing these technologies. You find a lot of information, often infused with marketing jargon and consultation offers, on what some call the fight-to-the death for container supremacy whereas others claim that the various technologies often solve different things and are rooted in very different contexts. If there is one thing to learn, then certainly to be very careful before building any complex system that is strictly bound to one or a specific set of technologies, tools, or platforms.

When investigating favored technologies that are used in applied research projects or system developments, it seems that Docker is the common denominator. Docker containers running in production environments need to be orchestrated across multiple machines; and with the orchestration engines things divert. One of the first orchestration engines was Marathon on top of Apache Mesos. Now we do have Nomad, Kubernetes, or Docker Swarm, which is now part of Docker Engine.

The container technology Docker (i.e., the file format and runtime engine) was quickly complemented with additional technologies, such as Docker Hub, Docker Registry, Docker Cloud or Docker Datacenter. Docker Hub allows public storage of Docker images whereas the Docker Registry can be used for storing it on-premise. Docker Cloud is a managed service for building and running containers and the Docker Datacenter is a commercial offering embodying many Docker technologies. And quickly we are on the market. That’s where the jargon and market speak starts and true believers espousing their faith are burning heretics who would dare to consider alternatives.

Kubernetes is a technology introduced by Google. It allows to orchestrate Docker containers without having to interact with the underlying infrastructure. It provides a standard deployment interface and primitives for a consistent app deployment experience and APIs across clouds, which can be very useful in our context of deploying and executing applications working on Earth observation data stored on mission exploitation platforms. Kubernetes modular API allows to integrate systems around the core Kubernetes technology. Experiences with Kubernetes, that is now offered as a service by many vendors, show that it could be a viable solution for EO Exploitation Platforms. It would free the application developer from the underlying infrastructure, though it is not quite clear what effort would be required on the platform side to setup and run Kubernetes deployments.

An alternative for infrastructure abstraction is Apache Mesos.

52°North is involved in several projects dealing with processing of large amounts of Earth Observation data. For example, the research project WaCoDis aims to implement a geo-information infrastructure for river basin management monitoring tasks including water quality control, water protection and protection of access to clean water. For this purpose, remote sensing data from the Copernicus program, weather data and in-situ sensor data will be combined, merged and analyzed. Another example is the RIESGOS research project, which aims to implement multi-risk assessment from natural hazards by utilizing standardized service interfaces.

Furthermore, 52°North has long-time experience in defining and implementing Web-based Geoprocessing tools and is actively contributing to standardizing geoprocessing. Benjamin Pross is currently leading the Web Processing Service 2.0 SWG at OGC. The 52°North WPS implementation fully supports WPS 1.0.0 and 2.0 and is widely used. It has also been used in Testbed-13 Earth Observation Clouds thread.

Since 52°North provides a WPS implementation and is involved in different projects dealing with Earth Observation data, 52°North has a high interest in the activities for standardizing the processing of large amounts of Earth Observation data and wants to contribute with its experience in developing Web-based Geoprocessing systems.

52°North has implemented an Application Deployment and Execution Service based on the 52°North javaPS framework. The framework fully supports the WPS 2.0 interface. The means to deploy and undeploy processes were implemented as WPS process themselves. The process descriptions were aligned to the ones specified in the OGC Testbed-13: Application Deployment and Execution Service Engineering Report (OGC 17-024). Figure 5 shows the general sequence of communication between client and server. The requests are described in the following.

Based on the process descriptions, execute requests could be send to the ADES.

Based on the Application Package, that is send along as execute-input, a new process will be created and made available in the capabilities. The process summary will be returned in the execute response.

The complete process description looks like the following:

The process itself is a mockup, but sending an example execute request is possible:

A single example output TIFF is returned by reference.

The concept of the ADES works well with WPS and the 52°North WPS could be easily extended with the necessary functionality. WPS 2.0 processes are a good means for deploying Application Packages and creating new WPS processes. The input- and output-formats can be described in a standardized way and generic clients are able to execute the Deploy- and UndeployProcess. The 52°North ADES was used for testing by several client developers during the Hackathon. It supports the deployment and execution of new processes based on Application Packages. The back-end implementation, i.e. deploying the Docker container specified in the Application Package required more time than initially anticipated, so we had to create a mock-up for this functionality. In the OGC Testbed-13: Application Deployment and Execution Service Engineering Report two approaches for implementing an ADES are described. One approach is using WPS processes. This was implemented during the EOEP Hackathon. An alternative approach used a transactional extension for WPS 2.0. In the OGC Testbed-14 different approaches for enabling WPS with transactional functionality are further explored. The Testbed-14 WPS-T ER will specify how transactional behavior can be implemented for WPS using additional dedicated operations for deploying and undeploying processes. Also REST-based approaches will be explored. Findings from the EOEP Hackathon will be taken into account for discussing these different approaches and will be considered when developing novel approaches for flexible Web-based processing of EO data in 52°North’s ongoing research projects.

The contribution of 52°North was supported by the research project WaCoDis (co-funded by the German Federal Ministry of Transport and Digital Infrastructure, program mFund, contract: 19F2038D) and by the research project RIESGOS (co-funded by the German Ministry of Research and Education, program CLIENT-II, contract: 03G0876).

To meet the needs for storage, processing and distribution of products to customers, CS is developing the GeoStorm platform. As part of a Pathfinder project developed for ESA for several years, CS has already integrated some image processing. GeoStorm is developed in accordance with good practices in particular regarding interoperability. We are therefore very attentive to the respect of the OGC standards into our developments.

During this Hackathon, we have implemented both client and server. Our client is the GeoStorm web application which provides rich interface to select processing parameters. Implemented WPS 1.0 services are DeployProcess, UndeployProcess and ExecuteProcess.

We didn’t choose the Kubernetes approach as we did on others similar projects because of our preoccupation to deploy our solution on architectures like HPC. We therefore have chosen to implement a docker like solution that is not requiring user’s privilege increase and allow the image to be used for several tasks. We were interesting in verifying that this implementation meets the requirements of the Application Package and can apply to both new cloud solutions and the traditional ones.

Here is an example of the output mosaic with some processed tiles on the study area displayed into GeoStorm platform:

We are striving after development of interoperable research independent of cloud processing facilities to foster exchange and repeatability. In particular the Institute for Earth Observation at EURAC Research has the need to be able to run transparently applications on different clusters, where data and computing resources are available. The proposed Hackathon was perfectly in line with what we currently try to achieve in the in the H2020 openEO project ( http://openeo.eu ), where we are already developing a driver to connect the openEO API with the OGC WC(P)S 2.0 and it processing extension. So participating in this Hackathon gave as the possibility to implement and test web processing services based on processing chains organized in containers and link it to the RESTful services that we are currently defining and developing in the openEO project.

The solution implemented by Eurac Research consists of an ADES server that exposes a set of operations as described in the OGC Testbeds, in form of a set of WPS 1.0.0 compliant operation offerings. Further an interface with the openEO API has been started to be implemented to access those web process services in a RESTful way and render it interoperable with the client API’s in python and R that are currently in development.

The reasons to use WPS 1.0.0 for the implementation are:

The operations actually implemented as a proof of concept and working prototype are the following:

Those processes are also discoverable via openEO using the /processes endpoint of the RESTful API.

The GetCapabilities result for the service (wpseoproc) implemented:

The implementation uses the proposed OWS Context Document (OWC) to describe and deploy an application. This seems a very good standard and convenient way to convey and validate the application capabilities on a given cluster.

The OWC document is passed as parameter to the WPS DeployProcess operation offering.

The ADES server, apart exposing the listed WPS services, will also perform the following tasks.

An alternative solution, would be to use the proposed micro-services within a container. In this way each application exposes the WPS Execute operation itself. On one side this solution would simplify the interaction with the container, but on the other side this would introduce some pitfalls for the application development and also for the ADES applications that control the container.

Some of possible pitfalls:

The overall experience with the implementation based on the Testbeds are good.

The Application OWS Context Document (OWC) should have a clear, minimum set of mandatory fields required for an application to be valid. This minimum set should than be validated and accepted on all clusters.

The process UUID generated should actually be the SHA-256 message digest of the OWC Application context document. This would help to have a unique, trackable and interchangeable mapping between the OWC document and the process identifier.

Solenix participated in the EOEP Hackathon with two main objectives.

The first objective was to demonstrate the validity and interoperability of its contribution to OGC Testbed-13 - in particular the Application Management Client (AMC) deliverable - and in general of the OGC Testbed-13 results. Two results in particular were relevant both for the Testbed activities and for the Hackathon: the Application Package encoding as a OWS Context document; and the use of WPS for the interface between the AMC and the Application Deployment and Execution Service (ADES).

The second objective was to bridge between Testbed-13 and Testbed-14, even if Solenix committed to participating in the Hackathon without knowing if it would be involved in Testbed-14. Indeed, several topics addressed in the Hackathon were also addressed in Testbed-13 and will be relevant and revisited in Testbed-14. Given that, during the Hackathon, Solenix’s participation in Testbed-14 was confirmed, the Hackathon provided a smooth flow between the two initiatives and a jump-start in Testbed-14.

Solenix brought the two following assets to the Hackathon:

Solenix’s AMC supports the Application Package (AP) in OWS Context format and implements generic WPS client functionality, which allows a user to browse, register and trigger execution of processes on several instances of the ADES. It also implements the basic OpenSearch catalogue flow integrated in the WebWorldWind 3D globe using FEDEO as endpoint. This implementation is very simplified, both in terms of the UI and in terms of functionality, since only the second step of the two-step search is implemented (i.e., the actual search for products. The collection discovery is bypassed and only well-known collection endpoints are used). It should be noted, in any case, that this functionality was not envisaged to be necessary for the Hackathon and in fact it was not required.

The examples of Application Package in OWS Context format are available at https://github.com/opengeospatial/EOEPHackathon2018/tree/master/AP .

The AMC software was deployed on a Virtual Machine hosted on CloudSigma’s Frankfurt data center, taking advantage of that Cloud Provider’s sponsoring of the Hackathon.

The AMC was then minimally tailored for the Hackathon, with no significant functional changes. The set of endpoints made available by participants providing ADES implementations was configured and a rectangle corresponding to the Canadian Northwestern Territories was determined (based on a Shape file provided by ESA/NRCAN) and drawn over the WebWorldWind 3D globe.

No other changes were made or required to be able to interoperate with available and working ADES providers, which is a demonstration of the validity of the Testbed-13 EOC results and the Solenix AMC implementation. In fact, in all attempted cases, once network connectivity (firewalls, blocked ports) and CORS (Cross-Site Scripting protection) issues were solved, the AMC managed to interact with the ADES implementations using the WPS functionality coming out of Testbed-13. This further attested to the maturity and interoperability power of WPS.

One of the AP examples mentioned above was modified as required for the Hackathon, several times during the final event, in agreement with other Participants. The AP used for the final demo is available at https://github.com/opengeospatial/EOEPHackathon2018/blob/master/AP/hackathon-ap.xml . It is reproduced here for completeness.

Besides the basic administrative information about the application (Burn Scar Detection), the following features are worth highlighting.

During the two days of the final demo event, it was possible to attempt integration with a few ADES implementations, with varying degrees of success. Using the AP above or slight variations, it is possible to register the AP after choosing one of the available ADES, using the following page:

On uploading the AP, the AMC prints-out some general information about the AP and asks for a confirmation. Once this is given, under the hood the AMC contacts the ADES using the WPS Execute operation on a specially prepared process called 'DeployProcess.'

After the application is registered, it can be selected for usage. Application execution can be triggered after filling-in the fields of the dynamically generated form built from the WPS Process Description returned by the ADES as a DescribeProcess response. The Figure below shows this for an earlier version of the Burn Scar Detection Application Package which considered a single time window of interest field instead of two separate fields, one for the start time and one for the stop time:

Besides the 52 North ADES, with which partial integration was successfully implemented before the final event (it was not possible to integrate the complete execution flow), it was possible to fully integrate and demonstrate the complete flow using the University of Timisoara’s ADES implementation during the final event.

For what concerns the Thales implementation, by the end of the final event there were still CORS and connectivity issues due to the fact that they were using the Boreal Cloud made available by NRCan, which is not easily accessible through the Internet. All parties agreed and were confident, however, that had these issues been addressed, integration would have been possible, since both the Solenix AMC and the Thales ADES implement standard WPS.

As can be seen in the Figure above, the University of Timisoara actually exposed two endpoints, one called ADES for the registration/unregistration of Application Packages and another one called WPS for the actual execution of processes. Even if this was not the original intention and none of the Testbed-13 ADES implementations or other Hackathon implementations did this, it does not pose any negative consequences and in fact it was agreed to keep it such to highlight that it is also a valid approach. It can actually be argued that this is a desirable split between two kinds of function which are fundamentally different and have different access requirements (the application registration/unregistration, for privileged users; and the application execution, for regular users).

It was mentioned during the final event that from the Testbed-13 ERs it was not evident that the two functions were supposed to be provided by a single endpoint, which is a point of improvement for future Tested ERs.

Finally, it should be mentioned that integration with EURAC’s backend was also attempted, but this was not possible for two main reasons: the fact that EURAC’s backend did not send appropriate headers and so the browser’s CORS protections did not allow the requests to complete (requires adequate server configuration to work); and the fact that also EURAC, similarly to NRCan with the Boreal Cloud, has its own private infrastructure which is not easily accessible through the Internet.

To allow access, EURAC put in place a proxy and implemented simple HTTP Authentication, but all attempts to change the client to send appropriate authentication headers did not succeed. The EURAC backend responded with a valid GetCapabilities response, but the Process offering was always empty (which EURAC confirmed was the expected result when authentication fails).

Space Applications Services has been developing a platform named ASB (for Automated Service Builder) for several years now. It has started before the draft Exploitation Platform Open Architecture was published and thus also before the Testbed-13 took place.

ASB has a mechanism to dynamically deploy Docker container images and run processes pre-installed in these containers in cloud environments. The actual deployment of the containers is delegated to Marathon (which relies on Mesos for the selection of the target worker nodes) and each container runs its own lightweight WPS service.

During the Hackathon, we have implemented a client component that dynamically generates an Application Package using the information we have in database (including processes, inputs/outputs, datatypes), then interacts with the WPS interface of the ADES implementations to deploy the application and trigger the executions.

The key differences between the Marathon-based and the ADES-based implementations are:

During the Hackathon, the new client component has been tested against the ADES implemented by Thales Alenia Space and 52°North.

The ASB core component in charge of requesting the deployment of the process/application images and their execution is the Tasks Manager. As most of the other ASB core components, it is implemented in the Python scripting language. It uses the WPS client package provided by the open-source library OWSLib (https://github.com/geopython/OWSLib/) to communicate with WPS servers.

The generation of the Application Packages did not pose specific problems. The EP Application Package E.R. is detailed enough to understand how the applications metadata must be encoded as Atom feeds.

The integration of the client component with the ADES of Thales Alenia Space has required some adaptations on both sides. The main incompatibilities are listed hereafter.

Our participation to the EOEP Hackathon was motivated by our aim of further extending the WPS 2.0 server developed as part of the ESA funded EO4SEE Project.

Particularly we are interested in insuring interoperability with third-party WPS implementations and adherence to the recommendations originating from the Testbed activities.

Our solution involved introducing a series of new conventions inside the Application Package, conventions agreed upon on-site during the Hackathon. Particularly the conventions have been endorsed by Solenix, 52North and VITO. The details of the modified Application Package are described in section Experiences with AP & ADES.

The implemented solution demonstrated complete interaction between the AMC developed by Solenix and the EWPS Server provided by UVT and supported by IeAT. The solution demonstrated that interaction between the various components envisioned in Testbed-13 is feasible.

The demonstration presented at the Hackathon involved the execution of the complete application lifecycle:

It is worth mentioning that the triggered application execution successfully completed the proposed use-case scenario. Runtime information regarding the execution can be seen in Figure 10.

As part of the Hackathon we agreed together with the other participants (particularly Solenix, 52North, and VITO) an modified version of the application package

The application package assumed the introduction of an mapping section inside the owc:offering node, as depicted below.

The mapping node was aiming two different purposes:

The Hackathon was of great value for UVT/IeAT as we have not been involved in the Testbed activities. It allowed us to test interoperation with different clients (Solenix) and ADES implementation (Solenix).

Our main recommendation would be that the next Hackathon activities should have a dedicated session for decision making and agreeing upon common rules.

Also complete (end-to-end) practical demonstrations would be great.

VITO implemented a backend (ADES), exploring both the WPS and OpenEO approach. The backend is based on pyWPS as an additional interface to the OpenEO backend.

First implementation is on the PROBA-V MEP processing cluster where we have:

Second implementation on CloudFerro: - Kubernetes for deployment and parallelization - CloudFerro catalog and archive

The proof-of-concept implementation is based on the latest version of PyWPS (4.0.0). This version implements WPS 1.0.0, thus support for 2.0.0 is not available.

The following experiments have been conducted.

1. Local execution

2. Docker app run

Remarks

Solution 2 is my preferred solution, as the deployment step can be replaced by uploading the image to a Docker registry (which can still be a customized marketplace for remote sensing related images). Standardization will be limited to specifying the generic docker run process itself. Most existing WPS’s already support this type of custom process, without modifying the WPS itself. The spec of the generic process can also be made fairly foolproof, as there is no need to have a command line template, clients should just specify the command line directly.

Our participation to the EOEP Hackathon was aimed for positioning inside the earth observation with different solutions using real products from Sentinel 1, working with common tools as SNAP or WPS server development and providing our background knowledge in different tools and services. Particularly, we are interested in developing a server back-end fully integrated with a client and make a portable solution, which can be installed in a cloud totally accessible.

At this chapter, it will be described the final Thales Alenia Space implemented solution, after testing different solution (can be seen in the next chapter).

During the Hackathon, several developments and configurations were done. First of all, the cloud required had to be configured in order to upload the developments there. At the beginning, it was chosen CloudFerro, but after uploading our developments there and due a lack of resources in the cloud, during the Hackathon meeting we migrate all the developments to Boreal Cloud, where we can have the same size machines.

CloudFerro configuration:

As can be seen the machines were different, therefore we had a lack of performance in those machines, having as maximum of 15 workers (SNAP executions) executed at the same time. The total performance for the whole cluster was 120vCPU and 832GB RAM.

Boreal cloud configuration:

Using this last configuration, we got an improved configuration using 3 nodes less and the total performance for the whole cluster was empowered 160vCPU and 120GB RAM compared with the previous option. After migrate to this configuration, the maximum number of workers were 35, therefore having 140 products to process, the whole process last around 45 minutes, using less than four executions.

The products from sentinel and the DEM were mounted on the machines as mount point, therefore were easily accessible. In the first cloud (CloudFerro), those mounting points were unmount automatically after some time, therefore this connection was not reliable enough to automate the process.

In that cloud environment were needed just to install docker-ce and include all the hosts in the same cluster using docker swarm. Once the docker swarm was initiated and working, the docker images can be downloaded in order to execute all the processes, which makes the solution fully portable. Thales Alenia Space has also written a docker compose file in order to execute the process automatically. This docker compose also limited the number of memory and CPU for the execution, therefore if the SNAP process needs 8vCPU and 24GB RAM and we have a machine with 16vCPU and 52 GB RAM, when up the docker compose, will limit the SNAP executions to two. On other machines, it will be executed more if fit in the machine requirements.

This made the migration really easy, having to install just docker and introduce the hosts in the same cluster if we want to migrate the solution.

At high scale, the implemented solution had to be based on the WPS server (2.0) and SNAP docker image. First of all an application package was designed in order to Deploy, Execute and UnDeploy commands. This xml file called the ADES, which connected with the WPS launched the SNAP application inside the cluster.

To make this possible, we used a queue message service (ActiveMQ) reused and modified from different Thales Alenia Space products in order to queue messages from the WPS and dequeue in the nodes (listener). After dequeue the messages the process was executed in the nodes calling the SNAP application, which requires two inputs (eodata and DEM) and after processing an output was stored in a public accessible bucket in .tif format. The processing time for a single product using SNAP is around 13 minutes.

The outline of the Thales solution can be seen in Figure 11.

The final solution was composed of a SOAP API to connect the WPS client with the North52 WPS server and this connects with the queue message service, which sends the execution to the workers (nodes). Also, in order to test the solution launching all the products to process, we used Jmeter to automate the process in case we could not integrate with a real client. The final implemented solution can be seen in Figure 12.

During the period of the Hackathon we tried several options to make the backend works. First of all our ideas was to implement FaaS (Function as a Service) integrated with docker swarm, in order to manage the balancing between all the nodes when running SNAP application to process the products. This implemented solution can be seen in Figure 13.

After the trial, the balancing in the FaaS solution was not working properly, so the solution had to be changed and the alternative approach was using JaaS (Jobs as a Service), which is able to run docker images balancing the load between the whole docker swarm cluster. Using this solution, in an automated environment where the tasks were run automatically, in some of the nodes those executions were not properly run, and therefore this solution was not valid to test the performance.This implemented solution can be seen in Figure 14.

Discarding the previous two solutions, we go on for the solution done in implemented solution section, using a Thales Alenia Space designed solution reusing as WPS server, the server is done by North52.

The demonstration developed by Thales Alenia Space involved the execution of the complete lifecycle:

The WPS modified server, returns the public url where the product processed was placed when the process finish, which will be returned to the client in order to display the product.

Different constrains have been found during the Hackathon implementation.

The SNAP application has two major improvements which will give more value to the tool. In the first place, the application size should be reduced in order to work properly in a docker clustered environment, which easily can be done in the docker file. The other improvement is the based on the kind of application, which right now is monolithic, which means that it cannot be split among different workers along a cluster. With this approach, the processing time of the execution cannot be reduced, so it will be needed more computers or larger computers in order to run more than one SNAP application at the same time. Using an environment based on big data (spark), and adapting the tool, this time can be reduced in order to make calls in near real time and decreasing the processing time exponentially.

Based on the size and the computing requirements of SNAP, the cloud where the process is run should be big enough to be able to run the process. Most of the issues regarding the cloud were in terms of performance and accessing through the different open ports. Also the mount point in the cloud has to be reliable enough to allow the execution of SNAP as stated in implemented solution.

The Hackathon was valuable for Thales Alenia Space as we have not been involved in previous Testbed activities. It allowed us to develop and test the communication with clients and ADES and develop in a low time frame a fully functional solution based on the WPS standard and SNAP application built on cluster.

This Hackathon provides a business opportunity for stakeholders to experiment with the latest results from Testbed-13 in the context of ESA’s Earth Observation (EO) Exploitation Platforms. Participants will establish design decisions that influence the upcoming Testbed-14 and future work in the context of application handling in cloud environments operated by the sponsoring organizations. Participants will have the opportunity to network and establish direct relationships with other participants and with sponsors. They will also be part of the essential initial stages of the standardization process. During the Demonstration Meeting, participants will have the opportunity to visit the ESA Operations Centre in Darmstadt (Germany), including the Main Control Room.

OGC Principles of Conduct will govern all personal and public interactions in this initiative. The OGC Innovation Program Policies and Procedures apply together with the OGC Principles of Conduct and the OGC Intellectual Property Rights Policy.

One Hackathon objective is to support the OGC Standards Program in the development and publication of open standards. Each Participant will be required to allow OGC to copyright and publish documents based in whole or in part upon intellectual property contributed by the Participant during Hackathon performance. Specific requirements are described under the "Copyrights" clauses of the OGC Intellectual Property Rights Policy.

Each Participant shall contribute to the Hackathon Results Engineering Report.

This Hackathon seeks to deliver technical demonstrations in the form of software demonstrations at the Demonstration Workshop. Recording and publication of these demonstrations will be discussed at the Kick-off workshop. Software implementations remain with the participants and do not need to be made available after the Hackathon.

Experiences and lessons learned from the Hackathon will be captured in the Hackathon Results Engineering Report. This report will be edited by OGC staff with contributions by all participants.

There are no reporting duties during the Hackathon. Communication is reduced to the kick-off workshop, a Final Decision web conference, a Demonstration Preparation web conference, and the physical demonstration meeting. Please find further details here.

The roles generally played in any OCG Innovation Program initiative are defined in the OGC Innovation Program Policies and Procedures, including Sponsors, Bidders, Participants, Observers, and the Innovation Program Team ("IP Team").

Compared to other Innovation Program initiatives, the Hackathon provides maximal flexibility and minimal organizational and communication overhead on the participants.

This Annex provides background information on the OGC baseline, describes the Hackathon architecture, and identifies all requirements and corresponding work items.

The Hackathon builds on results from the recently concluded Testbed-13 initiative and paves the way for Tested-14 and subsequent initiatives in the context of deployment and execution of applications in cloud environments. The goal is to demonstrate that the Testbed-13 results, described in the Engineering Reports OGC Testbed-13: Exploitation Platform Application Package, OGC Testbed-13: Application Deployment and Execution Service, and OGC Testbed-13: Cloud are fit for purpose. Given that these Engineering Reports state several options, this Hackathon shall identify the best solution and identify any missing elements as basis for Testbed-14 and future initiatives.

ESA has started in 2014 the Earth Observation Exploitation Platforms initiative that created an ecosystem of interconnected Thematic Exploitation Platforms (TEP) for Earth Observation data. Testbed-13 focused on two key aspects:

The metadata describing an application in its Docker container is bundled in a so called Application Package. Details on this Application Package are described in the OGC Testbed-13: Exploitation Platform Application Package Engineering Report. An Application Package encapsulates the description of the application itself, i.e. the application metadata, a reference to the application software container, metadata about the container itself and its resource types, deployment, execution, and mapping instructions of external data to container-specific locations for input and result data, and auxiliary information such as Web-based catalogues for data discovery and selection.

The Application Package developer uploads the Docker container that includes the application to a Docker Hub and provides the Application Package to the TEP (Thematic Exploitation Platform). The TEP uses an Application Deployment and Execution Service (ADES) as described in the OGC Testbed-13: Application Deployment and Execution Service Engineering Report. This service, implemented as a WPS v2.0 profile, allows the dynamic deployment and execution of the Docker container on cloud infrastructure.

The Hackathon shall verify the specifications provided in the Engineering Reports, shall develop recommendations where the Engineering Reports describe several options, and shall detect any missing elements or defects that need to be corrected in future initiatives.

Participants can provide either a client application or a server application or both to the Hackathon. Alternatively, participants can provide alternative applications if these applications work on Sentinel-1 input data (or the input data can be made available by the participant). In the case of application provision, the participant needs to provide the application together with the corresponding application package. In case a participant provides both the client and the server application, then the participant agrees to pro-actively engage with others participants to ensure proper interoperability testing. At the demonstration meeting, we will test all server instances with all clients. Given that we will agree on the interface between client and server during the initial phase of the Hackathon, clients and servers can be developed independently of each other. A simple CURL client will be used as reference. Each server will receive the same two calls as described in the Hackathon Scenario below. Clients will be evaluated based on functionality and ease of use. Servers will be evaluated based on performance.

Scenario

The Hackathon shall implement the following scenario:

Design Decisions

The Hackathon participants need to agree on the following design decisions:

Implementations

The Hackathon participants shall demonstrate the following elements:

The winning team will receive the Hackathon Trophy and prizes. The winner will be selected by all teams being present at the Demonstration Workshop.