Earth System Grid Federation (ESGF) Compute Challenge

The Thematic Exploitation Platform (TEP) open architecture advanced in previous testbeds tackled Earth observation data in the context of federated, heterogeneous infrastructures. The main requirements included support for application deployment, execution and chaining, as well as the agreed authentication and authorization mechanisms. All interfaces needed to be agreed upon and standardized so as to achieve the Sponsors' (European Space Agency, Natural Resources Canada) interoperability goals.

It was observed by certain testbed participants, observers and stakeholders that the same set of requirements applies to climate data, processes and infrastructure. As such, the U.S. Department of Energy’s (DOE) Office of Biological and Environmental Research (BER) sponsored additional engineering work to ESGF interoperability. The general objectives proposed and presented to ESGF stakeholders are as follows, in order of importance:

Previous work in OGC testbeds allowed definition of application packages in workflow-enabled environments, and their subsequent deployment and execution on distributed secured infrastructures. Data discoverability was superficially addressed through inclusion of OpenSearch capabilities to identify and select the appropriate data from catalogs. Applications developed for the occasion included stacking, subsetting, feature detection and index computation, both on optical and Synthetic-Aperture Radar (SAR) data. All these concerns are of particular interest for OGC’s Earth Observation Exploitation Platform (EO Ex Platform) Domain Working Group (DWG).

The work described in this ER improves interoperability of workflows with pre-deployed WPS 1.0 and WPS 2.0 processes. It also enables the inclusion of pre-deployed processes adhering to the ESGF Compute Working Team (CWT) Application Programming Interface (API). The workflows proposed in this work include climate data (CMIP6, statistically downscaled climate scenarios, variables and indexes), climate processes (index computation) and base processes (subsetting, averaging). Three different infrastructures were used for interoperability experiments.

The last two objectives of the ESGF concept, the use of EO in support of climate infrastructures and the integration of ML systems, were only partially and informally addressed in this work. It is expected that work planned and conducted in OGC Testbed-15, as well as in other collaborative innovation projects, will continue to support these efforts and drive future requirements. This report contains relevant content potentially supporting OGC request for information (RFI) on Earth Observation Big Data Architecture.

This section presents recommended potential future tasks and deliverables that can support advancement of requirements and expand research motivations. The reader can also refer to Section 9 of this Engineering Report for discussion and open issues for further details.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

In addition to participants and observers in this testbed activity, the following institutions collaborated in the ESGF Compute Challenge and coordinated their efforts:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

The figure below is taken from the Testbed-14 ADES & EMS Results and Best Practices ER [1] and shows the architecture adopted. The main building blocks included the Client, the Execution Management Service (EMS), the FEDEO OpenSearch gateway and an Application Deployment and Execution Service (ADES). While OpenSearch, ADES and the Client play an important role in the EOC thread of Testbed-14, they are not really considered in the current report. This report focuses on an extension to the EMS allowing support of a large variety of applications types. The desired outcome is a backward compatibility with existing infrastructure, all while extending compatibility with process providers using their own interfaces.

Led by the Office of Biological and Environmental Research (BER) of the U.S. Department of Energy (DOE), ESGF is an international multi-agency federation that develops, deploys, and maintains software to facilitate advancements in geophysical science. ESGF’s open-source, operational code base disseminates model simulation, observational, and reanalysis data for research assessments and model validation via secure storage and dissemination of petabytes of data.

Addressing “big data” challenges in Earth system research, ESGF is a collaboration of numerous computer scientists, data scientists, and climate researchers. The federation houses an enormous database of global observational and simulation data — more than five petabytes — and manages the high-performance computing (HPC) hardware and software infrastructure necessary for scientific climate research. In the nearly two decades since its launch, ESGF has grown to serve 25,000 users on six continents [8]. More details can be found in ESGF Brochure or in Discussion for other supporting material and citations.

The ESGF Compute Challenge participants completed various experiments and integrations. The EMS implementation described in this report is based on components from both Birdhouse and PAVICS. The Solution described in this report focuses on providing access to deployed ESGF CWT API processes and providers in application packages and workflows. Future work includes exposing application packages through ESGF CWT API, either manually or dynamically. Below is a list of the various systems and frameworks involved in the Compute Challenge.

EDAS can be used as an analytics engine for the ESGF through STRATUS, an integrative framework developed at NASA NCCS. STRATUS presents a unified API and workflow orchestration for varied climate data analytic services. See Discussion section for more information on STRATUS.

As of early April, only AIMS and EDAS offered endpoints implementing the ESGF Compute Working Team API. Interoperability experiments conducted on these providers for this work are documented in Section 7 and Section 8, while the API is briefly described below.

The ESGF Compute Working Team (CWT) is working to improve interoperability and compute capabilities within the federation using Web services technology. The OGC WPS standard has been selected to ensure machine-to-machine interoperability. A reference server implementation of a compute node is offered. Certification datasets are listed in a GoogleDocs living document. The process requirements are defined on GitHub. These requirements are partially duplicated below for convenience.

The goal of the extension presented in this report is to improve the EMS in order to increase compatibility across existing systems. The EMS provided for Testbed-14 offers a WPS-T 2.0 interface, yet should be able to run multiple application types and use them in heterogeneous workflows. Using a single interface should allow execution of a large array of existing applications and, moreover, use them inside workflows. Two new application types are considered in this project. The first is for backward compatibility and consists of execution of existing process served by WPS 1.0 endpoint. The second is to broaden furthermore the application scope by covering the ESGF Compute Working Team (CWT) API.

Multiple approaches have been considered to achieve this goal. The first approach is to keep the EMS as is, package every type of application into Docker images, and provide alongside them a CWL describing invocation mechanisms. However, packaging existing providers would yield huge Docker images with multiple processes. This could simply prove impossible for external providers for which the code is unavailable. The second approach, much simpler, however defeats one of the principles of Testbed-14 which is to bring the application to the data. This approach consists of packaging all the information required to make a standard WPS 1.0 or ESGF CWT API request.

A DeployProcess document is still provided but two new profile names have been introduced, wpsApplication and ESGFWpsApplication, that require fewer elements as everything needed can be extracted from the existing endpoints. In the WPS 1.0, the CWL file can even be generated transparently at deployment time since the parameters mapping is trivial. The CWL file itself allows to specify execution requirements so that the engine can change the execution unit and perform a classic WPS 1.0 or ESGF CWT API execute request, without a prior deployment request.

As described in Section 5, in Testbed-14, CRIM’s EMS implementation was all contained in a component named Twitcher. This component is part of the Open Source software framework Birdhouse [9]. In the current solution, Twitcher is separated into two components so that each one can focus primarily on its own role and make the architecture simpler. The previous diagram resumes the situation. Twitcher now operates as regular proxy and a Security Enforcement Point (PEP), alongside the permission provider, Magpie. The WPS-T 2.0 REST interface and the CWL engine are moved to a new component named Weaver.

This subsection describes the various application profiles developed and introduces their integration into workflow. Subsetting and climate indices calculation are also introduced as key processes relevant to ESGF.

Currently, only a small portion of the ESGF WPS processing is implemented in Weaver. This is largely due to the fact that ESGF WPS inputs are nested, and this nested structure must be translated to a flat one to correspond to standard WPS inputs or to Weaver’s inputs. Some of these nested parameters are easy to implement because they are always the same across each process, but others cannot be automatically queried. A full example JSON file, containing all required inputs and outputs for the process, can be found in Annex D. Below is an excerpt of an execution body.

The files and variable parameters correspond to the ESGF CWT API variable input, as described in ESGF section. The api_key must be obtained by creating an account on the Lawrence Livermore National Laboratory (LLNL) website. The other inputs are a flat representation of the ESGF 'Domain' input. So far, the time and lat/lon attributes are present in every NetCDF file encountered so they are easy to implement and re-use for each process. The ESGF processes successfully tested are CDAT.aggregate and CDAT.subset, deployed on AIMS2 servers.

For the application chaining, the CWL engine is now able to process all application types only by instantiating the proper job type. To demonstrate that interoperability, two workflows have been produced and will be presented in this section. Beforehand, utility applications will be introduced.

There is a need to (re-)establish a common vocabulary for the software artefacts involved. For instance, the terms Application, Application Package, Process, Service and Endpoint lend themselves to be used interchangeably. The potential confusion is even more important in context of Workflows that can chain applications and execute deployed processes.

Current adoption of WPS 2.0 standard is still very limited. For instance, 52°North implementations such as javaPS are based on OGC Web Processing Service standard version 2.0; PyWPS server is currently only planning support for this version. In the near future, one can still expect to see new WPS 1.0 endpoints appear. By enabling support of pre-deployed WPS 1.0 services into a WPS 2.0 workflow environment, and onwards to WPS 3.0, the Solution presented in this report has the potential to facilitate a transition phase for legacy deployments. This relative ease to provide WPS 2.0 RESTful interfaces comes at the expense of the increased software complexity involved with an EMS/ADES implementation such as Weaver.

It has been shown with the previous workflows that interoperability between WPS 1.0 and ESGF CWT interface can be achieved with the Weaver component. If climate processes could have been available as a Docker application during the short project time frame, a workflow composed of all WPS 1.0, ESGF CWT and WPS-T 2.0 interface could have been tested also.

In the process of adapting the EMS to climate applications, it became clear that having a Docker image of an application is not a sufficient condition to consider it an Application Package. Docker images are often packaged with ease of installation and deployment in mind, but not runtime considerations such as execution. In PAVICS [13] and Birdhouse [9], climate processes are packaged as Docker images, alongside a WPS server. These processes are intended to be run as a service, so that requests are made to it and result are fetched from it. The Docker container is deployed and then is always running. To convert this service into an Application Package as described in [2], the processes would need to be packaged without any web server, database or file server. The Docker image must define a run command that map inputs directly to the process function. The result must then be extracted from the container after it has been run. It is a completely different paradigm that require careful consideration to define clear entry points to the executable code.

In CWL, tools and workflows can take File types as input and produce them as output. This specification documents the use the tool and allows simple type-checking when creating parameter files. It is possible to reference existing ontologies, like EDAM, or reference a local ontology. Additional investigation for creation of such an ontology for EO and climate is required, and recommended as future work in the summary.

One recommendation from OGC Testbed-14: ADES & EMS Results and Best Practices Engineering Report [1] is to use Metalinks for multiple outputs. During the ESGF Compute Challenge, Ouranos and DKRZ produced an implementation that adds MetaFile and MetaLink classes, code that has been merged in PyWPS. The intent is to improve user experience with respect to WPS outputs storing multiple files under the same identifier. For example, when computing an indicator on an ensemble of climate simulations, the output is going to be a list of files. At the moment, a usual way to send that back to the user is to either zip these files and provide the reference to the zip file, or create a txt file storing the list of references.

In this implementation, a process developer declares an output as a FORMATS.METALINK, instantiates a MetaLink object and fills it with all the output files and their metadata (name, description). The server returns an XML file matching the metalink schema. Metalink files are recognized and individual files downloaded and converted into objects. The MetaLink format opens up interesting possibilities, including distributing files as torrents or use of a checksum. Additionally, impacts of Metalink support in workflow environments is to be investigated.

NASA Stratus Framework provides a set of standards and APIs to facilitate the integration of disparate analytic services into unified workflows with a common interface. The Stratus framework has been partially implemented at NCCS and a limited deployment is in the initial stages of testing and security review. As the framework addresses several challenges of EO platforms, it is presented to the reader of this report as a solution potentially complementing ESA Thematic Exploitation Platform (TEP) open architecture. Below are STRATUS primary requirements, as listed in its white paper:

In Testbed-15 Machine Learning tasks, there is a specification to “continue the work of cloud computing from Testbed 13, 14 with work on WPS and possibly CWL (Common Workflow Language), and continue the LiDAR best practices work from Testbed 14”. This applies for these deliverables:

It is to be noted that work documented in OGC Testbed-14: Machine Learning Engineering Report [4] presents a Dockerized, self-contained application package of PyTorch, including helper libraries. The work presented in the current report is therefore seen as potentially complementary to the ML common architecture to be defined in the Machine Learning thread.

In this work, a CWL descriptor such as the one in Annex B is generated automatically using DescribeProcess on a pre-deployed WPS 1.0 service. The process description is expected to explicitly define all inputs and outputs. In the case of ESGF CWT API compliant process, DescribeProcess effectively lists Domain, Variable and Operation inputs, as specified in the API and described in Section 6. The multiple values of these parameters have to be passed separately in a JSON file. As such, they values are not automatically available for the CWL generation. A CWL file has to be manually edited to explicitly add and map the all inputs. An example can be seen in Annex C. To remediate this, a specific set of inputs, or profile, could be defined as a shared interface between deployed ESGF processes. Additionally, the EMS could implement special logic to parse the multiple values of an ESGF CWT API compliant service.

The following publications offer a thorough view on various challenges and requirements of ESGF: