OGC Testbed-13: NA001 Climate Data Accessibility for Adaptation Planning

This client visualizes the result and the process steps of NA104 WCS and the NA103 Prediction WPS. For the agricultural scientist client scenario of the OGC Testbed-13 project, the agricultural researchers considering the effects of climate on crop production require access to climate and prediction data for ingest into crop forecasting models, potentially including temporal and spatial sub-sets of climate model variables relevant to rain and soil conditions. In this case, the focus is on providing the data access of NASA MERRA-2, exposed through the NA104 WCS 2.0, moreover, the client also supports the integration of on-demand crop yield prediction outputs from the NA103 Prediction WPS running the APSim crop model for the expert user. This client will greatly help researchers to access the correct datasets, with an effective API, and Geographic Information Systems (GIS) data formats including GeoTIFF and Geography Markup Language (GML).

The agricultural scientist client developed is based on the GMU Geobrain framework (http://www3.csiss.gmu.edu/OnAS/) , the client portal is located here: http://www3.csiss.gmu.edu/OnAS/

The area of interest in the scenario (described later) is the San Francisco bay area.

The client supports the following functions:

Support WCS MERRA-2 data as the crop yield prediction model input:

Support APSim WPS service as the crop yield prediction model

The expert user could interact with the WPS through the Agricultural Scientist Client as it provides the ability to produce predictions based upon finely grained parameterization of the service.

Visualization of the MERRA-2 data in the Modeling thread is accomplished through utilization of Portal for ArcGIS' (http://server.arcgis.com/en/portal/) and GeoServer (http://docs.geoserver.org). WCS is simply imported into GIS desktop software such as QGIS or ArcMap and the provided properties can then be used in standard analysis. However, visualizing WCS services through Esri Portal is not straightforward, as is the case with WMS services where images (or maps) are simply "displayed". Portal for ArcGIS does not include methods to include WCS services and GeoServer 2.9.1 does not have cascading service capability for external provided WCS services. A workaround is to display the NASA data directly in its native raster format, GeoTIFF or NetCDF, or use the WMS endpoint for each MERRA data coverage. Visualization of WCS data was tested using two methods:

An Esri web map has been setup in Portal for ArcGIS for testing purposes, “Testbed-13 Non-scientist” displays both the WMS’s mentioned above (see also Figure 7, where daily max precipitation is displayed for 1 day): https://diluvium.colorado.edu/arcgis/home/webmap/viewer.html?webmap=0856d14f047b4db3aca9f045241aaa62

Both the cascading and the direct incorporation of the WMS endpoint have the disadvantage of only showing a single time-slice of the input data. Input data is currently not hosted with WMTS capability, therefore no time dimension visualization is available.

The second use case for the non-scientist client is to have the user select a simple pre-configured APSim model parameterization file that can then be submitted as a WPS service that executes the model and returns the APSim model output as a point GML, the methodology for this is described in the following sub-sections.

The latest stable version of QGIS (version 2.18.13 http://qgis.org/en/site/) is used to test the connection and execution the non-scientist APSim WPS process. The plugin 'WPS Client' needs to be installed to enable WPS client connection capability, and can be installed using the standard QGIS plugin repository. Once installed, the plugin can be activated or deactivate under the 'web' tab, by selecting 'WPS-client'. All that needs to be done to connect to a WPS service, is to provide the URL, in this case: http://54.149.145.158/wps/WebProcessingService. The 'non-scientist client' was selected from all available WPS services that will display. Once a specific WPS service is selected a WPS input parameters window will appear (Figure 8).

The output format can be chosen, after populating all required input parameters, following the provided standard input parameters from the WPS test client URL. Given output formats from the WPS are currently as shapefile or as text, the shapefile option is suitable for GIS software and the text output provides a table with APSim yields for given dates.

After executing the WPS service, the status of running the WPS is provided, in this case 'Execution of APSim Non Scientist finished successfully', and the output is returned as shapefile and automatically read in and displayed by QGIS. Results of the APSim simulations can be viewed by opening the attribute table of the shapefile (Figure 9).

The WPS shapefile can be saved locally. Using the 'time manager' plugin of QGIS, the changing yield values of rice can be displayed as for example changes in size of the point shapefile.

The FME client is configured using an FME data transfer model called a workspace that submits requests and consumes responses from the APSIM WPS. This configuration has a choice to select either the scientist or non-scientist access mode. The scientist option employs the full parameter query and the non-scientist mode employs the abbreviated, non-expert query. The user also specifies the location in degrees for which APSim results are required. In both cases, when the client receives the WPS response, it parses the GML contained into features. See Figure 10 and Figure 11 from the reader workspace and Figure 12 showing the client display.

One of the challenges with consuming the response GML was that it was difficult to parse. Every time a query is submitted, the feature member container name changes. The typical feature container in GML is just <featureMember>. In this case, the WPS response after one query had a feature container of: <n52:Feature-fa8574c1-6bb8-4c21-84e0-e50cda71eb2a gml:id="ID21">. Then the next time the query was run it became: <n52:Feature-02cac30c-f5ac-44a0-9d2d-cbac72adf5b2 gml:id="ID21">. So the feature container object had a unique id embedded in the tag. This may not be fully GML compliant because the application schema (xsd) would have to change with every result in order to read this by the schema. However, FME has an 'ignore schema' mode in the GML reader so this was used with the query 'featureMembers/*'. This is essentially an XQuery that extracts all children of 'featureMembers'.

The other challenge was the time delay involved with using the non-scientist access mode. As discussed in more detail under the NA103 and NA104 sections, the non-scientist service mode involves longer run times on the server side due to the amount of data that is downloaded by the server to support each request. Also, the request process times appear to vary significantly, perhaps partially depending on the number of competing requests. Initially, access using the FME non-scientist client resulted in time-out errors. Increasing the timeout tolerance on the client resolved this, but testing is limited as every request / response iteration may involve a delay of several minutes.

To make this output available in a more accessible format, after GML feature extraction, the FME WPS client workspace was then used to generate a time step OGC KML file with a georeferenced, buffered circle where the circle is colored according to a gradient from blue = low to red = high. The buffer amount is also set to the value of field1 in meters + 1000 - in the case shown below this is the value of T2MMAX. This enables the functionality to hit play on the time slider and watch the values of T2MMAX change over time. This was then visualized in NASA’s WorldWind software - see Figure 13 below.

A custom WCS reader was implemented in FME to query the WCS that the WPS depends on directly. The goal was to test sub-setting the NetCDF time series from the WCS client to minimize the data required for transmission by defining the minimum x,y and t range required for the analysis. This sub-setting appeared to offer more responsive performance. For example, to retrieve a 4x4 grid subset of NetCDF data only took 2.5 seconds. This sub-setting approach should be considered in future testbeds in order to reduce the transmission and processing footprint for WCS client requests and downstream processing from any dependent WPS’s. Below are screenshots of the FME WCS custom reader workpspace, parameters and output as displayed in Data Inspector.

This is a sample WCS request generated by the above FME WCS client to request a subset of the MERRA-2 data: http://testbed-13.opendap.org:8080/WCS-2.0?service=WCS&version=2.0.1&request=GetCoverage&CoverageId=MERRA2_100.statD_2d_slv_Nx.19800101.SUB.nc4&format=netcdf&subset=latitude(37.2,40.2)&subset=longitude(-123.1,-120.1)&rangesubset=T2MMAX&format=netcdf

Note that the custom client evaluated did involve configuration setup of transformers within the FME workspace which define the data transformation workflow. The client requests are implemented by feeding values from published parameters into XML templates of WPS or WCS requests in the XMLTemplater transformer. These requests are then sent to the WPS or WCS server using HTTPCaller. The response is then parsed into features using FeatureReader or ImageFetcher depending on the response type. Although some configuration was needed, no coding was required, and the flexibility of this approach means that the client can easily be customized to generate a range of query types including spatial and temporal ranges, with results that can be readily transformed to meet a variety of output or external service requirements.

The Testbed-13 RFP requires: a "component accessible by Web Processing Service (WPS) will enable access and control of predictive models relevant to drought and agriculture prediction based on climate and other data."

This requirement has been clarified to mean a flexible, configurable crop model with academic credentials that is suitable for augmenting with external data. APsim (Agricultural Production Systems Simulator) is the crop model of choice as it is:

APSim is modular and is able to be extended to support different aspects of crop yield prediction, for example prediction of grape crop yield requires metrics on water retention for grape species, which is implemented as a module in the APSim and requires a large set of parameters in addition to the existing set. For Testbed-13 the base APSim model is used to demonstrate execution via WPS and interoperability with the WCS (work item NA104) provided by OPeNDAP.

The APSim crop model is usually accessed by a GUI interface provided with the software, it also has options for execution on a cluster for high performance computing, presumably for complex modeling. Behind the scenes, the crop model is configured using an XML (.apsim) file to store all of the values to execute and return results. This use of XML lends itself to external configuration without the use of the UI. The .apsim file also provides paths to other files for use in the simulation. The most important of these is the meteorological (Met) file that provides daily measures to simulate crop growth including:

The variables in the Met file can be augmented by extractions from the MERRA-2 data provided by the OPeNDAP WCS.

Modern Era Retrospective analysis for Research and Applications version 2 (MERRA-2) is a product released by NASA for climatic research and development projects. The datasets provide 1 degree squared rasters of the world with the following variables encoded:

Additionally, the data are available at different frequencies ranging from hourly to monthly, with the low frequency data derived from the high frequency data. As the APSim model creates a meteorological file, the data provided can be incorporated into the APSim model by updating the values in the meteorological file using MERRA-2 data. The lookups between the required data and the MERRA-2 data are as follows, please note that not all of the required variables within the crop model have a suitable corresponding variable in the MERRA-2 data, also, unit conversion is required in the available MERRA-2 data (for example Kelvin to Celsius):

Figure 17 outlines the use case diagram for the APSim WPS, which forms the software component to the Prediction WPS requirement outlined in the RFQ. There are three main actors in the use case diagram, these conform to the systems that will interact with the WPS. The Expert User interacts with the system via the Agricultural Scientist Client (NA102) and has the ability to produce predictions based upon finely grained parameterisation of the WPS. Essentially, all of the functionality of APSim is exposed to the user and can be manually configured via direct input into the client, or by uploading a series of files that are required by the crop model to run correctly. There is very little error catching available, as the user guidance is minimal beyond the process descriptions. Additional, shared functionality is included to enable the user to present the system with a latitude and longitude to produce climate data for a location over a given date range as this maybe an easier method of determining input climate data.

The second actor is the Non-Agricultural Scientist, that is, an individual or team who wants to run crop simulations using climate data, but may not have the expertise to produce the required files for the crop model. This actor is reliant on the WPS to configure the model using the information provided by the client at a high level. For example, the user may not have the knowledge to fully create a crop model, but they are likely to understand parameters such as the location, crop and date range of the predictions. Therefore, the WPS will attempt to take these parameters from the user and infer the required parameters, either using external sources of data or internal logic. These predictions are unlikely to be as robust as those for the agricultural scientist, but will still offer value to the users.

The third actor is the WCS (NA104) holding NASA data. This acts as a temporally enabled data repository for NASA data. The Prediction WPS makes a call to this service to query point values for temperature, precipitation and irradiance variables. Other data, such as soil water at different depths may be made available through this service, although this is marked as an extension to the current functionality.

The workflow for the WPS can be found in Figure 18. Essentially, the process workflow is determined according to what process is executed in the WPS. The scientist process is simplistic in that it translates the input variables into an APSim execution document, executes the process and then returns the result. The non-scientist process executes using few parameters and makes decisions on the type of template APSim file to use from the crop keyword (a restricted WPS string input). The meteorological file is specified in the scientist client, whereas the most suitable file is identified in the non-scientist client and then augmented with MERRA-2 data for a provided date range. Note that the non-scientist process contains two separate date ranges, one for the prediction and one for MERRA-2 meteorological data augmentation. This is for testing purposes to manage the responses of the WCS and avoid client time out, an improvement for future iterations is to produce a set of asynchronous processes.

The non-scientist process is designed to be used with the non-scientist client. Unlike the agricultural scientist process, it has few variables and gathers much of the required data from geospatial elements such as where the site is located. This is meant as a demonstrator of what is possible to automate, however there are shortcomings with the approach, which will be described later in the document.

The full input parameters are described here:

The output parameters are as follows: