Testbed-12 Implementing Asynchronous Services Response Engineering Report

Publication Date: 2017-06-30

Approval Date: 2017-06-29

Posted Date: 2016-12-08

Reference number of this document: OGC 16-023r3

Reference URL for this document: http://www.opengis.net/doc/PER/t12-A067

Category: Public Engineering Report

Editor: Benjamin Pross

Title: Testbed-12 Implementing Asynchronous Services Response Engineering Report

Testbed-12 Implementing Asynchronous Services Response Engineering Report (OGC 16-023r3)

COPYRIGHT

WARNING

This document is an OGC Public Engineering Report created as a deliverable of an initiative from the OGC Innovation Program (formerly OGC Interoperability Program). It is not an OGC standard and not an official position of the OGC membership.It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.

LICENSE AGREEMENT

Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.

If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.

THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.

This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.

Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.

This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.

None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.

Table of Contents

Abstract

Most of current OGC specifications define synchronous communication patterns, i.e. after sending a request to an OGC service, clients need to wait for the response. But several applications, e.g. delivery of information about events or executing complex environmental models with long runtime, need asynchronous client-server interaction pattern that do not require clients to keep the connection to the server continuously open in order to wait for responses. At the moment, there are several approaches how to add asynchronous communication to existing OGC services: One option is to use a WPS façade, as the WPS specification already defines asynchronous service responses. Another option is to add extensions to the different specifications and the third option is developed by the OGC Publish-Subscribe Working Group. This ER summarizes and compares the results from the different activities for asynchronous service responses and provides recommendations for future activities.

Business Value

This benefit of this ER will be recommendations for handling asynchronous communication for different OGC Web services. Based on the findings, a common baseline for asynchronous service communication could be derived.

Technology Value

The described approach for using WPS to enable synchronous services with asynchronous functionality is a new use case for web-based processing.

Keywords

ogcdocs, testbed-12, SOAP, asynchronous

Proposed OGC Working Group for Review and Approval

WPS 2.0 SWG, WFS/FES SWG, PubSub SWG

1. Introduction

1.1. Scope

Most of current OGC specifications define synchronous communication patterns, i.e. after sending a request to an OGC service, clients need to wait for the response. But several applications, e.g. delivery of information about events or executing complex environmental models with long runtime, need asynchronous client-server interaction pattern that do not require clients to keep the connection to the server continuously open in order to wait for responses. At the moment, there are several approaches how to add asynchronous communication to existing OGC services: One option is to use a WPS facade, as the WPS specification already defines asynchronous service responses. Another option is to add extensions to the different specifications. The extension specifically addressed by this ER was for WFS. The third option was developed by the OGC Publish-Subscribe Working Group. An example of using the Pub/Sub concepts with a Catalog Service is described in the Testbed-12 PubSub / Catalog ER [OGC 16-137]. The three methods are discussed in section 8. The goal of this ER is to summarize and compare the results from using a WPS facade and an extension for WFS for asynchronous service responses and to provide recommendations for future activities.

1.2. Document contributor contact points

All questions regarding this document should be directed to the editor or the contributors:

Table 1. Contacts
Name	Organization
Benjamin Pross	52°North GmbH
Peter Vretanos	Cubewerx

1.3. Future Work

See section 9.

1.4. Foreword

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

2. References

The following documents are referenced in this document. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. For undated references, the latest edition of the normative document referred to applies.

OGC 06-121r9, OGC® Web Services Common Standard
OGC 09-025r2 OGC® Web Feature Service 2.0 Interface Standard
OGC 14-065 OGC® WPS 2.0 Interface Standard
OGC 16-137 OGC® Testbed-12 PubSub / Catalog ER

3. Terms and definitions

For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard [OGC 06-121r9] shall apply. In addition, the following terms and definitions apply.

3.1. Facade

Software design pattern. The goal is to simplify access to underlying complex functionality.

4. Conventions

4.1. Abbreviated terms

Async Asynchronous
OWS OGC Web Service
SOAP Simple Object Access Protocol
Sync Synchronous
WCS Web Coverage Service
WFS Web Feature Service
WPS Web Processing Service

5. Overview

This ER describes two approaches to enable synchronous OGC service with asynchronous capabilities. First, a WPS facade approach is described and tested against WFS and WCS services. Second, an approach is presented, to enable WFS with asynchronous capabilities using additional parameters in the request.

First, some background information is given, followed by a description of the different asynchronous service implementations. Finally, recommendations for future work are given.

6. Background

6.1. Asynchronous Request Pattern

Three basic request-response patterns can be identified. (1) Synchronous requests:

Figure 1. Synchronous request sequence

The client has to keep the connection alive until the service has finished its work and is ready to return the response. If the connection is interrupted (e.g. when using a mobile device), also the result of the request is lost, as the client has no information about where to re-connect to get the specific result that was generated by the previous request.

(2) Asynchronous requests using a pull mechanism:

Figure 2. Asynchronous request sequence - pull

After the client sends the asynchronous request, the service immediately responds. The response must contain information about where the client can request status-updates and finally the result. As the response is sent immediately and is small, this pattern is less error-prone. The service uses its own infrastructure to store the status and result (using a distributed infrastructure would be possible, but this would introduce a source for errors). The client knows the location of the result and is able to get it at any time. In a mobile environment, the client could therefore wait for a moment of stable/strong connection.

(3) Asynchronous requests using a push mechanism:

Figure 3. Asynchronous request sequence - push

Following the push-based pattern, the client has to provide the service with information about an endpoint, that the service can connect to and push status-updates and finally the result. The client doesn’t need a direct response from the service at all, which could be beneficial in mobile environments. On the downside, the service has to actively connect to an endpoint, which introduces a source for errors. If the endpoint cannot be reached, the service has no means of informing the client. If the endpoint for pushing status/result is not tightly coupled to the client, the client has to pull the endpoint.

6.1.1. WPS facade

As the Web Processing Service interface standard specifies a pull-based mechanism for asynchronous execution, the WPS could be used as a facade in front of a service that offers no asynchronous request mechanism. The following image shows the sequence diagram of a basic facade approach:

Figure 4. WPS facade sequence

The task of keeping the connection alive is moved from the client to the WPS, which presumably resides in a well connected environment in contrast to e.g. a mobile client. Still, when requesting large amounts of data, the connection between WPS and data service could reach timeout-limits.

7. Implementations

7.1. WPS Facade

The WPS facade approach was tested with a SOAP WFS provided by the Arizona State University and a SOAP WCS provided by George Mason University/Jacobs University. Also, a secured SOAP WFS was accessed that was provided by Secure Dimensions/CubeWerx. The process description is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<wps:ProcessOfferings xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/2.0" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
  <wps:ProcessOffering processVersion="1.0.0" jobControlOptions="sync-execute async-execute" outputTransmission="value reference">
    <wps:Process>
      <ows:Title>testbed12.cmd.AsyncFacadeProcess</ows:Title>
      <ows:Abstract>Process acting as async facade for a SOAP endpoint</ows:Abstract>
      <ows:Identifier>testbed12.cmd.AsyncFacadeProcess</ows:Identifier>
      <wps:Input minOccurs="1" maxOccurs="1">
        <ows:Title>request</ows:Title>
        <ows:Identifier>request</ows:Identifier>
        <ns:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
          <ns:Format default="true" mimeType="application/soap+xml"/>
          <ns:Format mimeType="text/xml"/>
        </ns:ComplexData>
      </wps:Input>
      <wps:Input minOccurs="1" maxOccurs="1">
        <ows:Title>endpoint-url</ows:Title>
        <ows:Identifier>endpoint-url</ows:Identifier>
        <ns:LiteralData xmlns:ns="http://www.opengis.net/wps/2.0">
          <ns:Format default="true" mimeType="text/plain"/>
          <ns:Format mimeType="text/xml"/>
          <LiteralDataDomain>
            <ows:AnyValue/>
            <ows:DataType ows:reference="xs:anyURI"/>
          </LiteralDataDomain>
        </ns:LiteralData>
      </wps:Input>
      <wps:Output>
        <ows:Title>response</ows:Title>
        <ows:Identifier>response</ows:Identifier>
        <ns:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
          <ns:Format default="true" mimeType="application/soap+xml"/>
          <ns:Format mimeType="text/xml"/>
        </ns:ComplexData>
      </wps:Output>
    </wps:Process>
  </wps:ProcessOffering>
</wps:ProcessOfferings>

The Process was kept generic, so that it is able to work with a wide variety of services. The following table explains the inputs:

Table 2. WPS Async facade process inputs
Input name	Description	Format
request	The request that shall be send to the service	XML/SOAP
endpoint-url	The URL of the service endpoint	anyURI

The following outputs are provided by the process:

Table 3. WPS Async facade process outputs
Output name	Description	Format
response	The response	XML/SOAP

7.1.1. Experiments

In the following section, we describe some experiment we conducted with different SOAP services.

SOAP WFS

The following XML shows an example execute-request encapsulating a GetFeature request to a SOAP WFS provided by the Arizona State University:

<?xml version="1.0" encoding="UTF-8"?>
<wps:Execute xmlns:wps="http://www.opengis.net/wps/2.0"
  xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlink="http://www.w3.org/1999/xlink"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd"
  service="WPS" version="2.0.0" response="document" mode="async">
  <ows:Identifier>testbed12.cmd.AsyncFacadeProcess</ows:Identifier>
  <wps:Input id="request">
    <wps:Data mimeType="application/soap%2Bxml">
      <soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
        <soap:Head></soap:Head>
        <soap:Body>
          <wfs:GetFeature xmlns:wfs="http://www.opengis.net/wfs/2.0"
            xmlns:fes="http://www.opengis.net/fes/2.0" xmlns:gml="http://www.opengis.net/gml/3.2"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://schemas.opengis.net/wfs/2.0/wfs.xsd http://schemas.opengis.net/gml/3.2.1/gml.xsd http://schemas.opengis.net/filter/2.0/filterAll.xsd"
            service="WFS" version="2.0" outputFormat="application/gml+xml; version=3.2" count="3000">
            <wfs:Query typeNames="landuse"></wfs:Query>
          </wfs:GetFeature>
        </soap:Body>
      </soap:Envelope>
    </wps:Data>
  </wps:Input>
  <wps:Input id="endpoint-url">
    <wps:Data>
      <wps:LiteralValue>http://polar.geodacenter.org/services/ows/wfs/soap/1.2</wps:LiteralValue>
    </wps:Data>
  </wps:Input>
  <wps:Output id="response" transmission="value"
    mimeType="application/soap%2Bxml" />
</wps:Execute>

Note that the mode-attribute is set to async and the response to document. The later means, that the response of the upstream service will be encapsulated in a WPS Result document. The attribute could also be set to raw to obtain the upstream service response directly. The WPS directly returns a StatusInfo document containing a jobId that can be used to obtain further status updates and finally the result:

<?xml version="1.0" encoding="UTF-8"?>
<wps:StatusInfo xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
  <wps:JobID>ca291352-593a-4ddc-bdab-d1f079d7125c</wps:JobID>
  <wps:Status>Accepted</wps:Status>
</wps:StatusInfo>

For subsequent status requests the following URL-syntax must be used:

http://myhost/myWPSWebapp?Request=GetStatus&Service=WPS&version=2.0.0&jobid=ca291352-593a-4ddc-bdab-d1f079d7125c

After the WPS process has successfully requested the upstream service, the status will be updated:

<?xml version="1.0" encoding="UTF-8"?>
<wps:StatusInfo xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
  <wps:JobID>ca291352-593a-4ddc-bdab-d1f079d7125c</wps:JobID>
  <wps:Status>Succeeded</wps:Status>
</wps:StatusInfo>

To obtain the result from the WPS the following URL-syntax must be used:

http://myhost/myWPSWebapp?Request=GetResult&Service=WPS&version=2.0.0&jobid=ca291352-593a-4ddc-bdab-d1f079d7125c

Figure 5. The 3000 landuse features

The following table shows some statistics of the load time for loading 1000, 2000 and 3000 (all) features using the SOAP WFS directly and via the WPS facade (sync execute for testing). 100 requests were send during each test.

Table 4. Statistics for load times comparison requesting the SOAP WFS for 1000 features
Method	Min (ms)	Max (ms)	Mean (ms)	Standard deviation (ms)
Direct	4712	12314	5067,18	982
Via WPS Facade	7522	20897	9463,19	1474,69

Table 5. Statistics for load times comparison requesting the SOAP WFS for 2000 features
Method	Min (ms)	Max (ms)	Mean (ms)	Standard deviation (ms)
Direct	8122	14632	8624,91	1179,64
Via WPS Facade	13771	28492	15430,2	1956,01

Table 6. Statistics for load times comparison requesting the SOAP WFS for 3000 features
Method	Min (ms)	Max (ms)	Mean (ms)	Standard deviation (ms)
Direct	11056	28070	11779,59	1930,31
Via WPS Facade	18536	39433	20892,55	2851,57

SOAP WCS

The following XML shows an example execute-request encapsulating a GetCoverage request to a SOAP WCS provided by the George Mason University:

<?xml version="1.0" encoding="UTF-8"?>
<wps:Execute xmlns:wps="http://www.opengis.net/wps/2.0"
  xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlink="http://www.w3.org/1999/xlink"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd"
  service="WPS" version="2.0.0" response="document" mode="async">
  <ows:Identifier>testbed12.cmd.AsyncFacadeProcess</ows:Identifier>
  <wps:Input id="request">
    <wps:Data mimeType="application/soap%2Bxml">
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:ns="http://www.opengis.net/wcs/2.0">
   <soap:Header/>
   <soap:Body>
      <ns:GetCoverage service="WCS" version="2.0.0">
         <ns:CoverageId>GEOTIFF:"/home/zsun/testfiles/data/2010_305_30H.tif":Band</ns:CoverageId>
         <ns:format>image/GEOTIFF</ns:format>
      </ns:GetCoverage>
   </soap:Body>
</soap:Envelope>
    </wps:Data>
  </wps:Input>
  <wps:Input id="endpoint-url">
    <wps:Data>
      <wps:LiteralValue>http://cube.csiss.gmu.edu/axis2/services/GMU_SOAP_WCS_Service.GMU_SOAP_WCS_ServiceHttpSoap12Endpoint</wps:LiteralValue>
    </wps:Data>
  </wps:Input>
  <wps:Output id="response" transmission="value" mimeType="application/soap%2Bxml" />
</wps:Execute>

As response for this request, a the coverage (base64 encoded tiff image) is returned wrapped in a SOAP envelope.

The following table shows some statistics of the load time for loading two different coverages using the SOAP WFS directly and via the WPS facade (sync execute for testing).

The size of the first coverage is 416 KB. 100 requests were sent during each test.