OGC Testbed-13: Disconnected Networks Engineering Report

This ER is required to discuss the implications of operating core OWS services such as Web Coverage Service (WCS), Catalog Service for the Web (CSW), Web Feature Service (WFS), Web Mapping Service (WMS) and Web Map Tile Service (WMTS) on DDIL networks.

The targeted working group for this document is the Architecture Domain Working Group (DWG). This ER can help inform the Architecture DWG as they define improvements for the current OWS framework including how to integrate support for DDIL networks.

All questions should be directed to the editor or the contributors:

Future work should test:

Future work should not address transport layer issues directly. OWS services should provide adaptations in these extensions to meet the needs of quickly changing networks.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

An underlying assumption to Web technologies is the availability of a global, high-bandwidth communications infrastructure such as the Internet. That assumption is not always valid. The tasks described in this section examined ways OGC web services could be adapted for DDIL communications environments.

The Internet provides a reliable, high-speed, global information exchange service. Most of us take it for granted. Yet the Internet is a complex system. It can and does fail often. Furthermore, it is not truly global. There are communities within 70 miles of Washington D.C. which have little or no access to the Internet. Therefore, the OGC Web Services standards should be extended to address communications environments with the following properties:

While not directly affecting the content transmitted, transport application layers may interfere with effective transmission in DDIL environments. Some application layer protocols are more suited to these environments and would be favored in a non OWS client-server relationship. One example of an HTTP alternative is MQTT. Nodes on a properly configured MQTT network can cooperate to deliver information and retain messages. Because the basis for transmission in MQTT application networks is a publish-subscribe model, these networks are also suited for intermittent networks where transmission links are not reliable and asynchronous communication is preferred.

Clients operating in this environment cannot access a typically designed OWS service because the supported application protocol in the OWS specifications is HTTP, or higher level protocols, such as SOAP [OWS]. Clients and servers wishing to use an alternative application protocol would need a method of defining a new DCP format to support MQTT and similar protocols.

Switching services or data nodes requires consistent security mechanisms to be in place. In many cases new services can step in for a client if another is unavailable. Authenticating against this new service and verifying the validity of this new service requires consistent distribution of authentication and access control directories amongst potential OWS services as well as distributed trust information to verify new services. Additionally, any security mechanism implemented by an OWS service that is part of its advertised capabilities must be consistent across all implementing services.

A best practice is not to use linked data for critical information. Links to information such as XML schemas etc. are less likely to be important than relevant / current information. Critical information should be retrievable from the operating network of the client.

When networks are unreliable and connectivity is intermittent, it is possible that messages between two nodes can be partially transmitted. In this case, it is ideal to have message formats that while incomplete, can be parsed to recover as much information as possible. Typical document-based OWS request/response formats for operations in CSW and WFS are record-based and streamlike. For raster based services such as WMS and WCS, advertised formats may vary, but these formats are not dictated by the service type.

Additionally, CSW and WFS responses can be paged. This gives the client some measure of control over how much information can be downloaded in a single request/response cycle. Paging is useful when dealing with intermittent networks as it can support retries during synchronous communications with a server. A client can retry pages at a time and slowly build up the full response instead of retrying the full response only to have it always fail due to size and transmission time.

Server side compression has been proposed as useful capability to reduce bandwidth demands in low bandwidth networks. Testbed-13 and Testbed-12 both examined the application of various compression techniques such as Apache Avro and Google Protobuf to determine their applicability and the performance impact of compression on data exchange. The results of these experiments are presented in Solutions.

Traffic on a network can be bursty, intermittent, consistently low-bandwidth and predictably or un-predictably available. The nature of the network traffic flow can be characterized using Quality of Service (QoS) metrics. If clients are capable of adapting, there should be "hints" about traffic type to help the client make decisions. For example, if a client understood that a WCS provided multiple image formats, some formats would better support an intermittent network connection than others. Streaming image formats such as JPEG 2000 Interactive Protocol (JPIP) might be a better response type than one large GeoTIFF with a multi-resolution pyramid, or vice versa. A server could advertise its current traffic characteristics. Clients could use this information to determine what formats to request and how to request them. As another example, a WFS client might understand that it is reasonable to request a large amount of information from the WFS for purposes of caching, because the bandwidth is available for a quick download, but may not be consistently available.

When connectivity and bandwidth are available, clients may need to cache information they are interested in or anticipated being interested in. Clients that require access to constantly changing information will require "shelf-life" attributes for information retrieved from services, observation timestamps and depreciating confidence levels in the information become important. Caching also requires careful strategies for prioritizing information to make best use of available network resources. A client may begin by requesting "summary" data first, as this data is better than none at all, and then more detailed data as best it can. To support this strategy, OWS services would need to provide information at both levels of detail.

Similar to caching, replication of data implies that there is a complete copy of server data within an area/time range or topic of interest maintained on the client in similar service. When contact with the upstream/authoritative server is available, the client copy can be updated. When network changes prevent contact, the client can query the local service instead. Replication and caching both require extra storage capacity and this may limit their applicability in some DDIL clients. Just like caching, replication assumes some contact between servers to ensure data is updated to its latest version. Replication was discussed in Testbed-13 using the GSS bulk data transfer protocol. This work is discussed in Solutions.

Synchronization implies 2-way communication of transactional information between client and server. In completely disconnected networks, there must be a physical exchange of information between participating nodes. Clients often perform transactions against the central server data store. If a link is unavailable at the time of the transaction, the client must queue the transactions, and be cognizant of conflicting changes on the server when the link becomes available and the queued transactions are applied to the server.

In Testbed-13, Testbed-12 work was extended to explore compression and serialization using Apache Avro and Google Protobuf. The findings from the experiment indicated that compression using known techniques can mitigate the size impact of document-based responses from WFS implementations. In the table below, we can see that the application of zip to compress responses is one of the most efficient and ubiquitous compression techniques available. OWS services that provide text-based document responses would gain from providing compressed output in DDIL environments where bandwidth is poor.

In Testbed-12 and Testbed-13, work was done to support the use of Asynchronous methods using the "WPS Facade" approach for WFS services. These approaches are significant for the adaptation of OWS services to DDIL environments. They provide a method for a client to register a request for information and be able to retrieve the response for that request out of band. Applicability in DDIL environments is apparent, considering the interrupted network case. A client can register a request for information such as a GetFeature request, register a response handler for the request, get back an acknowledgement from the server, go offline, come back online and use the server acknowledgement to check the status of and retrieve the response. The response could be made available to the client by a response handler via email, download from the server or file system retrieval, or any applicable method that suits the client. Using the WPS approach, a "mode" variable is set to "asynchronous" to trigger the response to provide the information regarding the now registered job. In the experiment, the response handler sent an email to an email address specified by the client.

Currently, there is no universally-defined method for defining asynchronous request handling for all OWS services. WPS and CSW implement asynchronous request processing. The outcome of this experiment was to recommend investigation into extending OWS to support a general method for defining asynchronous processing.

Asynchronous processing is typically defined as a one-off process to define a single request-response cycle. There are other cases where requests may be long-lived and exist as standing orders. Typically, this would be accomplished with an OWS service by constantly polling a server to detect changes in features that match some attributes of interest for the client. In DDIL environments, this is impossible or challenging. To handle this concept, the OGC has defined a [PubSub] standard, built upon the Web Services Notification model of the Organization for the Advancement of Structured Information Standards (OASIS). Using this model, interested clients register an interest in a topic with the server or a broker that negotiates the protocol with the server. The broker can also intercept changes in the server data that matches this topic and notifies the registered client of the changes. The changes can be in the notification, or the notification can provide information on how to retrieve the data. In Testbed-13, a PubSub experiment was performed using a GeoSynchronization Server (OGC 17-031). In addition to synchronizing feature data, the server would notify interested clients of data changes as they occurred. In Testbed-12, the CSW specification was extended to support the PubSub protocol.

For Testbed-13, the application of the GeoSynchronization Server was to implement a change review process which included notifications of the changed geographic features. In the diagram below the Subscriber node subscribes to the Replication Feed. The Replication Feed provides the latest changes to the features stored in a remote WFS of interest to the Subscriber, once the changes have been pushed to the WFS by the reviewer. The subscriber registers an interest in replication events so that it receives only the latest changes as they occur.

A replication event signals that, as a result of a proposed change, a change has been made to a data provider’s data store and describes what that change is. Interested parties can then retrieve the description of the change and apply that change to their local copy of the data thus synchronizing their data store with the data provider’s data store.(OGC 17-031)

Subscribers use the publication mechanism to register for publication events like the replication event. Filter criteria, such as geographic area, constrain the publication of events to those features that conform to the specified filter.

In DDIL networks, this form of replication registration is useful when clients will be predominantly offline, offering a method to "catch-up" on feature changes when online, without replicating the entire contents of the WFS.

Since 2016, effort to develop an extension to OWS Capabilities responses that include QoS metrics has begun. Members of the OGC Quality of Service and Experience DWG have begun defining XML Schema Documents (XSD) to describe concepts such as Operational Status, Operation Anomaly, Quality of Service metrics etc. Using these new concepts, servers can advertise specific constraints on their service quality. Clients extending OWS to include these concepts will be able to react to advertised availability constraints that describe when a network is available, how much bandwidth is available and how reliable the network is. A client can develop strategies for dealing with advertised issues such as using small page sizes for results if a network is unreliable, or requesting smaller image formats from servers on low-bandwidth networks.

Many data stores that back services such as WFS, WCS, WMS and WMTS can be broken-down into subsets and stored in a portable stand-alone database format known as GeoPackage (OGC 12-128r14). GeoPackage is an OGC standard with extensions that supports the commonly used data formats provided by these services.

GeoPackage can store vector feature information in a manner similar to simple-features. Additionally, GeoPackage can store raster map data and imagery in a tile matrix pyramid similar to WMTS, this data can be used to compose map images at arbitrary scale to support WMS GetMap requests.

GeoPackage has a proven extension system that can be used to store additional information to help disconnected systems symbolize the stored vector data. Additionally, a new tiled gridded coverage extension has also been developed that can be used to provide coverage information such as elevation that would typically be served by a WCS (OGC 17-066r1).