I.  Abstract

The SensorThings API Extension: WebSub Asynchronous Messaging Standard (STA-WebSub) introduces an additional capability to the SensorThings API v1.1, allowing users to subscribe to a compliant SensorThings API endpoint using a request URL. This enables users to receive notifications when the outcome of a query changes, typically due to the arrival of new data. STA-WebSub is built upon the W3C WebSub Recommendation.

Specifically, the STA-WebSub Standard outlines how to determine if a request URL is suitable for subscription, how to initiate subscriptions using the URL, and how notifications will be delivered. This Standard mandates that a consumer implements an HTTP(S) Webhook to receive notifications from the STA-WebSub Hub(s) associated to a SensorThings API instance.

Implementing STA-WebSub enhances the versatility of SensorThings API services in creating asynchronous workflows over the HTTP protocol. The ability to establish trigger conditions and specify the structure of event data pushed to a Webhook via HTTP(S) POST allows for tailored event processing within workflows. For instance, when used alongside a STA-WebSub Subscriber that supports WebSockets, this Standard enables web applications running in modern web browsers to subscribe to and receive notifications (such as observations or other data) for seamless visualization, without the need for browser extensions. In an air quality example, a JavaScript application could alert the subscribed population about abnormal pollutant levels in real time on their mobile devices. In the weather sector, a process could be triggered that recalculates a weather forecast whenever sufficient new data is received through a subscription.

STA-WebSub provides two significant advantages. Firstly, subscribers can retrieve data from a SensorThings API service and then use the same URL to subscribe for updates. This approach eliminates the necessity for subscribers to poll the SensorThings API service. Instead, all updates tied to the specified URL are sent to the subscriber’s Webhook when an update event occurs. Secondly, the MQTT protocol used by the SensorThings API does not need to be exposed to third parties for receiving updates. With STA-WebSub, the MQTT protocol can be kept internal between the SensorThings API service and the associated STA-WebSub Hub(s), enhancing security and performance management of the MQTT broker.

A compliant STA-WebSub Hub implementation must be capable to convert a subscription URL into an MQTT topic, subscribe to the MQTT broker associated with the SensorThings API service, and receive MQTT notifications to distribute them to the STA-WebSub Subscribers’ Webhooks.

To fully leverage STA-WebSub’s capabilities, a SensorThings API service needs to support the MQTT topic pattern to include $filter and $expand functionalities through Open Data Protocol (OData) query parameters. The $filter parameter supports subscriptions based on defined triggering conditions, while the use of $select and $expand allows subscribers to receive precisely the required data and structure. This feature can be described as ‘receiving fit-for-purpose updates’.

To manage subscriptions according to business or governance policies, authentication and access control can be integrated into the STA-WebSub discovery process. For scalability and optimal processing of various notification types (updates), a single SensorThings API service may provide multiple STA-WebSub Hubs tailored to different types of updates or business requirements. The assignment of a specific STA-WebSub Hub to a subscriber may depend on the subscription URL and existing business policies.

Building trust in the received updates can be achieved through the W3C WebSub’s HMAC option, which offers subscribers a method to verify that the content distributed from a Hub is authentic and has not been altered.

INFO: The key work for crafting this OGC Standard was undertaken in the Enhancing Citizen Observatories for healthy, sustainable, resilient and inclusive cities (CitiObs) project, which received funding from the European Union’s Horizon Europe research and innovation program.

III.  Preface

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium SHALL not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

1.  Scope

This OGC SensorThings API Extension: WebSub Asynchronous Messaging Standard outlines the methods for extending the OGC SensorThings API v1.1 implementation to support asynchronous messaging over HTTP(S). This is achieved by applying the W3C WebSub Recommendation.

2.  Conformance

Requirements for one standardization target type are considered:

Conformance with this standard SHALL be checked using all the relevant tests specified in Annex A (normative) of this document. The framework, concepts, and methodology for testing, and the criteria to be achieved to claim conformance are specified in the OGC Compliance Testing Policies and Procedures and the OGC Compliance Testing web site.

In order to conform to this OGC® interface standard, a software implementation SHALL choose to implement:

All requirements-classes and conformance-classes described in this document are owned by the standard(s) identified.

3.  Normative references

The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

Steve Liang, Tania Khalafbeigi, Hylke van der Schaaf: OGC 18-088, OGC SensorThings API Part 1: Sensing Version 1.1. Open Geospatial Consortium (2021). http://www.opengis.net/doc/is/sensorthings/1.1.0.

WebSub, January 23, 2018. https://www.w3.org/TR/websub

M. Nottingham: IETF RFC 5988, Web Linking. RFC Publisher (2010). https://www.rfc-editor.org/info/rfc5988.

4.  Terms and definitions

This document uses the terms defined in OGC Policy Directive 49, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this document and OGC documents do not use the equivalent phrases in the ISO/IEC Directives, Part 2.

This document also uses terms defined in the OGC Standard for Modular specifications (OGC 08-131r3), also known as the ‘ModSpec’. The definitions of terms such as standard, specification, requirement, and conformance test are provided in the ModSpec.

For the purposes of this document, the following additional terms and definitions apply.

an implementation that is capable to interact with an OGC SensorThings API service to manage subscriptions and receive notifications via MQTT

a URL returned by the SensorThings API service as an HTTP Link header <Link "topic URL">; rel="self" in response to a request URL (if allowed for subscription).

a string used to subscribe to the OGC SensorThings API MQTT broker for receiving update events. The STA-WebSub Hub determines the MQTT topic from the request URL.

5.  Conventions

This sections provides details and examples for any conventions used in the document. Examples of conventions are symbols, abbreviations, use of XML schema, or special notes regarding how to read the document.

6.  Introduction to a STA-WebSub System (Informative)

This section illustrates how to build a STA-WebSub System using the OGC SensorThings API Extension: WebSub Asynchronous Messaging Standard (STA-WebSub) as defined in this document. This section further outlines a complete system that extends SensorThings API into asynchronous messaging over HTTP(S).

The STA-WebSub System is a specialized version of the asynchronous messaging system outlined in the W3C WebSub Recommendation. As a WebSub-based system, the STA-WebSub system consists of three components: Subscriber, Hub, and Publisher. The Publisher can be further divided into two parts: the HTTP interface of the SensorThings API service and the MQTT broker.

This STA-WebSub extension to the SensorThings API defines the requirements solely for the HTTP interface to implement the W3C WebSub Discovery protocol. Therefore, it does not affect the backward compatibility of SensorThings API version 1.1.

While the requirements for the Subscriber and Hub are also addressed in this document, they are not standardized. Appendix B offers guidance on how to implement a STA-WebSub Hub, enabling asynchronous messaging via HTTP for a SensorThings API service.

6.1.  Introduction to W3C WebSub

NOTE:  It is recommended to first read the W3C WebSub Recommendation.

WebSub is a W3C Recommendation that provides guidelines for implementing asynchronous processing on the web.

First, the discovery protocol enables a Subscriber to verify with a Publisher whether a self-defined topic (a resource URL) is available for subscription.

Second, the subscription protocol outlines how a Subscriber can interact with a Hub to register a topic for receiving notifications in the future.

Finally, WebSub specifies the process by which the Hub delivers updates to the Subscriber(s).

The following figure illustrates the basic functioning of the W3C WebSub Recommendation.

 

Figure 1 — W3C WebSub Flow Diagram

The notification protocol in the W3C WebSub Recommendation is unspecified (see WebSub §6 and “Publisher notifies the hub of new content” as illustrated above) so that basically any protocol can be used between a Publisher and a Hub. A STA-WebSub system leverages this openness to couple the Publisher (an OGC SensorThings API service) and a (STA-WebSub compliant) Hub using MQTT.

6.2.  STA-WebSub Overview

An implementation of the SensorThings API Standard supports two protocols: The HTTP interface and the MQTT interface. Compliant with W3C WebSub §6, STA-WebSub specifies to use the MQTT protocol between the Hub and the Publisher. In addition the STA-WebSub extension specifies how the SensorThings API HTTP interface implementation supports the W3C WebSub discovery requirements.

A STA-WebSub system consists of four components.

• SensorThings HTTP interface:

  • The W3C discovery protocol gets implemented here; and

  • The STA-WebSub error handling gets implemented here.

• SensorThings MQTT broker:

  • The Hub subscribes (and unsubscribes) for updates on MQTT topics; and

  • The broker sends MQTT notifications to the Hub.

• Hub:

  • The subscription functionality must be extended to transform a W3C topic URL into a SensorThings MQTT topic and the Hub must use the MQTT protocol to subscribe/unsubscribe for topics; and

  • The notification of distribution events takes place via MQTT. A STA-WebSub Hub must therefore listen to MQTT events received from the associated MQTT broker and distribute the event content to all subscribers using HTTP(S) POST.

• Subscriber:

  • The Subscriber discovers subscription topics by sending HTTP HEAD and GET request URLs to the HTTP interface of the SensorThings API service;

  • A URL that can be used for subscription with the Hub (via hub.topic) can be obtained from the HTTP Link header rel="self" included in the discovery response; and

  • A subscriber may use an api-key or x-api-key parameter for subscribing to the Hub.

The STA-WebSub — SensorThings API extension WebSub — defines additional capabilities in the context of SensorThings API that allow the propagation of updates via HTTP(S) using callback URLs to WebHooks as defined in the W3C WebSub Recommendation. The notification update is based on an event for a MQTT topic as defined in the SensorThings API Standard.

To make STA-WebSub work, the Subscriber, Publisher and Hub need to implement certain functionalities. In order to understand the requirements better, the following high-level flow diagram illustrates the additions for STA-WebSub.

 

NOTE:  As illustrated in the STA-WebSub flow diagram above, this standard has implications for the subscriber, the publisher’s implementations, and the Hub. However, only the publisher’s section (highlighted in the yellow box) is considered normative in this document, as the SensorThings API service is the target of standardization.

To ensure a functional solution, Appendix B offers guidance on the implementation-specific aspects necessary to adapt a W3C Hub and implement a STA-WebSub Hub.

Figure 2 — STA-WebSub Flow Diagram

The W3C WebSub specification does not define specific topics. Instead, a WebSub Subscriber can determine whether a resource URL is available for subscription. This flexibility is crucial for the STA-WebSub to enable subscribers to define:

• The actual trigger for an update event, and

• The data structure for the information delivered with an update event using ODATA query parameters.

Any implementation of the STA-WebSub extension must support the MQTT topic pattern as outlined in the SensorThings API v1.1 Standard, enhanced with ODATA queries. A SensorThings API service that accommodates ODATA options with MQTT topics allows subscribers to utilize $filter for specifying event triggers, as well as $select and $expand to define the content data structure.

The capacity to create notification conditions combined with the ability to specify the precise data structure required for a notification is a powerful feature. This capability is essential for executing workflows tailored to a subscriber’s Webhook.

NOTE:  The support for $select is mandatory for a SensorThings API service implementation. However, the support for $filter and $expand is optional.

6.3.  Introduction to Discovery, Subscription and Notification

Certain functional requirements regarding discovery, subscription and notification need to be defined for extending the W3C WebSub protocol to support STA-WebSub.

6.3.1.  Discovery

A STA-WebSub compliant SensorThings API HTTP interface (also known as a Publisher) supports W3C WebSub discovery by adding Link headers with rel="hub" and rel="self" to the HTTP response. W3C WebSub additionally requires that these Link headers be included either as HTTP response headers or inline in an XHTML encoded response.

Since the typical response format for a SensorThings API implementation is either JSON or GeoJSON, rather than XHTML, the STA-WebSub extension mandates that the Link headers be returned as HTTP response headers. Furthermore, W3C WebSub expects these discovery links to be available in response to both HTTP HEAD and GET requests.

To comply with this requirement for discovery, the STA-WebSub extension stipulates that a SensorThings API HTTP interface must support the HTTP HEAD method in addition to the already existing HTTP GET method.

The following sequence diagram illustrates the discovery for the URL http://localhost/sta/v1.1/Observations

 

Figure 3 — Discovery with URL for supported subscription

For this URL, the implementation returns the <Link>; rel="self" header.

6.3.2.  Discovery “Error” Handling

A STA-WebSub hub may receive a subscription request via the hub.topic, which translates into an MQTT topic. However, this topic may not be accepted by the SensorThings API MQTT broker. The accepted topic URLs are deployment-specific. For instance, subscribing to /Observations might generate an excessive load, resulting in it being disallowed. Additionally, using unsupported or disallowed OData options, such as $expand or $filter, can cause the absence of a Link rel="self" header in the response. The lack of this header indicates that a subscription for that particular topic URL is not possible. This behavior is perfectly in line with the W3C WebSub Recommendation. Nonetheless, any subscriber, whether a user or a service/process, may be puzzled as to why the discovery response does not include the Link rel="self" header.

It is important to note that the absence of a Link rel="self" header is not considered an error. Therefore, using HTTP 4xx status codes in this case would be inappropriate. Furthermore, the W3C WebSub specification does not provide guidelines for error handling in these scenarios. There should be guidance available to clarify why the Link rel="self" header is missing, which would support users or developers implementing a STA-WebSub Subscriber.

To address this issue, this STA-WebSub Standard introduces the Link rel="help" header. The URL associated with this relationship should direct users to a static help page that explains why a subscription to the specified topic URL is not permitted.

The following sequence diagram illustrates the discovery of a URL that is unsupported for subscription: http://localhost/sta/v1.1/Observations.

NOTE:  It is assumed that the topic v1.1/Observation is blacklisted.

 

Figure 4 — Discovery with URL not supported for subscription due to topic restriction

To acknowledge that the topic URL cannot be used for subscription, the implementation does not return the <Link>; rel="self". The <Link>; rel="help" URL URL explains the error: The topic URL is disallowed for subscription because it is using disallowed ODATA option (ODATA option $expand is blacklisted): http://localhost/sta/v1.1/Datastreams(4711)/Observations?$expand=FeatureOfInterest

A STA-WebSub compliant implementation does not return the <Link>; rel="self" but the <Link>; rel="help" header for the URL above.

6.3.3.  Subscriptions

A STA-WebSub Hub implementation must be capable of transforming the W3C WebSub hub.topic expressed as an HTTP(S) URL into an MQTT topic pattern that is accepted by the MQTT broker associated with the SensorThings API service. Given that the SensorThings API service and the Hub closely interact, this transformation should not be overly complicated.

To prevent subscriptions to unsupported MQTT topics, the Hub must utilize the discovery protocol and deny any requests if the discovery response does not include the Link rel="self" header.

If the Hub receives an unsubscribe request from a subscriber, it must confirm the intent with the subscriber and proceed to unsubscribe from the MQTT topic linked to the associated SensorThings API service.

6.3.4.  Notifications

Once the Hub subscribes to an MQTT topic, it awaits notifications from the MQTT broker associated with the SensorThings API service.

When a notification event occurs (consisting of both the topic and content), the Hub delivers the content to the callback URLs that have subscribed (the subscribers’ Webhooks) using an HTTP POST request.

{
    "serverSettings": {
        "conformance": {
            "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery"
        },
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": [],
            "odata_denied": []
        }
    }
}

{
    "serverSettings": {
        "conformance": {
            "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery"
        },
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": [],
            "odata_denied": ["$expand", "$filter"]
        }
    }
}

{
    "serverSettings": {
        "conformance": {
            "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery"
        },
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": ["v1.1/Observations"],
            "odata_denied": []
        }
    }
}

7.  SensorThings API — WebSub Extension (Normative)

As outlined in Clause 6, a system leveraging the SensorThings API Extension: WebSub Asynchronous Messaging (STA-WebSub) is a specialization of the asynchronous messaging system described in the W3C WebSub Recommendation. As a W3C WebSub system, the STA-WebSub system is comprised of three parts (Subscriber, Hub, Publisher). In the context of this OGC SensorThings API Extension: WebSub Asynchronous Messaging Standard, the Publisher is the SensorThings API service which can be split into two sub-components: the HTTP interface and the MQTT broker.

This STA-WebSub extension defines one requirements class:

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "odata_denied": <JSON Array of denied ODATA options>
        }
    }
}

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "odata_denied": ["$expand", "$skip", "$top", "$filter"]
        }
    }
}

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "odata_denied": []
        }
    }
}

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": <JSON Array of denied topics>
        }
    }
}

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": ["v1.1/Observations", "v1.1/Datastreams('very chatty')/Observations"]
        }
    }
}

{
    "serverSettings": {
        "https://www.opengis.net/spec/sensorthings-websub/1.0/conf/discovery": {
            "topics_denied": []
        }
    }
}

8.  Media Types for any data encoding(s)

This OGC Standard uses Link Header as defined in IETF RFC 5988 and W3C WebSub Recommendation.

In particular, the following HTTP Link headers are used:

Bibliography