Testbed-12 Data Broker Engineering Report

The purpose of this report is to research various topics related to Data Brokering in the Aviation thread. The report shall:

All questions regarding this document should be directed to the editor or the contributors:

Improvements in future testbeds are desirable to the following topics:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

AIXM - Aeronautical Information Exchange Model
AMDB - Aerodrome Mapping Data Base
AMXM - Aerodrome Mapping Exchange Model
AMXS - Aerodrome Mapping Exchange Schema
API - Application Programming Interface
BBOX - Bounding Box
CCI - Cross-Community Interoperability
COTS - Commercial Off the Shelf
CRUD - create, read, update, delete
CSW - Catalog Service for the Web
DNOTAM - Digital NOTAM
ECQL - Extended Common Query Language
EFB - Electronic Flight Bag
FIXM - Flight Information Exchange Model
FNS - Federal NOTAM Service
FPS - Feature Portrayal Service
GML - Geography Markup Language
HTML - HyperText Markup Language
HTTP - HyperText Transfer Protocol
ICAO - International Civil Aviation Organization
ISO - International Organization for Standardization
JDK - Java Development Kit
NOTAM - Notice to Airmen
OGC - Open Geospatial Consortium
RDF - Resource Description Framework
SBVR - Semantics of Business Vocabulary and Business Rules
SOA - Service Oriented Architecture
SQL - Structured Query Language
SWIM - System Wide Information Management
UUID - Universally Unique Identifier
WCS - Web Coverage Service
WFS - Web Feature Service
WFS-T - WFS-Transactional
WMTS - Web Map Tile Service
WPS - Web Processing Service
WXXM - Weather Information Exchange Model
XMI - XML Metadata Interchange
XML - Extensible Markup Language
XSLT - Extensible Stylesheet Language Transformations

In OGC Testbed 11, a significant effort was made to research and develop the core specification of a “Data Broker” concept for the Aviation thread, based on OGC technology. The overall goal was to enable the setup of cascading OGC web services to form a data source chain, in which one service is capable of providing information coming from other services.

The Data Broker implemented in OGC Testbed 11 was a pure WFS 2.0 that aggregated its data on-the-fly from other data sources. Clients connecting to this service could not distinguish it from a regular WFS 2.0 service.

The following topics were analyzed and implemented:

Building on top of the previous testbed’s report and implementation, the following new functional requirements have been stated and will be analyzed in this testbed:

Building on OGC Testbed 11, various targeted solutions were investigated and discussed during Testbed 12. At a high-level, these targeted solutions were:

Although not part of the requirements, the Data Broker developed in Testbed 12 also included brokering of FIXM 4.0 (with GML) data.

Here is a summary of the recommendations identified during Testbed 12. For details, refer to the following chapters.

Apart from these recommendations, a change request has also been identified for FIXM 4.0, i.e. the integration of an optional ISO 19115 / 19139 metadata property on a Flight feature. The need for this was revealed during the Data Broker’s data aggregation process, to allow the integration of lineage / provenance information on FIXM flight features. For details, refer to the FIXM GML Engineering Report, OGC 16-028.

The high-level architecture of the Data Broker in Testbed 12 is shown in the figure below. Similar to Testbed 11, it connects to WFS providers offering AIXM 5.1 data. New is the addition of providers offering AMXM 2.0 and FIXM 4.0 data, an integrated WMS and integration with a CSW to discover data providers.

On an interface level, the data broker is split up into two services: WMS and WFS. Both services offer different access to the same data, and the individual services can communicate between each other to exchange data where needed. It is possible for instance to request a layer from the WMS service that is a portrayed version of the feature types of a WFS service. This allows both WFS and WMS based clients access to the same data.

On the business level, the data broker uses a CSW for endpoint discovery. The CSW provides the Data Broker with a list of AIXM, AMXM and FIXM based WFS services, as well as existing WMS services. The list could even include other Data Brokers. The Data Broker uses the capabilities document of each of the services to aggregate and conflate the capabilities of all the services it mediates. This means that for each of the feature types available in each of the source WFS services, a feature type is created in the capabilities document of the Data Broker. If multiple WFS services are found with the same feature types, then their capabilities documents are merged, taking into account the enlarged bounds of the feature types.

For WMS, a portrayal layer is created for each of the WFS feature types available in the Data Broker. This allows clients without WFS capabilities to display feature types without any GML decoding abilities. Finally, the WMS also contains a layer for each of the layers inside an aggregated source WMS service.

To support the Data Broker research in Testbed 12, the implemented Data Broker of Testbed 11 was extended based on the use cases and architecture discussed in the previous sections. This particular implementation was built on top of Luciad’s COTS software product LuciadFusion. LuciadFusion offers a set of standards-based software components for server-side development, including an OGC Web Services Suite equipped with OGC-compliant WFS and WMS service components. One of these component’s benefits for the Data Broker task is its open data back-end API, allowing users to easily connect to any type of storage component – such as other OGC web services.

The Luciad Data Broker implementation is based on Java Servlet technology. To run, the Broker requires a Java servlet container or application server compatible with Java Servlet 3.0 or higher. Apache Tomcat 7 was used during Testbed 12. Other than being capable of running a Java Virtual Machine 1.7 (or higher) and an appropriate servlet container / application server, no requirements are posed on the underlying hardware or operating system.

An essential part of adding an OGC WMS to the Data Broker is the definition of the available WMS layers, listed in the capabilities. The Data Broker’s WFS defines a list of feature types representing the available feature data. This list can be mapped one-to-one on a list of WMS layers. By definition, the resulting list is grouped in one root WMS layer.

As an example, the following feature type is returned from the Data Broker when listing WFS feature types:

Then the resulting WMS Layer will look like this:

An important aspect of the Data Broker’s WFS is the integration of lineage information in the feature data to identify a feature’s data source. This is obviously no longer possible in a WMS context, since the result is bitmap. To still inform WMS clients about the underlying data source, we can include relevant metadata in the WMS layer definitions in the capabilities.

To enable WMS clients accessing more information about a visualized feature, the Data Broker’s WMS can offer GetFeatureInfo support. The WMS standard does not define any standardized exchange format, so we can define a few possibilities that match with the Data Broker’s use case:

A standard WMS uses one or more predefined styles for each offered layer, from which a client can choose. A WMS with support for the Styled Layer Descriptor (SLD) Profile extends this with the ability to include an SLD in a request, enabling users to customize the styling of a WMS layer.

Within a Data Broker, both capabilities are useful:

In contrast to the styling options, data filtering options are limited. In a standard WMS, data cannot be filtered, except for a geographical area of interest that is supplied with a map request. To filter data based on its properties - for instance, to filter all airports that are located in a given ICAO region - an SLD-enabled WMS is required: next to a styling definition, an SLD can include an OGC Filter to filter the data during the rendering process. One drawback of this approach is the fact that the filter is an integral part of the styling definition: it cannot be used independently of a customized style, so users will always need to define a style next to the filter - which might not be desirable, if the default, predefined style is preferred.

For WMS layers serving rendered vector data, such as the Data Broker’s WMS, it would be useful to let clients include filters in map requests to filter the layer’s data content. One practical example in the geospatial community can be found in GeoServer, which supports filter parameters in a map request - expressed as an OGC Filter or using ECQL.

In case the offered filtering options can be limited - such as only allowing to filter on ICAO region - it could also be an option to rely on WMS dimensions. WMS dimensions make it possible to parameterize a layer with one or more parameters. Typical examples are altitude and time, but it is equally possible to define a custom dimension - such as an ICAO region. Although this approach limits the filtering possibilities (you can only filter based on what is offered by the layer’s dimension(s), it is part of the OGC WMS standard - hence an interoperable solution.

An AIXM 5.1 data set essentially consists of a set of features, contained in a feature collection. In a stand-alone context, this feature collection is an AIXM BasicMessage. Within a WFS context, this feature collection is by definition a WFS FeatureCollection. In AMXM 2.0, there isn’t a feature collection element. Features can still be grouped, but within another, core feature: the AerodromeMappingDatabase. In theory, this AerodromeMappingDatabase could be added as single feature in a WFS context, ending up with a WFS FeatureCollection containing a single AerodromeMappingDatabase feature containing all available airport layout features (Aprons, Taxiways …). Aligned with the standard, the WFS only knows about the AerodromeMappingDatabase features - the underlying features are considered properties - so a response will either be empty or contain this AerodromeMappingDatabase feature, with all its subfeatures. This is not particularly useful, since it avoids users to be able to do a fine-grained selection on airport layout features. The Data Broker implementation follows a different approach, in which the AerodromeMappingDatabase element is left out and in which the contained airport layout features are directly put under a WFS FeatureCollection.

An important aspect of the Data Broker is to store information about the source of the data, to inform clients about the original provider. This is also known as lineage information. A common and standardized way to represent such information is the ISO 19115 metadata model. Within AIXM 5.1 [3], ISO 19115 metadata can be used throughout the data model:

Within the Data Broker, the need is primarily at the feature level. Looking at AMXM 2.0, we have slightly different metadata provisions. ISO 19115 metadata can be used, but only on the AerodromeMappingDatabase feature level [2]. As discussed in the previous section, we chose to leave this feature out to better align AMXM and WFS feature provision. Consequently, this has as drawback that we can no longer store metadata on a feature level.

We could address this by replacing the WFS FeatureCollection with an AMXM AerodromeMappingDatabase and use this to include Lineage information, but this would be a deviation of the WFS standard - which dictates that the output should be encompassed in a WFS FeatureCollection. Additionally, including it on an AerdromeMappingDatabase level would make it global for all encompassed AMXM features, while these may be provided by different WFS sources used by the Data Broker.

Based on the above analysis, a recommendation for the AMXM standard is to add an ISO metadata property on all AMXM features - similar to AIXM. This gives users the ability to store fine-grained metadata on an individual feature - such as lineage information.

A Data Broker is responsible to conflate features, to ensure that there are no duplicates in the resulting Broker’s response. The preferred way to do this is to rely on unique identifier information available in the features. Similar to AIXM 5.1, AMXM 2.0 enables the use of a unique identifier, so this requirement is fulfilled. The Data Broker implementation in Testbed 12 shows that conflation equally works for both AIXM 5.1 and AMXM 2.0.

With the Data Broker able to support two exchange formats that share a lot of concepts (airport layout entities) and underlying technology (GML 3.2), we can look at leveraging these similarities. One use case is to support data mediation capabilities and enable the broker to offer heterogeneous data formats into a single format to the client.

The Data Broker’s implementation of Testbed 12 supports transforming AMXM 2.0 to AIXM 5.1 data, on-the-fly during the brokering process. To include this capability, the existing Data Broker’s processing pipeline from Testbed 11 has been extended with a mediation component, as illustrated in the figure below.

The integration of data mediation does impact some of the Data Broker’s responsibilities:

Additionally, the Data Broker should offer the mediation capability to its clients in an easy-to-use and interoperable way, hiding the details related to heterogeneous data aggregation and mediation (apart from their description in the metadata). One logical way to do this is to define a feature type per aeronautical entity, such as AirportHeliport, Airspace, ApronElement, …​ and let the client choose the resulting format by means of the OUTPUTFORMAT request parameter. Each feature type might be an aggregation of multiple data sources using heterogeneous formats; using the OUTPUTFORMAT request parameter and the Data Broker’s mediation capability, the requested data is seamlessly collected and sent in a single format to the client.

As a CSW can expose metadata records about data sets, we can use it in the Data Broker to search for data sets of interest and consequently determine what the Data Broker can serve. The following example request searches for data sets based on a geographic region.

The resulting response includes a set of extrinsic objects, identifying data sets that correspond to the specified search criteria:

The extrinsic object includes a link to further identify the data set by means of an ISO 19115 data description, with information about the type of data, its geographic region and spatial reference and its provider.

This search capability enables a Data Broker to automatically find, aggregate and serve data sets based on user-defined data criteria.

Another approach for the Data Broker’s CSW integration can be to search for web services. Because the Data Broker relies on the WFS, it can simply look for any web services adhering to this geospatial web service standard. Example request:

The resulting response includes a set of service objects, identifying web services that comply with the search criteria:

Each service object includes service binding information, providing the necessary details to set up a connection with the service.

Within Testbed 12, the CSW has been used by the Data Broker to discover web services, following the approach described in the previous section. The main benefit is avoiding manual web service configuration, which was the approach followed in Testbed 11. The discovery step is executed once, during the startup of the Data Broker. In an operational Data Broker component, this could be further extended with capabilities such as automatically & regularly scheduled discovery tasks to update the current list of web services.

For OGC Testbed 11, we introduced a data enrichment feature for the data broker. As data was being transformed or aggregated, this metadata was encoded into the resulting AIXM 5.1 result. For Testbed 12, some of the new formats do not have support for embedding ISO 19115 metadata in the response.

Both AMXM and FIXM do not currently support the ISO 19115 model to add rich metadata into the file itself. Having this would enable the Data Broker to add highly detailed lineage information on a feature level.

Alternatively, it would also have been desirable to be able to add ISO 19115 metadata into the a WFS FeatureCollection object. While this is less fine-grained than adding it directly on a feature level, it would allow for a more generic solutions to lineage metadata.

When describing a WMS service, the filtering capabilities you are presented with are limited to:

Outside of these parameters, additional filtering on the a named layer is not supported. For Testbed 12, we experimented with using the dimension parameter of WMS and using it as a filter on the GUFI of an aircraft. While this is not the intended use of the dimension parameter, it is a feasible option.

For more complex filter descriptions, it would be desirable to be able to construct more complex filtering parameters on a WMS named layer.

In OGC Testbed 12, we found that file formats based on GML 3.2 make it easier for a client to decode, if that client already supports GML 3.2 geometries.

This includes both AMXM and FIXM. The addition of GML simplified the process of decoding, parsing and visualizing the data.