Published

OGC Engineering Report

Testbed-18: Features Filtering Summary Engineering Report
Sergio Taleisnik Editor
OGC Engineering Report

Published

Document number:22-023r2
Document type:OGC Engineering Report
Document subtype:
Document stage:Published
Document language:English

License Agreement

Use of this document is subject to the license agreement at https://www.ogc.org/license



I.  Abstract

This OGC Testbed-18 (TB-18) Features Filtering Summary Engineering Report (ER) summarizes the implementations, findings, and recommendations that emerged from the efforts to better understand the current OGC API-Features filtering capabilities and limitations and how filtering can be decoupled from data services.

This ER describes:

II.  Executive Summary

Previous OGC work has addressed the challenges of increasing interoperability between aviation data services. Recently, the OGC community has developed a new family of standardized OpenAPI-based Web API Standards for various geospatial resource types. These OGC APIs have the potential to enhance the way in which consumers can access geospatial data from various sources. Testbed-16 brought together previous work on the development of OGC API Standards, the use of semantics to enrich data, SWIM data processing, and demonstrated an OpenAPI-based API serving SWIM data. Testbed-17 used the lessons learned and recommendations from Testbed-16 and focused on further testing the value of standards-based APIs within the SWIM program.

OGC API-Features endpoints define their filtering capabilities. Filtering is standardized across different parts of OGC API-Features (see section Previous Work), with two parts still in draft status. Advanced filtering capabilities require sophisticated server software. Not all data providers will be able to operate such a powerful service endpoint. FAA SWIM Data Services currently produce data from the National Airspace System (NAS) that is provided to consumers using various protocols and service offerings in both synchronous and asynchronous messaging formats. Testbed-18 (TB-18) explored filtering mechanisms for feature data served by OGC API-Features instances. The experiments included filtering of native and fused SWIM data and experimented with filtering services.

The research questions for the Advanced Filtering of SWIM Feature Data Task were as listed below.

To answer these questions, this Testbed-18 Task was organized into the development and testing of a system of six interconnected components, as seen on Figure 1 and listed below.

Figure 1 — Component Diagram for the Advanced Filtering of SWIM Feature Data Task

All components were successfully developed and tested. The lessons learned throughout the Testbed are documented in this ER and help respond to the questions posed above. The following is a set of recommendations for future work.

III.  Keywords

The following are keywords to be used by search engines and document catalogues.

testbed, web service, api, standard, filter, SWIM, aviation


IV.  Preface

It is possible that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed upon by any implementation of the standard set forth in this document and to provide supporting documentation.

V.  Security considerations

No security considerations have been made for this document.

VI.  Submitters

All questions regarding this document should be directed to the editor or the contributors:

NameOrganizationRole
Sergio TaleisnikSkymantics, LLCEditor
Clemens Porteleinteractive instruments GmbHContributor
Eugene YuGeorge Mason UniversityContributor
Jérôme Jacovella-St-LouisEcere CorporationContributor
Patrick DionEcere CorporationContributor
Mohammad MoallemiConcepts Beyond LLCContributor

Testbed-18: Features Filtering Summary Engineering Report

1.  Scope

This OGC Testbed 18 Engineering Report (ER) summarizes the implementations, findings, and recommendations that emerged from Testbed-18 efforts regarding the current filtering capabilities and limitations with the OGC API — Features Standard and how filtering can be decoupled from data services. The ER describes two façades built to interface SWIM services and serve aviation data through APIs built using OGC API Standards.

2.  Normative references

The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

Open API Initiative: OpenAPI Specification 3.0.2, 2018 https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md

van den Brink, L., Portele, C., Vretanos, P.: OGC 10-100r3, Geography Markup Language (GML) Simple Features Profile, 2012 http://portal.opengeospatial.org/files/?artifact_id=42729

W3C: HTML5, W3C Recommendation, 2019 http://www.w3.org/TR/html5/

Schema.org: http://schema.org/docs/schemas.html

R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee: IETF RFC 2616, Hypertext Transfer Protocol — HTTP/1.1. RFC Publisher (1999). https://www.rfc-editor.org/info/rfc2616.

E. Rescorla: IETF RFC 2818, HTTP Over TLS. RFC Publisher (2000). https://www.rfc-editor.org/info/rfc2818.

G. Klyne, C. Newman: IETF RFC 3339, Date and Time on the Internet: Timestamps. RFC Publisher (2002). https://www.rfc-editor.org/info/rfc3339.

M. Nottingham: IETF RFC 8288, Web Linking. RFC Publisher (2017). https://www.rfc-editor.org/info/rfc8288.

H. Butler, M. Daly, A. Doyle, S. Gillies, S. Hagen, T. Schaub: IETF RFC 7946, The GeoJSON Format. RFC Publisher (2016). https://www.rfc-editor.org/info/rfc7946.

3.  Terms, definitions and abbreviated terms

This document uses the terms defined in OGC Policy Directive 49, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this document and OGC documents do not use the equivalent phrases in the ISO/IEC Directives, Part 2.

This document also uses terms defined in the OGC Standard for Modular specifications (OGC 08-131r3), also known as the ‘ModSpec’. The definitions of terms such as standard, specification, requirement, and conformance test are provided in the ModSpec.

For the purposes of this document, the following additional terms and definitions apply.

3.1.  Terms and definitions

3.1.1. Application Programming Interface (API)

an interface that is defined in terms of a set of functions and procedures, and enables a program to gain access to facilities within an application [7].

3.1.2. Façade Service

a component that fetches data from a specific data source and makes it available through its own interface [9]. The main reason for building this type of service is the difficulty or inability to modify the original data source with the intent of modifying either:

  • the underlying structure of the API; or

  • the format in which the data is made available.

3.1.3. Standardized API

an API that is intended to be deployed by multiple API providers with the same API definition.

Note 1 to entry: The only difference between the API definitions will be the URL(s) of the API deployment. All other aspects are identical (resources, content schemas, content constraints, business rules, content representations, parameters, etc.) so that any client that can use one deployment of the standardized API definition can also use all other deployments, too.

Note 2 to entry: If the API provides access to data, different deployments of the API will typically share different content.

3.1.4. Standards-based API

an API that conforms to one or more conformance classes specified in one or more standards.

Note 1 to entry: Since almost all APIs will conform to some standard, the term is usually used in the context of a specific standard or a specific family of standards. This ER considers Web APIs with a specific focus on the OGC API standards. Therefore, whenever the term is used in this ER, it is meant as an alias for an API that conforms to one or more conformance classes as defined in the OGC API standards.

3.1.5. SWIM Data

any data provided through the SWIM System.

3.1.6. Web API

an API using an architectural style that is founded on the technologies of the Web [8].

Note 1 to entry: Best Practice 24: Use Web Standards as the foundation of APIs in the W3C Data on the Web Best Practices [8] provides more detail.

Note 2 to entry: A Web API is basically an API based on the HTTP standard(s).

3.2.  Abbreviated terms

AIXM

Aeronautical Information Exchange Model

API

Application Programming Interface

CRS

Coordinate Reference System

ER

Engineering Report

FAA

Federal Aviation Administration

FIXM

Flight Information Exchange Model

NAS

National Airspace System

NOTAM

Notice to Airmen

OGC

Open Geospatial Consortium

SCDS

SWIM Cloud Distribution Service

SFDPS

SWIM Flight Data Publication Service

SWIM

System Wide Information Management

TB

Testbed

TFMS

Traffic Flow Management System

TIE

Technology Integration Experiment

WFS

Web Feature Service

WXXM

Weather Information Exchange Model

4.  Introduction

4.1.  Background

4.1.1.  SWIM

The System-Wide Information Management (SWIM) initiative supports the sharing of aeronautical, air traffic, and weather information. This is accomplished by providing communications infrastructure and architectural solutions for identifying, developing, provisioning, and operating a network of highly distributed, interoperable, and reusable services.

As part of the SWIM architecture, data providers create services for consumers for data access. Each service is designed to be stand-alone. However, the value of data increases when combined with other data. Real-world situations are often not related to data from one but instead from several SWIM feeds. Since consumers can retrieve data from several SWIM services there is the need for interoperability between the services.

4.1.2.  OGC API Standards

For several years, OGC members have worked on developing a family of OGC Web API standards for various geospatial resource types. These OGC API Standards are defined using OpenAPI. As the OGC API standards continue to evolve, are approved by the OGC, and are implemented by the community, the aviation industry can subsequently experiment with and implement them.

The following OGC API Standards and Draft Specifications were used for the development of APIs during Testbed 18.

OGC API – Features: A multi-part standard that defines the capability to create, modify, and query vector feature data on the Web and specifies requirements and recommendations for APIs to follow a standard way of accessing and sharing feature data. It currently consists of four parts and a fifth proposed part.

  • OGC API — Features — Part 1: Core. Approved September 2019, this standard defines discovery and query operations. [11]

  • OGC API — Features — Part 2: Coordinate Reference Systems by Reference. This standard, approved October 2020, extends the core capabilities specified in Part 1: Core with the ability to use coordinate reference system (CRS) identifiers other than the defaults defined in the core. [12]

  • Draft OGC API — Features — Part 3: Filtering. Part 3 specifies an extension to the OGC API — Features — Part 1: Core standard that defines the behavior of a server that supports enhanced filtering capabilities. [13]

  • Draft OGC API — Features — Part 4: Create, Replace, Update, and Delete. Part 4 specifies an extension that defines the behavior of a server that supports operations to add, replace, modify, or delete individual resources from a collection. [14]

  • Proposal OGC API — Features — Part 5: Search. The proposal is an initial draft for query resources that support queries on multiple collections in the same request, parameterized stored queries, and join queries. [5]

A Common Query Language (CQL2) is being developed together with Part 3 to standardize a language that is recommended for filter expressions. [15]

OGC API – Processes: An approved (August 2021) OGC API Standard that specifies requirements for implementing a Web API that enables the execution of computing processes and the retrieval of metadata describing their purpose and functionality. Typically, these processes combine raster, vector, coverage, and/or point cloud data with well-defined algorithms to produce new information. [1]

Draft OGC API – Tiles: This recent OGC API Standard defines how to discover which resources offered by the Web API can be retrieved as tiles, retrieve metadata about the tile set (including the supported tile matrix sets, the limits of the tiled set inside the tile matrix set), and how to request a tile. [2]

Draft OGC API – Styles: This draft OGC API specifies building blocks for implementing OGC Standards based Web APIs that enables map servers, clients, and visual style editors to manage and fetch styles. [3]

4.1.3.  Exploration of OGC API Standards by SWIM

Over the years, the FAA and OGC have jointly explored making SWIM data more easily accessible and more valuable. As part of these past efforts, Testbed-16 brought together previous work on the development of OGC APIs, the use of semantics to enrich data, and SWIM data processing. The objectives were to deliver the first demonstration of an OpenAPI-based API serving SWIM data, a component generating aviation Linked Data, and two client applications querying and displaying that data [14].

Two of TB-16 recommendations were to integrate OGC API requirement classes within SWIM Data Services and to demonstrate interoperability between diverse Aviation APIs [14]. In order to advance these recommendations, TB-17 focused on the development of eleven APIs based on OGC API Standards, and the completion of Technology Integration Experiments (TIEs) between these APIs.

During TB-16, the development of the API serving aviation data resulted in numerous lessons learned and recommendations [14]. TB-16 saw the development of one aviation-related API based on an OGC API Standard (OGC API — Features). The APIs developed during TB-17 ([4]) addressed many of those lessons learned and implemented additional OGC API Standards (draft and approved) which have been maturing since. This process is reflected in Figure 2.

Figure 2 — History of OGC experiments to enhance SWIM

4.2.  Requirements Statement

Testbed-18 required investigating the potential of filtering using OGC API Standards in the context of the SWIM Program.

The original goals of the TB-18 Advanced Filtering of SWIM Feature Data Task were as follows

  • Experiment with OGC API-Features filtering mechanisms.

  • Explore if best practices for advertising filtering capabilities are required beyond what is already defined in the various OGC API-Features Parts.

  • Demonstrate advanced filtering in situations where the data endpoints support only rudimentary filtering by introducing a new service type “Filtering Service.”

  • Allow filtering rules for a specific data service to be provided at runtime in a machine-readable manner.

The research questions for the Advanced Filtering of SWIM Feature Data Task were as follows.

  • How does filtering of SWIM data served by OGC API-Features endpoints work?

  • Is the metadata required by the various OGC API-Features parts sufficient to allow clients to fully “understand” the filtering capabilities of a service endpoint?

  • OGC API — Features — Part 3: Filtering and the Common Query Language (CQL) supports queryables that are not directly represented as resource properties in the content schema of the resource. Is it possible to identify best practices for their usage?

  • Clients may know the content schema of offered resources. How to use this knowledge for advanced filtering beyond what is defined in OGC API — Features — Part 3: Filtering and the Common Query Language (CQL)?

  • How does a filtering service look like that allows advanced filtering for rather simple OGC API-Features-based SWIM data endpoints?

  • How does such a service work in situations where a data publisher has restricted filtering on certain properties (for example, because the backend datastore has not been configured to allow high-performance queries on those properties)?

  • How can a client application support a customer who has knowledge of the content schema of an offered resource in the creation of filter statements? What are the key requirements for a developer GUI that supports visualization and management of these filtering tools?

  • Is easily creating a new filtered dataset possible by creating machine readable filtering rules based on the metadata required by the OGC API-Features standards? How can these rules be provided to the Filtering Service at runtime?

4.3.  Functional Overview

As shown in Figure 3, the Advanced Filtering of SWIM Feature Data Task architecture was organized into a system of seven interconnected components. All seven components were developed simultaneously throughout the Testbed, with permanent communication and cooperation among participant organizations.

The components can be divided into three groups.

  • Façades for SWIM services with simple filtering mechanisms. Retrieve aviation data from multiple SWIM services and serve these data through APIs built based on OGC API Standards featuring basic filtering mechanisms. Three Façades were built.

    • The OGC API-Features Façade 1 (identified collectively as D100): Four APIs built to serve NOTAMs, Airport Layouts, and Airspaces

    • The OGC API-Features Façade 2 (identified collectively as D101): Three APIs built to serve aeronautical, flight, and weather features.

    • An extra façade, not originally included in the Task architecture, was offered in-kind by the company Skymantics, and was named OGC API-Features Façade 3: An API built to serve flight plans from the SFDPS (FAA) Service.

  • Components that serve aviation data with advanced filtering mechanisms. Two filtering services were built, each one featuring an API.

    • The Filtering Service 1 (identified as D102): Built to serve SWIM data from D100 with advanced filtering mechanisms.

    • The Filtering Service 2 (identified as D103): Built to serve SWIM data from all three façades with advanced filtering mechanisms.

  • Client components to demonstrate consumption of filtered data, and configuration of filtering mechanisms. Two clients were built: One meant to serve an aviation domain expert and the other to serve a developer of aviation software applications.

    • The Business User Client (identified as D104): A client built to query filtering services and demonstrate the usage of advanced filtering mechanisms.

    • The Developer Client (identified as D105): A client built to define filter statements that can be expressed in a machine-readable way and can be exchanged with the filtering services.

Figure 3 — Component Diagram for the Advanced Filtering of SWIM Feature Data Task

4.3.1.  Component Interactions

The following two figures illustrate the intended interactions between the components described in the Work Items & Deliverables section of this ER. The two figures illustrate the workflows for using the filtering service for data subsetting (Figure 4) from the perspective of a business client and for configuring the filtering service at runtime (Figure 6) from the perspective of the filtering rules developer.

In the first workflow, illustrated in Figure 4, an OGC API-Features façade to SWIM Data Service data service offers insufficient filtering capabilities to its customers. The Business User Client does not want to access large data sets and then perform filtering itself. Instead, the client wants to make use of a Filtering Service that can handle the filtering of the data and provide the subset of the data that the client is interested in. If the filtering service receives a data request from the client, it connects to the data service to access the necessary data, filters out everything that is not requested by the client, and eventually delivers the result to the client.

Figure 4 — Workflow from the perspective of a business user that needs filtered data

Figure 5 — First Workflow Sequence Diagram

The second workflow, illustrated in Figure 6, demonstrates how a filtering service can be configured at run time. The assumption is that the Developer Client is aware of the API characteristics of the data service as well as the content schema of the data served by the data server. Based on both, the client supports the user with a GUI in the definition of the filtering rules. The user can then register these rules with the filtering service, which is now configured to run the data service specific filtering.

Figure 6 — Workflow from the perspective of a filtering rules developer

Figure 7 — Second Workflow Sequence Diagram

5.  OGC API-Features Façade 1 (D100)

The OGC API-Features Façade 1, identified as deliverable 100 or D100, is a component built to demonstrate a service consuming data from a SWIM data service and serving that data through an API built based on OGC API standards offering a limited set of filtering capabilities. The component was demonstrated by interactive instruments GmbH.

The Façade Service was originally set up in Testbed 17 (see the description of D104 in chapter 4 of [4]) and updated for Testbed 18.

5.1.  Internal Architecture

5.1.1.  Component Overview

This Façade Service features the following.

  • Two data retrieval subcomponents: one for the Federal Notice to Airmen System (AIM-FNS) and another one for airspace and airport static data sources.

  • A database to store the extracted data.

  • ldproxy which delivers the APIs that serve the extracted data. Figure 8 describes the Façade Service subcomponents, the data sources (note that for AIM-FNS the retriever also accesses static sources on some occasions), and the Testbed-18 components consuming from the Façade APIs.

Figure 8 — D100 Component Overview

5.1.2.  ldproxy (Web API)

The main component that delivers the Aeronautical Data APIs is ldproxy. ldproxy is a software product written in the Java programming language that implements the OGC API family of Standards. ldproxy enables sharing geospatial data using Web APIs based on OGC API Standards.

The component provides the following four Data APIs that have been deployed.

Responses with feature data could be requested in the following representations/formats.

  • GeoJSON (media type application/geo+json, query parameter f=json)

  • JSON-FG (media type application/vnd.ogc.fg+json, query parameter f=jsonfg)

  • HTML (media type text/html, query parameter f=html)

  • FlatGeobuf (media type application/flatgeobuf, query parameter f=fgb)

In addition, some of the APIs also provided access to vector tiles and styles for those tiles. These can be useful for exploring the data but are not directly related to the research questions.

For more information about the APIs, see section 4.2 of the Testbed 17 Aviation API Engineering Report.

5.1.3.  PostgreSQL/PostGIS (Database)

The data available for API access are stored in a PostgreSQL/PostGIS database. That is, the data are stored in tables with the geometries stored as Simple Features geometries in Well-Known-Text format.

The following are the aeronautical datasets.

  • All Controlled Airspaces of the Classes B, C, D, and E from the National Airspace System Resource (NASR) Subscription

  • Airport data for the major airports in the United States provided by Hexagon (AIXM 5.1)

  • Notices to Airmen (NOTAMs) received from a Federal Notice to Airmen System (AIM-FNS) subscription in FAA’s SWIM Cloud Distribution Service (SCDS), a cloud-based infrastructure dedicated to providing near real-time FAA SWIM data to the public via Solace JMS messaging

5.1.3.1.  Communication Between ldproxy and PostgreSQL

ldproxy converts API requests to SQL queries and processes the results to convert them to API responses in the GeoJSON, Features and Geometries JSON (JSON-FG), HTML, or Mapbox Vector Tile (MVT) formats.

Figure 9 — Information Flow for Data Requests

While the first two datasets are static and do not change, the NOTAMs are dynamic. New NOTAMs are added to the database as they are received from the SCDS subscription. Since a new NOTAM may change information that is cached in ldproxy for performance reasons (in particular, the spatial and temporal extent of the NOTAM dataset, but also vector tiles) a database trigger is used to notify ldproxy about a new NOTAM. For each new NOTAM, the spatial and temporal extents are evaluated and, if needed, the extents of the NOTAM feature collection are updated. Additionally, vector tiles that include the spatial extent are invalidated in the tile cache.

Figure 10 — Communicating Data Changes to ldproxy

The communication between ldproxy and PostgreSQL uses the standard PostgreSQL protocol, which is TCP/IP-based.

5.1.4.  Data Retrieval

See sections 4.1.4 and 4.1.5 of the Tested-17 Aviation API Engineering Report on details how the SWIM datasets are loaded into the database.

5.2.  Differences to the component from Testbed 17

The component was updated to the latest version of ldproxy and to the new version 1.0.1 of OGC API Features Part 1: Core.

The APIs mainly supported simple access to the data, based on OGC API Features Part 1: Core. That is, paging was supported as well as filtering using bbox, datetime, and selected feature attributes.

Some sample requests for data requests with simple filter capabilities were included in the description of the NOTAM API, by clicking on “Sample requests to filter NOTAM events” on the HTML landing page for more information. This gave clients an idea of how to construct data requests with simple but already useful filter capabilities.

Example — Example simple filtering requests in the NOTAM API

The HTML representation of the Features resource also included a simple HTML form to submit requests with these simple filters.