OGC Testbed-13: Aviation Abstract Quality Model Engineering Report

This document answers the requirement for an Abstract Data Quality Model to be designed to enable service advertising, service validation and assurance within the Aeronautical Domain.

In service advertising, a service makes known to a potential consumer the quality of the data provided by the service. Based on this information, the consumer can determine whether or not the service meets their needs.

In service validation, assurance is given that the quality of the data provided by a service is consistent with the quality that is explicitly defined in a service contract or any kind of agreement that may exist between a service provider and service consumer.

Both use cases share two common preconditions:

The Abstract Quality Model forms a basis for implementing data quality services specifically for the aviation domain. Prior to this ER, data quality services were reliant on the standards inherited into the three aviation exchange models (Aeronautical Information Exchange Model (AIXM), Flight Information Exchange Model (FIXM) and Weather Information Exchange Model (WXXM)), these vary from not having any capability to record data quality to the data quality metrics for ISO 19157. Although the ISO standard is recognized and provides some of the required data quality functionality, it is missing the key concepts of timeliness and precision. The abstract quality model described in this document introduces these concepts to the AQM in a standards-compliant way.

The ER contributes to the Data Quality DWG by providing a re-interpretation of the ISO elements and extensions to the ISO models to record mission critical information in the aviation domain. Additionally, the Aviation DWG should be consulted as to the utility of this ER.

All questions regarding this document should be directed to the editor or the contributors:

An emerging requirement mentioned throughout this ER is data quality metrics in heterogeneous data created through either informal or formal conflation processes. To address this emerging requirement, future work will need to consider:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

Aeronautical information services provide data concerning the physical infrastructure in the aeronautical domain. This data is encoded in AIXM 5.1.1, which is openly documented. The model is maintained by EUROCONTROL and the FAA and is under active development. AIXM is built on recognized, open standards including ISO. From a data quality perspective, AIXM inherits the data quality metrics of ISO 19157 directly into the model, therefore any abstract quality model should also conform to similar standards to ensure simple interoperability.

FIXM 3.0.0 is the flight and aircraft flow information model that is standardized across the sponsoring flight authorities. The flight information is a model that contains data about supporting aerodromes, flights and addresses. Many of the features are inherited from ISO standards, however, the model does not appear to contain any objects from the ISO 19115-1/19157 models, therefore the quality of the flight information is not considered as part of the model.

Meteorological data is required in the aviation domain for flight planning, monitoring and traffic management use cases. Weather data can be complex, therefore it has been standardized as WXXM 2.0. The standard is built on ISO 19156 Observations and Measurements (O&M) and includes references to the AIXM model.

Certain products and services are particularly important to military use, for instance military charting, and Digital Vertical Obstruction File data for low flying. Although not a focus of this ER, it is recognized that military users have their own data quality requirements compared to civilian flight requirements that should be addressed if the model is to be used in military services. The military operates under a different rule set compared to the compliance requirements for civil aviation bodies. However, this moratorium on regulation is due to shortly expire and compliance will be required. Additionally, the military have a right to fly over the nation to which they belong, however they often have a duty of care to civil air users that is not reciprocated. Flight planning for military aircraft is often short notice, does not have a solid, defined flight plan due to the response to threats. Therefore, the military, as users of air information, are likely to require access to timely flight data and other air information at all times with the added constraint of the duty of care. An additional requirement is information classification and the military reliance on disconnected networks that must be considered by an implementing system.

Commercial providers range from joint public and private cooperatives to fully private companies and even individuals providing aviation services. As many organizations shift from AIS to AIM, commercial vendors will provide a range of enabling services such as aeronautical charting, flight procedures and airspace services to customers. Although ideally, these services would comply with new ICAO standards on the management, handling and aggregation of data for publication as aeronautical information using automated AIM systems, in reality, service quality is variable. For certain purposes, data quality is less critical, whereas in others, high data quality is mandated. Although data services within a particular commercial system may display internal integrity and compliance with International Civil Aviation Organization (ICAO) standards, where they may be exported, aggregated and disseminated further, there is the potential for loss of integrity, or a loss of an auditable data trail. Data quality metadata assists in maintaining high data quality standards through individual measures, but also through the concept of lineage, or a record of the data originator and processing history.

Commercial, government and open source communities also provide free-to-use services to the public for interest or information. These tend to be flight trackers, aggregated from Automated Dependent Surveillance - Broadcast (ADS-B) and RADAR data, or airspace exploration. Data here can be provided as is without detailed metadata and quality information as it is not designed to be used beyond being a public information service. This type of service is not considered as part of the abstract data quality model due to its lack of accreditation possibilities and public target audience.

The ICAO Convention on International Civil Aviation, also known as the Chicago Convention (Doc 7300)[3] is a major defining endeavor within the domain. It is useful to consider the requirements for data quality from official aeronautical regulatory framework for data quality measures that may be suitable to display in data discovery operations.

The object of the AIS is to ensure the flow of information/data necessary for the safety, regularity and efficiency of international air navigation. Annex 15 states the Standards and Recommended Practices for Aeronautical Information Services. Within the document are a several definitions that are useful for aeronautical service quality elements:

Data quality: A degree or level of confidence that the data provided meets the requirements of the data user in terms of accuracy, resolution and integrity.

Accuracy: Accuracy is defined as the degree of conformance between the estimated or measured value and the true value. With measured positional data the accuracy is normally expressed in terms of a distance from a stated position within which there is a defined confidence of the true position.

Precision: Precision is defined by ICAO Aeronautical Information Services Annex 15 as: “The smallest difference that can be reliably distinguished by a measurement process.” This refers to the reproducibility of a measurement rather than the number of decimal places that are reported. For example, measuring the dispersion of a spatial reference from the Global Positioning System (GPS) many times to understand the likelihood of successful repeatability. In this case if the values are close together then it has a high degree of precision or repeatability.

Resolution: A number of units or digits to which a measured or calculated value is expressed and used, note that this definition is akin to many users' view of precision.

Traceability: Ability to trace the history, application or location that is under consideration. When considering product, traceability can relate to:

Timeliness: Annex 15 of the Convention on Civil Aviation refers to the timeliness of aeronautical information/data services:

"8.2.5 Where automated pre-flight information systems are used to provide the harmonized, common point of access by operations personnel, including flight crew members and other aeronautical personnel concerned, to aeronautical information/data and meteorological information, the civil aviation authority or the agency to which the authority to provide service has been delegated in accordance with 3.1.1 c) shall remain responsible for the quality and timeliness of the aeronautical information/data provided by means of such a system."

Assemble: The annex also talks about assembling: A process of merging data from multiple sources into a database and establishing a baseline for subsequent processing. The assemble phase includes checking the data and ensuring that detected errors and omissions are rectified. This concept has parallels with conflation, a set of methodologies for combining data from different sources and the data aggregation concepts discussed in this document.

The European Commission mandated Data Quality using Regulation 73/2010[5] that was adopted on 26 January 2010. Aeronautical Data Quality (ADQ) regulation 73/100 states requirements on the quality of aeronautical data and aeronautical information for the Single European Sky (SES). The overall objective of this rule is to achieve aeronautical information of sufficient quality, accuracy, timeliness and granularity as a key enabler of the European Air Traffic Management (ATM) Network. In terms of scope, the aeronautical information data process chain extends from the original data sources (e.g. surveyors, procedure designers, etc.) through AIS and publication to the end users of the data for aeronautical applications. It is noted in this regulation that quality requirements relating to aeronautical data and aeronautical information are not always met within the European Air Traffic Management Network (EATMN), in particular the accuracy and integrity requirements. The shift to digital data products and services aimed to address these requirements, among others. As in Annex 15, data quality is defined as a degree or level of confidence that the data provided meets the requirements of the data user in terms of accuracy, resolution and integrity, with the same definitions adopted. Step P-02 - Data integrity monitoring states: “Data integrity requirements introduced by safety objectives must be measurable and adequate.”

Regulations stipulating data quality may also be used to deliberately block data, or to add lag. An example is the FAA Block Aircraft Aviation Request (BARR) program to limit aircraft data displayed via Aircraft Situation Display to Industry (ASDI). This is due to security concerns, and many commercial flight trackers have at least a 5 minute delay. From a data quality perspective, delays in data service have ramifications for different measures of quality such as completeness, but also provide challenges for products produced using aggregation or conflation techniques as different parts of a resultant product will likely produce data with heterogeneous fitness for purpose. Measurement of varying quality across an image is discussed in the Testbed 12 Engineering Report OGC 16-050 Imagery Quality and Accuracy.

There are definitions of data quality throughout the reviewed documentation, the two that are considered most relevant are as follows:

Throughout the review of aviation data quality documentation, mandates from different regulatory bodies, a review of the existing standards and engagement with stakeholders, the following concepts are required for an AQM:

The Accuracy, Currency, Completeness and Consistency (A3C) model quality framework was introduced by DigitalGlobe[7] as an instrument to compare imagery from multiple sources and determine the best fitness for purpose. This model was initially suggested in the Request for Participation, Annex B of Testbed 12 and was elaborated upon in the Imagery Quality and Accuracy ER. A first consideration of this work is whether the A3C quality framework is suitable to describe aviation data services. Aviation services are wider in scope than purely imagery data, they contain a mixture of features (vector) and coverages (raster) data that may represent flight data, aviation infrastructure, sensor data or meteorological data. However, the main classes of the A3C quality framework are generic and are still appropriate. The following sections discuss these classes and compare with the aviation data case.

The ISO standards coming from TC 211[9] form a collection of documents dealing with different aspects of Geographic Information and Geomatics. Specific ISO standards considered are:

ISO 19115-1 defines general-purpose metadata for geographic information. The metadata model enables definition of domain-specific user extensions based on a common pattern to facilitate implementation of software and services using those extensions. The purpose of metadata is to describe resources, whether they are services or datasets. In the case of data, generally this description remains with the data and does not change. It can be used both to describe the data for interpretation and for discovery, the focus of this section is to discuss the ISO 19157 elements, as they are most relevant to the aviation domain.

The quality of a service can be determined by metrics in the SDCM or WSQM QoS metrics, however in this ER, service level quality refers to the contained data rather than the quality of the service provide by the endpoint. Measuring data at a service level is concerned with describing a group of datasets provided by a service. This differs from aggregation of datasets, which is where features from datasets are combined to produce a single, new dataset with quality that varies across a single dataset. Service level quality in this instance seeks to describe how the quality of the datasets offered by the service vary and may include an aggregation of the quality metrics from the datasets.

Dataset quality is the main level of quality used in the aviation domain, however there is often a debate as to what constitutes a dataset. Taking into account the concepts described in the Metaquality section, the homogeneity of a dataset has an effect on the dataset scope or grouping of features. If a dataset is homogeneous, i.e. contains features or tiles of identical origin and vintage, then the concept of a dataset is quite simple as it is a group of common features. However, if the dataset is a mosaic of features that have different origins, then quality metrics are likely to be affected. Additionally in modern systems, a single feature held in a database may belong to many different datasets, for example, a single flight over the UK will belong to a dataset of all flights and a dataset of flight across the UK. A metadata record (along with data quality metrics) should exist for each of these datasets for discoverability and qualification purposes.

Measuring the quality of individual features or elements is useful in a variety of use cases, notwithstanding the complications of grouping heterogeneous features or tiles, feature level quality is useful in processes that involve conflation of datasets. This is largely done to visualize differences in source and quality across a conflated dataset, additionally it provides a metric of usability at the feature level, which defines how a conflated dataset maybe used operationally. In the aviation domain, it is likely that datasets are more homogeneous in terms of their timeliness and source due to the constraints regarding air safety. The introduced concept of precision has obvious descriptive power to features or even attribute level metadata.

According to ISO 19157 The quality of a dataset may be represented by one or more aggregated data quality results (ADQR). The ADQR combines quality results from data quality evaluations based on different data quality elements or different data quality scopes. Interestingly, a dataset may be deemed to be of an acceptable aggregate quality even though one or more individual data quality results fails acceptance (i.e. a weighted sum over a particular threshold). Whether this is acceptable in an aviation dataset is a key question. There are a number of considerations for creating implementation rules in the aviation domain:

Aggregation should only be used when compelling reasons exist. The meaning of the aggregate data quality result should always be made clear.

QualityML (Quality Indicators Dictionary and Markup Language) is a hierarchical dictionary of data quality terms that operate across the board of standards. Essentially, QualityML offers the user the ability to maintain the semantic descriptions of quality (and meta-quality) measures no-matter what profile of the standards they are encoded in. QualityML is important to consider in the abstract data quality model as it is likely how the eventual AQM will be expressed in an implementation. However, the implications of implementing the AQM in QualityML is discussed other ERs in the Testbed 13 aviation thread.

The Accuracy metaclass has the ability to describe the quality of positional accuracy, thematic accuracy, quantitative accuracy and accuracy of a time measurement. This class is not simply concerned with positional accuracy as might be expected. The ISO 19157 elements included are:

Each of these elements describe the closeness of each of the aspects of the data to the universe of discourse. There are extensions proposed to the model that are concerned with describing precision of different aspects of the dataset.

These added elements address the requirements of the aviation domain to understand the precision of different aspects of the dataset.

The Traceability of a dataset is used to understand the lineage of the resource including the originator/person responsible and the processing steps taken on the dataset to produce its current form. Note that an EX_Extent class is included in the responsible party information to account for heterogeneity within the dataset, that is, processing may have been completed on part of the dataset, for example and update and other parts of the dataset may not have been updated. Traceability includes:

The Resolution class contains information about the resolution of a dataset including its spatial and temporal resolution. These aspects are recorded using standard ISO elements:

The completeness of an aviation dataset aligns with the definition outlined in the ISO standards. The Completeness class includes the following:

As mentioned previously, timeliness is a representation of the concepts of currency and fitness for purpose and is described as part of the metaclass Timeliness. The Timeliness element uses the MDMaintenanceInformation class to describe the update frequency, date of last update and other information about the update. The class also includes a TemporalExtent to describe the time period. The metadata update frequency value is restricted by the MD_MaintenanceFrequecyCode. The class also includes an extent element to restrict the timeliness measure by a geographic region, this provides the AQM with the ability to record timeliness metadata in mosiaced datasets. Timeliness is described using the following ISO compliant classes:

Integrity is a metaclass that is concerned with the internal consistency of data according to its defined concept, schema, format and topology. The ISO standards contain elements and implementation guidance that is suited to recording this metadata, the classes included are:

The packaging information for the data is recorded in the GP_Fixity class. As mentioned in ISO standards review, GP_Fixity is a class to record information about a dataset’s authenticity through signing. The class simply records the information, it does not provide a methodology for implementing authenticity checks as domains are likely to have their own restrictions and implementing systems.

A founding motivation for undertaking this work is the SDCM does not contain a reference to data quality, only to service quality. Therefore, the AQM is related back to the SDCM according to the Data Entity class with the data quality of an asset offered by a service described by the Quality of data class. Figure 5 outlines the relationship between the AQM and the SDCM. A full explanation of the relationship between the AQM and the SDCM is explained in Testbed 13 FA002 - Data Quality ER.

AIXM is concerned with the information exchange of physical infrastructure in the aeronautical domain and as mentioned previously, inherits some of ISO19157 fields. The AQM is easily implemented within AIXM due to the interoperability with ISO, however the extensions to the ISO model also have to be considered. Additionally, of interest to AIXM users is the scale that the data are captured at, as this will have an impact on many other aspects of data quality.

Aerodrome information is largely static with only aspects of the features likely to be of interest in many use cases. Requirements for airport information for flight planning and monitoring maybe restricted to aspects such as runway length, capacity, flying height and local flight restrictions rather than the precise geometry measurements for the terminal buildings or the location of amenities. Considering these requirements, the abstract quality model is less concerned with timeliness in this use case as the data provided is static, it is concerned with precision as the geometry of obstructions is paramount to safety. The crucial aspect of data quality in this use case may be the authority who is serving or publishing the data, as there is likely to be authoritative sources who have the mandate to produce this data for use in a safety critical context.

Flight information has aspects in common with the air information use case and also with the weather use case. Unlike weather, flights are planned with a specific route, therefore, planned flight routes will generally reflect the routes actually taken by the aircraft. However, flights are not static and aircraft position changes over time, therefore there is a predictive element to understanding the data quality of flight information. A major consideration for data quality in flight information is the sensor that is used to monitor the flights in transit. There are several methods for monitoring flight information and include humans spotting planes and report their location to technological means of monitoring aircraft such as RADAR and imagery. In these cases, the AQM could be used to record the properties of the sensor that made the observation. These parameters could be used to discern positional accuracy and precision as well as the all important timeliness parameter.

Weather information is changeable and based upon predictive modeling. Understanding the quality of weather information has a fitness-for-purpose element to it, as it is incorporated into flight planning at different stages. Weather information is utilized at all stages of flight planning, an example usage is as follows:

These different stages have different data quality requirements, the abstract data quality model is suited to providing information at each of these times, however the requirements for some of the quality measures are likely to differ. For example, long range flight planning elements are by their nature, not reliable, however they may still be timely as the user can be working with the most up-to-date predictions. Additionally, long range weather predictions are unlikely to be localized down to a single aerodrome operating space, as long range forecasts tend to be regional. Highlighting these aspects in the abstract data quality model provides users with information they require to make a decision on the best data depending on what stage of planning they are at.