OGC Testbed-13: Data Quality Specification Engineering Report

Usually, feature instance level metadata and even attribute instance level metadata is allowed by adding fragments of metadata types to the other attributes in the data model of the feature types.

The exploration of the Aeronautical Information Exchange Model (AIXM) 5.1 reveals that there are some quality measures that have already been considered for some feature types.

Most feature types in AIXM are derived from basic geometric primitives or from the elevated version of them. Next figure shows as an example one feature type for each geometric elevated primitives.

In addition to the horizontal and vertical accuracy, specific feature types for aviation objects can carry other quality elements for aspects other than positions. For example, Accuracy is included as an attribute of:

There are also some other feature elements that can be somehow related to quality, for example the CodeIntegrityLevelILSBaseType, defined as "A coded value indicating the quality which relates to the trust which can be placed in the correctness of the information supplied by the ILS facility", is included in Navaids Points|Navaids|Navaid.

On the other hand, the Meteorological Community Exchange model METCE, within the Weather Information Exchange Model (WXXM), talks about resolution in Procedure|MeasurementContext: resolutionScale, defined as "The attribute 'resolutionScale' specifies the smallest change (e.g. the 'resolution') in property value of the 'measurand' that is intended to be measured within this procedure, using the unit of measure 'uom'. It shall be provided as a scaling factor, e.g. scale = -2 implies a precision of 100 units.."

In the models, there are currently no indications about other data quality aspects regarding quality facets such a timeliness, etc.

A dataset provided by a service can have quality measures attached. These quality measures will be quantitative aggregations of the quality of the individual features included in the dataset, as it is explained in the sections below.

Services that are providing data can provide an overall estimation of the data quality of the datasets available in the service.

Positional accuracy in the aviation context is concerned with the recorded position of each feature compared to its actual position. Note that this also includes a measurement of precision, a separate measure for the aviation domain.

Within AIXM models, feature positional description use 2D elements (Point, Curve and Surface) or its elevated versions (ElevatedPoint, ElevatedCurve and ElevatedSurface). All of them have horizontalAccuracy and the latter also have verticalAccuracy. Thus, elevation accuracy is decoupled from horizontal accuracy, so that pure 3D quality measures do not apply to this model.

All types of accuracy are described as quality elements in this quality of data model. Quality category is used to describe (in an enumeration basis) the different quality elements described in ISO such as positional accuracy, quantitative attribute accuracy and so on, as well as the new elements described in this document such as positional precision. The list of category values can be seen in the SDCM extension model explanation, in the next section.

Feature level horizontalAccuracy (for points, curves and surfaces) is defined in AIXM models as "The difference between the recorded horizontal coordinates of a feature and its true position referenced to the same geodetic datum expressed as a circular error at 95 percent probability".

Feature level verticalAccuracy is defined as "The difference between the recorded elevation of a feature and its true elevation referenced to the same vertical datum expressed as a linear error at 95 percent probability".

There are two main features within the models, those representing actual phenomena on the Earth (e.g. tower), and those representing human conventions (e.g delimitation of areas tagged as forbidden).

On the first group, a validation campaign may be done (sampling on the ground) to check the correspondence of the dataset position for these elements, and their real position. These differences may be aggregated to compute a dataset quality measure Circular Map Accuracy for the whole dataset. For example, an AeronauticalGroundLight (that is used for markingTheSiteOf within AirportHeliport) does have an elevatedPoint position that may include its 95% circular error (CE95) in the horizontalAccuracy element and an elevationAccuracy that may include its linear error at 95% (LE95). Several CE95 and LE95 values (for several AeronauticalGroundLight features) will be respectively used to compute the dataset CE95 and LE95 values.

Sometimes, it is not possible to provide a numerical value for a quality category as the quantitative uncertainty of the instrument used to do the measure is not available. Nevertheless, usually most instruments have a known order of magnitude precision, thus it is useful to provide the instrument name as an indication of this precision. For example the position of an aircraft can be measured using different instruments on board or other estimations in the positional precision element. Typical examples are: GPS, ADS-B (meters), ADSBLostCoverageEstimation or TimeSpeedDistanceEstimation (100km or more).

A descriptive result for a data quality element can also be used to describe the spatial distribution of data quality. For example, it could be good to have a way to state that horizontal accuracy of the elements next to or inside an Airport are defined in a more certain way than the obstacles around it. This is the typical use of a descriptive result as defined in ISO 19157.

Thematic accuracy, is defined as the accuracy of attributes and it depends on the attribute type. In ISO it consists of three data quality elements: classification correctness, non-quantitative attribute correctness and quantitative attribute accuracy.

Regarding quantitative attributes, there are several relevant variables in the AIXM models covering angles (magnetic variations, bearing, slope), sizes (height, width) that are difficult in a single meaningful overall quality measurement. We suggest thematic accuracy is only provided for the dataset when it includes a coherent set of variables such as magnetic variations. The attribute that is being assessed should be defined in the scope of the quality element (or in the scope of the whole quality of data section if needed).

On the other hand, to cover classification correctness, the usual approach uses misclassification matrices to indicate thematic accuracy. An example of this can be giving the misclassification matrix for the attribute type in Routes/En-route/Route/type within AIXM Route feature:

Moreover, as recognized in OGC 17-032, not only the usual misclassification indication but an indication about unclassified items may be helpful. This can be described using NodataAreas metrics in QualityML, understanding the percentage of unclassified items is similar to percentage of unclassified area in a classification image:

The resolution class contains information about the resolution of a dataset including its positional (or spatial), vertical, temporal and attribute resolutions. ISO describes spatial resolution through the MD_Resolution class and temporal resolution through the TM_Duration class.

SDCM quality of data model describes a general Quantity class that can be used to describe the value and units of measure of any of these resolutions in a general way, and that is specified for each of the four resolution types. Positional resolution has also a equivalentScale element in order to be able to describe the most important elements in ISO (as distance, angularDistance and levelOfDetail can be described within positional resolution using the value and the needed units). For temporal resolution, the value and units are enough to describe the elements contained in ISO TM_Duration.

Regarding attribute resolution, the element resolutionScale in Procedure|MeasurementContext from the Meteorological Community Exchange model METCE (within WXXM) is another indication of resolution, and thus should be "translated" to resolution quality element within SDCM extension model. This is a direct translation as it is a scaling factor and thus the resolution value can be computed using the formula:

And for example for a feature with a resolutionScale of -2, an attribute resolution with value 100 and attributes units of measure should be defined.

Traceability describes the ability to trace the history, application or location of that which is under consideration. The closest match within ISO standards is Ll_lineage element. Information on the originator/person responsible should be included.

A first approach to lineage of a dataset would be given as a statement. This is a simple way to describe traceability but it is not the most recommended, as it does not easily state sources used and process steps related to the dataset history. According to ISO, a full flexible system is described where several options are possible:

Even though the power of ISO schema should be recognized, allowing hierarchical definition of process steps and sources, creates a very flexible situation for each dataset of each service that would lead to a probably complex situation with hardly comparable datasets within a service or among services. Moreover, the originator of the data entity is also recognized in OGC 17-032 as relevant for traceability.

Thus, the proposal for SDCM extension described in this ER is to recommend to include a flat structure for sources, process steps and originator in the traceability section. To follow this aim, current elements on SDCM are reviewed in order to decide if they can be used to describe these elements, or if new elements are needed.

SDCM Operation element may seem a good candidate to be reused (even if an extension is needed) to be mapped to ISO processStep element. Unfortunately, stepDateTime and processor are missing in SDCM operation class and these are important elements to describe traceability as describe who and when. Moreover, several Operation elements make no sense on a process step description. Thus, a new Process Step element in SDCM is created to fulfill the process step requirements, including process step description, date and time, organization carrying out the process and an optional reference to the process description. The processor description can be omitted if it is the same as the data originator (or even the service originator). The process step can refer to a process description document, similarly to the data description document from a data entity.

Regarding source descriptions for the lineage and traceability, it can be done reusing the "Data Entity" class within SDCM model and then describing the name and description of the Data entity and giving a reference document as well. The extended SDCM model includes a Quality of data section that is aggregated to Data Entity, the source description could also include its originator or its spatial resolution through Data quality sub-elements.

As an example, a service can provide a feature collection composed of an airport description, the approach and departure procedures, and obstacles around the airport. This feature collection conforms to a data entity delivered by one service. In the Quality of Data entity, the traceability of this feature collection can be stated as:

A second example is related to gridded weather products such as the ones delivered by CSS-Wx Web Coverage Service (WCS), that enable National Airspace System (NAS) systems to access high-resolution aviation weather data to meet their individual needs and to support NAS operations.

Completeness describes the amount of data in a dataset according to a comparison with the expected data, it is described using a measure of commission and omission. Completeness may be described at a dataset level or at feature level, for example regarding omission and commission errors among categories in a classification.

Several measures are described in ISO 19157 to describe completeness. The measures they represent use mainly excess, duplicate and omission measures. QualityML groups and extends the list of measures in order to aggregate the same concept but different metric. All the measures related to the same quality measure are grouped and use a metric called items which results can be expressed as a boolean, count or rate. In fact ISO 19157 suggest several options, in this case for the rate elements, when states that "[Error rate / Correct items rate] can either be presented as percentage or as a ratio. The value unit in the quantitative result (see 7.5.4.2) can be used to specify that the result is presented in percentage or as a ratio". To standardize these options for the rate as well as to combine the other two options (boolean and count), QualityML describe the Items metrics as a choice among "indicator" (for boolean), "count" or "rate". For the last one a parameter is described in order to include the maximum value of the rate. Thus, a value of 100 in this attribute will be used to express that the value is a percentage. Default value for this attribute is 1, representing a pure ratio.

Moreover, usually measures based on errors and on correct items are described in ISO 19157. Both definitions are exactly the same, the only difference being "which elements" the measure is counting. This, in QualityML is described by the Domain of the Quality measure, allowing then a higher aggregation schema relating several ISO measures to the same QualityML measure with several metrics and domains:

These measures are mainly related to a dataset level, as they describe the differences among the elements described in the dataset and the elements in the universe of discourse, finding those that are missing, duplicated or existing only in the dataset. As an example, for a dataset describing obstacles in an airport, an omission indicator may be described as:

Time is an essential aspect of the aeronautical information world, where change notifications are usually made well in advance of their effective dates. Aeronautical information systems are usually requested to store and to provide both the current situation and the future changes.

In order to satisfy the temporal requirements of aeronautical information systems, AIXM must include an exhaustive temporality model, which enables a precise representation of the states and events of aeronautical features. A general temporal model should be uniformly applied to all aeronautical feature types and the temporality concept should be abstracted from the task of modeling object properties. At the conceptual level, the model should describe the temporal evolution of the features, as they occur in the real world.

The AIXM Temporality Model describes two levels at which aeronautical feature instances are affected by time: 1) Every feature has a start of life and an end of life; and 2) The properties of a feature can change within the lifetime of the feature[3]. It is considered that any feature property may change in time, except for the global unique identifier. This is a key assumption of the AIXM Temporality model.

Within FIXM models, there are several elements describing time, such as runwayTime (in FIXM 3.0.1.Base.Aerodrome.RunwayPositionAndTime.runwayTime) or standTime (in FIXM 3.0.1.Base.Aerodrome.StandPositionAndTime.standTime). FIXM also includes the Base.Time schema that provides representations for time elements.

What AIXM temporality and FIXM models do not cover is the accuracy of a time measurement, that is fixed to 1 minute, as the validTime on the model is a date and time element. The accuracy time measurement or the precision of time measurement could be described for the whole dataset or for several temporal elements described in the AIXM or FIXM models.

When accuracy elements are described within AIXM and FIXM models, the selected measure is related to 95% probability (as explained for example for trueBearingAccuracy attribute in AIXM Navaids Points|Navaids|Azimuth). Thus, the recommendation to describe temporal accuracy is to use "accuracy of a time measurement" also with 95% probability, for example:

A temporal precision element has been also added as a quality category as suggested by OGC 17-032.

Timeliness is a representation of the concepts of currency and fitness for purpose and is described including date and time of capture, maintenance date and time and maintenance frequency. Using these elements a dataset can be described in terms of when it was produced, and whether it is valid.

For a dataset that is updated quarterly, for example, capture date, last maintenance and maintenance frequency can be described.

Moreover, for a service, an aggregated measure related to timeliness can be computed if a requirement on timeliness is defined, such as that this data should not be more than a certain number of days old. This could be described using a Temporal validity data quality category, using an 'item' metric (a boolean indicator, number of items or rate) and a domain requirement to define the time limit to consider and element to be conformant or not to this time requirement . To describe the conformance or non-conformance domain requirements, the domain can describe a range ("domain min" and/or "domain max" parameters) to define the requirement, as needed.

Using the AIXM temporality model, several data quality measures can be computed to describe the temporal validity of the dataset, for example describing if there are elements that have already ended their lives (and thus, do not follow timeliness requirements).

For example, if the feature requirements considers that features older than 15 days are not current enough (so they do not conform to the rule), a quality indicator with a temporal validity category can report the rate of elements non conformant with the domain with an specific requirement as a parameter in the domain of the quality measure (3% in this example):

Within the WXXM model, there are also highly dynamic features that should be properly described in order to be able to assess its timeliness. Inside the model, there is an AIRMET package that reports the occurrence and/or expected occurrence of specified en-route weather phenomena which may affect the safety of aircraft operations, and of the development of those phenomena in time and space. These weather phenomena are reported as impacted regions of airspace.

This package contains candidate representations for eventual adoption by ICAO Meteorological Information Exchange Model (IWXXM). Representations are based upon ICAO Annex 3 Amendment 76 / WMO No. 49. These representations should be considered unofficial until incorporated into IWXXM. Each observation/forecast phenomenon includes its own period of validity for described meteorological conditions, which is represented as the O&M Observation validTime. These elements can be similarly used to compute aggregated measures such as percentage of features covering a certain time requirement (like the last example).

Integrity describes the degree of assurance that can be given that the dataset has not been altered or lost since creation or update from the required body. There are several commonly used strategies to describe and ensure data integrity (some recognized in ICAO Annex 15 or other standards [5]), and those are included in the Integrity class in the model:

Moreover, several quality elements can be described in order to include the audit results as well as other integrity parameters related to logical consistency elements in ISO (and its four types) and temporal consistency:

There are several qualityML measures and metrics that can be used to describe consistency quality parameters, all of them using items measure metrics, for example:

The quality of service class could be applied directly to Quality of Data. This approach is the simplest one in the way that the model is only slightly modified, but no new classes or class modification would be needed. On the contrary, this approach maybe too simple to contain all the elements or to provide enough semantic information about the quality measures needed to describe aviation services in general, and specifically those registered that includes a Quality Element in NASS services.

In this case, SDCM extensions are more important. The idea is to modify SDCM following ISO 19115 and 19157 concepts, but keeping as much as possible SDCM simplicity. Instead of directly adopting the ISO 19115 and 19157 classes, a reinterpretation (usually simplifying the ISO classes) is suggested.

A new "Quality of Data" class is created that includes several child elements:

The new element "Quality of Data" can be used to illustrate the level of granularity "dataset" or "service". Details are described in next sub-sections.

OGC 17-032, OGC® Testbed-13 Abstract Data Quality ER, explained how to describe aviation concepts using ISO elements. With this approach an XML file describing the full ISO metadata record for a dataset (or service) may be described.

SDCM v2.0 "Data Definition" element on "Data Entity class" could be used to link to the full MD_metadata class.

The drawback of this option is that ISO standards are quite complex and several mandatory information elements that would be required are potentially not relevant or it’s harder to find that in a simplest model that will be more easily comparable.

The complete UML model for the SDCM is shown in the next figure.

Within the new Quality of Data class, an element "scope" is defined. This element is intended to include the description of the scope of elements to whom this quality information should apply.

Typically, "dataset" level is the proper value if this quality report is describing the whole dataset, but sometimes an attribute or feature type may be specified if this quality report only applies to them, for example "FlightObject/Flight/EnRoute/Position/AircraftPosition/position".

The first element inside Quality of Data is Quality element. This element is intended to describe quality reports about the dataset, feature or attribute. It is inspirited in ISO 19157 DQ_Element.

The element contains a quality category, that should be selected from the namesake enumeration. Values for this enumeration come from ISO 19157 DQ_Elements subclasses, as well as from OGC 17-032 or this ER proposals:

The second element is quality scope. This only needs to be populated if its value is not the same as the one for the general "scope" element (described in the previous section). This allows, for example, the generation of a single "Quality of Data" section for the whole dataset, that includes "Quality elements" for the whole dataset (thus "quality scope" will be not defined) or other for specific features or attributes (and thus "quality scope" will be described for them).

The result of the quality element can be described as three options:

When a quantitative result is provided, several elements should be described:

More information on QualityML use to describe measure name, domain, metrics and values (including parameters when needed) can be found in OGC 16-050 Imagery Quality and Accuracy ER.

The resolution classes contain information about the resolution of a dataset including its positional (or spatial), vertical, temporal and attribute resolutions.

ISO describes spatial resolution through the MD_Resolution class and temporal resolution through the TM_Duration class.

All of them are specified from a general Quantity element, and thus include a value and a unit of measure. This is enough for vertical, temporal and attribute resolutions. For positional resolution, also an equivalentScale element is added in order to allow describing resolution in this approach, like ISO standards recognize. Note that other options described by ISO, such as distance, angularDistance and levelOfDetail can be described using the general Quantity class, describing the proper units.

The traceability class includes a first textual element, statement, to describe in a general way the lineage of the element it is describing (usually a dataset). Moreover, traceability class has three elements that can be described:

Several data sources can be described as being part of the dataset creation. The data source is a specified class from Data Entity class, and thus it may have, if necessary a quality of data section to describe its traceability or other quality elements.

Several process steps describing the algorithms or processes that had been applied to obtain the dataset can be described. A description and date-time of execution needs to be described for each process step. Moreover, the organism carrying out the process (processor) may be described. It can be omitted if it is the same than the data originator (or even the service originator). A process description document can be also referenced.

Finally, the originator of the dataset may also be described as a part of the lineage. It can be omitted if it is the same than the service provider.

Both originator and processor classes are specified from the SDCM Organization class and thus can be described with a name, description and website, and at least one point of contact.

Timeliness class on SDCM extension is intended to describe the data and time of the information, the date and time of its maintenance and also its expected maintenance frequency. Possible values for the last element are the ones in the ISO MD_MaintenanceFrequencyCode codelist.

Moreover, and also very important especially for dataset and service comparison, timeliness is related to quality elements categorized with time validity.

Integrity class includes several elements to certify integrity of the data and data authorship, following ICAO and widely used rules:

Moreover, and also very important especially for dataset and service comparison, integrity is related to quality elements categorized with several consistency categories, i.e. conceptual consistency, domain consistency, format consistency, topological consistency and temporal consistency.

The first set of quality parameters refer to the completeness of the server and datasets documentation, taking into account how many datasets of the service provide a specific data quality element.

A similar approach was previously used in GeoViQua project on the description of the GeoLabel, that is a graphic representation (i.e., a static image) which generated for each dataset based on the quality information that is available for that dataset. This idea evolved and has been later used as a voluntary label part of GEO Data branding strategy to give visibility to the effort data providers put into making their processes conformant with the Data Management Principles (DPM) in the intergovernmental Group on Earth Observations (GEO).

In the same line, NOAA defined and uses the Completeness Rubric in addition to ISO compliance, to provide an extra level of assessment to help metadata authors provide more thorough information about their data [12].

Next table describes proposed Quality of Service parameters based on Quality of Data parameters and describing completeness about the quality of the data documentation. There are several parameters that describe completeness of different aspects of the SDMC data quality extension. One table is presented below for each aspect of the data quality extension.

Of course, a high value on these indicators it is not per se an indicator that the datasets served by a server are better than others, as the presence of documentation has nothing to do with its value (for example a higher or lower attribute accuracy) or its usefulness (having a process step description that says "unknown" or an unintelligible character string identifying a process in a certain software may not be very informative). This needs to be complemented with some quantitative parameters (next section), when possible.

The second set of quality parameters are those quantitatively describing services and their datasets. Several measures can be computed using dataset values for a certain quality parameter, in order to give some insight on how to compare services regarding the quality of its datasets.

For many of the parameters a set of summary statistics (or metrics) can be provided, each of them bringing its approach to the measure. The most common value to summarize is the average (or arithmetic mean) but the count of values is also meaningful to know how many elements this value is representing. The median is also interesting as it may differ from the average depending on the data distribution. Minimum value and maximum value are helpful in order to describe the maximum range of the values for this variable in a certain service. Finally, the standard deviation is also representative as a dispersion measure and, thus, to obtain confidence intervals centered on the average (if a normal distribution is assumed).

As this set of parameters can be used for several quality parameters, its definition is presented once in the next table and then referred to from the table describing the QoS parameters proposed:

The next table describes proposed Quality of Service parameters based on Quality of Data parameters and describing summary metrics about the quality of the data within a service: