OGC Testbed-13: Quality Assessment Service ER

The Abstract Data Quality ER (FA001)[4] presents an abstract or conceptual model for data quality in the context of SOA services in general and OGC-compliant services in particular. This model articulates the concept of data quality as well as associated facets, concepts and relationships. It is based on widely-used industry standards and models, such as the Service Description Conceptual Model (SDCM) [8], the Quality Indicators Dictionary and Markup Language (QualityML)[7] and the ISO 19157[15] standard of the International Organization for standardization (ISO).

The Data Quality Specification ER (FA002)[5] presents a Data Quality Assessment Specification. This specification defines a set of data quality parameters as well as the methods and units of measure employed for measuring these parameters. The specification is information domain neutral, i.e., it specifies data quality characteristics and methods that can be applied to all aviation information domains: weather, flight, and aeronautical.

The following considerations were included in the Call for Participation (CFP) so as to help implementers to see the “whole picture”:

This table provides the formal list of requirements:

Currently, neither in the aviation domain nor the OGC standards baseline, is there an adequate mechanism to describe the quality of (geospatial) data provided by a service as a subset of overall Quality-of-Service (QoS) attributes/taxonomies. It is not possible to express the expectations of service consumers regarding the quality of data or for service providers to advertise their services from the data quality perspective. The claims of service providers regarding quality of data cannot be validated nor assessed due to the lack of governance tools for data validation.

The AQM forms a basis for implementing data quality services specifically for the aviation domain. Prior to this testbed and FA001, data quality services were reliant on the standards inherited into the three aviation exchange models such as the Aeronautical Information Exchange Model (AIXM)[11], Flight Information Exchange Model (FIXM)[12] and Weather Information Exchange Model (WXXM)[13]. These vary from not having any capability to record data quality to the data quality metrics as defined in ISO 19157[15]. Although the ISO standard is recognized and provides some of the required data quality functionality, it is missing the key concepts of timeliness and precision. The abstract quality model described in FA001[4] document introduces these concepts to the AQM in a standard-compliant way, which is used in the data quality assessment service specification.

The outcome for this engineering report is the description of an Application Programming Interface (API) and design for the data quality assessment service. The ER investigates the options, proposes a solution and explains the use cases with the service interface suitable to meet all requirements put on this service. Following questions have been answered in this ER:

The ER also provides the WPS 2.0 profiles for data quality assessment service implementations.

In the area of aviation data quality, this engineering report, together with associated reports, provides a general blueprint for the implementation of the quality assessment service. While the ER considers aviation data formats and quality parameters specific for that business domain, it also remains general and open for further extensions.

All questions regarding this document should be directed to the editor or the contributors:

This engineering report describes the practical application of several aviation metadata and ISO standards (from the area of data quality) for an interface to access the data quality assessment service implementation. The WPS 2.0 standard plays here an essential role as the OGC Web Services (OWS) compatible, common purpose, extensible interface for accessing of geospatial computation routines. Data quality assessment fits very well into the concept of web “processing” service.

The following are recommendations for future efforts in standardization, that have been identified during this testbed, towards establishing the standard for data quality assessment service implementations:

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

This section provides an introduction into stakeholders and basic functions provided by the ADQS.

This section provides the list of components recommended for the ADQS implementation. The architecture is based on two pillars. The first one is the Service Endpoint, which is responsible for mediation between service consumers and quality assessment logic. The second one is the Workflow Engine, which provides high implementation flexibility and service availability.

Essential for a successful implementation is the integration of the service front-end component and the workflow engine. The WPS 2.0 specification supports synchronous and asynchronous service execution offering the operations for metadata retrieval, task execution, status check, and result retrieval. A WPS-based service interface reduces options for workflow design and implementation because WPS 2.0 does not support arbitrary interactions between assessment processes and human actors.

The OGC WPS 2.0 Interface Standard provides rules for standardizing how inputs and outputs (requests and responses) for geospatial processing services can be structured in a standard way. WPS also defines how a client can request the execution of a process, and how the output from the process is handled. WPS also defines a standardized interface that facilitates the discovery of and binding to those processes by clients.

Processes can include any algorithm, calculation or model that operates on spatially referenced data. For example, a quality assessment process could be understood as an applicable calculation.

Publishing means making available machine-readable binding information as well as human-readable metadata that allows service discovery and use.

The version 2.0 of the WPS specification allows three types of input data:

The WPS does not specify the kind of backend business logic that could be implemented. Instead, it specifies a generic mechanism that can be used to describe and web-enable any sort of (geospatial) process. That also means that the service interface was kept generic, which makes it always applicable but also less expressive of one particular use case.

The main advantage of WPS is that it allows delivery of calculations (in this case the quality assessment) to consumers independent of the underlying software, implementation, algorithms etc. This independence helps to ensure the longevity of code.

Other advantageous WPS features are:

The WPS allows several different approaches for executing a quality assessment process:

The WPS 2.0 specification can be utilized for data quality assessment service (as a service endpoint blueprint) by exposing the following operations to service consumers:

The execution of quality assessment is then performed through the following interaction sequence that may be initialized by a WPS client and performed via a WPS front-end:

WPS discovery and binding mechanisms follow the OWS model [2] in that the WPS defines a GetCapabilities operation, and requests are based on HTTP Get and Post.

In case of asynchronous process execution, as it has been depicted on the second diagram, two additional operations are responsible for the execution monitoring:

Translated into the ADQS terminology, the usage workflow has following typical steps:

The structure of a service contract document (API) is depicted in the following UML diagram (from the WPS 2.0 specification):

To demonstrate the basic DescribeProcess syntax for ADQS, the following XML listing provides process description (response to DescribeProcess) for a minimalistic positional accuracy assessment service to assess the quality of data provided by some hypothetical flight track service. The input parameter is a flight trajectory encoded in the GML. This blueprint could easily be modified to use more appropriate AIXM 5.1 format:

A more advanced example would also contain a second input parameter for a flight number and a bounding box with geographic coordinates for a rectangular area of interest.

The diagram below repeats the WPS-based application’s component diagram provided above using the ADQS terminology.

A WPS 2.0 API implementation acts as a reverse proxy towards service consumers encapsulating backend service implementation details. Jobs are the WPS units of works, which are assigned to a particular assessment process instance. The WPS manages the assessment process execution lifecycle and in case of asynchronous data quality assessment evaluation offers an interface to consumers, which enables them to control and monitor running jobs.

This ER makes the distinction between two major types of data quality processes:

An example of the quality claims are metadata documents hosted in an aviation service registry. The data quality statements are expected to be embedded into an SDCM 2.0 QoS document, as it was proposed in FA002. Two types of output document are applicable for the data quality assessment service report, one for quality estimation and another for quality assessment.

Due to the universal applicability of QualityML, the assessment output document could be created by extending the input document and adding additional elements. The input document shall be extended in the way that for every quantitative or qualitative element provided in the input, an additional element with calculated metric values is set in the output. These metric calculations could be placed in original QualityML elements or they could be integrated into the output document separately. Further, the assessment output shall provide additional data elements for encoding of assessment relevant metadata. The information regarding the input data set (which is the object of the quality assessment) shall be integrated into the output document. The following list depicts the basic structure of the assessment report (output) document:

In the case of quality estimation, the input document shall contain all types of quality attributes that a service consumer might be interested to get the report about. The attributes shall be encoded using QualityML statements. The metric values shall remain empty. The resulting output document structure contains the QualityML elements from the input document with populated values for metric attributes. If the metrics are not specified in the input document, the assessment service might calculate values for all applicable metrics. In this case a metadata description shall describe the approach and be attached on the output document.

The code example below shows the minimalistic data quality document borrowed from the Quality ML web page. The metric for quantitative attribute accuracy used to make the quality statement is encoded using an URI (distributions/normal). This URI could be used during the service invocation to communicate the type of assessment required to be performed. A decision on which elements inside of the data set shall be considered for metric calculations has to be made based on the type of input data. For example, positional accuracy is related to geospatial information and in the case of AIXM 5.1 data quality assessment, for every single entity specified in the AIXM 5.1 the ADQS shall be filled up with information about elements that were used to encode positions.

The QualityML statements embedded into input documents shall be used by service execution logic to identify data quality assessment activities required to be performed for an individual assessment process:

The code example below demonstrates a simple QualityML output document containing a single metric. The highlighted area in the code points out to the two characteristic attributes of a normal distribution metric, which, if they are populated in the input document, represents the data quality claim made by service provider (the claim which shall be assessed by the ADQL), or if the values were omitted, that shall be the indicator for ADQS service execution logic, that those metric values shall be calculated and presented as part of data quality assessment.

The quality assessment strategy (quality parameters) could also be specified by service consumers using control attributes in service requests. A consumer specifies what to assess using control attributes or lets the service perform all sorts of assessments/investigations it is able to calculate. Therefore, the following are options for process execution control:

Another approach for assessment execution control would be based on having multiple service endpoints with support for different subsets of metrics. The service implementations are therefore provided in several versions of different complexity. Each one is equipped with a subset of quality metrics so the service provider can decide which one to apply. This approach seems to be particularly useful considering the overall number of possible metrics and computational effort to calculate them all. Full-quality assessment might not always be required. Having said that, a service might offer the following three assessment process types:

The light profile version would contain only the common metrics, the standard one would be extended with additional metrics, while the full version would host all metrics applicable for every applicable data quality assessment case. The decision of which quality parameters/metrics shall belong to which of types/categories should be made with input from domain experts.

The input/output document formats and the list of supported quality metrics shall be modeled using the WPS 2.0 DescribeProcess document and the concept of service profiles.

The processing service specification includes the concept of an Application Profile. It is a sort of process implementation blueprint. Processes that are advertised as compliant with the same Application Profile are intended to provide identical functionality. An Application Profile is essentially the same as the ProcessDescription document that is returned in response to a DescribeProcess request. The concept of profiles shall be utilized for the design of an aviation data quality assessment profile for WPS 2.0 as it was elaborated in the next chapter.

A process concept describes the functionality of a process or process group on a very high level. The WPS 2.0 standard recommends that this could be done using a Hypertext Markup Language (HTML) page. The high-level concept for data quality assessment is recommended to be based on the results of this engineering report and made publicly available as an HTML page hosted for example by an aviation service registry.

A generic process profile shall consist of an XML document similar to a process description but without data formats specified for inputs and outputs, and an HTML page with descriptions for the process itself and for the inputs and outputs. The formal generic data quality assessment process profile description does not exist for ADQS, so the recommendation is to create and publish the description in accordance with this engineering report and the requirements from the WPS 2.0 specification.

An implementation profile consists of a WPS process description and an HTML page with details about the implementation, as well as inputs and outputs of the process. The HTML page for the ADQS process profile does not exist yet and the recommendation would be to create and publish the process description in accordance with this engineering report and the WPS 2.0 specification.

The implementation profile extends the generic assessment profile and additionally defines the input and output formats. As a blueprint, the process description for a buffer WPS process from the WPS 2.0 examples folder was used.

To encode the atomic statements about the data quality the FA002 specifies the taxonomy called QualityML. A Quality element in QualityML is a combination of following taxonomy elements:

Quality metrics further include:

The combination of a quality domain and quality metrics are commonly known as quality measures. These concepts are taken from ISO 19139 and ISO 19115-3 standards.

Summarized, the data quality model is defined based on the QualityML, as well as ISO recommendations and extensions developed in Testbed-13 FA002[5] for the aviation domain. Here is an example of how to encode the WPS output with quality metadata in ISO 19139 metadata documents using the QualityML framework and the qualityML URIs. This format is suitable for initial data quality evaluation and not for assessment of claimed data quality by a service provider.

Here, some examples on how to encode quality metadata in ISO 19139 metadata documents using the qualityML framework and the qualityML URIs:

In a WPS profile the input parameters are optionally specified as complex types defined in external, imported schemas. For QualityML this option would mean that the corresponding XML schema shall be referenced in the profile document.

If we take a look at the WPS result above we will see that a lot of XML tags and quite complex structure is used to encode following:

Obviously, a lot of XML text was used to encode several simple attributes: the name of measurement, the method, the two values and the unit of measurement. Some information such as the measure (normal distribution) was redundantly encoded twice. The following code example, borrowed from QualityML, presents even more bloated data quality assessment in XML used to encode missing ratio of an element per 100 items (completeness omission)

Considering the two examples given above, alternatively, the QualityML elements from the corresponding XML schema might be encoded using the WPS 2.0 taxonomy elements. The motivation is to try to utilize the same taxonomy in a more compact format.

For quality measures we need at least:

The following code provides an example of how to use WPS instead of dedicated QualityML XML schema. The concept of 'allowed values' is used to incorporate QualityML taxonomy in the WPS service endpoint description.

In this listing, WPS literal data types are used to define the taxonomy of metrics and measurements methods instead of QualityML schema with semantically equivalent elements. This approach might be preferable in combination with WPS because it better utilizes the expressiveness of the WPS and provides better readable service contract document.

WPS 2.0 offers a standardized service interface which has two characteristics of interest from the extensibility point of view. The WPS was designed using a generic interface specification, which basically allows any kind of input data.

The specification defines three types of input data:

The bounding box based input is standardized as the smallest common denominator between all potential OWS services.

The literal data, simple data types, varies depending on the service implementation and its contract. Literal data types can be used for interface definition in the DescribeProcess document. Simple data types are standardized in the sense that there is an enumerated set of allowed types (integer, double, datum, string) and values they could take.

Complex data, provided as input data, is a placeholder for any sort of structured data. This could be a document described by an XML schema (for example an AIXM or FIXM document), for example.

Combinations of all three input data types to describe the input are allowed and definitively imaginable. For example, we could have a service with simple data definitions used for quality attributes, any kind of structured document might be used for assessment input data and the bounding box could carry the spatial information in order to reduce quality assessment evaluation to one specific area.

Obviously, a WPS endpoint allows the coexistence of several different service contracts bound on different background quality assessment processes. Service instances could be configured to work with the same background process but to be described using a different service contract. They also could be deployed in several operational versions. As part of the deployment process, every process instance will get a dedicated DescribeProcess document with a well-defined input data set.

The data inputs feature from DescribeProcess allows a lot of flexibility but also introduces a certain overhead and makes the service consumption more difficult. This is due to the need to invoke metadata operations such as GetCapabilities and DescribeProcess in order to check the options and figure out how to consume the service.

Digital NOTAMs are expected to be delivered in an AIXM 5.1 message envelope containing aeronautical objects they refer to. Since the NOTAMs have different originators and could be delivered with or without infrastructure entities or aeronautical objects, some of them might be considered incomplete for particular use cases (fit for purpose).

Quality assessment would follow certain predefined rules of completeness and monitor the absence of certain objects in Digital NOTAM messages.

Possible outcome: At the end, the completeness could be described as a missing ratio, for example 2%, which would signal that 2% of all NOTAM messages delivered by service have been incomplete, because they were missing certain related elements.

Metadata could, for example, specify (among others) a data quality attribute such as the mean absolute deviation: MAD =.025 sec. for timeliness.

Let’s assess the timeliness for a live flight tracker (of commercial airline flights) service. Assume that the service claims (in its data quality attributes) for a chosen flight to provide position information every second with mean absolute deviation in timeliness of MAD=0.25. In other words, more than 99% of positions are going to be delivered (pushed) to service requestors within an interval between 0.75 and 1.25 seconds.

Quality assessment: In order to assess the timeliness performance, several flight tracks will be collected (in some observation time or permanently). As part of the assessment process, an algorithm component of DQAS responsible for timeliness assessment will calculate the times between provided positions (or retrieved from position items if they are part of them). Mean values will also be calculated, as well as the mean absolute deviation from that value.

Possible outcome: the service provides positions with time accuracy of one second (mean value) with dispersion of 0.23 seconds. That means that 99.3% of values were sent within a time range of 0.76 - 1.23 seconds. Final conclusion of the assessment process would be that the result was in accordance with claimed quality of data!

Quality metadata could also specify (among others) a data quality attribute such as the positional accuracy for a flight tracker service. It is about the deviation of a provided flight trajectory from the flown one.

The quality assessment might also include the common flight trajectory characteristics of commercial airline flights. For example, the trajectory cannot (it is very uncommon) have big differences in speed, altitude or heading between time intervals. Such deviations might indicate bad quality of flight tracking positional information.