Testbed-12 Imagery Quality and Accuracy Engineering Report

The A3C defined the 4 quality classes (Accuracy, Currency, Completeness, and Consistency) as:

A3C model is designed with the fit for purpose in mind and includes elements that are considered as data quality by ISO 19157 complemented by other aspects that can be found in other classes of the more general ISO 19115-1 metadata describing the datasets.

This ER links the A3C framework to ISO 19157 and QualityML vocabularies, and proposes encodings compatible with common metadata standards. In addition to this ER, implementations of this combined framework have been developed in Testbed 12 such as a WMTS server that implements the time accuracy part and a WCS that provides support to the complete framework.

Different sources of information come with different location accuracy (focusing in horizontal accuracy).

One of the common indications to define this accuracy is the CE90. It is defined in the ISO 19157 in Table D.46 — "Circular map accuracy standard".

For example, in the constellation of DigitalGlobe, the older satellites (Quickbird and Ikonos) have a positional accuracy of 20m CE90. In the new generation, they have a positional accuracy of 3m and in the future we could have 1m for satellites orbiting at 400 miles.

To be able to estimate this positional accuracy, several factors have to be taken into account giving us different approaches to determine positional accuracy.

It is important to clarify that the same quality measure would be used for any of these approaches, thus it is also important to provide information on the evaluation method of a particular metric.

The immediate aspect of currency is how recently the scene was taken. This can be found in the time stamp when the image was collected (initial and end time). If the image is being used as the "current" one, this introduces uncertainties in the features we are seeing in the images, if they are not recent enough (and some elements may have changed or moved).

A more elaborated use case with currency issues appears with temporal uncertainty related to a large image created by mosaicking multiple paths and scenes. For example, temporal accuracy can be provided as "90% of the mosaic is as current as 3 days or less". The temporal uncertainty can be bigger in mosaics over large territories (e.g. the African continent) . It can also be described as an overall measure (aggregation: "no pixel is older than a value t"). This can be provided at the pixel level as well.

This is in line with the WMTS service that is developed in this Testbed 12 and explained in WMTS Time Accuracy. In this case each, tiles can be tagged with a time accuracy estimation. For low scale tiles we will have time accuracy estimation for a small area. For higher scales we will have more aggregated values and in the extreme we will end in a scale where the whole extent fits in a tile and in this case the tagged quality will be the global time accuracy or the composed mosaic.

This is particularly important for the the disaster management use case, where currency is the most important factor.

This is also related with mosaics of large areas (e.g. a mosaic of all African continent). For large extension it could not be possible to generate a completely cloud free image. In this case the percentage of the data tagged as NoData can be one of this metrics (e.g. 0.5 % of area is missing). We can go further defining the percentage of the mosaic covered by clouds. We could do the same for cloud shadows but this is much more difficult and the industry is not able to report this yet.

The presence of cloud information can also be exposed by a WMTS. We could try and use this information in the WMTS. Some times there are products that have a "cloud" raster layer. Some other times the clouds are stored as polygons in a database.

Consistency in a GIS vector context is easily understandable and measured through a conformity test to the data model specification. In a remote sensing context there is no theoretical pattern to compare to, and only a reduced number field measures (e.g. field radiometry records) are usually available to correct image radiometry and make it consistent over time.

The weather and atmosphere status changes over time. Important factors that influence the image quality are the presence of aerosols and water vapor. This can be different even in areas of a single scene and changes the quality of portions the image.

There are different levels of radiometric corrections involved:

Consistency may be intended to be within a single image, the whole product series (all the historical archive of the instrument), the complete constellation (among Worldview satellites) or even within different constellations (e.g. Landsat 8 and Sentinel 2).

According to ISO 19157 there are seven quality elements (or classes) describing certain aspects of the quality of a geographic dataset:

These quality classes (or elements) can be elaborated in several quality indicators (or sub-elements).

The definition of DQ_Element and its classes and sub-classes as described in ISO 19157 can be seen in next figure:

When describing the quality of geographic data, different quality elements and different subsets of the data may be considered. In order to describe these, data quality units are used. A data quality unit is the combination of a scope and data quality elements.

The scope of the data quality unit(s) specifies the extent, spatial and/or temporal, and/or common characteristic(s) that identify the data on which data quality is to be evaluated. One data quality scope shall be specified for each data quality unit.

Moreover, an evaluation of a data quality element is described by the following:

Thus, to describe a quality subelement 7 items should be provided:

Results can be:

There are several non quantitative elements on the metadata that include information somehow related to data quality:

In FP7 GeoViQua (http://www.geoviqua.org) the usage concept was extended to geospatial user feedback (GUF). GUF is a form of metadata derived from the experience that users gain by using the data. The GUF.SWG developed an standard in two separate documents:

Recently, the OGC TC approved these two documents as international standards. This work is probably out of scope in this ER but is worth to mention it for completeness.

There are some elements that can be used for "fit for purpose" assessment but are not considered parts of the data quality in ISO:

A3C accuracy can be expressed using DQ_PositionalAccuracy quality element from ISO 19157 (including AbsoluteExternalPositionalAccuracy, RelativeInternalPositionalAccuracy or GridedDataPositionalAccuracy). This is valid for X, Y. If the CRS is a 3D CRS, Z dimension can be also included. If Z is considered an attribute, ISO 19197 includes a DQ_QuantitativeAttributeAccuracy that could also be used the describe, for example, the elevation accuracy in a Digital Elevation Model (DEM).

In the original definition of the A3C model in the Annex B of the Request for Quotation, there is no mention of thematic accuracy. This could be an important component for vector data but also for imagery representing quantitative variables (e.g. temperatures) and categorical variables (e.g. Land cover classes).

So that, proposed measures will include thematic accuracy classes: i.e.: DQ_QuantitativeAttributeAccuracy, DQ_ThematicClassificationCorrecteness and DQ_NonQuantitativeAttributeCorrectness

Currency refers to the temporal extent of the imagery products used to cover the associated area, since multiple dates of images are typically required to cover a large area. This is covered by ISO 19115-1 in EX_Extent and the specialization EX_TemporalExtent.

In addition, other elements of the ISO 19115-1 model could be used to reflect the frequency in which the producer plans to update the dataset. This is not directly related with the currency of the data, but can be a proxy to it, in case the EX_TemporalExtent is not indicated. It is clear that if the producer intends to update the dataset every 15 days, the probability of having a current dataset is higher than if the maintenance is planned for each 10 years. Please note that the use of these elements to infer currency is not perfect: even if the dataset is updated frequently, the updated version could be another dataset and not the one at hand.

In the figure below, resourceMaintenance points to a MD_MaintenanceInformation with:

Data quality classes can also be used in combination with EX_TemporalExtent. This is recorded using DQ_TemporalQuality. Temporal quality can represent different aspects concerning the temporal tags of the data, exemplified in the following use cases:

ISO defines completeness as commission (DQ_CompletenessComission) and omission (DQ_CompletenessOmission).

In imagery products, cloud cover is usually documented as quality flags adding a quality band. A quality band can be added defining a quality band in MD_Band (new in ISO 19115-1) or by adding a quality measure with a QE_CoverageResult (ISO 19115-2) (alternatives are described in Pixel level Quality).

In some cases, the sensor specifications forces the collection of data with geometrical restrictions that can leave gaps between scenes or paths that will create data omissions. For similar reasons, the revisiting time of some zones of the Earth in satellite imagery coming from near polar orbits is less frequent in the equator leaving gaps for some time periods. Other reasons for omissions are the lack of some bands due to sensor failure. In some situations measures can saturate the dynamic range of the sensor or the digital representation of them resulting in omission of information.

On one hand, the description of the radiometric processing performed over the image should be documented on lineage (statement or processStep). This metadata element can help understanding the consistency that each image can have related to the whole time series.

Moreover, DQ_ConceptualConsistency and DQ_DomainConsistency can be used to record inconsistencies over time that can have its origin in lack of good calibration of the sensors and bias. This will generate inconsistency of colors and relative inaccuracy over time. Temporal series can be built combining old and new platforms and sensors resulting in non-homogeneous results. Spectral and spatial error propagation from collection to production can also influence in consistency (in particular if processing chains change over time).

In 2009, OGC announced the release of UncertML as a discussion paper (http://www.opengeospatial.org/pressroom/pressreleases/1002). "UncertML is a conceptual model and XML encoding designed for encapsulating probabilistic uncertainties and may be used to quantify and exchange complex uncertainties in data. Most data contains uncertainty arising from sources such as measurement error, observation operator error, processing/modeling errors, or corruption. Processing uncertain data propagates and often increases uncertainty. Thus there is a need for a standard way of characterizing uncertainty that is readily interpreted by software systems. UncertML is based on a number of ISO and OGC standards, such as ISO 19138 Data Quality Measures, and addresses the ISO/IEC guide to the expression of uncertainty in measurement (GUM). UncertML utilizes the OGC Geography Markup Language (GML) Standard and the OGC Sensor Web Enablement Common (SWE) Standard."

Later, UncertML was adopted by the UncertWeb project and a version 2.0 of it emerged as a well documented proposal maintained by a small community: http://www.uncertml.org.

The problem with UncertML is that it only deals with one way of measuring the data quality: uncertainty. This works well with some measurements of accuracy but does not cover other quality measures like consistency, completeness etc. We needed a way to extend the UncertML approach to all other quality indicators and to harmonize them with the measurement definitions that ISO 19157 is providing and many more that are used in other domains.

QualityML (http://www.qualityML.org) is a data model, a vocabulary and an encoding for data quality.

To report quantitative or conformance quality in QualityML, you need to combine the following concepts:

The main idea behind QualityML is that it can be used to describe quality elements in metadata in a standardized way that it also includes semantics (as it also points to the included definitions), allowing easy comparison of products.

On one hand, the QualityML is a profile of the ISO geospatial metadata standards (e.g. ISO 19157) providing a set of rules for precisely documenting quality indicator parameters that is structured in 6 levels. On the other hand, QualityML includes semantics and vocabularies for the quality concepts.

Whenever possible, QualityML uses statistic expressions from the UncertML dictionary (http://www.uncertml.org) encoding. However it also extends UncertML to provide list of alternative metrics that are commonly used to quantify quality beyond the uncertainty concept, for example the ones used in ISO 19157.

Quality metrics are used to compute the result of each quality measure value, when applied to a certain domain. QualityML provide a matrix of the combinations of indicators, measurements, domains and metrics commonly used.

The main idea behind this structure is to unlink measures, domains and metrics description, in order to maximize generalization of descriptions and increase coherence among several measures using the same metrics (even with different domains), or several quality indicators using the same measures.

Traditionally, quality reports are associated to product specifications resulting in quality metrics common for all scenes of the same product.

Each individual scene (associated here to the concept of dataset) can be validated against what is considered ground truth and can have quality metric reports e.g. about the geospatial accuracy of the positions of the individual pixels (a process that is commonly done applying GCP). This results in an statistical dataset quality.

Recently, per pixel or per object qualities are also being included (e.g MODIS and Sentinel). In this case, an uncertainty model is associated to an scene that allows to compute individual uncertainties for each pixel. These uncertainties can be of any kind but e.g a set go GCP can be used to model the uncertainty distribution of the scene or an illumination model can be used to model the radiometric uncertainty of each pixel. This error modeling is particularly interesting when there is a need to propagate uncertainty when the same scene is used to derive high level products.

Another source of per pixel uncertainty is introduced when several scenes are combined to create a mosaic product. In this case the pixels of the mosaic have different uncertainty depending on the source of the original data (depending on the original scene).

To improve performance of visualization portals, data is structured on map tiles that are served by a tile service (e.g. based on the OGC WMTS). Tile layer services are expected to be easy to use providing access to a product that is a spatial and temporal mosaic of a long time series. The quality of each tile depends on the quality of the source scenes. Potentially each tile can have a different quality information depending on the source scenes. More details are discussed in WMTS Time Accuracy.

Metadata of each image can describe the positional uncertainty of satellite position using: ISO 19157 D.29 — Mean value of positional uncertainties. This measure is a measure describing the whole dataset series for a certain amount of time (as the measure is using measures from validation stations during a certain amount of time), and thus this need to be specified using and specific EX_TemporalExtent on the scope of the quality measure.

In QualityML, this measure is called Mean Absolute Error (MAE) and the domain and metrics are different depending if a 1D, 2D or 3D approach is needed.

Metadata of each imagery can describe the ground location accuracy using: ISO 19157 D.44 to D.49 — Circular error at different significance levels. This measure is a measure describing each dataset, and is usually computed according to satellite ephemerides.

A common measurement for positional accuracy is called the circular error. The circular error estimates probability for the true value to lie in an circular area assuming a two-dimensional density function of the normal distribution. The circular area is characterized by its radius. For the same set of values, the confidence radius can be returned at different probabilities.

Different probabilities are considered different measures in ISO 19157. As explained in QualityML quality vocabulary and encoding, some ISO measures are generalized in QualityML as single measure (with a parameter), called Circular Map Accuracy in this case (with a parameter "levels" to describe the confidence).

Thus, the description of ground location accuracy using QualityML would use:

Moreover, metadata of the imagery can describe the product accuracy meaning the positional accuraccy of the final product at a dataset level. Usually the evaluation of the positional accuracy for gridded data is carried out by the definition of ground control points (with known true, or accepted as true, positions).

In this case, most common measure for positional accuracy is root mean square error of planimetry RMSEP, that is equivalent to a bidimensional standard deviation (or σ)):

(that, assuming a normal distribution of errors, describes a probability of 63.8%).

In QualityML, this measure is called Root Mean Square Error and, for planimetry, is applied to a 2D Differential Error Measure domain, and uses Root Mean Square Error 2D metrics.

A Digital Elevation Model is a useful product and describing its accuracy on the height values is related to describing thematic accuracy. In this case a DQ_QuantitativeAttributeAccuracy quality element is the convenient, and the proposal is to use the UncertML Normal Distribution description to describe the mean and the variance of the uncertainty on this element:

A Normalized Difference Vegetation Index (NDVI), for example, can be derived from imagery, and documenting its thematic accuracy is important for ulterior modeling based on this biophysical variable.

In this case a DQ_QuantitativeAttributeAccuracy quality element is the convenient, and measures such as Root Mean Square Error, Mean Absolute Error or Quantitative attribute correctness can be used.

In generic terms, currency is defined by the distance between the date the data was captured and the current time. In the ISO 19115 and 19115-1 the acquisition date can be documented as the temporal extent of the resource.

Then to calculate the currency, a difference between the temporal extent and the current date will give us a currency estimation.

Note that a temporal extent can be expressed not only by a single date but by a range. A range is useful for a single scene that is captured by a push-broom sensor. Sometimes a dataset is composed by juxtaposition/mosaic of scenes with a variety of acquisition time. In this case, the temporal extent of the result could be quite wide. Depending on the circumstances, it could diminish the quality of the data.

If a currency requirement is described in a certain environment, or for a certain product specification, a Temporal validity indicator could be used to state the degree of agreement between image temporal extent and the requirement.

For example, if the product specifications has a requirement that considers pixels that are older than a limit (e.g. 24 months old) as not current enough, the temporal validity quality indicator can report this, describing the requirement as a parameter in the domain of the quality measure, as explained (with the same example) in QualityML domain (with parameters).

Note, that metrics/items allow for both a boolean indicator, the number of pixels or a percentage. Thus, usually when a pixel-value result is used, the items is used as a boolean to indicate (for each pixel) if a certain characteristic is present (e.g. is this pixel covered by snow). When the result is given as a quantitative result, usually rates (or number of pixels) are used (e.g. 26% of the image is covered by clouds).

In the next sections QualityML Flag areas measure is used to describe snow cover, including pixel-level encoding examples. On the other hand, cloud cover is provided using the general purpose metadata cloudCoverPercentage:

Next sections will show and example of image processing level documentation to describe the values of the image according to a general codeSpace (liked to a QualityML dictionary), as well as to include a quality measure to describe the rate on pixel values that are not conformant to its value domain, because they have reflectance values higher that the maximum (e.g. due to sensor saturation).

ISO 19157 gives examples of how DQ_MeasureReference should be described in metadata (e.g. in section E.4.1.2 Reporting commission, table E.12):

In this example, the namespace (in the 'codeSpace' element of the MD_Identifier) is not defined. Moreover, the identifier is not directly linked to a catalogue defining the measure.

Encoded in XML following ISO 19115 this example looks like:

According to ISO 19139, Anchor is substitutable of CharacterString, so this ability can be used when instantiating the web environment extensions for properties having CharacterString type in ISO 19115-1. The main advantage of using 'Anchor' element is that it allows to define an "xlink:href" attribute to include a link to the catalogue definition of the code or the codeSpace when describing a measure.

Our general proposal to give semantics to a Measure reference (by means of directly link it to a catalogue) is the use of Anchor element in:

to include direct links to QualityML measures as a recommended option (but any other dictionary that a user would like to use is also possible).

Next examples are covering the quality measure conceptually described in QM_3 Accuracy Processed pixel location QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_3 Accuracy Processed pixel location old 19115 XML and QM_3 Accuracy Processed pixel location 19115_1 XML.

The encodings of a nameOfMeasure, measureIdentification and measureDescription for this quality measure are described here for convenience (as an exemplification of the Anchor element usage):

Quality Metrics are described in QualityML as the elements used to describe the result for a certain Quality Measure. A single quality measure can be expressed by different metrics, and this is the reason for the split between the two concepts. More details can be found in QualityML quality vocabulary and encoding or in QualityML.

Quantitative result may be a single value or multiple values, depending on the values of attributes valueType and valueStructure defined in the description of the measure applied. The attribute valueRecordType is used to describe how the valueType and valueStructure defined in the measure are implemented to provide the value of the quantitative result.

QualityML metrics schema (available at http://qualityml.geoviqua.org/schemas/qualityml/1.0/qualityml.xsd) describes the metrics defined in QualityML that can be used in an XML encoding within the gmd:value/gco:Record.

When describing a quantitative result, the recommendation to give semantics is to:

Next examples are still covering the quality measure conceptually described in Product accuracy (x,y,z): Processed pixel location accuracy on the ground. The full XML encoding for this quality measure can be found in XML examples of encoding for Quality Measures for Raster, in ISO 19115 encoding and ISO 19115-1 encoding.

The encodings of a RecordType, Record and valueUnit for this quality measure result are described here for convenience (as an exemplification of the QualityML metrics elements usage):

Some metrics, or some metrics options need the description of some parameters to be fully described. A first example of this is the description of the parameter "max" when an Items/rate metrics is used, or the parameter "level" when the CircularError metrics is used. More details can be found in QualityML quality vocabulary and encoding or in QualityML.

Next examples are covering the quality measure conceptually described in QM_2 Accuracy Pixel location QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_2 Accuracy Pixel location old 19115 XML and QM_2 Accuracy Pixel location 19115_1 XML.

The encodings of a RecordType, Record and valueUnit for this quality measure result are described here for convenience (as an exemplification of the QualityML metrics elements, and its parameters, usage):

Some metrics, or some metrics options need the description of some parameters to be fully described. A firs example of this is the description of the parameter "max" when an Items/rate metrics is used, or the parameter "level" when the CircularError metrics is used. Quality Metrics are described in QualityML as the elements used to describe the result for a certain Quality Measure. A single quality measure can be expressed by different metrics, and this is the reason for the split between the two concepts. More details can be found in QualityML quality vocabulary and encoding or in QualityML.

The use of the Domain of the Quality measure, un QualityML has been described in QualityML vocabulary encoding, allowing then a higher agrupation schema. QualityML describes several domains to which the quality measures can be applied.

Next examples are covering the quality measure conceptually described in QM_7 Consistency SurfRefl value domain QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_7 Consistency SurfRefl value domain old 19115 XML and QM_7 Consistency SurfRefl value domain 19115_1 XML.

The encodings of a RecordType, Record and valueUnit for this quality measure result are described here for convenience, as an exemplification of the QualityML metrics elements, its parameters, and a general "rule" for the quality metric domain:

Sometimes restrictions on the domain used needs to be described, and thus some parameters are needed for the domain.

Next examples are covering the quality measure conceptually described in QM_5 Currency TemporalValidity QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_5 Currency TemporalValidity old 19115 XML and QM_5 Currency TemporalValidity 19115_1 XML.

The encodings of a RecordType, Record and valueUnit for this quality measure result are described here for convenience, as an exemplification of the QualityML metrics elements, its parameters, and also a domain and its parameters:

Related to temporal quality element, the TemporalExtent general metadata element is usually documented, sometimes giving information needed to better understand the quality measure.

Next examples are covering the description of temporal extent that is related to the quality measure conceptually described in QM_5 Currency TemporalValidity QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_5 Currency TemporalValidity old 19115 XML and QM_5 Currency TemporalValidity 19115_1 XML.

The encodings of temporalExtent (following ISO 19115) is described here for convenience (other sections of the same example have been shown previously and the full XML is also referenced above):

Related to Completeness quality element, the cloudCoverPercentage general metadata element is usually documented (in MD_ContentInformation | MD_ImageDescription) if only a dataset-level value is available. If a clouds flag mask (pixel-level) is available, it can be described in metadata as explained in Pixel level Quality can be used if a single percentage is used.

Next examples are covering the description of cloudCoverPercentage that is related to the quality measure conceptually described in QM_6 Completeness FlagAreas QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_6 Completeness FlagAreas old 19115_XML and QM_6 Completeness FlagAreas 19115_1 XML.

The encodings of temporalExtent (following ISO 19115) is described here for convenience (other sections of the same example will be discussed in Pixel Level):

One way to document part of this information is using general purpose metadata to describe the processing level of the image. In the ISO 19115-1 introduces a new element in MD_CoverageDescription called processingLevelCode that is an MD_Identifier and thus the proposal is to link to a code described in the QualityML vocabulary, using the Anchor element (as explained in other examples above).

Next examples are covering the description of image processing level general metadata, that is related to the quality measure conceptually described in QM_7 Consistency SurfRefl value domain QMR. The full XML encoding for this quality measure can be found in Appendix QM Raster, in QM_7 Consistency SurfRefl value domain old 19115 XML and QM_7 Consistency SurfRefl value domain 19115_1 XML.

The encoding of processingLevel (following ISO 19115) is described here for convenience (other sections of the same example have been shown previously and the full XML is also referenced above):

ISO 19115-2 adds a new subclass to DQ_Element specifically designed to include a coverage that contains a raster file that has values indicating the error or the uncertainty for each pixel.

The coverage result has several additional attributes of type MD_SpatialRepresentation (to describe the grid; its dimension etc), MD_CoverageDescription (to describe the semantics of the coverage and a description of several bands grouped in attribute groups), MD_Format (describes de format of the data) and MX_DataFile (that allow for the inclusion of the file itself).

Example:

Limitations:

ISO 19115-1 adds a new MD_CoverageContentTypeCode called "qualityInformation" specifically designed to include a "band" that contains a raster file that has values indicating the error or the uncertainty for each pixel.