IV.A.  Foreword

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

3.3.  Abbreviated terms

ADES

Application Deployment and Execution Service

AI

Artificial Intelligence

AP

Application Package

ARD

Analysis Ready Data

AWS

Amazon Web Services

CEOS

Committee on Earth Observation Satellites

DML

Data Markup Language

EMS

Exploitation Platform Management Service

EO

Earth Observation

ER

Engineering Report

ESA

European Space Agency

FAIR

Findability, Accessibility, Interoperability, and Reuse

ML

Machine Learning

OGC

Open Geospatial Consortium

STAC

SpatioTemporal Asset Catalog

SWG

Standards Working Group

TD

Training Data

TDS

Training Dataset

TrainingDML-AI

Training Data Markup Language for Artificial Intelligence

UML

Unified Modelling Language

5.1.  Defining AI and ML

As a field, AI covers “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages” Oxford Reference. ML is then a subset of the AI field, specifically focusing on the creation of algorithms that learn from data without explicit programming. The output of these algorithms is then a trained ML model, which can process new inputs.

Within the domain of ML, there are three categories of application, each distinguished by their learning method.

• Supervised learning: The algorithm is provided with a labeled TDS which pairs input data with a training label. The algorithm creates an output from the input data and compares this with the training label then iteratively updates itself to maximize its accuracy in comparison to the desired output.

• Unsupervised learning: The algorithm is provided with input data and attempts to identify commonalities and differences in the data that allow it to be grouped. Once groups are established, the algorithm is able to identify which group new input data should belong to.

• Reinforcement learning: The algorithm is exposed to an environment to which it must respond and is then rewarded if it responded appropriately.

This Engineering Report (ER) primarily focuses on supervised learning applications. Supervised learning algorithms include linear regression, decision trees, support vector machines, and neural networks. In particular convolutional neural networks are commonly used in EO applications due to their ability to identify features in images. Convolutional neural networks can be used for semantic segmentation (pixel-level classification) and object detection (bounding box identification).

5.2.  Typical formats for TDSs in EO Applications

A TDS is made up of two components: input data and training labels. For EO applications, the input data are remote sensing observations: Multispectral and hyperspectral satellite imagery, red, green, and blue (RGB) aerial photography, drone-mounted LiDAR point clouds, and so on. The training labels capture the classification and location of known features in the input data, such as the location and bounds of a building, or the type of crop at a given location. Training labels can be provided in vector or raster format, depending on the ML application. Supervised learning applications require both the input data and the training labels whereas unsupervised learning applications only require the input data.

5.3.  Example use cases

This ER section describes some ML for EO use cases to provide context. The use cases are based on Testbed participants’ experience particularly where ML is used to create spatial data (vector and raster) from EO data for a broad range of use cases.

The use cases are from efforts from state and national governments to create spatial data from ML on jurisdictional areas as a method of automating the creation of these products. In these use cases, reusability is a key consideration. These agencies (for example) may wish to later extend the existing training set to keep the dataset current, as well as to explore new questions and produce new information products.

As part of each use case, the authors provide a summary of implications for a TDS standard. For a specific review of the suitability of the proposed TraningDML-AI standard for use cases, see Table A.1.

5.4.  Opportunities

As highlighted in the above case studies, ML has become widely used in the automated creation of insightful data products from EO data. As TDSs form the basis of ML approaches, a TDS standard has the potential to improve the quality and consistency of the application of ML to EO. This section covers specific opportunities that a TDS standard could enable.

The generation of a TDS is context-specific. The process is directly linked to the geospatial and temporal domain over which it is created, as well as the features it includes. A TDS standard would encourage TDS creators to provide this context along with the TDS. This allows future users to understand the TDS’s applicability to a new domains, or to refresh or augment the TDS to capture different features of interest. As TDSs are often time- and resource-intensive to create, improved reusability of TDSs would be valuable for the EO community.

Our world is constantly changing, and features captured by a TDSs may become outdated over time. As such, the ability to describe and trace changes to features, along with versioning of TDSs, is important for ensuring ML applications are using valid data. A TDSs standard can aid the cataloging and versioning of TDSs.

ML processes need high quality and consistent TDSs to perform well. This may relate to either consistency in labeling across the TDS, or measures of its similarity to associated ground truth data. Having provenance and automated quality metrics captured by the standard would serve to help creators serve reliable and consistent TDSs and provide users with confidence in the TDS.

As agencies begin to rely on ML to produce automated products from EO data, it is critical that they are well-informed when creating or procuring TDSs. A TDS standard would support these agencies to request metadata that enables use and reuse as described above, without needing deep ML expertise. Clear descriptions of TDS metadata would also allow ML projects to be worked on by multiple providers, helping set clear expectations between the TDS creator and the TDS user, and allowing for transfer of a TDS across multiple parties.

5.5.  Challenges

The case studies presented in this ER also highlight several challenges that must be considered in the development of a TDS standard. This section describes specific challenges that arise when working with TDSs and how these relate to the creation of a standard.

The use cases demonstrate that TDSs are created for highly specific domain problems. The challenge for a TDS standard will be to support creators in providing sufficient information about the domain. Without this, a new user cannot easily assess whether the TDS can be leveraged in their domain. Relevant domain information includes the following.

• Total geographic extent

• Spatial distribution of individual TDS elements

• Date and time of labeling

• Date and time of input data capture

• Properties of the input data and labels, including (but not limited to):

  • the source of the input data (e.g., a specific satellite or LiDAR instrument);

  • any corrections applied to the source data (e.g., terrain correction, top of atmosphere correction);

  • the features of the input data (e.g., spectral bands, derived features);

  • any properties of those features (e.g., spectral range, definition of any derived features, spatial resolution); and

  • uncertainties associated with the input data or labels (e.g., positional uncertainty from GPS, depth uncertainty from SDB).

• Designation to training, validation or test set, for individual TDS elements

• Description of sampling strategy

• Description of methods used to stratify the data

• Description of class imbalances present in the TDS

There are many methods that can be used to create training labels or input data, and multiple of these may be used within a single TDS. This may affect the overall quality of a TDS (discussed further in Clause 9), and a new user may wish to include, exclude, or revise particular elements based on their creation method. A TDS standard will need to ensure each element in a TDS can be labeled with the following information.

• Who created the label (with each individual assigned a unique ID), including (but not limited to)

  • a domain expert

  • a non-expert

  • a machine learning process

• The process for creating the label, including (but not limited to)

  • labeled from imagery by a human

  • generated by a machine learning process

  • collected in the field by a human

• Version history of the label

• Any accuracy measures related to the label (e.g., GPS accuracy for field-collected labels)

• The path to corresponding input data

• The process for creating the corresponding input data, including (but not limited to)

  • direct from source

  • augmented from source (e.g., rotated, shifted, mirrored)

  • synthesized (e.g., generated by a Generative Adversarial Network (GAN) or from simulations)

The development of a standard for TDSs should anticipate that TDSs will evolve over time, as new algorithms are developed and popularized. The challenge for a TDS standard will be in capturing the critical domain information described above while remaining flexible enough to accommodate future changes in the way TDSs are generated.

By having a TDS standard there is potential for TDSs to become more interoperable This is due to users having information on the limits of the domain of application for a given TDS. As such, the standard needs to address the idea that a new TDS could comprise selected elements of existing TDSs and that the lineage is appropriately recorded.

6.1.  Training Data Markup Language for Artificial Intelligence Draft Standard

[peng] recognized that existing TDS, including open source benchmarks, can lack discoverability and accessibility plus there is often no unified method to describe the Training Data (TD).

The Training Data Markup Language for Artificial Intelligence (TrainingDML-AI) standard, released for internal review on August 2, 2022 is a conceptual model defined using Unified Modelling Language (UML) as a series of modules.

Figure 1 — TrainingDML-AI module overview.

TrainingDML-AI is designed as a universal information model that defines elements and attributes which are useful for a broad range of AI/ML applications. Any TD element may be augmented by additional attributes and relations whose names, data types, and values can be provided by a running application without requiring extensions to the TrainingDML-AI conceptual schema and respective encodings.

TrainingDML-AI builds on the ISO 19100 family of standards.

Figure 2 — Use of ISO Standards in TrainingDML-AI.

Annex B of the specification showcases the JSON encoding of example TDSs.

6.2.  SpatioTemporal Asset Catalog (STAC)

The goal of the SpatioTemporal Asset Catalog (STAC) family of specifications is to standardize the way geospatial asset metadata is structured and queried. Of relevance to this Testbed-18 activity are the following extensions, which are currently (as of 03 August 2022) classed as Work In Progress.

• ML AOI: An Item and Collection extension to provide labeled training data for ML models.

• ML Model: An Item and Collection extension to describe ML models that operate on Earth observation data.

The ML AOI (Area of Interest) extension relies on, but is distinct from, the existing label extension. STAC items using the label extension link label assets with the source imagery for which they are valid. This is often as a result of human labeling effort. By contrast STAC items using the ‘ml-aoi’ extension link label assets with raster items for each specific ML model that is being trained.

6.3.  ESA funded initiatives and projects such as AIREO

The European Space Agency (ESA) funded Artificial Intelligence (AI) Ready Earth Observation activity AIREO provides resources and tools to data creators and users to ensure their TDS are FAIR and to standardize aspects of TDS such as quality assurance and metadata completeness indicators.

The aim is for the AIREO TDS Specification to be applicable to all levels of predictive feature data and to other target variable types. The purpose of the first version of the specification was to generate feedback on the content and requirements from the community to ensure a more useful and relevant V1 specification. As such, the focus has been on a limited number of datasets with examples in the AIREO pilot datasets.

Figure 3 — Overview of the AIREO TDS specification.

Figure 4 shows the STAC implementation of the data model.

Figure 4 — STAC implementation of the AIREO data model.

There are also the AIREO Best Practice Guidelines that outline how to generate and document AIREO-compliant datasets following the AIREO specifications.

Separately, the AI4EO initiative supports the aims of ESA’s Φ-lab that are to accelerate the future of EO by means of transformational innovations. AI4EO hosts challenges that bring AI and EO together.

6.4.  ANZLIC considerations of TDSs as foundational data

In the geospatial domain, the United Nations Committee of Experts on Global Geospatial Information Management (the UNGGIM) has identified 14 Global Fundamental Geospatial Data Themes. These data themes are considered fundamental to support global initiatives, such as reporting on the Sustainable Development Goals. These themes are being adopted by nations across the world and are driving the need to create and maintain these datasets in an efficient and effective manner. In Australia, the Australian and New Zealand Land Information Council (ANZLIC) has adopted these themes and challenged jurisdictions to develop and maintain these data. The ANZLIC community is considering the role of ML in this activity (as described in Clause 5.3.1) and also how the TDSs themselves may be considered as a critical building block, and perhaps may in future be recognized as “foundational,” leading to the consideration of whether TDSs might in future become a recognized and authoritative data product.

6.5.  Public TDS repositories

6.6.  Previous OGC activities

7.1.  Current structure and usage of metadata in ML TDS

As outlined in Clause 6, most current ML TDS models use the STAC family of specifications as the basis to structure the TDS and related metadata. The STAC specification defines only a limited set of ‘STAC Core Metadata’ elements used for STAC Catalog ‘Collection’ and STAC Catalog ‘Item’. The following core metadata elements are required.

• Basic metadata to provide an overview of a STAC Item

  • title

  • description

• Date and Time definition

  • datetime, created and updated — to allow recording of temporal capture via information about

  • start_datetime and end_datetime — to allow specification of ranges of capture datetimes

• License information for data and metadata

• Provider information — to allow defining information about provider (e.g., name, description, and url) and their roles (e.g., processor, producer, licensor, host)

• Instrument information — to allow specifying the information about platform, instrument, mission, constellation and ground sampling distance used for data acquisition

Given its modular nature, the STAC specification allows enhancing the metadata definition of STAC objects through extensions. One of the stable STAC extensions recommended for definition of ML items and collections (see Clause 6) is the ‘Scientific Citation Extension’. This extends STAC core metadata elements with reference information about which publication a STAC object originates and how it should be cited or referenced. Additional scientific citation metadata, such as the Digital Object Identifier (DOI), citation, and indication of relevant publications help to increase reproducibility and findability of a STAC object, and thus improving its FAIRness (more detail in Clause 10).

7.2.  Review and application of ISO metadata standards for ML TDS

This section discusses four key ISO metadata standards: ISO 19115-1, ISO 19115-2, ISO 19157-1, and ISO 19157-3.

7.2.1.  ISO 19115-1 and ISO 19115-2 for geographic information

ISO 19115-1:2014 Geographic information — Metadata — Part 1: Fundamentals

ISO 19115-1:2014 defines the schema required for metadata about geographic datasets and services. The standard defines the structure for information about data and metadata identification, spatial and temporal extent, quality, distribution, and licenses. This standard is applicable to the definition of metadata catalogs (typically used in a Spatial Data Infrastructure — SDI) as well as for describing geographic resources of various kinds (i.e., datasets or services, maps, charts, or textual documents about geographic resources) and at various levels of detail (e.g., dataset, feature, or attribute). Figure 5 illustrates the metadata schema defined in ISO 19115-1.

Figure 5 — ISO 19115-1 Metadata schema.

ISO 19115-1:2014 also identifies the minimum metadata set required to serve most metadata applications, including data discovery, access, transfer, and use, and a decision on dataset’s fitness for use (see Table 1).

Table 1 — Metadata for the discovery of geographic datasets

Metadata element	Obligation	Comment
Metadata reference information	Optional	Unique identifier for the metadata
Resource title	Mandatory	Title by which the resource is known
Resource reference date	Optional	A date which is used to help identify the resource
Resource identifier	Optional	Unique identifier for the resource
Resource point of contact	Optional	Name, affiliation, and role of the person responsible for the resource
Geographic location	Conditional	Geographic coordinates or description of metadata location — Mandatory if the described resource is not a ‘dataset’.
Resource language	Conditional	Language used to describe the resource — Mandatory if other than default (English).
Resource topic category	Conditional	A selection from the list of topics defined in ISO 19115-1 — Mandatory if the described resource is not a ‘dataset’ or a ‘dataset series’.
Spatial resolution	Optional	The nominal scale and/or spatial resolution of the resource
Resource type	Conditional	ISO 19115-1 standard code (e.g., dataset, feature, attribute, product) the metadata describe — Mandatory if the described resource is not a ‘dataset’.
Resource abstract	Mandatory	A brief description of the content of the resource
Extent information for the dataset	Optional	Temporal or vertical extent of the resource
Resource lineage/provenance	Optional	Source and production steps used in producing the resource.
Resource on-line link	Optional	URL for the resource.
Keywords	Optional	Words and phrases describing the resource to be indexed and searched.
Constraint on the resource access and use	Optional	Restrictions on the access and use of the resource.
Metadata date stamp	Mandatory	Reference date for the creation (and update) of metadata
Metadata point of contact	Mandatory	The party responsible for the metadata.

Note in Table 1 that if a described resource is a ‘dataset’ or a ‘dataset series’ ISO 19115-1 mandates only four metadata elements to describe a such resource.

1. Resource title

2. Resource abstract

3. Metadata date stamp

4. Metadata point of contact

Arguably, this is insufficient to ensure findability, accessibility, interoperability, and reuse of a resource, especially by machines, which is often the case in ML.

ISO 19115-2:2019 Geographic information — Metadata — Part 2: Extensions for acquisition and processing

ISO 19115-2:2019 extends ISO 19115-1:2014 by defining the schema required for describing the acquisition and processing of geographic information, including imagery. This standard defines the structure for describing properties of measuring systems and the numerical methods and computational procedures used to derive geographic information from the acquired data. Figure 6 illustrates the metadata schema defined in ISO 19115-2.

Figure 6 — ISO 19115-2 Metadata schema.

ISO 19115-1 and ISO 19115-2 are intentionally generic. Although they are applicable in most geospatial domains, some specific disciplines may need extensions or modifications to the model. ISO 19115-1:2014 allows the creation of metadata extensions. The following are the permitted types of extensions in the current metadata standard.

1. Adding a new metadata package

2. Creating new metadata codelist elements (expanding a codelist)

3. Adding new metadata elements

4. Adding new metadata classes

5. Imposing a more stringent obligation on an existing metadata element

6. Imposing a more restrictive domain on an existing metadata element

An extension mechanism defined in item five in the list above would ensure more comprehensive discovery metadata, and thus help increase the FAIRness of a resource.

ISO 19157-1 Geographic information — Data quality — Part 1: Fundamentals

In the 19100 series of standards, data quality information is considered to be a specialized type of metadata about the quality of a geographic information resource. ISO 19157-1 provides a framework for defining the quality of geographic data. This includes principles for evaluating quality, a conceptual model for handling quality information, a structure and content of data quality measures, and guidelines for reporting a quality evaluation. The framework is extensible, with rules for how to add additional data quality measures including a provision for complex dimensions of data quality. Figure 7 illustrates the data quality model defined in ISO 19157-1.

Figure 7 — ISO 19157-1 Conceptual model of quality for geographic data.

According to the standard, working with data quality includes:

• understanding the concepts of data quality related to geographic data;

• defining data quality conformance levels in data product specifications or based on user requirements;

• evaluating data quality and metaquality; and

• reporting data quality and metaquality.

Where metaquality is a way of describing the quality of the data quality evaluation.

A data quality evaluation can be applied at various scopes (see Figure 11) to dataset series, a dataset, or a subset of data within a dataset, sharing common characteristics so that its quality can be evaluated. Data quality elements and their descriptors are used to describe how well a dataset meets the criteria set forth in its data product specification or user requirements and provide quantitative quality information.

Data quality elements and their descriptors are used to describe how well a dataset meets the criteria set forth in its data product specification or user requirements and provide quantitative quality information. ISO 19157-1 defines the following data quality elements.

• positional accuracy: Closeness of agreement between a measured position of features and a position accepted as true within a spatial reference system. The following are the types of positional accuracy.

  • absolute positional accuracy

  • relative positional accuracy

  • gridded data positional accuracy

• thematic accuracy: The accuracy of quantitative attributes and the correctness of non-quantitative attributes and of the classifications of features and their relationships. Depending on the type of attribute, this can be expressed with the following.

  • classification correctness

  • non-quantitative attribute correctness

  • quantitative attribute correctness

• completeness: The presence or absence of features, attributes, or relationships in a resource and can be expressed by the following.

  • omission

  • commission

• logical consistency: The degree of adherence to logical rules of data structure, attribution, and relationships as follows.

  • conceptual logical consistency

  • domain consistency

  • format consistency

  • topological consistency

• temporal quality: The quality of the temporal attributes and temporal relationships of features. The following aspects can be used to express temporal accuracy.

  • accuracy of time measurement

  • temporal consistency

  • temporal validity

In addition to the elements describing the quality of a geographic resource, the standard defines a way of describing the quality of the quality evaluation, i.e., metaquality of a resource. Confidence, representativity, and homogeneity are example elements suggested in the standard.

ISO 19157-1 recognizes that for many domain-specific purposes it is necessary or convenient to extend the standard data quality information model as defined in the standard. An extension includes adding data quality elements (e.g., ‘class balance degree’) and data quality measures (e.g., ‘number of imbalanced classes in the training dataset’).

ISO 19157-3 Geographic information — Data quality — Part 1: Data quality measures register

To facilitate comparisons of datasets expressing the quality in a comparable way and having a common understanding of the data quality measures that have been used is essential. These data quality measures provide descriptors of the quality of geographic data through comparison with the universe of discourse. ISO 19157-1 standardizes the components and structures of data quality measures, and ISO 19157-3 (currently under development) establishes a machine-actionable data quality measures register. An ISO 19157-3 compliant register will contain an extensible curated set of data quality measures. The structure of a data quality measure (as defined in ISO 19157-1) is illustrated in Figure 8 and an example is in [igure-DQmeasureExample].

Figure 8 — Data quality measure structure as defined in ISO 19157-1.

Figure 9 — Example of a data quality measure.

7.2.2.  Metadata reporting at various granularity levels

ISO 19115-1:2014, ISO 19115-2:2019, and ISO 19157-1 are applicable at various levels of detail. These are defined by the MD_Scope property in the metadata information (MD_Metadata) and of the resource (see Figure 10).

Figure 10 — ISO 19115-1 Metatadata on Metadata.

The default scope for metadata is the ‘dataset’, but the domain of the scope is not restricted to the dataset. The MD_ScopeCode (see Figure 11) defines an extensible list of possible types of resources. The choice of suitable type is left to the data provider.

Figure 11 — ISO 19115-1 Metatadata scope types

7.3.  Examples of human and machine-readable metadata for a TDS

Producers use metadata to advertise their resources. This is typically done through metadata catalogs or via embedded metadata in the resource that can be indexed by search engines. Metadata also serve as the essential vehicle for users’ (humans or machines) decision on resources’ fitness for use. Each type of user requires their own way of metadata expression, i.e., human readable (as in the example in Figure 12) or machine readable (as in the example in Figure 13).

Figure 12 — Human readable metadata documentation

Figure 13 — Machine readable metadata documentation

8.1.  What is a catalog?

In an OGC White Paper on standards and cloud computing McKee et al. 2011, the cloud is described as based on a standards framework for service-oriented architectures that provides a “publish, find, bind” model.

• Publish: Resources can be hosted and their description, network location, and interfaces can be published in standards-based registries or catalogs.

• Find: Client applications can search the registries or catalogs to find a resource.

• Bind: The client application can invoke the server through standard interfaces.

As a result, catalogs are important for both publishing and finding information.

ISO 19115-1 and ISO 19115-2 are applicable to the metadata for cataloging geographic services/datasets as previously detailed in Clause 7.

OGC-API Records provide discovery and access to metadata about geospatial resources (e.g., data, services, ML models, etc.), and has three main building blocks: record, collection, and records Application Programming Interface (API) as a web interface. A record provides a description (i.e., metadata) about a resource that the provider of the resource wishes to make discoverable. A collection is used to describe a collection of resources. There are several ways that records can be deployed as a “collection of records” or a catalog. Three deployment patterns are envisioned as follows.

1. A catalog deployed as a crawlable collection of records

2. A catalog deployed as searchable endpoint(s)

3. A local resources catalog

For a SpatioTemporal Asset Catalog (STAC), the terms static and dynamic are used to describe these deployment patterns. However, a static catalog is not really static since additional records can be added at any time. Therefore, the terms crawlable and searchable are proposed for OGC-API Records.

8.2.  Version control for TDS

The Data on the Web Best Practices (DWBP) states that datasets published on the Web may change over time and describes example scenarios. Even for small changes, it is important to keep track of the different dataset versions to make the dataset trustworthy. Publishers should remember that a given dataset may be in use by one or more data consumers. Therefore they should take reasonable steps to inform those consumers when a new version is released. Also, the publisher should take a consistent, informative approach to versioning, so data consumers can understand and work with the changing data.

For TDSs, questions include the following.

• In the context of maintaining fundamental data, new imagery sources may be added, and a trained model would need to be exposed to this imagery. Training data collection will expand, or may be updated. How is this tracked?

• Reprocessing of the used imagery: Many satellite missions have their data reprocessed, which creates a consistent dataset. When a TDS is made available, users may be downloading older versions of extracted data, which have anomalies that have since been corrected. Alternatively, if new versions of such data are download the ML model may not work as expected, e.g., if there is a change to the radiometric calibration of how the data is stored as has occurred with the Copernicus Sentinel-2.

• Temporal element: Training data is tied to imagery from a given date. What happens when the underlying landscape changes? Training data cannot necessarily be used on new imagery if the landscape has changed.

• Research papers would need to state which version of a training data set were used for future reproducibility of results.

For the Training Data Markup Language for Artificial Intelligence (TrainingDML-AI) standard, and as shown in Figure 1, there is a Changeset module that defines changes in the Training Data (TD) between different versions. Three kinds changes to training samples are included: add, modify, and delete.

Public repositories, such as Zenodo (see Clause 6) provide a means to archive dataset versions with version-specific Digital Object Identifiers (DOIs). However, EO data is often processed, including sub-setting, so will not be in its original form and hence will probably have lost its original metadata. Therefore, keeping a record of the input dataset versions can be complicated as datasets may not be consistently processed. For example, Copernicus Sentinel-2 is a frequently used dataset where the processor has been updated during the mission lifetime and the archive has not currently (as of August 2022) been reprocessed consistently. Therefore, imagery from different dates will have different corrections applied that may influence the ML inputs, and hence, model.

8.3.  Splitting source data and annotated training data

Source data may change, either intentionally or unintentionally. Therefore, in storing a TDS, there is a question about what should be contained versus referenced by the catalog.

For example, if the underlying imagery is not provided, only a query for it or a description of how it was generated is? Is this the best approach, or should the extracted source data be included as well as any derived products that act as inputs to the ML model?

8.4.  Making TDS catalogs self-explanatory

“Self-explanatory” should equate to sufficient metadata within the catalog, including onward web links, that enables a user to answer questions they might have. Difficulties can occur as time passes because web links become out-of-date/broken and the information becomes lost — a particular problem with online storage where storage is reliant on individuals/organizations paying for storage. Another example would be a disaster response scenario where users do not have access to the internet, or setups where the TDS is distributed to a disconnected environment.

The AIREO Best Practice Guidelines encourages data creators to describe the metadata and documentation in the catalog richly to enable attribute-based searches and facilitate data findability. The AIREO Best Practice Guidelines also discusses the need for eternally persistent identifiers, with the metadata containing machine-processable and verified elements for citation and version control.

Gebru et al. 2021 proposed datasheets for datasets to support a perceived gap in the standardized process for documenting ML TDS, which was applied by the WorldStrat TDS. The datasheet is as an appendix to the accompanying paper, and part of the accompanying documentation within the WorldStrat Zenodo archive. It contains more detailed information than is potentially feasible to store in the metadata alone.

9.1.  Biases and domains in TDS

The domain in which training data is created has significant influence on whether it can be reused. For example, the training dataset of roofprints created for Clause 5.3.2 was created for urban areas in the state of Victoria, Australia. Whether it can be reused in a new domain depends on the similarities between the original domain and the new domain. For example, the roofprint dataset might be reusable in other urban settings of Australian cities but might not be suitable to rural areas of Australia, due to differences in size, density, ground cover, or roof material.

When delineating and predicting features from Earth observation data, the year and season may also impact whether the training data can be applied to a new domain. This is particularly true in the case of training datasets that label the presence of vegetation, such as in Clause 5.3.1 and Clause 5.3.4, as vegetation will have different spectral qualities depending on the time of year the associated satellite data was captured. By capturing domain information in training dataset metadata, future users can make informed decisions about whether existing training data can be considered accurate enough for a new application.

Domains for consideration include the following.

• Geographical extent

• Capture date of reference or ground truth data

• Seasonality of reference or ground truth data

Understanding any biases that may be present in the data is important. Bias can be thought of as any process that occurred during training data creation that led to the data being unrepresentative of the real world. For example, when conducting ground surveys such as in Clause 5.3.4, the most efficient collection method may be to work along main roads, but this may mean the distribution of classes in the training data is not representative of the distribution across the whole area of interest. Lack of domain knowledge when producing labels may also lead to biases in the training data, such as through the systematic mislabeling of certain examples due to annotator confusion.

Along with the domain description, training dataset metadata should also describe the method used to generate the training dataset, especially sampling strategies that were used to ensure the training dataset was representative of the real world.

9.2.  Auto-generation of quality indicators

Consistency, measured as the ratio of consistently labelled instances to all instances, can be measured automatically, but will require each training data instance to be tagged with each annotator’s label.

Accuracy is more challenging and may require a subset of the training data to be identified as the benchmark for assessing correctness. With such a benchmark set, it would be possible to compare the agreement between labels from annotators with those from the benchmark. This would only give a representative idea of the accuracy but may be helpful in assessing the appropriateness of a training dataset for a new application.

10.1.  The FAIR guiding principles

According to Wilkinson2016 resources are FAIR when they are findable, accessible, interoperable and reusable. A summary of the FAIR principles are found in Table 3:

Table 3 — The FAIR principles

In more detail, data are findable when they are sufficiently described by their metadata and when they are registered and indexed in a searchable resource that is known and accessible to potential users (EC2018; Wilkinson2016).

Digital resources are accessible, when anyone (human or machine) with access to the Internet understands via provided metadata exactly how to access the digital resource and what the conditions on its reuse are (EC2018; Wilkinson2016). A common misinterpretation of this concept is the expectation that accessible (and hence FAIR) digital objects should be ‘open’ and/or ‘free’. This is not what FAIR guiding principles define. The only condition for FAIR digital objects is the clarity and transparency on the conditions of access and reuse of these objects (Mons2017).

Resources are interoperable when they use “normative and community recognized specifications, vocabularies, and standards that determine the precise meaning of concepts and qualities that the data represent” (EC2018, p.19). Although presence of these vocabularies and standards in a format compatible with semantic web would undoubtedly increase their interoperability (vanDenBrink2019;Mons2017), this requirement does not mean that vocabularies and standards used to describe the resource have to ‘be on the web.’ Use of a well-defined community profile (e.g., an ISO 19115-1:2014 Metadata Profile) and providing metadata in a machine-readable format (e.g., XML) ensure sufficient interoperability of a resource.

License information and the description of the provenance are the two crucial factors determining the reuse of a digital resource by both humans and machines (Mons2017; EC2018). This requires that the description of the license and provenance information need to be provided in a suitable format (e.g., XML or RDF).

Implementation of FAIR varies by community regarding which data should be FAIR and to what degree. In this sense, FAIR needs to be understood as scale and various degrees of FAIRness for different types of digital objects, such as datasets, are hence possible.

10.2.  Metadata – crucial element for ensuring FAIRness

Metadata are crucial for ensuring FAIRness of digital resources (Wilkinson2016; Musen2022). Metadata can be intrinsic or user-defined (Mons2017). Intrinsic metadata is created automatically during data capture (e.g., timestamp of a data record, or an automatic label of a data production software) and user-defined metadata is added to provide context for understanding digital object’s creation through provenance information. Both types of metadata should be added to a digital resource to ensure its FAIRness.

10.3.  Defining a TDS standard that enables FAIR Principles

In addition to general metadata requirements to meet the FAIR principles, a TDS standard should the following.

• Have all labels and input data (and any other descriptive metadata) link to the assigned globally linked and persistent identifier (F3)

• Provide clear links between labels and input data (R1)

• Capture provenance for both labels and input data (R1.2)

A review of compliance of the proposed TrainingDML-AI standard with the FAIR principles is in Table A.2.

11.1.  Standards

• Underlying geospatial standards have been developed by ISO that support the definition of a TDS, including the the data quality.

• These ISO standards are being built upon by the Training Data Markup Language (DML) for AI (TrainingDML-AI) Standards Working Group alongside community activities such as the AI Ready EO (AIREO) activity and SpatioTemporal Asset Catalog (STAC).

11.2.  Next steps

• Feedback from this Tested-18 activity has been provided to the community drafting the TrainingDML-AI standard and this should continue alongside providing inputs to/collaboration with the broader community activities. A summary of open recommendations are within Annex A.

• TrainingDML-AI introduces proposals to standardize both Provenance and Versions for TDS. These two elements of a TDS standard could end up being exceptionally important in the future because of the centrality of TDS to AI/ML applications, and also because there is an increasing focus on reproducability as part of open science.

• Establish how the metadata of a ML model can link to the metadata of a TDS used to train the ML model.

11.3.  Best practice ideas

• A TDS defines and delimits the operational competence of an ML model inferred from it. Therefore, to enable reusability, the domain of the TDS, such as geographical extent, date, and seasonality, must be sufficiently described so a new user can judge the suitability of a TDS for their application. Also, for applications such as object detection, it could also be helpful to describe the range of sensor orientations represented or the kinds of sensor distortions.

• There would be a benefit in having controlled vocabulary and semantics to specialize, generalize, or combine, etc., existing ML TDS within application domains. Current labeling regimes are almost always taxonomically “flat,” i.e., they seldom (if ever) reflect hierarchical concepts such as cars/volkswagons or animals/dogs. Yet such labeling will be required to sustain a healthy market for, and long-term maintenance and extension of, TDSs. One example of a hierachal system is the CORINE land cover nomenclature.

• It may be useful to distinguish the quality of particular labels in the TDS, such as those delineated by experts, as these could be used as a benchmark to assess the accuracy of labels from other annotators.

• To enable the automated generation of metrics such as TDS consistency, Training Data (TD) labels must be attributed with an ID that is unique for each annotator. This is so the number of annotators per label can be counted, as well as whether all unique annotators agreed on the label.

11.4.  GeoEthics

• Encouraging the open availability of TDSs so training models can reuse an existing TDS rather than developing a new one. This is costly and time consuming. Still, application-specific TD may be required, with more general TDs being a helpful starting point.

• Alternatively, it may be helpful to store and make the trained ML models publicly available and then “transfer learning” is applied for the specific application. This is a significant approach to reuse in AI/ML with reuse carried out at the model rather than TD level. In this case, knowledge of the used TDS would remain vital as additional TD would need to follow the same input specification. Also, it would be important to have developed an open standard for interoperability between ML architectures, e.g., as developed by ONYX.

• Supporting ML scientists in storing data within standard formats by developing open-source tools/code that allows formatting data in standard formats and ingesting said data into typical ML tools/packages.