OGC Testbed-14: Machine Learning Engineering Report

The AI landscape is rapidly evolving, leading to new possibilities for geospatial analytics on ever larger Big Data. Current standards are being modernized to take into account these novel methods and the new data available in federated infrastructures. The goal of this work is to develop a holistic understanding of AI, ML, and DL in the context of geospatial. By this work, it is expected to advance or derive best practices for integrating ML, DL, and AI tools and principles in the context of OWS. The following figure presents an overview of the research motivations to be addressed.

Throughout their experiments, the participants of this task needed to answer the following questions:

Several approaches were proposed for consideration in this work:

A goal of this task and its analysis was also to suggest potential future activity where these results could be investigated through recommended future tasks, deliverables, components, and Engineering Reports. This section presents recommended potential future tasks and deliverables that can support advancement of requirements and expand research motivations. The reader can also refer to Section 9 this Engineering Report for discussion and open issues for further details.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

Several examples of supervised, unsupervised, and semi-supervised ML applications can be found in [3] and [4]. Typical problem classes are illustrated by [5] in the following figure.

Major performance metrics of ML systems include precision and recall, training time and execution time, and metaparameters and analyst feedback. Similarly, the CFP lists the following metadata as key elements to consider in the ML task:

This section briefly presents key GeoINT challenges coherent with this work’s scope, with respect to the role of the analyst, to the processes and workflows, and to scenarios and applications. Further reading material can be found in [6], [7], [8], [9] and [10]. Below is a definition of GeoInt as found on Wikipedia.

This section briefly presents key Earth Systems challenges coherent with this work’s scope with respect to the data characteristics as well as scenarios and applications. Here this engineering report notes a large overlap with GeoInt applications presented above, as Earth Systems studies natural phenomena that impact life.

In our scenario, an analyst that must annotate imagery in order to train or retrain models has several modalities offered to him or her. For example, annotation clients and methods found in [4], [17] and [23] use point points, patches, bounding boxes, tiles, contours, and regions. The figure below illustrates the use of polygons to compactly delimitate objects of interest and may be considered as state-of-the-art.

Semantic aspects of annotation, including discussions on classes, labels, vocabularies and their granularity, can be found in [21] and [24]. The ML application in [20] describes the use of hierarchies, trees, ontologies, and directed graphs such as WordNet.

A large number of popular authoritative annotated data sources and datasets are referenced in [24], [25], [3], [26], [15], [20], [27] and [28]. These datasets include, but are not limited to, ImageNet, COCO, LandScan3, Urban 3D Challenge, IARPA, SpaceNet, PASCAL, Semantic Segmentation Challenge, Forest Recognition Competition, Flickr, GoogleImages, and OpenStreetMap. Instructions on how to construct training, validation, and test datasets while taking into account bias and diversity can be found in [17], [5] and [21].

This section gives an overview of the main characteristics of DL models and highlights special considerations for geospatial intelligence tasks.

Deep learning refers to the practice of training large, multi-layered, artificial neural networks for supervised or unsupervised ML problems. Since the 90’s, neural networks have been known to be a universal function approximator, and stacking several layers was thought to be able to model complex non-linear relations, but training such models remained out of reach. It is not until 2006 that, due to a combination of factors (compute power, suitable datasets, and new training methods), DL models started surpassing the accuracy of other state-of-the-art ML models [29]. Since then, DL has become one of the most active research subjects in the field of ML. It is now at the core of many state-of-the-art models and has yielded many applications in the industry including in the imagery and geospatial data analysis domain.

In comparison with other methods in the field of ML, DL models presents several advantages:

In practice, DL has shown some limits [5] including:

Deep learning models excel at processing 1D, 2D, 3D, dense, and homogenous data such as raster images, sound signals, text data, and time series (i.e.: data that can easily be represented as tensors). This effectiveness has led to many breakthroughs in accuracy for image classification, object detection, segmentation, speech recognition, and language modelling tasks. Deep learning-based methods have also yielded some impressive results on generative tasks, in particular with the use of GANs [15] (e.g,: increasing image resolution and generating maps from satellite images).

At the onset of the testbed, the concept was refined to include major data sources, systems, and clients. Below is an example of the work produced to prepare the subsequent development efforts and demonstrations.

For the proof-of-concept, the testbed participants decided to focus on disaster interoperability scenarios and, in particular, on urban floods. The disaster management cycle is well-suited for this study, as it is well-documented and encompasses several activities that may involve the use of ML services:

Those tasks involve multiple data sources (e.g., punctual sensors, ground cameras, high and medium resolution imagery, SAR images) and have high requirements on system interoperability and decision traceability.

Those scenarios are further refined for the demonstration section. In this section, particular scenarios are not described, but rather a background that drives architecture and API design choices. Below is a figure created in the concept generation phase of Testbed-14.

In the scope the of the ML system, four categories of actors were identified: Image analysts, software engineers, ML model developers, and end users. Theses actors are often split over several public and private organizations, but they all may interact with the same ML system at some point: when building, updating, deploying or using the service APIs.

Their interactions with the system are depicted in the following figure. In the most basic use case, the Image Analyst (IA) uses prior- and post-event data to identify areas destroyed by a natural disaster such as a hurricane. As temporal aspects were put aside in the current work, the image analyst identifies feature types that do not take any change detection into account.

Characteristics of the work of an analyst can be found in [26]. Among the major challenges of these users, as noted through operational timelines, are ambiguity in data labelling and trust of models. For the latter, data curation good practices are required but not sufficient for transparency of ML solutions. Additional discussion on assumptions and validations of AI can be found in [11] and [6]. The future analyst is also seen using advanced environments such as Virtual Reality (VR). NGA’s 2020 Analysis Technology plan (pdf) presents some evolution on this important role:

The following deliverables were produced.

This subsection presents two profiles considered for the training and execution of ML models.

This subsection describes the two other Deep Learning applications designed, implemented, and documented in accordance with the Proof of Concept developed in Testbed-14. The first one relates to VHR imagery semantic segmentation, the second involves transfer of the learning of SAR remote sensing onto popular DL architectures and trained models.

This section presents the metadata schemas proposed by the participants and used in their Technology Integration Experiments (TIEs).

The CFP also lists the following schemas and references:

The demonstration of the current work can be considered a success. The client application was indeed shown to execute processes offered by the ML system and display its results found in the Image and Feature Repository. As expected, the ML system provided classification results that in turn were referred to in the KB. The proposed schemas did integrate controlled vocabulary references.

Participants diligently interacted through the GitHub issue tracker in order to describe and solve problems and complete their interoperability experiments.

Difficulties in the followings aspects were noted, all mostly due to technical complexity:

It is expected that through future deliverables proposed in the future work subsection of the summary, these difficulties would be solved in early stages of a subsequent testbed.

The reader can find additional hints on problematic, outstanding or limited elements in discussion and open issues.

The Machine Learning Knowledge Base is effectively a catalogue of metadata about models, available images, available features, and information on previous model execution/training. In an OGC context, CSW could be used as a standardized interface in the Knowledge Base. This should also consider whether the information stored in the Knowledge Base could be handled using the OGC CSW-ISO Application Profile of CSW, which utilizes the ISO 19115 metadata model. If the ISO 19115 model is not rich enough to support the Knowledge Base use cases, then a more extensible registry model, such as the one offered by the CSW-ebRIM Application Profile, could be considered.

The Testbed-14 Next Generation Services (NextGen) thread addresses both new capabilities at the service level as well as complex interaction patterns between services. One of the tasks is to investigate Next Generation OGC Web Services RESTful interfaces by experimenting with the new WFS 3.0 specification and to add security mechanisms based on OAuth2.0. The use of the OpenAPI style of interaction in the ML Knowledge Base API is similar to the approach being investigated in NextGen. In order to forge links between the ML task and NextGen tasks, it is recommended to experiment with Next Gen CSW services using the findings from Testbed 14 on WFS 3.0.

In line with Testbed-14 EOC Application Packaging best practices [2], it is recommended that future work include packaging of ML systems for major AI framework. One advantage is the easier discoverability, distribution and management of various ML systems such as TEP and MEP architectures [32]. The major systems described in this work, the Execution Management System (EMS) and the Application Deployment and Execution System (ADES), both implement WPS-T 2.0 REST/JSON bindings. Additionally, work conducted in the EOC thread of Testbed-14 describe best practices for workflows, including data fetching, data preparation, creation of handcrafted features, publication of outputs, etc.

A post-processing step is required to segment the resulting regions into individual classified features (e.g., a car or a park) that can be stored with WFS. For the ML system, this can be achieved by transformation of the output into blobs. For the DL detector, the point location of an object is determined by the mode (maximal value) of each blob of the probability map for each class, but the contours remain fuzzy.

Several approaches to classification assume a flat structure to labels. Performance of DL can be significantly increased by combining popular datasets. This can be achieved in several ways. For example, using a hierarchical tree generated from WordNet, labels from ImageNet and COCO can be merged. It is recommended to continue experiments aiming to merge annotation datasets.

Supervised learning algorithms implemented and described in the testbed for ML system include Decision Trees and Deep Neural Networks. Once trained, both the outputs of both methods can be described as models, but their parameters, associated metadata, and performance differ significantly. In most cases, training time for a specific dataset and inference time for a given input are applicable. For performance, precision and recall for a given input are also applicable when ground truth data is available. It is recommended that participants consider adding training time, inference time, precision, and recall in the interfaces of their deliverable to better demonstrate use in a disaster scenario or, more generally, in GeoInt use cases.

In the Testbed-14 ML task, the participants developed a descriptive analytics use case in the spatial domain: "What has happened here? What do we see?". The scenario did not make extensive use of data with variety on the temporal axis. It is recommended that future experiments in Machine Learning includes a support for a variety of temporal data, potentially including IoT timeseries, satellite imagery, weather forecasts, climate projections, etc. Such variety enables predictive and ultimately prescriptive scenarios that are highly valued to AI users. Support for a wide array of temporal data will also advance geospatial standards, for example by establishing good practices for timestamp management of data or catalog querying and filtering. For example, Testbed-14 also looked at swaths - vertical collections of atmospheric wind velocity. Each "pixel" is a stack of wind direction and magnitude measurements at a specific place and time.

Interfaces developed in the testbed for ML systems favor interoperability at the data level (get source data, get training data, store) and at the process level (train, classify). Currently, no interoperability is possible at the model level (load model, get model), as algorithms such as Decision Trees and DNNs, both applicable as classifiers, are fundamentally different in their structure. Initiatives like the Open Neural Network eXchange (ONNX) aim to enable interoperability of models into various popular AI toolsets and frameworks ^[1]. It is recommended to consider publishing neural network capabilities of ML systems, thus allowing richer use cases, for example exportation from one ML system to another one through a workflow via get model and load model operators.

Implementations in the ML task used an Image Repository mainly to store data inputs and inference outputs of the model, facilitating their retrieval. In order to increase transparency and ultimately improve trust of algorithms, it is suggested to experiment with the use of OWS to facilitate visualization and interaction with the internal layers of CNNs. This assumes that artificial neural network models' weights and architecture can be assimilated to structured, dense spatial data (WMS, WCS) containing patterns and features (WFS) used for AI’s decisions or recommendations for a specific query (WPS).

The Controlled Vocabulary Manager (CVM) provides a service for finding terms by various parameters. The service is used by the Machine Learning Clients to provide a list of terms for classifying features in an image. The CVM could be fully integrated into the OGC AI/ML architecture and used to provide a common vocabulary to all components within the architecture: ML System (WPS), Knowledge Base, Machine Learning / Image Analyst Clients, and Image/Feature Store. Other topics for consideration include standardization, from an OGC context, of the service interface and data exchange formats used by the CVM service, for example potential use of the OGC Catalogue Service interface (CSW) and data encodings such as GML.

Inline with Testbed-14 EOC Execution Management System (EMS) best practices, consider exploration of advanced platform-based use cases for AI. GeoInt literature describes "query interpreters" and subsequent "query conductors" for federated subsystems. Query interpreters use NLP, a form of AI, to understand text-based user queries and automatically produce federated workflows. Currently, work in EOC describes use of graphical workflow editors. Future work for EMS could include experiments on automatic generation of workflows or selection of catalogued workflows that are closest to a user query, thus advancing "query interpreter" concept. Experiments in the creation of Analysis Ready Data and Dynamic Analysis Ready Data could support such efforts.

Experiment with use of tiles (WMTS) to directly feed training patches for DL. Several DL implementations for supervised semantic segmentation use raw GeoTiff images as input. Patches are often extracted from generating a bounding box around a labelled pixel. An experiment using tiles would either allow annotations at the tile level or allow efficient fetching of tiles belonging to a labelled feature (point, region). Considering that efficient subsetting and fetching of imagery data is one of the biggest advantages of using tiles, this suggests an experiment involving the use of OGC Web Services such as WMTS.

Tiles can also enable efficient DL training for several levels of resolution. In this experiment, annotations made on VHR imagery could be cascaded to lower resolutions. In such an experiment, there are tradeoffs to the use of tiles. For instance, only 3 bands are fetched per request by default, limiting usefulness with multispectral imagery. Bias can be introduced by data preprocessing, hindering use for scientific studies. Consider extensions to WMTS to support larger number of bands or to describe metadata related to image pre-processing.

Experiment with the use of 3D tiles in Augmented Reality (AR) or Mixed Reality environments in order to produce interoperable annotations between 2D, grid-based annotations and their 3D environment counterparts. For example, assuming grid-based annotation are more mature, an experiment using 3D tiles could allow filtering, retrieval and visualization of 2D annotations produced in spatial proximity of the AR user. The user could then enrich the labels according to his tasks, either by adding an elevation component or triggering image capture from the ground. Also, assuming labelled ground-level imagery is abundant (Flicker, Google Image, etc.) and related ML practices in the literature abounds, the image capture from AR users could be structured similarly and be consumed by classical computer vision models.

An experiment in an AR environment could also benefit from semantic contextualization. For instance, if the user is a humanitarian worker on the field during a disaster scenario, he/she would be interested in recent satellite imagery annotations in his area related to structural damages, injuries or logistics.

Every time a user runs an image classification model, the output image is automatically sent into the Image Repository – meaning it comes up as a source/training target for the next user. This is not a significant issue in a demonstration environment, but a mechanism to control this is required in production to avoid flooding the Image Repository with indecipherable output data.