OGC Testbed-16: Full Motion Video to Moving Features Engineering Report

OGC Moving Features is an easy to read and write format with csv, json and xml implementations. The standrd specifies how to encode the result of a tracking algorithm in a simple and readable fashion and enables a one to one mapping of all the metadata from the original MISB 0903 VMTIs. However, the mapping of certain properties like embedded Geography Markup Language (GML) and Web Ontology Language (OWL) classes can be complex and will be difficult to define an encoding for those properties that can be interpreted by a generic decoder.

OGC O&M offers similar capabilities relative to OGC Moving Features with the difference that specific attention is given to map ontologies. This is an advantage of OGC O&M over OGC Moving Features but comes at a cost of complexity and is only useful for MISB 0903 data that carries relevant metadata.

SensorThings is a useful API in which to encode results from Full-Motion Video and Moving Features sources due to its versatile support for displaying readings from varied sensors. The toolset that converts FMV sources to Moving Features and SensorThings (ST) follows a consistent high-level workflow for conversion the outputs to be accessible via the ST API and convey them to dashboards for user observation.

Web Video Map Tracks (WebVMT) is an open web format based on JavaScript Object Notation (JSON) and W3C Web Video Text Tracks (WebVTT). WebVMT offers some similarities to its OGC counterparts for moving objects and sensors with mapping to MISB 0903 VMTI metadata and the additional benefit of encapsulation to support any format that can be encoded as a json object. Generic decoders can identify data by type and seamless integration with HTML DataCue enables metadata access in web browsers. The DataCue API enables web pages to associate arbitrary timed data with audio or video media resources, and for exposing timed data from media resources to web pages.

This testbed work contributed business value by providing a semantic model which integrates MISB, National Imagery Transmission Format (NITF), Sensor Web Enablement (SWE), Semantic Sensor Network (SSN) ontology, Moving Features, STANAG 4676, SensorThings, WebVMT, and other sensor models.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

This section provides a brief description of the standards that are used and evaluated both as input and output to the proof of concept.

STANAG 4609 describes an exchange format for motion imagery. It is the official format for motion imagery (video data, image sequences, FMV - full motion videos) exchange within the NATO nations. Motion imagery is defined by MISB to be video of at least 1 Hz image frequency together with metadata. STANAG 4609 describes the encoding of the video and the metadata (geographical data) for different usages. This includes the supported video codecs, bit rates, frame rates, container formats, metadata content, metadata encoding and hardware to distribute the motion imagery.

The standards which make television and cable networks possible are established and maintained by the Society of Motion Picture and Television Engineers (SMPTE).

SMPTE ST 336 defines a byte-level data encoding protocol for which can be multiplexed with a video stream. Synchronization between the key-value pairs and the associated video frames is maintained using the same mechanism as is used to synchronize the audio and video streams. SMPTE also defines a small set of Keys in SMPTE ST 335.

The Motion Imagery Standards Board (MISB) has built on those standards to address additional requirements identified by the Defense and Intelligence communities. Those standards are codified in the Motion Imagery Standards Profile (MISP) and STANAG 4609. The MISB standards most relevant to this effort are MISB 0601 and MISB 0903. MISB 0601 defines the basic set of keys for use by UAS systems. MISB 0903 defines additional keys for Video Moving Target Indicators (VMTI). Moving Target Indicators are reports on detections of objects in a FMV frame which appear to be moving, along with any additional descriptive information that the detector can provide.

This OGC Standard specifies standard encoding representations of movement of geographic features. The primary use case is information exchange. The encodings specified in the OGC® Moving Features suite of standards conform to the ISO 19141:2008, Geographic information – Schema for moving features standard.

A feature is an abstraction of a real-world phenomenon. A geographic feature if it is associated with a location relative to the Earth is a geographic feature. ISO 19141:2008 represents moving features by defining a geometry for a feature that moves as a rigid body. This allows moving and stationary features to be analyzed and exploited using the same algorithms and software.

Sensor Model Language (SensorML) provides a robust and semantically tied means of defining processes and processing components associated with the measurement and post-measurement transformation of observations. This includes sensors and actuators as well as computational processes applied pre- and post-measurement. The main objective is to enable interoperability, first at the syntactic level and later at the semantic level (by using ontologies and semantic mediation). This is so sensors and processes can be better understood by machines, utilized automatically in complex workflows, and easily shared between intelligent sensor web nodes.

The Observations and Measurements (O&M) Standard defines a conceptual schema for sensor observations, and sampling features produces when making observations. These provide models for the exchange of information describing observation acts and their results, both within and between different scientific and technical communities. Observations commonly involve sampling of an ultimate feature of interest. O&M defines a common set of sampling feature types classified primarily by topological dimension, as well as samples for ex-situ observations. The schema includes relationships between sampling features (sub-sampling, derived samples).

The SensorThings API (STA) OGC standard defines the interconnection and communication of Internet of Things (IoT) devices. It is comprised of a sensing part, which allows data observed by multiple IoT sources to be conveyed using a standard JSON-based format, and a tasking part, which allows events to be activated based on the values of these observations. All observations, metadata, and messages generated in this API can be forwarded to MQTT or to RESTful endpoints for later use by clients.

This section provides a general description of a Multiple Object Tracking algorithm that computes tracks from the individual MISB 0903 Video Moving Target Indicators (VMTI). The algorithm uses a global nearest neighbor approach that takes into account a speed and bearing estimate of the moving object. This algorithm has the advantage of being able to run on a video stream, rather than needing to collect all the detections first, as well as being fast enough to run in real-time.

MISB 0903 concentrates on video tracking as a VMTI is essentially a location and bounding box. Tracking through Lidar sensors is not prohibited. The standard is not meant for radar tracking. The standard for Radar moving target detections is STANAG 4607. The algorithm makes the assumption that a single object is described by a single target and that a single target describes a single object (Point Object Tracking). Missing detections and false positives are dealt with as corner cases.