Publication Date: 2021-01-13
Approval Date: 2020-12-15
Submission Date: 2020-11-20
Reference number of this document: OGC 20-036
Reference URL for this document: http://www.opengis.net/doc/PER/t16-D021
Category: OGC Public Engineering Report
Editor: Emeric Beaufays, C.J. Stanbridge, Rob Smith
Title: OGC Testbed-16: Full Motion Video to Moving Features Engineering Report
COPYRIGHT
Copyright © 2021 Open Geospatial Consortium. To obtain additional rights of use, visit http://www.opengeospatial.org/
WARNING
This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Public Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.
LICENSE AGREEMENT
Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.
If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.
THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.
This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.
Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.
This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.
None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.
- 1. Subject
- 2. Executive Summary
- 3. References
- 4. Terms and definitions
- 5. Description of standards
- 6. Tracking Algorithm
- 7. Encoding to OGC Moving Features
- 8. WebVMT: Exporting MISB Metadata for the Web
- 9. Software Design and Demonstrator overview
- 10. Test Scenarios
- 11. Discussion
- Appendix A: Revision History
- Appendix B: Bibliography
1. Subject
This OGC Testbed-16 Engineering Report (ER) evaluates the suitability of existing OGC standards for the generation of Moving Features from Full Motion Video (FMV) that has an embedded stream of detected moving objects.
This ER presents several proof of concept applications that accept FMVs, with multiple encoded Video Moving Target Indicators (VMTI), and combines the VMTIs into separate tracks that are then encoded to OGC Moving Features.
In addition, the ER explores the generation of records encoded according to OGC Sensor Model Language (SensorML) 2.0 standard describing the collection platform and relevant telemetry information from the key-value stream content encoded according to the MISB 0601 and 0903 specifications of the Motion Imagery Standards Board (MISB).
2. Executive Summary
This OGC ER documents work completed in the OGC Testbed-16 Full Motion Video (FMV) thread.
Unmanned Aerial Vehicles (UAVs) are increasingly used in military and civil applications. A video camera is often the primary sensor on a UAV, providing a torrent of imagery of the overflown ground. From this imagery, detecting the presence of moving objects on the ground, such as vehicles and people, can be of strategic importance.
Modern Motion Imagery platforms are capable of extracting a considerable body of information from the raw collection and then streaming that information in-band with the video stream. However, this information comes in a form which is not readily exploitable except by specialized systems.
STANAG 4609, MISB Std. 0601 (UAS metadata) and MISB Std. 0903 (VMTI) define encoding a video with frame by frame metadata that may contain any number of VMTIs. However, the individual moving object detections can be made more useful when combined into moving features or "tracks". This process is summarized in the tracking algorithm chapter. This process is a non-trivial problem. The experimental methods and results are further described in this document.
(a) Each frame of the video can contain several detections. (b) The detections are decoded and referenced according to the camera telemetry. (c) The detections throughout the video are accumulated. (d) The different tracks are inferred.
In addition, the OGC has developed several standards to serve as the interoperability medium for the outcome and description of the above process which are assessed in their suitability to the FMV to moving feature scenario.
This report explores the following topics:
-
Assessment of the suitability of OGC Moving Features and OGC O&M as interoperability formats for moving features inferred from VMTI detections.
-
Implementation of a process to generate tracks from VMTI detections.
-
Encoding to OGC Moving Features, OGC O&M, and WebVMT.
-
Generation of OGC SensorML 2.0 records describing the collection platform and relevant telemetry information from the key-value stream (0601 and 0903 content).
-
Implementation of demonstrators for the generation of OGC Moving Features, Observations, and SensorML documents from sample Full Motion Video streams.
-
Subsequent conversion of OGC Moving Features data into the SensorThings API, permitting access to JSON-encoded Moving Features observations through SensorThings-based sites and architectures.
2.1. Key findings and results
OGC Moving Features is an easy to read and write format with csv, json and xml implementations. The standrd specifies how to encode the result of a tracking algorithm in a simple and readable fashion and enables a one to one mapping of all the metadata from the original MISB 0903 VMTIs. However, the mapping of certain properties like embedded Geography Markup Language (GML) and Web Ontology Language (OWL) classes can be complex and will be difficult to define an encoding for those properties that can be interpreted by a generic decoder.
OGC O&M offers similar capabilities relative to OGC Moving Features with the difference that specific attention is given to map ontologies. This is an advantage of OGC O&M over OGC Moving Features but comes at a cost of complexity and is only useful for MISB 0903 data that carries relevant metadata.
SensorThings is a useful API in which to encode results from Full-Motion Video and Moving Features sources due to its versatile support for displaying readings from varied sensors. The toolset that converts FMV sources to Moving Features and SensorThings (ST) follows a consistent high-level workflow for conversion the outputs to be accessible via the ST API and convey them to dashboards for user observation.
Web Video Map Tracks (WebVMT) is an open web format based on JavaScript Object Notation (JSON) and W3C Web Video Text Tracks (WebVTT). WebVMT offers some similarities to its OGC counterparts for moving objects and sensors with mapping to MISB 0903 VMTI metadata and the additional benefit of encapsulation to support any format that can be encoded as a json object. Generic decoders can identify data by type and seamless integration with HTML DataCue enables metadata access in web browsers. The DataCue API enables web pages to associate arbitrary timed data with audio or video media resources, and for exposing timed data from media resources to web pages.
2.2. Business value
This testbed work contributed business value by providing a semantic model which integrates MISB, National Imagery Transmission Format (NITF), Sensor Web Enablement (SWE), Semantic Sensor Network (SSN) ontology, Moving Features, STANAG 4676, SensorThings, WebVMT, and other sensor models.
2.3. Document contributor contact points
All questions regarding this document should be directed to the editor or the contributors:
Contacts
Name | Organization | Role |
---|---|---|
Emeric Beaufays |
Hexagon |
Editor |
Frederic Houbie |
Hexagon |
Editor |
C. J. Stanbridge |
Compusult |
Contributor |
Rob Smith |
Away Team Software |
Contributor |
2.4. Foreword
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.
4. Terms and definitions
- ● FMV|Full-Motion Video
-
high-fidelity digitally encoded video, defined as that whose data is stored at a rate of 25 or more frames per second
- ● MF|MovingFeatures
-
OGC API for defining the paths of moving points, lines, and solid geometrical entities as well as the static and dynamic properties of sensors that are located on those moving objects. MovingFeatures data can be expressed as either XML or JSON.
- ● MISB|Motion Imagery Standards Board
-
agency established by directive of the United States Department of Defense “to formulate, review, and recommend standards for Motion Imagery”
4.1. Abbreviated terms
API |
Application Programming Interface |
BER |
Basic Encoding Rules |
BER-OID |
Basic Encoding Rules, Object IDentifiers |
ER |
Engineering Report |
FMV |
Full-Motion Video |
FOI |
Feature of Interest |
GML |
Geography Markup Language |
HTML |
HyperText Markup Language |
IMAPB |
[floating-point to] Integer MAPping using starting point B (defined by MISB 1201) |
I/O |
Input/Output |
IoT |
Internet of Things |
ISO |
International Organization for Standardization |
JSIL |
Joint System Integration Laboratory (United States Department of Defense) |
JSON |
JavaScript Object Notation |
JSP, .jsp |
Jakarta Server Pages (formerly JavaServer Pages) |
KLV |
Key-Length-Value |
LDS |
Local Data Set |
LS |
Local Set |
MF |
MovingFeatures |
MIME |
Multipurpose Internet Mail Extension |
MISB |
Motion Imagery Standards Board |
MISP |
Motion Imagery Standards Profile |
MPEG-2 |
Moving Picture Experts Group standard 2 |
MQTT |
Message Queuing Telemetry Transport |
NITF |
National Imagery Transmission Format |
OGC |
Open Geospatial Consortium |
O&M |
Observations and Measurements |
PAT |
Program Association Table |
PES |
Program Elementary Stream |
PMT |
Program Map Table |
POC |
Proof of Concept |
REST |
REpresentational State Transfer |
SDO |
Standards Development Organization |
SensorML |
Sensor Model Language |
SMPTE |
Society of Motion Picture and Television Engineers |
SSN |
Semantic Sensor Network |
STA |
SensorThings API |
STANAG |
STANdardization AGreement |
SWE |
Sensor Web Enablement |
TCP |
Transmission Control Protocol |
TS |
Transport Stream |
UAS |
Unmanned Air System |
UDP |
User Datagram Protocol |
UDS |
Universal Data Set |
UI |
User Interface |
UML |
Unified Modeling Language |
UOM, uom |
Unit of Measure |
URI |
Uniform Resource Identifier |
VMTI |
Video Moving Target Indicator |
WebVMT |
Web Video Map Tracks |
XML |
eXtensible Markup Language |
5. Description of standards
5.1. Overview
This section provides a brief description of the standards that are used and evaluated both as input and output to the proof of concept.
5.2. STANAG 4609
STANAG 4609 describes an exchange format for motion imagery. It is the official format for motion imagery (video data, image sequences, FMV - full motion videos) exchange within the NATO nations. Motion imagery is defined by MISB to be video of at least 1 Hz image frequency together with metadata. STANAG 4609 describes the encoding of the video and the metadata (geographical data) for different usages. This includes the supported video codecs, bit rates, frame rates, container formats, metadata content, metadata encoding and hardware to distribute the motion imagery.
The standards which make television and cable networks possible are established and maintained by the Society of Motion Picture and Television Engineers (SMPTE).
5.3. SMPTE ST 336
SMPTE ST 336 defines a byte-level data encoding protocol for which can be multiplexed with a video stream. Synchronization between the key-value pairs and the associated video frames is maintained using the same mechanism as is used to synchronize the audio and video streams. SMPTE also defines a small set of Keys in SMPTE ST 335.
The Motion Imagery Standards Board (MISB) has built on those standards to address additional requirements identified by the Defense and Intelligence communities. Those standards are codified in the Motion Imagery Standards Profile (MISP) and STANAG 4609. The MISB standards most relevant to this effort are MISB 0601 and MISB 0903. MISB 0601 defines the basic set of keys for use by UAS systems. MISB 0903 defines additional keys for Video Moving Target Indicators (VMTI). Moving Target Indicators are reports on detections of objects in a FMV frame which appear to be moving, along with any additional descriptive information that the detector can provide.
5.4. OGC Moving Features
This OGC Standard specifies standard encoding representations of movement of geographic features. The primary use case is information exchange. The encodings specified in the OGC® Moving Features suite of standards conform to the ISO 19141:2008, Geographic information – Schema for moving features standard.
A feature is an abstraction of a real-world phenomenon. A geographic feature if it is associated with a location relative to the Earth is a geographic feature. ISO 19141:2008 represents moving features by defining a geometry for a feature that moves as a rigid body. This allows moving and stationary features to be analyzed and exploited using the same algorithms and software.
5.5. OGC SensorML
Sensor Model Language (SensorML) provides a robust and semantically tied means of defining processes and processing components associated with the measurement and post-measurement transformation of observations. This includes sensors and actuators as well as computational processes applied pre- and post-measurement. The main objective is to enable interoperability, first at the syntactic level and later at the semantic level (by using ontologies and semantic mediation). This is so sensors and processes can be better understood by machines, utilized automatically in complex workflows, and easily shared between intelligent sensor web nodes.
5.6. OGC Observations and Measurements
The Observations and Measurements (O&M) Standard defines a conceptual schema for sensor observations, and sampling features produces when making observations. These provide models for the exchange of information describing observation acts and their results, both within and between different scientific and technical communities. Observations commonly involve sampling of an ultimate feature of interest. O&M defines a common set of sampling feature types classified primarily by topological dimension, as well as samples for ex-situ observations. The schema includes relationships between sampling features (sub-sampling, derived samples).
5.7. OGC SensorThings API
The SensorThings API (STA) OGC standard defines the interconnection and communication of Internet of Things (IoT) devices. It is comprised of a sensing part, which allows data observed by multiple IoT sources to be conveyed using a standard JSON-based format, and a tasking part, which allows events to be activated based on the values of these observations. All observations, metadata, and messages generated in this API can be forwarded to MQTT or to RESTful endpoints for later use by clients.
6. Tracking Algorithm
6.1. Overview
This section provides a general description of a Multiple Object Tracking algorithm that computes tracks from the individual MISB 0903 Video Moving Target Indicators (VMTI). The algorithm uses a global nearest neighbor approach that takes into account a speed and bearing estimate of the moving object. This algorithm has the advantage of being able to run on a video stream, rather than needing to collect all the detections first, as well as being fast enough to run in real-time.
MISB 0903 concentrates on video tracking as a VMTI is essentially a location and bounding box. Tracking through Lidar sensors is not prohibited. The standard is not meant for radar tracking. The standard for Radar moving target detections is STANAG 4607. The algorithm makes the assumption that a single object is described by a single target and that a single target describes a single object (Point Object Tracking). Missing detections and false positives are dealt with as corner cases.
6.1.1. Initialization
The first phase of the algorithm is the initialization or "seeding" of tracks where new detections are used to instantiate tracks.
VMTIs from the first frame or VMTIs that cannot be matched to an existing track each begin a new track. The VMTIs from the following frames are loaded. Each track looks for VMTIs within a given radius of the last known position. The new VMTIs that are matched to existing tracks are used to extend them. VMTIs that don’t match to any existing track start a new track
6.1.2. Expansion
Existing tracks are expanded by matching detections to the track’s predicted location based on speed and bearing.
Based on the direction and speed of the moving object, their predicted location is computed. The algorithm looks for the next track location within a given radius of the predicted location and expands the track. If no VMTI is found within the radius, the track is finalized after some time.