1.1.  Terms and definitions

Moving feature

A representation, using a local origin and local ordinate vectors, of a geometric object at a given reference time (ISO 19141:2008). In the context of this ER, the geometric object is a feature, which is an abstraction of real world phenomena (ISO 19109:2015).

Tracking

Monitoring and reporting the location of a moving object (adapted from ISO 19133:2005).

Tracklet

A fragment of the track followed by a moving object.

Trajectory

Path of a moving point described by a one parameter set of points (ISO 19141:2008).

Trajectory mining

The study of the trajectories of moving objects in order to find interesting characteristics, detect anomalies and discover spatial and spatiotemporal patterns among them.

1.2.  Abbreviated terms

API

Application Programming Interface

MF

Moving Feature(s)

MISB

Motion Imagery Standards Board

ML

Machine Learning

MPEG

Moving Picture Experts Group

MSE

Mean Squared Error

VMTI

Video Moving Target Indicator

WebVMT

Web Video Map Tracks

WICG

Web Platform Incubator Community Group

WMS

Web Map Service

W3C

World Wide Web Consortium

4.1.  Requirements

Testbed 16 demonstrated that Motion Imagery derived Video Moving Target Indicators (VMTI) can be extracted from an MPEG-2 motion imagery stream and represented as OGC Moving Features or Web Video Map Tracks (WebVMT).

The work performed in the TB-17 MF task formalized an architecture for integrating moving object detections, proposed standards for the required APIs and content encodings, expanded the sources of moving object detection that can be supported, and explored exploitation and enhancement capabilities which would leverage the resulting store of moving features.

The architecture includes the following components :

1. Detection ingest: This component will ingest data from a moving object detection system, extract detections and partial tracks (tracklets), and export the detections and tracklets as OGC Moving Features.

2. Tracker: This component ingests detections and tracklets as OGC Moving Features, then correlates them into longer tracks. Those tracks are then exported as OGC Moving Features

3. Data Store: provides persistent storage of the Moving Feature tracks.

4. Machine Analytics: software which enriches the existing tracks and/or generates derived information from the tracks

5. Human Analytics: software and tools to help users exploit the Motion Imagery tracks and corresponding detections or correlated tracks. For example, a common operational picture showing both static and dynamic features.

This work was demonstrated using a real-time situational awareness scenario.

4.2.  Scenario use case

This ER describes the detection and tracking of moving buses in front of a school. The video was acquired by a light-weight Unmanned Aerial Vehicle (UAV) or drone, courtesy of University of Calgary.

A separate autonomous vehicle use case was analyzed to detect and track people and vehicles moving nearby with WebVMT. Video and lidar data were captured from a moving StreetDrone vehicle and provided courtesy of Ordnance Survey UK.

6.1.  Introduction

The Ingestion service receives raw data from sensors or edge computers and converts it into Observations (See SensorThing’s specification for details). The service then posts these observations to the Storage service where it is interpreted and stored as Features (See Feature specification for details). There are three components to an Ingestion service; the receiver, the convertor and the sender. The receiver reads in the raw data. The convertor parses the data and turns it into SensorThing’s Observations. Finally, the sender component publishes the Observations to the Storage service via the MQTT protocol.

6.2.  Ingestion process architecture

As the developed ingestion service ingests tracklets from detected objects in video frames, the following architecture was designed to handle the ingestion tasks.

Figure 4 — Ingestion process architecture

In Testbed-17, raw data providers were drones and stationary cameras that push raw video frames to the deep learning computer. The deep learning computer detects moving features (buses) using a pre-trained model in each frame and then builds tracklets one by one using a prediction and estimation algorithm for consecutive frames. The result of this procedure is the object bounding box (bbox), class, track_id, color, and the detection time. Then these tracklets are sent to the ingestion service using http POST. The ingestion service was implemented on an AWS EC2 service with Ubuntu server x64 OS. The flask web framework Nginx, was used as the web server, and Guncorn used as the web server gateway to handle the ingestion service. The EC2 service takes the data in the format that is mentioned in the Input section . Ingestion service includes a camera registration module which lets users register cameras with their metadata in the ingestion service. To register a camera, a route in the service based on the following format was created:

http://52.26.17.1:5000/register_cam

To register a camera the following payload should be posted.

{
    "id":"name of the camera",
    "cam_location":[longitude,latitude],
    "image_coords":[[x0, y0], [x1, y1], [x2, y2],...,[xn, yn]],
    "ground_coords":[[longitude0, latitude0],
                        [longitude1, latitude1],
                        [longitude2, latitude2],
                        [longitude3, latitude3],...,[longitude_n, latitude_n]]
}

The camera metadata is stored in a MongoDB (nosql) database which can be used for transformation of image coordinates to geographic coordinates. Also, the ingestion service uses this data to let the storage service know what the source of the observations is. Based on the Testbed-17 Moving Feature architecture, cameras are registered as Things in the SensorThings API (STA) model. The names of Thing instances are used to let other services, such as machine analytics, access the raw or process video streams.

After the camera registration in the ingestion service, tracklets are posted to the relevant camera route in the ingestion service. To do so, the following route should be used.

http://52.26.17.1:5000/ingestion/Camera_name

The payload input format of this POST request is mentioned in the Input section.

After receiving the tracklets from the deep learning computer, based on the camera that records the video, coordinates are transformed into longitude and latitude . For the transformation, image and the corresponding ground control points were employed. The details are explained in the Transformation section.

Finally, tracklets are enriched with the geographic coordinates and are published in the STA observation format (which is discussed in the Output section) in MQTT payloads. The storage service endpoint details for receiving the tracklets from ingestion services is:

{
    "broker": "tb17.geomatys.com",
    "port": 30170,
    "topic": "/Observations",
    "Datastream": 1
}

6.3.  Transformation

The final task was to transform objects’ locations from the video space into geographic space. To do so, a homomorphic model which is similar to 2D projective transformation, was used. Homography includes 8 main variables in a 3×3 transformation matrix. So, to resolve the transformation, at least 4 GCP (Ground Control Point) are required. This approach is widely used for transforming objects from one planar space to another planar space. [3] In the following Figure, one of the plains can be considered as the image frame and the other one would be the ground plane in the geography space.

Figure 5 — Homography transformation (source: http://man.hubwiz.com)

6.4.  Input

Sensors should push tracklets data into the ingestion service while the tracklet payload must have bbox, time, and trackid as attributes. Also, the ingestion service can process bbox and color of objects as optional attributes. The input format of the ingestion service:

{
    "class":"bus",
    "track_id":23,
    "bbox":[23,12,34,51],
    "time":"2021-05-13T00:09:18Z",
    "color":[255,255,0]
}

definitions:

class: The moving object type which can be car, bus, bicycle, person, …​

track_id: The unique id of the tracked moving object

bbox: Bounding box of objects in the camera coordinate system. [upper-left x, upper-left y, width, height]

time: The phenomenon time or the time that object has been detected on video frames

color: The mean RGB color code of the detected object

6.5.  Functions

The ingestion service uses a NoSQL database to register cameras. They are stored in the camera table as json objects. These objects include coordinates transformation parameters as well as camera metadata. This information is used in the conversion part of the ingestion service when image coordinates are received by the ingestion service. The output of this conversion is the geographic coordinate of the moving object. The following shows the payload for registering cameras in the ingestion service.

{
    "id":"GoProTestbed17",
    "cam_location":[ -114.16064143180847, 51.085716521902036],
    "image_coords":[[629, 881], [1201, 695], [855, 604],[1808, 572]],
    "ground_coords":[[-114.160505, 51.085698],
                    [-114.160515, 51.085604],
                    [-114.160166, 51.085476],
                    [-114.160769, 51.085106]]
}

After points transformation, the STA function converts the received data to the STA format and sends it to the storage service endpoint in MQTT format. The following shows an example of the output of the ingestion service to the storage service.

{
    "phenomenonTime": "2021-07-27T04:03:03Z",
    "resultTime": "2021-07-27T04:03:03Z",
    "result": 48,
    "FeaturesOfInterest": {
        "name": "48,-114.16081989038098,51.085107555986895",
        "description": "BusMovingObject",
        "encodingType": "application/vnd.geo+json",
        "feature": {
            "type": "Feature",
            "properties": {
                "image_bbox": [
                    944,
                    765,
                    123,
                    59
                ],
                "image_color": [
                    42,
                    38,
                    41
                ],
                "class": "bus"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -114.16081989038098,
                    51.085107555986895
                ]
            }
        }
    },
    "Datastream": {
        "@iot.id": 60987
    }
}

The other module is the Compusult Component which feeds the D135 ingestion service.

6.6.  Data

To show the capabilities of the ingestion service, two videos were recorded from two different points of views. One of them was recorded from a drone’s point of view and the other one was recorded from a fixed GoPro camera. The first camera points to the bus station and the second camera points to the street which ends at the bus station. Using the methods which has been described in the next chapter, objects are detected and tracked. Then the detected objects are sent to the ingestion service, as soon as they are observed in a frame, using the HTTP POST method. Figure 1 illustrates a frame of detected objects by GoPro camera and Figure 2 shows a snapshot of a bus station recorded by the drone. These two videos are almost synced.