OGC Testbed-14: CityGML and AR Engineering Report

This ER provides findings and recommendations which may impact a number of OGC Working Groups that deal with standards for working with 3D geospatial data.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

As the use of CityGML was an important objective of this initiative, attempts were made to identify useable data sets, as well as CityGML Application Domain Extensions (ADEs) which may provide information more particularly useful for Augmented Reality experiments. Existing CityGML ADEs identified included:

Also relevant are the open source BIM collective [4], Indoor Spatial Data Model Using CityGML ADE [5], application of large 3D CityGML models on smartphones [6] and Augmented Reality Games with CityGML [7].

However, as it was already difficult to find a basic usable CityGML data set in an area of interest, let alone one defining any of these ADEs, no such data sets were identified.

Being able to see "hidden" features is a very useful aspect of AR, displaying underground utilities such as pipelines, is a good use case scenario. A data set of a pipeline in Kaohsiung, Taiwan available in the COLLADA format was used for some experiments in the Testbed. The location of the content however, proved problematic as no testbed participant was normally located in Kaohsiung.

The Testbed sponsors originally specified requirements for supporting 3D Tiles and/or I3S. They also hoped for a continuity of the work performed in Testbed-13 on 3D client performance. As a result, the focus shifted on the presentation of 3D buildings in an AR application. One problem with the presentation of 3D buildings in AR is that the actual buildings are normally directly visible, without the need for augmentations. As such, a 3D model would simply obstruct a real, better view of the actual building. However, having the 3D model could potentially be useful in an emergency response scenario where thick fog or smoke could severely restrict visibility. Alternatively, presenting the buildings with a reduced opacity setting may be useful to verify the accuracy of a data set with the real world, and/or for mapping one’s surroundings. The vast majority of the experiments performed in this initiative dealt with this 3D buildings scenario.

Unfortunately, the data sets used had little attribute information available. However, the approach implemented for transforming and transmitting buildings data kept into consideration the capability for preserving attributes when better attributed data is available.

In order to experiment with the capacity to display actual annotations apart from 3D models, separate and combined experiments were done using OpenStreetMap data for presenting street maps, including buildings and streets names.

Although desktop experiments were done with terrain elevation and aerial imagery, the very limited nature of mobile hardware, as well as the desire to not hide the majority of the camera feed made it difficult to find a proper use case for these data in a mobile Augmented Reality scenario. In one case however, information from the elevation model data was incorporated within the buildings position information.

See Chapter 10 for a detailed description of the data sets used for all experiments, as well as how they were processed and integrated for use by the services and Augmented Reality clients.

In this initiative, generic geospatial datasets, with no special consideration for Augmented Reality, were used so that AR applications developers could readily make use of any geospatial datasets such as those defined or transmitted using OGC standards. Similarly, current GIS applications developers can leverage their existing tools, data and expertise to start deploying Augmented Reality solutions.

Many of the same concepts for styling geospatial features apply whether dealing with a cartographic or 3D view (whether doing Augmented Reality or not). As an example, any flat feature can be simply draped on the ground, and the same stroke or fill symbolizers can be applied to the feature(s). Labels and markers (annotations) typically still face the user as billboards, even if their 3D position serves as a basis for projecting to a screen position.

Some additional capabilities are possible however, such as defining styles specific to solid shapes / volumes, or using 3D models as markers. There is also the concept of depth of objects, which is sometimes useful to have objects sorted back to front with actual depth values.

With this in mind, rather than defining something entirely distinct, it makes sense to extend the classic GIS styling model with 3D capabilities.

A great advantage of unifying styling representations is to be able to easily integrate any styled GIS content within an AR application.

This is done through the same mechanism as specifying cartographic styling for vector features in a typical cartographic 2D view, with a symbolizer concept associated with a feature described in a generic manner which accommodates 3D views just as well as 2D views.

As part of the ongoing next iteration of defining a standard conceptual model for styling capabilities, these possibilities were also considered within the Testbed-14 portrayal thread activities.

This conceptual model for styling features, as well as an encoding for it, is detailed in Appendix C.

By configuring how the Augmented Reality content is displayed on either the camera feed, or the rendering of a virtual 3D view of the integrated urban model represented by the CityGML, entirely on the client, a great level of flexibility is gained. This linking can be established by specifying styling rules, and referencing available attributes associated with the geospatial data. In the approach used in this Testbed, the description of annotations elements was not hardcoded in either the source CityGML or the transmission format.

Capabilities very specific to Augmented Reality could be extensions to a conceptual model for styling geospatial data. This would provide a mechanism by which to re-integrate ARML capabilities within this conceptual model.

At a very close range, the GPS precision is not enough to accurately register the camera with the surroundings. Indoor, the GPS signal is blocked, and devices cannot communicate with satellites. Virtual sensors such as the Android fusion sensor providing 6 degrees of freedom integrating the gyroscope, accelerometer, magnetometer in addition to input from the camera may solve some of these challenges. However, potentially recognizing features from a pre-existing data set and/or leveraging more advanced computer vision techniques on the camera feed could help solve some of these challenges.

Access to detailed in-door data sets is key for looking at these challenges in the context of Augmented Reality. Perhaps insights from the Indoor mapping and Navigation pilot would prove useful. Additionally, hardware with stereoscopic cameras might offer better ways to more accurately register the view. It is also possible to apply pattern recognition, and place markers in the real world at specific locations where augmentations should appear.

Finally, augmentation of reality at a smaller scale is generally more interesting and useful, making it possible to display more relevant content which is part of the user’s immediate surroundings. The heavy reliance on the complex topic of computer vision however implies more involved processes, and for this reason this was reserved for a future initiative.

The (rear-facing) camera of the mobile device provides a live feed of the user’s surroundings. It supplies an application with the backdrop in which augmentations can be integrated so that the user feels as though these are part of their actual surroundings. In computer vision-based Augmented Reality, the camera feed is not merely a backdrop but is analyzed in real time to identify anchoring patterns or to precisely locate features. Through the use of virtual fusion sensors such as Android’s six degrees of freedom sensor (TYPE_POSE_6DOF), the camera can also play an integral part in maintaining accurate information about the position and orientation of the device.

The Ecere client directly accessed the Android camera using the Android Camera 2 API. The Ecere client is built using the native eC programming language rather than Android’s Java development language. Further the NDK API for the Camera 2 was only added in Android API Level 24 (corresponding to Android 7.0 — Nougat). As there was a desire to support devices with an earlier version of Android, the Java Native Interface (JNI) had to be used. This decision required writing extra code to make use of that Camera API, thus complicating the task of implementing camera support. By using the Camera 2 API through JNI, it was possible to support Android devices starting from API Level 21 (corresponding to Android 5.0 — Lollipop). This code will likely be published under an open-source license as part of the Ecere SDK, and may be re-usable in other projects (including in projects written in other native languages such as C or C++, with minor modifications).

That camera code uses a capture session set up with repeating requests to continuously capture images. Using a lower resolution than the camera is capable of capturing can provide smoother performance. The performance of the current setup is still being investigated as it is believed that performance can be further improved, potentially by taking a more direct path between the captures and displaying the image on the device.

Obtaining the GPS coordinates is the most accurate means for retrieving the device’s location. However, using the on-board GPS chip only works outdoors, and requires significant power. The network location mechanism can use the cell tower and Wi-Fi triangulation, but these computed locations are fairly inaccurate.^[1]

Ideally for a geospatial AR application, the GPS unit should be used to provide coordinates. This could prove a real challenge for indoor Augmented Reality applications (possibly the location could first be obtained as close as possible from outside the building). Using the GPS provides the absolute frame of reference for the device’s position in relation to the Earth.

Because the Android NDK does not provide an API for obtaining the GPS coordinates and location updates, the JNI also had to be used to set up location updates and access the GPS chip in the Ecere Android client, through the LocationManager class.

In addition to latitude and longitude, the on-board GPS chip can provide altitude information. The altitude returned by Location.getAltitude() will be the altitude above the WGS84 reference ellipsoid, not the mean sea level. The latter can be obtained by taking into account the difference between the EGM96 geoid and the WGS84 reference ellipsoid, which varies between a difference of -100 and 80 meters).

This altitude information from the GPS has not yet been used in the Ecere client.

A guide to the Android motion sensors can be found here.

The use of location updates, sensors and the camera capabilities is particularly costly in terms of power. Turning off such capabilities when not required is essential to minimize this impact. This is in addition to potentially heavy use of the Graphical Processing Unit (GPU) for rendering a large number of items and/or 3D geometry. This makes Augmented Reality applications rather power hungry, as they can quickly drain a device’s battery. Correspondingly, it also produces a lot of heat, and the size of a device can be related to both its ability to dissipate this heat and the capacity of its battery. As purely anecdotal information, the Testbed-14 participants saw two laptops and one tablet perish during this initiative!

These are things to consider for field operations where devices would be required to be used for long periods of time without the ability to charge the devices, especially if operations increasingly rely on such Augmented Reality applications. It was beyond the scope of this testbed initiative to perform an analysis of factors such as power usage, heat and battery duration of devices. The small team was kept busy for the entire duration of the testbed by the already significant workload of the primary objectives. These objectives included processing and serving relevant data sets, achieving interoperability between the different services and clients, as well as displaying augmentation properly registering with the real world. This could be investigated in future AR initiatives. However, this would require strictly determining many variables and evaluation criteria so as to produce useful results.

Ecere built a client targeting Android mobile devices, leveraging their cross-platform Ecere and GNOSIS SDKs and making use of OpenGL ES, the Android sensors, and the Android Camera 2 API directly. The client would automatically connect to both services and request 3D contents data in a tiled manner based on the current view position. It provided buttons to jump to fixed locations for the datasets (Washington, D.C. and New York City), as well as to the last known GPS position, or to constantly reposition the camera based on GPS location updates.

The client currently requires an Android 5.0+ device, as well as support for OpenGL ES 3.0. It will be possible to lower this requirement to OpenGL ES 2.0 after some fallbacks are properly implemented in the Ecere graphics engine. Application packages (APKs) were built for ARM CPUs, both 32-bit (armeabi v7a) and 64-bit (arm64-v8a), and tested on multiple devices.

There are four buttons in the application:

If Follow mode is not toggled on, the user can:

Whether Follow is on or not, the user can simply rotate the device around freely to update the camera view orientation (yaw/pitch/roll).

These experiments focused on rendering 3D buildings, as streamed by the services.