OGC CDB Vector Data in GeoPackage Interoperability Experiment

Three complete CDB data stores were provided for use in this Interoperability Experiment. The provision of CDB data stores and the subsequent conversion of Shapefiles into GeoPackages was defined in the IE Activity Plan in Experiment 1 [2]. The participants identified a number of activities associated with executing Experiment 1. A key objective of this experiment was to capture and compare metrics as well as to identify and document any differences in the Shapefile to GeoPackage conversions that resulted from the use of different conversion tools.

This section provides a short summary of the CDB data stores and tools used by each of the participants in this OGC Interoperability Experiment. Greater detail for each participants work can be found in Section 6 (Results) of this report.

Given the size of the CDB data stores used in this IE, there was discussion related to how best to provide data stores updated with GeoPackage content. The general consensus was that downloading the entire data store just to get the GeoPackages was non-optimal and time wasteful. Therefore, the participants discussed using Version metadata (Volume 1 CDB Standard, Clause 5.1.8). They determined that the original CDB data with Shapefiles would be Version 1 and that CDB enhanced with GeoPackages would be Version 2.

This experiment researched the direct one-to-one conversion of Shapefiles in a CDB data store into a corresponding set of Geopackages. GDAL and various commercial tools were used to do the transfer of Shapefile content to a GeoPackage. Characteristics of the Option 1a approach are:

This experiment researched the approach of using normalized SQL in the conversion of Shapefiles into GeoPackages. This approach has the following characteristics:

This experiment researched the approach of using flattened attribution in the conversion of Shapefiles into GeoPackages. This approach has the following characteristics:

This experiment researched the approach of using flattened attribution plus the GeoPackage related Tables extension in the conversion of Shapefiles into GeoPackages. The reason for using the extension was to enhance the ease of moving data in both directions (Shapefile to GeoPackage and visa-versa) using existing tools and without any data loss. This approach has the following characteristics:

The following is a description of a key use case in the Aechelon content processing workflow to use in an image generator.

The publishing software is in Python, with invocations of C++ EXEs for performance-critical processing. The feature scan step is entirely in Python, version 3.5. Please note that for the image generation workflow, only metadata fields that affect the appearance of features are considered and remainder of the CDB content is ignored, such as tactical data, or the entire geopolitical dataset. The hydrography network dataset is also ignored since the RMTexture dataset is used to identify areas of water.

For the purposes of this experiment, five feature types were considered and processed: Cultural, Lights, Powerlines, Railroads, and Trees. All of the performance information provided in the tables below is related to these five feature types.

The following is a concise description of the methodology used for execution of the Aechelon committed tasks in this Interoperability Experiment.

The following three tables provide basic performance metrics for the three CDB data stores processed in Experiments 2, 3, and 4. Providing performance metrics is one of the tasks identified in Experiment 1 of the GeoPackage in CDB IE.

In the tables below, "Baseline" refers to metrics based on the source Shapefiles. Option 1A, Option 1C, and Option 1D refer to the sub-options for Experiment 2 (one to one transformation of Shapefiles into GeoPackages).

CAE focused on Experiment #2 as described in Section 5 - Details of the Experiments Performed. Specifically, CAE focused on the part of the experiment where each Shapefile is replaced by one GeoPackage file.

CAE performed a series of tests to measure the impact of Shapefile-to-GeoPackage conversion on file size, file count, network transfer, and decode performance. These tests are detailed in the following sections.

CAE conclusion with some observations and recommendations for futher development of the standard can be found in Section 7 - Recommendations.

The first step of the CAE experiment was to compare other file formats that would be appropriate to store CDB Vector Data. These formats include GeoJSON and GML in addition to Shapefiles and GeoPackages. The table below lists the observed file sizes.

Notes and observations:

Notably, CAE also observed that every new table added to a SQLite database increases the size by a minimum of 4 KB, which is presumably an internal allocation unit intended to support real-time addition of data rows.

CAE also did a simple read-performance test on each of these files. For this test, CAE measured the time taken by GDAL to open the file, iterate through all features, then close the file. All measurements represent the mean of 8 runs.

Notes and observations:

In preparation for more comprehensive tests, CAE then made a few minor modifications to the GDAL library to ensure that the Shapefile-to-GeoPackage conversion was sufficiently lossless for our purposes.

CAE made the following modifications:

CAE did not use the sample CDB provided by the participants because their goal was to compare the performance of CAE internal applications when running with a CDB produced by CAE before and after the replacement of Shapefiles with GeoPackage files.

However, CAE believes that their findings may apply equally well to other databases.

CAE built a customized script that invoked the ogr2ogr executable to convert all vector files in two CDBs. Class DBF files and junction DBF files were converted to standalone GeoPackage files.

CAE found that the conversion to GeoPackages can substantially increase the amount of disk space required for vector data.

Notes and observations:

The CAE application of CDB involves networked client/server systems, so a key performance factor is the time required to request and transfer files over a network. For this test, CAE measured the time taken for a client to request and transfer a selection of vector data files from a networked server. Files were loaded "cold": the CDB data volume was freshly mounted immediately before each test to ensure that the OS file cache was clear. This test loaded files from CDB 2.

Notes and observations:

The final test was to benchmark the loading time for a certain geographical region in a real time system. CAE measured the decode time, number of files and data transfer volume. The real time system is the client device consuming OGC CDB data over the network. This test loaded files from both CDB 1 and CDB 2.

[*] Our observed GeoPackage decode time is split into GDAL-related processing and internal format conversion (which is not optimized). Although our total GeoPackage decode time was measured at 9.09 seconds slower than Shapefile decode time, the GDAL-only portion of the decode time was 1.72 seconds faster. Writing a well-optimized GeoPackage decoder that decoded directly from SQLite into CAE’s internal representation should expect a small performance win.

Notes and observations:

FlightSafety has developed both a CDB generation tool and a CDB Publisher client. The performance requirements of the CDB Publisher are much greater than CDB generation, so this report will focus on loading and consuming a CDB dataset. The CDB Publisher uses a CDB data store as the source data for building the synthetic environment for FlightSafety’s VITAL 1100 image generator system. These systems are used for pilot training on a variety of flight simulator systems. The Publisher does not do any preprocessing of the CDB dataset; all CDB data that it consumes is discovered and loaded during the publishing. This approach was chosen due to the world-wide scope of CDB and unknown quantity of content. The CDB well defined storage structure makes it easy to find the file(s) containing the data needed for the synthetic environment creation. Based on the flight training system requirements, an appropriate level of detail of vector and model data is discovered and loaded. The Publisher adapts to the available levels of detail of vector data, and the flight characteristics of the training device. The publishing system is primarily in C, and the testing was all performed with C libraries and code. The Shapefile API that is tested is a custom FlightSafety library, optimized for faster performance.

The experiments were focused on just-in-time uses of CDB, similar to how a FlightSafety visual system would use the data. Statistics were collected on the original CDB dataset, and the converted GeoPackage CDB datasets. These were used to infer the cost of database configuration management and transmission/deployment to a training device. Testing was done on both currently encoded CDB Shapefiles and on converted GeoPackage encoded files (Experiment 2, options 1, 3, and 4). Tests focused on the latency of loading files, processing data, and closing files. Tests were done on different conversion options and settings to come up with optimal recommendations

The following describes the methodology used to evaluate, convert and test the CDB datasets using GeoPackage vector encoding.

Experiment 1 - While the initial direction of the Interoperability Experiment involved utilizing the provided open source scripts to facilitate the conversion, there was an interest expressed from a Data Creator role in performing this operation with other software. CDB Studio features the capability to both ingest and generate CDB data stores so this was a natural fit for this application and Hexagon US Federal’s participation. The conversion process was developed to align closely with the provided workflow but was slightly altered in specific ways in line with how the application ingests data. Although an exhaustive analysis of the differences between the custom conversion logic and the provided scripts was not performed, where possible any differences are noted in this report. For Experiment 1, the Shapefile data was converted by adding the feature geometry and instance-level attributes into a GeoPackage using built-in capabilities in the LuciadLightpseed API. Class attributes were stored in a separate table using the pattern in Option 1b where the CNAM attribute for each feature is a foreign key for the class attributes table. The extended attributes were stored in a separate RTE table as in Option 1d and linked by a mapping table in accordance with the RTE spec.

Experiment 2 – For this experiment the vector feature and attribute information for each tile was converted into a single GeoPackage dataset. The CDB directory structure was maintained with folders under each vector component for the levels of detail and U designation and individual GeoPackages for each R offset.

Experiment 3 – In this experiment all datasets for a specific vector component were combined into a single GeoPackage. The resulting dataset differed slightly from the suggested approach in that the individual tile datasets were not combined into a feature layer in the GeoPackage and instead were kept in separate layers. This facilitated the current architecture of CDB Studio which was built to ingest the data in a tiled manner. It is entirely possible to adjust this pattern to utilize combined features and leverage spatial queries against this larger feature table which might produce an ingest performance increase, but this modification was beyond the scope of the Interoperability Experiment. Class attributes were consolidated for the component into a single table.

Experiment 4 – For the final experiment all the component datasets were further combined to create a single GeoPackage per geocell. As in Experiment 3, the class attributes were combined into a single table per component.

The visualization methodology was consistent between all experiments and relied heavily on the existing logic of the CDB Studio application with minor modifications. Standard visualization metrics were recorded and did not differ greatly between the original and the experimental datasets which was expected as the LuciadLightspeed API was designed in the Model-View-Controller (MVC) paradigm and keeps the source data and display components independent. Data ingestion was shown to be the more pertinent metric affecting the overall performance of the visualization software. CDB data stores are ingested using a lazy loading strategy to alleviate memory concerns. The current implementation did not utilize the extended attributes so although these were added to the converted GeoPackages they had no impact on the data ingest for visualization. CDB Studio was also designed to match the CDB data stores’ tiled architecture and data loads are done as tiles are needed to populate the current display area and scale. Further modifications could be done to improve efficiency with the vector data stored in the combined GeoPackages as in experiments 3 and 4 and produce faster data ingestion but this was out of scope for this Interoperability Experiment.

Visualization metrics were gathered on the initial load of the CDB datasets which involves an animated pan/zoom to the dataset area and an initial display of the data in a view encompassing the entire dataset bounds and at a coarse detail level.

Time to Convert – This is the total time to process the conversion of the dataset from the original version containing Shapefile data into GeoPackages. Deletion of the original Shapefile data was done as a secondary manual step and was not included in this metric.

Size on Disk – The size on disk was obtained by viewing the Windows Explorer properties window on the Tiles folder of the generated data. It is not inclusive of unmodified data components such as imagery and elevation. The size reported for the original Shapefile dataset was a measure of the replaced files.

File Count – Similar to the size on disk, this metric was gathered by viewing the Windows Explorer properties view on the Tiles folder of the generated data and does not include data components that were unaffected by the GeoPackage conversion process.

Data Ingest – Data ingest is the time taken to load all vector data needed the initial display of the CDB data. The visualization logic loads data on demand for the given tiles so this is a subset of the full dataset and could vary greatly depending on the areas and levels of detail being viewed. This metric also only reflects the modified vector datasets.

Memory Usage – Similar to the data ingest metric, the memory usage is a measure of the heap memory footprint of the ingested data repeated across the different sample datasets. The number reported was the delta of general system heap memory utilized before loading the CDB dataset and after. Note that this is not the final memory footprint of the datasets but instead can include temporary data structures used during the ingest process that could later be garbage collected. As such this metric is meant to show the typical memory overhead involved with loading and visualizing a CDB dataset and not the continuing persistent state.

Frames Per Second (FPS) – The frames per second values were gathered by utilizing a diagnostic overlay in the CDB Studio application during the data ingest. This value fluctuates during the tests and a visual inspection of the overlay data was used to determine the typical range of these fluctuations.

Our view is that a form of Experiment 2 is backwards compatible with the existing specification, for the following reasons:

Although previous experiments with one-to-one Shapefile-to-GeoPackage conversion have been very positive, our experiences indicate mixed results. On one hand, we observe substantial benefit in reducing the number of files stored and loaded, and we also observe the possibility of a comparable (or faster) decode time. However, on the other hand, we see potentially-problematic increases in the amount of storage space required.

The negative effects are amplified for CDBs with large numbers of Shapefiles that have small numbers of features. The GeoPackage format (especially as outputted by GDAL) is very space-inefficient when it comes to encoding tiles with small quantities of data. Notably, even if we moved to a 1-tile-per-table instead of a 1-tile-per-file approach, the underlying SQLite format seems to allocate a minimum of 4 KB per table regardless of how little data is in it.

Please see CAE recommendations

Standard Visualization Metrics - Overall visualization performance was similar between all experiments for each dataset. Converting to GeoPackage has no significant impact either positive or negative on visualization once the data is loaded.

Data Ingest - Data ingest for visualization was highly dependent on the size and structure of the underlying GeoPackage datasets and the architecture of the data ingest code. Both load times and heap memory usage were much greater for the combined GeoPackages on large datasets. While major adjustments to the GeoPackage interactivity was beyond the scope of this IE, minor modifications to reduce file I/O on the combined datasets showed potential for improvement in data loading time.

Data Conversion - For data conversion the best results were also obtained from Experiments 3 & 4. The caveat here is that some memory issues were encountered with the provided datasets as more layers and data were added to a single GeoPackage which introduces questions of scalability for these approaches.

To achieve the improvements in storage while also maintaining the speeds comparable with sub-option 1C and addressing the file-per-LOD issue, Aechelon recommends two additional experiments: (a) where each component selector of each LOD is in its own GeoPackage file. This is effectively a variant of sub-option 1C where the U and R references of the same component selectors are combined into one file; and (b) where each dataset’s LODs are in a separate geopackage file. This is effectively a variant of Experiment 3 where instead of storing each LOD in a separate layer in the same GeoPackage file, each LOD is a separate file.

If Aechelon were to recommend only one processing alternative, among those in this experiment, for inclusion as an alternate primary vector format in a future OCG CDB revision, it would be option 1C in Experiment 2.

Further investigation into the optimal structure of combining multiple sub-datasets (rail, hydrography, etc) and multiple levels of detail into a single GeoPackage could alleviate several encountered performance drawbacks with the CDB datasets in Experiments 3 & 4. At this point without this investigation and guidance Hexagon US Federal recommends the CDB architecture presented in Experiment 2 which provides both a reduction in the dataset’s file count and minimizes the impact to existing CDB applications accessing data in the current tiled directory structure.