B.1.  Informal observation on the cost of copies

A GeoHEIF reader was developed for Testbed 20 and (at the time of writing this Report) is available in the “incubator” group of modules of the Apache SIS code repository. The initial prototype, committed October 2024, shared only partially the code that could be shared with the existing GeoTIFF reader. Initial benchmarks (not reported in this Report) showed that this GeoHEIF prototype was approximately two times as slow as GeoTIFF for reading the same image. This difference was assumed to be caused by the GeoHEIF prototype performing more copies and data reorganizations from one array to another. A major refactoring of the prototype, committed in February 26th, has brought the GeoHEIF reader to the same level as the GeoTIFF reader in terms of efforts for reducing array copies. Benchmarking on a local file then showed differences within the standard deviations. This observation explains the emphasis regarding array copy operations documented in this Annex.

B.1.2.  Cross-language transfers of arrays

The interfacing between Java and a native library has some performance costs. A part of the cost come from an additional copy of arrays. Copies may come from the mechanism used for crossing the language barrier (e.g., Java Native Interface) or may come from interactions between this mechanism and the API of the native library. When a client application and a library need to transfer large arrays such as large raster image, there are at least three possible strategies.

1. When invoking a library function, the client application provides a pointer telling the library where to read or write the data. This may be a pointer to the application’s internal buffers. This approach requires that the application trusts the library.

2. Conversely, the library may return a pointer to its internal buffer, in which case the client application reads or copies the data itself. This approach requires that the library trusts the application.

3. Alternatively, the library may copy data to a temporary buffer, in which case the application reads or copies data from that buffer. With this approach, neither the library nor the application needs to trust the other.

The figure below illustrates these three strategies. Each arrow represents an array copy operation. The direction is from the software component that performs the read or write operation, toward the software component which exposes its internal for allowing that operation.

 

Figure B.1 — Strategies for transferring data from a library to a client applications

The GDAL native library generally applies the first strategy in its C/C++ API. The second strategy is also available through the GetLockedBlockRef function, which provides direct access to the GDAL internal cache. However, the latter method is available only through the C++ API, while the binding framework used in OGC Testbed 20 can use only the C API.

Since the Java Virtual Machine (JVM) does not give pointers to Java arrays, the first strategy is not possible (except in some special cases called “critical”) with a binding though ‘java.lang.foreign’ . Said otherwise, the JVM generally does not allow a C/C++ library to write directly in a Java array, except in exceptional circumstances called “critical functions”. One reason is because the garbage collector may move the array to a new memory location at any time, and C/C+&#43 libraries are not prepared to handle such memory address changes in the middle of an operation. Therefore, the SIS-GDAL binding must allocate a temporary buffer, ask GDAL to copy some raster data to that buffer, then let Java copy the buffer’s content to the final Java array. This is the third strategy in list above. The extra copy could be avoided with the second strategy, but the latter is not accessible in the SIS-GDAL binding case.

In summary, extra copy operations (compared to pure Java or pure C/C+&#43 implementations) were necessary in Testbed 20 when using a native library because of the interaction between the native API and the framework used for accessing it. In Java context, extra copies could be avoided if the native library allowed the client application to perform copies itself (strategy 2) through a C API.

B.2.  Benchmarking method

The benchmarks described in this Annex were implemented by a Java application available as a Maven project on the Testbed code repository. The application read a text file which describes thousands of queries following a scenario. The scenarios implemented in the benchmarking code were:

Each scenario can be generated once and saved in a text file for reproducibility. A different text file must be generated for each image to test, because the queries depend on the geospatial extent of the image. Each image should be available in two formats, GeoTIFF and GeoHEIF. The tester should verify that the following characteristics are identical in the GeoTIFF and GeoHEIF images (this was not verified by the benchmarking application).

Due to lack of time, the benchmark described in this annex used a single image with the following characteristics.

The Testbed 20 participants did not verify that the two images had the tiles written in the same order. This verification should be added in the future if this benchmarking effort is continued. An overview of the image is below (location is in Australia):

Figure B.2 — Test image

The following snippet shows the header of a scenario file followed by the 4 first queries. Each scenario file contains 10,000 such queries. The scenario shown below is the one for queries of random sizes, positions and resolutions.

# OGC Testbed 20 — GIMI and COG benchmark
# https://gitlab.ogc.org/ogc/T20-GIMI/
#
# GridGeometry
#   ├─Grid extent
#   │   ├─Column: [0 … 9727] (9728 cells)
#   │   └─Row:    [0 … 9215] (9216 cells)
#   ├─Geographic extent
#   │   ├─Lower bound:  35°19′23″S  149°06′03″E
#   │   └─Upper bound:  35°18′54″S  149°06′44″E
#   ├─Envelope
#   │   ├─Easting:   690,999.950000002 … 691,972.750000002   ∆E = 0.1 m
#   │   └─Northing: 6,089,078.45000004 … 6,090,000.05000004  ∆N = 0.1 m
#   ├─Coordinate reference system
#   │   └─EPSG:28355 — GDA94 / MGA zone 55
#   └─Conversion (origin in a cell center)
#       └─┌                              ┐
#         │ 0.1   0     691000.000000002 │
#         │ 0    -0.1  6090000.000000040 │
#         │ 0     0          1           │
#         └                              ┘
#
# Scenario: RANDOM

{
    areaOfInterest = 691263.650000002 6089473.6500000395, 691678.3500000021 6089681.6500000395
    resolution     = 1.3 1.3
    gridExtent     = 2626 3172, 6760 5239
    subsampling    = 13 13
    subsampledSize = 319 160
}

{
    areaOfInterest = 691652.950000002 6089078.45000004, 691948.950000002 6089826.45000004
    resolution     = 1.6 1.6
    gridExtent     = 6528 1728, 9472 9200
    subsampling    = 16 16
    subsampledSize = 185 468
}

{
    areaOfInterest = 691155.550000002 6089121.55000004, 691891.750000002 6089575.15000004
    resolution     = 1.8 1.8
    gridExtent     = 1548 4248, 8892 8766
    subsampling    = 18 18
    subsampledSize = 409 252
}

{
    areaOfInterest = 691304.450000002 6089272.45000004, 691625.050000002 6089920.6500000395
    resolution     = 1.4 1.4
    gridExtent     = 3038 784, 6230 7252
    subsampling    = 14 14
    subsampledSize = 229 463
}

Listing B.1 — Snippet of scenario file for the random case

While all scenarios listed at the beginning of this section were implemented and were available in the benchmarking application, only the two first scenarios were executed in Testbed 20: Reading tiles of 512 × 512 pixels sequentially, then at random positions but always at full resolution. Each scenario of 10,000 queries was executed about 100 times. The machine running the benchmark application had the following characteristics:

The Apache SIS version used was a snapshot of the main development branch of version 1.5, built from commit 3950fbe8d5fc2dfac23a4b4a7bd0eb1d1ddea90b (April 11, 2025).

B.3.  Running the benchmark

The benchmark application expects a specific directory structure. As a general rule, each directory contains data in one specific geographic area at one specific resolution, at least approximately. Then, an arbitrary number of sub-directories encode the same data in different formats, using different compressions, different tile sizes, etc. While different sub-directories were planned for testing different combinations of image layouts and compressions, only the following directory structure has been used in Testbed 20, with the ‘ACT2017*’ images downloaded and renamed from here.

<data directory>
└── COG versus HEIF
    └── No pyramid
        ├── ACT2017.heif
        └── ACT2017.tiff

Listing B.2 — Directory structure of test data

For building the benchmarking application, clone the Testbed 20 repository and execute the following commands (replace cd by chdir on Windows):

cd D011/code
mvn package

Listing B.3 — Build benchmarking application (Unix)

For generating the scenario files, execute the following commands (replace <data directory> by the root of above directory tree):

./run.sh generate-scenarios \
    --image-directory <data directory> \
    --images COMPARE_COG,COMPARE_HEIF \
    --scenarios SERIAL

Listing B.4 — Generate scenario files (Unix)

For running the scenario files, execute the following commands:

./run.sh benchmark \
    --image-directory <data directory> \
    --images COMPARE_COG,COMPARE_HEIF \
    --scenarios SERIAL \
    --num-passes 110 \
    --ignored-passes 10

Listing B.5 — Run benchmarking application (Unix)

Replace SERIAL by RANDOM_WITH_FULL_RESOLUTION for testing the scenario reading tiles at random positions. Add --use-gdal true for testing with GDAL instead of Apache SIS (it requires a GDAL installation to be reachable on the library path of the host computer).

sudo systemctl start httpd.service

./run.sh benchmark \
    --scenario-directory <data directory> \
    --image-directory "http:​//localhost/" \
    --images COMPARE_COG,COMPARE_HEIF \
    --scenarios SERIAL \
    --num-passes 110 \
    --ignored-passes 10

B.4.  Benchmarking results

The following table shows the execution times with the GeoTIFF and GeoHEIF pure-Java readers. 10,000 reads of tiles of 512 × 512 pixels were done, then the whole process was repeated about 100 times. The statistics of the first 10 iterations were discarded as the Java Virtual Machine warmup (this was conservative, as one iteration is usually sufficient). The scenarios shown in the table are:

Table B.1 — Execution times (milliseconds) of queries of 512 × 512 pixels on the local file system

The above table shows large variations, with relatively large maximum values and with standard deviations that are much larger than the average values. Note that even with the sequential scenario reading tiles from left to right then top to bottom, there is no guarantee that the tiles in the files are organized in that order. It is possible that the reading of some tiles where slower because they required a seek to a distant position in the file. This hypothesis has not been verified. For avoiding this uncertainty, the following table shows the statistics of execution times of iterations over all queries instead of statistics over the reading time of individual query. Since the exact same scenario was repeated for each iteration, there were no variation between each iteration, contrary to the time measurements of individual queries. The times shown in the table are the average times per query for making the comparison with the above table easier.

Table B.2 — Execution times (milliseconds) of image read operations on the local file system

The above table shows much more stable results. It also shows that GeoTIFF and GeoHEIF performances are equivalent for sequential readings of tiles of an uncompressed image on the local file system. A difference larger than the standard deviation is observed in the case of random access, with GeoTIFF appearing faster than GeoHEIF. This difference was unexpected, as the two readers were intended to be of equivalent performance. The cause was not investigated in Testbed 20.

B.5.  Binding to native code

Read operations using the GDAL native library (C/C++) were also benchmarked. This benchmark used a “GDAL to Java” binding implemented with the Panama technology (the java.lang.foreign package). Panama, available since Java 22, replaced Java Native Interface (JNI) (Java 1.1). This benchmark did not use the OSGeo GDAL-Java binding, which is based on SWIG (a generator of JNI bindings). An advantage of Panama is that is provides greater control on copy operations between Java heap and native memory.

The benchmarking results are not reported for two reasons. First, the GDAL 3.9.3.0 version used for the benchmark did not include GeoHEIF support (but getting that support would have been possible by building GDAL locally), while the benchmark’s goal is to compare GeoHEIF against GeoTIFF rather than to compare libraries. Second, the time measurements were unexpectedly large, which suggest that this benchmark did not used GDAL correctly. The cause may be not enabling the GDAL cache, missing or incorrect calls to GDALRasterAdviseRead(…), or parameters given to GDALRasterIO(…) forcing more work than intended (e.g., queries too large or not aligned on tiles). The issue was not investigated in Testbed 20. Nevertheless, some observations are possible by running the benchmark application in the NetBeans profiler.

The time spent creating empty rasters (22%) is not normal and was not observed in the pure-Java implementations of the GeoTIFF and GeoHEIF readers. The cause was not investigated and is unlikely to be related to GDAL. Those empty rasters are created by calls to Java’s Raster.createWritableRaster(SampleModel, 'Point') method before to fill the raster’s content with data provided by GDAL using strategy 3 of Figure B.1. A hypothesis for the slowness may be related to the fact that this Java method tries to use the video card memory, which is an overhead in the context of the Java-C binding.

A similar benchmark of the pure-Java GeoTIFF reader showed 8% of time in the GeoTIFF reader and 90% of time in the benchmarking application. The source of the latter inefficiency was identified as the use of an unnecessarily generic algorithm for a matrix inversion in the Apache SIS library. This will be fixed in a future version. However, as those matrix inversions happened outside the read operations that were measured, it did not impact the time measurements reported in this annex.

C.1.  Executive Summary

Silvereye Technology contributed to Testbed 20 benchmarking in two forms:

The small tiles evaluation demonstrated that high efficiency codecs, when combined with an MPEG proposed low-overhead image file format, can provide significant reduction in file size over current formats such as PNG.

C.2.  Test data

As part of Silvereye Technology’s contribution to Testbed 20, GeoTIFF source data was converted to HEIF, including GeoHEIF metadata. The data was Red-Green-Blue true color aerial imagery of a residential area in Canberra, Australian Capital Territory. The source data has a CC-BY license (Jacobs Group (Australia) Pty Ltd and Australian Capital Territory government). Silvereye Technology acknowledges the ACT governments Open Data policy.

Silvereye Technology converted the data using custom tools and produced several files with different encoding options (e.g., tiled or not, pyramid overview or not). This allowed benchmarking to evaluate the effect of these options under different use cases, without needing to compensate for the effects of different data.

C.3.  Small tile size comparison

Silvereye Technology conducted an evaluation of the applicability of HEIF to small images, such as those used for raster tiles. This was motivated, in part, by the low-overhead image file format changes proposed in ISO/IEC 23008-12 CDAM 2.

An example of this is OpenStreetMap, where 256 pixel by 256 pixel tiles are rendered for visualization of the underlying data. The usual encoding of those tiles is as PNG images, which works well for the sharp edges expected in a raster map.

This kind of tiling is potentially the result of an OGC API for Tiles request. Since this API supports content negotiation, there is a clear path for adding HEIF-encoded files as an option where it could provide benefit.

Silvereye Technology selected two tiles from OpenStreetMap — one that is very simple (essentially a single color, from a forested area to the west of Canberra, Australian Capital Territory), and one that was more complex (multiple roads, building outline, symbols and text) from a commercial area in Canberra, Australian Capital Territory.

Figure C.1 — Simple OSM test tile

Figure C.2 — Complex OSM test tile

The tiles were compressed using the “avifenc” command line tool from libavif, with different compression quality settings (essentially trading off compression artifacts for file size). The same compression was then repeated using the experimental low-overhead option (--mini). In both cases the same AV1 codec was used, being the reference AOM codec.

Results are shown in the graphs below.

Figure C.3 — Simple tile compression size for AVIF and PNG

Figure C.4 — Complex tile compression size for AVIF and PNG

As a lossless compression system, there is no quality variation for PNG — it is provided in the above graphs only as a reference level.

In both cases, the low-overhead option reduces the file size by approximately 238 bytes from the full ISO/IEC 23008-12:2022 header. The relative importance of that reduction depends on the overall size of the file, being more obvious for the simple tile.

Given the simple tile has essentially only one color, it compresses very well with both PNG (which can use a single entry in the color palette), and with AV1. However the use of the low-overhead header is required to bring AVIF into a comparable file size with PNG on this simple tile content. As would be expected from the very small difference in compressed size, there is essentially no visual difference across the quality level settings.

The complex tile shows much more file size variation across the (arbitrary) quality scale, from substantially smaller than PNG to substantially larger than PNG. This is associated with much more noticeable visual differences, as shown in the samples below. Very low quality settings (0 or 10, not shown) produce distortion to the extent that the file is essentially unusable. Settings in the range 20 to 50 produce files that have some artifacts but are likely usable for many purposes. Settings above this level are visually identical to the PNG source image, at potentially half the file size.

Figure C.5 — Complex tile - quality=20

Figure C.6 — Complex tile - quality=30

Figure C.7 — Complex tile - quality=50

Figure C.8 — Complex tile - quality=60

Figure C.9 — Complex tile - quality=80

Figure C.10 — Complex tile - quality=100

By way of comparison, using an uncompressed RGB TIFF file for either test tile produces a file just under 200 kilobytes (256 x 256 x 3 = 196608 plus file overhead). Using a palette format reduces that to approximately 8 kilobytes for the simple tile, and 67 kilobytes for the complex tile.

Applying a lossless compression mechanism with TIFF helps significantly, although both PNG and AVIF are generally smaller in terms of file size:

While a full evaluation of HEIC (i.e., H.265) was not included, an informal check suggested that it would be less compact than AV1, although still better than PNG for the complex tile example and somewhat worse than PNG for the simple tile example.

In conclusion, the use of HEIF with a high efficiency codec can produce significant file size reduction over PNG tiles, and in combination with the proposed low-overhead image file format, it is likely that it will probably be of the same order as PNG even for very simple files.

Future work could:

D.1.  Benchmarking Data

Two types of COG data were used in the benchmarking procedure: real data and synthetic data. Real data are useful for Real Data Benchmarking as they provide an honest representation of data values, which affects the data compression formatting variables. Common occurrences such as NoData areas outside of swaths are captured in real data. Synthetic data are useful for Synthetic Data Benchmarking as they remove uncertainty that arises from unpredictable data content and its effect on compression.

D.1.1.  Real Data

The COGs used in this benchmark represent real datasets, such as satellite images or elevation models. In Helyx’s benchmarking tests an emphasis was placed on geospatial raster data over photographic raster data as this is the target of the COG format. Geospatial raster data requires a wider range of bit depths and bands as it has a broader range of data to represent (e.g., scientific imagers generating multi- and hyperspectral images, and derivatives like elevation models). Geospatial rasters may also be much larger than their photographic counterparts (in column/row space), as they may be mosaics of several images, or generated by large sensors (such as a pushbroom scanner on an Earth Observation satellite). Photographic rasters place an emphasis on storage and transmission efficiency given their ubiquity. Unlike geospatial rasters, photographic rasters are normally used for qualitative purposes, so are less limited in compression formats (lossy compression in acceptable).

Geospatial raster data are widely available from web accessible data repositories such as the Microsoft Planetary Computer, or can be generated using GDAL binaries (gdal_translate), or other conversion softwares. Examples of real data identified for benchmarking include:

1. Copernicus Sentinel-1 Ground Range Detected (GRD) Synthetic Aperture Radar (SAR) satellite imagery. These data contain two bands/channels representing the dual radar polarities used during acquisition (Vertical-Transmit, Vertical-Receive, or VV and Vertical-Transmit, Horizontal-Receive, VH). Sentinel-1 data provided by Copernicus are not provided in COG format by standard so have to be converted first.

2. NASA EO-1 Hyperion Hyperspectal satellite imagery. These data contain 242 bands/channels, so represent an edge-case for formats like COG.

3. USGS Landsat Near-Real-Time (NRT) multispectral imagery. These data are dynamically retrieved by querying a Spatiotemporal Asset Catalog (STAC) and randomly downloading a recent Landsat Level-2 image.

Real data can be re-saved to COGs using GDAL, which provides parameters to change the structure of the COG. These include tiling schemes, zoom levels, overview and compression options. Testbed 20 participants were consulted as to the range of options to use to create different variants of COG datasets for benchmarking.

 

Figure D.1 — Sentinel-1 image of Rotterdam and the Southern North Sea.

D.2.  Benchmarking Objectives

D.3.  Benchmarking Scenarios

The Testbed 20 GIMI benchmarking scenarios define the specific tests that were used to benchmark COGs and are outlined in Java code of the Scenario enumeration. The Helyx benchmarking activity aimed to follow these scenarios as closely as possible.

Each scenario modifies either a format variable, the image content, or an operational variable. Format variables relate to how files and their metadata are formatted. Image content relates to the data type and pattern of the raster data, as these effect compression. Operational variables relate to how the files are operated (interacted with). Scenarios are derived from a combination of possible operations and formatting options. Possible operations derived from basic read/write and cover COG-specific operations, such as subset reading. Possible formatting options are derived from the GDAL COG driver options.

D.4.  Benchmarking Approach

Helyx used Jupyter Notebooks for benchmark implementation.

Real benchmarking tests measured read/write times against pre-defined data files where there is no control over file content and creation options.

Synthetic benchmarking tests measured read/write times and file sizes against file content and creation options. Possible synthetic data creation options are as follow.

D.5.  Analysis and Observations

Read and write operations were performed in a single thread.

D.5.1.  Synthetic Data: Formats and Drivers

To minimise variables in assessing format and driver read/write times and file size, the following minimal creation options were specified:

• Dimensions: 3 x 512 x 512

• Bit depth: Unsigned 8-bit (Byte)

• Pattern: Random/Gaussian

A significant factor in read and write times was the compression algorithm, which differ significantly from HEIF to non-HEIF formats. To make the comparison as fair as possible the GDAL and PIL-based TIFF files were generated using JPEG compression. It is acknowledged that these are two very different technologies and the results should be interpreted as such.

D.5.1.1.  Result Graphics

 

Figure D.2 — Image write times by format and drivers.

 

Figure D.3 — Image read times by format and drivers.

 

Figure D.4 — File size in memory and on disk by image format and drivers.

D.5.1.2.  Findings

• PIL’s TIFF format was the fasted for both read and write times.

• PIL’s TIFF format was also the least efficient file size, suggesting compression is not being performed correctly, or at all. Creation options should meet the requirement for JPEG compression in a PIL TIF, see the PIL doc, but this may be user error on the author’s part.

• RasterIO’s COG write time is marginally slower than PILs (0.1s Vs 0.02s).

• RasterIO’s COG read time is significantly slower than PILs (0.046s Vs 0.005s).

• PIL’s HEIF format take considerably longer to write than the TIFF format. This is probably because of the additional computational cost of HEVC compression.

• PIL’s HEIF format is the most efficient from a file size perspective.

D.5.1.3.  Recommendations

• It is difficult to fairly compare the speed of HEIF to TIFF or COG because the compression algorithms are so different. A fairer assessment should also consider storage efficiency. It is likely that the computational cost (compute resource and time) of generating HEIFs balance against file size.

• In cases where speed matters most no compression, or computationally simple compression algorithms may be more suitable.

D.5.2.  Synthetic Data: Image Dimensions

To minimise variables in assessing image dimensions on read/write times, the following minimal creation options were specified:

• Formats: PIL_HEIF | PIL_TIF | RIO_COG

• Dimensions: 1|3 x 512|1024|4096 x 512|1024|2048

• Blocksize: 256 x 256

• Bit depth: Unsigned 8-bit (Byte)

• Pattern: Random/Gaussian

D.5.2.1.  Result Graphics

 

Figure D.5 — Image write times by format and dimensions.

 

Figure D.6 — Image read times by format and dimensions.

 

Figure D.7 — Image file size in memory (uncompressed) and on disk (compressed) across a range of image sizes.

D.5.2.2.  Findings

• Read/write times have an exponential correlation to image dimensions. Smaller images are written and read exponentially faster than larger images.

• File sizes in memory (uncompressed) compared to on disk (compressed using JPEG for TIF/COG, and HEVC for HEIF) have an unexpected relationship. Generally HEIF is the most efficient, however in some cases RasterIO’s TIFF with JPEG compression is better (3072 KBs in memory, 3×1024x1024 shape). This may be because quality is set to 100% and the image is random, meaning there is little any compression algorithm can improve.

D.5.2.3.  Recommendations

• Investigating multi-threaded read/write times for larger images would be useful to reflect real-world scenarios. This will probably have an impact on the time it takes to perform more complex compression/decompression.

• Investigating the effect of lossy compression would be beneficial as this is critical to qualitative image processing workflows (rendering basemaps etc). It is less relevant for scientific images where lossless compression is necessary.

D.5.3.  Synthetic Data: Bit Depths

To minimise variables in assessing bit depths on read/write times, the following minimal creation options were specified:

• Formats: PIL_HEIF | PIL_TIF | RIO_COG

• Dimensions: 3 x 1024 x 1024

• Blocksize: 256 x 256

• Bit depth: Unsigned 8-bit (Byte) | Unsigned 10-bit | Unsigned 12-bit | Unsigned 16-bit | Signed 16-bit | Unsigned 32-bit | Signed 32-bit | Float 32-bit | Float 64-bit | Complex 64-bit

• Pattern: Random/Gaussian

D.5.3.1.  Result Graphics

 

Figure D.8 — Image write times by format and bit depth.

 

Figure D.9 — Image read times by format and bit depth.

 

Figure D.10 — Image file size in memory (uncompressed) and on disk (compressed) across a range of bit depths.

D.5.3.2.  Findings

• Read and write times are proportional to bit depth for the GDAL-generated COGs. 8-Bit images are read and written the fastest with Complex and floating point images being the slowest.

• Read and write times are similar for the HEIF images, so not proportional to bit depth.

• 64-bit floating point COG images took slightly longer to write than 64-bit complex floating point COG images, which wasn’t expected.

• HEIF was only comparable at 8-bit/Byte for a 3-band image. In this case it was much slower to read and write than COGs, but produced much smaller files.

• Tests were constrained by Numpy’s support for bit depths. Numpy covers most integer and float bit depths used by HEIF and COG. However for complex numbers only supports 64-bit.

D.5.3.3.  Recommendations

• HEIF’s main advantages in file size are reduced when addressing quantitative image processing operations (high bit depths, lossless compression).

D.5.4.  Synthetic Data: Bit Depth and Compression

To minimise variables in assessing bit depth and compression on read/write times, the following minimal creation options were specified:

• Formats: PIL_HEIF | PIL_TIF | RIO_COG

• Dimensions: 3 x 1024 x 1024

• Blocksize: 256 x 256

• Bit depth: Unsigned 8-bit (Byte) | Unsigned 10-bit | Unsigned 12-bit | Unsigned 16-bit | Signed 16-bit | Unsigned 32-bit | Signed 32-bit | Float 32-bit | Float 64-bit | Complex 64-bit

• Compression: None | LZW | Deflate | LERC | JPEG | HEVC

• Pattern: Random/Gaussian

D.5.4.1.  Result Graphics

 

Figure D.11 — Image read and write times, and size by bit depth and compression. Marker size represents size on disk, colour represents bit depth and marker shape represents compression type.

D.5.4.2.  Findings

• HEIF read/write times and file sizes are consistent, irrespective of bit depth.

• COG read/write times and file sizes vary by bit depth and compression.

• COG file sizes do not correlate to bit depth but do correlate with compression.

• No compression is the fastest to write, but 8-bit JPEGs are faster to read.

D.5.4.3.  Recommendations

• Benchmarking did not consider lossless compression, where the HEVC algorithm is likely to excel. Running additional benchmarks on lossy compression is recommended.

D.5.5.  Real Data

The real data benchmarking workflow is as follows.

1. Query a STAC server using a dynamic query (e.g., a period of time from present). This will improve the likelihood different images are downloaded each time the benchmark is run. High cloud images are filtered out as these will have reduced image content. The Landsat-C2-L2 collection from the Microsoft Planetary Computer is used in this benchmark. Images with 10% or less cloud acquired within 2 days of the current time (when the test was run) are selected.

2. Select a random set of items returned by the query. This further reduces the likelihood of the same images being downloaded. In this case, 10 images were selected.

3. For each STAC item, iterate a set of assets and download the assets. Time how long it takes to download the asset, how long it takes to save the asset locally to disk, and how large the asset is on local disk. The red, green and blue Landsat bands were used as they are all of the same resolution and have similar spectral properties (so will contain similar information).

All available image footprints are shown in grey, footprints randomly selected for download are shown in red. image::../images/stac_item_footprints.png[align=center, width=400]

D.5.5.1.  Result Graphics

 

Figure D.12 — A scatterplot showing the relationship between read/retrieve time of STAC items over the open internet (into memory) and the time taken to write them locally from memory to disk. Marker colours define assets (bands), marker sizes define local file size on disk.

D.5.5.2.  Findings

• As expected, there is no clear relationship between read/write time or file size when retrieving STAC items over the open internet.

• There is a very weak pattern between asset type, read time and file size. Red assets take slightly longer to read and are slightly larger. Blue assets are slightly quicker to read and are slightly smaller. Green assets are in the middle speed/size wise. There are many factors that could affect this, from how images are selected from the STAC server (low cloud images tend to be over arid environments), to the way Landsat is configured, to how sunlight is reflected by the ground.

D.5.5.3.  Recommendations

• More thorough benchmarking will probably show that read times are subject to broad external factors, such as overall internet activity.

• Comparing a geo- and cloud-enabled HEIF format would be more meaningful than an isolated assessment of COGs, although should still be performed in a rigorous manner (e.g., over a longer time period) to isolate time-variant factors (like internet traffic).