B.1.1.  Contributions

The OGC Testbed 20 activity provided an excellent opportunity to work on libheif enhancements and bring it a big step forward in supporting the GIMI profile and also improving its overall architecture, which will be beneficial to the general user base.

Dirk Farin Algorithmic Research wants to express thanks to Silvereye Technology for the valuable contributions, especially for the ISO/IEC 23001-17:2024 decoder implementation, but also for improvements in various other parts and for the integration work to other software libraries, like GDAL (see Annex E).

B.2.1.  Types of Image Tiling

When reading or writing high-resolution images keeping the whole image in memory may not be feasible. For those situations, libheif supports reading and writing images as individual tiles. When reading, only the accessed tiles are read from the file and decoded. Similarly, when writing, each tile can be encoded separately and added to the image file. Writing the image tiles in arbitrary order is also possible.

libheif supports three methods to store tiled images.

Reading any of these tiling formats with libheif uses the same API. Therefore users do not have to care about the format used internally in the HEIF file.

Note: the functions for writing tiled ‘unci’ and ’tili’ images are still in the ‘heif_experimental.h’ header until the API is considered stable. If libheif is installed through the distribution’s package manager, this experimental file might not be installed. Please compile libheif from source and enable ‘WITH_EXPERIMENTAL_FEATURES’ in the cmake config.

B.2.2.  API for Writing Tiled Images

Irrespective of the tiling format used, the tiled image will always be generated first and then individual tiles added to it. The API for generating the tiled image depends on the tiling format used. Adding the tiles to the image is always done with the same API call.

struct heif_error
heif_context_add_grid_image(struct heif_context* ctx,
                            uint32_t image_width,
                            uint32_t image_height,
                            uint32_t tile_columns,
                            uint32_t tile_rows,
                            const struct heif_encoding_options* encoding_options,
                            struct heif_image_handle** out_grid_image_handle);

struct heif_error
heif_context_add_image_tile(struct heif_context* ctx,
                            struct heif_image_handle* tiled_image,
                            uint32_t tile_x, uint32_t tile_y,
                            const struct heif_image* image,
                            struct heif_encoder* encoder);

struct heif_unci_image_parameters {
  int version;

  // --- version 1

  uint32_t image_width;
  uint32_t image_height;

  uint32_t tile_width;
  uint32_t tile_height;

  enum heif_metadata_compression compression;

  // ...
};

struct heif_error
heif_context_add_unci_image(struct heif_context* ctx,
                            const struct heif_unci_image_parameters* parameters,
                            const struct heif_encoding_options* encoding_options,
                            const struct heif_image* prototype,
                            struct heif_image_handle** out_unci_image_handle);

struct heif_error
heif_context_add_image_tile(struct heif_context* ctx,
                            struct heif_image_handle* tild_image,
                            uint32_t tile_x, uint32_t tile_y,
                            const struct heif_image* image,
                            struct heif_encoder* encoder);

struct heif_tiled_image_parameters {
  int version;

  // --- version 1

  uint32_t image_width;
  uint32_t image_height;

  uint32_t tile_width;
  uint32_t tile_height;

  uint32_t compression_type_fourcc;

  uint8_t offset_field_length;   // one of: 32, 40, 48, 64 (bits)
  uint8_t size_field_length;     // one of:  0, 24, 32, 64 (bits)

  uint8_t number_of_extra_dimensions;  // 0 for normal images, 1 for volumetric (3D), ...
  uint32_t extra_dimensions[8];        // size of extra dimensions (first 8 dimensions)

  uint8_t tiles_are_sequential;  // (bool) hint whether all tiles are added in sequential order
};

struct heif_error
heif_context_add_tiled_image(struct heif_context* ctx,
                             const struct heif_tiled_image_parameters* parameters,
                             const struct heif_encoding_options* options,
                             struct heif_image_handle** out_tiled_image_handle);

struct heif_error
heif_context_add_image_tile(struct heif_context* ctx,
                            struct heif_image_handle* tile_image,
                            uint32_t tile_x, uint32_t tile_y,
                            const struct heif_image* image,
                            struct heif_encoder* encoder);

B.2.3.  API for Reading Tiled Images

Reading images works the same for all tiling schemes and even for non-tiled images. Non-tiled images will appear as images consisting of a single tile.

Before decoding a tile, the first step should be to get the tiling information with:

struct heif_error
heif_image_handle_get_image_tiling(const struct heif_image_handle* handle,
                                   int process_image_transformations,
                                   struct heif_image_tiling* out_tiling);

Listing B.7 — API for requesting image tiling information

The Boolean parameter process_image_transformations indicates whether libheif should take care of all image transformations (rotations, mirroring, cropping) internally, or whether want to handle them. If ‘process_image_transformations’ is enabled, libheif will also convert the tile coordinates such that it looks to the client application as if the image geometry is not transformed.

The above function returns the following tiling information:

struct heif_image_tiling
{
  int version;

  // --- version 1

  uint32_t num_columns;
  uint32_t num_rows;
  uint32_t tile_width;
  uint32_t tile_height;

  uint32_t image_width;
  uint32_t image_height;

  // Position of the top left tile.
  // Usually, this is (0;0), but if a tiled image is rotated or cropped, it may be that the top left tile should be placed at a negative position.
  // The offsets define this negative shift.
  uint32_t top_offset;
  uint32_t left_offset;

  uint8_t number_of_extra_dimensions;  // 0 for normal images, 1 for volumetric (3D), ...
  uint32_t extra_dimension_size[8];    // size of extra dimensions (first 8 dimensions)
};

Listing B.8 — Image tiling information structure

In general, assume that the image_width and image_height are no integer multiples of the tile size. This constraint will be the case for ‘unci’ images, but not for the other tiling types. When a tile extends beyond the image border, the client application should not draw the part of the tile that is outside of the image border. Libheif will not crop the border tiles.

Internally, tiles may extend beyond the right and bottom borders, but when process_image_transformations, the image may be rotated and cropped and tiles may extend beyond all four borders. For this reason, the fields top_offset and left_offset indicate how much of the top and left border should be removed when displaying the image.

Figure B.1 — Tiled image with ignored right and bottom border.

Figure B.2 — Image has been rotated by 90 degrees. Now the left_offset is greater than 0.

Now individual tiles can be decoded with:

struct heif_error
heif_image_handle_decode_image_tile(const struct heif_image_handle* in_handle,
                                    struct heif_image** out_img,
                                    enum heif_colorspace colorspace,
                                    enum heif_chroma chroma,
                                    const struct heif_decoding_options* options,
                                    uint32_t tile_x, uint32_t tile_y);

Listing B.9 — API for decoding a single image tile

As with encoding, tile_x and tile_y specify the tile position index, not the pixel coordinate. The other parameters are the same as for the usual heif_image_handle_decode_image() function. Make sure that the heif_decoding_options value ignore_transformations is set to the opposite as process_image_transformations.

B.3.1.  Decoding the Multiresolution Pyramid Information

The pymd entity group can be retrieved with

struct heif_entity_group
{
  heif_entity_group_id entity_group_id;
  uint32_t entity_group_type;  // this is a FourCC constant
  heif_item_id* entities;
  uint32_t num_entities;
};

heif_entity_group*
heif_context_get_entity_groups(const struct heif_context*,
                               uint32_t type_filter,
			       heif_item_id item_filter,
			       int* out_num_groups);

Listing B.12 — API for getting the entity group information

More specifically, the type_filter to heif_fourcc('p','y','m','d') can be set to directly get the entity group of the pyramid if it is present. The ‘item_filter’ can be set to the primary image ID to get the main image pyramid. However, there will often be only one pyramid in the file, but consider the case that there are multiple pyramids in one file. The image item IDs (’entities’) in the returned entity group are ordered from lowest resolution to highest resolution.

B.3.2.  Encoding a Multiresolution Pyramid

First, encode all the resolution layers as separate images. Each layer image in the pyramid can be a tiled image and image types can also be mixed. For example, the highest resolution layer could be an unci image that stores the lossless compressed data, while the overview images use tili or grid with an efficient image codec. It also helps that software that cannot read unci images can at least show the overview images.

Once all layer images are created and their item IDs are known, call:

struct heif_error
heif_context_add_pyramid_entity_group(struct heif_context* ctx,
                                      const heif_item_id* layer_item_ids,
                                      size_t num_layers,
                                      heif_item_id* out_group_id);

Listing B.13 — API for adding an image pyramid

This will group the images into a pymd group. The image IDs can be listed in any order. libheif sorts the layer images according to their size and automatically generates the meta-information stored in the pymd entity group.

Note that tiles can still be added after the pymd entity group has been generated. For example, first create “empty” images with heif_context_add_tiled_image() for each layer, generate the pymd entity group, and then start encoding the image tiles.

B.4.1.  Implementing a File Reader Interface

Out of the box, libheif supports reading HEIF files from either a file stored in the filesystem or from memory. However, libheif also provides a generic reader interface that can be implemented by the client application to connect libheif to other data sources. This ‘heif_reader’ interface can be used to implement a direct download of the HEIF file from a network.

Apart from the usual ‘read’ and ‘seek’ functions, which were previously available, the interface was extended in Testbed 20 to a version 2 that also includes the function heif_reader_range_request_result request_range(uint64_t start_pos, uint64_t end_pos, void* userdata). libheif will call this function to tell the reader which file range it is going to read next. This should be a blocking call in which you can download the data from the network. The advantage of downloading the file in the read() function is that the read() function may read many small chunks, while request_range() will request larger file ranges that are more efficient to download. More data than requested may be downloaded and then let libheif know in the result that this data is available. libheif may decide to use that extra data if useful.

Another optional function is ‘void preload_range_hint(uint64_t start_pos, uint64_t end_pos, void* userdata)’. With this function libheif indicates that it may need this file range in the future. Contrary to the above, this function should be non-blocking and return immediately to the calling function. So that the content is ready in case the range is requested later a download in the background may be useful.

If you are caching network data, consider the callback void ‘release_file_range(uint64_t start_pos, uint64_t end_pos, void* userdata)’ which is used to let the application know if libheif does not need a specific file range anymore and the content can be removed from the cache. Note any data can be removed from the cache whenever desired. However, remember that the content may need to be reloaded if libheif requires that content again.

B.5.1.  Encoding Tiled Images

When encoding a high-resolution tiled image, heif-enc expects each tile image in a separate input image with filenames that contain two numbers to denote the tile row and column position. For example: tile-2-3.jpg or image-002-003.jpg.

To switch heif-enc into tile input mode, use the option -T / --tiled-input and specify one tile image as input. heif-enc will scan the directory to search for images with the same name pattern. heif-enc will detect the number range for the row and column position. This means that the position numbers may start from 0 or 1 (or any other number). heif-enc will also detect whether to use leading zeros for the numbers.

Usually, heif-enc expects that the first number is the vertical position and the second number the horizontal position. If swapping is required, use --tiled-input-x-y.

The generated tiled image will have a size which is the sum of all the tile widths and heights. However, the input image might have some padding at the right and bottom border. In that case, overwriting the total image size with --tiled-image-width # and `—tiled-image-height #`is possible.

B.5.2.  Multi-Resolution Pyramids

heif-enc can also encode a multi-resolution pyramid stack of images. In this case, just specify the input images for each resolution layer on the command line and provide the option --add-pyramid-group.

For a hierarchical pyramid of tiled images, provide the option ‘—tiled-input’ and specify the filename of one tile for each pyramid layer. An example command line would be:

heif-enc --tiled-input layer1-001-001.jpg layer2-01-01.jpg layer3-01-01.jpg -o output.heic --add-pyramid-group

Listing B.15

The input layer images can be specified in any order; heif-enc will sort them by size and add them in the correct order to the pyramid group.

B.8.1.  Introduction

HEIF files can contain image sequences in addition to still images. The HEIF specification builds upon the ISO14496-12 ISOBMFF standard with a few extensions from ISO23008-12 (HEIF) and GIMI specific definitions from NGA.STND.0076_1.0.

During the Testbed 20 GIMI Task Extension, support for HEIF image sequences was added to libheif. In particular, this includes intra-coded sequences in the JPEG2000, HTJ2K, and ISO23001-17 uncompressed format. However, the implementation also covers H.264, H.265, H.266, AV1, and JPEG (only decoding for H.264). Motion-predicted video sequences are not yet supported.

Sequences may consist of several media tracks. A media track is either a visual track or a metadata track. Other track types (e.g., audio) are currently not supported.

A track contains timed samples. Metadata samples can contain any arbitrary data and the track header specifies what kind of data is contained through an URI-based naming scheme. Samples from different tracks do not necessarily come in pairs. Instead, they can be timed independently. For example, while the visual track may carry a 30 fps video stream, the metadata track may contain acceleration sensor data, sampled at a higher rate.

Additionally, each sample (image or metadata) can contain sample auxiliary information (SAI). This is a usually small data package with auxiliary information that is directly attached to the sample. Libheif currently supports TAI timestamps and textual content IDs as defined in the GIMI standard.

Finally, metadata can also be attached to a track as a whole. Currently, libheif support the TAI timestamp configuration block and GIMI track content IDs.

Relations between tracks are defined by track references. For example, a ‘cdsc’ (Content Description) reference from the metadata track to the visual track denotes that this metadata track is further describing the referenced visual track.

The API specific to sequence processing can be found in the libheif/heif_sequences.h header file. This document only describes the basic steps to decode and encode HEIF sequences. For more specific functionality, the reader is referred to that header file.

B.8.2.  Reading HEIF Sequences

After opening the HEIF file as usual and obtaining a heif_context object, the application can check whether there is an image sequence in the file with:

int heif_context_has_sequence(heif_context*)

Listing B.16 — API for checking whether HEIF file contains a sequence

Note that a file can contain still images and a sequence at the same time.

To get an overview what tracks are contained in the HEIF file, the application can use the functions:

int heif_context_number_of_sequence_tracks(const struct heif_context*);
void heif_context_get_track_ids(const struct heif_context* ctx, uint32_t out_track_id_array[]);

Listing B.17 — API for getting the sequence track IDs of an HEIF file

To then get access to a specific track with

struct heif_track* heif_context_get_track(const struct heif_context*, uint32_t id);

Listing B.18 — API for accessing sequence tracks of an HEIF file

Alternatively, for convenience, you can call heif_context_get_track(ctx, 0) with a zero ID to get the first (and usually only) visual track in the file.

To check what kind of data the track contains:

heif_track_type heif_track_get_track_handler_type(struct heif_track*);

Listing B.19 — API for getting a sequence track type

This can be an image sequence, a video, or metadata. Other track types (e.g., audio) are currently not supported by libheif, but the raw data can still be read to some extent with the same API as for metadata tracks.

struct heif_error heif_track_get_image_resolution(heif_track*, uint16_t* out_width, uint16_t* out_height);

struct heif_error heif_track_decode_next_image(struct heif_track* track,
                                               struct heif_image** out_img,
                                               enum heif_colorspace colorspace,
                                               enum heif_chroma chroma,
                                               const struct heif_decoding_options* options);

uint32_t heif_image_get_duration(const heif_image*);

struct heif_error heif_track_get_next_raw_sequence_sample(struct heif_track*,
                                                          heif_raw_sequence_sample** out_sample);

const uint8_t* heif_raw_sequence_sample_get_data(const heif_raw_sequence_sample*, size_t* out_array_size);

uint32_t heif_raw_sequence_sample_get_duration(const heif_raw_sequence_sample*);

B.8.3.  Writing Tracks

struct heif_error heif_context_add_visual_sequence_track(heif_context*,
                                                         uint16_t width, uint16_t height,
                                                         struct heif_track_info* info,
                                                         heif_track_type track_type,
                                                         heif_track** out_track);

struct heif_track_info* heif_track_info_alloc();

struct heif_error heif_track_encode_sequence_image(struct heif_track*,
                                                   const struct heif_image* image,
                                                   struct heif_encoder* encoder,
                                                   const struct heif_encoding_options* options);

void heif_image_set_duration(heif_image*, uint32_t duration);

struct heif_error heif_context_add_uri_metadata_sequence_track(heif_context*,
                                                               struct heif_track_info* info,
                                                               const char* uri,
                                                               heif_track** out_track);

heif_raw_sequence_sample* heif_raw_sequence_sample_alloc();
heif_error heif_raw_sequence_sample_set_data(heif_raw_sequence_sample*, const uint8_t* data, size_t size);
void heif_raw_sequence_sample_set_duration(heif_raw_sequence_sample*, uint32_t duration);

struct heif_error heif_track_add_raw_sequence_sample(struct heif_track*,
                                                     const heif_raw_sequence_sample*);

B.8.4.  Sample Auxiliary Information (SAI)

As mentioned in the introduction, we can attach Sample Auxiliary Information (SAI) to samples in the track. The type of the SAI is specified with a four-character code. Libheif currently supports TAI timestamps and GIMI content IDs as SAIs. The SAI data is assigned to the sample objects. For example, to the heif_image in the case of visual tracks, or heif_raw_sequence_sample in the case of metadata tracks. The SAI data is assigned to the sample objects, i.e., to the heif_image in the case of visual tracks, or heif_raw_sequence_sample in the case of metadata tracks.

When reading a file, content_IDs these can be retrieved from the samples with

const char* heif_image_get_gimi_sample_content_id(const heif_image*);
const char* heif_raw_sequence_sample_get_gimi_sample_content_id(const heif_raw_sequence_sample*);

Listing B.33 — API for getting the GIMI content ID from an image or metadata sample

TAI timestamps can be retrieved using:

const struct heif_tai_timestamp_packet* heif_raw_sequence_sample_get_tai_timestamp(const struct heif_raw_sequence_sample*);
struct heif_error heif_image_get_tai_timestamp(const struct heif_image* img,
                                               struct heif_tai_timestamp_packet* timestamp);

Listing B.34 — API for getting the TAI timestamp from an image or metadata sample

There exist similar functions to attach the SAIs to the samples before encoding the samples into the track.

There are also functions to get the list of SAIs that are available in a read HEIF file:

int heif_track_get_number_of_sample_aux_infos(struct heif_track*);
void heif_track_get_sample_aux_info_types(struct heif_track*, struct heif_sample_aux_info_type out_types[]);

Listing B.35 — API for getting the TAI timestamp from an image or metadata sample

B.8.5.  Track-level Metadata

A track that contains TAI timestamp SAIs also needs a TAIC configuration box. This is assigned to the track as a whole. More correctly, the box is assigned to a cluster of samples, but since there is usually only one cluster of metadata samples, the TAIC is assigned to the whole track. The configuration box can be retrieved using:

const struct heif_tai_clock_info* heif_track_get_tai_clock_info_of_first_cluster(struct heif_track*);

Listing B.36 — API for getting the TAI clock information for the track

When writing tracks with SAIs attached to the samples, libheif offers two possibilities as to how the data is structured in the file:

It is also possible to assign a track-level GIMI content ID to the track. This can be obtained with

const char* heif_track_get_gimi_track_content_id(const struct heif_track*);

Listing B.37 — API for getting the GIMI content ID for the track

and when writing the track, it is also specified in heif_track_info.

B.8.6.  SAI Interleaving

When writing tracks with SAIs attached to the samples, libheif offers two possibilities how the data is laid out in the file.

Which mode to choose can be specified in the heif_track_info structure.

B.8.7.  Track References

Usually, metadata tracks provide additional information related to a visual track. This can be indicated in the HEIF file by setting a track reference. A track reference has a type, which in this case is ‘cdsc’ (Content Description). When writing a file, the track references can be added with

void heif_track_add_reference_to_track(heif_track*, uint32_t reference_type, heif_track* to_track);

Listing B.38 — API for adding a track reference to another track

In order to indicate that a metadata track is describing a visual track:

heif_track_add_reference_to_track(metadata_track, heif_track_reference_type_description_of, visual_track);

Listing B.39 — Specifying that a track contains metadata for another track

For the corresponding functions to list the track references when reading files, consult the libheif/heif_sequences.h header file.

B.9.1.  Encoding Sequences with heif-enc

Two options were added to the all-around command-line tool for encoding HEIF files ‘heif-enc’ to encoding image sequences:

  -S, --sequence            encode input images as sequence
                            (input filenames with a number will pull in all files with this pattern).
      --timebase #          set clock ticks/second for sequence
      --duration #          set frame duration (default: 1)
      --fps #               set timebase and duration based on fps
      --vmt-metadata FILE   encode metadata track from VMT file

Listing B.40

Option -S switches heif-enc to sequence encoding mode. The input images are read from individual image files. However, all input files do not need to be specified on the command line. heif-enc assumes that the input files are numbered like image0001.jpg to image4200.jpg. Only one filename has to be specified. The tool will automatically detect the number part in the filename and search for the first and last number in that directory.

The frame-rate can be specified in two ways.

As an experimental option, --vmt-metadata FILE was added. This reads the VMT metadata from the given file and adds it as a separate metadata track. This option will only be available when libheif has been compiled when using the option -DENABLE_EXPERIMENTAL_FEATURES=on during cmake configuration.

B.9.2.  Decoding Sequences to Separate Images

The existing heif-dec command line tool has been extended to decode sequences and write the images to individual images. Decoding of sequences (instead of decoding the still images) is enabled with the command line option -S or --sequence. Execution of this command will output a numbered list of images.

B.9.3.  Getting File Information with heif-info

The heif-info command line tool will output summary information about the image sequence HEIF file such as the following:

MIME type: image/heis
main brand: heis
compatible brands: msf1

sequence time scale: 30 Hz
sequence duration: 24 seconds
track 1
  handler: 'vide' = video
  resolution: 1024x1024
  sample entry type: hvc1
  sample auxiliary information: stai, suid
track 2
  handler: 'meta' = metadata
  sample entry type: urim
  uri: urn:smpte:EXAMPLE
  sample auxiliary information: stai, suid
  references:
    cdsc: track#1

Listing B.41

B.9.4.  Displaying Image Sequences

A new command line tool heif-view was added to show image sequences in real-time. This command is called with the HEIF file and shows the sequence in a UI window. The Simple DirectMedia Layer (SDL) library is required as a compilation dependency for this tool.

The tool has the following options:

heif-view  libheif version: 1.19.7

Usage: heif-view [options]  <input-file>

Options:

  -h, --help                     show help
  -v, --version                  show version
      --list-decoders            list all available decoders (built-in and plugins)
  -d, --decoder ID               use a specific decoder (see --list-decoders)
      --speedup FACTOR           increase playback speed by FACTOR
      --show-sai                 show sample auxiliary information
      --show-frame-duration      show each frame duration in milliseconds
      --show-track-metadata      show metadata attached to the track (e.g., TAI config)
      --show-metadata-text       show data in metadata track as text
      --show-metadata-hex        show data in metadata track as hex bytes
      --show-all                 show all extra information

Listing B.42

With --speedup #, the playback speed can be increased or decreased.

The options starting with --show* enable output of the respective data in the terminal window. As metadata tracks contain custom data, it can either be shown as plain text with --show-metadata-text or as binary hex dump with --show-metadata-hex depending on the metadata format used.

D.1.1.  Read operations strategies

Apache SIS offers different strategies for reading a subset of the image:

The latter strategy (deferred execution) is inspired from Java Advanced Imaging (JAI), which was built on top of Java2D. Despite being 25 years old, JAI is still quite advanced even compared to modern libraries. JAI is not maintained anymore, but Apache SIS reuses many of its ideas, including the use of Java2D as a foundation. Apache SIS provides a custom implementation of the Java2D java.awt.image.RenderedImage interface. That interface can represent tiled images backed by different types of java.awt.image.SampleModel (data layouts) such as banded (each band in a separated array), pixel interleaved (values of each pixel in consecutive elements of the same array), packed (multiple values packed in a single integer using bitmasks), etc. The image upper-left pixel is not necessarily associated with the (0,0) coordinates. The complexity of accessing data in such a diversity of layouts is hidden behind standard Java2D API such as Raster.getSample(x, y, band). Apache SIS adds another optional abstraction level inspired from JAI with PixelIterator. The Java2D standard library has specialized implementations of SampleModel and Raster for optimizing the most common data layouts, with delegations to the video memory and GPU when possible.

Strategy #2 (deferred execution) can be seen in action with the SIS viewer. The figure below shows one of the images used in Testbed 20. The rectangle outline on the right of the “ACT2017.tiff” label represents the whole file. The filled rectangles inside the outline represent the ranges of bytes that have actually been read. This representation is updated dynamically as the user zooms and pans the image. This on-demand reading of range subsets works with file systems, HTTP ranges, and Amazon S3. The figure below shows the result after a few of such user’s actions.

Figure D.1 — Visualizing the ranges of bytes read from a file

Contraryy to the viewer application, the benchmarking described in the OGC Testbed 20 GIMI Benchmarking Report OGC [[_4-041]] uses strategy 1 because of its more predictable nature.

D.1.2.  Implementation perspective on GeoTIFF versus GeoHEIF

The structure of a GeoTIFF file is a table of 32-bits (with standard TIFF) or 64-bits (with BigTIFF) pointers. The data referenced by these pointers can be located anywhere in the file in any order, not necessarily near the locations where the pointer values were provided. These pointers are used for various kinds of data, not only the tile pixel values. For example, the data can be information about bands (number of bits per sample, number of samples per pixel, minimum and maximum values), the color map, and the GeoKeys.

For an in memory GeoTIFF image, dereferencing these pointers is very efficient. For a GeoTIFF image on the local file system, dereferencing can be done with file seek operations, which are still reasonably efficient on modern hardware and operating systems. But for an image read from the cloud, dereferencing is more challenging because bytes are received sequentially and seek operations (closing a connection and sending a new HTTP range request) are costly. In Apache SIS, the data read from the cloud transit in a buffer of arbitrary size (currently 32 kilobytes). All pointers referencing data that are present in the buffer are dereferenced immediately. Other pointers are put on a waiting list and opportunistically dereferenced later, when the buffer is filled with new data. If, for answering a user’s request, the value referenced by a pointer on the waiting list become mandatory, and if that value is far from the current stream position (“far” being an arbitrary concept), only then Apache SIS will perform a file seek or HTTP range request. However, it will consult the waiting list for opportunistically grouping nearby values with the same request.

A Cloud-Optimized GeoTIFF (COG) file is a GeoTIFF file organized following a small set of conventions. Some of those conventions are recommendations about the positions in the file of the data referenced by the pointers. Those recommendations aim to reduce the scattering of data, thus giving a chance for implementations to reduce the amount of seek operations or HTTP range requests that they will have to perform. But COG does not completely eliminate the effort that implementations must do.

The structure of a GeoHEIF file is different. Data are stored in boxes of various types, and some boxes can contain other boxes. As each box contains all its data, there are generally no pointers to arbitrary positions in the file except in a few cases designed for large data such as ChunkOffsetBox and ItemLocationBox. A box can be skipped if its content is not of interest. Skipping a container box skips also all boxes contained inside that container. This structure is easy to implement in an object-oriented language, because each type of box can be mapped to a class in the programming language. There is no need to implement something like a waiting list of pointers as in the GeoTIFF case. However, an inconvenience is that it is not always possible to jump directly to information of interest. A reader can only move from one box to the next in a sequential read, except for large data referenced by pointers in ‘ChunkOffsetBox’ and similar special cases. But this inconvenience is mitigated when all the boxes to skip are contained in a larger box, since the latter can be skipped in one operation.

D.2.1.  Tiling

GDAL provides a GDALDatasetRasterIO(…) function for reading arbitrary sub-regions with arbitrary sub-samplings. If a user’s request spans many tiles, that function takes care of reading all needed tiles and assembling the result in a single output array. Alternatively, GDAL also provides the GDALReadBlock(…) function for reading on a tile-by-tile basis, which is potentially more efficient. Apache SIS uses the former function, but requests only regions that should correspond to a whole tile or an integer number of tiles. These requests are computed by the same TiledGridCoverage Java class as the one used by the pure-Java GeoTIFF and GeoHEIF implementations (Annex D.1), using the tile size given by GDALGetBlockSize(…). This strategy relies on the assumption that GDALDatasetRasterIO(…) will detect that the requests match the tile layout and therefore can be executed as efficiently as if GDALReadBlock(…) were invoked.

The GDAL-Java binding uses the tile cache inherited by Apache SIS TiledGridCoverage class. The GDAL tile cache is not enabled by default. The Apache SIS tile cache is based on soft references (a mechanism that supports detecting when a tile is not referenced anymore by the client application). The C/C++ smart pointers would have some similarities (while not fully equivalent) but are not used by the GDAL raster API.

D.3.1.  Differences in extent of returned data

When the user requests a subregion of a raster, GDAL returns the raster data for exactly the requested subregion. This is necessary because GDALDatasetRasterIO(…) copies the raster data in an array allocated by the user. Apache SIS does the opposite: It allocates the array itself and returns the raster data encapsulated in data structures of the java.awt.image package. By leveraging the abstraction provided by those data structures, Apache SIS takes the liberty of returning more data than requested if doing so is more efficient. In particular, Apache SIS generally rounds a user request to an integer number of tiles, which enables caching the tiles, return them directly (without copy) and reuse them (again without copy) if needed in subsequent requests. Compared to GDAL, it is as if the GDALRasterBlock::GetLockedBlockRef(…) function (the function for accessing directly the internally cached blocks) was used transparently for all I/O operations with Apache SIS, but without the need to lock/unlock tiles on user’s side.

The fact that users may receive something different than what they requested is partially transparent. This is because the image coordinate system is adjusted for having pixel coordinates (0,0) at the beginning of users’ request. In other words, the upper-left corner may start at negative pixel coordinates if the image is larger than requested. Users can ignore this detail for some operations such as accessing pixel values or rendering the image. See, for example, listing E.4 in GIMI Lessons Learned and Best Practices Report [OGC 24-040] Annex §E.2.2.1.

D.3.2.  Differences in zoom levels

Since users may request an image of any size, GDALDatasetRasterIO(…) may resample the image for fitting in the requested region. If the resampling method is nearest neighbor and a pyramid of images at different resolutions is available, GDAL may use a pyramid level slightly less accurate than the requested resolution with a tolerance factor of 1.2. Apache SIS is more conservative: It may apply subsampling but never resamples during I/O operations (resampling is available as a separated operation) and unless there is no such data always selects a pyramid level providing at least the requested resolution. The cost of this strategy is potentially slower I/O operations if Apache SIS selects a finer pyramid level than GDAL This difference is not always visible at rendering time.

D.3.3.  Differences in georeferencing

Both GDAL and Apache SIS use affine transforms for describing the conversions from pixel coordinates to CRS coordinates. However, the details differ in three ways.