2.1.1.  Prediction structures

Figure 1 — Prediction Structure in Video Coding

When coding video sequences, three different encoding modes can be coarsely differentiated:

While intra only and low-delay sequences can be decoded in the natural frame order, unrestricted video sequences may reorder the original images. Thus, the input video has to be buffered for a few frames such that the reordering can take place. This introduces delay on both the encoder side and the decoder side. Images in video without reordering can be encoded and decoded immediately, hence the name “low-delay”.

See Figures Figure 1 and Figure 2.

Figure 2 — Frame Ordering in Video Coding

The implementation in libheif provides the option to choose between these three coding modes. If a mode is not supported by the selected video codec, the library will fall back to a supported mode (for example “intra only” for JPEG-2000). More specifically, it depends on the exact video encoder whether a mode is supported. There can be different encoders for the same compression format that differ in which modes they support.

2.1.2.  Example: per-frame sizes for the three prediction structures

To illustrate the effect of the prediction structure on compression efficiency, a 100-frame video sequence was encoded three times with H.264 (x264), once for each prediction mode, while keeping all other encoder parameters identical. The resulting per-frame sizes are shown in Figures Figure 3, Figure 4, and Figure 5.

Figure 3 — Per-frame sizes for the "intra-only" mode. Total file size: 6,299,508 bytes.

Figure 4 — Per-frame sizes for the "low-delay" mode. Total file size: 2,842,387 bytes.

Figure 5 — Per-frame sizes for the "unrestricted" mode. Total file size: 2,193,068 bytes.

In the “intra-only” mode, every frame is coded independently, so the per-frame sizes are nearly equal and uniformly large. This mode produces the largest file because no inter-frame redundancy is exploited. On the other hand, since each frame can be decoded independently of all others, this mode allows random access to any frame without first decoding preceding frames.

In the “low-delay” mode, periodic keyframes (visible as the tall bars at the start and at frame indices 40 and 90) are coded as intra frames, while the frames in between are predicted from preceding frames and are therefore much smaller. The total file size drops to about 45% of the intra-only file.

In the “unrestricted” mode, the encoder uses a more complex prediction pattern with both forward and backward references. This produces a characteristic alternation between larger reference frames and very small frames that are predicted from both neighbours, resulting in roughly half of the frames being extremely small. The total file size is the smallest of the three modes, at about 35% of the intra-only file.

Note that the exact file-size differences between these modes are highly content-dependent. Sequences with little change over time and slow motion benefit much more from inter-frame prediction than sequences with rapid scene changes or strong motion, where prediction is less effective and the relative gain over intra-only coding shrinks.

2.3.1.  New plugins

During the OGC Testbed-21 initiative, a new plugin based on the x264 library was written to encode H.264 sequences. Decoding of H.264 sequences was previously done with a plugin based on the OpenH264 library. Additionally, the existing H.265 decoding plugin based on the ffmpeg library was extended to also decode H.264 streams.

2.6.1.  Sidecar items

Arbitrary data can be embedded into a HEIF file as non-image items. Functions for adding such custom items to a HEIF file and to extract them from existing files were already added to libheif in Testbed-20. During Testbed-21, options were added to heif-enc to easily add a sidecar file with a MIME type and optional name string:

--add-mime-item TYPE       add a mime item of the specified content type (experimental)
--mime-item-file FILE      use the specified FILE as the data to put into the mime item (experimental)
--mime-item-name NAME      assign the name to the embedded item (experimental)

Libheif and heif-enc will not do any processing on this data (except optional compression), it will be included verbatim into the file. The optional compression method can be specified by

--enable-metadata-compression ALGO  enable metadata item compression (experimental)
                                    Choose algorithm from {off,deflate,zlib,brotli}.

There also exists a new option for heif-dec to extract such a sidecar file from the HEIF file by specifying the MIME type of the item that should be extracted:

--extract-mime-item TYPE   extract the MIME item with the given content type into a file (mime-item.data)

2.12.1.  TIFF loader

Reading the TIFF input is implemented in heif-enc as an additional loader, just like the existing JPEG or PNG loaders. Reading TIFF turned out to be much more complex since there is a wide variety of TIFF files that all need special code. The following input formats and TIFF file organizations are supported:

The TiledTiffReader does not read the full image at once, but provides an interface to iterate through all overview images and image tiles such that the TIFF image organization can be mirrored in the GIMI output, with each tile being encoded independently. This allows to convert images much larger than the available memory.

The following TIFF features are not supported yet:

2.12.2.  JPEG loader changes

The JPEG loader was extended to support CMYK colorspace in the input. This is converted to RGB since libheif has no CMYK encoding support yet.

2.12.3.  RAW image loader

Since some Testbed participants wanted to convert raw data, especially floating point data, a reader for raw data files was also implemented. This is also a loader module for heif-enc, but the user has to specify the image dimensions since there is no header in these files:

--raw                      force raw pixel data input (for files without .raw/.img suffix)
--raw-width #              width of raw input image (computed from file size if omitted)
--raw-height #             height of raw input image (computed from file size if omitted)
--raw-type TYPE            pixel data type: uint8, sint8, uint16, sint16, uint32, sint32, float32, float64
--raw-endian ENDIAN        byte order of input data: little (default), big

This is currently the only method to input 32 bit signed/unsigned integers and 64 bit floats into heif-enc. The --raw option can be omitted if the input filename ends with a .raw or .img suffix. It is also sufficient to specify either a width or height. The other dimension is computed automatically from the file size and the known dimension.

2.13.1.  Shell wrapper usage

The basic syntax of the wrapper script is:

geotiff_to_gimi.sh <input.tif> [options...] -o output.heif

Options handled by the script itself are:

--tiles                    generate per-tile geometry metadata (forwarded to the Python script)
--ttl NAME                 save the Turtle metadata to NAME instead of embedding it
--metadata-script FILE     use FILE instead of the default `geotiff_metadata_to_turtle.py` script

All other options are forwarded verbatim to heif-enc. This means that any heif-enc option, such as the compression format, quality, or tiling mode, can be appended directly.

# Simple conversion with default settings (H.265, embedded Turtle)
geotiff_to_gimi.sh image.tif -o output.heif

# Tiled conversion with per-tile metadata and quality 50
geotiff_to_gimi.sh image.tif --tiles -q 50 -o output.heif

# Keep the Turtle file as a sidecar
geotiff_to_gimi.sh image.tif --tiles --ttl metadata.ttl -o output.heif

2.13.2.  Metadata generation

The Python script geotiff_metadata_to_turtle.py uses the rasterio library to read the GeoTIFF and emits RDF/Turtle to standard output. The script requires rasterio (pip install rasterio).

The script can be used stand-alone:

python3 geotiff_metadata_to_turtle.py image.tif            # full-image bounds only
python3 geotiff_metadata_to_turtle.py image.tif --tiles    # add per-tile bounds

The generated Turtle consists of:

If the GeoTIFF’s coordinate reference system is not EPSG:4326, the bounds are automatically reprojected to WGS 84.

An example of the generated Turtle for a single tile:

@prefix cco: <https://www.commoncoreontologies.org/> .
@prefix geosparql: <http://www.opengis.net/ont/geosparql#> .

# image
<urn:uuid:7d29f93-f42f-4cd8-9078-fc1a9a208bfc> a cco:ont00002004 .

# tile 0 0 0
<urn:uuid:430a4166-5079-4477-888b-bc23857931b4> a geosparql:Geometry ;
    geosparql:asWKT "POLYGON((-64.83 18.37,-64.83 18.33,...))"^^geosparql:wktLiteral ;
    cco:ont00001808 <urn:uuid:7d29f93-f42f-4cd8-9078-fc1a9a208bfc> .

When heif-enc reads this Turtle file via --turtle, it matches the geometry UUIDs to the corresponding GIMI content IDs assigned to the tiles in the HEIF file, establishing the link between the geospatial metadata and the image tiles.

2.13.3.  Tiling and overview pyramids

Without the --tiles flag (or when the input GeoTIFF is not internally tiled), the script emits a single geometry covering the full image bounds.

With --tiles, the script iterates over the GeoTIFF’s internal tile grid and emits one geometry per tile. It also processes all overview (pyramid) levels stored in the GeoTIFF. For each overview level, the pixel-to-coordinate mapping is scaled by the overview’s downsample factor, and the reduced-resolution tile grid is iterated in the same manner. This produces a complete set of geospatial geometries for every tile at every resolution level, which heif-enc can then use to populate the GIMI content IDs across the full image pyramid.

When converting a tiled GeoTIFF, heif-enc reads the tiles individually through its TiledTiffReader, allowing conversion of images much larger than available memory. The resulting GIMI file mirrors the GeoTIFF’s tiling structure and overview hierarchy as a pyramid entity group.

2.14.1.  Encoder/decoder architecture

The previous implementation of the ISO 23001-17 codec already had separate code paths for each interleave type, but these were selected by conditional branches within a single function and shared much of their surrounding logic. As the number of supported configurations grew in interleave types, bit depths, padding modes, endianness, and data types, this became increasingly difficult to maintain and extend. The complex, very general implementation also made the decoding unnecessarily slow. During Testbed-21, both the decoder and encoder were refactored into a modular architecture where a subset of configuration variants is implemented in its own class. A factory class selects the appropriate module at runtime based on the full set of uncC (uncompressed configuration) box parameters, not just the interleave type. This means that, for example, the same interleave mode can have multiple decoder implementations: one optimized for byte-aligned components with arbitrary data types, another for sub-byte bit depths with block packing, and so on. This results in an easier implementation and much faster runtime for the commonly used formats.

An important design principle is that the heif_image representation used by libheif and the application does not expose the full combinatorial space of uncC configurations. Instead, libheif maintains a small set of choices for pixel formats in heif_image:

All the many possible uncC data layouts are decoded into one of these formats, and conversely, these formats are encoded into the appropriate uncC configuration by the encoder factory. This keeps the API simple for users, who only need to work with a small, well-defined set of image formats. Furthermore, libheif provides automatic conversion between all supported pixel formats, so an application can request any output format regardless of how the data is stored in the file.

This modular architecture improves maintainability: adding support for a new uncC configuration requires only adding a new codec module without touching the existing decoders or encoders.

Decoder architecture.

When decoding an ISO 23001-17 image, a factory class inspects the uncC box parameters and selects the first decoder module whose requirements match the configuration. This enables having a set of simple but fast decoders that are preferably selected when they are capable of decoding the input image, and only switches to more complex implementations when required. Seven concrete decoder modules are currently implemented, each in a separate decoder class:

All decoder modules support any values for row_align_size (row padding) and tile_align_size (tile padding). The decoder modules are generally independent of the image colorspace. They operate on the data layout described in the uncC box and do not need to know whether the components represent RGB, YCbCr, or arbitrary sensor channels. The exception is the mixed-interleave decoder, which is specific to YCbCr images with chroma subsampling.

The bytealign_component decoder is the only module that supports data types beyond unsigned integers. It handles signed integers, IEEE 754 floating-point, and complex numbers at bit depths of 8, 16, 32, 64, and 128 bits (128-bit for complex numbers only, representing two 64-bit floats). This module also supports both little-endian and big-endian component byte order.

The component decoder also handles the tile_component interleave mode (mode 4), where tile data for each component is stored separately in the file rather than being concatenated into a single block per tile.

Encoder architecture.

The encoder follows a symmetric factory class design. The factory class selects the appropriate encoder based on the image’s colorspace, chroma format, data type, and encoding options. The selected encoder decides which ‘uncC’ parameters are best for encoding the given image. The encoder’s primary goal is to write the data as compactly as possible.

Four concrete encoder modules are implemented:

The native libheif image formats are either separate pixel planes for each component or interleaved RGB. Three of the encoder modules are specific for the RGB interleaved case with different bit depths. The component encoder is the universal fallback that accepts any colorspace and data type. It is the only encoder that supports signed integers, floating-point, complex numbers, and YCbCr subsampling. The three RGB-specific encoders handle pixel-interleaved output in various packed formats.

The following figures illustrate the bit-level data layout produced by the rgb_pixel and rgb_block_pixel encoders for a single RGB pixel at different bit depths. The colored blocks represent the bits of the red, green, and blue components. Bit numbers within each byte are shown at the top.

Generally, for bit-depths smaller than 8 bits per component, the values are padded to 8 bit. For bit-depths 9 to 13, the RGB values are packed densely into 4 or 5 byte blocks and then padded to the next byte boundary. The rationale for this is that the data should be arranged for a good compressibility with the “deflate” or “brotli” algorithms. These algorithms work on a per-byte basis, detecting runs and repetitions of byte sequences, and their efficiency would be reduced if pixel values are not aligned to byte boundaries. For values with 9 to 13 bits per component, on the other hand, padding would inflate the amount of data so much that it probably cannot be compensated by a better compression. Thus, the RGB pixel values are packed into 4 or 5 bytes. This allows to still efficiently compress runs of similar RGB pixels.

How much can be gained by this arrangement depends on the input data content. A thorough evaluation is still open for research.

2.14.2.  Data types and multi-channel images

Extended data type support. Prior to Testbed-21, the uncompressed codec only supported unsigned integer components at 8 to 16 bits per sample. During Testbed-21, the codec was extended to support a wider range of data types as defined in ISO 23001-17:

These data types are expressed in the ISO 23001-17 uncC box via the component_format field, with values component_format_unsigned (0), component_format_signed (3), component_format_float (1), and component_format_complex (2). On the C API side, the new heif_channel_datatype enum represents these four data types. Each data type has a corresponding pair of pixel accessor functions (read-only and read-write) in heif_uncompressed.h, all indexed by component index:

const float*  heif_image_get_component_float32_readonly(image, component_id, &stride);
float*        heif_image_get_component_float32(image, component_id, &stride);
// similarly for uint8, uint16, uint32, uint64, int8, int16, int32, int64,
//   float64, complex32, complex64

Multi-channel image model.

Alongside the data type extensions, the internal image representation was changed from a fixed map of heif_channel enum values (such as heif_channel_Y, heif_channel_Cb, heif_channel_R) to a list of components identified by integer index. This new model directly reflects the ISO 23001-17 component definition box (cmpd), where each component has an index and a type.

The key API additions for the multi-channel model are:

// Add a component with arbitrary type, data type, and bit depth
heif_image_add_component(image, width, height, component_type, datatype, bit_depth, &out_idx);

// Query the number of components with pixel data and the total cmpd components
heif_image_get_number_of_used_components(image);
heif_image_get_total_number_of_cmpd_components(image);

// Get the valid component ids (which may be non-contiguous)
heif_image_get_used_component_ids(image, out_ids);

// Query per-component properties
heif_image_get_component_width(image, component_id);
heif_image_get_component_height(image, component_id);
heif_image_get_component_bits_per_pixel(image, component_id);
heif_image_get_component_type(image, component_id);

Not all components declared in the cmpd box need to have pixel data planes. For example, a Bayer-pattern image may have a single filter_array component with pixel data, but the cmpd box also declares reference components (red, green, blue) that are used by the color pattern box (cpat) to describe which color each filter array pixel represents. These reference components have cmpd entries but no pixel planes.

The set of supported component types was extended to cover the full ISO 23001-17 Table 1:

monochrome, Y, Cb, Cr, red, green, blue, alpha,
depth, disparity, palette, filter_array, padded,
cyan, magenta, yellow, key_black

This enables images with arbitrary component configurations, such as CMYK images (cyan/magenta/yellow/key_black), depth maps, or disparity images, alongside the traditional RGB and YCbCr colorspaces.

Since backwards compatibility has to be maintained, images can be constructed and read using either the new component types or with the established heif_colorspace / heif_channel combination. When reading an ISO 23001-17 image, libheif analyzes what components have been defined. If these match a common pattern that can be expressed with heif_colorspace and heif_channel, otherwise, it assigns the new heif_colorspace_nonvisual to it. For the opposite direction, when the user generates an image based on a heif_colorspace preset, libheif creates the corresponding ISO 23001-17 components internally.

2.14.3.  Bayer-image decoding

A color filter array (CFA) sensor captures a single color per pixel in a repeating pattern. The most common arrangement is the 2×2 Bayer pattern (RGGB), but other patterns such as RGBW (with a panchromatic/white channel) or Quad Bayer Coding (QBC) with 4×4 patterns are also used. ISO 23001-17 represents such images using a single filter_array component for the pixel data and a color pattern box (cpat) that describes the repeating pattern layout.

Encoding Bayer images.

To encode a Bayer image, the application creates an image with the new heif_colorspace_filter_array colorspace, which uses a single channel for the filter array pixel data. The Bayer pattern is attached to the image via:

typedef struct heif_bayer_pattern_pixel
{
  uint32_t component_id;
  float component_gain;
} heif_bayer_pattern_pixel;

heif_error heif_image_add_bayer_component(heif_image*,
                                          uint16_t component_type,
                                          uint32_t* out_component_id);

heif_error heif_image_set_bayer_pattern(heif_image*,
                                        uint32_t bayer_component_id,
                                        uint16_t pattern_width,
                                        uint16_t pattern_height,
                                        const heif_bayer_pattern_pixel* patternPixels);

where each heif_bayer_pattern_pixel specifies a component ID and an optional gain value. To get the component IDs, a Bayer component first has to be added with heif_image_add_bayer_component(), which maps the component type to a new component ID. These can then be used in the heif_bayer_pattern_pixel.

Note that heif_image_set_bayer_pattern() requires the bayer_component_id to which filter-array component this pattern belongs, but ISO 23001-17 only allows one filter-array component with a cpat per image. Since this might be an oversight in the ISO 23001-17 standard, the API was designed to be future-proof for potential updates of the standard.

Decoding Bayer images.

On the decoder side, the cpat box is read and its component indices are validated against the cmpd table. The Bayer pattern is then attached to the decoded heif_image. An application can access the raw filter-array pixel data directly, or it can request automatic demosaicing during the color conversion step. A bilinear interpolation demosaicing algorithm was implemented that converts the filter-array image to interleaved RGB, supporting both 8-bit and 9-16 bit images. The demosaicing algorithm is integrated into the color-conversion pipeline such that it operates transparently if the application specifies a specific output colorspace that libheif should convert the image to.

The following functions have been added to the API to read the Bayer pattern information. The function heif_image_get_bayer_pattern() returns the pattern with the component IDs. These can be resolved to component types, with heif_image_get_component_type().

int heif_image_get_bayer_pattern_size(const heif_image*,
                                      uint32_t bayer_component_id,
                                      uint16_t* out_pattern_width,
                                      uint16_t* out_pattern_height);

// Get the Bayer / filter array pattern pixels.
// The caller must provide an array large enough for pattern_width * pattern_height entries
// (use heif_image_get_bayer_pattern_size() to query the dimensions first).
// Returns heif_error_Ok on success, or an error if no pattern is set.
heif_error heif_image_get_bayer_pattern(const heif_image*,
                                        uint32_t bayer_component_id,
                                        heif_bayer_pattern_pixel* out_patternPixels);

uint16_t heif_image_get_component_type(const heif_image*, uint32_t component_id);

The heif-gen-bayer tool.

To facilitate testing and demonstration of Bayer-pattern image support, a new command-line tool heif-gen-bayer was implemented. This tool takes an RGB PNG image as input, simulates a Bayer-filtered sensor capture by sampling the appropriate color channel at each pixel according to the selected pattern, and writes the result as an ISO 23001-17 uncompressed image with the cpat box.

The tool supports several built-in filter array patterns:

Custom patterns can also be specified as a string of R/G/B characters (e.g., -p BGGR).

heif-gen-bayer [options] <input.png> <output.heif>
heif-gen-bayer -S [options] <frame_NNN.png> <output.mp4>

Options:
  -b, --bit-depth #       output bit depth (default: 8, range: 8-16)
  -p, --pattern <name>    filter array pattern (default: rggb)
  -S, --sequence          sequence mode (expand numbered PNGs)
  -V, --video             use video track handler (vide) instead of pict
      --fps <N>           frames per second (default: 30)

In sequence mode (-S), the tool encodes multiple numbered PNG frames as an uncompressed image sequence or video track with the Bayer metadata attached to each frame.

Table 7 shows the result of encoding an RGB image through heif-gen-bayer with the RGGB pattern and then decoding it back with heif-dec, which applies bilinear demosaicing.

Table 7 — Bayer encoding/decoding example. Left: original RGB image. Right: image after Bayer-pattern encoding with heif-gen-bayer and decoding/demosaicing with heif-dec.

Figure 15

Figure 16

2.14.4.  ISO 23001-17 image metadata

ISO 23001-17 defines several optional metadata boxes that can be associated with uncompressed image items. During Testbed-21, support was added for five of these metadata types, covering sensor characteristics and imaging parameters. All metadata types survive the encode/decode roundtrip: the encoder reads the metadata from the HeifPixelImage and writes the corresponding ISO boxes, while the decoder reads the boxes and attaches the data to the decoded image.

Color pattern box (cpat). This box describes a repeating color filter array pattern as used in Bayer-type sensors. Each position in the pattern references a component index in the cmpd table and optionally specifies a gain value. The encoding and decoding of this box is described in detail in the Bayer-Image Decoding section above.

Chroma sample location (cloc). This box specifies the spatial position of chroma samples relative to luma samples in 4:2:0 subsampled images. It uses integer values 0 through 5 from ISO 23091-2 / ITU-T H.273, plus an additional value 6 defined by ISO 23001-17 for a two-row specification of chroma positions. The API provides a straightforward set/has/get interface:

heif_image_set_chroma_location(image, location);   // location: 0-6
heif_image_has_chroma_location(image);              // returns non-zero if set
heif_image_get_chroma_location(image);              // returns location value

Polarization pattern (splz). This box describes a polarization filter array over the sensor pixels. Each position in the pattern specifies a polarization angle in degrees (0.0 to 360.0), or a special IEEE 754 NaN value (bit pattern 0xFFFFFFFF) to indicate “no polarization filter” at that position. Multiple polarization patterns can be added, each associated with a different group of components. The API supports querying which pattern applies to a given component index:

heif_image_add_polarization_pattern(image, num_component_ids, component_ids,
                                    pattern_width, pattern_height, polarization_angles);
heif_image_get_number_of_polarization_patterns(image);
heif_image_get_polarization_pattern_info(image, pattern_index, ...);
heif_image_get_polarization_pattern_data(image, pattern_index, ...);
heif_image_get_polarization_pattern_index_for_component(image, component_id);

Sensor non-uniformity correction (snuc). This box stores per-pixel gain and offset correction tables. The correction equation is y = gain * x + offset, where x is the raw sensor value and y is the corrected value. The gain and offset arrays are full image-sized arrays of float values. A flag indicates whether the correction has already been applied to the pixel data. Multiple NUC tables can be added, each for a different group of components:

heif_image_add_sensor_nuc(image, num_component_ids, component_ids,
                          nuc_is_applied, image_width, image_height,
                          nuc_gains, nuc_offsets);
heif_image_get_number_of_sensor_nucs(image);
heif_image_get_sensor_nuc_info(image, nuc_index, ...);
heif_image_get_sensor_nuc_data(image, nuc_index, ...);

Sensor bad pixels map (sbpm). This box records known defective sensor positions: entire bad rows, entire bad columns, and individual bad pixels identified by (row, column) pairs. Like the NUC box, a flag indicates whether the bad-pixel correction has already been applied. Multiple maps can be added for different component groups:

heif_image_add_sensor_bad_pixels_map(image, num_component_ids, component_ids,
                                     correction_applied,
                                     num_bad_rows, bad_rows,
                                     num_bad_columns, bad_columns,
                                     num_bad_pixels, bad_pixels);
heif_image_get_number_of_sensor_bad_pixels_maps(image);
heif_image_get_sensor_bad_pixels_map_info(image, map_index, ...);
heif_image_get_sensor_bad_pixels_map_data(image, map_index, ...);

All of these API functions are declared in the public header heif_uncompressed.h, with supporting type definitions in heif_uncompressed_types.h.

2.14.5.  Mapping of component IDs to cmpd and uncC indices

A challenge during implementing multi-spectral or Bayer-encoded images according to the ISO 23001-17 standard was that there is no consistent way to identify the components. Furthermore, the implementation has to be backwards compatible to the existing libheif API. The existing libheif implementation was targeting either monochrome, RGB, or YCBCr images with optional auxiliary channels like alpha, depth, or disparity. The pixel data was organized in planes that directly mapped to the colorspace, e.g., R, G, B, Y, Cb, Cr as planar pixel planes, indexed by the color channel. As a special case, there was also an interleaved RGB(A) format as a single plane, since many applications exchange image data in this organization.

For a complete ISO 23001-17 implementation, this scheme does not work anymore since an ISO 23001-17 image may contain several monochrome components, or even user-defined components with no pre-defined type. On the other hand, there is no direct correspondence to an “interleaved RGB” component in ISO 23001-17. Another difficulty was that ISO 23001-17 may contain Bayer images, which define components with no directly coded image data, but metadata can still be assigned to these non-coded components.

When choosing a suitable ID for identifying components in the API, it was noticed that

Thus, a new “component ID” had to be introduced that is used in the library API to uniquely identify the components, working both for coded image components and Bayer pattern components. There is no direct, easy correspondence between the “component ID” and a cmpd or uncC index. Instead, libheif will use the component ID exclusively in the API and generate suitable cmpd and uncC entries when encoding. Trying to combine similar entries to save space.

A very simple case, without any of the aforementioned difficulties, is shown in Figure 17. It shows a heif_image three image planes containing the red, green, and blue components (yellow boxes). Each of these heif_channel components maps directly to an ISO 23001-17 component (red boxes) and receives its unique component ID. A snuc metadata box (Sample Non-Uniform Correction) is assigned to the green and blue component. Since all three components have a different type, they receive their own cmpd entry and the snuc metadata box is stored with references to cmpd indices 1 and 2.

A more complex case is shown in Figure 18. It shows a multi-spectral image with four monochrome channels. A snuc metadata box is assigned to the components with IDs 1,2,3 and a sbpm metadata box is assigned to the components with IDs 2 and 4. Now, even though all components have the same type, they cannot be mapped to the same cmpd entry because the ISO 23001-17 standard says that metadata boxes are assigned to the components indirectly through the cmpd index. Thus, libheif examines which components have the same type and the same metadata assigned to it. In this example, this is the case only for components 1 and 3. These two component IDs can be both mapped to the cmpd index 1. The snuc metadata box will finally reference cmpd entry 0 and 1, while sbpm will reference 0 and 2.

The example in Figure 19 depicts an image with an interleaved RGB plane and an independent Bayer filter image. The interleaved RGB plane is stored in a single heif_image plane inside libheif, but since there is no direct correspondence to that in ISO 23001-17, this interleaved RGB plane, maps to three ISO 23001-17 components (1-3). The Bayer filter image only has one coded component (4) for the filter array, but the cpat box in this example signals that the filter array should decode to three R,G,B components (not to be confused with the independent interleaved image). These Bayer RGB components have no directly coded pixel data, but they receive their own component IDs (5-7) such that it is possible to assign metadata to them. In this example, libheif recognizes that the R,G,B components have the same type, but since the blue component in the interleaved image has a sbpm metadata box assigned to it, a separate cmpd entry is required. In the figure of this example, a line with the uncC indices has been added to show that there is a second mapping inside libheif that maps the component IDs to uncC indices.

When decoding a HEIF file with an ISO 23001-17 image, the whole process runs in the opposite direction. The cmpd indices are replaced with unique component IDs that are then used in the API. Note that the component IDs are volatile and may be different at the encoding and decoding side.

2.14.6.  “unci” compression benchmark

During image encoding experiments, it was observed that saving “unci” images with brotli compression was much slower than with deflate compression. Both standard implementations of these encoding algorithms come with a selectable compression level setting, with 0-11 for brotli and 1-9 for deflate (excluding deflate level 0, which is uncompressed). The current implementation in libheif always uses the default levels, which is 11 (highest compression) for brotli and 6 (balanced) for deflate.

To assess the trade-off between file size and encoding time offered by the compression options of ISO 23001-17 (“unci”), a benchmark was run on a large test image (approximately 6.2 GB uncompressed) using both the deflate and brotli general-purpose compressors at all supported compression levels. Measurements were taken on an AMD Ryzen 9 9950X CPU. Both compression algorithms utilize only a single CPU core.

Figure 20 shows file size versus encoding time for the relevant range of compression levels. The uncompressed case (about 6.2 GB, 11 s) and the two slowest brotli settings (level 10: 16 min; level 11: 35 min) are omitted from the plot so that the interesting region, where brotli and deflate can be directly compared, is visible. The numbers next to each data point indicate the compression level.

For this image, brotli clearly dominates deflate: across the entire range, the brotli curve lies below and to the left of the deflate curve, meaning brotli produces smaller files in less time. Already at lowest settings (brotli level 1 to 4), brotli compresses faster than any deflate level while producing a file comparable in size to deflate at moderate levels. At higher quality settings, brotli reaches file sizes that deflate cannot match at any level.

It is also worth noting that brotli level 0 and 1 (6 s and 7 s) is faster than storing the uncompressed data (11 s): the compression effort is more than offset by the time saved writing a roughly six times smaller file to the SSD. In other words, using light brotli compression is not only smaller but also faster than writing “unci” data without any compression.

These results confirm that for this type of imagery, brotli is the preferred choice among the general-purpose compressors offered by ISO 23001-17. It should be reconsidered, however, whether the default level 11 for Brotli is the best choice given that its computation time is significantly larger, but also producing smaller file sizes. We keep this setting for the moment since encoding an image is a one-time cost. Whether there are any differences in decoding speed is yet to be explored.

2.15.1.  The libheif reader interface

In Testbed-20, the heif_reader callback interface in libheif was extended to version 2, adding functions specifically designed for on-demand network streaming. The version 1 interface already provided basic callbacks for sequential reading (read, seek, get_position, wait_for_file_size), but these are inefficient over a network because libheif issues many small reads during parsing. Each of these would translate into a separate HTTP request with significant round-trip latency.

The version 2 interface addresses this by adding the following callbacks.

The typical interaction between an application, the heif_reader implementation, and libheif is illustrated in Figure 21. When the application opens a remote HEIF file, libheif first requests the file header and the metadata box (meta) through request_range() calls. From the metadata, libheif learns the byte offsets of all tiles. When the application later requests decoding of a specific tile, libheif issues a request_range() for that tile’s data range, and the reader implementation translates this into an HTTP range request.

2.15.2.  Connecting libheif to curl

The tiled-image-viewer demonstration application, which previously only supported loading HEIF files from the local filesystem, was extended during the first of the Testbed-21 Code Sprints to also load and display HEIF files served over HTTP. The heif_reader version 2 callbacks are implemented in C++ classes that use libcurl to perform the actual HTTP communication.

When the viewer is started with a URL, the reader performs the following initialization.

After initialization, the reader is passed to libheif via heif_context_read_from_reader(). From this point on, libheif drives all data access through the callback functions. Each request_range() call translates to an HTTP range request using curl’s CURLOPT_RANGE option, and the fetched data is stored in a local cache for subsequent read() calls.

Two caching strategies were implemented, described below.

2.15.3.  Trivial cache

The first and simpler implementation stores each fetched byte range as a variable-length entry in a list. When libheif calls request_range(), the reader fetches the requested range from the server and appends it to the cache. Subsequent read() calls search the cache linearly for an entry that covers the requested position.

The release_file_range() callback removes cache entries that fall within the released range, freeing memory for data that libheif no longer needs.

This implementation is straightforward and was developed first as a proof of concept. However, it can be inefficient in practice because some compressed tiles are very small, resulting in many small HTTP requests with high per-request overhead.

2.15.4.  Block cache

The second, more advanced implementation divides the file into fixed-size blocks (64 KB by default, configurable via command-line option). At initialization, an empty block vector is pre-allocated with one entry per block covering the entire file.

When libheif calls request_range(), the reader:

Because the blocks have a fixed size, looking up whether a particular file position is already cached is an O(1) index computation rather than a linear search. More importantly, fetching data in large, fixed-size blocks means that a single HTTP request often covers several small compressed tiles at once, significantly reducing the total number of network round-trips compared to the trivial cache.

Figure 22 illustrates how this caching scheme works in practice. The top row represents the compressed tile data as it is laid out in the file, with each segment corresponding to one tile (A through W). The bottom row shows the fixed-size cache blocks (1 through 8) into which the file is divided. Note that tile sizes vary considerably: highly detailed tiles may occupy a large portion of a cache block, while smooth or uniform tiles compress to only a few bytes.

In the first example, suppose the application needs to decode tile E. Libheif issues a request_range() for tile E’s byte range, indicated by the two green arrows. The reader extends this range to the enclosing cache block boundaries and fetches cache blocks 2 and 3 in a single HTTP request. Because tiles C through H are all small and located close together in the file, this single request also brings in the data for tiles C, D, F, G, and H (shown in the green highlighted region). When any of these neighboring tiles are requested later, their data is already present in the cache and no additional network request is needed. Tiles that are spatially close in the image tend to be stored close together in the file, so this read-ahead effect is particularly beneficial when panning across a region of the image.

In the second example, tile P is requested. Its data (indicated by the two red arrows) falls entirely within cache block 6, which is fetched in a single HTTP request. This also loads part of tile Q’s data. When tile Q is subsequently requested, the reader finds that part of its data is already cached in block 6. Only the remaining portion needs to be fetched (yellow arrows), which triggers a request for cache block 7. This request, in turn, also brings in the data for tiles R and S (yellow highlighted region), again providing read-ahead for future tile requests in that area.

Figure 22 — Example of the block-based caching scheme. The top row shows compressed tile data (A-W) in the file. The bottom row shows fixed-size cache blocks (1-8). Green arrows and highlighting: requesting tile E fetches cache blocks 2 and 3, which also cover tiles C-H. Red arrows and highlighting: requesting tile P fetches cache block 6. Yellow arrows and highlighting: the subsequent request for tile Q only needs cache block 7, since part of Q’s data was already loaded with block 6.

Figure 23

In both implementations, tile loading and network transfer run on a background worker thread so that the user interface remains responsive while data is being fetched.

2.15.5.  Download progress visualization

When operating in URL mode, the viewer displays a progress bar at the top of the window indicating which parts of the file have been fetched. The bar shows unfetched byte ranges in red and cached ranges in green, giving the user immediate visual feedback on how much of the file has been downloaded.

Figure 24 — The tiled-image-viewer displaying a remote HEIF file. The progress bar at the top shows fetched byte ranges in green and unfetched ranges in red.

2.15.6.  Command-line options

The following command-line options were added to the tiled-image-viewer for network streaming support:

-u, --url              treat input as HTTP/HTTPS URL instead of local file
-t, --trivial-reader   use trivial cache reader instead of block cache (URL mode only)
-b, --block-size <kB>  block size in kB for block cache reader (default: 64)
-p, --primary          start with primary image (default: start at overview image)

Example usage:

# View a remote HEIF file with default block cache (64 KB blocks)
./tiled_image_viewer -u https://example.com/large_image.heif

# Use larger blocks for high-latency connections
./tiled_image_viewer -u -b 256 https://example.com/large_image.heif

# Use the trivial cache for comparison
./tiled_image_viewer -u -t https://example.com/large_image.heif