1.1.  About this Report

GeoHEIF is a container file format based on HEIF and ISOBMFF that compresses images into smaller files while maintaining high quality and provides an alternative to traditional image formats. The flexibility of the HEIF format supports different methods for storying images and their metadata. However, not all methods will be efficient and interoperable.

This Report provides a list of use cases where HEIF format can be used and how the data and the metadata can be structured. The use cases are based on discussions and testing done in OGC Testbed 20. Some cases are describe how to store different image types such as panchromatic or grey scale image and extend beyond photographic images. Others are more applicable to model outputs such as land cover classifications, atmosphere evolution, and weather forecast runs. Some other use cases consider actions such as map browsing (pan, zoom, clip…​) and the use of tiles and overview as well as confidentiality or extraterrestrial imagery. Finally, a set of use cases related with images sequences is provided and described.

1.2.  Aims

The focus of the OGC Testbed 20 GIMI Coverage format selection activity was to enumerate HEIF applications and select among the characteristics (items and properties) that need to be used from the long list of existing and proposed characteristics to improve the efficiency and the interoperability of the image format.

1.3.  Objectives

This Report collects the use cases discussed during the development and benchmarking activities conducted in OGC Testbed 20 that were focused on the HEIF and the GeoHEIF extensions.

2.1.  Introduction

This section provides an overview of use cases where the HEIF format can be used. For each case, the different items and item properties provided by the HEIF specification or by the proposed GeoHEIF extensions are selected, and an explanation on how to combine the items and the item properties is provided.

This is the list of use cases that are analyzed in this Report:

The following subsections describe each use case. Please note that the cases are based on the capabilities of the GeoHEIF format defined in the OGC Testbed 20 GIMI Specification Report (OGC 24-038) (e.g., datacube) and in the GIMI Lessons Learned and Best Practices Report (OGC 24-040) (e.g., tiling). Many of these “capabilities” are newly defined and were not implemented in current HEIF related libraries (such as libheif) at the time of this Report was written. In particular, the properties edim and pcel to define extra-dimensions and cell properties were not available. Support for non-integer data types might not exist yet (so currently, cell properties such as temperature and wind speed such be scaled and rounded to integers).

2.2.  Panchromatic or grey scale image

Panchromatic or grey scale imagery is one of the simplest cases. The HEIF file has an item containing a single channel image.

When using ISO/IEC 23001-17 (uncompressed codec), this is a component type 0 (monochrome). In JPEG-2000, the image is stored as a one channel image. When using video-oriented codecs (h.264, h.265, h.266, AV1) and JPEG, this is a YCbCr image with chroma format 4:0:0 (i.e., luminance without color information).

These various encodings may be abstracted away. For example, in the libheif API, all cases are mapped to images with a heif_colorspace_monochrome channel.

Additionally, if the image contains geospatial information, the ogeo brand must be included in the FileTypeBox (ftyp) to signal that the file structure follows the GeoHEIF. In this case, the coordinate reference system (CRS) is defined using the mcrs item property in the ItemPropertyContainerBox, and, if the image is georeferenced, the matrix transformation model is defined using the mtxf item property in the ItemPropertyContainerBox.

2.3.  Photographic landscape (RGB image neither georeferenceable nor georectified)

Photographic landscape is another simple use case. The HEIF file has an item containing a three-channel image (a.k.a. three compound image). This is typically in a format that supports RGB internally, such as JPEG-2000 or ISO/IEC 23001-17, or a format that is readily converted to RGB. For example, video codecs will usually store color images not in the RGB space but first transform those images into a YCbCr color space because that results in better compression ratios. Often, chroma channels are additionally subsampled to a lower resolution. However, images can be stored in native RGB by storing a nclx color profile in the metadata with a matrix_transform=0. This stores the green channel in the Y (luminance) channel and red, blue in the two chroma channels without subsampling (chroma 4:4:4). JPEG always uses YCbCr internally.

2.4.  Multiband satellite image (georeferenceable)

In the HEIF format, multiband satellite images can be structured in two ways. The HEIF file can have an image item in a format that internally supports multiple bands or can have multiple image items, one item for each band. The ogeo brand must be included in the FileTypeBox (ftyp) to indicate that the file contains geospatial information and to indicate a GeoHEIF file. In a GeoHEIF file, the images can be georeferenceable by associating them to the CRS in the mcrs item property and to a set of tie-points in the tiep item property, both in the ItemPropertyContainerBox and related to the image items in the ItemPropertyAssociationBox. The fact that some images share the same associations is an indication that they are part of the same multiband image. The description of the characteristics of each band can be added in the property CellPropertyTypeProperty (pcel) in the ItemPropertyContainerBox. There is a cell property type property for each image item of a different type. The specifics of the description of the band, such as the maximum and minimum frequency of the band, are expected to be found by following the ‘definition’ uri.

When a compressing video codec is used, it is necessary to store each band in a separate image as described in the greyscale case above because these codecs do not natively support multiple bands. When using JPEG-2000 or ISO/IEC 23001-17, it is possible to store the bands as separate, custom image components within the same image.

2.5.  Hyperspectral satellite image

Multiband and hyperspectral imaging are two distinct remote sensing technologies that differ primarily in their spectral resolution and number of bands captured. Multiband imaging captures data in a limited number of broader spectral bands, typically 3 to 20. These bands are discrete and separated, focusing on specific wavelengths of interest. Multiband sensors commonly capture visible light (RGB) and a few infrared bands. In contrast, hyperspectral imaging captures hundreds to thousands of narrow, contiguous spectral bands. This results in a much higher spectral resolution, allowing for more precise identification and characterization of materials.

Hyperspectral imagery can be arranged as a datacube. A datacube is a multi-dimensional (“n-D”) array of values designed for efficient data analysis and querying. The data is arranged in dimensions and cell properties. Dimensions can be coordinates, other continuous variables or even enumerations, e.g., categories.

In the case of hyperspectral imagery, the datacube is a 3D case with two spatial dimension and an extra dimension representing the narrow radiometric frequency interval of the spectral bands.

The third dimension is the spectral dimension that is represented by several image items, at least one for each of the N radiometric frequency values in the spectral dimension.

The third dimension, spectral, is defined as an extra-dimension in a ExtraDimensionProperty (edim) and the extra-dimension values ​is defined in ExtraDimensionValueProperty (edvl) as the central frequency of the interval. All images are associated with the same cell property type property (pcel). This property contains the description of the intensity of light in each radiometric frequency interval.

2.6.  Orthophotography (georectified)

Orthorectified imagery is a special case of a georectified image. In addition to the georeference process, orthorectification removes distortions caused by terrain variations, camera angle, and other factors. The resulting orthorectified image appears as if each cell were captured from directly above (vertical).

As in the case above, the HEIF file can have an image item in a format that internally supports multiple bands or can have multiple image items, one item for each band. The ogeo brand must be included in the FileTypeBox (ftyp) to indicate that the file contains geospatial information and to indicate a GeoHEIF file. In a GeoHEIF, the images can be georectified by associating them to the CRS in the mcrs item property and by defining the affine transformation matrix in the mtxf item property. These are both in the ItemPropertyContainerBox and related to the image items in the ItemPropertyAssociationBox. If the components of the image are only “colors” the association to a cell property type property (pcel) may not be necessary.

2.7.  Stereogram image

In a HEIF file, a stereogram image has two images with the same timestamp. The ogeo brand must be included in the FileTypeBox (ftyp). In GeoHEIF, the images should have the same association to the CRS in the mcrs item property and the two images should have their respective association to a set of tie-points in the tiep item property. The two sets of tie points should define two areas that have a significant overlap.

The fact that it is a stereo image pair is indicated by adding an EntityGroup of type ster with the first entity ID identifying the left image and the second entity ID referencing the right image. HEIF also includes the item properties cmin and cmex for specifying the camera intrinsic and extrinsic parameters that describe the spatial relationship between the two cameras.

2.8.  Temperature image (continuous cell property)

In a HEIF file, a temperature image has an image item in a format that supports floating point numbers. This is currently only possible with the ISO/IEC 23001-17 codec.

The ogeo brand must be included in the FileTypeBox (ftyp). In a GeoHEIF, the image is georectified by associating it to the CRS in the mcrs item property and by defining the affine transformation matrix in the mtxf item property. The description of the temperature cell property (or any other continuous cell property) should be added as an association to a cell property type property (pcel). The definition URI should point to the definition of temperature such as: https://qudt.org/vocab/quantitykind/Temperature. The unit, should point to the units of measure of the temperature such as https://qudt.org/vocab/unit/DEG_C and the unitLang such as https://qudt.org/vocab/unit/.

2.9.  Land cover classification image (categorical cell property)

For a land cover land classification image, a HEIF file has an image item, typically with unsigned short integer values (uint8). While in theory using video codecs for this image type should be possible, using the ISO/IEC 23001-17 codec is advised because this scheme by definition is lossless. Even though ISO/IEC 23001-17 is often called the ‘uncompressed’ codec, it in fact supports lossless compression using the ‘deflate’ or ‘brotli’ algorithms. Brotli compression is a lossless data compression algorithm that reduces the size of files such as HTML, CSS, and JavaScript. It is primarily used to make web traffic to make it faster and more efficient to transmit information over the internet. Some video codecs that provide lossless compression can be used. However, great care must be taken to set the parameters correctly. This is because these codecs are not primarily designed for lossless operation and this part is often overlooked.

The ogeo brand must be included in the FileTypeBox (ftyp)…​ In a GeoHEIF file, the description of the categorical cell property should be added as an association to a cell property type property (pcel) (e.g., “Land cover”); the definition URI should point to the definition of land cover such as: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:Land_cover. Each possible land cover class should be provided as an association to a cell property category property (pcat) where the list of possible land cover classes are listed (e.g., “forest”, “farm”, “water”…​).

2.10.  Weather variables images (surface temperature, relative humidity…​)

Images such as surface temperature are a typical case of a continuous cell properties multiband image. As with the multiband case above, the HEIF file either has several image items: One for each weather variable typically in a format that supports floating point numbers (i.e., ISO/IEC 23001-17), or it contains a single ISO/IEC 23001-17 with separate components for the physical parameters. The ogeo brand must be included in the FileTypeBox (ftyp). In a GeoHEIF file, all images are georectified by associating them to the same CRS in the mcrs item property and to the same affine transformation matrix in the mtxf item property. The description of the continuous cell properties is different for each cell property.

2.11.  Weather forecast

The Global Forecast System (GFS) is a global numerical weather prediction system containing a global computer model and variational analysis run by the United States’ National Weather Service (NWS). The model, run four times a day, generates data for dozens of atmospheric and land-soil variables, including temperatures, winds, precipitation, soil moisture, and atmospheric ozone concentration. The system couples four separate models (atmosphere, ocean model, land/soil model, and sea ice) that work together to accurately depict weather conditions. This is an example of the data that comes out of numerical forecasting: https://www.nco.ncep.noaa.gov/pmb/products/gfs/gfs.t00z.pgrb2.0p25.f003.shtml

Numerical forecasts utilize two kinds of times:

2.12.  Atmosphere characterization (height)

This is a typical case of a 3D datacube. We characterize the atmosphere by measuring parameters such as water vapor content, temperature…​ at several heights. The HEIF file has several image items, at least one for each of the N height values in the height dimension. Each image item is a multiband image, each component containing an atmospheric parameter (water vapor content, temperature…​).

If the 3D datacube need to be tiled for performance reasons, there is a tili image (defined in “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B”) with one extra dimension for the height. Each tile is an ISO/IEC 23001-17 image with one component for each atmospheric parameter. Alternatively, one can use one separate tili image for each atmospheric parameter and use images with just one component.

The ogeo brand must be included in the FileTypeBox (ftyp). All images are georectified by associating them to the same CRS in the mcrs item property and to the same affine transformation matrix in the mtxf item property. The third dimension, height, is defined as an extra-dimension in a ExtraDimensionProperty (edim) and the values ​​that this extra-dimension takes are defined in ExtraDimensionValueProperty (edvl). All images are associated to the same cell property type property (pcel), that contains the description of the array of the atmosphere parameters measured.

2.13.  Atmosphere evolution (height + time)

Atmospheric characterization is a typical 4D datacube use case. The atmosphere is characterized by measuring parameters such as water vapor content and temperature at several heights and several times during a month.

The HEIF file has several image items, at least one for each of the N height values multiply by the M time values, resulting in at least N*M images. Each image item is a multiband image, each component containing an atmospheric parameter (water vapor content, temperature…​).

If the 4D datacube need to be tiled for performance reasons, the HEIF file has a tiled tili image (defined in “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B”) with two extra dimensions. One extra dimension is the height and the other extra dimension the time. As above, the representation can also be turned inside-out by storing multiple tili 4D datacubes, one for each parameter.

The ogeo brand must be included in the FileTypeBox (ftyp). All images are georectified by associating the images to the same CRS in the mcrs item property and to the same affine transformation matrix in the mtxf item property. In this case, the file has two extra-dimensions defined in edim: height and time and two sets of values, one for each extra-dimension defined in edvl. All images are associated with the same cell property type property (pcel), that contains the description of the array of the atmosphere parameters measured.

2.14.  Weather forecast runs (modeling ensemble)

The ensemble case requires a complex datacube with several extra-dimensions. In an ensemble weather forecast, several models are used to issue the weather prediction. Then there is the time when the calculations of the model are done (this created a time series) and the anticipation when the forecast predictions apply (another time series). This generated a 5D datacube with three additional dimensions: model, time and anticipation. The ‘model’ dimension is a categorical dimension (in this case the definition of the dimension values is included in edvl item property), the ‘time’ dimension is continuous, and the ‘anticipation’ dimension is an ordinal dimension.

The HEIF file would have several image items, at least one for each combination of model, time and anticipation, resulting at least N*M*O images (where N is the number of models, M the number of times and O is the number of anticipations). Each image item is a multiband image, each component containing a predicted parameter (temperature, precipitation, wind direction, wind speed, cloud cover, etc.).

If the datacube need to be tiled for performance reasons, the HEIF file has a tiled tili image (defined in “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B”) with several extra dimensions: one for height, time and anticipation.

In this case it is important to define the workflow by including the metadata and provenance information as a metadata item. To relate each image with its corresponding metadata an Item Reference Box (iref) is needed.

 

Figure 1 — Still image and its metadata related by ItemReferenceBox. Picture taken from NGA 0076 v0.6 and simplified to show only the accessory metadata

2.15.  Crop yield prediction as a function of level of fertilizers and species and plant variety (modeling outputs depending on several input parameter combination)

The results of a model predicting crop yield depend on the initial parameters such as fertilizer quantity, irrigation intensity, and type of crop. Depending on the sophistication of the model, the execution time could be too long for an easy browsing application that explores the effects of adjusting the parameter values. A possible solution to this problem is to execute the model for all possible combinations of parameters. This way, model parameters become extra-dimensions in a datacube where cell property values are crop yields inside the cell. The practical implementation of this case is done in the same way as described in the weather forecast run. Again, continuous extra dimensions (fertilizer quantity, irrigation intensity, etc.) and categorical dimensions (type of crop, type of fertilizer, etc.) can be used.

If the datacube need to be tiled for performance reasons, the HEIF file has a tiled tili image (defined in “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B”) with the initial parameters as extra dimensions. If there is no simulation run for a specific combination of parameters, the output image (tile) can be marked as ‘undefined’.

2.16.  Browsing (pan, zoom, clip…​) through high resolution and large extent image in the Web or in the Cloud

A classical HTTP interaction requesting a resource results in downloading the whole resource. Traditionally, obtaining a part of a results such as a part of large image was implemented by offering imagery through a web service on top of HTTP server. The web service extracts a requested subset of original information hiding the complexities of the data structure. An HTTP Range request offers a different alternative, sending parts of a resource back to a client. In practice, it is possible to read parts of a resource in a different order than the original storage, allowing for a random access. As such, a client can random access the file in the same way as can be done on local drives. In this approach, the details of an optimized internal data structure of the file formats become visible and known to the client.

Cloud-optimized formats are a set of data storage and access approaches that enable more efficient handling of large geospatial datasets for visualization and analysis in cloud environments. The cloud optimized format may or may not be random accessible. However, the typical implementation may use HTTP range request. The key benefits of these formats include:

A cloud-optimized format provides a data structure for imagery that enables clients to make HTTP range requests (HTTP requests to ranges of bytes) to the server to fetch desired subregions and sub-resolutions on the fly.

There are two common data structure approaches: chunks and tiles. Chunks divide the file into a linear sequence of fragments of the same size without taking the geometric characteristic into account. Tiles divide the file into regular geometric shapes consisting of a single connected “piece” (topological disc) without “holes” or “lines” (commonly 2D) (definition adapted from OGC Abstract Spec on Tiling of 2D Euclidean space, OGC 19-014r3). Tiles are usually complemented by overviews (a lower resolution version of the same data, also divided into tiles of the same size).

The screen visualization use case can be considered a particular one, as the amount of data required to fill the screen is lower than a typical size of a GIS or remote sensing image. In this case, there are two extreme cases:

In both extremes, the performance is conditioned by the ability to select the right ranges of bytes from the original image. Since the tiled image pyramid is pre-generated, the amount of data transmitted will depend on the size of the tiles used in the original tiling, and how closes the sub-resolutions in the original file are from the actual resolution required by the visualization. The amount of data transmitted will also depend on the efficiency of the compression of the original tiles.

 

Figure 2 — Extracting the needed amount in 3 resolutions. (The red square represents the actual needed data for the screen, and the green square represent the minimum data to be transferred from the server)

A user viewing the image may do frequent pan and zoom operations. When a strategy for caching previous responses is implemented, the performance will also be conditioned by how much of the already transmitted data can be recycled, thereby reducing the amount of new information transmitted. This is conditioned by the original structure of the tiling and sub-resolutions.

The HEIF format provides a tiling mechanism using the grid image format (ISO 14496-12:2022 Section 6.6.2.3), that saves a tiled image as a collection of small images. Each tile is stored in the HEIF file as a separate image item, which are then combined into a large grid image. This format has some limitations: The number of tiles in each dimension (i, j) cannot exceed 256. In addition, every source image used as part of the grid derived image must be referenced to the grid image. Further, the number of references is limited to 65535, so at most one dimension can be 256. Assuming typical tile sizing (e.g., 256 pixels wide and 256 pixels high), this limitation is likely to be encountered in the geospatial domain. For example, when representing a long strip from a line-scan sensor, or a mosaic of a composite image. Note that this limitation does not occur in TIFF, or in the derived Cloud Optimized GeoTIFF (COG) format.

In “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B” a solution to support larger images that extends the HEIF format and provides a tili image type was proposed as an alternative to grid.

Also, the ISO/IEC 23001-17 lossless image format has built-in support for efficient tiling and can be used when lossless data or non-integer datatypes are required.

Also, as part of Testbed 20, Silvereye Technology evaluated options for Coverage selection from HEIF files related to this use case described in Annex B.

Testing if the tile approach is a good strategy is not easy as it involves several HTTP range requests and a good implementation of a caching or buffer mechanism. Some details on how to conduct benchmarking on tiled images are explain in Annex C.1 and Annex D.

2.17.  Confidential or private image (metadata)

During Testbed 20 no item property was proposed to implement the security information for an image.

The GIMI profile, NGA 0076 v0.6, specifies a way to label the security content based on the use of an Information Security Markings (ISM.XML) document and UUID’s, ISM.XML providess a way to label each item.

 

Figure 3 — Media Configuration and ISM labeling for the XML code example. Picture taken from NGA 0076 v0.6

 

<?xml-model href="../ISM/Schematron/ISM/ISM_XML.sch" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<GIMISecurity xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xmlns="urn:us:mil:nga:stnd:0076:ism"
              xmlns:ism="urn:us:gov:ic:ism"
              xmlns:gimi="urn:us:mil:nga:stnd:0076:ism"
              xmlns:arh="urn:us:gov:ic:arh"
              xsi:schemaLocation="urn:us:mil:nga:stnd:0076:ism Sample_imagery_ism.xsd"
              ism:DESVersion="202405"
              ism:ISMCATCESVersion="202405"
              gimi:GIMISecVer="1">
  <File>
    <arh:Security ism:compliesWith="USGov USIC"
                  ism:resourceElement="true"
                  ism:createDate="2006-05-04"
                  ism:classification="U"
                  ism:ownerProducer="USA"/>
    <Items>
      <Item gimi:itemID="ee4ea7a9-27a1-522f-9e6d-182618b2609e">
        <gimi:Security ism:classification="U"
                       ism:ownerProducer="USA"/>
      </Item>
      <Item gimi:itemID="e03d1743-5664-586e-916f-9fa731d0315f">
        <gimi:Security ism:classification="U"
                       ism:ownerProducer="USA"/>
      </Item>
    </Items>
    <Tracks>
      <Track gimi:trackID="60f00d81-52c3-5c3c-9258-5f0e07a63076">
        <gimi:Security ism:classification="U"
                       ism:ownerProducer="USA"/>
        <TrackPortions>
          <TrackPortion gimi:trackportionID="d83b7576-4fc0-59e3-9303-9d1a9830c0ed">
            <gimi:Security ism:classification="U"
                           ism:ownerProducer="USA"/>
          </TrackPortion>
          <TrackPortion gimi:trackportionID="2245b213-9d44-5c8f-8124-645df7a65110">
            <gimi:Security ism:classification="U"
                           ism:ownerProducer="USA"/>
          </TrackPortion>
        </TrackPortions>
      </Track>
      <Track gimi:trackID="8b40ebf8-c278-5d7a-88c6-95aa89b61deb">
        <gimi:Security ism:classification="U"
                       ism:ownerProducer="USA"/>
      </Track>
    </Tracks>
  </File>
</GIMISecurity>

Listing 1 — Example of ISM.XML Instance File taken from NGA 0076 v0.6

2.18.  Sparse images

Images that cover a very large spatial area may only contain small areas of interest. For example, imagery of oceans may show mostly water without objects such as small islands, ships, or sea mammals. Providing high level images of the large area with detailed imagery only where an object has been detected may be useful.

The sparse images case can be addressed using the tili image item described in the “OGC Testbed 20: GIMI Lessons Learned and Best Practices Report, Annex B”. This supports the implementation of tile space with empty tiles. In this case an empty tile is a tile that is enumerated in the TiledImageOffsetTable with a tile_start_offset 0 indicating the tile content is not encoded. All tile images should be compressed to optimize the space.

 

Figure 4 — File structure of a tili image item. Picture taken from OGC 24-040

2.19.  Planetary image (extraterrestrial CRS)

For images referenced to a planetary reference system, the mcrs item property provides ways to reference or describe the CRS. In the GeoHEIF format, the CRS is not only a number (such as in the case of GeoTIFF) but a UTF8 text preceded by a numeric crsEncoding. It is likely that relevant organizations create registries for reference systems on other planets and celestial objects. When there is an URL that links to a registry that provides a definition of the planetary CRS, a 4CC crsu code is used. When there is not CRS URI, but the CRS can be described as a combination of datum, projection and ellipsoid, a wkt2 code is used. Finally, when the image is a picture taken by a sensor, an Engineering CRS may be necessary. In this case, the Engineering CRS is centered on the spacecraft’s camera and the coordinate system class is restricted to Cartesian (3D), Spherical (3D), and Spherical (2D). See more details in in OGC Testbed 19 Extraterrestrial GeoTIFF Engineering Report [OGC 23-028]. Currently the definition of an Engineering CRS is not specified in the GeoHEIF format.

2.20.  Satellite images time series

When a series of images are taken from an earth observation platform as it tracks over the planet, the images potentially form an overlapping mosaic where each image has a different imaging time.

The serie of images can be stored preferably in an image sequence. The respective absolute capture time for each image can be stored as TAI timestamp, which is assigned to each image sample as a Sample Auxiliary Information (SAI). TAI timestamps are defined in ISO/IEC 23001-17 Amendment 1. In the GeoHEIF format, each image can also be georeferenceable or georectified by associating it with a CRS in a mcrs property that is assigned to the track’s sample description box. This CRS is constant for the whole track or at least a sample cluster (time range of the track). The camera position matrix can be stored in a property similar to an affine transformation matrix (mtxf) data packet. The property needs to be defined in futures activities, and that is assigned to each image as sample auxiliary information.

Alternatively, this case can be considered as a 3D datacube where the extra-dimension is the image time. In the GeoHEIF format, each image can be georeferenceable or georectified by associating it to a CRS in the mcrs item property and to a set of tie-points in the tiep item property or to a affine transformation matrix in the mtxf item property respectively, both in the ItemPropertyContainerBox and related to the image item in the ItemPropertyAssociationBox. In this case each image has a different tiep or mtxf item property, and typically all images have the same mcrs. The image time extra-dimension is implemented as ‘edim’ item property and the set of times that this dimension can take is implemented as edvl.

2.21.  Burst Photography

In HEIF, a burst of images can be more efficiently coded in an image sequences. HEIF provides additional constraints that enable fast random access to these images. For example, predictive coding can be restricted to have just one reference frame. Thus, each frame in the burst can be extracted by decoding at most two images (the reference image and the desired image that uses the reference image for prediction, resulting in a composed image). It is also possible to store the image burst together with a composed image and relate the image burst and the composed image together. For example, a single file could contain a sequence of exposure-bracketed input images and the resulting HDR image.

2.22.  Videos from a camera with a fixed position

The video with a fixed position use case could be a surveillance camera, a fixed camera used for traffic monitoring by taking images or video streams, or satellite images or orthophoto time series.

In a GeoHEIF file, these images can be encoded as image sequences with the camera CRS mcrs and the transformation matrix mtxf stored in the sample description box of the track. Note that, different from the “Satellite images time series” use case, the mtxf is stored in the sample description. This is because the camera is fixed and therefore not necessary to transmit a new transformation matrix for each frame.

In a HEIF file, these images can be encoded as image sequences using the MovieBox (moov) with a TrackBox (trak). In the TrackBox, the MediaBox (mdia) includes a MediaInformationBox (media) with timescale and the duration of the sequence and other boxes such as the SampleTableBox (stbl) and the SampleDescriptionBox (stsd) where there is a description of what type of samples are in the images, which codec is used. There are also other boxes, such as DecodingTimeToSampleBox (stts) which provides the duration of each frame, or the SampleSizeBox (stsz) which provides the size of each frame.

2.23.  Videos from a moving camera

This videos from a moving camera use case could be images or video tracks from a camera installed on a helicopter for traffic monitoring, or images or video tracks from a drone.

Similar to the Satellite images timeseries, the camera position and pose can be stored with a fixed CRS mcrs in the sample description box and the transformation matrix mtxf as sample auxiliary information attached to each frame. If the high-precision TAI timestamps are not needed, they can be omitted and the sample timing of the ISOBMFF format can be used, which is suitable for fixed frame-rate video cameras.

In a HEIF file, if additional metadata is required, this can be added as a separate metadata track. The content of the data is application specific and can include various sensor data (like acceleration sensors or temperature). This results in two track boxes (trak): a track for the video and a track for the metadata. The metadata track includes a reference to a video track using a cdsc (Content Description) reference to denote that the metadata track is further describing the referenced visual track. This allows metadata to be synchronized with the video (or visual) track. Both tracks can include a DecodingTimeToSampleBox (stts) that provides the duration of each frame. The duration can be different for the metadata frames and for the visual frames.

2.24.  Videos from multiple cameras

The videos from multiple cameras use case covers videos when the same event is being tracked from multiple cameras or when it is desired to create a 3D movie with stereographic cameras.

A HEIF file can contain multiple video tracks, one for each camera’s samples. Even though the video is stored in separate tracks, they can use the same time base, such that the camera images stay synchronized. This can be achieved even if the cameras run at different framerates.

The camera positions can be stored in a way similar to the CRS and the affine transformation matrix is stored for the still images, using a mcrs and mtxf as described in previous use cases (and defined in GeoHEIF), or using the cmin, cmex boxes (camera internal matrix, camera external matrix) defined in the HEIF standard. The latter is specifically designed for stereoscopic cameras, but can also be used in other use cases when only the relative pose of the cameras in relation to each other is important.

2.25.  Videos with a depth sensor

A depth image, also known as a depth map or range image, is a visual representation where each pixel encodes the distance from a sensor or camera to points in a scene. Unlike regular color images, the value at each pixel in a depth image corresponds to the relative or absolute depth (distance) of the scene’s surfaces from a specific viewpoint. The depth images usually are accompanied with “depth representation information” (stored, for example, as H.265 SEI messages). This information defines the near and far plane of the depth data and whether the distances are sampled linearly or stored as the disparity between left and right image.

A depth image can be obtained directly by special cameras with a time-of-flight sensor or by constructing it with an algorithm that use color images from stereo camera pair. These depth images can be stored in a separate video track. The image resolution of this depth image can (and often will) be different from the main color image track. The depth track should use a track reference to the color track to indicate that it contains auxiliary depth data instead of visual content.

As depth images have different characteristics than color images, typical compression artifacts that are acceptable for color images can be undesired for depth images. This should be considered when choosing the right video codec for the depth data. For example, there is a special coding mode for depth images in H.265, but it is not widely supported. Another choice is to use a lossless codec or a codec where the maximum error can be controlled (e.g., JPEG2000).

2.26.  Videos with reconstructed 3D point cloud

Some cameras reconstruct a sparse 3D point cloud that accompanies the 2D color image. Since the format of this 3D point cloud varies, storing this as custom sample auxiliary information, attached to each image is recommended. There is no standardized four-character code to identify these yet and a proprietary code has to be used.