4.5.1.  Source Code

In a Reproducible FAIR Workflow scenario, ensuring that the code behind the application is uniquely identified and traceable is necessary. Even if software configuration management systems have become a common tool for tracking and controlling changes in the software, the FAIR publication of applications also needs methods for software citation, software retrieval, and long-term preservation. This is achieved with the issuing of a persistent identifier.

The persistent identifier provides a long-lasting reference to source code and is usually understood as an actionable and accessible reference over the Internet that retrieves the necessary files. As such, the persistent component is dependent on the service commitment to resolve the identifier and the dedicated storage lifespan.

Several initiatives provide this persistent dual functionality of identification and archivation. This Best Practice identifies one solution with Software Heritage. Software Heritage is a non-profit multi-stakeholder initiative unveiled in 2016 by the French Institute for Research in Computer Science and Automation (Inria) and supported by UNESCO. The mission of Software Heritage is to collect, preserve, and share all software that is publicly available in source code form, with the goal of building a common shared infrastructure at the service of industry, research, culture, and society.

Software Heritage provides SoftWare Heritage persistent IDentifiers (SWHIDs) and ensures preservation of the software source code by crawling code hosting platforms, like GitHub, GitLab.com, or Bitbucket, and package archives, like npm or PyPI, ingested into a special data structure, a Merkle DAG, that is the core of the archive.

The use of Merkle DAG is a generalization of a hash list and a hash chain similar to a Merkle tree (i.e. a tree of hashes) but while a Merkle tree connects transactions by sequence, the Merkle-DAG connects transactions by hashes. In a Merkle-DAG, addresses are represented by a Merkle hash. The use of Merkle DAGs allows efficient and secure verification of the contents of a large data structure and verifies any kind of data stored, handled, and transferred in and between computers.

To obtain a SWHID, the code must be hosted on a publicly accessible repository (Github, Bitbucket, a GitLab instance, an institutional software forge, etc.) using one of the version control systems supported by Software Heritage (Subversion, Mercurial, and Git).

The top level of the source code tree most contain a linked data metadata file with the CodeMeta vocabulary (i.e., codemeta.json). This file helps indexing the source code in the Software Heritage archive and provides an easy way to link to other related research outputs.

CodeMeta is a minimal metadata schema for science software and code with the goal to create a concept vocabulary that can be used to standardize the exchange of software metadata across repositories and organizations. With the CodeMeta schema it is possible to map between the different services (GitHub, figshare, or Zenodo) and compare software metadata used across multiple repositories. The software metadata concepts are arranged into a JSON-LD context for serialization.

CodeMeta developed the translations from the different vocabularies. The CodeMeta vocabulary is an extension of the SoftwareApplication and SoftwareSourceCode classes found in the vocabulary of the Schema.org initiative [schema]. Metadata information conformant to the CodeMeta vocabulary can be represented in JSON format, named codemeta.json, like the example below with the information for Water Bodies Detection application.

{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "license": "https://spdx.org/licenses/CC-BY-NC-SA-4.0",
    "dateCreated": "2022-09-01",
    "datePublished": "2022-09-25",
    "dateModified": "2022-09-25",
    "name": "Water Bodies Detection",
    "version": "1.0.0",
    "description": "The Water Bodies Detection is an Application that uses the NDWI index and the Otsu threshold to detect water bodies using Sentinel-2 data",
    "developmentStatus": "concept",
    "funder": {
        "@type": "Organization",
        "name": "OGC Testbed 18"
    },
    "keywords": [
        "NDWI"
    ],
    "programmingLanguage": [
        "Python",
           ],
    "softwareRequirements": [
        "container runtime"
    ],
    "author": [
        {
            "@type": "Person",
            "givenName": "Jane",
            "familyName": "Doe",
            "email": "jane.doe@acme.earth"
        },
        {
            "@type": "Person",
            "givenName": "John",
            "familyName": "Doe",
            "email": "john.doe@acme.earth"
        }
    ]
}

For convenience and readability, the following files also need to be present on the top level of the source tree.

• README containing a description of the software (name, purpose, pointers to website, documentation, development platform, contact, and support information, etc.).

• AUTHORS, a list of all the persons credited for the software.

• LICENSE, the project license terms.

The following image shows the contents of a repository with the needed application elements for it to be uniquely identified and traceable with Software Heritage.

Figure 3 — Gitlab repository for the Water Bodies Detection application with the needed application elements for it to be uniquely identified and traceable by Software Heritage.

Once the code repository is properly prepared it can be submitted to Software Heritage to be archived with its full development history. If the repository is hosted on one of the major known forges the process takes just a few hours. The submission process can be done through the Software Heritage web site or by requesting the archival programmatically using the dedicated Software Heritage API entry point.

After submitting the software repository for archival at the Software Heritage it obtains the following identifier:

and the permanent link:

https://archive.softwareheritage.org/swh:1:dir:afaf7ddf712f8f3f0c66bd16de495d31a67106b9; origin=https://gitlab.com/terradue-ogc/testbed-18/water-bodies-detection.git; visit=swh:1:snp:79dd9bcb0327d0cd75abceccbb7bccbe51e889a3; anchor=swh:1:rev:b69b47d3dc663031cee7793813e01a2db6f582e7

Figure 4 — Application Package on Software Heritage

4.5.2.  Container

Together with the source code, a Reproducible FAIR Workflow scenario also requires that the application container(s) is(are) uniquely identified and traceable. Initially thought of as a straightforward responsibility of the container registry, it became clear that the common practices in well-known container formats like Docker do not guarantee the unique identification and traceability of the container.

For example, when building container images the developer tags the image with a specific name such as:

docker build -t docker.terradue.com/crop:1.1.6 .

And, when performing a Docker pull that tag is used to retrieve the required container. However, if by mistake the developer or continuous integration tool builds another container image with the same tag but with changed content, the reproducibility cycle would be broken.

To address this issue a deterministic approach is required using the sha256 signature of the container which will act as an immutable identifier. To obtain this with Docker would require execution of the following command:

docker inspect docker.terradue.com/crop:1.1.6 | jq '.[0]["RepoDigests"][0]'

This returns a Docker identifier guaranteed to be unique.

docker.terradue.com/crop@sha256:ec2d8e71ab5834cb9db01c5001bde9c3d6038d0418ad085726b051b4359750e1

This approach ensures a deterministic identification of a container image and build instance. In fact, two separate builds from the same Docker file result in differing hash values. === Package

As seen in the previous section, developers build a container image with the application’s command line tool(s) and respective runtime environments and then publish the container image on a repository.

Subsequently, following the OGC Best Practices for EO Application Packages (OGC 20-089), the developers create an Application Package encoded in a Common Workflow Language (CWL) document to describe the data processing applications, provide information about the parameters, software items, executables, dependencies, and metadata. Aside from the role of the CWL in packaging existing applications, the Application Package can orchestrate the execution of an application of several pre-packaged processing steps. In this scenario the Application Package combines and/or chains multiple processing services, potentially offered by different developers/organizations, in parallel or sequentially.

The Application Package documents the orchestration steps to invoke the command line tool(s) included in the image(s). This document ensures that the application is fully portable among all supporting processing scenarios and supports automatic deployment according to the OGC API - Processes (OGC 18-062).

The OGC Best Practices for EO Application Packages (OGC 20-089) recommends that to enrich the application package with new concepts, these should originate from schema.org and be linked with their RDF encoding. While OGC 20-089 only enforces the version element as a mandatory, it already suggest several additional elements that are key for the Reproducible FAIR Workflows scenario such as the following.

• author: The main author of the Application Package - https://schema.org/author.

• citation: A citation or reference to a publication, web page, scholarly article, etc. https://schema.org/citation.

• codeRepository: Link to the repository where the Application code is located (e.g., SVN, github). https://schema.org/codeRepository.

• contributor: A secondary contributor to the Application Package https://schema.org/contributor.

• dateCreated: The date on which the Application Package was created. https://schema.org/dateCreated.

• keywords: Keywords used to describe this application. Multiple entries in a keywords list are delimited by commas. https://schema.org/keywords.

• license: An URL to the license document that applies to this application. https://schema.org/license.

• releaseNotes: Description of what changed in this version. https://schema.org/releaseNotes.

All these elements can be obtained from the CodeMeta vocabulary used in the previous section to characterize the Application (i.e., codemeta.json) and can be directly added when defining the Application Package.

In the Reproducible FAIR Workflow scenario, it is necessary to ensure that the Application Package is uniquely identified and traceable. As such, this ER recommends the assignment of a DOI to the application package and extends the metadata section of the Application Package CWL with:

• sameAs: URL of a reference Web page that unambiguously indicates the item’s identity. https://schema.org/sameAs.

For the previous example of the Water Bodies Detection application, the metadata section of the Application Package encoded in CWL would become:

cwlVersion: v1.0
$graph:
...
$namespaces:
  s: https://schema.org/
s:author:
- class: s:Person
  s:affiliation: Planet Earth
  s:email: john.doe@somedomain.org
  s:name: Doe, John
s:contributor:
- class: s:Person
  s:affiliation: Planet Earth
  s:email: jane.doe@somedomain.org
  s:name: Doe, Jane
s:sameas: https://doi.org/10.5072/zenodo.1107209
s:softwareVersion: 1.1.6
schemas:
- http://schema.org/version/9.0/schemaorg-current-http.rdf

4.6.1.  Input EO Data

As defined in OGC 20-089, the Application can take as input an array of Spatio-Temporal Asset Catalog (STAC) Items URLs aggregated into a STAC Collection describing input EO data. The STAC Collection Specification defines a set of common fields to describe a group of Items that share properties and metadata. This definition fits the purposes of aggregating the STAC Items into a single element that can potentially be cited and assigned a digital identifier.

To comply with the Reproducible FAIR Workflow scenario, the STAC collection defining the input EO data must use the STAC Scientific Citation Extension [STAC Citation] to indicate from which publication the data originates and how the data itself should be cited or referenced. The referencing capability of the Scientific Citation Extension is a key element to meet the goals of reproducibility and citation of an application package execution. With the assignment of a DOI to the STAC collection that aggregates the STAC Items used as inputs, the input dataset becomes citable and findable.

{
  "type": "Collection",
  "id": "collection-id",
  "stac_version": "1.0.0",
  "description": "this is the collection description",
  "links": [
    {
      "rel": "cite-as",
      "href": "https://doi.org/10.5072/zenodo.1102087"
    },
    {
      "rel": "self",
      "href": "https://sandbox.zenodo.org/api/files/f1cb1db1-1ece-4c1f-858c-d753e99b36d1/collection.json",
      "type": "application/json"
    }
 ...

In the case of an input EO Data from a datacube, the STAC Datacube Extension specifies the datacube related metadata and enriches the input dataset STAC Collection with additional information on the temporal, spatial, and radiometric dimensions. The temporal enrichment provides the list of acquisition dates, the spatial provides the extent of x and y dimensions, and finally the radiometry provides clear and detailed information about the radiometric characteristics of the input data.

{
  "type": "Collection",
  "id": "collection-id",
  "stac_version": "1.0.0",
  "description": "this is the collection description",
  "links": [
    {
      "rel": "root",
      "href": "https://sandbox.zenodo.org/api/files/f1cb1db1-1ece-4c1f-858c-d753e99b36d1/collection.json",
      "type": "application/json"
    },
    {
      "rel": "item",
      "href": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_10TFK_20210713_0_L2A",
      "type": "application/json"
    },
    {
      "rel": "item",
      "href": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_10TFK_20220524_0_L2A",
      "type": "application/json"
    },
    {
      "rel": "cite-as",
      "href": "https://doi.org/10.5072/zenodo.1102087"
    },
    {
      "rel": "self",
      "href": "https://sandbox.zenodo.org/api/files/f1cb1db1-1ece-4c1f-858c-d753e99b36d1/collection.json",
      "type": "application/json"
    }
  ],
  "stac_extensions": [
    "https://stac-extensions.github.io/scientific/v1.0.0/schema.json",
    "https://stac-extensions.github.io/datacube/v2.0.0/schema.json"
  ],
  "cube:dimensions": {
    "x": {
      "axis": "x",
      "extent": [
        -121.8343226741975,
        -120.51956282559038
      ],
      "reference_system": 3857
    },
    "y": {
      "axis": "y",
      "extent": [
        39.63588071728383,
        40.64479052153662
      ],
      "reference_system": 3857
    },
    "time": {
      "extent": [
        "2021-07-13T19:03:24Z",
        "2022-05-24T19:03:29Z"
      ],
      "values": [
        "2021-07-13T19:03:24Z",
        "2022-05-24T19:03:29Z"
      ]
    },
    "spectral": {
      "type": "bands",
      "values": [
        "B1",  "B2", "B3",  "B4",
        "B5",  "B6", "B7",  "B8",
        "B8A", "B9", "B10", "B11",
        "B12", "QA10", "QA20", "QA60"
      ]
    }
  },
  "sci:doi": "10.5072/zenodo.1102087",
  "extent": {
    "spatial": {
      "bbox": [
        [
          -121.8343226741975,
          39.63588071728383,
          -120.51956282559038,
          40.64479052153662
        ]
      ]
    },
    "temporal": {
      "interval": [
        [
          "2021-07-13T19:03:24Z",
          "2022-05-24T19:03:29Z"
        ]
      ]
    }
  },
  "license": "proprietary"
}

4.6.2.  Output EO Data

As defined in OGC 20-089, the output dataset as generated by the application package execution is a STAC Catalog with links to STAC Items representing and describing the resulting files.

Similarly, to the input EO data, to comply with the Reproducible FAIR Workflow scenario the output EO data must be defined as a STAC Collection with the Scientific Citation Extension to convey the DOI. Additionally, and for extra traceability and reproducibility validation, the output STAC Collection must use the STAC File Extension to provide the file MD5 for each asset.

In the case of an output EO Data that publishes to a GeoDataCube, the STAC Datacube Extension specifies the GeoDataCube related metadata and enriches the output dataset STAC Collection with additional information on the temporal, spatial, and radiometric dimensions. The temporal enrichment provides the list of acquisition dates, the spatial provides the extent of x and y dimensions, and finally the radiometry provides clear and detailed information about the radiometric characteristics of the output data.

{
  "type": "Collection",
  "id": "collection-id",
  "stac_version": "1.0.0",
  "description": "this is the collection description",
  "links": [
    {
      "rel": "root",
      "href": "https://sandbox.zenodo.org/api/files/00ba172e-c9f0-4552-91e1-342a63cf3477/collection.json",
      "type": "application/json"
    },
    {
      "rel": "item",
      "href": "https://sandbox.zenodo.org/api/files/00ba172e-c9f0-4552-91e1-342a63cf3477/S2B_10TFK_20210713_0_L2A/S2B_10TFK_20210713_0_L2A.json",
      "type": "application/json"
    },
    {
      "rel": "item",
      "href": "https://sandbox.zenodo.org/api/files/00ba172e-c9f0-4552-91e1-342a63cf3477/S2A_10TFK_20220524_0_L2A/S2A_10TFK_20220524_0_L2A.json",
      "type": "application/json"
    },
    {
      "rel": "cite-as",
      "href": "https://doi.org/10.5072/zenodo.1102089"
    },
    {
      "rel": "self",
      "href": "https://sandbox.zenodo.org/api/files/00ba172e-c9f0-4552-91e1-342a63cf3477/collection.json",
      "type": "application/json"
    }
  ],
  "stac_extensions": [
    "https://stac-extensions.github.io/scientific/v1.0.0/schema.json",
    "https://stac-extensions.github.io/datacube/v2.0.0/schema.json"
  ],
  "cube:dimensions": {
    "x": {
      "axis": "x",
      "extent": [
        -121.413752588606,
        -120.71922542174708
      ],
      "reference_system": 3857
    },
    "y": {
      "axis": "y",
      "extent": [
        39.83402935827303,
        40.47202226335379
      ],
      "reference_system": 4326
    },
    "time": {
      "extent": [
        "2021-07-13T19:03:24Z",
        "2022-05-24T19:03:29Z"
      ],
      "values": [
        "2021-07-13T19:03:24Z",
        "2022-05-24T19:03:29Z"
      ]
    },
    "spectral": {
      "type": "bands",
      "values": [
        "B1", "B2", "B3", "B4",
        "B5", "B6", "B7", "B8",
        "B8A", "B9", "B10", "B11",
        "B12", "QA10", "QA20", "QA60"
      ]
    }
  },
  "sci:doi": "10.5072/zenodo.1102089",
  "extent": {
    "spatial": {
      "bbox": [
        [
          -121.413752588606,
          39.83402935827303,
          -120.71922542174708,
          40.47202226335379
        ]
      ]
    },
    "temporal": {
      "interval": [
        [
          "2021-07-13T19:03:24Z",
          "2022-05-24T19:03:29Z"
        ]
      ]
    }
  },
  "license": "proprietary"
}