B.1.1.  Cloud Optimized GeoTIFF (COG)

COG products are pixel-interleaved GeoTIFF files with internal tiles that can be fetched individually. The files may include multiple reduced resolution pages for efficient viewing, though these are not relevant for full-resolution processing tasks. Key aspects of COG benchmarking include:

B.1.2.  GeoZarr

GeoZarr is a format that stores geospatial data in chunked, compressed arrays. GeoZarr is designed for efficient cloud storage and access, with each chunk stored as a separate file. This supports better partitioning and parallel access, which can be advantageous in HPC environments.

B.1.3.  GeoParquet

GeoParquet[36] is an extension of the Apache Parquet format[1], optimized for geospatial data. The GeoParquet format supports efficient columnar storage, which can be beneficial for certain types of geospatial queries and analyses. Some benchmarking opportunities are:

B.2.1.  COG

The following are some specific benchmarking tests for COG in HPC:

C.2.1.  Data Collection

C.2.2.  Evaluation Criteria

C.2.3.  Tools and Techniques

C.2.4.  Analysis Methods

C.3.1.  Workflows handling COG files

This section describes the setup of the High-Performance Computing (HPC) environment for the evaluation of COG files. This environment is relevant for other cloud-optimized geospatial data formats covered by this report.

To encapsulate the workflow, the Testbed participants used the Common Workflow Language (CWL), and to run on HPC they relied on the Toil CWL runner [5]. The CWL is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments. Toil is a Python-based workflow engine that can execute CWL workflows on a variety of execution environments, including HPC clusters and especially SLURM [16] [14] [15] which is the available scheduler on the HPC the Testbed used for these experiments.

The Toil CWL runner [5] [17] was installed on the HPC environment provided by I-GUIDE for this investigation.

 

python3 -m virtualenv ~/venv
source ~/venv/bin/activate
pip install "toil[cwl]"

Listing C.1 — Install Toil CWL runner

Since GDAL[18] supports COG, GDAL was used to read and write COG files. The GDAL library is a widely used open-source library for reading and writing geospatial data formats.

The code snippet below shows the use of gdal_translate in a CWL workflow that reads a COG file, performs a geographic subset, and writes the result to a new COG file.

The CWL file defines the command-line tool gdal_translate with input parameters for the bounding box, EPSG code, and asset URL. The tool uses inline JavaScript expressions to construct the command-line arguments dynamically based on the input parameters.

 

class: CommandLineTool

requirements:
  InlineJavascriptRequirement: {}
  DockerRequirement:
    dockerPull: osgeo/gdal:alpine-normal-3.6.2

baseCommand: gdal_translate
arguments:
- -projwin
- valueFrom: ${ return inputs.bbox.split(",")[0]; }
- valueFrom: ${ return inputs.bbox.split(",")[3]; }
- valueFrom: ${ return inputs.bbox.split(",")[2]; }
- valueFrom: ${ return inputs.bbox.split(",")[1]; }
- -projwin_srs
- valueFrom: ${ return inputs.epsg; }
- valueFrom: |
    ${ if (inputs.asset.startsWith("http")) {

         return "/vsicurl/" + inputs.asset;

       } else {

         return inputs.asset;

       }
    }
- valueFrom: ${ return inputs.asset.split("/").slice(-1)[0]; }

inputs:
  compress:
    type: string
  blocksize:
    type: int
  asset:
    type: string
  bbox:
    type: string
  epsg:
    type: string
    default: "EPSG:4326"

outputs:
  tifs:
    outputBinding:
      glob: '*.tif'
    type: File

cwlVersion: v1.0

Listing C.2 — Example CWL file for gdal_translate

The CWL file defines the input parameters for the gdal_translate tool, including the asset URL, bounding box, and EPSG code. To set values to these input parameters, a YAML file is used.

 

bbox: "136.659,-35.96,136.923,-35.791"
asset: https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B8A.tif
epsg: "EPSG:4326"

Listing C.3 — Example YAML file for gdal_translate inputs parameters

Toil CWL runner [5] [17] was used to execute the workflow defined in the CWL file. The runner is configured to use the SLURM batch system [16], disable caching, and run the workflow in a Singularity (renamed Apptainer) container [22] [20] [21] [19].

 

TOIL_CHECK_ENV=True toil-cwl-runner \
        --batchSystem slurm \
        --disableCaching true \
        --workDir . \
        --jobStore ./store_stats \
        --clean=never \
        --singularity \
        example.cwl example-job.yaml

Listing C.4 — Execute gdal_translate through Toil CWL runner

The execution of the workflow through toil-cwl-runner [5] generated the following debug messages on the standard output.

 

[2024-11-01T06:04:04-0400] [MainThread] [I] [cwltool] Resolved 'example.cwl' to 'file:///home/x-gfenoy/TB20-Benchmark/gdal_translate/gdt1/example.cwl'
[2024-11-01T06:04:05-0400] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 new
[2024-11-01T06:04:05-0400] [MainThread] [I] [toil.job] Processing job 'CWLJob' gdal_translate kind-CWLJob/instance-wm_sphxd v0
[2024-11-01T06:04:05-0400] [MainThread] [I] [toil] Running Toil version 5.5.0-b0ff5be051f2fd55352e00450b7848dcf8354a3b on host login07.anvil.rcac.purdue.edu.
[2024-11-01T06:04:05-0400] [MainThread] [I] [toil.leader] Issued job 'CWLJob' gdal_translate kind-CWLJob/instance-wm_sphxd v1 with job batch system ID: 0 and cores: 1, disk: 3.0 Gi, and memory: 2.0 Gi
[2024-11-01T06:04:07-0400] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2024-11-01T06:05:55-0400] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 (0 failures) [01:12<00:00, 0.01 jobs/s]
{
    "tifs": {
        "location": "file:///home/x-gfenoy/TB20-Benchmark/gdal_translate/gdt1/B8A.tif",
        "basename": "B8A.tif",
        "nameroot": "B8A",
        "nameext": ".tif",
        "class": "File",
        "checksum": "sha1$679b939185992198bfd2d0ed15f1178fc661472a",
        "size": 2275155
    }
}[2024-11-01T06:05:56-0400] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/home/x-gfenoy/TB20-Benchmark/gdal_translate/gdt1/store_stats)

Listing C.5 — Output generated by the execution

Toil[17] comes with a stats command that can be used to generate statistics about the execution of the workflow. This command provides information about 4 categories: memory, wait, time, and clock. The following command generates statistics about memory usage.

 

toil stats store_stats --categories memory --human

Listing C.6 — Generate statistics with Toil stats

The stats command generates detailed information about the execution of the workflow.

 

Batch System: slurm
Default Cores: 1  Default Memory: 2.0G
Max Cores: 9.22337e+18
Total Clock: 2s  Total Runtime: 74s
Worker
    Count |                                   Memory
        n |      min     med     ave     max   total
        1 |       0K      0K      0K      0K      0K
Job
 Worker Jobs  |     min    med    ave    max
              |       0      0      0      0
    Count |                                   Memory
        n |      min     med     ave     max   total
        0 |       0K      0K      0K      0K      0K

Listing C.7 — Output generated by the stats command

This example run was not intended to be representative of a real-world testing scenario, but rather to illustrate the process of executing a CWL workflow on an HPC environment using Toil[17].

In the previous example the gdal_translate tool was used to read COG files. The same approach can be used to write COG files with GDAL.

C.3.2.  Toil Workflow Execution Service (WES)

The Toil Workflow Execution Service (WES) is a RESTful API that provides a standardized interface for executing CWL workflows on a variety of execution environments. The WES API allows users to submit, monitor, and retrieve the status of CWL workflows.

To use the WES API, the Toil WES server must be installed and configured on the HPC environment. Toil WES requires a RabbitMQ server [23] [24] that can be started with the following command:

 

apptainer run \
    --bind $(pwd)/etc:/opt/bitnami/rabbitmq/etc/rabbitmq/ \
    --bind $(pwd)/var_lib:/opt/bitnami/rabbitmq/var/lib/rabbitmq \
    --bind $(pwd)/.rabbitmq:/opt/bitnami/rabbitmq/.rabbitmq/ \
    --bind $(pwd)/tmp:/bitnami/rabbitmq/ \
    docker://bitnami/rabbitmq:latest

Listing C.8 — Start the RabbitMQ server

The toil.server.celery_app can be found in the file venv3.11/lib/python3.11/site-packages/toil/server/celery_app.py. If the environment variable TOIL_WES_BROKER_URL is defined, it is the one used to access the RabbitMQ server (per default the user can use user as username and bitnami as password).

 

source ~/venv3.11/bin/activate
TOIL_WES_BROKER_URL="amqp://user:bitnami@127.0.0.1:5672" celery -A toil.server.celery_app worker --loglevel=INFO

Listing C.9 — Start Celery workers

The WES server can be started with the following command:

 

source ~/venv3.11/bin/activate
TOIL_WES_BROKER_URL=amqp://user:bitnami@127.0.0.1:5672//  toil server \
  --opt=--stats \
  --opt=--batchSystem=slurm \
  --opt=--defaultMemory=2Gi \
  --opt=--maxMemory=100Gi \
  --opt=--singularity \
  --opt=--jobStore=/home/x-gfenoy/TOIL/toil_storage/example/job_store/$(uuidgen)

Listing C.10 — Start the Toil WES server

From there the WES API can be accessed through the URL: http://localhost:8080/ga4gh/wes/v1/service-info and provide information about the service.

 

{
  "version": "7.0.0",
  "workflow_type_versions": {
    "py": {
      "workflow_type_version": [
        "3.7",
        "3.8",
        "3.9"
      ]
    },
    "cwl": {
      "workflow_type_version": [
        "v1.0",
        "v1.1",
        "v1.2"
      ]
    },
    "wdl": {
      "workflow_type_version": [
        "draft-2",
        "1.0"
      ]
    }
  },
  "supported_wes_versions": [
    "1.0.0"
  ],
  "supported_filesystem_protocols": [
    "file",
    "http",
    "https"
  ],
  "workflow_engine_versions": {
    "toil": "7.0.0"
  },
  "default_workflow_engine_parameters": [],
  "system_state_counts": {
    "SYSTEM_ERROR": 2
  },
  "tags": {
    "wes_dialect": "standard"
  }
}

Listing C.11 — WES service-info URL

By using the Toil WES API, client applications can submit CWL workflows to the HPC environment for execution and monitor their progress through the Toil WES API.

The client application used for this benchmark was ZOO-Project with support for Deploy, Replace, Undeploy (DRU) and CWL, a reference implementation of the OGC API — Processes — Part 1: Core Standard. ZOO-Project provides a web processing service that can execute CWL workflows in a HPC environment. In the Earth Observation Exploitation Platform Common Architecture (EOEPCA) project, an implementation of the zoo-wes-runner and its associated cookiecutter template was used to execute CWL workflows on the HPC environment using the Toil WES.

To enable access from the ZOO-Project Kubernetes cluster to the Toil WES server, SSH tunneling was used to forward the port 8080 from the HPC environment to port 8100 on the local machine.

 

ssh -L 0.0.0.0:8100:127.0.0.1:8080 username@hpc-hostname

Listing C.12 — SSH tunneling to forward the port 8080

To enable access from the HPC environment to the ZOO-Project Kubernetes cluster, SSH tunneling was used to forward the port 4566 from the local machine to the port 4900 on the HPC environment.

 

ssh -R 127.0.0.1:4900:127.0.0.1:4566 username@hpc-hostname

Listing C.13 — SSH tunneling to forward the port 4566

The ZOO-Project instance can then be configured to use the Toil WES API to execute CWL workflows on the HPC environment.

 

workflow.inputs.WES_URL: "http://<WES-HOSTNAME>>:8100/ga4gh/wes/v1/"
workflow.inputs.WES_USER: "test"
workflow.inputs.WES_PASSWORD: "$$2y$$12$$ci.4U63YX83CwkyUrjqxAucnmi2xXOIlEF6T/KdP9824f1Rf1iyNG"

Listing C.14 — ZOO-Project configuration

To deploy the ZOO-Project-DRU instance with the WES support on a local Kubernetes cluster, the following command can be used:

 

git clone https://github.com/ZOO-Project/ZOO-Project.git
cd ZOO-Project
skaffold dev -p dru-wes

Listing C.15 — Deploy the ZOO-Project-DRU with WES support on a local cluster

The ZOO-Project-DRU instance with WES support can then be accessed through the URL: http://localhost:8080/.

The ZOO-Project-DRU instance provides an API implementation compliant with the OGC API — Processes — Part 1: Core Standard, that can execute CWL workflows on the HPC environment using the Toil WES API. In addition to the execution capability, the ZOO-Project-DRU provides also an API implementing the OGC API — Processes: Part 2: Deploy, Replace, Undeploy candidate Standard to manage the deployment of CWL workflows.

C.3.3.  Vulnerability Analysis for Aging Dams Workflow

The workflow to be supported is available here: https://github.com/I-GUIDE/vulnerability_analysis_agingdams and was proposed by the I-GUIDE project.

The workflow is available as a Jupyter notebook: https://github.com/I-GUIDE/vulnerability_analysis_agingdams/blob/main/iguide_agingdams.ipynb.

The workflow is composed of several steps that are able to be converted into CWL.

Nevertheless, the ability to read the data from some external servers was required, but was not possible due to restricted access.

For example, for Step 0 described in the notebook, the data is read from the following server: https://fim.sec.usace.army.mil. However, this URL is not accessible.

As the required data were available on the HPC environment, these data could be used for testing purposes to replicate Step 1 of the workflow. Step 1 is titled Extract “Inundation Classification for Census Tracts for this Dam”.

The part of the notebook that is of interest is the following:

 

import cybergis_compute_client
from cybergis_compute_client import CyberGISCompute

cybergis = CyberGISCompute(url="cgjobsup.cigi.illinois.edu", isJupyter=True, protocol="HTTPS", port=443, suffix="v2")
cybergis.show_ui(defaultJob="Extract_Inundation_Census_Tracts_Processor",input_params=params_classify_tracts)

Listing C.16 — I-GUIDE CyberGISCompute invocation

The cybergis_compute_client is a Python client library that provides an interface to the CyberGISCompute API. The CyberGISCompute class is used to create a connection to the CyberGISCompute server and submit a job to the server.

In order to replicate this step using a CWL workflow, a Docker image for the Job named “Extract_Inundation_Census_Tracts_Processor” needs to be created. For doing so, the following repository can be used: https://github.com/I-GUIDE/container_images/. The I-GUIDE project uses the HPC Container Maker from NVIDIA to build the Docker or Apptainer images.

For converting the recipe into a Dockerfile, use the following command.

 

hpccm --recipe recipe.hpccm --format docker > Dockerfile

Listing C.17 — Use HPC Container Maker to convert a recipe to a Dockerfile

See below the slightly modified generated Dockerfile.

 

FROM geoedf/framework

COPY ./GeoEDF /extractinundationcensustracts/GeoEDF
COPY ./setup.py /extractinundationcensustracts/setup.py

RUN apt-get update && \
    apt-get install -y unzip

RUN cd /extractinundationcensustracts && \
    pip3 install .

RUN echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
    echo "conda activate geoedf" >> ~/.bashrc

ENV PATH="/opt/conda/envs/geoedf/bin:${PATH}"
COPY ./main_multi_scenarios.py /app.py

Listing C.18 — Generated Dockerfile

To test the container for executing the job, follow the steps from the manifest.json available from: https://github.com/cybergis/population_vulnerable_to_dam_failure_compute. The steps are as follows: pre_processing_stage, execution_stage, post_processing_stage.

 

python setup.py
python main_multi_scenarios_final_JP.py
python cleanup.py

Listing C.19 — Steps defined in the manifest.json

A CWL encapsulating the workflow is provided. Execute the CWL using cwltool.

 

# cwltool --leave-tmpdir --tmpdir-prefix=$(pwd)/ ../app-pkg.cwl#extrac-inundation-census-tract --dam_id_array="CA10022"
INFO /home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/venv/bin/cwltool 3.1.20250110105449
INFO Resolved '../../app-pkg.cwl#extrac-inundation-census-tract' to 'file:///home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/app-pkg.cwl#extrac-inundation-census-tract'
INFO [workflow ] start
INFO [workflow ] starting step step_1
INFO [step step_1] start
WARNING [job step_1] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
WARNING [job step_1] Skipping Docker software container '--cpus' limit despite presence of ResourceRequirement with coresMin and/or coresMax setting. Consider running with --strict-cpu-limit for increased portability assurance.
INFO [job step_1] /home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0$ docker \
    run \
    -i \
    --mount=type=bind,source=/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0,target=/LtLiFg \
    --mount=type=bind,source=/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/z7d58kky,target=/tmp \
    --workdir=/LtLiFg \
    --read-only=true \
    --user=1001:1001 \
    --rm \
    --cidfile=/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/f92bqcma/20250131164539-392399.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/LtLiFg \
    geoedf/extractinundationcensustracts4cwl \
    /bin/bash \
    run.sh \
    CA10022 2> /home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/std.out
/LtLiFg
Output Directory:  /tmp/Multi_F_Results/N_0
Temp Directory:  /LtLiFg
Total Dams: 345
Creating output file that is 197P x 137L.
Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff.
Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff to destination /LtLiFg/MH_F_CA10022_resample.tiff.
...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
Creating output /LtLiFg/MH_F_CA10022.json of format GeoJSON.
100 - done.
Creating output file that is 190P x 129L.
Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff.
Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff to destination /LtLiFg/TAS_F_CA10022_resample.tiff.
...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
Creating output /LtLiFg/TAS_F_CA10022.json of format GeoJSON.
100 - done.
Creating output file that is 148P x 103L.
Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. -9999) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff.
Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff to destination /LtLiFg/NH_F_CA10022_resample.tiff.
...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
Creating output /LtLiFg/NH_F_CA10022.json of format GeoJSON.
100 - done.
Archive:  /tmp/tl_2020_06_tabblock20.zip
 extracting: tl_2020_06_tabblock20.cpg
  inflating: tl_2020_06_tabblock20.dbf
  inflating: tl_2020_06_tabblock20.prj
  inflating: tl_2020_06_tabblock20.shp
  inflating: tl_2020_06_tabblock20.shp.ea.iso.xml
  inflating: tl_2020_06_tabblock20.shp.iso.xml
  inflating: tl_2020_06_tabblock20.shx
CA10022: Step 1, 1/4, Identifying associated regions for multiple scenarios
CA10022: Step 1, 2/4, Search states associated
-- CA10022 impacts 1 States, ['06']
CA10022: Step 1, 3/4, Extracting GEOID of census blocks
CA10022: Step 1, 4/4, Assigning geometry to census blocks
CA10022: Step 2, 1/2, Merging census data to FIM geoid
CA10022: Step 2, 2/2, Calculating Moran's I and LISA
Merging results for 1 dams
Saving results Multi_F_fim.geojson
Saving results Multi_F_mi.geojson
Saving results Multi_F_lm.geojson
Done
INFO [job step_1] Max memory used: 6980MiB
INFO [job step_1] completed success
INFO [step step_1] completed success
INFO [workflow ] completed success
{
    "stac": [
        {
            "location": "file:///home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0",
            "basename": "uzihaaj0",
            "class": "Directory",
            "listing": [
                {
                    "class": "File",
                    "location": "file:///home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_mi.geojson",
                    "basename": "Multi_F_mi.geojson",
                    "size": 1510,
                    "checksum": "sha1$99fa0fdb37b17ad7cdbd10159acc24f5e3ec0a0b",
                    "path": "/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_mi.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_lm.geojson",
                    "basename": "Multi_F_lm.geojson",
                    "size": 2019858,
                    "checksum": "sha1$455d6d5d6c54d58c9d2d817704978ea445c4d2a2",
                    "path": "/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_lm.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_fim.geojson",
                    "basename": "Multi_F_fim.geojson",
                    "size": 17870,
                    "checksum": "sha1$228acbc5fb546fc448cbbf881c0b621b62de8ea3",
                    "path": "/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0/Multi_F_fim.geojson"
                }
            ],
            "path": "/home/gerald-fenoy/WORK/2024/TB20/cybergis-extract-inundation-census-tract-app-pkg/runs/tmp1/uzihaaj0"
        }
    ]
}INFO Final process status is success

Listing C.20 — Run log from cwltool

Below is the log generated by the execution of the workflow using the toil-cwl-runner.

 

TOIL_CHECK_ENV=True toil-cwl-runner --jobStore=/home/x-gfenoy/TOIL/toil_storage/example/job_store/$(uuidgen) --batchSystem=slurm         --singularity      --defaultDisk=6G --maxDisk=8G app-package.cwl#extrac-inundation-census-tract example-job.yaml
[2025-02-01T06:21:48-0500] [MainThread] [I] [cwltool] Resolved 'app-package.cwl#extrac-inundation-census-tract' to 'file:///home/x-gfenoy/cybergis/app-package.cwl#extrac-inundation-census-tract'
[2025-02-01T06:21:49-0500] [MainThread] [I] [toil.cwl.cwltoil] Importing input files...
[2025-02-01T06:21:49-0500] [MainThread] [I] [toil.cwl.cwltoil] Importing tool-associated files...
[2025-02-01T06:21:49-0500] [MainThread] [I] [toil.cwl.cwltoil] Creating root job
[2025-02-01T06:21:49-0500] [MainThread] [I] [toil.cwl.cwltoil] Starting workflow
[2025-02-01T06:21:49-0500] [MainThread] [I] [toil] Running Toil version 7.0.0-d569ea5711eb310ffd5703803f7250ebf7c19576 on host login07.anvil.rcac.purdue.edu.
[2025-02-01T06:21:51-0500] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2025-02-01T06:21:51-0500] [MainThread] [I] [toil.leader] Issued job 'CWLJob' extrac-inundation-census-tract.0.process kind-CWLJob/instance-0udrq2p2 v1 with job batch system ID: 2 and disk: 5.9 Gi, memory: 7.9 Gi, cores: 1, accelerators: [], preemptible: False
[2025-02-01T06:23:51-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 02-01-2025 06:23:51: CWL step complete: extrac-inundation-census-tract.0.process
[2025-02-01T06:23:51-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] extrac-inundation-census-tract.0.process.stdout follows:
=========>
        /zaPGJB
        Output Directory:  /tmp/Multi_F_Results/N_0
        Temp Directory:  /zaPGJB
        Total Dams: 345
        Creating output file that is 197P x 137L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_CA10022.tiff to destination /zaPGJB/MH_F_CA10022_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /zaPGJB/MH_F_CA10022.json of format GeoJSON.
        100 - done.
        Creating output file that is 190P x 129L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_CA10022.tiff to destination /zaPGJB/TAS_F_CA10022_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /zaPGJB/TAS_F_CA10022.json of format GeoJSON.
        100 - done.
        Creating output file that is 148P x 103L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff [1/1] : 0Using internal nodata values (e.g. -9999) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_CA10022.tiff to destination /zaPGJB/NH_F_CA10022_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /zaPGJB/NH_F_CA10022.json of format GeoJSON.
        100 - done.
        Archive:  /tmp/tl_2020_06_tabblock20.zip
         extracting: tl_2020_06_tabblock20.cpg
          inflating: tl_2020_06_tabblock20.dbf
          inflating: tl_2020_06_tabblock20.prj
          inflating: tl_2020_06_tabblock20.shp
          inflating: tl_2020_06_tabblock20.shp.ea.iso.xml
          inflating: tl_2020_06_tabblock20.shp.iso.xml
          inflating: tl_2020_06_tabblock20.shx
        CA10022: Step 1, 1/4, Identifying associated regions for multiple scenarios
        CA10022: Step 1, 2/4, Search states associated
        -- CA10022 impacts 1 States, ['06']
        CA10022: Step 1, 3/4, Extracting GEOID of census blocks
        CA10022: Step 1, 4/4, Assigning geometry to census blocks
        CA10022: Step 2, 1/2, Merging census data to FIM geoid
        CA10022: Step 2, 2/2, Calculating Moran's I and LISA
        Merging results for 1 dams
        Saving results Multi_F_fim.geojson
        Saving results Multi_F_mi.geojson
        Saving results Multi_F_lm.geojson
        Done
<=========
[2025-02-01T06:24:04-0500] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 (0 failures) [02:09<00:00, 0.03 jobs/s]
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Collecting workflow outputs...
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_fim.geojson...
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_mi.geojson...
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_lm.geojson...
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Stored workflow outputs
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Computing output file checksums...
{
    "stac": [
        {
            "location": "file:///home/x-gfenoy/cybergis/tmp-out518lljks",
            "basename": "tmp-out518lljks",
            "nameroot": "tmp-out518lljks",
            "nameext": "",
            "class": "Directory",
            "listing": [
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_fim.geojson",
                    "basename": "Multi_F_fim.geojson",
                    "size": 17870,
                    "checksum": "sha1$228acbc5fb546fc448cbbf881c0b621b62de8ea3",
                    "nameroot": "Multi_F_fim",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_fim.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_mi.geojson",
                    "basename": "Multi_F_mi.geojson",
                    "size": 1514,
                    "checksum": "sha1$e5c8788ddd4e44dd59c1ebc72208a97f485b5af4",
                    "nameroot": "Multi_F_mi",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_mi.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_lm.geojson",
                    "basename": "Multi_F_lm.geojson",
                    "size": 2019858,
                    "checksum": "sha1$455d6d5d6c54d58c9d2d817704978ea445c4d2a2",
                    "nameroot": "Multi_F_lm",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-out518lljks/Multi_F_lm.geojson"
                }
            ],
            "path": "/home/x-gfenoy/cybergis/tmp-out518lljks"
        }
    ]
}
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.cwl.cwltoil] CWL run complete!
[2025-02-01T06:24:05-0500] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/home/x-gfenoy/TOIL/toil_storage/example/job_store/445e5d39-99a0-4b17-b4ac-5fb9e05efdd6)

Listing C.21 — Run log from toil-cwl-runner

In order to execute the workflow via toil using SLURM as a batch system, publish the following docker image on DockerHub: geolabs/cybergis-eoap.

The scatter mechanism offered by CWL was used and is described in the following section as to how to process multiple dams in parallel.

C.3.4.  Scatter mechanism in CWL

The CWL scatter mechanism supports parallel execution of workflow steps, where each scattered execution instance receives a different subset of the input data. This improves performance and resource utilization, especially when dealing with large-scale datasets. CWL provides different scattering methods:

By leveraging these scatter methods, geospatial workflows can efficiently process large datasets in a distributed computing environment. To achieve this on HPC clusters, CWL is executed through Toil WES, which maps the scattered processes onto SLURM job submissions, ensuring optimal utilization of computational resources.

 

(venv3.11) (base) x-gfenoy@login07.anvil:[cybergis] $ TOIL_CHECK_ENV=True toil-cwl-runner --jobStore=/home/x-gfenoy/TOIL/toil_storage/example/job_store/$(uuidgen) --batchSystem=slurm         --singularity      --defaultDisk=6G --maxDisk=8G app-p
ackage.cwl#extrac-inundation-census-tract example-job1.yaml
[2025-02-03T13:12:04-0500] [MainThread] [I] [cwltool] Resolved 'app-package.cwl#extrac-inundation-census-tract' to 'file:///home/x-gfenoy/cybergis/app-package.cwl#extrac-inundation-census-tract'
[2025-02-03T13:12:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Importing input files...
[2025-02-03T13:12:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Importing tool-associated files...
[2025-02-03T13:12:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Creating root job
[2025-02-03T13:12:05-0500] [MainThread] [I] [toil.cwl.cwltoil] Starting workflow
[2025-02-03T13:12:05-0500] [MainThread] [I] [toil] Running Toil version 7.0.0-d569ea5711eb310ffd5703803f7250ebf7c19576 on host login07.anvil.rcac.purdue.edu.
[2025-02-03T13:12:07-0500] [MainThread] [I] [toil.leader] 0 jobs are running, 1 jobs are issued and waiting to run
[2025-02-03T13:12:07-0500] [MainThread] [I] [toil.leader] Issued job 'CWLJob' extrac-inundation-census-tract.1.process kind-CWLJob/instance-vh_7qnf1 v1 with job batch system ID: 2 and disk: 5.9 Gi, memory: 11.8 Gi, cores: 4, accelerators: [], preemptible: False
[2025-02-03T13:12:07-0500] [MainThread] [I] [toil.leader] Issued job 'CWLJob' extrac-inundation-census-tract.0.process kind-CWLJob/instance-nqs8ej8a v1 with job batch system ID: 3 and disk: 5.9 Gi, memory: 11.8 Gi, cores: 4, accelerators: [], preemptible: False
[2025-02-03T13:13:29-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 02-03-2025 13:13:29: CWL step complete: extrac-inundation-census-tract.0.process
[2025-02-03T13:13:29-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] extrac-inundation-census-tract.0.process.stdout follows:
=========>
        /UndHuj
        Output Directory:  /tmp/Multi_F_Results/N_0
        Temp Directory:  /UndHuj
        Total Dams: 345
        Creating output file that is 363P x 267L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_SD01097.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_SD01097.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_SD01097.tiff to destination /UndHuj/MH_F_SD01097_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /UndHuj/MH_F_SD01097.json of format GeoJSON.
        100 - done.
        Creating output file that is 363P x 267L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_SD01097.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_SD01097.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_SD01097.tiff to destination /UndHuj/TAS_F_SD01097_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /UndHuj/TAS_F_SD01097.json of format GeoJSON.
        100 - done.
        Creating output file that is 363P x 267L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_SD01097.tiff [1/1] : 0Using internal nodata values (e.g. 0) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_SD01097.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_SD01097.tiff to destination /UndHuj/NH_F_SD01097_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /UndHuj/NH_F_SD01097.json of format GeoJSON.
        100 - done.
        Archive:  /tmp/tl_2020_46_tabblock20.zip
         extracting: tl_2020_46_tabblock20.cpg
         extracting: tl_2020_46_tabblock20.dbf
         extracting: tl_2020_46_tabblock20.prj
         extracting: tl_2020_46_tabblock20.shp
         extracting: tl_2020_46_tabblock20.shp.ea.iso.xml
         extracting: tl_2020_46_tabblock20.shx
         extracting: tl_2020_46_tabblock20.shp.iso.xml
        SD01097: Step 1, 1/4, Identifying associated regions for multiple scenarios
        SD01097: Step 1, 2/4, Search states associated
        -- SD01097 impacts 1 States, ['46']
        SD01097: Step 1, 3/4, Extracting GEOID of census blocks
        SD01097: Step 1, 4/4, Assigning geometry to census blocks
        SD01097: Step 2, 1/2, Merging census data to FIM geoid
        SD01097: Step 2, 2/2, Calculating Moran's I and LISA
        Merging results for 1 dams
        Saving results Multi_F_fim.geojson
        Saving results Multi_F_mi.geojson
        Saving results Multi_F_lm.geojson
        Done
<=========
[2025-02-03T13:13:35-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 02-03-2025 13:13:35: CWL step complete: extrac-inundation-census-tract.1.process
[2025-02-03T13:13:35-0500] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] extrac-inundation-census-tract.1.process.stdout follows:
=========>
        /mrUvUX
        Output Directory:  /tmp/Multi_F_Results/N_0
        Temp Directory:  /mrUvUX
        Total Dams: 345
        Creating output file that is 367P x 200L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_MS03356.tiff [1/1] : 0Using internal nodata values (e.g. -9999) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_MS03356.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//MH_F_MS03356.tiff to destination /mrUvUX/MH_F_MS03356_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /mrUvUX/MH_F_MS03356.json of format GeoJSON.
        100 - done.
        Creating output file that is 366P x 199L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_MS03356.tiff [1/1] : 0Using internal nodata values (e.g. -9999) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_MS03356.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//TAS_F_MS03356.tiff to destination /mrUvUX/TAS_F_MS03356_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /mrUvUX/TAS_F_MS03356.json of format GeoJSON.
        100 - done.
        Creating output file that is 366P x 199L.
        Processing /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_MS03356.tiff [1/1] : 0Using internal nodata values (e.g. -9999) for image /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_MS03356.tiff.
        Copying nodata values from source /vsicurl?list_dir=no&empty_dir=yes&url=https://www.hydroshare.org/resource/1de14e8622564aa5bd645b92d899a201/data/contents//NH_F_MS03356.tiff to destination /mrUvUX/NH_F_MS03356_resample.tiff.
        ...10...20...30...40...50...60...70...80...90...100 - done.
        0...10...20...30...40...50...60...70...80...90...Several drivers matching json extension. Using GeoJSON
        Creating output /mrUvUX/NH_F_MS03356.json of format GeoJSON.
        100 - done.
        Archive:  /tmp/tl_2020_28_tabblock20.zip
         extracting: tl_2020_28_tabblock20.cpg
         extracting: tl_2020_28_tabblock20.dbf
         extracting: tl_2020_28_tabblock20.prj
         extracting: tl_2020_28_tabblock20.shp
         extracting: tl_2020_28_tabblock20.shp.ea.iso.xml
         extracting: tl_2020_28_tabblock20.shx
         extracting: tl_2020_28_tabblock20.shp.iso.xml
        MS03356: Step 1, 1/4, Identifying associated regions for multiple scenarios
        MS03356: Step 1, 2/4, Search states associated
        -- MS03356 impacts 1 States, ['28']
        MS03356: Step 1, 3/4, Extracting GEOID of census blocks
        MS03356: Step 1, 4/4, Assigning geometry to census blocks
        MS03356: Step 2, 1/2, Merging census data to FIM geoid
        MS03356: Step 2, 2/2, Calculating Moran's I and LISA
        Merging results for 1 dams
        Saving results Multi_F_fim.geojson
        Saving results Multi_F_mi.geojson
        Saving results Multi_F_lm.geojson
        Done
<=========
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 (0 failures) [01:38<00:00, 0.05 jobs/s]
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Collecting workflow outputs...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_fim.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_mi.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_lm.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_fim.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_mi.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Saving file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_lm.geojson...
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Stored workflow outputs
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] Computing output file checksums...
{
    "stac": [
        {
            "location": "file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55",
            "basename": "tmp-outyfqnlg55",
            "nameroot": "tmp-outyfqnlg55",
            "nameext": "",
            "class": "Directory",
            "listing": [
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_fim.geojson",
                    "basename": "Multi_F_fim.geojson",
                    "size": 133506,
                    "checksum": "sha1$1fec430665800a5c637ed36892d391b39c95fd0c",
                    "nameroot": "Multi_F_fim",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_fim.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_mi.geojson",
                    "basename": "Multi_F_mi.geojson",
                    "size": 1488,
                    "checksum": "sha1$11eaf14b4b33a5587e95c18ed3b252d85e862bd1",
                    "nameroot": "Multi_F_mi",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_mi.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_lm.geojson",
                    "basename": "Multi_F_lm.geojson",
                    "size": 423877,
                    "checksum": "sha1$e4ff6b15fe30177882d6fa686f8bfbcbb67cbb6d",
                    "nameroot": "Multi_F_lm",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outyfqnlg55/Multi_F_lm.geojson"
                }
            ],
            "path": "/home/x-gfenoy/cybergis/tmp-outyfqnlg55"
        },
        {
            "location": "file:///home/x-gfenoy/cybergis/tmp-outajnb5eui",
            "basename": "tmp-outajnb5eui",
            "nameroot": "tmp-outajnb5eui",
            "nameext": "",
            "class": "Directory",
            "listing": [
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_fim.geojson",
                    "basename": "Multi_F_fim.geojson",
                    "size": 20087,
                    "checksum": "sha1$cfb14c04ae8c75eb5c1b0f160bff33fd49f5df77",
                    "nameroot": "Multi_F_fim",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_fim.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_mi.geojson",
                    "basename": "Multi_F_mi.geojson",
                    "size": 1479,
                    "checksum": "sha1$573512a7e01873e0a5bdfcfeac094431b3c68235",
                    "nameroot": "Multi_F_mi",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_mi.geojson"
                },
                {
                    "class": "File",
                    "location": "file:///home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_lm.geojson",
                    "basename": "Multi_F_lm.geojson",
                    "size": 195914,
                    "checksum": "sha1$02666ae0d29e8c1a11a5f8dd2fd736baebfabcf9",
                    "nameroot": "Multi_F_lm",
                    "nameext": ".geojson",
                    "path": "/home/x-gfenoy/cybergis/tmp-outajnb5eui/Multi_F_lm.geojson"
                }
            ],
            "path": "/home/x-gfenoy/cybergis/tmp-outajnb5eui"
        }
    ]
}
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.cwl.cwltoil] CWL run complete!
[2025-02-03T13:13:48-0500] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/home/x-gfenoy/TOIL/toil_storage/example/job_store/e630922c-ece4-4c92-940a-b21a9369bdc8)

Listing C.22 — Run log from toil-cwl-runner with multiple inputs

C.3.5.  Performance Evaluation

The performance evaluation of the COG files in HPC environments was conducted using a set of benchmarks and metrics to assess the efficiency and scalability of COG in geospatial data processing workflows. The evaluation included the following key aspects:

The evaluation was conducted using open-source geospatial libraries and software to work with COG files. Benchmarking tools were used to collect CPU and memory resource utilization metrics and timing metrics for performance evaluation. The HPC environment provided by I-GUIDE was utilized for this investigation.

The performance evaluation included a comparison of COG’s performance against traditional GeoTIFF files in terms of speed, scalability, efficiency, and functionality. The evaluation results provide insights into the benefits and drawbacks of using COG files in HPC environments and recommendations for adopting COG in geospatial data processing workflows.

C.3.6.  Conclusion

If using Scatter permits parallelization of tasks, the workflow was encapsulated as a single entity when splitting a workflow into smaller parts as some tasks that run internally to the workflow could have been executed in parallel. For instance, executing every scenario in parallel rather than running the whole workflow for each individual dam.

The workflow that was chosen for this activity should have been split into smaller parts for enhancing scalability and optimizing performance. Also, the workflow used different formats that are accessed at every run online. Further, some datasets are provided in zipped shapefiles format. Having the data available in a more standard format such as that used in the OGC API — Features service with a backend based on GeoParquet would have been of interest.

With the system in place, toil to execute the CWL workflow, and the capability to ask for statistical information about CPU time, memory consumption, disk usage, and so on; with the `toil —stats ` command, it is possible to compare the processing time of the workflow with inputs from the initial GeoTIFF format to COG with the different tilesize and compression options.

The next step should be to evaluate workflows with more granularity to ensure that the task can be clearly identified within a workflow execution. This will allow for better monitoring and optimization of the workflow execution.

D.1.1.  Example Use Case:

Example

D.1.2.  Advantages of GeoZarr:

D.2.1.  Data Collection

D.2.2.  Selection Criteria

D.2.3.  Evaluation Criteria

D.2.4.  Tools and Techniques

D.2.5.  Analysis Methods

D.5.1.  Summary

Running the benchmark took around three seconds and created 2,184 data files. The dataset contained three indexes (dimensions) for time, x coordinate and y coordinate with a value of temperature. The data was partitioned by the time dimension as defined by the chunk property in the dataset schema. All the data files were 204 bytes in size. A coder could choose different partitioning schemes to create fewer or more data files.

A drawback of using Zarr formatted datasets is that the number of files in a large dataset could become very large. A large number of files can affect reading of the dataset by reducing I/O performance. In addition, a large number of small files leads to inefficient partitioning on storage devices. However, the benefit of using Zarr formatted datasets is that small partitions support adding or updating data easily without massive amounts of memory, compute time, or I/O. Another advantage to reading Zarr formatted datasets is that an application reading from the dataset can choose to read only a few partitions. In this example, an application may only be interested in observing temperatures from a small subset of time periods. Thus, the application only needs to read a few files from the underlining storage device.

During the examination of Cloud Optimized File Formats, the OGC Testbed-20 participants evaluated two APIs for working with Zarr files. Python XArray is easy to use. However, XArray seems to require knowing how large the dataset will be and requires that a coder load all the data at once. This is okay for smaller datasets and datasets that do not change overtime. However, XArray does not seem to be a good choice for larger datasets. The OGC Testbed-20 participants considered the Python Zarr module. The code in the Zarr module is more flexible. Nevertheless, participants agreed that this module is more difficult than it needed to be. Therefore, the participants created an open-source API for writing GeoZarr files as described in the evaluation section above.

D.5.2.  Recommendations

The OGC Testbed-20 participants provide the following guidance and recommendations for using the geozarr API [geozarr-lib] on HPC environments:

In general, the participants felt that GeoZarr is well-suited for processing on High Performance Computing (HPC) clusters. Distributed programming and parallel processing will support the efficient writing and reading of large or streaming datasets. The main issue that coders will face is knowing how to partition the data. Large partitions support better compression rates. However, smaller partitions lend themselves to distributing the processing to HPC cluster nodes.

D.6.1.  Sharding Index

One such feature is the sharding index [27]. Sharding enables storing very large, chunked arrays, which currently would have inefficient access characteristics or might even be impossible to store as generic Zarr arrays. The significance of sharding indexes in Zarr stems from the limitations imposed by storage devices and operating systems on the number and size of files stored. In particular, GeoZarr often deals with small chunk sizes, leading to the creation of numerous small files. While this is advantageous for the distribution and modification of datasets in High Performance Computing (HPC) environments, it poses challenges related to storage and operating system constraints on cluster nodes.

The Zarr version 3 sharding index mitigates the issue of small files by allowing a single file to hold multiple chunks of data. This approach is similar to how GeoTIFF files support internal tiling. By sharding chunks, data creators can organize data to achieve the best of both worlds: Small chunks and appropriately sized files. The term “appropriately sized” indicates that data creators can group chunks into files optimized for storage devices without combining all chunks into a single file.

This method allows for better distribution of processing while also reducing the complexity involved in adding and modifying data.

E.1.1.  Example Use Case:

Example

E.1.2.  Advantages of GeoParquet

The advantage of using GeoParquet on a High-Performance Computing (HPC) cluster is the ability to partition data. The more the data can be partitioned, the better performance a distributed database controller can achieve. The fact that data is partitioned by columns allows the data curator to add additional columns without having to reindex any existing files. Column portioning in GeoParquet allows a data curator to put column/row group data in separate files. This chunking of data into smaller files is suited to the processing of data on distributed nodes of the HPC.

The indexing concept in GeoParquet enables a database engine to skip the reading of data files that are not relevant to a query. In addition, GeoParquet does not store null data making the storage data more compact. When a database controller is asked to store null data, the controller stores a Boolean bit in a definition level indicating if the value exists or is null. Thus, the controller only needs to adjust the definition level when adding null values.

E.2.1.  Data Collection

E.2.2.  Selection Criteria

E.2.3.  Evaluation Criteria

E.2.4.  Tools and Techniques

E.2.5.  Analysis Methods

E.5.1.  Summary

E.5.2.  Recommendations

The benefit of containerization is that coders can keep the containers up to date with new toolsets and everyone can have customized toolsets that meet their needs. The drawback of containerization is that networked applications are harder to write and work with. The initial results of the Testbed-20 investigations showed that while a status display of the ray cluster shows multiple nodes attached, the nodes were not able to communicate correctly. One recommendation would be to run the application on the HPC hardware without containerization. However, with the hardware/software provided for this benchmark, the code needed to be refactored to run on an older version of Python. The other option is to update the software stack on the HPC. However, that could affect other current projects and probably would not be considered. So, this leaves the coders with the option of either figuring out what the communication barrier is or looking at other forms of distributed programming. Then, the only real option that presents itself with the current code is to run the container on a single node. This works and still takes advantage of multiprocessing.

Partitioning and compression are other issues to consider for data curators. Compressing small amounts of data does not provide much storage savings. In addition, the storage hardware has its own partitioning scheme. One recommendation for the curator is to define page sizes that match the storage partitioning scheme and are big enough to offer real compression value. The curator should also consider the size of the row group in records. Knowledge of the potential storage size of a record along with the page size should allow a curator to determine how many records to put into a row group. The curator needs to weigh the partition size against the need for parallel reading/writing and the amount of change that will occur with the dataset. Row Groups for a column can be stored in separate files. Smaller partitions tend to allow for better distribution access in distributed programming. In addition, smaller partitions mean less refactoring of data when records are inserted, updated or deleted. However, if the data is fairly static and queries return large record sets, the curator will then want to create large partitions that can be compressed.

Another consideration for fairly static datasets is record grouping. Since GeoParquet deals with spatial data, the curator may want to define row groups that group spatially near data together. This is because indexing of spatial data is done with a bounding box. A large bounding box with different row groups overlapping each make for very inefficient queries. Considering “spatial” awareness to the data also applies to the paging scheme within the row group as each page can have its own bounding box index. The one exception to this is when you have a large volume of data where all features are spatially near each other. In this situation a data curator may want to have some overlapping bounding boxes as multiprocessing or distributed processing could query several row groups and pages at the same time.

The OGC Testbed-20 HPGC task participants recommend the use of HPC clusters for processing GeoParquet data. The distributed and multiprocessing characteristics of an HPC make HPC clusters a good candidate for processing large GeoParquet datasets. The issue comes in the curation of data to determine record placement and partition sizes.