4.1.  TestNG Test

4.1.1.  Overview

The executable test for OGC API — Processes 1.0 is available at the OGC GitHub repository:

https://github.com/opengeospatial/ets-ogcapi-processes10

The test has been developed in TestNG[https://testng.org/doc/], the current test framework used by the OGC Compliance Program.

The following standards have been used:

• OGC API — Processes — Part 1: Core (OGC 18-062r2)

Each conformance class is represented as a TestNG Java class within its own package. The package structure is shown in the following:

|
|_ org.opengis.cite.ogcapiprocesses10
|_ org.opengis.cite.ogcapiprocesses10.conformance
|_ org.opengis.cite.ogcapiprocesses10.general
|_ org.opengis.cite.ogcapiprocesses10.jobs
|_ org.opengis.cite.ogcapiprocesses10.landingpage
|_ org.opengis.cite.ogcapiprocesses10.openapi3
|_ org.opengis.cite.ogcapiprocesses10.process
|_ org.opengis.cite.ogcapiprocesses10.processlist
|_ org.opengis.cite.ogcapiprocesses10.util

Figure 1

The following conformance classes were implemented:

• Landing Page /

• API Definition /api

• Conformance Path /conformance

• HTTP 1.1

• Processes /processes

• Jobs /jobs

The following conformance classes were not implemented:

• Joblist

• Dismiss

• Callback

The method stubs and code comments (JavaDoc) for the Java classes were created out of the abstract test suite using an automated script.

A demonstration instance of the test suite is available here:

https://17.testbed.dev.52north.org/teamengine/

For the validation of the JSON requests/responses the OpenAPI4J library was used.

The following image shows the start screen of the user interface. The user can specify the endpoint of the landing page of the OGC API — Processes implementation, as well as the identifier of a testable process.

Figure 2 — Start screen of the TEAM Engine tests for the OGC API - Processes - Part 1 standard

Figure 3 — Result screen of the TEAM Engine tests for the OGC API - Processes - Part 1 standard

During the testbed, 30 tests of the OGC API — Processes standard were implemented. An unresolved issue with the validation of the schema for results prevents the completion of a number of tests. The following table shows the test results using the OGC API — Processes instance running at: http://tb17.geolabs.fr:8101/ogc-api using the process published by the server that has the identifier “echo”:

Table 1

Test	Result	Reason
test Job Results Sync	Skipped	Did not find Link with value rel=monitor, skipping test.
test Job Results Async Document	Passed	- =
test Job Creation Input Inline Binary	Failed	Not implemented yet.
test Job Creation Sync Raw Mixed Multi	Failed	Not implemented yet.
test Job Results No Such Job	Passed	-
test Job Results Exception Results Not Ready	Failed	Not implemented yet.
validate Conformance Operation And Response	Passed	-
test Job Creation Input Inline	Passed	-
test Job Creation Sync Raw Value One	Failed	Got unexpected status code: 500
test Job Exception No Such Job	Passed	-
test Job Creation Inputs	Passed	-
test Job Creation Input Ref	Skipped	No input with href detected.
test Job Creation Request	Passed	-
test Process Success	Passed	-
test Job Op	Passed	-
test Process Exception No Such Process	Passed	-
test Job Results	Failed	body: Type expected ‘string’, found ‘object’. (code: 1027) (…​)
test Job Results Failed	Failed	body: Field ‘type’ is required. (code: 1026) From: body.<required>
test Job Success	Passed	-
test Process List Success	Passed	-
landing Page Validation	Passed	-
test Process	Passed	-
test Job Creation Success Async	Passed	-
test Job Creation Input Array	Failed	Not implemented yet.
test Pl Links	Passed	-
test Job Creation Input Inline Mixed	Failed	Not implemented yet.
test Job Creation Auto Execution Mode	Passed	-
test Job Creation Input Validation	Failed	expected <br>[400] but found <br>[200]
test Job Creation Default Outputs	Passed	-
test Pl Limit Response	Passed	-
test Job Creation Sync Document	Passed	-
test Job Results Async Raw Ref	Passed	-
test Job Creation Input Inline Bbox	Passed	-
test Job Creation Input Inline Object	Passed	-
test Process List	Passed	-
test Job Results Async Raw Value One	Failed	Java exception
test Job Results Async Raw Mixed Multi	Failed	Not implemented yet.
test Job Creation Sync Raw Value Multi	Failed	Not implemented yet.
test Job Results Async Raw Value Multi	Failed	Not implemented yet.
test Pl Limit Definition	Passed	-
landing Page Retrieval	Passed	-
test Job Creation Sync Raw Ref	Failed	Not implemented yet.
test Job Creation Default Execution Mode	Passed	-
test Job Creation Op	Passed	-

5.1.  ETF test framework

The ETF test framework is an open-source application framework that can be used by the tester to execute tests through a web interface. The ETF uses a modular software-architecture and is designed to deliver user-friendly, self-explanatory test reports.

It is successfully used as a basis for the European INSPIRE Validator and in several projects for German mapping agencies.

5.2.  NeoTL DSL

For web service testing, ETF has leveraged SoapUI[https://www.soapui.org/], but for various reasons a new solution was evaluated in 2020. A new ETF test driver for geoservice and Web API testing was implemented in a prototype based on a new domain specific language (DSL). The DSL was named NeoTL.

The goal was to simplify the definition of tests with the DSL as much as possible and thus make it maintainable and accessible to subject matter experts without knowledge of a specific programming language. Language concepts were adapted for Testbed 17 and the special requirements for creating OGC API Processes tests.

To lower the entry barrier cloud-based tool support was leveraged. The test cases were developed in the browser, a local installation was not required.

Another goal was to simplify the communication between the various experts; the SWG members, test authors and implementers by using a DSL.

5.3.  Comparison with CTL

Prior to 2014, executable test suites in the OGC Validator used the OGC Compliance Test Language (CTL). Due to limitations of CTL, since 2014 all new executable test suites developed for the OGC Validator are built using the TestNG framework instead of CTL. This section presents a comparison between CTL and NeoTL, however, it should be noted that CTL is a legacy technology that is no longer used to develop new executable test suites in the OGC Compliance Program.

Domain Specific Languages are distinguished between internal and external DSLs.

Examples of internal DSL include domain-specific UML profiles, domain-specific XML Schema, the Gradle DSL of the Gradle build tool and also the CTL.

Internal DSL essentially use the language concepts of their host language. With Gradle these language concepts are based on Groovy[https://groovy-lang.org/], with CTL the concepts are based on XML elements and various XML schema. This means that the grammar is restricted to the syntax of the host language and it adopts all the nice but also the less pleasant syntactic features, as the following CTL test illustrates:

<test name="wfs:wfs-1.1.0-Basic-GetCapabilities-tc1">
    <param name="wfs.GetCapabilities.get.url"/>
    <assertion>The GET method request must be supported (using HTTP GET).</assertion>
    <comment>GetCapabilities by GET. Pass if all of the following conditions are true:
        (1) the response is schema valid;
        (2) the root document is an wfs:WFS_Capabilities document.
    </comment>
    <link title="wfs-1.1.0-Basic-GetCapabilities-tc1">http://cite.opengeospatial.org/te2/about/wfs/1.1.0/site/ats-wfs11-basic-cc/GetCapabilities/GET/BasicGetCapabilities-GET-tc1.html</link>
    <link>OGC 04-094, 13.1, p.79</link>
    <code>

        <xsl:variable name="request1">
            <request>
                <url>
                    <xsl:value-of select="$wfs.GetCapabilities.get.url"/>
                </url>
                <method>get</method>
                <param name="service">WFS</param>
                <param name="version">1.1.0</param>
                <param name="request">GetCapabilities</param>
                <p:XMLValidatingParser.GMLSF1/>
            </request>
        </xsl:variable>

        <xsl:choose>
            <xsl:when test="not($request1/*)">
                <ctl:message>FAILURE: Missing or invalid response entity.</ctl:message>
                <ctl:fail/>
            </xsl:when>
            <xsl:otherwise>
                <xsl:variable name="expression">//wfs:WFS_Capabilities</xsl:variable>
                <ctl:call-test name="ctl:assert-xpath">
                    <ctl:with-param name="expr" select="$expression"/>
                    <ctl:with-param name="doc" select="$request1"/>
                </ctl:call-test>
            </xsl:otherwise>
        </xsl:choose>

    </code>
</test>

Figure 4 — CTL test case example

An external DSL is a language that’s parsed independently of the host general purpose language. For example, regular expressions and CSS. The concrete syntax and the semantics are freely defined. This means that external DSLs can be more flexible and expressive. The CTL example would look something like this in NeoTL (the Request definition is externalized and referenced):

TestCase "The GET method request must be supported (using HTTP GET)" {
    id: wfs.1.1.0.Basic-GetCapabilities-tc1
    description: "GetCapabilities by GET. Pass if all of the following
        conditions are true:
            (1) the response is schema valid;
            (2) the root document is an wfs:WFS_Capabilities document."

    references:
        - "Abstract Test Case wfs-1.1.0-Basic-GetCapabilities-atc3, p.79"
            "https://portal.ogc.org/files/?artifact_id=8339"
            AbstractTestCase

    ValidationStep "GetCapabilities with GET method" {
        id: step
        description: "GetCapabilities with GET method and validate response against
            GMLSF1 schema"

        given:
            - Service is "WFS 1.1.0"

        when: Request requests.wfs1.GetCapabilities executed

        then:
            - Assert XPath {
                /wfs:WFS_Capabilities/* exists
            }
            - Assert XmlSchema {
                schema "http://schemas.opengis.net/wfs/1.1.3/wfs.xsd"
                validates
            }
    }
}

GetRequest "GetCapabilities" {
    id: requests.wfs1.GetCapabilities

    query:
        - "service" = "WFS"
        - "version" = "1.1.0"
        - "request" = "GetCapabilities"
}

Figure 5 — NeoTL test case example

Furthermore, it is possible to tailor the IDE closely to the language. Thus, it is possible to perform semantic checks in addition to syntactic checks, syntax highlighting and to simplify the implementation of tests for the test developer with several other IDE services. Figure 6 shows the editor with some IDE services like the Outline and the Problem views as well as the syntactically highlighted test definition for an OGC API — Processes test case.

Figure 6 — NeoTL IDE example

All currently implemented IDE services are described in the language workbench documentation.

5.4.  Structure of a NeoTL Test Case

The individual language concepts are described in detail in the documentation and additionally also displayed directly in the IDE as help. The section briefly describes how a NeoTL Test Case is structured and what concepts it is based on.

The basis for the structuring is the ISO 19105:2021 — Geographic information — Conformance and testing , and thus executable tests are composed of the following concepts:

• test suites

• test modules

• and test cases.

TestSuite "OGC API - Processes" {
    id: org.opengis.ets.ogcapi.processes
    version: 0.9.2-snapshot
    description: "Executable Test Suite for validating Web APIs that implement
            the 'OGC API - Processes - Part 1: Core' standard. The Test Suites
            are based on the normative Abstract Test Suites from Annex A of the
            OGC Implementation Specification draft 1.0-draft.7-SNAPSHOT."

   references:
        - "OGC Implementation Specification"
            "https://docs.ogc.org/DRAFTS/18-062.html"
            ImplementationSpecification

    executes:
        - oapi.processes.core
        - oapi.processes.joblist
        // ...

    defines:
        - URL $schemaUrl = "https://raw.githubusercontent.com/opengeospatial/ogcapi-processes/master/core/openapi/schemas"
}

Figure 7 — Test Suite

TestModule "Job List" {
    id: oapi.processes.joblist

    description: "The Job list requirements class specifies how to retrieve a job list from the API"

    references:
        - "Core ATS"
            "http://docs.ogc.org/DRAFTS/18-062.html#_conformance_class_core"
            AbstractTestSuite
        - "Conformance Class Job list"
            http://www.opengis.net/spec/ogcapi-processes-1/1.0/conf/job-list
            ConformanceClass

    executes:
        - joblist.list
        - joblist.list.links
        // ...
}

Figure 8 — Test Module

TestCase "Validate links section in job list" {
    id: joblist.list.links
    description: "Validate that the proper links are included in a response."

    references:
        - "Conformance Class Job list"
            "https://docs.ogc.org/DRAFTS/18-062.html#_conformance_class_job_list"
            AbstractTestCase

    ValidationStep "Validate self link" {
        // ...
    }

    ValidationStep "Validate HTML link" {
        // ...
    }
}

Figure 9 — Test Case

The syntax for enumerations (see executes keyword in Figure 7) is influenced by YAML, but unlike YAML, the number of spaces before indentation is ignored.

A test case can contain one or multiple validation steps. Validation steps define the interaction with the implementation and the expected behavior. Their structure is based on the well-known Given-Then-When scheme used by many test frameworks.

ValidationStep "Request the Landing Page as JSON document" {
    id: step
    description: "Request an JSON document with an Accept application/json header"

    given:
        - ConformanceClass http://www.opengis.net/spec/ogcapi-processes-1/1.0/conf/json

    when: Request requests.landing.json executed

    then:
        - Assert OpenAPI3 {
            schema "${schemaUrl}/landingPage.yaml"
            validates
        }
        - Assert HTTP { statusCode "200" }
        - Assert HTTP { contentType "application/json" }
}

Figure 10 — Validation Step

Related literature [1] suggests that the three main sections of a Validation Step are:

• the given section which describes the precondition for the whole Validation Step

• the when section is the action that is executed to interact with the implementation

• the then section verifies the results of the previous action with assertions

5.4.1.  Given

If all preconditions in the given section are met, then the action defined in the when section is executed. The result of the action must satisfy all assertions defined in the then section to result in a passed validation. If at least one precondition is not fulfilled, the result of the validation step is the status skipped, if a conformance class is prerequisite for the test case, which is not implemented by the tested implementation, the status is not_applicable. Figure 11 shows how the status skipped is displayed in an ETF test report when a test case with required information has failed.

Figure 11 — ETF skipped status

    PostRequest "Execute Echo Process" {
        id: requests.echo

        path: "processes/${echoId}/execution"

        headers:
            - "Accept" = "application/json"
            - "Prefer" = ${preferHeader}

        body {
            "inputs": ${inputs},
            "outputs": ${outputs},
            "response": "document"
        } as application/json
    }

    - Assert JSON {
        $.jobs[?(@.type == 'process')] exists
        or $.jobs empty
        otherwise FAIL with "No job entry exists with the
            requested process type 'process'."
    }

    - Assert JSON {
        $.jobs count == 1
        otherwise FAIL with "Expected exactly one entry in the Job List"
    }

    - Assert JSON {
        $.conformsTo contains some
            "http://www.opengis.net/spec/ogcapi-processes-1/1.0/conf/core"
        otherwise FAIL with "Expected an 'conformsTo' array containing
            the value
            'http://www.opengis.net/spec/ogcapi-processes-1/1.0/conf/core'."
    }

5.5.  Dynamic tests in NeoTL

In addition to the declaration of conformance classes as precondition, the results of other validation steps can also be specified as preconditions. During the interaction with the implementation, further tests can be created based on the results. The concept of generators is used for this purpose, shown in Figure 16.

TestCase "Process descriptions" {
    id: process.description
    description: "Validate that a process description can be retrieved from the expected location."

    ValidationStep "Generate Requests" {
            id: process.desc.generator
            description: "Generate Process Description Requests"

            given:
                - ConformanceClass http://www.opengis.net/spec/ogcapi-processes-1/1.0/conf/ogc-process-description
                - Response from process.list.encoding.json
            when: Generator generators.process.ids executed
            then:
                - Assert JSON {
                    ${processIds} not empty
                    otherwise FAIL with "No IDs in process list found"
                }
    }

    ValidationStep "Validate Process Description" {
        id: process.desc
        description: "Validated responses of the generated process descriptions requests"

        given:
            - One ${processId} of ${processIds} from process.desc.generator
        when: Request requests.processes.description executed
        then:
            - Assert HTTP { statusCode "200" }
            - Assert HTTP { contentType "application/json" }
            - Assert OpenAPI3 {
                schema
                "${schemaUrl}/process.yaml"
                validates
            }
            - Assert JSON {
                $.id equals ${processId}
                otherwise FAIL with
                "The id in the returned response does not match the requested id"
            }
    }
}

Generator "Generate Process IDs" {
    id: generators.process.ids

    from RESPONSE
        as ${processIds} query $..id
}

GetRequest "Get Process Description" {
    id: requests.processes.description

    path: "processes/${processId}"

    headers:
        - "Accept" = "application/json"
}

Figure 16 — Generators

The generator is specified within the when section. With the generator, a set of values is extracted and a validation step is created and executed for each value in the set. The result in the report can be seen in Figure 17.

Figure 17 — NeoTL Generator report

If only one test is to be executed, an extractor must be used.

TestCase "Check for an echo process" {
    // /conf/core/job-creation-op
    id: jobs.echo
    description: "Check that an echo process with the id 'echo' or 'EchoProcess' exists"
    references:
    - "Processes /processes"
        "https://docs.ogc.org/DRAFTS/18-062.html#_jobs"
        AbstractTestCase

    ValidationStep "Extract EchoProcess ID" {
        id: extract
        description: "Check that an echo process with the id 'echo' or 'EchoProcess' exists"

        given:
            - Response from process.list.encoding.json

        when: Extractor extractors.echo.id executed

        then:
            - Assert JSON {
                ${echoId} exists
                otherwise FAIL with "No echo ID could be extracted"
            }
    }
}

Extractor "Extract Echo Process ID" {
    id: extractors.echo.id

    from RESPONSE
        as ${echoId} query $.*[?(@.id == 'echo' || @.id == 'EchoProcess')].id
}

TestCase "Request the process description of the the echo process" {
    id: jobs.echo.description
    description: "Check that the echo process description can be retrieved"
    references:
    - "Processes /processes"
        "https://docs.ogc.org/DRAFTS/18-062.html#_jobs"
        AbstractTestCase

    ValidationStep "Request EchoProcess description" {
        id: request
        description: "Check that the echo process description can be retrieved"

        given:
            - Value ${echoId} from jobs.echo.extract

        when: Request requests.process.description executed

        then:
            - Assert OpenAPI3 {
                schema
                "${schemaUrl}/process.yaml"
                validates
            }
    }
}

Figure 18 — Extractors

In the first test case, the ID of the echo process is queried and it is checked that such a process exists. If the validation step of the first test case does not fail, the validation step in the second test case jobs.echo.description is executed, which uses the extracted job ID in a request.

Figure 19 — NeoTL Extractor report

5.6.  Extension Points

A DSL usually describes what should happen, but not exactly how it should happen. Up to a certain level, the NeoTL DSL allows to define assertions, such as:

• Does an element exist?

• Does there exist at least / exactly one element in a sequence / array?

• Does a property possess the given value?

• Does a JSON object have properties / is an array empty?

• How does it compare with other elements?

• Does a response validate against a schema?

• Is a specific content type returned?

• Is a specific status code returned?

• Does an HTML message validate against a W3C markup validator?

For many tests, these assertions are sufficient and they help the test developer to stay on an abstract level. For more complex queries, a general purpose language, such as Java, is inevitably required.

During the implementation of executable test cases, there were two areas where general purpose languages had to be used:

1. A Java library was integrated to generate input data from a JSON schema definition

2. A special case had to be considered for the generation of inputs for Qualified Values. This was implemented in XQuery 3.1. Although XQuery is originally designed for processing XML, it has a generic data model. It can be used to process both XML and JSON, enabling processing of new OGC API services as well as services that serve XML.

The listing in Figure 20 shows the second case with the XQuery extension.

(:~
 : Check if schema defines a Qualified Value.
 :
 : see https://docs.ogc.org/DRAFTS/18-062.html#req_core_process-execute-input-inline-object
 :)
declare %private function oapip:mustBeQualified($schema) {
  exists($schema[type='object' and not(properties/bbox)]) or exists($schema[oneOf or anyOf or allOf])
};

(:~
 : Wraps each qualified value into a 'value' JSON object
 :
 : see https://docs.ogc.org/DRAFTS/18-062.html#req_core_process-execute-input-inline-object
 :)
declare function oapip:qualifiedSampleValues($nodes, $genFct as function(item()) as item()* ) {
  for $n in $nodes/*
      return element { $n/name() } {
        let $generated := $genFct($n/schema)
        return if (oapip:mustBeQualified($n/schema)) then
          element { "value" } {
                (: qualified value :)
                $generated
          }
        else
          $generated
      }
};

Figure 20 — XQuery extension

The defined function is called an extractor.

Extractor "Extract the input and output values from the schema object" {
    id: extractors.inputs

    from RESPONSE
        as ${inputs} query {
            oapip:qualifiedSampleValues( $.inputs, etf:sampleFromSchema )
        }
}

Figure 21 — Extractor calling XQuery

The etf:sampleFromSchema function is an XQuery wrapper for the Java library mentioned in the first case and is not described here.

5.7.  Modularization

All definitions are stored in files. For better overview, higher level structure definitions are only allowed once per file, otherwise the test developer is free to split the definitions into multiple files.

Assertions can also be externalized, combined into assertion groups and used in multiple validation steps:

AssertionGroup "JSON response received" {
    id: assertions.json.success

    assertions:
        - Assert HTTP { statusCode "200" }
        - Assert HTTP { contentType "application/json" }
}

AssertionGroup "Processes Conformance Class in Json Array" {
    id: assertions.processes.cc.json.array

    assertions:
        - AssertionGroup assertions.json.success must pass
        - Assert OpenAPI3 {
            schema
            "${schemaUrl}/confClasses.yaml"
            validates
        }
}

Figure 22 — Assertion Groups

If assertions in another file are referenced, this must be specified via an import statement.

import "Assertions.neotl"

Figure 23 — Import statement

The following directory structure was created for the Executable Test Suite. The content is described after the directory or file name.

.
├── ProcessesApi-testsuite.neotl # the Test Suite definition
├── extensions # the extension libraries that are used in this test suite
│   └── SampleGenerator.xqm
├── shared # definitions that are used in several Validation Steps
│   └── Assertions.neotl
├── tests
│   ├── 1_Core
│   │   ├── 1_Landing
│   │   │   ├── Actions.neotl
│   │   │   └── Landing-tcs.neotl # Test Case definitions
│   │   ├── 2_API
│   │   │   ├── Api-tcs.neotl
│   │   │   └── Actions.neotl
│   │   ├── 3_Conformance
│   │   │   ├── Actions.neotl # externalized Actions
│   │   │   ├── Assertions.neotl # externalized Assertions
│   │   │   └── Conformance-tcs.neotl # Test Case definitions
│   │   ├── 4_Processes
│   │   │   ├── Actions.neotl
│   │   │   ├── Assertions.neotl
│   │   │   └── Processes-tcs.neotl
│   │   ├── 5_Jobs
│   │   │   ├── Actions.neotl
│   │   │   └── Jobs-tcs.neotl
│   │   └── Core.neotl # the Test Module definition
│   ├── 2_Process_Description
# etc.

Figure 24 — File structure

Currently, Java libraries can also be placed as JAR files in the extension folder and be imported. They are loaded at test runtime with a separate Java ClassLoader to avoid conflicts between different ETSs.

5.8.  Summary of executable Test developed

The tests developed in NeoTL have been published in a public GitHub repository [2]

At the beginning of the implementation, the definitions of the abstract test cases were parsed and templates were generated for the DSL. Since some abstract test cases describe the validation of several aspects at once, they could not be implemented one-to-one, but had to be adapted to the DSL.

A further degree of automation could have been achieved if:

• the dependencies between tests would have been hyperlinked in a structured way

• the interaction with the implementation would have been separated from the validation of the results

Some abstract test cases could not be implemented because they require a sequence of events with certain test data or require internal control of the implementation under test’s behavior that cannot be realized with a black box test (see Abstract Test Case /conf/core/job-results-failed or /conf/dismiss/job-dismiss-op).

Certain tests require the test framework to receive and analyze certain test data from the application (e.g. Server) being tested, via HTTP Post or HTTP Get Request (see /conf/callback/job-callback or /conf/core/job-results-async-raw-ref). These far-reaching framework functionalities were also not implemented.

The following table contains an overview of all test cases implemented during Testbed 17. The mapping of the abstract test case identifiers to the NeoTL executable test case identifiers can be seen in the second and third columns.

Table 2

5.9.  TIE results

Out of the 82 abstract test cases, 65 executable test cases were implemented. The implemented test cases were run against the OGC API — Processes implementation of GeoLabs which passed 47 tests.

Other implementations were tested, but these did not implement the latest version of the OGC API — Processes standard and therefore stopped the test runs at a very early stage.

5.10.  Findings and Recommendations

1. During the implementation phase of the tests, maintenance of the used open-source validation library OpenAPI4J was discontinued. As an alternative, the KaiZen OpenAPI parser was tested, which did not reliably return errors when validating more complex schemas, especially when combining schemas with oneOf, anyOf and allOf. The OpenAPI4j library provided verifiable results with the GeoLabs implementation and therefore continued to be used.

2. Since a large number of JSON libraries are written in JavaScript, one could consider supporting this language alongside Java and XQuery as an additional extension language for NeoTL.

3. Currently, Java libraries can only be stored as JAR files in an ETS folder. Referencing specific versions as known from Maven or Gradle could be considered. However, the consideration could also be influenced by the second question, whether JavaScript libraries should also be usable.

4. Since only one ETS was implemented, it was not possible to evaluate how well the reusability of definitions between different ETSs works.

5. The abstract test cases should be referenceable via an URL so that the URL can be embedded in the executable tests. This might be performed using the OGC definition server[https://www.ogc.org/def-server].

6. The abstract test cases could be made more machine-readable so that dependencies can be extracted and that a distinction can be made between interaction with the implementation and expected behavior. A fully automated transformation into tests is not be possible, but a number of information could be pre-filled in templates.

7. Certain language concepts can be improved or simplified. This would require further feedback from users of NeoTL.

8. The IDE currently lacks a function to execute the test cases or individual requests locally. For a test, the ETS must currently be deployed.

9. In case of missing language concepts, the following approach has proven successful during development: in the first step, the extension is developed in a general purpose language. If it is used more often and could be useful for other executable test suites, one can think about including it directly as a language concept in the DSL.

10. The performance of the IDE can be improved, especially by caching constraint checks.

11. With a high number of generated validation steps, saving the responses, generating the report and viewing it in the browser takes a relatively long time. A test run with the GeoLabs implementation involves the execution of over 700 generated requests and the validation of over 2800 assertions. The runtime of the tests takes less than 40 seconds, while the whole report is available after almost 1.5 minutes. This could be optimized in the framework.

6.1.  Automatization of Tests from OGC Standards

6.2.  Echo Process

In some cases, certain requirements of the specification (e.g. jobs conformance class) requires a special mechanism for testing. For example, when a process has to run asynchronously for some time. The test engine should monitor the execution and the intermediate status messages. But, if the process is not known, the implementation of the test execution can become a challenge.

If the echo process is implemented, it will provide a scenario that a test engine can follow to validate part of the test.

Here is an example definition of the echo process:

{
   "id":"EchoProcess",
   "title":"Echo Process",
   "description":"This process accepts and number of input and simply echoes each input as an output.",
   "version":"1.0.0",
   "jobControlOptions":[
      "async-execute",
      "sync-execute"
   ],
   "outputTransmission":[
      "value",
      "reference"
   ],
   "inputs":{
      "pause":{
         "title":"Number of Pause Seconds",
         "description":"The number of seconds the EchoProcess process should pause execution, to simulate actually doing something, and thus give the test engine time to run the async execution tests.  For example, get statuses from the server, etc.  If the server does not implement async-execute then this parameter shall have no effect on the execution of the EchoProcess process and test engines should simply ignore it.",
         "minOccurs":0,
         "schema":{
            "type":"double",
            "default":4.0
         }
      },
      "stringInput":{
         "title":"String Literal Input Example",
         "description":"This is an example of a STRING literal input.",
         "schema":{
            "type":"string",
            "enum":[
               "Value1",
               "Value2",
               "Value3"
            ],
            "example":[
               "Value2"
            ]
         }
      },
      "measureInput":{
         "title":"Numerical Value with UOM Example",
         "description":"This is an example of a NUMERIC literal with an associated unit of measure.",
         "schema":{
            "type":"object",
            "required":[
               "value",
               "uom"
            ],
            "properties":{
               "measurement":{
                  "type":"number"
               },
               "uom":{
                  "type":"string"
               },
               "reference":{
                  "type":"string",
                  "format":"uri"
               }
            },
            "examples":[
               {
                  "meansurement":10,
                  "uom":"m"
               }
            ]
         }
      },
      "dateInput":{
         "title":"Date Literal Input Example",
         "description":"This is an example of a DATE literal input.",
         "schema":{
            "type":"string",
            "format":"dateTime",
            "examples":[
               "2021-10-31T23:59:59"
            ]
         }
      },
      "doubleInput":{
         "title":"Bounded Double Literal Input Example",
         "description":"This is an example of a DOUBLE literal input that is bounded between a value greater than 0 and 10.  The default value is 5.",
         "schema":{
            "type":"number",
            "format":"double",
            "minimum":0,
            "maximum":10,
            "default":5,
            "exclusiveMinimum":true,
            "examples":[
               3.14159
            ]
         }
      },
      "arrayInput":{
         "title":"Array Input Example",
         "description":"This is an example of a single process input that is an array of values.  In this case, the input array would be interpreted as a single value and not as individual inputs.",
         "schema":{
            "type":"array",
            "minItems":2,
            "maxItems":10,
            "items":{
               "type":"integer"
            },
            "examples":[
               [
                  1,
                  7
               ],
               [
                  2,
                  4,
                  6
               ]
            ]
         }
      },
      "complexObjectInput":{
         "title":"Complex Object Input Example",
         "description":"This is an example of a complex object input.",
         "schema":{
            "type":"object",
            "required":[
               "property1",
               "property5"
            ],
            "properties":{
               "property1":{
                  "type":"string"
               },
               "property2":{
                  "type":"string",
                  "format":"uri"
               },
               "property3":{
                  "type":"number"
               },
               "property4":{
                  "type":"string",
                  "format":"dateTime"
               },
               "property5":{
                  "type":"boolean"
               }
            },
            "examples":[
               {
                  "property1":"Some string.",
                  "property2":"http://www.opengis.org",
                  "property5":true
               }
            ]
         }
      }
   },
   "outputs":{
      "stringOutput":{
         "schema":{
            "type":"string",
            "enum":[
               "Value1",
               "Value2",
               "Value3"
            ]
         }
      },
      "measureOutput":{
         "schema":{
            "type":"object",
            "required":[
               "value",
               "uom"
            ],
            "properties":{
               "measurement":{
                  "type":"number"
               },
               "uom":{
                  "type":"string"
               },
               "reference":{
                  "type":"string",
                  "format":"uri"
               }
            }
         }
      },
      "dateOutput":{
         "schema":{
            "type":"string",
            "format":"dateTime"
         }
      },
      "doubleOutput":{
         "schema":{
            "type":"number",
            "format":"double",
            "minimum":0,
            "maximum":10,
            "default":5,
            "exclusiveMinimum":true
         }
      },
      "arrayOutput":{
         "schema":{
            "type":"array",
            "minItems":2,
            "maxItems":10,
            "items":{
               "type":"integer"
            }
         }
      },
      "complexObjectOutput":{
         "schema":{
            "type":"object",
            "required":[
               "property1",
               "property5"
            ],
            "properties":{
               "property1":{
                  "type":"string"
               },
               "property2":{
                  "type":"string",
                  "format":"uri"
               },
               "property3":{
                  "type":"number"
               },
               "property4":{
                  "type":"string",
                  "format":"dateTime"
               },
               "property5":{
                  "type":"boolean"
               }
            }
         }
      }
   },
   "links":[
      {
         "href":"https://processing.example.org/oapi-p/processes/EchoProcess/execution",
         "rel":"http://www.opengis.net/def/rel/ogc/1.0/execute",
         "title":"Execute endpoint"
      }
   ]
}

Figure 25 — Example definition of the echo process

Another example is as follows:

{
   "id":"echo",
   "title":"Echo input",
   "description":"Simply echo the value provided as input",
   "version":"2.0.0",
   "jobControlOptions":[
      "sync-execute",
      "async-execute",
      "dismiss"
   ],
   "outputTransmission":[
      "value",
      "reference"
   ],
   "links":[
      {
         "rel":"execute",
         "type":"application/json",
         "title":"Execute End Point",
         "href":"http://tb17.geolabs.fr:8108/ogc-api/processes/echo/execution"
      },
      {
         "rel":"alternate",
         "type":"text/html",
         "title":"Execute End Point",
         "href":"http://tb17.geolabs.fr:8108/ogc-api/processes/echo/execution.html"
      }
   ],
   "inputs":{
      "a":{
         "title":"Literal Input (string)",
         "description":"An input string",
         "schema":{
            "type":"string",
            "default":"Any value"
         }
      },
      "b":{
         "title":"Complex Input",
         "description":"A complex input ",
         "schema":{
            "oneOf":[
               {
                  "type":"string",
                  "contentEncoding":"utf-8",
                  "contentMediaType":"text/xml"
               },
               {
                  "type":"object"
               }
            ]
         }
      },
      "c":{
         "title":"BoundingBox Input ",
         "description":"A boundingbox input ",
         "schema":{
            "type":"object",
            "required":[
               "bbox",
               "crs"
            ],
            "properties":{
               "bbox":{
                  "type":"array",
                  "oneOf":[
                     {
                        "minItems":4,
                        "maxItems":4
                     },
                     {
                        "minItems":6,
                        "maxItems":6
                     }
                  ],
                  "items":{
                     "type":"number",
                     "format":"double"
                  }
               },
               "crs":{
                  "type":"string",
                  "format":"uri",
                  "default":"urn:ogc:def:crs:EPSG:6.6:4326",
                  "enum":[
                     "urn:ogc:def:crs:EPSG:6.6:4326",
                     "urn:ogc:def:crs:EPSG:6.6:3785"
                  ]
               }
            }
         }
      },
      "pause":{
         "title":"Literal Input (double)",
         "description":"An optional input which can be used to specify the number of seconds to pause the service before returning",
         "schema":{
            "type":"number",
            "default":10,
            "format":"double",
            "nullable":true
         }
      }
   },
   "outputs":{
      "a":{
         "title":"The output a",
         "description":"The output a returned",
         "schema":{
            "type":"string",
            "default":"Any value"
         }
      },
      "b":{
         "title":"The output b",
         "description":"The output b returned",
         "schema":{
            "oneOf":[
               {
                  "type":"string",
                  "contentEncoding":"utf-8",
                  "contentMediaType":"text/xml"
               },
               {
                  "type":"object"
               }
            ]
         }
      },
      "c":{
         "title":"BoundingBox output ",
         "description":"A boundingbox output ",
         "schema":{
            "type":"object",
            "required":[
               "bbox",
               "crs"
            ],
            "properties":{
               "bbox":{
                  "type":"array",
                  "oneOf":[
                     {
                        "minItems":4,
                        "maxItems":4
                     },
                     {
                        "minItems":6,
                        "maxItems":6
                     }
                  ],
                  "items":{
                     "type":"number",
                     "format":"double"
                  }
               },
               "crs":{
                  "type":"string",
                  "format":"uri",
                  "default":"urn:ogc:def:crs:EPSG:6.6:4326",
                  "enum":[
                     "urn:ogc:def:crs:EPSG:6.6:4326",
                     "urn:ogc:def:crs:EPSG:6.6:3785"
                  ]
               }
            }
         }
      }
   }
}

Figure 26 — Another example definition of the echo process

At the exception of the pause input parameter, every input is returned with the same name as an output. In the second case three inputs are present to illustrate the use of the three data types that were available in the WPS specification:

• a: a simple string (LiteralData),

• b: a complex data (can be application/json or text/xml),

• c: a bounding box

The pause parameter has been added to give the opportunity to test long running jobs. This way, the job list can be tested during the execution, especially the status and progress changes but, also to dismiss the conformance class, if implemented.

Implementation guidance:

• Servers SHALL implement sync-execute.

• Servers SHOULD implement async-execute.

• Servers SHALL implement value for outputTransmission.

• Servers SHOULD implement reference for outputTransmission.

• If async-execute is not supported then the text engine should ignore the pause input. The execution endpoint (link with rel=ogc:execute) should be adjusted to reflect the server actual execution endpoint for the EchoProcess process.

The test engine should then follow each branch in the following response summary tables: - https://docs.ogc.org/DRAFTS/18-062.html#sc_execute_response - https://docs.ogc.org/DRAFTS/18-062.html#_response_7 and verify that the server behaves accordingly. So, for example, in Table 7, the echo process is executed with:

• sync-execute

• response=document

• outputTransmission=reference

• and N outputs are returned

The server should respond with an HTTP status code of 200. The body of the response should be of type application/json and it should validate against the results.yaml schema. The test engine will also have to parse the response to verify that the value of each output in the response matches the input value specified by the engine.

6.3.  Execution Process

When implementing a very trivial echo service and deploying it on the GeoLabs prototype Server Instance, the participants realized that it may be a great help for service developers to expose a fully-featured scenario that can be used for testing a service execution. The initial echo service accessible on the GeoLabs Server Instance was simply returning the output without potentially converting the input into another format or reprojecting it, depending on its type. In the sample implementation provided, 3 inputs are supported: one string, one complex, and one bounding box. So, depending on the service complexity or the level of development of a specific service, the Test Suite itself may have difficulty producing a relevant request automatically. In consequence, experiments were performed including the definition of multiple paths corresponding to services’ execution path with their associated request’s body examples directly within the exposed OpenAPI.

So, we may consider adding one option to recommendation 24:

• Provide one or more example requests for a specific service in the exposed API.

Benefits of such an addition:

• Ease interactions with the published API using traditional OpenAPI tools, such as SwaggerUI;

• Ensure working scenarios for specific services;

• Advertise specific server implementation capabilities, such as automatic OGC Web Services publication;

• Give direct access to the Test Suites to a list of testable services without relying on fetching every single process description;

• Provide the Executable Test Suites with a set of pre-generated request bodies that can be used rather than having to generate the requests at run-time;

• Use external tools such as spectral to validate OpenAPI definition documents including the bodies of the requests as examples implying requests body’s validation.

Identified issue:

• Introducing the requirement of testing external execute request body provided within the OpenAPI from the Test Suite;

• Keep the specification and the request bodies in sync;

• Increased OpenAPI definition size.

Figure 27 — SwaggerUI without examples

Figure 28 — SwaggerUI with additional paths and associated examples