Published

OGC Engineering Report

•
OGC Testbed 17: CITE Engineering Report
Luis Bermudez Editor
Submission Date: 2021-11-19 Approval Date: 2021-12-09 Publication Date: 2022-04-08 External identifier of this OGC® document: http://www.opengis.net/doc/PER/t17-D045 Internal identifier of this OGC® document: 21-044
OGC Engineering Report

Published

Notice

This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard.

Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.

Document number:21-044
Document type:OGC Engineering Report
Document subtype:
Document stage:Published
Document language:English

Copyright notice

Copyright © 2022 Open Geospatial Consortium
To obtain additional rights of use, visit http://www.ogc.org/legal/

Note

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.


License Agreement

Permission is hereby granted by the Open Geospatial Consortium, (“Licensor”), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.

If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.

THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.

This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.

Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications. This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.

None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.



I.  Abstract

This OGC Testbed 17 Engineering Report (ER) documents the result of the work performed in the CITE thread of the OGC Testbed-17 initiative. CITE is the Compliance Interoperability & Testing Evaluation Subcommittee that provides a forum for an open, consensus discussion regarding approaches and issues related to conformance and interoperability testing as part of the OGC standards process. This ER provides information about the development of a test suite for the OGC API — Processes Standard (OGC18-062r2) to be executed in the OGC Test Evaluation tool (TEAM Engine). The ER also documents an evaluation of an alternative environment for OGC compliance testing.

II.  Executive Summary

This Engineering Report (ER) captures the result of the work performed in the CITE thread as part of the OGC Testbed-17 initiative. The document provides information about the development of a test suite for the OGC API – Processes standard to be executed in the OGC Validator tool (which is implemented using open source TEAM Engine software product) and evaluation of an alternative environment for OGC testing. The work is done under the umbrella of the Compliance Interoperability & Testing Evaluation (CITE) Subcommittee (SC) that provides a forum for an open, consensus discussion regarding approaches and issues related to conformance and interoperability testing as part of the OGC Standardization process.

The content of this ER will inform OGC Standards Working Groups (SWGs) about how to structure and write compliance tests for the most recent OGC API Standards. The ER also documents the pros and cons of using a possible alternative testing environment. It will facilitate enhancement of interoperability by providing the infrastructure to test and improve more implementations that are seeking compliance certification towards OGC standards.

As at 2021, several OGC SWGs are developing API standards that enable easier interaction of modern clients with servers. The OGC API — Processes standard is designed to enable a client to explore and run processes available over the web. The work done in Testbed 17 provides both the TestNG scripts and an alternative environment to write the tests to check if implementations are compliant with the OGC API — Processes.

The OGC Validator is currently implemented using a Java-based application called TEAM Engine to perform testing of software that claim compliance with one or more OGC standards. TEAM Engine executes one or more tests written in either the Compliance Test Language (CTL) or implemented using the TestNG Framework. The Testbed-17 participants explored the challenges of using this testing framework for the most recent OGC APIs.

The work in this thread responded to the following research questions:

An overview of findings and recommendations is as follows:

III.  Keywords

The following are keywords to be used by search engines and document catalogues.

ogcdoc, OGC document, CITE, API, compliance


IV.  Preface

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

V.  Security considerations

No security considerations have been made for this document.

VI.  Submitting Organizations

The following organizations submitted this Document to the Open Geospatial Consortium (OGC):

VII.  Submitters

All questions regarding this document should be directed to the editor or the contributors:

Name Organization Role
Luis Bermudez GeoSolutions Editor
Peter Vretanos CubeWerx Contributor
Benjamin Pross 52°North Contributor
Gérald Feony Geolabs Contributor
Jon, Herrmann interactive instruments GmbH Contributor

OGC Testbed 17: CITE Engineering Report

1.  Normative references

The following documents are referred to in the text in such a way that some or all of their content constitutes requirements of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

OGC 18-062r2, OGC API — Processes — Part 1: Core, Open Geospatial Consortium (2021)

2.  Terms and definitions

This document uses the terms defined in OGC Policy Directive 49, which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this document and OGC documents do not use the equivalent phrases in the ISO/IEC Directives, Part 2.

This document also uses terms defined in the OGC Standard for Modular specifications (OGC 08-131r3), also known as the ‘ModSpec’. The definitions of terms such as standard, specification, requirement, and conformance test are provided in the ModSpec.

For the purposes of this document, the following additional terms and definitions apply.

2.1. Abstract Test Suite (ATS)

A set of testable assertions about the functionality of a standard, which an implementation must support in order to achieve compliance to the standard. ATS are based on the conformance clauses defined in the standard.

2.2. Compliance

A state of a specific software product, which implements an OGC Standard and has passed the Compliance Testing Evaluation.

2.3. Executable Test Suite (ETS)

A set of code (e.g. Java and CTL) that provides runtime tests for the assertions defined by the ATS. Test data required to do the tests are part of the ETS.

2.4. Process

A process p is a function that for each input returns a corresponding output

p : X → Y

where X denotes the domain of arguments x and Y denotes the co-domain of values y. Within this specification, process arguments are referred to as process inputs and result values are referred to as process outputs. Processes that have no process inputs represent value generators that deliver constant or random process outputs.

The term process is one of the most used terms both in the information and geosciences domain. If not stated otherwise, this specification uses the term process as an umbrella term for any algorithm, calculation or model that either generates new data or transforms some input data into output data as defined in section 4.1 of the WPS 2.0 standard.

3.  Introduction

The Compliance Program provides the resources, procedures, and policies to certify products for compliance with one or more OGC standards. Amongst the resources provided by the program are executable test suites that enable developers to test whether their products implement OGC Standards correctly. This Engineering Report (ER) provides the following major sections:

4.  Executable Test Suite

This section provides information about the development of the Executable Test Suite (ETS) for OGC API — Processes. The executable test suite was developed as a module for deployment into TEAM Engine software. The following standards development organizations are known to offer compliance testing using TEAM Engine:

The European Union’s INSPIRE Validator is also known to use the OGC’s TEAM Engine instance for some INSPIRE validation tests.

4.1.  TestNG Test

4.1.1.  Overview

The executable test for OGC API — Processes 1.0 is available at the OGC GitHub repository:

https://github.com/opengeospatial/ets-ogcapi-processes10

The test has been developed in TestNG[https://testng.org/doc/], the current test framework used by the OGC Compliance Program.

The following standards have been used:

  • OGC API — Processes — Part 1: Core (OGC 18-062r2)

Each conformance class is represented as a TestNG Java class within its own package. The package structure is shown in the following:

|
|_ org.opengis.cite.ogcapiprocesses10
|_ org.opengis.cite.ogcapiprocesses10.conformance
|_ org.opengis.cite.ogcapiprocesses10.general
|_ org.opengis.cite.ogcapiprocesses10.jobs
|_ org.opengis.cite.ogcapiprocesses10.landingpage
|_ org.opengis.cite.ogcapiprocesses10.openapi3
|_ org.opengis.cite.ogcapiprocesses10.process
|_ org.opengis.cite.ogcapiprocesses10.processlist
|_ org.opengis.cite.ogcapiprocesses10.util

Figure 1

The following conformance classes were implemented:

  • Landing Page /

  • API Definition /api

  • Conformance Path /conformance

  • HTTP 1.1

  • Processes /processes

  • Jobs /jobs

The following conformance classes were not implemented:

  • Joblist

  • Dismiss

  • Callback

The method stubs and code comments (JavaDoc) for the Java classes were created out of the abstract test suite using an automated script.

A demonstration instance of the test suite is available here:

https://17.testbed.dev.52north.org/teamengine/

For the validation of the JSON requests/responses the OpenAPI4J library was used.

The following image shows the start screen of the user interface. The user can specify the endpoint of the landing page of the OGC API — Processes implementation, as well as the identifier of a testable process.

Figure 2 — Start screen of the TEAM Engine tests for the OGC API - Processes - Part 1 standard

Figure 3 — Result screen of the TEAM Engine tests for the OGC API - Processes - Part 1 standard

During the testbed, 30 tests of the OGC API — Processes standard were implemented. An unresolved issue with the validation of the schema for results prevents the completion of a number of tests. The following table shows the test results using the OGC API — Processes instance running at: http://tb17.geolabs.fr:8101/ogc-api using the process published by the server that has the identifier “echo”:

Table 1

TestResultReason
test Job Results SyncSkippedDid not find Link with value rel=monitor, skipping test.
test Job Results Async DocumentPassed- =
test Job Creation Input Inline BinaryFailedNot implemented yet.
test Job Creation Sync Raw Mixed MultiFailedNot implemented yet.
test Job Results No Such JobPassed-
test Job Results Exception Results Not ReadyFailedNot implemented yet.
validate Conformance Operation And ResponsePassed-
test Job Creation Input InlinePassed-
test Job Creation Sync Raw Value OneFailedGot unexpected status code: 500
test Job Exception No Such JobPassed-
test Job Creation InputsPassed-
test Job Creation Input RefSkippedNo input with href detected.
test Job Creation RequestPassed-
test Process SuccessPassed-
test Job OpPassed-
test Process Exception No Such ProcessPassed-
test Job ResultsFailedbody: Type expected ‘string’, found ‘object’. (code: 1027) (…​)
test Job Results FailedFailedbody: Field ‘type’ is required. (code: 1026) From: body.<required>
test Job SuccessPassed-
test Process List SuccessPassed-
landing Page ValidationPassed-
test ProcessPassed-
test Job Creation Success AsyncPassed-
test Job Creation Input ArrayFailedNot implemented yet.
test Pl LinksPassed-
test Job Creation Input Inline MixedFailedNot implemented yet.
test Job Creation Auto Execution ModePassed-
test Job Creation Input ValidationFailedexpected <br>[400] but found <br>[200]
test Job Creation Default OutputsPassed-
test Pl Limit ResponsePassed-
test Job Creation Sync DocumentPassed-
test Job Results Async Raw RefPassed-
test Job Creation Input Inline BboxPassed-
test Job Creation Input Inline ObjectPassed-
test Process ListPassed-
test Job Results Async Raw Value OneFailedJava exception
test Job Results Async Raw Mixed MultiFailedNot implemented yet.
test Job Creation Sync Raw Value MultiFailedNot implemented yet.
test Job Results Async Raw Value MultiFailedNot implemented yet.
test Pl Limit DefinitionPassed-
landing Page RetrievalPassed-
test Job Creation Sync Raw RefFailedNot implemented yet.
test Job Creation Default Execution ModePassed-
test Job Creation OpPassed-

4.1.2.  Recommendations

  • The TestNG framework works well with the executable test suite for the OGC API — Processes — Part 1: Core. Except for the validation issue for the result schema no critical issues were detected. The OpenAPI4J library reliably validates the JSON requests/responses.

  • OGC API — Processes — Part 1: Core consists of several conformance classes with currently 44 tests. The tests are listed directly beneath each other. This way, it can be hard to get an overview of the passing/failing tests. Thus, a possibility for better structuring of the tests is recommended.

  • The creation of new tests requires to execute the test repeatedly. Currently, single tests cannot be run, only the complete test suite, which can take a considerable amount of time. It is therefore recommended to investigate how single tests can be run.

  • A large number of execute requests are sent to the implementation under test to evaluate various combinations of parameters. It is recommended to investigate whether responses to execute requests can be reused to cover different test cases.

  • A number of the tests that fail seem to be a cascaded effect of the fact that the TestNG-based ETS does not recognize the specification relation types. For example, the test “landing Page Validation” fails because the ETS is looking for a link with relation “processes” but the correct link relation according to the specification is “http://www.opengis.net/def/rel/ogc/1.0/processes”. The ETS should be modified to address this. The problem has been reported in the GitHub Issues log.

  • The TestNG-based ETS does not seem to recognize the content type application/problem+json as a valid content type for an exception response. The OGC API — Processes specification, however, cites RFC 7807 where it specifies this as the correct MIME type for error report. A GitHub issue has been recorded for implementing support for the MIME type specified in RFC 7807.

  • Improve feedback to the users. Good feedback enables a pseudo-interactive engagement with the OGC Validator supporting incremental refinement of the server being tested. The feedback should include the headers passed to the server, the URL that was accessed, and the content of the body sent to the server under test, if applicable. It is recommended to include this level of feedback on the TestNG-based ETS.

5.  Alternative Test Environment

An alternative test environment based on the ETF test framework was used to validate implementations towards OGC standards, in particular focusing on the new OGC API standards. The standard selected was OGC API — Process.

5.1.  ETF test framework

The ETF test framework is an open-source application framework that can be used by the tester to execute tests through a web interface. The ETF uses a modular software-architecture and is designed to deliver user-friendly, self-explanatory test reports.

It is successfully used as a basis for the European INSPIRE Validator and in several projects for German mapping agencies.

5.2.  NeoTL DSL

For web service testing, ETF has leveraged SoapUI[https://www.soapui.org/], but for various reasons a new solution was evaluated in 2020. A new ETF test driver for geoservice and Web API testing was implemented in a prototype based on a new domain specific language (DSL). The DSL was named NeoTL.

The goal was to simplify the definition of tests with the DSL as much as possible and thus make it maintainable and accessible to subject matter experts without knowledge of a specific programming language. Language concepts were adapted for Testbed 17 and the special requirements for creating OGC API Processes tests.

To lower the entry barrier cloud-based tool support was leveraged. The test cases were developed in the browser, a local installation was not required.

Another goal was to simplify the communication between the various experts; the SWG members, test authors and implementers by using a DSL.

5.3.  Comparison with CTL

Prior to 2014, executable test suites in the OGC Validator used the OGC Compliance Test Language (CTL). Due to limitations of CTL, since 2014 all new executable test suites developed for the OGC Validator are built using the TestNG framework instead of CTL. This section presents a comparison between CTL and NeoTL, however, it should be noted that CTL is a legacy technology that is no longer used to develop new executable test suites in the OGC Compliance Program.

Domain Specific Languages are distinguished between internal and external DSLs.

Examples of internal DSL include domain-specific UML profiles, domain-specific XML Schema, the Gradle DSL of the Gradle build tool and also the CTL.

Internal DSL essentially use the language concepts of their host language. With Gradle these language concepts are based on Groovy[https://groovy-lang.org/], with CTL the concepts are based on XML elements and various XML schema. This means that the grammar is restricted to the syntax of the host language and it adopts all the nice but also the less pleasant syntactic features, as the following CTL test illustrates:



<test name="wfs:wfs-1.1.0-Basic-GetCapabilities-tc1">
    <param name="wfs.GetCapabilities.get.url"/>
    <assertion>The GET method request must be supported (using HTTP GET).</assertion>
    <comment>GetCapabilities by GET. Pass if all of the following conditions are true:
        (1) the response is schema valid;
        (2) the root document is an wfs:WFS_Capabilities document.
    </comment>
    <link title="wfs-1.1.0-Basic-GetCapabilities-tc1">http://cite.opengeospatial.org/te2/about/wfs/1.1.0/site/ats-wfs11-basic-cc/GetCapabilities/GET/BasicGetCapabilities-GET-tc1.html</link>
    <link>OGC 04-094, 13.1, p.79</link>
    <code>

        <xsl:variable name="request1">
            <request>
                <url>
                    <xsl:value-of select="$wfs.GetCapabilities.get.url"/>
                </url>
                <method>get</method>
                <param name="service">WFS</param>
                <param name="version">1.1.0</param>
                <param name="request">GetCapabilities</param>
                <p:XMLValidatingParser.GMLSF1/>
            </request>
        </xsl:variable>

        <xsl:choose>
            <xsl:when test="not($request1/*)">
                <ctl:message>FAILURE: Missing or invalid response entity.</ctl:message>
                <ctl:fail/>
            </xsl:when>
            <xsl:otherwise>
                <xsl:variable name="expression">//wfs:WFS_Capabilities</xsl:variable>
                <ctl:call-test name="ctl:assert-xpath">
                    <ctl:with-param name="expr" select="$expression"/>
                    <ctl:with-param name="doc" select="$request1"/>
                </ctl:call-test>
            </xsl:otherwise>
        </xsl:choose>

    </code>
</test>

Figure 4 — CTL test case example

An external DSL is a language that’s parsed independently of the host general purpose language. For example, regular expressions and CSS. The concrete syntax and the semantics are freely defined. This means that external DSLs can be more flexible and expressive. The CTL example would look something like this in NeoTL (the Request definition is externalized and referenced):



TestCase "The GET method request must be supported (using HTTP GET)" {
    id: wfs.1.1.0.Basic-GetCapabilities-tc1
    description: "GetCapabilities by GET. Pass if all of the following
        conditions are true:
            (1) the response is schema valid;
            (2) the root document is an wfs:WFS_Capabilities document."

    references:
        - "Abstract Test Case wfs-1.1.0-Basic-GetCapabilities-atc3, p.79"
            "https://portal.ogc.org/files/?artifact_id=8339"
            AbstractTestCase

    ValidationStep "GetCapabilities with GET method" {
        id: step
        description: "GetCapabilities with GET method and validate response against
            GMLSF1 schema"

        given:
            - Service is "WFS 1.1.0"

        when: Request requests.wfs1.GetCapabilities executed

        then:
            - Assert XPath {
                /wfs:WFS_Capabilities/* exists
            }
            - Assert XmlSchema {
                schema "http://schemas.opengis.net/wfs/1.1.3/wfs.xsd"
                validates
            }
    }
}

GetRequest "GetCapabilities" {
    id: requests.wfs1.GetCapabilities

    query:
        - "service" = "WFS"
        - "version" = "1.1.0"
        - "request" = "GetCapabilities"
}

Figure 5 — NeoTL test case example

Furthermore, it is possible to tailor the IDE closely to the language. Thus, it is possible to perform semantic checks in addition to syntactic checks, syntax highlighting and to simplify the implementation of tests for the test developer with several other IDE services. Figure 6 shows the editor with some IDE services like the Outline and the Problem views as well as the syntactically highlighted test definition for an OGC API — Processes test case.