Testbed-12 CITE User Guide - Profiles

This document offers guidance about how to develop and execute a test suite that verifies an implementation of some OGC application profile that is based on one or more existing standards. It is primarily intended for developers and other technical professionals who seek to test implementations that claim conformance to an OGC profile. For more general information about how to develop OGC conformance tests, see the TEAM Engine documentation site.

In ISO 19106, a profile is defined as a

In practice, a profile imposes additional constraints on an implementation such as mandating support for an optional capability. Or it may define extensions where permitted by the base standards. However, a profile can never contradict any base standard, such that conformance to a profile implies conformance to all of the base standards from which it is derived.

Various user communities have developed profiles based on OGC or ISO standards. For example:

The DGIWG WMS profile is based on the OGC WMS 1.3 specification, and the NSG WMS profile is in turn based on the DGIWG profile. A profile always has one or more dependencies that must be heeded when developing an implementation or a conformance test suite. As indicated in WMS profile dependencies, a test suite for any WMS profile will include tests covering one or more conformance classes in the OGC WMS conformance test suite. The DGIWG profile requires conformance at the "Queryable WMS" level; the associated test suite also includes tests that cover the specific requirements of that profile.

A conformance class is essentially a container for a coherent set of requirements that address some functional capability or area of concern. A conformance level is a kind of conformance class in the which the requirements of a higher level contain all the requirements of the lower levels (see Conformance classes).

A test suite for a profile generally selects one or more conformance classes (or levels) from the relevant set of base specifications; these tests then implicitly become part of the dependent test suite. The base tests are invoked in the course of running the profile-specific tests. However, the manner in which this is done depends on how a test suite is implemented.

A test suite that covers the requirements of an application profile is accessed and executed just like any other test suite. It appears in the listing of available suites, and it can be selected and run in the same manner. See the following figure: Profile-specific test suites.

Most of the OGC conformance test suites developed in recent years are based on the TestNG framework. The framework provides a simple declarative mechanism for incorporating tests from other suites in a test suite definition, which is an XML document that conforms to this document type definition.

The root <suite> element contains one or more child <test> elements that correspond to conformance classes. These conformance classes may be drawn from other test suites. For example, the WFS 2.0 test suite uses two conformance classes from the GML 3.2 test suite. The following listing shows how this dependency is declared.

The <test> sets are run in document order by default, which is usually the preferred behavior. If any tests fail in a base conformance class, this can cause profile-specific tests to be skipped since there is already evidence of non-conformance.

The OGC Compliance Test Language (CTL) specifies an XML grammar for defining a test suite. Some older OGC test suites were implemented using CTL scripts. In recent years test suites have been developed using the TestNG framework, and this is the recommended approach for new test suites. But some current profiles are based on older OGC standards for which only CTL-based test suites exist.

Tests may be organized into separate packages denoted by the ctl:package element; the main package must contain a ctl:suite element that identifies the starting test. The ctl:profile element may be used to formally define a test group that corresponds to a conformance class. A profile must refer to its base test group (usually the main test suite), and it must also identify the starting test. The sample listing below displays a profile definition for the GET method binding in the WMTS test suite.

The <base> element refers to the suite or profile (by name) that this profile depends on. This establishes an implicit ordering, such that the root suite is run first and test execution follows the dependency graph.

A RESTful API was introduced in order to provide a common mechanism for invoking a test suite regardless of its implementation. In effect, a test suite becomes a kind of "microservice" that can be accessed as indicated in the table below.

The URI parameters are delimited by curly brackets; these are explained below.

The list of available test suites is presented as a brief HTML document (XHTML syntax) that contains links to the deployed test suites. While the document can be displayed in a web browser for human viewers, it can also be consumed and parsed by other software applications in order to facilitate test execution. For example, a service description in a registry could be automatically annotated with information about its conformance status by running a test suite and inspecting the results.

When the link for a particular test suite is dereferenced a summary document is obtained. This document briefly describes the test suite and contains a table of test run arguments. Each input argument is a separate entry in the body of an HTML table as shown in the listing below.

A test run is initiated by submitting a request to the test run controller. The summary description lists the test run arguments (mandatory, conditional, optional) that are recognized by the controller. A test suite can be invoked using a simple GET request in most cases. For example, to test a WFS 2.0 implementation the target URI is constructed as follows (replace localhost:8080 with the actual host name and port number of an available teamengine installation):

where {wfs-capabilities-url} is the URL to retrieve the capabilities document for the implementation under test (IUT). Note that this need not be obtained directly from the IUT—​it could be fetched from elsewhere (e.g. a service registry), as long as it describes the same service.

With TEAM Engine 4.9 or later it is also possible to invoke a CTL test suite in this manner. However, a controller must be available in order to do this. The WMS 1.3 test suite contains a CtlController class that serves as an example of how to enable this capability in other CTL test suites.

As a concrete example, consider the DGIWG WMS profile which is based on the OGC WMS 1.3 standard (also published as ISO 19128:2005). The profile requires implementation of the Queryable WMS conformance class as defined in the base standard. To verify this, the OGC test suite can be invoked using the REST API by submitting a GET request with the following query parameters (the target URI has been abbreviated to emphasize the query component):

A successful response contains an XML entity that represents the test results. The root element contains a <log> child element for each test that was run. The first log entry indicates the overall verdict; the value of the endtest/@result attribute is an integer code that signifies a test verdict (see table below).

If a constituent test failed, the overall verdict is set as Inherited Failure (5). In general, a failed subtest will "taint" all of its ancestor tests in this manner.

The default format of the test results is framework-specific: for TestNG, this is an XML representation having <testng-results> as the document element. The results of running a CTL test suite also produce XML output, with <execution> as the document element. Support for the W3C Evaluation and Report Language (EARL) 1.0 Schema has been introduced. The specification (currently a late stage working draft) defines an RDF vocabulary for describing test results:

The following listing shows how conformance classes are described using the EARL vocabulary. An earl:TestRequirement instance represents a conformance class; it has one or more constituent tests (earl:TestCase). Furthermore, a dependency may be expressed using the dct:requires property. In this example, Conformance level 2 is based on Conformance level 1 and thus establishes a higher level of conformance.

The EARL vocabulary does not define any terms that pertain to a test run by itself. A custom vocabulary was introduced for this purpose. A cite:TestRun resource provides basic summary information about a test run, including the input arguments and an overall tally of test verdicts. Standard Dublin Core metadata terms are employed where appropriate. For example, the dct:extent property reports the temporal extent of the test run; that is, its total duration represented using the XML Schema duration datatype.