OGC® Catalogue Services support the ability to publish and search collections of descriptive information (metadata records) for geospatial data, services, and related information. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources within an information community.

This part of the Catalogue Services standard describes the common architecture for OGC Catalogue Services. This document abstractly specifies the interfaces between clients and catalogue services, through the presentation of abstract models. This common architecture is Distributed Computing Platform neutral and uses UML notation. Separate (Part) documents specify the protocol bindings for these Catalogue services, which build upon this document, for the HTTP (or CSW) and OpenSearch protocol bindings.

An Abstract Conformance Test Suite is not included in this document.  Such Suites shall be developed by protocol bindings and Application Profiles (see 8.5, ISO/IEC TR 10000-2:1998) that realize the conformance classes listed herein. An application profile consists of a set of metadata elements, policies, and guidelines defined for a particular application[1].

OGC document number 14-014 – HTTP Protocol Binding – Abstract Test Suite is available to address conformance with the provisions of OGC document number 12-176r2 – HTTP Protocol Binding. All annexes to this document are informative.

The following are keywords to be used by search engines and document catalogues.

OGC Catalogue Services, metadata, geospatial data, geospatial services, search, discovery, abstract model, general model, HTTP, CSW, OpenSearch, Abstract Conformance Test Suite, ogcdoc, OGC document, asynchronous, catalogue, CQL, client, csw:Record, distributed, Dublin Core, federated, filter, GetCapabilities, GetDomain, GetRecords, GetRecordById, Harvest, http, https, KVP, metadata, record, request, resource, response, server, schema, spatial, temporal, Transaction, UnHarvest, XML, XML-Schema.

This document is one part of the OGC® Catalogue Services version 3.0 Implementation Standard. Unlike previous versions, Catalogue 3.0 is now divided in multiple parts, with this part specifying the abstract model and another to describe the HTTP protocol binding known as Catalogue Service for the Web (CSW).

This version of the Catalogue Standard has been significantly improved, largely based on  change requests submitted by both Open Geospatial Consortium (OGC) members and the public. The changes made in this version relative to version 2.0.2 (OGC document 07-006r1) are summarized in Annex B.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

The following organizations submitted this Document to the Open Geospatial Consortium (OGC):

con terra GmbH

National Research Council of Italy (CNR)

Cubewerx Inc.

Intergraph Corporation

Joint Research Centre (JRC), European Commission

U.S. Geological Survey

Earlier versions of this Standard were submitted to the OGC by the following organizations:

BAE SYSTEMS Mission Solutions (formerly Marconi Integrated Systems, Inc.)

Blue Angel Technologies, Inc.

Environmental Systems Research Institute (ESRI)

Geomatics Canada (Canada Centre for Remote Sensing (CCRS)

Intergraph Corporation

MITRE

Oracle Corporation

U.S. Federal Geographic Data Committee (FGDC)

U.S. National Aeronautics and Space Administration (NASA)

U.S. National Imagery and Mapping Agency (NIMA)

All questions regarding this submission should be directed to the editor or the submitters:

1. Scope

This document abstractly specifies the interfaces and a framework for defining bindings and application profiles required to publish and access digital catalogues of metadata for geospatial data, services, and related resource information. These Catalogue Services support the use of one of several identified query languages to find and return results using well-known content models (metadata schemas) and encodings.

This Standard is applicable to the implementation of interfaces on catalogues of a variety of information resources. The target audience for this standard is the community of software developers who are implementers of OGC compliant Catalogue servers and clients.

2. Conformance

Conformance to the mandatory catalogue service abstract interfaces is described in section 8. It is the requirement of protocol-specific bindings and application profiles to provide concrete tests and validation in conformance with these abstract conformance classes. Test data and queries may be included in Application Profiles associated with this abstract model and with specific protocol bindings.

3. Normative References

The following normative documents contain provisions that, through referenced in this text, constitute provisions of this document. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. For undated references, the latest edition of the normative document referred to applies.

IETF RFC 2045 (November 1996), Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, Freed, N. and Borenstein N., eds., http://www.ietf.org/rfc/rfc2045.txt

IETF RFC 2141 (May 1997), URN Syntax, R. Moats, http://www.ietf.org/rfc/rfc2141.txt

IETF RFC 2396 (August 1998), Uniform Resource Identifiers (URI): Generic Syntax, Berners-Lee, T., Fielding, N., and Masinter, L., eds., http://www.ietf.org/rfc/rfc2396.txt

IANA, Internet Assigned Numbers Authority, MIME Media Types, available at http://www.iana.org/assignments/media-types/

ISO/IEC 8825:1990, Information technology – Open Systems Interconnection – Specification of Basic Encoding Rules for Abstract Syntax Notation One (ASN.1)

ISO/IEC TR 10000-1:1998. Information Technology – Framework and taxonomy of International Standardised Profiles – Part 1: General principles and documentation framework. Technical Report, JTC 1. Fourth edition, Available [online]: http://standards.iso.org/ittf/PubliclyAvailableStandards/c030726_ISO_IEC_TR_10000-1_1998(E).zip.

ISO/IEC 10746-2:1996. Information Technology – Open Distributed Processing – Reference Model: Foundations. Common text with ITU-T Recommendation X.902, Available [online]: http://standards.iso.org/ittf/PubliclyAvailableStandards/s018836_ISO_IEC_10746-2_1996(E).zip.

ISO 8601:2000(E), Data elements and interchange formats - Information interchange - Representation of dates and times

ISO 19101:2002, Geographic information – Reference model

ISO 19103 (DTS), Geographic information – Conceptual schema language, (Draft Technical Specification)

ISO 19106:2003, Geographic Information – Profiles

ISO 19108:2002, Geographic information – Temporal schema

ISO 19109:2002 (DIS), Geographic information – Rules for application schema

ISO 19110:2001 (DIS), Geographic information – Methodology for feature cataloguing

ISO 19113:2002, Geographic information – Quality principles

ISO 19114:2001, (DIS) Geographic information – Quality evaluation procedures

ISO 19118:2002, (DIS) Geographic information – Encoding

ISO/IEC 14977:1996, Information technology – Syntactic metalanguage – BNF

ISO 19115:2003, Geographic Information – Metadata

ISO 19119:2005, Geographic Information – Services

ISO/TS 19139:2007, Geographic Information – Metadata  -Implementation Specification

OASIS/ebXML Registry Services Specificationv2.5

OGC 99-113, OGC Abstract Specification Topic 13: Catalogue Services

OGC 02-112, OGC Abstract Specification Topic 12: OpenGIS Service Architecture

OGC 09-026r1, OGC Filter Encoding 2.0 Encoding Standard,

OGC 06-121r9,OGC Web Service Common Implementation Specification, Version 2.0.0

OMG UML, Unified Modeling Language, Version 1.3, The Object Management Group (OMG): http://www.omg.org/cgi-bin/doc?formal/00-03-01

OGC 12-176r2, OGC^® Catalogue Services specification – HTTP protocol binding (v3.0.0)

4. Terms and Definitions

This document uses the terms defined in Sub-clause 5.3 of [OGC 06-121r8], which is based on the ISO/IEC Directives, Part 2, Rules for the structure and drafting of International Standards. In particular, the word “shall” (not “must”) is the verb form used to indicate a requirement to be strictly followed to conform to this standard.

For the purposes of this document, the following additional terms and definitions apply.

4.1 client

software component that can invoke an operation from a server

4.2 data clearinghouse

collection of institutions providing digital data, which can be searched through a single interface using a common metadata standard [ISO 19115]

4.3 data level

stratum within a set of layered levels in which data is recorded that conforms to definitions of types found at the application model level [ISO 19101]

4.4 dataset series

collection of datasets sharing the same product specification [ISO 19113, ISO 19114, ISO 19115]

4.5 feature catalogue

catalogue containing definitions and descriptions of the feature types, feature attributes, and feature relationships occurring in one or more sets of geographic data, together with any feature operations that may be applied [ISO 19101, ISO 19110]

4.6 geographic dataset

dataset with a spatial aspect [ISO 19115]

4.7 geographic information

information concerning phenomena implicitly or explicitly associated with a location relative to the Earth [ISO 19128 draft]

4.8 identifier

a character string that may be composed of numbers and characters that is exchanged between the client and the server with respect to a specific identity of a resource

4.9 interface

named set of operations that characterize the behaviour of an entity [ISO 19119]

4.10 metadata dataset

metadata describing a specific dataset [ISO 19101]

4.11 metadata entity

group of metadata elements and other metadata entities describing the same aspect of data

NOTE 1           A metadata entity may contain one or more metadata entities.

NOTE 2           A metadata entity is equivalent to a class in UML terminology [ISO 19115].

4.12 metadata schema

conceptual schema describing metadata

NOTE               ISO 19115 describes a standard for a metadata schema. [ISO 19101]

4.13 metadata section

subset of metadata that defines a collection of related metadata entities and elements [ISO 19115]

4.14 operation

specification of a transformation or query that an object may be called to execute [ISO 19119]

4.15 parameter

variable whose name and value are included in an operation request or response

4.16 profile

set of one or more base standards and - where applicable - the identification of chosen clauses, classes, subsets, options and parameters of those base standards that are necessary for accomplishing a particular function [ISO 19101, ISO 19106]

4.17 qualified name

name that is prefixed with its naming context

EXAMPLE              The qualified name for the road no attribute in class Road defined in the Roadmap schema is RoadMap.Road.road_no. [ISO 19118]

4.18 request

invocation of an operation by a client

4.19 response

result of an operation, returned from a server to a client

4.20 resource

an object or artefact that is described by a record in the information model of a catalogue

4.21 schema

formal description of a model [ISO 19101, ISO 19103, ISO 19109, ISO 19118]

4.22 server

service instance

a particular instance of a service[ISO 19119 edited]

4.23 service

distinct part of the functionality that is provided by an entity through interfaces [ISO 19119]

capability which a service provider entity makes available to a service user entity at the interface between those entities [ISO 19104 terms repository]

4.24 service interface

shared boundary between an automated system or human being and another automated system or human being [ISO 19101]

4.25 service metadata

metadata describing the operations and geographic informationavailable at a server[ISO 19128 draft]

4.26 state

condition that persists for aperiod

NOTE               The value of a particular feature attribute describes a condition of the feature [ISO 19108].

4.27 transfer protocol

common set of rules for defining interactions between distributed systems [ISO 19118]

4.28 version

version of an Implementation Specification (document) and XML Schemas to which the requested operation conforms

NOTE               An OWS Implementation Specification version may specify XML Schemas against which an XML encoded operation request or response shall conform and should be validated.

5. Conventions

5.4 URN notation

All requirements listed in this document are relative to the root URL: http://www.opengis.net/doc/IS/cat/3.0  . Wherever there is a stated requirement and the work “req” is shown, “req” can be replaced with http://www.opengis.net/doc/IS/cat/3.0 to define the complete requirement URL.

6.      Catalogue abstract information model

6.1    Introduction

The abstract information model specifies a BNF grammar for a minimal query language, a set of core queryable[2] attributes (names, definitions, conceptual datatypes), and a common record format that defines the minimal set of elements that should be returned in the brief and summary element sets.

The geospatial community is very broad and works in many different operational environments, as shown in the information discovery continuum in Figure 1 - Information discovery continuum. On one extreme there are tightly coupled systems dedicated to well-defined functions in a tightly controlled environment. At the other extreme are Web based services that know nothing about the client. This document provides a specification that is applicable to the full range of catalogue operating environments.

6.2    Query language support

6.2.1    Introduction

The query capabilities of the OGC General Catalogue Model provide a minimum set of data types and query operations that can be assumed of OGC Compliant Catalogue implementations. In addition, these Query Capabilities provide a high degree of flexibility enabling alternate styles of query, result presentation, and the potential support of any geo-enabled query language. This flexibility is provided by the query operation that contains the parameters needed to select the query result presentation style and to provide a query expression that includes the actual query with an identification of the query language used. The query operation, query expression, and other related operations are further discussed in Clause 7.2.4.

The interoperability goal is supported by the specification of a minimal abstract query (predicate) language, which shall be supported by all compliant OGC Catalogue Services. This query language supports Boolean queries, text matching operations, temporal data types, and geospatial operators. The minimal query language syntax is based on the SQL WHERE clause in the SQL SELECT statement. The OGC Filter Specification is an implementation of a query language that is transformable to the OGC Catalogue Common Query Language (OGC CommonQL).

This minimal query language assists the consumer in the discovery of datasets of interest at all sites supporting the OGC Catalogue Services. The ability to specify alternative query languages allows for evolution and higher levels of interoperability among more tightly coupled communities of Catalogue Service Providers and Consumers.

6.2.2    OGC Catalogue Common Query Language (OGC CommonQL)

This sub-clause defines the OGC_Catalogue Common Query Language (OGC CommonQL) (BNF to be found in 9). OGC_CommonQL is the primary query language to be supported by multiple OGC Catalogue Service bindings in order to support search interoperability.

Assumptions made during the development of OGC CommonQL:

1.  The query will have syntax similar to the SQL “Where Clause.”
2. The expressiveness of the query will not require extensions to various current query systems used in geospatial catalogue queries other than the implementation of some geo operators.
3.  The query language is extensible.
4. OGC CommonQL supports both tight and loose queries. A tight query is defined for the case when a catalogue doesn’t support an attribute/column specified in the query, no entity/row can match the query and the null set is returned. In a loose query, if an attribute is undefined, it is assumed to match.

6.2.3    Extending the OGC CommonQL

The OGC CommonQL BNF can be extended by adding new predicates, operations, and datatypes. The following discussion is an example of extending the BNF to include a CLASSIFIED-AS operator using the patterns identified in OASIS/ebXML Registry Services Specification v2.5. This extension could appear in a protocol binding or an Application Profile.

This standard makes no assumptions about how taxonomies are maintained in a catalogue, or how records are classified according to those taxonomies. Instead, this specification defines a routine, CLASSIFIED-AS, in order to support classification queries based on taxonomies.

The CLASSIFIED-AS routine takes three arguments. The first argument is the abstract entry point whose classification is being checked. The second argument is the key name string that represents a path expression in the taxonomy. The last argument is the key value string that represents the corresponding path expression containing key values that are the targets of the query. In both cases, the first element of the path expression for the key name argument and key value arguments shall be the name of the taxonomy being used. The normal wildcard matching characters, ‘_’ for a single character and ‘%’ for zero or more characters, may be used in the key value expression which is the last argument of the CLASSIFIED_AS routine.

The following set of productions defines the CLASSIFIED-AS routine.

/* The following example:                                            */
/*                                                                   */
/* RECORD CLASSIFIED AS CLASSIFICATIONSCHEME=’GeoClass’  */
/*      =’/GeoClass/North America/%/Ontario’                 */
/*                                                                   */
/* Will find all records in all the Ontario’s in North America.      */

The following are the required BNF specializations:

<classop argument list> ::= <left paren> <entry_point> <comma>
        <Classification Scheme> <comma><Classification Node> <right paren>
 
<entry_point> ::= <identifier>
<Classification Scheme> ::= <identifier>
<classop name> ::= CLASSIFIED_AS
 
<Classification Node> ::= <identifier> | <solidus><path element>[<solidus><path element>]…
<path element> ::= <character pattern>
 
<routine invocation> ::= | <geoop name><georoutine argument list>
                         | <relgeoop name><relgeoop argument list>
                         | <routine name><argument list>
                         | <classop><classop argument list>

Consider the following example:

CLASSIFIED_AS(’RECORD’, ‘GeoClass’, ‘GeoClass/NorthAmerica/%/Ontario’)

In this example, we are searching records classified according to the GeoClass taxonomy. Specifically, we are looking for all catalogue records classified as Continent=NorthAmerica, Country=any country and State=Ontario. Notice how the wildcard character ‘%’ is used to search for any Country node.

Here is the same example encoded using XML:

<ogc:Filter xmlns:ogc=“http://http://www.opengis.net/ogc”>
  <ogc:ClassifiedAs>
     <ogc:TypeName>csw:Record</ogc:TypeName>
     <ogc:ClassificationScheme>GetClass</ogc:ClassificationScheme>
     <ogc:ClassificationNode>/GeoClass/NorthAmerica/%/Ontario
       </ogc:ClassificationNode>
  </ogc:ClassifiedAs>
</ogc:Filter>

In order for catalogue clients to be able to determine which taxonomies are available, a catalogue implementation should advertise the list of available taxonomies in its capabilities document. If a query is executed against a non-existent taxonomy, then an exception should be raised.

6.2.4    Query language realization

Many OGC service operations have the requirement to pass and process a query as a structure to perform a request. There are several query languages and messaging mechanisms identified within OGC standards. Application Profiles should be explicit about the selected query languages and any features peculiar to a scope of application. The following items should be addressed in the preparation of an Application Profile with respect to query language support.

Support for “abstract” query against well-known queryable entry points (e.g. OGC Core). Some standards promote or require the exposure of well-known field-like objects as common search targets (queryables), allowing interrogation of a service without prior negotiation on information content. The mandatory queryable attributes which shall be recognized by all OGC Catalogue Services are discussed in Subclause 6.3.2.

Selection of a query language. Some standards describe one or more query languages that can be supported. Identify the name and version of required query language(s) anticipated by this Application Profile for use.

Supported data types (e.g. character, integer, coordinate, date, geometry) and operator types (e.g. inequality, proximity, partial string, spatial, temporal). Query languages may be restricted in their implementation or extended with functions not described in the base standard. This narrative should provide lists or reference documents with the enumerated data types and operator types required by this Application Profile. In addition, any description of special techniques (e.g. supporting joins or associations) that are expected by an Application Profile should be described.

6.3    Core catalogue schema

6.3.1    Introduction

Metadata structures, relationships, and definitions – known as conceptual schemas – exist for multiple information communities. For the purposes of interchange of information within an information community, a metadata schema may be defined that provides a common vocabulary which supports search, retrieval, display, and association between the description and the object being described. Although this standard does not require the use of a specific schema, the adoption of a given schema within an information-sharing community ensures the ability to communicate and discover information.

The geomatics standardization activity in ISO Technical Committee 211 include formal schemas for geospatial metadata that are intended to apply to all types of information. These metadata standards, ISO 19115:2003[4] and ISO 19115-1:2014[5] include proposals for core (discovery) metadata elements in common use in the geospatial community. ISO/TS 19139:2007 defines a formal encoding and structure of ISO 19115:2003 metadata for exchange. Where a catalogue service advertises such application schemas, catalogues that handle geographic dataset descriptions should conform to published metadata standards and encodings, e.g. ISO 19115:2003, and support XML encoding per ISO 19139 or profiles thereof. Service metadata elements should be consistent with ISO 19119[6] or 19115:2014[7].

6.3.2    Core queryable properties

The goal of defining core queryable properties is query interoperability among catalogues that implement the same protocol binding and query compatibility among catalogues that implement different protocol bindings, perhaps through the use of “bridges” or protocol adapters. Defining a set of core queryable properties also enables simple cross-profile discovery, where the same queries can be executed against any catalogue service without modification and without detailed knowledge of the catalogue’s information model. This requires a set of general metadata properties that can be used to characterize any resource.

Tables 1, 2 and 3 define a set of abstract queryables that binding protocols shall realize in their core queryable schemas. Binding protocols shall further specify a record identifier (ID) based on the native platform ID types. Binding protocols shall also specify how the values of core queryable properties shall be encoded in service requests. Binding protocols may choose to use a single comma-separated list for compound datatypes or may label each sub-element for clarity and order flexibility. Application profiles may further modify or redefine the realization of the core queryables and how their values are encoded.

All realizations of the core queryable properties in a binding protocol shall include all the properties listed in Tables 1, 2, or 3 even if the underlying information model does not include information that can be mapped into all properties. Core properties that cannot have a value assigned to them because the information is not available in the information model of the catalogue shall be considered as having a value of NULL.

The properties “Title”, “Identifier” and the pseudo-property “AnyText” shall be supported as mandatory queryables in all implementations.  Protocol bindings shall describe mechanisms to identify and elaborate on the queryables and operations supported by a given catalogue service.

6.3.3    Core returnable properties

A set of core properties returned from a metadata search is encouraged to permit the minimal implementation of a catalogue service independent of a companion application profile, and to permit the use of metadata returned from different systems and protocol bindings. The core metadata is returned as a request for the Common Element Set. The Common Element Set is a new group of public metadata elements, expressed using the nomenclature and syntax of Dublin Core Metadata, ISO 15836. Table 4 provides some interpretation of Dublin Core elements in the context of metadata for geospatial data and services.

The core elements are recommended for a response but do not need to be populated. The support for a common syntax for the returnable properties as a “common” Summary Element Set is defined in the protocol binding clauses.

<?xml version=“1.0” encoding=“UTF-8”?>
<GetRecords
  service="CSW"
  version="2.0.2"
  maxRecords="5"
  startPosition="1"
  resultType="results"
  outputFormat="application/xml"
  outputSchema="http://www.opengis.net/cat/csw/2.0.2"
  xmlns="http://www.opengis.net/cat/csw/2.0.2"
  xmlns:csw="http://www.opengis.net/cat/csw/2.0.2"
  xmlns:ogc="http://www.opengis.net/ogc"
  xmlns:ows="http://www.opengis.net/ows"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:dct="http://purl.org/dc/terms/"
  xmlns:gml="http://www.opengis.net/gml"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2
                      ../../../csw/2.0.2/CSW-discovery.xsd">
  <Query typeNames="csw:Record">
     <ElementSetName typeNames="csw:Record">full</ElementSetName>
     <Constraint version="1.1.0">
        <ogc:Filter>
           <ogc:And>
              <ogc:PropertyIsLike escapeChar="\" singleChar="?" wildCard="*">
                 <ogc:PropertyName>dc:title</ogc:PropertyName>
                 <ogc:Literal>*Elevation*</ogc:Literal>
              </ogc:PropertyIsLike>
              <ogc:PropertyIsEqualTo>
                 <ogc:PropertyName>dc:type</ogc:PropertyName>
                 <ogc:Literal>Service</ogc:Literal>
              </ogc:PropertyIsEqualTo>
              <ogc:PropertyIsGreaterThanOrEqualTo>
                 <ogc:PropertyName>dct:modified</ogc:PropertyName>
                 <ogc:Literal>2004-03-01</ogc:Literal>
              </ogc:PropertyIsGreaterThanOrEqualTo>
              <ogc:Intersects>
                 <ogc:PropertyName>ows:BoundingBox</ogc:PropertyName>
                 <gml:Envelope>
                    <gml:lowerCorner>14.05 46.46</gml:lowerCorner>
                    <gml:upperCorner>17.24 48.42</gml:upperCorner>
                 </gml:Envelope>
              </ogc:Intersects>
           </ogc:And>
        </ogc:Filter>
     </Constraint>
  </Query>
</GetRecords>

The response to such a query, might be:

<?xml version=“1.0” encoding=“UTF-8”?>
<csw:Record
  xmlns:csw="http://www.opengis.net/cat/csw/2.0.2"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:dct="http://purl.org/dc/terms/"
  xmlns:ows="http://www.opengis.net/ows"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.opengis.net/cat/csw/3.0.0
                      ../../../csw/3.0.0/record.xsd">
   <dc:creator>U.S. Geological Survey</dc:creator>
   <dc:contributor>State of Texas</dc:contributor>
   <dc:publisher>U.S. Geological Survey</dc:publisher>
   <dc:subject>Elevation, Hypsography, and Contours</dc:subject>
   <dc:subject>elevation</dc:subject>
   <dct:abstract>Elevation data collected for the National Elevation Dataset (NED) based on 30m horizontal and 15m vertical accuracy.</dct:abstract>
   <dc:identifier>ac522ef2-89a6-11db-91b1-7eea55d89593</dc:identifier>
   <dc:relation>OfferedBy</dc:relation>
   <dc:source>http://myserver.com/csw? SERVICE=CSW&REQUEST=GetRecordById&RECORD=dd1b2ce7-0722-4642-8cd4-6f885f132777</dc:source>
   <dc:rights>Copyright © 2011, State of Texas</dc:rights>
   <dc:type>Service</dc:type>
   <dc:title>Elevation Mapping Service for Texas</dc:title>
   <dct:modified>2011-03-01</dct:modified>
       <dc:language>en</dc:language>
   <ows:BoundingBox>
      <ows:LowerCorner>-108.44 28.229</ows:LowerCorner>
      <ows:UpperCorner>-96.223 34.353</ows:UpperCorner>
   </ows:BoundingBox>
</csw:Record>

6.3.4    Information structure and semantics

Some services that implement OGC Standards expect a rigid syntax for the information resources to be returned, whereas others do not. This subclause allows an Application Profile to be specific about what information content, syntax, and semantics are to be communicated over the service. The following items should be addressed in an Application Profile.

1.  Identify information resource types that can be requested. In the case of a catalogue service, the information resources being described by the metadata may include geographic data, imagery, services, controlled vocabularies, or schemas among a wide variety of possible types. This subclause allows the community to specify or generalise the resource types being described in metadata for their scope of application.
2. Identify a public reference for the information being returned by the service (e.g. ISO 19115:2003 “Geographic Information – Metadata “). Include any semantic resources including data content model, dictionary, feature type catalogue, code lists, authorities, taxonomies, etc.
3.  Identify named groups of properties (element sets) that may be requested of the service (e.g. “brief,” “summary,” or “full”) and the valid format (syntax) for each element set. Identify valid schema(s) with respect to a given format to assist in the validation of response messages.
4. Specialise the core queryable properties list by making some optional queryable attributes mandatory, deleting other optional attributes and adding queryable attributes that should be standard across all profile users
5.  Optional mapping of queryable and retrievable properties against other public metadata models or tags.
6.  Expected response/results syntax and content Message syntax and schemas (e.g. brief/full, individual elements).

7.      General catalogue interface model

7.1    Introduction

The General Catalogue Interface Model (GCIM) provides a set of abstract service interfaces that support the discovery, access, maintenance and organization of catalogues of geospatial information and related resources. The interfaces specified are intended to allow users or application software to find information that exists in multiple distributed computing environments, including the World Wide Web (WWW) environment. 

Implementation design guidance is included in specified protocol binding Parts of this standard. Each protocol binding includes a mapping from the general interfaces, operations, and parameters specified in this clause to the constructs available in a chosen protocol. In most, but not all, protocol bindings, there may be restrictions or refinements on implementation of the General Model agreed within an implementation community. This sub-clause provides an overview of the portions of the GCIM that are realised by implementations described in other Catalogue Service Part documents.

Application profiles are intended to further document implementation choices. An Application Profile is predicated on the existence of one protocol binding as a Part of this standard.

Figure 2 - Reference model architecture shows the Reference Architecture assumed for development of the OGC Catalogue Interface. The architecture is a multi-tier arrangement of clients and servers. To provide a context, the architecture shows more than just catalogue interfaces. The bold lines illustrate the scope of OGC Catalogue.

The Application Client shown in Figure 2 - Reference model architecture interfaces with the Catalogue Service using the OGC Catalogue Interface. The Catalogue Service may draw on one of three sources to respond to the Catalogue Service request: a Metadata Repository local to the Catalogue Service, a Resource service, or another Catalogue Service. The interface to the local Metadata Repository is internal to the Catalogue Service. The interface to the Resource service can be a private or OGC Interface. The interface between Catalogue Services is the OGC Catalogue Interface. In this case, a Catalogue Service is acting as both a client and server. Data returned from an OGC Catalogue Service query is processed by the requesting Catalogue Service to return the data appropriate to the original Catalogue request. See Annex A for more about Distributed Searching.

7.2    Interface definitions

7.2.1    Overview

Figure 3 - General OGC catalogue UML static model is a general UML model of OGC catalogue service interfaces, in the form of a class diagram. Operation signatures have been suppressed in this figure for simplicity but are described in detail below. This model shows the Catalogue Service class plus five other classes with which that class are associated. A Catalogue Service is a realization of an OGC Service. Each instance of the Catalogue Service class is associated with one or more of these other classes, depending on the abilities included in that service instance. Each of these other classes defines one or several related operations that can be included in a Catalogue Service class instance. The Catalogue Service class directly includes only the serviceTypeID attribute, with a fixed value for the service type.

In Figure 3 - General OGC catalogue UML static model, an instance of the CatalogService type is a composite object that is a high-level characterization of a catalogue service. Its constituent objects are themselves components that provide functional behaviours to address particular areas of concern. A protocol binding may realise specific configurations of these components to serve different purposes (e.g. a read-only catalogue for discovery, or a transactional catalogue for discovery and publication).

The associated classes shown in this figure are mandatory or optional for implementation as indicated by the association multiplicity in the UML diagram. Therefore, a compliant catalogue service shall implement the OGC_Service, CatalogService, and Discovery classes. An application profile or protocol binding can implement additional classes associated with the Catalogue Service class. A catalogue implementation shall recognise all operations defined within each included class, and shall generate a message indicating when a particular operation is not implemented.

The protocol binding clauses of this standard provide more detail on the implementation of these conceptual interfaces. For example, the names of the classes and operations in this general UML model are changed in some of the protocol bindings. The names of some operation parameters are also changed in some protocol bindings.

Application Profiles may further specialise the implementation of these interfaces and their operations, including adding classes. In general, however, the interfaces and operations described here shall have the same semantics and granularity of interaction regardless of the protocol binding used.

The Catalogue Service class can be associated with the following classes.

a)     OGC_Service class, which provides the getCapabilities operation that retrieves catalogue service metadata and the getResourceById operation that will retrieve an object by query on its identifier only. This class is always realised by the Catalogue Service class, and is thus always implemented by a Catalogue Service implementation.

b)    Discovery class, which provides three operations for client discovery of resources registered in a catalogue. This class has a required association from the Catalogue Service class, and is thus always implemented by a Catalogue Service implementation. The “query” operation searches the catalogued metadata and produces a result set containing references to all the resources that satisfy the query. This operation returns metadata for some or all of the found result set. The optional describeRecordType operation retrieves the type definition used by metadata of one or more registered resource types. The optional getDomain operation retrieves information about the valid values of one or more named metadata properties.

c)     Manager class, which provides two operations for inserting, updating, and deleting the metadata by which resources are registered in a catalogue. This class has an optional association from the Catalogue Service class; this interface is implemented by the Catalogue Service implementation. The transaction operation performs a specified set of “insert”, “update”, and “delete” actions on metadata items stored by a Catalogue Service implementation—this enables a “push” style of publication. The harvestResource operation requests the Catalogue Service to retrieve resource metadata from a specified location, often on a regular basis—this behaviour reflects a ‘pull’ style of publication.

The three classes associated with the Catalogue Service class allow different OGC catalogue services to provide significantly different abilities. A particular protocol binding is used by each Application Profile and a particular set of these catalogue service classes is specified by each Application Profile.

Each of the catalogue classes is described further in the following subclauses. These subclauses discuss the operations and parameters of each operation in this general model. Specific protocol bindings or application profiles can define additional parameters. For example, the HTTP Protocol Binding adds the Service, Request, and Version parameters to all operation requests to be consistent with other OGC Web Services.

7.2.2    Catalogue Service class

The Catalogue Service class provides the foundation for an OGC catalogue service. The Catalogue Service class directly includes only the serviceTypeID attribute, as specified in Table 5. In most cases, this attribute will not be directly visible to catalogue clients.

7.2.3    OGC_Service class

7.2.3.1    Introduction

The OGC_Service class allows clients to retrieve service metadata by providing the getCapabilities operation. This class is always realised by the Catalogue Service class, and is thus always implemented by a Catalogue Service instance. Capabilities are described further in OGC Web Service Common Implementation Specification 2.0.

NOTE               This getCapabilities operation corresponds to CatalogueService.explainServer operation in OGC Catalogue version 1.1.1.

7.2.3.2  getCapabilities operation

The getCapabilities operation is specified in Table 6.

The getCapabilities operation is inherited from OWS Common 2.0 and is specialized to describe service capabilities of a catalogue.

The normal GetCapabilities operation response is a service metadata document that includes the “section” attributes listed and defined in Table 7, as selected by the “section” attribute in the operation request.

NOTE 1           The term “Capabilities XML” document was previously used for what is here called “service metadata” document. The term “service metadata” is now used because it is more descriptive and is compliant with OGC Abstract Specification Topic 12 (ISO 19119).

NOTE 2           This general model assumes that operation failure will be signalled to the client in a manner specified by each protocol binding.

7.2.3.3    getResourceById operation

The getResourceById operation is inherited from OWS Common and supports the request of one or more resources – in this case full, structured metadata records – from the catalogue. Records are discovered through the query operation whose response includes the identifier(s) of the record(s) meeting the conditions of the query. These identifiers are passed via the getResourceById to retrieve records from the catalogue in bulk.

7.2.4    Discovery class

7.2.4.1    Introduction

The Discovery class allows clients to discover resources registered in a catalogue, by providing three operations named “query”, describeRecordType, and getDomain. This class has a required association from the Catalogue Service class, and is thus always implemented by all Catalogue Service implementations. All Discovery class operations are stateless.

7.2.4.2     “query” operation

NOTE               This general model assumes that operation failure will be signalled to the client in a manner specified by each protocol binding.

7.2.4.3    describeRecordType operation

The describeRecordType operation is more completely specified in Table 15 — Definition of describeRecordType operation. 

Table 15 provides a UML model of the optional describeRecordType operation that shows the complete Discovery class with the DescribeRecordTypeRequest and DescribeRecordTypeResponse classes and the class they use. The operation request includes the attributes and association role name listed and defined in Table 16. The normal operation response includes the attributes and association role name listed and defined in Table 17.

NOTE               The describeRecordType operation corresponds to CG_Discovery.explainCollection operation in OGC Catalogue version 1.1.1.

7.2.4.4    getDomain operation

The optional getDomain operation is more completely specified in Table 18, which provides a UML model of the getDomain operation that shows the complete Discovery class with the GetDomainRequest and GetDomainResponse classes and the class they use. The operation request includes the attributes listed and defined in Table 19. The normal operation response includes the attributes and association role name listed and defined in Table 20.

7.2.5    Manager class

7.2.5.1    Introduction

The Manager class allows a client to insert, update and/or delete catalogue content. This class has an optional association from the CatalogueService class; it is not required that a catalogue service implement publishing functionality. Two operations are provided: “transaction” and “harvestResource”. Both are optional operations.

The “transaction” operation allows a client to formulate a transaction, and send it to the catalogue to be processed. The transaction may contain metadata records and elements of the information model that the catalogue understands. To use the transaction operation, the client must know something about the information model that the catalogue implements.

The “harvestResource” operation, on the other hand, directs the catalogue to retrieve an accessible metadata record and processes it for inclusion in the catalogue, perhaps periodically re-fetching the metadata records to refresh the information in the catalogue. The client does not need to be aware of the information model of the catalogue when using the “harvestResource” operation, since the catalogue itself is doing the work required to process the information. The client is simply pointing to where the metadata resource to be harvested is.

7.2.5.2    ”transaction” operation

The “transaction” operation is more completely specified in Table 21. Figure 7 provides a UML model of the “transaction” operation that shows the complete Manager class with the TransactionRequest and TransactionResponse classes and the classes they use.The operation request includes the attributes listed and defined in Table 22. The normal operation response includes the attributes listed and defined in Table 23.

7.2.5.3    harvestResource operation

The harvestResource operation facilitates the retrieval of remote resources from a designated location and provides for optional transactions on the local catalogue. The harvestResource operation is described in Table 24. Figure 8 provides a UML model of the “harvestResource” operation that shows the complete Manager class with the HarvestResourceRequest and HarvestResourceResponse classes.The operation request includes the attributes listed and defined in Table 25. The normal operation response includes the attributes listed and defined in Table 26.

This general model assumes that operation failure will be signalled to the client in a manner specified by each protocol binding.

8.      Conformance classes and specialisation

8.1    Introduction

This subclause provides an overview of the core elements of the General Catalogue Model and how these may be used in protocol bindings and application profiles.

The General Catalogue Model consists of an abstract model and a General Interface Model. The abstract query model specifies a BNF grammar for a minimal query syntax and a set of core search attributes (names, definitions, conceptual datatypes). The General Interface Model specifies a set of service interfaces that support the discovery, access, maintenance and organization of catalogues of geospatial information and related resources; these interfaces may be bound to multiple application protocols, including the HTTP protocol that underlies the World Wide Web.

Implementations are constrained by the protocol binding parts of this standard, which depend on this general model. Each protocol binding includes a mapping from the general interfaces, operations, and parameters specified in this clause to the constructs available in a chosen protocol. Application profiles are intended to further document implementation choices.

An Application Profile is based on one of the protocol bindings in the base specification. In the case of the Catalogue Services Standard, a profile should reference the HTTP/1.1 protocol binding unless others are defined/recognized. In most, but not all, protocol bindings, there may be restrictions or refinements on implementation agreed within an implementation community. A graphic model of the relationships is shown in Figure 9.

8.2    List of requirements and conformance classes

The general model conformance classes are presented in Table 27 — Catalogue Service Requirements (Conformance Classes) along with the corresponding requirement classes.  These conformance classes shall be realized in the protocol binding parts of this standard. Test suites are to be included in protocol binding parts of the specification that reference these general conformance clauses (requirements).

The following conformance classes are recommended for Catalogue Services, with respect to the Requirements listed within this standard.

8.3    Interface definitions

The various elements of the General Catalogue Interface Model provide functional behaviours and capabilities to address particular areas of concern. A protocol binding may realise specific configurations of these components to serve different purposes (e.g. a read-only catalogue for discovery, a transactional catalogue for discovery and publication).

A compliant protocol binding of the catalogue service is required to implement the OGC_Service, Catalogue Service, and Discovery classes. A protocol binding may also include any of the optional classes associated with the Catalogue Service class. A compliant implementation of a protocol binding shall recognise all operations defined within each class included in the protocol binding, and shall generate a service exception report indicating when a particular operation is not implemented (in such cases the operation is abstract—an implementation is not required).

The protocol binding parts of this standard provides more detail on the implementation of the general interfaces. In effect, each binding maps these interfaces to a particular application protocol. For example, the names of the classes and operations in this general UML model are changed in some of the protocol bindings. The names of some operation parameters are also changed in some protocol bindings. However, the interfaces and operations specified in all Protocol Bindings shall be consistent with the semantics and granularity of interaction specified in the General Interface Model.

Application profiles, which will appear as separate documents may further specialise the implementation of these interfaces and their operations, including adding classes and parameters. However, the application profile is a specialization of the parent protocol binding, in that the names of the operations and the parameters cannot be changed.

8.4    Query model components

8.4.1    Query language/model

Many OGC service operations have the requirement to pass and process a query as a structure to perform a search. There are several query languages and messaging mechanisms identified within OGC standards. Binding protocols and application profiles should be explicit about the selected query languages and any features peculiar to a scope of application. The following items should be addressed in specialization of a Protocol Binding or an Application profile with respect to query language support.

1.   Support for “abstract” queries, against well-known access points (e.g. core search properties). Some standards promote or require the exposure of well-known field-like objects as common search targets (queryables), allowing interrogation of a service without prior negotiation on information content. The mandatory queryable attributes which shall be recognised by all OGC Catalogue Services is discussed in Subclause 6.3.
2. Selection of a query language. Identify the name and version of required query language(s) anticipated by this Protocol Binding or Application Profile for use.
3.   Supported data types (e.g., character, integer, coordinate, date, polygon) and operator types (e.g., inequality, proximity, partial string, spatial, temporal). Query languages may be restricted in their implementation or extended with functions not described in the base specification. This would need to be done if the base query language did not support a data type required by the OGC CommonQL discussed in Clause 6 such as envelope.

In addition, an application profile may extend the OGC CQL or Filter syntax with functions not described in the base specification through use of the “function “construct in CQL and the “Filter“ language. If an application protocol uses this extension method, the profile documentation should include an updated BNF grammar in addition to lists or reference documents with the enumerated data types and operator types required by this Application Profile. In addition, any description of special techniques (e.g. supporting joins or associations) that are expected by an Application Profile should be described.

8.4.2    Common search and retrieval elements

The abstract information model is discussed in Clause 6; this model consists of a small set of abstract search elements and the specification of a common “summary” element set to allow queries across protocol bindings and even from outside the OGC domain. Each Protocol Binding should specialize this model by:

1.   Specify the syntax of the globally unique Identifiers including any registration authority information
2. Map the core search (queryable) elements into a concrete syntax based on the chosen record format(s)
3.   Define a “summary” element set that corresponds to the “summary” element set in the Catalogue general model

An application profile is expected to fully specify the conceptual information model adopted by the user community. This process and resulting artefacts are further discussed in Subclause 6.2.5 and the remainder of this clause.

8.5    Catalogue Application Profiles

ISO TR 10000-1:1998 describes a general framework for functional standardization and defines the concept of a profile. A profile identifies the use of particular options available in one or more base standards and it also provides a basis for developing conformance tests; a compliant profile shall not contradict the base specifications or otherwise give rise to non-conforming conditions. An application profile specifies the use of an application-layer protocol (e.g. HTTP/1.1) in order to provide for the structured transfer of information between systems (ISO/IEC TR 10000-2:1998).

A catalogue application profile binds a set of functional components (with interfaces specified as part of a protocol binding) to an abstract information model—expressed using UML—that has one or more concrete representations of catalogue content. Each representation is an Internet media type that conforms to a schema defined using some schema language (e.g., ASN.1, XML Schema, RDF Schema). An application profile specifies a set of functional components that are provided by a conforming implementation (Figure 10).

An application profile is derived from one or more base specifications in order to address particular needs or requirements. The general OGC catalogue model defines common behaviours and interfaces that have general utility, but in practice there is no single solution that fits everyone’s needs. Catalogue application profiles specify refinements or extensions that are targeted toward specific implementation communities; for these communities it is the application profile that represents the standard for conformance. Following ISO 19106, a Level 1 profile is defined as a pure subset of one or more ISO standards; a Level 2 profile includes allowable extensions and may also depend on non-ISO standards.

Clause 10 in the ISO 19119 standard distinguishes platform-neutral from platform-specific specifications and assumes that one of the former will constitute the basis for one or more of the latter. That is, a single platform-neutral specification will give rise to multiple platform-specific specifications each of which is bound to a particular distributed computing protocol (i.e. HTTP). The OGC catalogue framework upholds this basic distinction: the general interface model is a platform-neutral description of catalogue operations; each application profile is platform-specific—it makes use of one of the protocol bindings defined in the catalogue standard.

Note that in Figure 10 - Application profiles specify concrete catalogue services application profiles will reflect differing degrees of “thickness”. For example, if a profile employs a very simple conceptual model that embodies a limited set of simple properties then its ‘native’ representation may include little more than the common search and retrieval elements. Profiles that utilize more sophisticated models will define a native representation that provides more information; in this case the common search and retrieval elements shall be mapped to the catalogue information model.

8.6    Structure and format

8.6.1    Introduction

All application profiles shall be structured as shown in Table 28.  This organization complies with clause 12.3 of ISO 19106 (Geographic information – Profiles). A profile may introduce additional (sub)clauses as required.

Clauses 6 through 8 convey the particulars of the application profile in terms of three ‘views’ (these correspond to the following standard ODP viewpoints: Enterprise, Information, and Computational). The three views describe various aspects of the catalogue service with respect to the base specifications; taken together they constitute the basic application architecture. The essential content of these views is summarized in the following subclauses; additional guidance can be found in the annotated profile template (OGC Document 03-101).

8.6.2    System context

This view focuses on the purpose, scope, and policies of the catalogue service (i.e., what is the system used for). It documents special requirements[8] and describes the context of use as suggested in Table 29.

8.6.3    Information models

This view primarily focuses on the information structures and the semantics of information processing (i.e., what the system is about); it describes the public information model that is employed by the catalogue service. The syntax for all supported representations of the catalogued resources shall also be defined (Table 30).

A section describing the MIME-types to be used is mandatory for any standard involving data encodings. If no suitable MIME type exists in http://www.iana.org/assignments/media-types/index.html then this section may be used to define a new MIME type for registration with IANA.

8.6.4    External interfaces

This view primarily focuses on documenting the externally visible behaviour of the system, including the interfaces provided by its components and the supported protocol binding(s). This view shall define the request and response message structures as part of the operation signatures; it also documents supported query facilities and any relevant security considerations (Table 31). Most of the request and response message elements are imported with the protocol binding, but a (Level 2) profile may introduce extensions to meet more specialized requirements.

The inclusion of a UML diagram is recommended to provide an overview of the interfaces provided by a given service type, where each type provides a different—perhaps overlapping—set of interfaces (e.g. a read-only catalogue, a catalogue that allows a ‘push’ style of publication).

8.7    Compliance

A compliant application profile shall:

1.  Include the (sub)clauses indicated in Table 31 (additional clauses MAY also be included);
2. Define the supported catalogue information model using UML as the conceptual schema language;
3.  Define a set of mappings for the common XML record format data format;
4. Specify the ‘native’ representation of information model elements (additional representations MAY also be specified);
5.  Define any extensions to the imported protocol binding;
6.  Indicate how the elements of the general model are related to the corresponding elements of the profile-specific interfaces; and
7.  Include a conformance test suite (web-based services can do so using the OGC CITE notation).

Annex A - BNF Definition of OGC CommonQL (Normative)

A.1 Introduction

Federation” is “a concept in information technology referring to the lack of central authority over software design or configuration[9].” For the purposes of this standard a catalogue federation can be defined as a loosely coupled union of catalogues that share some common characteristics regarding their content.

Figure 11 identifies some of the relationships between individual OGC Catalogues in a catalogue federation. A “star” symbolizes a Catalogue instance. Federations A, B, C, and D are made up of [2..N] Catalogue instances (only a few are pictured). A Catalogue instance may exist outside any federation, or may be a member of multiple federations.

• A Catalogue instance may be a member of a single federation e.g., the blue instance is a member of Federation A.
• A Catalogue instance may be a member of two federations, one of which is a subset of the other e.g., the red instance is a member of Federation A and its subset Federation B.
• A Catalogue instance may be a member of two federations which share a union set of members e.g. the green instance is a member of Federations C and D.

A.2 Distributed search alternatives

In general, there exist minimally the following options for a distributed search on a catalogue federation.

A.3 Search controlled by catalogue client

The client derives the catalogue topology (the federation) behind one or more known catalogue servers by recursively discovering the “federated catalogues sections” of their capability documents and collecting all the catalogues within the federation. Then the client controls the searches on the catalogues itself.

•   Disadvantages:
  •   Every client has to determine the catalogue topology from time to time.
  •   The search control must be processed by every client (it is not transparent to the client).
  •   Catalogues which are not directly accessible (e.g. running behind a firewall in an intranet) cannot be accessed.
•   Advantages:
  •   Search control can be processed by the client: so the client can decide by its own how the search is operated.
  •   The response time of a single search request may be more predictable as no hidden requests to third party catalogues are involved.

A.3.1    Search controlled by catalogue server

Distributed Search in this sense allows for a Catalogue Server to accept a request from a client and distribute the request to other Catalogues within a federation. A Catalogue is acting as both: as a server and as a client (for another Catalogue – see Figure 13).

A catalogue can propagate a search request to 0, 1 or N other catalogues within the federation and the distributed catalogues can forward the request to 0, 1 or N other catalogues as well. Data returned from a Catalogue query is processed by the requesting Catalogue to return the data appropriate to the original Catalogue request. With that it becomes possible for a client to start a search from only one known location and to search as many catalogues as possible with the same filter statement. In this case, the metadata entries managed by the other catalogues become available to their own clients.

•   Disadvantages:
  •   More enhanced query request- and response-structures needed.
  •   Search control must be processed by every catalogue server (which provides access to federated catalogues).
  •   The response time for a single request may be less predictable as possibly hidden requests to (potentially slow) third party catalogues are involved[10].
  •   For consecutive requests (belonging together) the catalogue server must handle some state information.
•   Advantages:
•   The catalogue must only to know its direct “child-catalogues.”
•   Catalogues behind a firewall can be accessed.
•   Search control has not to be processed by every client.

In the following we assume a distributed search controlled by a catalogue server.

A.4         Distributed search preconditions

To enable Distributed Searching, the following preconditions must be assured.

•   A multi-tier Reference Architecture as defined in Figure 14.
•   A data model to define to which federated catalogue server searches can be distributed. A Catalogue may specify those federated catalogue servers in its capabilities document. For a specific catalogue server these are at a maximum all catalogues to which a query will be distributed if no restriction to specific federated catalogues is defined by the client (see below).
•   A data model to define how searches are distributed to the federated catalogue servers: The discovery request and response messages define elements that allow for the retrieval and comprehension of a distributed result set. The request and response messages contain elements that allow for understanding the status of distributed searches. These elements (which were already introduced in the general interface model) are explained in the following sections.

For a substantiation and for a better understanding how problems of distributed searches are solved by these elements and how distributed searches can be implemented, the elements used in the following sections pertain to the HTTP protocol binding.

A.5         Support of Distributed Search within Discovery Request Messages

The GetRecords message contains elements that allow the client to request certain search behaviour with respect to distribution.

The main parameter is DistributedSearch.

• The default query behaviour, if the DistributedSearch parameter is set to FALSE (or the parameter is not available), is to execute the query on the local server.
• A DistributedSearch parameter set to TRUE (or if the distributedSearch substructure is available in a request) indicates that the query should be distributed. In general all catalogues within the federation would be searched upon. Effectively the number and range of requested result items (defined by the entries already found and the maxRecords[11]-andstartPosition-parameters) would limit the search on a few catalogues within the federation.

Distributed searches can cause specific problems that are addressed within this catalogue interface standard. Some problems result from:

• First: the possibility that within a single distributed search the amount of approached catalogue service nodes is very large, causing long response times; and
• Second: the catalogue service node may be approached multiple times, resulting in closed loops causing the whole distributed system to potentially fail. In this case the loop causes infinite recursion – the same query is sent again and again resulting in system failure and/or timeout.

A third potential problem is the duplication of results when a catalogue service node may be approached multiple times. This is especially hard to detect when searching different catalogues in parallel, as the parallel searches have no idea what entries are already in the parallel resultSets. Figure 15 (see also the green node contained in both Fed.C and Fed.D in Figure 11) displays a case resulting in duplicates due to the same catalogue service node being queried twice.

Unnecessary duplicates are a nuisance but do not normally cause the system to fail.

A method to discover if any of the problems may occur would be to control the network topology manually. Before a query is issued, the query topology is checked for duplicates or loops.

It is important to notice that the problems depicted so far can be solved by restricting the search hierarchy to two levels: a client queries a catalogue services which is allowed to cascade once. Therefore the hopCount parameter was introduced (see Figure 16 - Extended search request structure). With the hopCount parameter of the discovery request message a specific control is put in place to:

• prevent closed-loops of searches; and
• terminate propagation when a certain number of catalogues have been reached.

If the value of the hopCount parameter is greater than 2[12] it cannot prevent the same catalogue service node being queried twice.

To allow an automatic solution to this problem an idea would be to track the nodes already accessed: a cascading catalogue service would make sure that the list of already accessed nodes of the query gets added its own identifier (URI). But this approach does not solve the problem when searching different catalogues in parallel. A better solution would be that every catalogue server must store the unique requestId[13] for every request already processed. In this case a catalogue server can decide if a request (represented by its globally unique requestId, i.e. a UUID) was already processed and does not process the query once again. The server would return an empty result set if the request has been seen before. If the client did not include the requestId in the request, the first catalogue server accessed should generate a unique one.

Example:

ServerA cascades (in parallel) the client’s request (with the requestId 1013) to Servers B and C. Servers B and C process the request and each cascade the request to Server D. Server D will process the first request and sends a full response to the client. On the second request Server D detects that a request with id 1013 was already processed and sends an empty response thus preventing duplication.

Another difficulty is caused by duplicate metadata entries in a resultset that are served by different catalogue servers. But this is not really a problem because every metadata entry is uniquely addressed by the catalogue URL-prefix^[14] of the Catalogues getCapabilities HTTP-GET operation from which it originates plus its identifier (which must be unique within the catalogue). So every metadata entry contained in the resultset of a distributed search can be accessed unambiguously by a subsequent getRecordById call on the catalogue which address is defined by its URL-prefix of it´s getCapabilities HTTP-GET operation and defining the Id (which is additionally included in the resultset).

Other problems are as follows:

• a client does not always want to search all catalogues which are listed within the FederatedCatalogues sections of a catalogue server’s capabilities documents; and
• it is not possible to control the timeouts of searches on federated catalogues.

To restrict the number of catalogues of a federation which should be searched upon in a distributed query an optional list of those catalogues can be provided within the federatedCatatalogues parameter (see Fig 34) of the discovery request message. Every catalogue is represented in the list by its url as defined within the FederatedCatalogues constraint of the capabilities document. For every catalogue in this list an optional timeout definition (in msec) can be provided within the timeout parameter.

On forwarding the request to a federated catalogue the catalogue should remove its own url from the list of the targeted catalogues (federatedCatalogues).

A big problem for a catalogue server within a server controlled distributed search is to assure that when the client repeats the same request (including the same targeted federated catalogues, the same filter statement and restricting the results by the same startPos and maxRecords values and under the condition that the content of the catalogues has not changed meanwhile) the resultset should be the same. Only if this is guaranteed it can be assured that a client is able to iterate block by block (defined by startPos/maxRecords) through the complete resultset.

If the catalogue servers should additionally have the possibility to provide the fastest results on every request or every sequence of requests that belong together, the protocol must provide the means to relate requests together. For example, with these means it is possible for a catalogue server to temporarily store results, re-order a predefined sequence of requested catalogues involved in a distributed search, record which results were delivered to which client within a distributed, etc. The means to enable this is the introduction of the following three conditional request parameters (which become mandatory in the case of a distributedSearch, see Figure 17:

• distributedSearchId: an Id which uniquely identifies a complete client initiated distributed search sequence/session;
• clientId: an Id which uniquely identifies the requestor; and
• distributedSearchIdTimout: defines how long (in sec) the distributedSearchId should be valid, meaning how long a server involved in distributed search should minimally store information related to the distributedSearchId.

The disadvantage of the introduced distributedSearchId is that the interface becomes more stateful.

Example:

The following example should clarify how a distributed search with these means can correctly be processed (see Figure 18). The scenario consists of a client (clientId: ‘Client’) and a network of 6 distributed catalogues which are involved in the distributed search. The client runs a distributed search session (distributedSearchId=’123’) that consists of a sequence of 4 requests. All requests include the same filter (for a new filter a new distributed search session would have to be initiated). The filter is not further considered here. The maximum hopCount is set to ‘4’, the list of federated catalogues (CSW) that should be considered beside “A” is [“B1”,”C1”,”B2”,”B3”,”C2”].

The bigger circles in Figure 17 mark the catalogue server nodes itself. The smaller circles alongside of each catalogue node describe how much result items can be provided by the catalogue server concerning the filter statement for the request. The rectangles mark how fast (relative to the other CSWs) the responses of each catalogue are provided. The arrows mark the direction of the searches. Above and below the arrows you´ll find the requestId, startPos, maxRecs and clientId parameter values of the requests. In the boxes you can see the information that is stored locally by a catalogue server to correctly execute subsequent requests. 

The client sends ‘request1’ of distributedSearch ‘123’ with startPos=’1’ and maxRecs=’10’ to CSW A. Because CSW A can only contribute 5 records to the resultset it starts a search on its associated CSWs. Usually A would deliver the results of its associated catalogues in a predefined order (B1, B2, B3). To be able to deliver the fastest results first CSWA starts the searches on B1, B2 and B3 in parallel threads and checks which results are first available. If e.g. B3 is the fastest responding catalogue, CSW A integrates the results of B3 first (in case of request1 5 items: 4 from B3, 1 from C2). Therefore request1 is already satisfied and A returns the 10 result items. For request1 B3 provides only 4 local items and started a distributed search on its associated catalogue C2 with startPos=’1’ and maxRecs=’1’. CSWC2 delivered 1 item to B3 which is a subset of C2’s resultset. CSWC2 and CSWB3 stored the information which result items were delivered to which client.

To allow the client to iterate through the complete resultset A must memorize e.g. the order of the distributed catalogues used within distributedSearch ‘123’ (order now: B3, B1, B2) and/or memorize which result block (startPos, maxRecs) originates from which distributed catalogue. It may also optionally cache results returned from the distributed catalogues.

For request2 CSW A can derive (by the distributedSearchId) that B3already delivered its result items 1-5 (resultset status: subset) and starts now requesting CSW B3 for the results with startPos=’6’ and maxRecs=’10’. CSW B3 checks that it already delivered its complete items (1-4) as well as the first item of C2 to A. As B3 knows that C2 did not deliver its full results it tries to search with startPos=’2’ and maxRecs=’10’. But CSW C2 could only deliver 7 items which are now bookmarked by B3 and (later) returned to CSW A. A checks that 7 items (but not 10) where delivered, memorizes this and starts a search with startPos=’1’ and maxRecs=’3’ (3 items are still missed for request2) on B1 (which is number 2 in the re-ordered list of associated catalogues of CSW A). CSW B1could only provide 1 local item and so starts a distributed search on its associated catalogue C1 with startPos=’1’ and maxRecs=’2’. CSW C1 could not provide any item and starts a distributed search with startPos=’1’ and maxRecs=’2’ on its associated catalogue B3. Now comes the newly introduced parameter clientId into play: CSW B3 detects by the clientId that within distributedSearch session ‘123’ the results 1-2 where already delivered to Client CSW A and denies the response of any items. Because in this case the whole subtree below CSW  B1 could only provide 2 items CSW A must start a distributed search with startPos=’1’ and maxRecs=’2’ on B2 which delivers the two items (resultset of CSW B2: subset) and so on….

The scenario described in this example is only one possible implementation. Others may cache parts of the results, do not request their associated catalogues in a steady order, harvest the whole content of their associated catalogues, and so forth.

Another point to consider is that in the scenario explained so far the clients wait as long as all requested (available) items are delivered. This makes the client implementation easier. It does also imply that the status of every response is either complete (no more items are available) or subset (more items are available). To further speed up distributed searches the catalogue servers may deliver partial resultsets while further trying to acquire the outstanding result items. In an appropriate scenario CSW A could deliver the first 5 items and define the status of the response message as processingmeaning that for faster response the requested number of items was not fully returned although more items are expected in the requested startPosition/maxRecords range. In this case the client can already display the first 5 items (although requested 10) and the server can continue with processing the distributed searches and temporary caching those which are already acquired. In the next step the client must adjust its request: next n records starting at position 6…. Consequence of this is that clients must generally check the response status and if necessary adjust their startPos and maxRecs request parameters.

For further speeding up the processing it is strongly recommended to not query for hitCount in the case of a distributed search as this would start a full distributed search slowing down the search speed.

A.6         Support of distributed search within discovery response messages

The results of every catalogue involved in a distributed search result are grouped within the federatedSearchResult element (which is of type FederatedSearchResultType) of the searchResults. Every federatedSearchResultelement includes the catalogueURL (the URL-prefix[15] of the getCapabilities HTTP-GET operation of the catalogue). This URL is also required for a subsequent getRecordByID request to be sent by the client.

Further the federatedSearchResultelement again includes an element of the type SearchResultsType  (see Fig. xx) so that trees of results can be described. Important information of the SearchResultsType with regard to federated search are:

• The result items (resultEntry);
• information how many results are delivered by the catalogue(numberOfRecordsReturned);
• information how many results are matched within the catalogue regarding the request (numberOfRecordsMatched); and
• runtime information of the search within the federated catalogue (elapsedTime).

If a federated catalogue has thrown an exception an entry of type FederatedExceptionTypeis included instead of type FederatedSearchResultType. FederatedExceptionTypeincludes the URL- prefix of the getCapabilities HTTP-GET operation of the catalogue (catalogueURL) as well as one or more elements of type ows:ExceptionReport.

A.7         Distributed search with common information model

Distributed searches are not only possible on OGC CSW base catalogues but also on catalogues implementing an OGC CSW profile (without having to know anything about the profile). This is achieved by using the OGC CSW common profile information model which does not only include a list of core queryable properties but also the common record response schema (a subset of Dublin Core metadata elements). All OGC CSW compliant catalogues must support a mapping of the core queryables to their information model and vice versa a mapping of their information model to the common record response schema.

Thus an OGC CSW client should be able to query any OGC CSW catalogue, regardless of the underlying information model, using the elements defined in the common record schema and understand the response (common record schema). With this model it is possible for an OGC CSW Client to query an OGC CSW AP ISO Catalogue.

Footnotes

[1] http://dublincore.org/documents/2001/04/12/usageguide/glossary.shtml#A

[2] That can be queried.

[3] http://en.wikipedia.org/wiki/Backus-Naur_Form

[4] http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=53798

[5] http://www.iso.org/iso/catalogue_detail.htm?csnumber=53798

[6] OGC Abstract Specification Topic Volume 12 (http://portal.opengeospatial.org/files/?artifact_id=1221) and ISO 19119 http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=39890

[7] hhttp://www.iso.org/iso/catalogue_detail.htm?csnumber=53798

[8] Clause 7 of ISO 19106 stipulates that a profile must clearly identify the specific user requirements that are satisfied by the profile.

[9] Source: http://en.wikipedia.org/wiki/Federation_(disambiguation)

[10] To speedup very slow responding remote catalogues, a catalogue has the possibility to harvest their content from time to time (creating an entire local cache of the metadata) and perform locally all filtering on all cached results of such a catalogue.

[11] The maxRecords is per total and not per catalogue.

[12] Each catalogue decrements this value by one when the request is received and does not propagate the request if the hopCount=0.

[13] So the requestId becomes mandatory in the case of a distributed search

[14] As defined in OWS Common a URL prefix is defined as a string including, in order, the scheme (“http” or “https”), Internet Protocol hostname or numeric address, optional port number, path, mandatory question mark ‘?’

[15] As defined in OWS Common a URL prefix is defined as a string including, in order, the scheme (“http” or “https”), Internet Protocol hostname or numeric address, optional port number, path, mandatory question mark ‘?’

[16] This change is inspired in SQL in order to accept local character sets as property name. Moreover, it resolves the clash problem between cql keywords and property names.