OGC Testbed-14: WMS QoSE Engineering Report

Quality of Service (QoS) and Quality of Experience (QoE) as they are intended and described at the OGC are two related concepts which require very specific treatment and characterization. Citing the definitions provided by the Domain Working Group (DWG) charter document:

QoE focuses on the usability of the information that is conceived via OGC services to end users or other client application and therefore is concerned more with qualitative aspects of such services like presence of metadata, proper and descriptive namings, appropriate styling and so on (a more thorough treatment is present in the QoE discussion paper OGC 17-049 entitled "Ensuring Quality of User Experience with OGC Web Mapping Services" available at https://portal.opengeospatial.org/files/?artifact_id=74403&version=1).

QoS focuses on providing reliable (i.e. quantitative ) measures of spatial data service metrics which can be used to characterize how a service ( one or more specific datasets exposed by a certain service) is performing both in near real-time as well as historically. It touches concepts like availability, scalability (also known as capacity), absolute performance (i.e. speed) and can be used to assess also perceived performance by final clients. As mentioned above, it is typically measured using metrics like error rates, throughput, availability and delay or request response time.

Quite often the QoS and QoE aspects of spatial data services are underestimated if not simply ignored due to lack of resources as well as lack of awareness, resulting in services which are difficult to exploit (i.e. QoE very low) and/or unstable or very slow (i.e. QoS very low). The result is that few users end up using them after the initial launch and this is especially true for services targeting end users who are used to interact with services a-la Google Maps which delivers extreme performance and scalability as well as bullet-proof usability.

The ability to combine and visualize location-based data using a Web Map Service (WMS) is a key value proposition of the Federal Geospatial Platform (FGP). The FGP is a collaborative online environment where a collection of the government of Canada’s most relevant geospatial information can be found easily and viewed on maps to support evidence-based decision-making, foster innovation, and provide better service for Canadians. The FGP includes the capability of selecting datasets with WMS to view individually or combined with other services in the FGP viewer.

This functionality, as is the general case with a WMS, is provided to allow users to immediately visualize and analyze geospatial data. Unfortunately, user feedback has proven that these geospatial web services are not always easy or intuitive to navigate, combine or understand. Because the FGP’s primary end users are government policy analysts who are not always highly familiar with mapping and web mapping technologies, it is important to make WMS, and the content they make available, as user-friendly as possible.

In 2016, to help alleviate this issue, the FGP developed a web service quality assessment methodology that supported WMS, ran an assessment and developed recommendations and best practices to support a user-friendly experience for all employees and citizens using WMS. Assessments to date have shown that key considerations are often very simple, but very impactful on QoE. The results of this study were used as the primary input into an OGC Discussion Paper created by the Quality of Service Experience (QoSE) DWG. The OGC QoSE-DWG has developed a discussion paper (OGC 17-049) entitled "Ensuring Quality of User Experience with OGC Web Mapping Services" [https://portal.opengeospatial.org/files/17-049] that identifies and describes issues often encountered by users of OGC WMS that affect the quality of their experience, and also provides an assessment framework for identifying issues and measuring quality, along with potential solutions and guidance to improve the usability of services.

The assessment framework for measuring QoE and the associated recommendations for improving service quality are intended to benefit human end-users who need to rapidly assimilate and use web mapping visualizations to answer questions or input into analysis. In other words, they need to be able to make sense of the information an OGC WMS provides them.

In addition, Testbed-13 addressed QoS aspects in the aviation domain. Though specific to a particular domain, The Testbed13: Data Quality Specification Engineering Report [http://docs.opengeospatial.org/per/17-018.html] addressed a number of general aspects that apply to this task nevertheless.

Testbed-14 has addressed the following WMS usability aspects:

1. Develop a revision of the OGC Discussion Paper 17-049: Develop a revision of the OGC Discussion Paper 17-049 as per the review and assessment of the current OGC Discussion Paper 17-049. This revision shall be performed in coordination with OGC QoSE DWG to address potential new requirements such as QoS metadata (Capabilities Extensions) server implementation and monitoring client implementations that allows improved service quality and measurement. The revision is part of this ER.

2. Quality of Service Experience Assessment Framework: The fourteen assessment criteria described in Discussion Paper 17-049 and summarized in Figure 1 are all aimed at assessing the quality of a web service in terms of the degree to which it conveys clearly understood information to the user. The user is assumed to be a non-expert in geospatial web services, but in most cases the criteria are equally valid for all classes of users. A selected number of services in the Testbed will be assessed against the QoSE assessment criteria, and retested once recommendations have been applied. Comparison of results will help to validate the effectiveness of the assessment criteria and the corresponding recommendations to improve usability, and allow for feedback, improvement or correction. Services are provided as Testbed-14 deliverable D115 and, a dedicated client as D116.

3. Quality of Service Experience Practices to Alleviate Usability Issues: The fourteen assessment criteria all have corresponding recommendations that describe practices that, once implemented, should help to alleviate user confusion and improve the usability of WMS as a means of visualizing and simple visual or query-based analysis of geospatial data. Once selected services have been assessed against assessment criteria, all or as many QoSE recommendations will be applied, then services will be subject to retesting.

4. Quality of Service Experience Implementing Best Practices: Man or Machine?: Analysis required here on the making of a geospatial WMS and the opportunities of human operator or programmatic response to make decisions that impact service quality of experience. Assessment of which usability issues are determined/caused by human input or what determined/caused by default via the programmatically generated aspects of the service (not due to direct human input or decisions) via the standard specification implementation. Results are captured in this ER.

5. Quality of Service Experience Test Suite: Develop test suite to programmatically test and validate that best practices have been successfully implemented. The test suite will automatically assess the quality of service according to the assessment criteria and validate or flag services that do not comply to best practices/recommendations. The results of the test suite (D117) are also captured in this ER.

Figure 2 illustrates the QoSE work items as reflected by the OGC Testbed-14 thread.

Note: The purpose of D122 is to provide a Client to meet requirements for QoSE and Portrayal as well as measure statistics on response time, number of layers, number of features, error codes generated, etc. within the client. Since subjective measurement criteria, focusing for example on legibility, intuitiveness, consistency of symbology, may be more difficult to automate, participant will gather QoSE statistics manually for those criteria or elements, and then propose recommendations for an automated service for those assessment criteria.

The OGC QoSE DWG provides a forum for discussing issues related to QoS and QoE of spatial data services and applications relying on these services for delivering timely and accurate spatial information to the end-users. Key business goals of the Working Group include the following:

The task described in this ER addressed the QoE assessment criteria laid out in the DWG Discussion paper referred to earlier and also implemented a QoS performance assessment. Specifically, an interactive user-friendly Graphical User Interface (GUI) was developed based on the QoE assessment criteria defined by the DWG. This GUI also lists performance statistics from a WMS server.

A goal of this task and its analysis was also to suggest potential future activity where these results could be investigated through new tasks in future testbeds.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

Figure 4 illustrates the connections and interoperability between the various component parts discussed in this ER.

A discussion paper entitled "Ensuring Quality of User Experience with OGC Web Mapping Services" provides an assessment framework for identifying issues and measuring quality, which can be downloaded at public link. It outlines fourteen criteria to measure QoSE, and recommendations are given for each of them, which are shown as components using a rectangle in Figure 5.

D116 requires implementation of a client dedicated to testing the assessment criteria of WMS instances. Therefore, a Graphical User Interface (GUI) was designed and is shown in Figure 6. It consists of three basic components, including a WMS layer selector, metadata information viewer, and QoSE assessment panel. Users can select a layer of interest from the WMS layer selector. Once a layer is selected, its metadata is loaded and shown in the metadata information viewer, which is located in the bottom-left. Finally, users can give their evaluation for each QoSE indicator based on the layer metadata contents. At the same time, users can also leave feedback about the QoSE framework or the client using the feedback panel.