OGC Testbed-14: WMS QoSE Engineering Report

In Figure 5, this ER also explains and works out the necessary metadata elements for the QoSE assessment. According to the WMS specifications, all metadata elements could be extracted as shown in Figure 7.

Metadata cache, through pre-parsing the metadata elements and storing them, could enhance the user experience by reducing user wait time. Therefore, some metadata elements contained in the capability documents are pre-parsed and stored in the local database. However, some metadata elements are not suitable for this procedure. For example, the layer map may be configured with different rendering strategies at different zoom levels, but it is difficult to know how many rules exist. Also, the WMS only supports pixel-wise GetFeatureInfo operation, which makes it hard to cache all feature information, and it is also a lot of effort to manage the vector information. For this, the live request operation is provided which will be described in the sections below.

There are tens of assessment criteria and recommendations for assessing the quality of service. Some of them are easy to programmatically test, but some others are too difficult to automatically test, even for people. This section describes which QoSE indicators have been implemented in the current test suite and briefly describes how to test them programmatically. The other QoSE indicators remain unsolved and reasons are summarized in the future research section.

Because the test suite adopts the same QoSE indicators set as that in the client, the human evaluation results from the client could be used to assess that from the test suite. Therefore, a test suite results presentation module has been implemented, which is located below the statistical graph (Figure). Users can open or close it by checking or unchecking the checkbox named D117 Result in the top-right.

In group discussion, some QoSE indicators are too subjective or too hard for computers to programmatically evaluate them. For example, the feature attribute criteria measure the number and relevance of attributes provided for each feature. To assess it, an expert knowledge database is in demand which should contain attributes’ requirements for each feature type in different applications. In the test suite result presentation module, descriptions of difficulty for programmatically assessing them are presented instead of test suite results.

Note that test results of the JMeter (see next section) component have been integrated into the portal, under the tab “Performance Load Reports”. GeoSolutions' QoS monitor, outlined later, is also integrated into the portal, under the tab “QoS Monitor”.

Stress testing is crucial to investigating the behavior of infrastructure or an application under heavy load. It usually involves measuring the following:

Note: There needs to be a discussion or agreement on stress testing against operational services with the server and service provider in order to avoid formal blocking of requests by the provider.

From an implementation/deployment standpoint the goal of stress testing is to gather concrete numbers about the characteristics that can be used over time to check the effect of evolutions. It can also be used to dimension hardware infrastructure and redundant deployments as well as to impose QoS restrictions to protect production deployment (e.g. limit number of parallel requests to a value that is closer but lower than the maximum throughput).

From a developer standpoint stress testing is also important in order to investigate software and architectural bottlenecks. As a developer trying to assess scalability, performance and availability of an infrastructure the goal is to hit the latter with a load that can force it to its limits (i.e. maximum resource utilization) and then analyze the effect on the underlying resources, find the bottleneck, fix it and then move to the next one. As an instance, bottlenecks can be identified in a WMS implementation (e.g. unnecessary synchronizations or inefficient code of sort) by not being able to fully exploit Central Processing Unit (CPU) and Memory under huge load. By experience, if at increasing load the throughput falls and the response time significantly increases but CPUs and/or memory are not fully utilized this is a clear indication that unless the implementation is being affected by slow Input/Output (slow disk, slow DBMS), then the implementation has some bottlenecks that need fixing.

Given what is written above, stress testing and benchmarking is an activity that should encompass the entire lifecycle of an infrastructure; it should be done during development. To do a sanity check on written code and chosen components, stress testing should also be done at deployment time to ensure proper dimensioning, it should also be done periodically when in production to double check and reconcile assumptions and expectation with reality. All this falls under the umbrella of Application Performance Management (https://en.wikipedia.org/wiki/Application_performance_management).

A useful case for stress testing is whenever server software, data sources or their configurations are changed: stress testing before and after modifying the service would verify if everything is still working as expected, or if the made changes had a real effect on service performance.

Apache JMeter (http://jmeter.apache.org/) is a popular open source tool used to load test (or stress test) web applications. JMeter lets us send an arbitrary amount of custom tailored OGC WMS requests to server applications to derive information about response time, throughput and availability of the services against heavy load conditions. Some of JMeter points of strength are mentioned below:

Within JMeter, a web performance test can be built, executed and analyzed.

Figure 32 shows an example of a JMeter load test plan, showing the global variables, a set of threads (1-100) and some result output options (summary reports and graphs).

For a more thorough discussion on JMeter and its capabilities, the reader is referred back to the official JMeter documentation as well as to the GeoSolutions training which contains a section on how to stress test GeoServer using JMeter (https://geoserver.geo-solutions.it/edu/en/enterprise/jmeter.html).

As mentioned above, performance tests have the objective of probing the swiftness and behavior of a service under various load levels. Performance tests also reconstruct the throughput and average response time curves under increasing concurrent load to simulate different real-world conditions. In order to do so, JMeter test plans send an increasingly higher number of concurrent WMS requests to test services performance under various load conditions.

Below the most important points to consider when setting up a stress test plan; this ER will then discuss how one of these test plans is set up.

Generally speaking, software developers tend to associate a single user to a single thread in JMeter to send requests. When it comes to WMS this is partially true since, if an end-user is interacting with a WMS service via a web-based applications that uses map-tiling, a single user can send up to 6 requests in parallel (depending on the web browser of choice) and even more if tricks like setting up multiple names for the same cluster under tests have been set up.

So the equivalent between threads in a JMeter ThreadGroup and end-users needs to be evaluated on a case by case basis. However, when if doubt it is safe to assume that JMeter can simulate a single user interaction with a single thread.

JMeter uses multithreading to simulate concurrent users, by having multiple threads sending requests in parallel and therefore when trying to simulate a huge load it can become quite heavy on the underlying infrastructure. Rule of thumb is therefore to avoid adding extra load to the infrastructure that is being stress tested by having JMeter run on a different one.

It is important to remark on the fact that a number of online services that can be used to generate a huge load based on a JMeter plan. In the past GeoSolutions have used https://flood.io/ for testing a large infrastructure, but there are other options like https://www.blazemeter.com/. In addition, JMeter can be used to run distribute tests by exploiting multiple slave machines from a single master as explained in the JMeter documentation (https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf).

When planning and executing a stress test with JMeter, carefully evaluate the impact of all the tools that sit between JMeter and the infrastructure that is being tested. This includes:

It is obviously crucial that the requests used to generate the stress test load are as representative as possible of real-world usage patterns. This is possible for systems that have already been launched and made accessible to end-users (e.g. by collecting and curating access logs) but it is not always possible for pre-flight stress tests; moreover it is always important to somehow randomize requests (see also below) in order to discover bad behaviors that are not always obvious during normal usage.

In general, so-called synthetic requests tend to be created starting from valid WMS requests and then randomizing in a controlled manner some of the parameters like BBOX, width, height and so on. Often, mixed stress tests tend to be used where some of the requests are real-world requests and others are synthetic requests.

Some of the parameters of each request (width, height, bounding box) are varied continuously to avoid aggressive caching on the server-side, this is done using JMeter "CSV Data Set Config" element that allows us to fetch the above-mentioned parameters from pre-generated Comma-Separated Value (CSV) files containing pseudo-random values for each of the parameters as described below.

A way to perform this randomization is given by using a feature within JMeter that allows use of an external CSV file to override certain default parameters of a WMS request using the values taken from the CSV file itself. An example is shown below(Figure 33). Web Map Services can be set up as tiled and untiled services. For each an external CSV with chaning boundary boxes has to be specified.

The config in the previous figure is responsible for fetching parameters from CSV files (file system path set in the TILED_CSV_FILE variable). An example of an external CSV file is given in Figure 34.

Fields are separated by a semicolon, the first two represent the values for width and height, next is the coordinates for the bounding box followed by the reference system for these coordinates.

A number of scripts exist already for the generation of the randomized parameters. GeoSolutions has put together Python scripts that can be used to generate parameters for WMS requests (and also for other services). These are available in a public GitHub repository (https://github.com/geosolutions-it/scripts/tree/master/performance_tests/WMS.

JMeter can produce the results in Hypertext Markup Language (HTML) outputs but this output is somewhat cumbersome and requires some plugins and an installation hosted on a web server. GeoSolutions typically uses spreadsheets to record the output of the JMeter plans (shown below in Figure 35) using the number of threads as the independent variable and the throughput and response time as the dependent variable. Below are some examples charts for throughput and response time.

To demonstrate the use of JMeter for this study we choose to do a load performance test on the WMS that is made available through: https://open.canada.ca/data/en/dataset/be0a3350-f755-418e-b04b-7ff9fd2ebeac. And choose a layer named “0”. This layer was selected using QGIS and represents geospatial data of “Pelagic Seabird Atlas, West Coast of Canada - Average Grid Cell Density, 2009” and has the following boundary box: -147.0,45.1666669999999968 : -123.0833329999999961,56.0 (see Figure 36).

This maximum boundary box was used to generate tiled and untiled variable boundary boxes with changing height and with running the script provided by GeoSolutions to a CSV file, similar as is shown in Figure 34. Once the tiled and untiled CSV files are in place the load performance test plan is built in JMeter (similar as shown in Figure 32). All in-place test runs should be conducted to make sure that JMeter receives data about the WMS when querying for tiled and untiled services. Make sure to log successful connections. These test runs can be done on a desktop in the JMeter GUI environment, and if all works well should show a snapshot of the geospatial data each time JMeter sends out a data request, see Figure 37.

Potentially, parameters need to be tweaked to be able to receive spatial data. Testbed-14 participants found that they could not create the variable boundary boxes captured by the CSV files for all WMS instances provided by Natural Resources Canada. Especially for datasets that cover only a very small spatial area these variable boundary boxes could not be generated and therefore the WMS site could not be tested. Also, for some WMS the tiled option was not activated and therefore this WMS capability could not be tested either.

Successful load performance tests are then moved and executed in the cloud. Running in the cloud, a summary report informs (see Figure 38) the user of the state of the execution and a graphical report (see Figure 39) provides a first indication of the performance of the WMS instance over time.

Two informative diagrams (Transaction Throughput and Response Time) are generated for each WMS instance that a load performance test is created for in the cloud (Figure 40). The graph on the left shows how transaction throughput time increases with increasing number of threads (concurrent users) until it levels out. So basically, it shows the statistical maximum possible number of transactions based on number of users accessing the WMS. Any more users that query the WMS at the same time will not affect the transaction throughput time anymore. In this case that happens after 35 to 40 users for the tiled service (orange line) and already at between 5 to 8 users for the untiled service (blue line). The graph on the right shows 2 interesting things. First, the untiled service (blue line) has significant slower responses with increasing threads (from 2 to 14 seconds). Secondly, for the tiled WMS service, the response time does start to increase with increasing users at around 30-35 threads but stays below 2 seconds, even with a 100 threads (users, each querying the WMS with 50 requests).

JMeter stress tests are ran, over time, for a variety of WMS provided by Natural Resources Canada (NRCan). NRCan is in the process of migrating some of their provided WMS to another platform, and this provides a perfect opportunity to see if the hosted WMS will perform better when migrated over. The stress tests reports will be made available online, such that they can be easily integrated into the "GUI for WMS Service Quality Assessment" effort mentioned above.

It is very important to make the QoS service very reliable, credible and transparent in terms of metadata declaration. Thus, GeoSolutions have built a Live Monitoring Service based on GeoHealthCheck that monitors the health and the availability of the GeoServer services which will be declared in turn in the QoS module of GeoServer and therefore in the GetCapabilities file of the service.

The figure above shows a schema of how to explicitly monitor external services in the QoSE module in GeoServer in order to offer a reliable service to the users.

The upper side of the diagram in Figure 45 represents an external monitoring service which continuously sends GetCapabilities or GetMap requests to GeoServer on the entire workspace or on single layers to control the endpoints of the WMS and WFS services of GeoServer.

The lower side shows that the monitoring service can be declared in the QoSE module of GeoServer linking it in the Operation Anomaly Feed section of the module to demonstrate the reliability of the service to the users.

Accordingly, once a user requests the WMS or WFS GetCapabilities of the server, the user will have a way to directly check the monitoring service.

The component overview graphic at the beginning of this section illustrates how the different components described above fit together. The following scenario will be used to demonstrate this in an example demo video.

After setting the story for the demonstration video including the material and short individual recordings describing each component as outlined at the start of this chapter, the Technology Integration Experiment (TIE) showcasing followed the steps below:

Step 1: Set up a couple of WMS instances that will be used to demonstrate the workflow.

Step 2: Introduce (briefly) a couple of WMS instances

Step 3: Use those instances to run through the Assessment GUI for user-based evaluation/rating

Step 4: Pull in the JMeter performance statistics of those WMS instances/services

Step 5: Showcase the GeoServer Extension for those WMS instances

Step 6: Illustrate the HTML Landing Page and its value to the community.