Testbed-12 Compression Techniques Engineering Report

As background to the project, Testbed 12 participants assessed a briefing provided to geospatial working groups of the US Department of Defense titled ‘Efficient XML Interchange (EXI), Taking XML to the Edge’. This briefing presented advantages of EXI for US military message exchange.

This presentation indicated that significant ‘bandwidth’ may be saved by using EXI to compress military messages such as TDL J-Series and STANAG 4607 messages.

This briefing states that EXI "Achieves New Levels of Performance Required for Tactical Environment'. It’s premise is that XML transfer requires significantly more bandwidth than compressed data formats such as EXI. Specifically, the briefing states that 'tactical messages' are reduced in size from approximately 8,000 bytes to approximately 200 bytes using EXI.

Testbed 12 participants also reviewed a 2010 study from the Naval Postgraduate School titled ‘Efficient Xml Interchange (EXI) Compression and Performance Benefits: Development, Implementation and Evaluation’. The abstract of this study indicates that EXI ‘delivers significant file size savings and processing efficiencies compared to existing practices’. The research concludes that 'for XML-based data, a doubling of bandwidth potential is achievable and central processing unit (CPU) burdens minimized when EXI is applied.'

This study indicated that the US Department of Defense (DoD) Network-Centric data sharing strategy is to 'XMLize' all data. The goal of this strategy is to ensure all data is visible, usable and interoperable, when and where needed, to accelerate decision cycles. However, the study indicated that this XML-based data approach comes at the cost of limiting real-time network edge device connectivity because they are seldom able to meet the necessary bandwidth and processing requirements due to XML’s intrinsic nature of being verbose and often complex to process.

The study also indicated that the EXI format removes redundant tags and values from XML documents and encodes numeric content in a binary format. Experiments were conducted evaluating the effectiveness of EXI for DoD tactical use and is followed with a recommended optimal EXI configuration. This research concluded that 'for XML-based data, a doubling of bandwidth potential is achievable and CPU burdens minimized when EXI is applied.' However, the report seems to present other test results immediately after this statement. Specifically, it is important to note the results from the perspective of schema-informed verses schema-less EXI compression.

For example, it notes a 'unique occurrence happened within this case set, the Web XML document was compressed better with GZip than EXI schema-informed by 0.8 of a percent. This is a minor difference, but of all the tested documents from the three test categories, it is the only document where EXI schema-informed was outperformed.'

Furthermore the report discusses that in the more general case observations, only the WEB xml and the DOCS xml test case files did the EXI schema-less technique resulted in a larger compressed file than GZip, though just slightly. The Zip technique delivered on two occasions with a resulting file greater than the original document, DOCX xml and HELLOWORLD xml.

In the end, the general conclusion of the report was that with or without a schema, EXI delivers noticeable file size savings averages compared to GZip, and far exceeded the Zip results.

This finding should be compared to the findings of Testbed 12 using geospatial data encoded in XML.

In addition, Testbed 12 participants reviewed work in OGC Web Services, Phase 8 (OWS-8) that assessed EXI. Some results from this prior OGC Testbed report indicated that for Aeronautical Information Exchange Model (AIXM) data:

With the review of the OWS-8 report, Testbed 12 participants noticed that different reports seemed to be coming to different conclusions regarding the efficiency of EXI for compressing XML.

Finally, Testbed 12 participants reviewed work done by the United Kingdom Defence Science and Technology Laboratory (Dstl). This assessment which argued that schema compression is critically dependent on the design of the schema as well as the XML data itself. In particular, the report noted that coordinates in GML 2 may be defined in a schema as 'text' and that schema optimization cannot improve on 'text.' With the results being poor compression with lots of coordinates. However, the report noted that coordinates in GML 3 may be defined in a schema as 'float' and that schema optimization can improve on 'float'. With the results being good compression with lots of coordinates.

The key point being that compression is critically dependent on the design of the schema as well as the XML data itself. Accordingly, Dstl argued that data sets with many geographic coordinates may compress well with EXI.

The OGC Web Feature Service (WFS) Implementation Specification allows a client to retrieve geospatial data encoded in Geography Markup Language (GML) and other formats from multiple Web Feature Services. The specification defines operations for data access and manipulation operations on geographic features, using HTTP as the distributed computing platform. Via these interfaces, a Web user or service can combine, use and manage geodata — the feature information behind a map image.

This International Standard specifies the behavior of a service that provides transactions on and access to geographic features in a manner independent of the underlying data store. It specifies discovery operations, query operations, locking operations, transaction operations and operations to manage stored parameterized query expressions:

This International Standard defines eleven operations:

Some WFS servers may also support additional non-GML feature encodings and client applications may access them using the outputFormat parameter domains. However, the WFS International Standard does not describe how a server would operate upon such encodings. This is an important distinction for TopoJSON interoperability testing, demonstration and operational implementation.

The OGC Filter Encoding Implementation Specification describes an XML and KVP encoding of a system neutral syntax for expressing projections, selection and sorting clauses collectively called a ‘query expression’. As background, a fundamental operation performed on a set of data or resources is that of querying in order to obtain a subset of the data which contains certain desired information that satisfies some query criteria and which is also, perhaps, sorted in some specified manner.

This International Standard defines the XML encoding for the following predicates.

Despite W3C statements that EXI processors are ‘relatively simple’, Efficient XML Interchange (EXI) is a very complex topic. The reader is encouraged to review the following W3C documents for a complete background -

Review of the references above indicates the W3C describes EXI as not 'dependent on schemas'. However, prior investigations have assessed that EXI may compress XML more efficiently if schemas exist describing the format of the expected XML. As background, it is important to understand there are two main ways in which EXI encodes XML documents -

EXI uses a set of built-in grammars to encode XML documents and XML fragments when no schema information is available.

The two modes, schema-less and schema-informed, are important for GML and WFS compression because prior reports noted that coordinates in GML 2 may be defined in a schema as string ('text'). Since schema optimization cannot improve on 'text' this may result in poor compression when there are many coordinates in GML data described by a GML 2 schema. However, the report noted that coordinates in GML 3 may be defined in a schema as float. Schema optimization can improve on float, with the results being good compression with lots of coordinates. The key point being that compression performance may be dependent on the design of the schema as well as the XML data itself.

EXI represents the contents of an XML document as an EXI stream. An EXI stream consists of an EXI header followed by an EXI body.

The EXI header conveys format version information and may also include the set of options that were used during encoding. If these options are omitted, it is assumed that the decoder has access to them out of band.

The EXI body comprises an event sequence describing the document (or document fragment) that is encoded.

In addition to the different compression performance that may be obtained with or without schemas, different types option values may be used while encoding XML documents in an EXI stream. Option values are part of the EXI header and provide a way to specify the options used to encode the body of an EXI stream. There are many option values outlined in the W3C EXI specification and the reader is again encouraged to review them as needed.

The most significant option values for Testbed 12 WFS Compression testing are presented and defined in the following table.

In OGC Testbed 12 EXI participants extended WFS with software capable of producing an output format in EXI. The software tested in WFS implementations included the packages listed in the following table.

For compression testing Arizona State University (ASU) implemented Compression Tests WFS Servers, Compression Clients, EXI Pre-Processors, EXI Processors in the following architecture -

The Compression WFS in Test Suite 1 was based on GeoServer, extended with EXIficient using Java and Javascript wrappers on the server side as needed. EXIficient was employed for encoding plain-text based GML and GeoJSON content into binary EXI files on the server side. The data preparation process is: features -→ XML-based data stream (GML) -→ Encoding into .exi file -→ Transfer to client application for decoding and rendering.

Additional types of compressing methods were implemented on the server including GZIP and LZMA(.7z) for comparison.

For compression testing ASU implemented both Schema-less and schema-informed modes.

Feature data over San Francisco representing points (schools_public_pt.shp), lines (stclines_streets.shp) and polygons (schools_public.shp) formed the test baseline. Other data sets were assessed as well.

The ASU client was developed using EXIficient with Javascript. To request geospatial features in EXI format just change the parameter of outputFormat into “gml2exi” or "json2exi", you will be able to get the same dataset in exi format from the original GML or GeoJSON file.

The performance recording module was implemented for comparison of EXI performance. Three formats are supported, including GeoJSON -LZMA, GML3.1.1-EXI and GML3.1.1-LZMA. Browser side data decompressing functions were implemented with the format of GeoJSON -ZIP, GeoJSON -LZMA, GML3.1.1-ZIP and GML3.1.1-LZMA realized.

ASU attempted to integrate Nagasena into Compression WFS but did not continue the TIE due to performance issues.

For compression testing The Carbon Project implemented Compression Tests WFS Servers, Compression Clients, EXI Pre-Processors, EXI Processors in the following architecture -

The Compression WFS in Test Suite 2 was based on CarbonCloud WFS, extended with OSS using .NET on the server side as needed. OSS was employed for encoding plain-text based GML and GeoJSON content into binary EXI files on the server side. The data preparation process was: features -→ XML-based data stream (GML) -→ Encoding into .exi file -→ Transfer to client application for decoding and rendering.

Additional types of compressing methods were implemented on the server including GZIP for comparison.

For compression testing The Carbon Project implemented both Schema-less and schema-informed modes.

Feature data over San Francisco representing points (schools_public_pt.shp), lines (stclines_streets.shp) and polygons (schools_public.shp) formed the test baseline. Other data sets were assessed as well.

The Carbon Project’s client was developed using OSS with .NET. To request geospatial features in EXI format the Accept-Encoding header can be used by the client to request compressed output and get the same dataset in exi format from the original GML or GeoJSON file.

Points:

Lines:

Polygons:

The performance recording module was implemented for comparison of EXI performance. Three formats were supported, including GeoJSON -GZIP, GML3.1.1-EXI and GML3.1.1-GZIP. Browser side data decompressing functions were implemented.

The Carbon Project attempted to integrate Nagasena into Compression WFS but did not continue the TIE due to performance issues.