OGC Vector Tiles Pilot: Summary Engineering Report

The requirements for the Pilot were to address the issues identified from previous OGC testbeds and propose draft standards to conceptualize tiled feature data and extend WFS, WMTS and GeoPackage to support tiled feature data based on the Mapbox Vector Tile (MVT) specification [1].

This ER directly addresses requirement D011 from the VTP Call For Participation (CFP). The requirement states that: “D011: Summary Engineering Report - A report that summarizes the initiative including outputs, lessons learned and recommendations”.

Prior to the pilot, vector tiles in the OGC was utilized in testbeds, but there was not an overall, agreed upon framework to implement against. The Pilot has now produced a Draft Standard of the Tiled Feature Data Conceptual Model [2] and tiling extensions for WMTS [3], WFS [4] and GeoPackage [5] as well as this summary report that grounds the findings in an on-going OGC context.

Although comprehensive, requirements for future work have been generated from this Pilot. These recommendations are to:

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

The OGC has three main organizational programs within its remit; the Standards Program, the Outreach and Communication Program and the Innovation Program. The Innovation Program defines several initiative types that enable the evolution or definition of draft standards through practical applications at increasing Technology Readiness Levels (TRLs). Figure 1 shows the initiatives within the innovation program, these are briefly described as follows:

The OGC also runs a number of hackathons each year. The hackathons are organized to bring groups of developers together, for a number of days, to focus on a specific problem. Hackathons also take place formally or informally during OGC testbeds and other initiatives as a delivery framework to rapidly produce results.

As mentioned previously, the work described in this ER is a summary of an OGC Pilot, therefore success criteria is based upon utilizing COTS components in an operational context.

This section covers some of the previous work done in OGC testbeds via a short literature review with the aim of providing an overview to ground the work done in this Pilot. The main documents of interest are:

Vector Tiles were originally discussed in Testbed-12 and documented in the Vector Tiling ER. That document characterized tiles in terms of their definition, use cases, utility among Geographic Information System (GIS) users and route to adoption within the OGC. At that time, there was no agreed method of tiling vector data. Tiling of raster data is accomplished (broadly) by associating a pixel with a geographic area and a geographic area with a specific tile in a 1:1 relationship for a single scale. However, vector features can vary and agreed approaches have to be adopted to successfully allocate a vector feature to a particular vector tile. Additionally, raster tiles are generally scale appropriate. The approach with vector data is different, as it is likely to undergo a transformation through generalization to produce scale appropriate tiles. Styling of vector tiles is also done differently to raster equivalents. Raster tiles are made up of simple images that generally have their styling 'baked in', whereas vector data can be styled on-the-fly using a variety of techniques. Styling approaches in vector tiles offer both overhead and opportunity to produce audience appropriate tiling.

The Testbed-12 Vector Tiling ER companion is the Vector Tiling Implementation ER that documents the implementation of vector tiles in a GeoPackage format. Prior to the Vector Tiling ER, GeoPackage supported raster tiling procedures with excellent reported results. The goal in Testbed-12 was to prove the approach for vector tiles in GeoPackage to replicate the results seen with raster tiles. Tiled feature data in GeoPackage was implemented using a vector tile pyramid concept, where vector tiles are stored in a pyramid, much in the same way raster tiles are. There are two approaches that were considered for storing tiled data in pyramids; render based tiling and feature based tiling. Render-based tiling changes the original geometry which was considered unacceptable by the sponsors, therefore a feature based tiling approach was used with a GeoJSON encoding.

The work done on vector tiles in Testbed-13 was a comprehensive study of approaches to tiling vector data including:

The stated goal of the Testbed-13 work on vector tiles is to move closer to standardization by producing a set of draft standards and implementations to demonstrate and test the draft standards, this VTP can be seen as a continuation of the work done in Testbed-13.

Tiling as a method of data delivery has been around since the 1960s, although in many instances, the focus was on raster-based products such as imagery. There are several terms associated with tiling that should be considered:

The main advantage of vector tiles is the speed at which data can be delivered from a server to a client. Much of this speed increase over traditional methods is achieved through efficient delivery of areas of interest requested by the client. As mentioned in the review of previous work, vector tiles have several advantages over existing vector distribution technologies. The push for vector tiles has come about in part because of the explosion of geographic data available to users. This is often termed Big Data and does not simply refer to data volumes, but also velocity and veracity. One also has to consider technologies and techniques devised specifically to take advantage of the wealth of data available. An often cited data source that may be considered geographic Big Data is OpenStreetMap (OSM) ^[1]. OSM has around 30GB of data available for the entire world and is a community driven project that anyone can contribute information to. Distribution of this data is problematic. Therefore strategies such as change-synchronization are often adopted to keep a dataset up to date server-side. Distribution of such data to clients can be challenging due to the data size. Therefore tiles are often used to provide the user with the smallest volume for their use case.

Vector tiles offer additional advantages through decoupling of peripheral aspects such as styling. Unlike most raster tiles that are pre-generated and styled according to a single schema, vector tiles have a single base dataset of features and can be styled upon request and on-the-fly. This enables feature data to be used for a multitude of clients. Client-side styling is also possible if suitable. Styling can refer to colors as well as other aspects such as point symbol size, coloring and shading.

Although very efficient, vector tiles have had issues with cross-tile alignment of features. If cross-tile features are not aligned, then there are downstream errors in the results of topological operations such as routing, as the features are no-longer connected. Additionally, cartographic convention is broken when producing products as the output is unsightly and may dissuade audiences from using a service.

The Testbed-12 Vector Tiles ER outlined several challenges with vector tiles, these were:

In addition to these, other challenges identified since include:

The Pilot was designed to be completed within a short time frame and utilized a phased approach, Phase 1 consisted of delivery of draft components and ERs and Phase 2 required delivery of final components and ERs. The overall goal of the Pilot was to define and test approaches for vector tiles extensions to existing OGC standards. This was done through profiling and providing extensions to the existing OGC standards. This section contains a short description of each of the delivered components, ERs and an overall architecture.

The architecture of the pilot was designed to cover the three main server client relationships identified above, shown in Figure 2, these include:

This architecture addressed vector tiles in each of the client server relationships to simultaneously enable tiling across the relevant suite of OGC standards. This approach provided implementers with guidance for VT no matter their OGC use case.

Web Feature Service 3.0 is revolutionary rather than evolutionary as it does not start with the same assumptions as previous versions of WFS. Instead of building upon the previous service based interface, the standard has migrated to a REST interface, a move that may be followed across the OGC suite of standards where appropriate. WFS 3.0 supports the same operations as the previous version, except they are now exposed as an Application Programming Interface (API) based on the OpenAPI (https://www.openapis.org/) specification.

The objective of the Vector Tiles Pilot for this component is to extend the WFS 3.0 operations to support vector tiles, notably the emphasis is on read access therefore the only required supported HTTP method is HTTP GET, however it is noted that servers with VT enabled WFS3.0 should also independently support HEAD and OPTIONS for each resource.

There are two types of resources that are supported by the WFS 3.0 Server, these are direct access to tiles and access to features within the tiles. The two different resources have different approaches and calls associated with them. The interface is based entirely upon http verbs, which is a major shift from utilizing service calls such as GetCapabilities and DescribeFeature. The http verbs used are:

The resources exposed via WFS 3.0 are accessed via REST and the HTTP verbs mentioned. Architecturally, WFS 3.0 is not implicitly bound by a single interface to a single server, which was the assumption in previous versions of WFS. Instead, resources are fronted via an API and the actual mechanics of storing the resources at rest are left to the implementer to choose the type of infrastructure performing the work behind the scenes.

There are two approaches to offering vector tiles in the WFS 3.0 extension:

As part of the pilot effort, a Vector Tiles extension has been created for GeoPackage. Prior to this extension there was no defined interoperable, agreed upon method to store vector tiles in a portable container. The primary use case for GeoPackage is to be a portable container for use primarily in Denied, Degraded, Intermittent, or Limited Bandwidth (DDIL) networks. DDIL use cases can utilize GeoPackage and vector tiles as the size and therefore delivery of vector tiles is orders of magnitude smaller than the raster equivalents. However, initial load of the GeoPackage onto a mobile device is required prior to utilization in a disconnected environment meaning that remote updating remains problematic.

Overall, the extension is based upon the MVT specification and uses Google Protocol buffers to encode the content within each tile. The extension requires GeoPackage core, and like all extensions, will run with clients that do not support it.

The extension is defined by creating new metadata tables within a GeoPackage:

There are several Vector Tiles extensions that have been developed as part of the pilot that are discussed in the next sections. There is no single Vector Tiles extension to GeoPackage, instead the extensions are modularized for simple implementation and to increase interoperability with services supporting the core standard only.

WMTS is an OGC service designed to offer a fixed set of graphical images in pixel space to a client. From the perspective of clients there is therefore no opportunity to dynamically style tiling on-the-fly. One of the objectives of the WMTS aspect of the pilot is the ability to provide vector tiles as a standard output of any service. The focus of this work item is Mapbox vector tiles, however there is also a requirement to implement GeoJSON vector tiles as a separate output. Several implementations were produced, as with the other work items in the Pilot.

Three separate GeoPackages were produced:

It should be noted that MVT and GeoJSON benefit greatly from compression shrinking by 90% and 96% respectively. The capability to Deflate compress individual tiles inside the SQLite database was added to GeoPackage near the end of the Pilot, unfortunately after the development of many of the test GeoPackages.

Storing attributes for vector tiles in GeoPackage has been subject to debate in the community. Two main methods have been identified:

The power of relational databases is the ability to relate tables to de-duplicate storage requirements, therefore there is a preference for the RTE rather than the verbose method of storing a table per tile. Additionally, performing queries across tiles can be problematic as overlapping features would appear in some form in multiple tables. The RTE is the GeoPackage method of managing many-to-many relationships, a typical use case for vector tiles and features that all have attributes. Currently, the RTE does not support requirements specific to vector tiles, that is, the tiles acting as the base data and the attributes as the relationship, therefore two tables are proposed for the RTE:

An advantage of vector tiles is the ability to de-couple elements, this has already been discussed with respect to tiling and attributes. Another requirement is to de-couple styling from tiling, there are two main proposals for accomplishing this in GeoPackage:

Across the implementations, participants noted that Mapbox Vector Tiles offer a significant performance increase over other tiling approaches due to the mandatory inclusion of Google Protocol Buffers as the encoding format. GeoJSON vector tiles do not experience a performance increase over other formats but are an emerging format for exchange and were implemented for interoperability reasons.

As mentioned previously, DDIL networks are a concern in many information communities. Traditionally, OGC services assumed consistent and stable internet connectivity. Delivery of data in DDIL environments requires assumptions regarding connectivity be readdressed. There are several strategies that can be adopted to compensate for DDIL environments that include:

If a network is available, then techniques such as compression can be used to deliver the data required. Mapbox vector tiles are well suited to these use cases due to the implementation of Google Protocol Buffers that provide optimal compression for delivery across a network. Another strategy used to take advantage of intermittent networks is the use of asynchronous services, where the server processes the request in the background and delivers the result when notified of client connectivity. Possible extensions to each of the standards considered are outlined in the individual ERs. Another method considered is the publish/subscribe method (pub/sub) that utilizes polling to synchronize changes across a network when required. This method is suited to more frequent updates and forgiving DDIL environments.

Overall, some elements of the VTP have shown that the technologies described are suited to these use cases, but there is more work to be done in explicitly extending the standards to DDIL environments.

Throughout the Pilot project there were discussions regarding the remit of the different OGC services and tiled data. Essentially, tiled feature data and supporting products are built from features that have attributes, much like any other vector data. Therefore, much of the exercise of the Pilot was to produce tiled feature data using the three OGC standards of interest. The standards used are not comparable, for example GeoPackage 1.2 is essentially a container and WFS 3.0 defines access protocols for feature data on a server. WMTS provides access to tiled matrix datasets, but in contrast to WFS 3.0, these tile matrices are pre-rendered. The WFS 3.0 extension proposed in this Pilot also has the ability to access pre-rendered tiles. This utilization of OGC services begs the question as to whether WFS 3.0 can fulfill all of the service requirements outlined in this Pilot; data for mobile, data for desktop and data for the web. The mobile data could be created by replacing the GeoPackage producer with WFS 3.0 with tiled data encoded with Google Protocol Buffers in a GeoPackage format.

This discussion may have further consequences for OGC services such as whether WFS 3.0 can be utilized to serve tiled data in terms of their features. The question is also whether WFS 3.0 can be extended to serve other files as well? This has obvious consequences for WCS, WMS and WMTS as their functionality could be re-produced using a WFS 3.0 style architecture based upon OpenAPI.

As OpenAPI is an interface description, it does not mandate what happens on the server, which can be treated as a black box. For example, there is no required "GetCapabilities" call that would be exposed via a REST endpoint as OpenAPI is concerned with the exposure of resources, there is no one interface to one server relationship implied or mandated. Therefore, there is no reason that geoprocessing cannot be covered by an extended version of the WFS 3.0 interface. In a strict interpretation of geoprocessing, this is already happening under the current WFS 3.0 specification because of the operations that can be performed, for example, filtering and applying styles to vector tiles requires some level of geoprocessing to be happening server side to deal with the request. One suggestion for a converged capability is the Web Object Service (WOS), although it remains to be seen whether a converged service will garner the required response from the Technical Committee. The reason for the lack of support for a converged capability is that modularization is a key principle of OGC Service Architecture that makes it possible to build a variety of solutions due to the ‘separation of concerns’.

Testbed-14 contains a study on the implications of technologies such as Docker, Kubernetes and Cloud Foundry being utilized in the standards work of the OGC. Essentially these technologies have the ability to dynamically allocate resources using container technology to deliver results. A REST based interface could be put in front of such an architecture for efficient and dynamic processing and data delivery. The implications of this on tiled data and the wider OGC should be explored in future initiatives.