2.1.  Terms and definitions

2.1.1. Application Programming Interface

An Application Programming Interface (API) is a standard set of documented and supported functions and procedures that expose the capabilities or data of an operating system, application or service to other applications (adapted from ISO/IEC TR 13066-2:2016).

2.1.2. CI/CD

Continuous Integration, Continuous Delivery is a method to frequently deliver apps to customers by introducing automation into the stages of app development. (source: RedHat)

2.1.3. DevOps

a set of practices that combines software development (Dev) and IT operations (Ops).

2.1.4. Cloud

a network of computing resources, such as storage, servers, applications and services, that can be used on-demand anywhere and anytime via the Internet NIST.

2.1.5. Cloud-native

The concept of building and running applications to take advantage of the distributed computing offered by the cloud delivery model. Cloud native apps are designed and built to exploit the scale, elasticity, resiliency, and flexibility the cloud provides, Oracle.

2.1.6. Code Generation

Source code generation is the process of creating programming code from a model such as one of OGC APIs data schemas.

2.1.7. Containerization

the packaging of software code with just the operating system (OS) libraries and dependencies required to run the code to create a single lightweight executable—called a container—that runs consistently on any infrastructure, IBM.

2.1.8. CRUD

In computer programming, create, read (aka retrieve), update, and delete are the four basic functions of persistent storage.

2.1.9. OGC APIs

Family of OGC standards developed to make it easy for anyone to provide geospatial data to the web.

2.1.10. Technology Stack

A technology stack or tech stack refers to a set of technologies, software, and tools that are used in the development of applications.

2.2.  Abbreviated terms

API

Application Programming Interface

CI/CD

Continuous Integration / Continuous Delivery

CNCF

Cloud Native Computing Foundation

COTS

Commercial Off The Shelf

DevOps

Software development (Dev) and IT operations (Ops)

EDR

Environmental Data Retrieval

IDL

Interface Definition Language

JSON

JavaScript Object Notation

OGC

Open Geospatial Consortium

OWS

OGC Web Services

STAC

Spatio temporal Asset Catalog

4.1.  Who are our developers?

More than 80% of the respondents reside in Europe and the United States of America; and many of the respondents were experienced developers.

Figure 1

• Almost 75% of developers are older than 35!

• More than 70% of the respondents have Masters degrees or/and doctorates.

• Almost 80% are full-time employed, 20% work for public agencies, 30% for small businesses.

• 53% of the respondents currently reside in Europe and almost 29% in North America

• 60% of them stated that they spend 21 hours or more per week on tasks related to geospatial technologies (Q19).

• 71% of the respondents have been coding for more than 8 years.

4.2.  Technology Stack

As expected, Python and JavaScript are the most popular programming languages.

Figure 2

• Regarding programming languages preferences (Q11), developers are most familiar with Python (about 80%) followed by JavaScript (58%). Many of them are also fluent in more traditional C/C++ programming languages (about 38%). Java is still well presented in their technology stacks.

• Developers are very familiar with the mainstream open-source geospatial projects. The most popular is the QGIS framework (for 90.40% of survey participants) followed by PostGIS 82.40%, GeoServer 65.60%, and MapServer 48.80%. This also reflects the fact that features and maps are the most significant and prominent geospatial data formats.

Figure 3

4.3.  Learning Habits

• The preferable way to learn a technology seems to be to “follow a documented how-to/tutorial” (30%) and “to read a tutorial” (20%).

• When learning about a new topic from the internet, “full written descriptions” are the most preferable sources.

• Portal Stack Overflow (Q29) seems to be the most important tool for getting support from the community (80%).

4.4.  Familiarity with OGC (API) Standards

• About 20% of the respondents have contributed to the development of OGC API Standards on GitHub. Still, these standards are not yet widely used in the community because, although 75% of respondents have either used an OGC Standard or read one of the documentations (Q24), a high percentage (30%) answered that they don’t have any experience involving OGC API Standards (Q25).

  

  Figure 4

• Features, Maps and Tiles are the most important geospatial standards and therefore the corresponding OGC APIs are of equal importance (Q27).

• Some of the respondents believe OGC API Standards are not easy to learn. On the question of whether OGC API Standards are easy to learn (Q30) almost 30% answered that “Mastering the first API is hard, but then it becomes easier to learn” while 40% thinks that “Some are very easy and others are extremely difficult to learn”.

• About 57% believes that there is a lack of best-practice guides that help differentiate the dos and don’ts”, as well as 38% that technical documentation is hard to read or understand (Q32)!

4.5.  Summary

The survey results suggest that a typical developer from the OGC/geospatial domain is an experienced, well-educated individual focused on stable, well-established technologies. Traditional and new technologies are equally used. For example they follow the recent trends regarding scripting programming languages as they are predominant today.

Considering the results, the API experiments focused on developing examples and how-to documentation as well as newer technologies and approaches to fill the gap between “stable maturity” and “contemporary/emerging trends”.

Annex A lists the questions used to conduct developers survey.

5.1.  Participants

To test all components and guidance material, the work in this task was shared by participants acting as:

1. Client developers

2. Server developers

3. Data providers

Client developers: Organization O1 implemented and delivered a client-side software library in Python for both OGC API Features and OGC API Environmental Data Retrieval (EDR). Organization O2 delivered a similar library in JavaScript and TypeScript. Both organizations demonstrated their library in a client application.

Server developers: Organization O3 implemented and delivered a server-side software library in Python for both OGC API — Features and OGC API — EDR. The library supported access and exploration of both OGC API — Feature and OGC API — EDR resources and supported data hosted on cloud object storage, cloud databases, and OGC Web Feature Service (WFS) instances (with constraint functionality). O3 provided a deployable instance together with deployment and execution documentation that allows third parties to test the deliverable(s) as a microservice in different cloud environments. Organization O4 delivered a similar library in JavaScript (alternatively TypeScript).

Data providers: Organization O5 deployed a microservice based on deliverables provided by O3 and O4 in at least two different cloud environments (to the extent possible, as some data backends are cloud specific, others work across cloud environments or are fully cloud independent). In addition, O5 provided data in a database, cloud object storage, and as a Web Feature Service instance to test data access (reuse of existing WFS and/or Web Coverage Service (WCS) instances, even outside of the target cloud environment, is admitted). All of the data sources were available to all organizations in this task and are freely accessible (i.e. object storage and WFS/WCS endpoints are fully accessible, together with cloud native databases as much as possible). Organization O6 performed similarly to O5.

Participants:

• Data providers:

  • Skymantics: D167 Data Backend and Deployment 1

  • Pixalytics: D168 Data Backend and Deployment 2

• Services (server developers):

  • 52N: D165 API Experiments Server (Python)

  • GMU: D166 API Experiments Server (JavaScript)

• Client developers:

  • Skymantics: D175 API Experiments Client (Python)

  • Solenix: D176 API Experiments Client (TypeScript)

5.2.  API experiment scenarios

The figure below illustrates a generic Testbed-17 OGC APIs experiment component diagram with interactions. A client with a dedicated OGC API software library (1) interacts with a data service that implements one of the OGC APIs (2) at the front-end and includes a cloud native data storage software library (3) at the backend that interacts with cloud native storage, such as cloud databases, cloud object storage, or existing OGC Web Service instances such as WFS.

Figure 6 — Generic API experiments scenario

6.1.  Key Features of a Cloud-Native Application Architecture

“Cloud native” is a design paradigm governed by the Cloud Native Computing Foundation (CNCF). Cloud native applications can be defined as applications running in a containerized environment, capable of moving in and out of the cloud, and that scale horizontally based on varying workloads. They are deployed as microservices, abstracted to underlying infrastructure, and delivered through a DevOps pipeline. Often, cloud-native applications are orchestrated by an orchestration infrastructure. Following are the characteristics as defined by CNCF:

Containerization

Cloud native applications are infrastructure agnostic and use containers. Containers provide the application with a lightweight runtime, libraries, and dependencies that allow the application to run as a stand-alone environment able to move in and out of the cloud—independent of the nuances of a certain cloud provider’s virtual servers or computer instances. With this, containers are able to increase mobility between different environments.

API Driven

The term means to ensure compatibility across platforms and simplify complexity by using APIs. The most common API pattern used in the industry is RESTful API. RESTful APIs are a component of loosely coupled architecture. REST systems interact through standard operations on resources, and do not rely on the implementation of interfaces.

Microservices

The term “microservice” is a design paradigm used in cloud-native applications to break down large components into multiple stand-alone deployable parts. The component is divided and deployed into smaller functional units. This way, maintenance in a module does not affect the other functionalities, since they can independently work on their own.

Horizontal Scaling

A key property of cloud-native applications is the ability to horizontally scale in and out based on the size of their payload.

Service Mesh

A service mesh is a dedicated infrastructure layer for handling service-to-service communication. The service mesh is responsible for the reliable delivery of requests through the potentially complex network topology of services. In practice, the service mesh is implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.

Delivery Pipeline

The state-of-the art software delivery pipeline is supported by the concepts of Continuous Integration/Continuous Delivery (CI/CD). CI/CD has the following stages/components:

1. Source code is stored in a Repository

2. Each time new code is pushed to a Repository, CI is automatically triggered to create a build (image).

3. The images are then stored in a Registry or Storage Bucket.

4. Through CD, images are pulled from the registry for deployment.

5. CD then pushes the image to the target environment (e.g., the Kubernetes cluster).

6.2.  Component Description Template

This section provides a template for formal description of components involved in the API experiments. The template is structured based on the following characteristics:

• Functional Description

• Implemented, supported or required OGC APIs

• Used data sources

• Technology Stack

• Components, libraries, configurations

• Development paradigm/process/approach:

  • Model Drives Software Development (MDSD), Domain Specific Language (DSL), Code/API generation?

  • Microservices? Service mesh?

  • Cloud native design/implementation?

• CI/CD:

  • Pipeline

  • Containerization

  • Container orchestration

  • Cloud deployment

7.1.  D165 API Experiments Server (Python)

The API Experiments Server implementation comprised separate implementations of OGC API — Features (Features) and OGC API — Environmental Data Retrieval (EDR). Both server implementations were developed in the Python programming language and around the Flask web framework. Flask provides the core capabilities to implement web interfaces. The basic implementation of both OGC API Standards is supported by the use of code generators. The model classes and controller classes (server stubs) are automatically generated from the respective OpenAPI specifications of Features and EDR by using the OpenAPI Generator project. OpenAPI Generator offers a specific code generator that produces basic Flask-based executable server applications.

The source code, corresponding documentation and deployment instructions are available on Github.

7.1.1.  OGC API — Features Server

The architecture of the Features implementation is structured into modules. The modular approach enables different types of data backends to be easily integrated in a similar way. Within this task, connectivity’s for OGC Web Feature Service (WFS) and Elasticsearch backends were implemented.

7.1.1.1.  Core Architecture

• Controller: Controller classes accept API requests, choose the correct backend to handle a request and return the result to the client. The controller classes (empty stubs) are generated by the OpenAPI code generator along with the model classes.

• Request transformer: A request transformer is specific to the backend type. Request transformers convert a request to the OGC API —  Features into a corresponding request to the specific data backend. For example a request to the /items endpoint is converted into a WFS GetFeature request.

• Query transformer: A query transformer is a backend specific implementation that translates query parameters into the domain specific language of the backend. For example a query transformer translates the datetime filter parameter of the /items endpoint into a WFS filter expression.

• Format transformer: A format transformer is optionally interposed if a data backend has no support for the GeoJSON format.

• Backend: A backend stores the configuration for an instance of a backend type (e.g URL of a WFS instance) and determines which type of request transformer has to be used for that specific backend type.

Figure 7 — schematic representation of the architecture of the OGC API — Features implementation with two data backends

7.1.1.2.  Data Backends

7.1.1.2.1.  OGC Web Feature Service Backend

The WFS backend can be used to create an OGC API — Features wrapper for an arbitrary WFS (2.x) instance. A request to the /items endpoint is converted to a GetFeature WFS request. The metadata for the collection endpoints of Features is automatically obtained and parsed from the WFS capabilities document.

7.1.1.2.2.  Elasticsearch Backend

The Elasticsearch backend implementation is mainly meant to be used with the OGC API — Records GeoJSON features that were generated and indexed in the D168 Data Backend and Deployment task. With the Elasticsearch backend the Features implementation can be used as an OGC API — Records implementation at the same time since the API endpoints are identical. The Elasticsearch backend is able to connect to Amazon Web Service (AWS) Elasticsearch Service/OpenSearch Service instances. Therefore the Elasticsearch Python package version 7.13.4 must be used because newer versions of the package do not connect to the AWS flavor of Elasticsearch.

In general the Elasticsearch backend is able to serve generic GeoJSON features indexed in an Elasticsearch index. An Elasticsearch index corresponds to a collection. The assumption is made that the GeoJSON features match the general structure of the OGC API — Records definitions in order to implement the filter capabilities (e.g. datetime filter) of OGC API — Features.

7.1.2.  OGI API — EDR (Environmental Data Retrieval) Server

In the same way as the Features implementation, the EDR implementation has a modular structure in order to enable seamless integration of different data backend types. Within this task the capability to perform data queries on NetCDF files was implemented. The EDR server application supports the position, radius and area data query endpoints of the EDR Standard. Though the OGC API — Features and — EDR implementations are separate applications the overall architecture is roughly the same.

When using the code generator there were originally many errors in the generated code for the EDR OpenAPI specification. The main reasons for the errors are the frequent use of the anyOf and oneOf keywords in the OpenAPI specification and the fact that the EDR specification is more comprehensive and complex compared to the Features specification. In order to mitigate these issues a custom, lean and simplified version of the OpenAPI EDR Standard was created. This custom version supports only two-dimensional, temporal data and only serves netCDF data as response. All data query endpoints that were not planned to be implemented —  that means all except position, radius and area  — were removed from the implementation to keep it as lean as possible. This resulted in fewer errors in the generated code which could be fixed manually afterwards.

7.1.2.1.  Core Architecture

• Controller: Controller classes accept API requests, choose the correct backend to handle a request and return the result to the client. The controller classes (empty stubs) are generated by the OpenAPI code generator along with the model classes.

• Request transformer: A request transformer is specific to the backend type. A Request transformer converts a request to the OGC API — EDR interface into a corresponding request to the specific data backend. A request transformer delegates the execution of data query requests to a data query transformer.

• Data query transformer: A query transformer is a backend specific implementation that executes data query requests for data served by a data backend.

• Format transformer: A format transformer is optionally interposed if a data backend has no support for the NetCDF format. Since the OGC API — EDR implementation only serves netCDF files and has only a NetCDF backend there are no format transformers used.

• Backend: A backend stores the configuration for an instance of a backend type (e.g location of a NetCDF file) and determines which type of request transformer has to be used for the specific backend type.

Figure 8 — schematic representation of the architecture of the OGC API — EDR implementation with a single data backend

7.1.2.2.  Data Backends

7.1.2.2.1.  netCDF backend

The NetCDF backend parses NetCDF files. Each file corresponds to an instance of a collection. The backend is adapted to the use of NetCDF land cover classification data provided by the D168 Data Backend and Deployment task. The general assumptions about the internal structure of these files is that each file contains a single data array, which has the dimensions x, y and time.

To implement data queries the NetCDF backend implementation mainly uses the Python packages Rasterio, xarray and Shapely. In addition, a package called rioxarray is used. Rasterio is used to clip raster data to a specific geometry and xarray is used to implement filter capabilities like the optional datetime filter. The internal data structure of xarray resembles the structure of a NetCDF file. rioxarray combines the functionalities of Rasterio and xarray which supports convenient use without transformation between (internal) data formats. Shapely is used to parse WKT geometries that are passed to the data query endpoints as query parameters.

7.2.  D166 API Experiments Server (JavaScript)

This section describes the server component as developed, documented and delivered by task D166 (JavaScript) and communicates all experiences and lessons learned back to the server provider to help optimizing documentation and scripts.

7.2.2.  Architecture

Figure 9 — Overall architecture

git clone https://github.com/opengeospatial/T17-API-D166.git

npm install

sudo pm2 start index.js

sudo pm2 stop index.js

"tl_2020_us_county": {
  "type": "collection",
  "title": "Tiger Line US Counties",
  "description": "Counties, US Census, TIGER/Line",
  "keywords": [
    "counties",
    "US",
    "TIGER/Line"
  ],
  "links": [
    {
      "type": "text/html",
      "rel": "canonical",
      "title": "information",
      "href": "https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html",
      "hreflang": "en-US"
    }
  ],
  "extents": {
    "spatial": {
      "bbox": [
        -179.231086,
        -14.601813,
        179.859681,
        71.439786
      ],
      "crs": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
    }
  },
  "providers": [
    {
      "type": "feature",
      "name": "PostGIS",
      "data": "postgresql://tluser:tl2020@localhost:5432/tl",
      "id_field": "ogc_fid",
      "title_field": "name",
      "table": "tl_2020_us_county",
      "geometry":{
        "geom_field":"wkb_geometry",
        "geom_format":"ewkb"
      }
    }
  ]
},

"DC_Building_Footprints": {
  "type": "collection",
  "title": "DC Building Footprints",
  "description": "DC building footprints.",
  "keywords": [
    "DC",
    "US",
    "building",
    "footprint"
  ],
  "links": [
    {
      "type": "text/html",
      "rel": "canonical",
      "title": "information",
      "href": "https://cubewerx.pvretano.com/cubewerx/cubeserv/default/ogcapi/usbuildingfootprints/collections/DC_Building_Footprints",
      "hreflang": "en-US"
    }
  ],
  "extents": {
    "spatial": {
      "bbox": [
        -77.115085,
        38.810444,
        -76.909707,
        38.99561
      ],
      "crs": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
    }
  },
  "providers": [
    {
      "type": "feature",
      "name": "WFS202",
      "id_field": "gml_id",
      "typename": "DC_Building_Footprints",
      "data": "https://www.pvretano.com/cubewerx/cubeserv?datastore=usbuildingfootprints&"
    }
  ]
},

"daraa": {
  "type": "collections",
  "title": "CubeSERV WFS - Daraa",
  "description": "CubeSERV WFS - Daraa service, support version 2.0.2.",
  "keywords": [
    "WFS",
    "feature",
    "capabilities"
  ],
  "links": [
    {
      "type": "text/html",
      "rel": "canonical",
      "title": "information",
      "href": "https://www.pvretano.com/cubewerx/cubeserv?DATASTORE=daraa&SERVICE=WFS",
      "hreflang": "en-US"
    }
  ],
  "providers": [
    {
      "type": "collection",
      "name": "WFS202Capabilities",
      "data": "https://test.cubewerx.com/cubewerx/cubeserv/demo?DATASTORE=Daraa&SERVICE=WFS&REQUEST=GetCapabilities&AcceptVersions=2.0.2&AcceptFormats=text/xml",
      "removeprefix":"cw"
    }
  ]
},

npm install

sudo pm2 start index.js

sudo pm2 stop index.js

"noaa_global_hourly_surface": {
  "type": "collection",
  "title": "The Integrated Surface Dataset (global, hourly)",
  "description": "The Integrated Surface Dataset (ISD) is composed of worldwide surface weather observations from over 35,000 stations, though the best spatial coverage is evident in North America, Europe, Australia, and parts of Asia. Parameters included are: air quality, atmospheric pressure, atmospheric temperature/dew point, atmospheric winds, clouds, precipitation, ocean waves, tides and more. ISD refers to the data contained within the digital database as well as the format in which the hourly, synoptic (3-hourly), and daily weather observations are stored. The format conforms to Federal Information Processing Standards (FIPS). ISD provides hourly data that can be used in a wide range of climatological applications. For some stations, data may go as far back as 1901, though most data show a substantial increase in volume in the 1940s and again in the early 1970s. Currently, there are over 14,000 'active' stations updated daily in the database. For user convenience, a subset of just the hourly data is available to users for download. It is referred to as Integrated Surface Global Hourly data, see associated download links for access to this subset.",
  "keywords": [
    "Integrated Surface Dataset",
    "Global",
    "NOAA"
  ],
  "links": [
      {
        "type": "text/html",
        "rel": "canonical",
        "title": "information",
        "href": "https://www.ncdc.noaa.gov/isd",
        "hreflang": "en-US"
      }
  ],
  "extents": {
    "spatial": {
      "bbox": [
        -180.00,
        -90.00,
        180.00,
        90.00
      ],
      "crs": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
    },
    "temporal": {
      "interval": [{
          "begin":"1972-01-01T00:00:00Z",
          "end":"1972-12-31T23:59:59Z"
      }],
      "trs": "TIMECRS[\"DateTime\",TDATUM[\"Gregorian Calendar\"],CS[TemporalDateTime,1],AXIS[\"Time (T)\",future]"
    }
  },
  "data_queries" : {
      "position": {
          "link": {
              "href": "/collections/noaa_global_hourly_surface/position?coords={coords}",
              "hreflang": "en",
              "rel": "data",
              "templated": true,
              "variables": {
                  "title": "Position query",
                  "description": "Position query",
                  "query_type": "position",
                  "coords" :{
                      "description": "Well Known Text POINT value i.e. POINT(-120, 55)"
                  },
                  "output_formats": [
                      "CoverageJSON",
                      "GeoJSON",
                      "IWXXM"
                  ],
                  "default_output_format": "IWXXM",
                  "crs_details": [
                      {
                          "crs": "CRS84",
                          "wkt": "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.01745329251994328,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
                      }
                  ]
              }
          },
          "linkrel":"relative"
      },
      "radius": {
          "link": {
              "href": "/collections/noaa_global_hourly_surface/radius?coords={coords}",
              "hreflang": "en",
              "rel": "data",
              "templated": true,
              "variables": {
                  "title": "Radius query",
                  "description": "Radius query",
                  "query_type": "radius",
                  "coords" :{
                      "description": "Well Known Text POINT value i.e. POINT(-120, 55)"
                  },
                  "output_formats": [
                      "CoverageJSON",
                      "GeoJSON",
                      "IWXXM"
                  ],
                  "default_output_format": "GeoJSON",
                  "within_units": [
                      "km",
                      "miles"
                  ],
                  "crs_details": [
                      {
                          "crs": "CRS84",
                          "wkt": "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.01745329251994328,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
                      }
                  ]
              }
          },
          "linkrel":"relative"
      },
      "area": {
          "link":                 {
              "href": "http://www.example.org/edr/collections/hrly_obs/area?coords={coords}",
              "hreflang": "en",
              "rel": "data",
              "templated": true,
              "variables": {
                  "title": "Area query",
                  "description": "Area query",
                  "query_type": "area",
                  "coords" :{
                      "description": "Well Known Text POLYGON value i.e. POLYGON((-79 40,-79 38,-75 38,-75 41,-79 40))"
                  },
                  "output_formats": [
                      "CoverageJSON",
                      "GeoJSON",
                      "BUFR",
                      "IWXXM"
                  ],
                  "default_output_format": "CoverageJSON",
                  "crs_details": [
                      {
                          "crs": "CRS84",
                          "wkt": "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.01745329251994328,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
                      }
                  ]
              }
          }
      },
      "locations": {
          "link": {
              "href": "/collections/noaa_global_hourly_surface/locations",
              "hreflang": "en",
              "rel": "data",
              "templated": false,
              "variables": {
                  "title": "Location query",
                  "description": "Location query",
                  "query_type": "locations",
                  "output_formats": [
                      "application%2Fgeo%2Bjson",
                      "text%2Fhtml"
                  ],
                  "default_output_format": "application%2Fgeo%2Bjson",
                  "crs_details": [
                      {
                          "crs": "CRS84",
                          "wkt": "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.01745329251994328,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
                      }
                  ]
              }
          },
          "linkrel":"relative"
      }
  },
  "crs": [
      "CRS84"
  ],
  "output_formats": [
      "application/geo+json",
      "text/html"
  ],
  "parameter_names": {
  },
  "providers": [
    {
      "type": "feature",
      "name": "PostGIS",
      "data": "postgresql://noaauser:noaa2020@localhost:5432/noaa",
      "id_field": "ogc_id",
      "locid_field": "station",
      "title_field": "station",
      "table": "noaa_global_surface",
      "geometry":{
        "geom_field":"the_geom",
        "geom_format":"ewkb"
      },
      "time_field":{
        "time_format":"datetime",
        "datetime":"date"
      }
    }
  ]
},

docker run -p 8080:8080 -d eugenegmu/ogc-api-features-javascript

docker exec -it <container id or name> /bin/bash

docker run -p 8080:8080 -d eugenegmu/ogc-api-edr-javascript

docker exec -it <container id or name> /bin/bash

7.2.6.  OGC API — Features Test Service

The test service is deployed at API-Features.

Landing Page

The implementation of the OGC API — Features Standard supports two media content types by default: application/json and text/html. Encoding negotiation is supported. For browsers, “text/html” would be the matching format. The following figure shows one example landing page.

Figure 10 — Landing page of the demonstration OGC API — Features server

Testing API using SwaggerUI: The SwaggerUI is enabled for users to test each API path. This is the response of an API document in text/html.

Features in a collection: The response for collection data supports encodings in both application/geo+json and text/html. The text/html shows a linked map and table for browsing items by pages. The following shows one example of a collection in the browser.

Figure 11 — Collection item page of the demonstration OGC API — Features server

7.2.7.  Test Service for OGC API — EDR

The test service is deployed at API-EDR.

Landing Page

The implementation of the API — EDR supports two formats by default: application/json and text/html. Encoding negotiation is supported. For browsers, “text/html” would be the matching format. The following figure shows one example landing page.

Figure 12 — Landing page of the demonstration OGC API — EDR server

Testing API using SwaggerUI: The SwaggerUI is enabled for users to test each API path. This is the response of an API document in text/html.

Data query in a collection: Several data queries are supported for each collection in the API-EDR service. The following figure shows one example collection which supports queries of position, radius, area, and location.

Figure 13 — The Collections page of the demonstration OGC API — EDR server

The response for collection data supports encodings in both application/geo+json and text/html. The text/html shows a linked map and table for browsing features by pages. The following shows one example of a collection in the browser. The request is https://aws4ogc17.webmapengine.com/edr/collections/noaa_global_hourly_surface/items?f=text%2Fhtml&datetime=1972-07-25T00:00:00.000Z/1972-07-25T23:59:59.000Z

Figure 14 — Collection item page of the demonstration OGC API — EDR server

7.3.  D167 Data Backend and Deployment

The D167 Data Backend and Deployment consists of three different components:

1. A tiles server generator, generating the tiles from OpenStreetMap data and publishing them through an Apache Web Server

2. An OpenStreetMap dataset generator, storing the data in PostGIS datasets (in AWS, CloudFerro or CloudSigma) or as cloud objects in AWS S3 buckets

3. A system to publish array-oriented datasets in a tiled manner, through a STAC catalog and storing the resulting NetCDF files in AWS S3 buckets

Figure 15 — Component diagram

The focus for all the components was in automation and ease of execution, following a DevOps approach and lowering the learning curve for new developers to deploy advanced environments. For example, the script for the tiles server generator covers the following actions:

1. Installs/compiles all the software dependencies.

2. Installs a PostgreSQL/PostGIS database and sets it up.

3. Downloads and configures a stylesheet.

4. Downloads OSM data for the specific country and inserts it into the database.

5. Downloads additional shapefiles and fonts.

6. Sets up Apache HTTP Server with rendered.

7. Launches Apache Server.

The process for deploying a tiles server using a Docker container was also tested and documented. In addition, a simple HTML/JavaScript client to test the tiles server was developed and added to the repository.

The idea of tiling NetCDF files was to experiment with solutions for data providers to offer array oriented datasets to service providers that can be consumed on demand or to select the data of interest based on the bounding boxes extension. As the concept of tiling is well understood in the geospatial world, instead of publishing a large NetCDF covering the whole world, the content can be tiled by levels, chopping the content into smaller chunks that are more easily transferable through the network. These smaller NetCDF files can be published through a STAC catalog that documents the extent of each file, allowing service providers to automate the search for the data of interest.

Figure 16 — NetCDF tiling concept

The source code and detailed documentation on how to deploy these components are available at https://github.com/opengeospatial/T17-API-D167. The datasets are deployed in the following endpoints:

Database Access

• AWS:

  • IP: 18.189.112.137

  • Port: 5432

  • User: apipostgres

  • Pwd: @P$_2021!

  • Database: dbapi4

  • Tables: geojson_places, geojson_waterways, geojson_polygons

• CloudSigma:

  • IP: 162.213.36.126

  • Port: 5432

  • User: dbasigma

  • Pwd: Db@_S$gma2021!

  • Database: gis

  • Tables: geojson_places, geojson_waterways, geojson_polygons

• CloudFerro:

  • IP: 185.178.85.176

  • Port: 5432

  • User: apipostgres

  • Pwd: Db@_F$rro2021!

  • Database: dbferro01

  • Tables: geojson_places, geojson_waterways, geojson_polygons

  • Example: psql -W -U apipostgres -p 5432 -h 185.178.85.176 dbferro01

AWS S3 buckets

• Polygons (2883 elements):

  • https://ogc-polygons.s3.us-east-2.amazonaws.com/

  • Examples: https://ogc-polygons.s3.us-east-2.amazonaws.com/332195921.json, https://ogc-polygons.s3.us-east-2.amazonaws.com/704263360.json

• Points (1776 elements):

  • https://ogc-points.s3.us-east-2.amazonaws.com/

  • Examples: https://ogc-points.s3.us-east-2.amazonaws.com/1143922944.json, https://ogc-points.s3.us-east-2.amazonaws.com/1242125200.json

Tiles server

• Area: Northeast Spain

• End point: http://3.143.92.80/tileserver/{z}/{x}/{y}.png

• Example: http://3.143.92.80/tileserver/0/0/0.png

7.4.  D168 Data Backend and Deployment

The aim of D168 was to be the provider of data and deploy a server component as developed, documented and delivered: Communicating all experience and lessons learned back to the server provider to help with optimizing documentation and scripts.

The testing involved storing the data in specific cloud environments, primarily AWS with some parallel testing of CreoDIAS, with a specific focus on cloud object storage. The API Experiments Server tested was D165.

7.4.1.  Development & Experiments

7.4.1.1.  Experiment 1: Data catalogs stored within AWS S3 bucket & OpenSearch

Pixalytics has an Earth Observation (EO) dataset that is currently stored in a directory/file structure on an AWS Expandable File Storage (EFS).

Copernicus Sentinel-1 and Sentinel-2 data were used to generate a land cover classification at 10-meter resolution to investigate sand dams as part of a project that focuses on sustainable sand extraction in Kenya. In addition, the datasets include other products derived from Sentinel-2 (vegetation related) and products derived from other satellite missions at coarser spatial resolutions.The dataset can be viewed online at https://voila.notebook.eo4sas.com/ as CaseStudy1 (one of three Jupyter Notebooks running as interactive webpages).

Within the framework of this activity, the dataset was transitioned to an accessible S3 bucket with the catalogue defined through the two options of STAC (https://stacspec.org/) and OGC API — Records. The image files were originally in GeoTIFF format but transitioned to Cloud Optimized GeoTIFF (COG) and NetCDF for the experiments.

This experiment involved five steps, with the code stored as shown in Figure 1:

1. Convert GeoTIFF files to COG and NetCDF format using convert_gtiff.py in utils folder

2. Create OGC Record or STACcatalogs using create_catalog.py in build_catalog folder of D168

3. Manual push of catalogs to AWS S3 buckets

4. Push catalogs to Elasticsearch using upload_esearch.py in deploy_catalog folder of D168

5. Deployment of D165 server providing API access to the catalogs with the specification in JSON files

Figure 17 — Representation of the D168 GitHub repository with the code folders (light blue) and code components (dark blue)

7.4.1.1.1.  Catalogs within AWS S3 bucket

The catalogs were created using Python code shared via the T17-API-D168 GitHub repository, under the build_catalog subfolder. The code uses yaml files (test-configuration.yaml and eo4sas-record.yml) to store configuration information, and rely on existing open-source code for STAC (pystac) and OGC API — Records (modified version of pygeometa to create the main catalog file) catalog creation. The GitHub ReadMe describes how to install the used anaconda environment and run the catalog creatin code.

Once created, the static catalogs are manually uploaded to the specifically setup S3 bucket that has public read access.

Examples of what is currently available via the S3 bucket include:

• STAC catalog v0-8 created using pystac for GeoTiFFs:

  • main JSON: https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-stac-v0-8/collection.json

  • image JSONs, e.g. https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-stac-v0-8/20200831T101156_rgb_classification/20200831T101156_rgb_classification.json

• STAC catalog v0-8 created using pystac for NetCDFs:

  • main JSON: https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-stac-nc-v0-8/collection.json

• OGC API — Records catalog v0-8 created using pygeometa for GeoTiFFs:

  • main JSON: https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-records-v0-8/catalog.json

  • image JSONs e.g.: https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-records-v0-8/eo4sas-record1.json

• OGC API — Records catalog v0-8 created using pygeometa for NetCDFs:

  • main JSON: https://pixalytics-ogc-api.s3.eu-west-2.amazonaws.com/eo4sas-catalog-records-nc-v0-8/catalog.json

7.4.1.1.2.  Catalogs deployed via Amazon OpenSearch (previously called Elasticsearch) Service with D165 Server

To support querying the catalogs, they were also deployed to Elasticsearch using Python code shared via the T17-API-D168 GitHub repository, under the deploy_catalog subfolder. The code uses a YAML file (es_upload_conf.yaml) for configuration of the deployment and then upload to Elasticsearch.

At the start of the Testbed 17 activity, OpenSearch was called Elasticsearch but since then the two have diverged. To maintain compatibility with what was developed the AWS deployment to Elasticsearch 7.10 was used. Also, version 7.13.4 of the Python Elasticsearch library needs to be used as more recent versions produce critical error messages.

• Elasticsearch endpoint on AWS [needs an IAM user account with authentication for access]: https://search-ogc-t17-d168-yhvlgzft2zhuvdssiaejkyq5lq.eu-west-2.es.amazonaws.com

  • STAC Catalog indices (for version 0-8): STAC-index (GeoTIFFs) STAC-index-nc (NetCDFs)

  • OGC Records indices (for version 0-8): records-index (GeoTIFFs) records-index-nc (NetCDFs)

7.4.1.1.3.  Deployment using D165 server

The deployment as an API using the D165 server is available at:

• OGC API — Features server with three catalogs (CubeWerx’s alongside Elasticsearch versions of Records and STAC GeoTiFF catalogs): http://ogcapi.pixalytics.com:8080/

• OGI API — EDR implementation with a single multi time-step NetCDF: http://ogcapiedr.pixalytics.com:8080/

7.4.1.2.  Experiment 2: Catalogs created and deployed using the CloudFerro CreoDIAS platform

As a second test, the D168 code was also run on the CreoDIAS platform. The data was still stored in the AWS S3 bucket, but pulled to CreoDIAS and the catalogs generated and deployed using the D165 server. These deployments have not been kept active so cannot be accessed.

7.5.  D175 API Experiments Client — Python

The D175 API Experiments Client — Python implements three different OGC APIs, each available in its own GitHub repository:

• OGC API — Features, available at https://github.com/opengeospatial/T17-API-D175-Features

• OGC API — EDR, available at https://github.com/opengeospatial/T17-API-D175-EDR

• OGC API — Routes, available at https://github.com/opengeospatial/T17-API-D175-Routes

All of the implemented APIs share a common structure and base code, as well as an automated and encapsulated deployment process that pulls the code from the respective repository. These implementations were conceived as data-centric clients, focusing the efforts in making them easy and quick to deploy and providing immediate access to the data.

Experiments were carried out in the auto-generation of client libraries for OGC API — Features and — EDR, based on the definition of a specific server API. The software used to auto generate the client libraries was OpenAPI Generator, in particular the python client (experimental). One of these experiments, auto generated against CubeWerx’s Daraa API, has been registered in its own repository and is available for deployment and tests at https://github.com/opengeospatial/T17-API-D175-Features_autogenerated.

Each client implementation consists of two different components:

1. A backend, built in Python 3 on top of Flask, a popular micro web framework. The backend implements the OGC API client calls to interact with the API server and fetch the data.

2. A frontend, built in HTML/JavaScript, receiving the interactions from the user through a web browser and translating those to the client backend, as well as visually displaying the data fetched from the server.

Figure 18 — Client architecture

This two-layered architecture enables separating the browser-client interface from the client-server interface, providing a greater flexibility in the design of the user interface. Clients can be deployed locally, but also remotely, at a different location than the API server or the user’s browser.

Clients use two common design patterns with this framework: The application dispatching pattern and the application as a package. This allows for a clean directory structure and readable code, making the different clients easy to maintain and deploy. Each client is launched in its own Python virtual environment, installing its own version of the required libraries without messing with the existing installation in the operative system.

Once the client is launched, it automatically reaches the server landing page and fetches all the important information, which is offered to the user via buttons, texts and drop-down menus. The goal is for the user to explore the API capabilities in a graphical and convenient way.

The User Interface for all clients have similar designs, with a main map to display results, a list of buttons to easily explore the API and a series of buttons and fields to make the most common queries. This user interface is stored in HTML, CSS and JavaScript files with a very simplistic design, but there is no limit as to the level of sophistication they can attain.

7.6.  D176 API Experiments Client — TypeScript

The TypeScript API Experiment client consists of three main components:

1. The API clients that are generated based on the corresponding OpenAPI specifications. (https://github.com/opengeospatial/T17-API-D176)

2. A connector for Web WorldWind that ties in the API responses into visual layers on WorldWind. (https://github.com/karriojala/WebWorldWind-OGCTB17)

3. A demonstrator that shows the API client’s capabilities in a real-world use case integrating with WorldWind, which bridges to plain JavaScript usage. (https://github.com/opengeospatial/T17-API-D176-dev)

The demonstrator is available to try here: https://ogctb17-apis-breithorn.solenix.ch/#/earth

Figure 19 — Demonstrator application using NASA WebWorldWind

The most interesting aspects about this experiment are:

• Execution in a Browser
  Browsers exhibit particular challenges for the execution and communication with remote services as they are executed in the restricted JavaScript sandbox in a browser. Considerations such as HTTP vs. HTTPS access, Cross-Origin Resource Sharing (CORS) and limitations in processing of data are to be considered.

• Exploration of the API Client in a real world scenario, using Web WorldWind.

• TypeScript with strong typing that significantly improves the available tooling for developers.

TypeScript is suitable for NodeJS environments and browsers, which spans a wide range of use cases.

The demonstration and discussed use cases focus on front-end development and visualizations in browsers, and less on complex processing or workflows of the results. Consequently, the developed use cases demonstrate the integration and common data request patterns for such web-based applications.

In addition to the OGC APIs Client source code, the documentation accompanying the source code provides examples for common use cases and explains the utility of specific end points and collections.

An important aspect of getting started with the OGC APIs is to get the initial bearings and identify where to find specific data, to learn the names and concepts used, and to learn how to explore available data sets. The documentation touches on these points as well, guiding a new user in getting started with the OGC APIs, data providers and data sets alike.

7.6.2.  OGC Code Sprint and Discrete Global Grid Systems

During October 2021 OGC API Code Sprint the client was extended with a demonstration for retrieving and visualizing DGGS (Discrete Global Grid Systems) zones. For this purpose, auto-generated TypeScript client was used in the same way as with OGC API — EDR and OGC API — Features.

Figure 20 — Demonstrator application showing DGGS zones

The top-level H3 zones were retrieved from the server and visualized on the globe. Clicking on the zones loads the children of the zones.

This demonstrates that code-generation can be a quick way to extend the capabilities of an existing client to support new APIs.

8.1.  Overview of OpenAPI

There are two goals for using OpenAPI:

1. Create comprehensive documentation of an intended communication interface.

2. Automatic generation of server and client-side code stubs as well as data transfer objects or structs.

The first point aids in the design, discussion and formalization of APIs during their inception and can be refined or adapted based on later implementation details.

The second point is particularly useful for creating compatible and comprehensive clients in a new programming language without requiring one to develop the boilerplate code for communication, message exchange, data representation, encoding and other well-understood challenges.

The OGC has embraced OpenAPI for documenting the new generation of OGC APIs and has used OpenAPI to design, iterate and ratify the new versions of the respective successor APIs to WMS, WFS, WPS and others.

OpenAPI is also an excellent means of providing the currently applicable capabilities of a particular server, where the server can provide an OpenAPI document on one of its endpoints. This is mandated by the OGC API — Common candidate standard.

In the frame of this testbed activity, the focus was on the use of OGC API — Features and OGC API — EDR and in demonstrating in an understandable fashion how to use those APIs in a manner that is accessible to developers that are not necessarily familiar with geospatial processing in general or the new OGC APIs in particular.

The following sections discuss various aspects that were identified in the course of this activity.

8.2.  Examples of Code Generation

OpenAPI based code generators are popular throughout the industry, wherever REST APIs are used and where OpenAPI has been used to formalize the interface.

Examples for successful use of generated clients are:

• The Kubernetes API clients that are available in a variety of languages. Similar to OGC API based servers, Kubernetes provides access to the extensible REST API that is currently available via OpenAPI specification endpoint.

• Cloud providers (Amazon, Google, Microsoft) provide REST APIs for communicating with their services. Software that interfaces with said services will often use generated communication stubs to interface with those services.

Large API surfaces — the part of a program that your users can openly interact with — having a large number of endpoints, data types, and/or implementations in different programming languages can benefit from code generation of the ‘boilerplate’ code to interface with a particular API. This leaves developers with more time to develop the added value by consuming or providing the services behind those APIs.

As mentioned in the introduction, the OGC API compliant servers are similarly set up with detailed APIs, which make it more difficult to create a comprehensive client implementation. Similarly, server implementations need to start somewhere as well and can benefit from a generated foundation.

8.3.  Caveats with Generated Code and Code Generators

The OpenAPI specification is very accommodating for API designers and developers in getting their ideas across and provides many degrees of freedom. Some of the aspects that would be beneficial for a more formal specification are optional and can be left out.

There are a limited number of functional OpenAPI code generators, which in turn have a large number of different back-ends that support the programming language, framework and programming style of choice.

The generated code is generally self-contained with proper namespaces, which is the desired behavior for any client library, or starting point for a server implementation.

There are a few caveats with using generated code:

• A new self-contained set of code with namespaces is generated for each OpenAPI specification.
  In the case of the OGC APIs, the OGC API — Features and OGC API — EDR standards are two separate such specifications.
  Any declarations in the API, even if they were included from the same underlying fragment, become part of the specific namespace. A Coverage response data structure for Features and for EDR will become two different structs or classes. There was no automated way to consolidate such shared declarations at the time of writing this ER.

• Manually changed code may be destroyed on re-generation.
  Inheritance or composition can work around these issues, where the needed manual changes are added to the generated functionality.

• Not all of the features of OpenAPI are easily transferable to all programming languages.
  A specific case identified during this activity were responses with anyOf or oneOf declarations, which would require union types, a higher-level abstraction or appropriate ‘holders’ and ‘markers’ that indicate which of the possible data types was returned.

Finally, there were also caveats with using the code generators themselves:

• The OpenAPI specification, which started as Swagger and was originally developed by the commercial entity SmartBear, is now an open standard. Multiple implementations of code generators are available with various back-ends of varying completeness and maturity.

• Because programming languages have different functionality, paradigms, supporting runtime frameworks and varying degrees of developer participation, the generated code can vary significantly in terms of resilience, error detection, recovery and verbosity of error messages during code generation.

Fortunately, there were no issues identified with the software license of the resulting generated code, which should encourage experimentation and use of OpenAPI code generators.

8.4.  Generating Code based on OGC OpenAPI Specifications

Code generators rely on consistent and clear instructions. Any ambiguity that would be fine for a human developer may trip up the code generator. This includes typos in declarations, missing links to other resources or incompatible data structure declarations.

Considering that the various OGC API Standards can be combined into a single server that provides the functionality of different APIs, some of the commonly available resources are shared or augmented with additional fields by the various APIs.

The approved OGC APIs, or those that are in development at the moment, are primarily aimed at human implementers. When using code generators, a few typos, inconsistent specifications of common endpoints and limitations of the code generators (as mentioned before) were identified.

There are three possible approaches for handling this circumstance:

1. Create distinct API clients for the different OGC API Standards and have the application decide, based on configuration, which client to use for a particular endpoint, the metadata provided for a particular collection or using other heuristics. This is the most standards-compliant approach that covers the functionality provided by the OGC API Standards. However, at the same time functionality is limited as it will not cover any server-specific extensions.

2. Generate clients for different APIs and manually merge them into a single client. This is the most work for a developer and is closest to developing a client from scratch. At the same time, it might be the most interesting solution for a server implementation, where the majority of work is in filling out the business logic behind the various resource handlers.

3. Implement a server endpoint that provides an OpenAPI definition document that is specific to this server. This definition document can also be used to bootstrap server development. A client based on such a server-specific OpenAPI definition document can utilize any specific extensions provided by the server, but is not guaranteed to work with any other implementation. The solution could be considered for highly-specialized servers that push the boundaries of what an existing OGC API provides.

The right approach depends on the scenario, desired level of interoperability and control that developers exert over the resulting code.

Approach 1 is the closest to the standards but also the most susceptible to minor deviations and errors in the OpenAPI definition documents used.

Approach 2 is the standards-adhering way to develop a multi-API server based on generated code.

Approach 3 leaves the most flexibility and gets the most out of the generated code in a coherent and unified code base, but may be limited in terms of interoperability if the developer is not careful.

In the frame of this activity, the generators for Python, JavaScript and TypeScript were exercised in more detail and information on those is provided in detail in the respective component descriptions in the Components Overview.

8.5.  Recommendations to the OGC for using OpenAPI

The main recommendations as outcome from using OpenAPI code generators as part of this activity are the following:

• Use code generators, in addition to SwaggerUI, ReDoc and other OpenAPI tools, in order to verify the consistency, completeness and correctness of OGC APIs in accordance with OpenAPI specifications.

• Provide feedback to the generator developers when bugs or issues are found in the general generator framework or a particular language back-end.

• Include commonly used declarations for common endpoints, data transfer types and responses in order to ensure full consistency.

• Find an agreed upon approach for the composition of different APIs in a single server and/or client, which allow implementation in a single server and single client side by side.

These recommendations, if considered, may lead to such OGC API Standards (or profiles of those standards), which would be more suitable for composition and code generation.

11.1.  Data backend and deployment

Table 1 — Server-Backend TIE Summary Table

Table 2 — TIE Functional Test

11.2.  Service invocations and data consumptions

Table 3 — Server-Client TIE Summary Table

Table 4 — TIE Functional Test