OGC Testbed-15: Machine Learning Engineering Report

This type of learning is usually implemented in game play applications and in use cases that include autonomous vehicle navigation as there is no "correct" answer to a particular problem. Instead the ML model looks to make the best decision given the circumstances with a view to maximizing cumulative reward. The reinforcement aspect is the application of the reward within the system, if cumulative reward increases then the system has a notion of a good decision and will seek to perform similar actions to further increase reward.

A Convolutional Neural Network (CNN) uses convolutions to extract features from local regions of an input. CNNs have gained popularity particularly through their excellent performance on visual recognition tasks. CNNs use relatively little pre-processing compared to other image classification algorithms. This means that the network learns the filters that in traditional algorithms were hand-engineered. This independence from prior knowledge and human effort in feature design is a major advantage. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, and natural language processing.

Recurrent Neural Networks (RNN) are a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows the model to exhibit temporal dynamic behavior. Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs. This makes them applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition.

The D100 component exposes a dedicated WPS enabled server (implementing the WPS 1.0 standard) for the requests. The WPS server is running on Flask, a Python lightweight Web Server Gateway Interface (WSGI) web application framework using PyWPS. The endpoint handles the following four different requests:

Considering that both network training and mosaic generation are demanding activities, these types of requests are queued in order to avoid blocking the WPS server. All the other requests are immediately served. For the cloud-free mosaic generation, requesting either one of the two defaults: ready network (one trained with 3 bands for Red-Green-Blue (RGB) and one with 4 bands for RGB + Near Infrared), or a new network trained (either 3 or 4 bands) with a dedicated WPS request is possible.

The queue mechanism relies on Remote Dictionary Server (Redis), an in-memory data structure and object persisting system supporting different kinds of abstract data structures. Every time the WPS server receives the training and mosaic generation requests , the new request is pushed on a Redis Queue (RQ), a Python library for queuing jobs and processing them in the background with so called "workers". The relevant job is not automatically run and its execution is remanded to RQ. A worker is another Python process running in the background as a work-horse to perform lengthy or blocking tasks instead of performing the task inside a web process. At least one worker is always up-and-running, but more than a single request instantiating more workers as needed according to application loading and hardware resources available (default configuration is 5 workers) can be served. Every time a worker is available, the job is retrieved from the RQ in a First In - First Out (FIFO) order and executed in a new dedicated process.

The Internal Storage component contains two different kinds of data: Trained network models and, inside what is called Workspace, all downloaded bands, tiles, intermediate cloud masks generated by the ML model, and the final mosaic for each request. By default, two models are present (as stated before one trained with 3 bands for RGB and one with 4 bands for RGB + Near Infrared). Any custom training requested by dedicated WPS call is also stored to be called later. All the models (stored as PyTorch checkpoints) are persistent in time. The Workspace content, instead, is preserved for each job only for a specific retention time. When time expires the specific folder is deleted to save storage.

When a WPS request is received for cloud mosaicking, the worker runs a dedicated job named Orchestrator. This was the core part of the D100 component and was composed of several different subcomponents as follows:

The mosaic generation was triggered by a specific WPS request. The following figure shows the sequence diagram for a generic mosaic generation process.

The D100 component receives a WPS execution request containing several input parameters (e.g. time window, ML model to be used and so on). The request is queued and waits for the first available worker to run the job. The client is notified with a response message indicating that the request was received and a new job was created with a specific Job ID. This ID is later used by the client to query the status of the request’s progress.

Any subsequent request about Job status will return one of the following status:

As soon as a worker is available, it queues a Job and runs the Orchestrator. As a first step, the Orchestrator queries the WCS server to retrieve all products covering the requested time window. The result list is then sorted in descending percentage coverage order, and all the products having cloud coverage greater than 70% are discarded. For each product found, the relevant bands are downloaded, sliced into tiles, inferred in the ML Model, and reassembled to generate a single cloud mask for the whole product. The Mosaic Builder also takes the product bands plus the cloud mask and merges one product at a time, stacking the different results respecting the descending sorting order. This process, from downloading the bands to image stacking is performed iteratively (i.e. handling one product at a time) until either the area is entirely cloud free or no more products are available. Finally, the resultant GeoTIFF RGB file is downloaded by the client.

The D100 implementation was delivered using two different networks having the same architectures but coping with tiles defined by either 3 (RGB) or 4 (RGB + NIR) bands. This approach is consistent with the two process profiles described in the OGC Testbed-14 Machine Learning ER and the ML best practices work from Testbed-14. This means that training or retraining of a network can be triggered by a dedicated WPS call. Considering the nature and context of the cloud detection system, having a dynamic dataset (mainly to validate the quality and accuracy of the network) is quite complex. Instead what can be requested is training a new instance of the network (choosing 3 or 4 bands architecture), asking for specific batch size and number of epochs. This new model is then stored in the Internal Storage and can be later recalled for the generation of a cloud free mosaic.

In order to handle all the WPS execute calls, at least one running worker shall be present. Run the following command line to start a new worker.

In order to assure that each job queued is served, the parameter -worker-ttl is set to -1 to disable expiration of the job.

Reviewing the status of the workers (i.e. how many are running and their queue status) can be achieved with the rqinfo command.

To stop the workers either kill the processes or close the prompt.

As described above, the WPS server is based on Flask and handles four different types of requests: Network training, cloud free mosaic generation, generation status query, and cloud free mosaic download. Beside these, the WPS instance exposes a generic interface such as the standard GetCapabilities.

In order to query the system about the status of a Job, a specific WPS execute service is exposed with the following parameters.

An example of this request is:

The status of the job follows the flow defined in Figure 9.

When the status query indicates processing has completed for the required Job ID, the URL for downloading of generated mosaic can be retrieved. A specific WPS execute service is exposed with the following parameters:

An example of this request is:

If the requested job is finished and the completion time is within the retention time period, the URL of the GeoTIFF RGB cloud free mosaic is returned. The URL is used to download the mosiac via standard HTTP protocol.

This software component is the core of the ML system and is in charge of searching and downloading product bands, loading and triggering the model, and creating the final cloud free mosaic in GeoTIFF RGB format. In order to optimize the execution performance of the Orchestrator (considering also that downloading of product bands is time consuming), the Orchestrator was designed with the following requirements:

The NRCan forestry CSW service endpoint is located at https://saforah2.nfis.org/geonetwork-main/srv/eng/csw.

Following a GetRecords request, the WCS Server returns all matching products tagged with the prf string and that were acquired within the range of the start / stop parameters provided in the job request.

The main configuration parameters for the Orchestrator are stored inside the config.yaml file:

First the Orchestrator checks if the area of the current product still contains clouds. If this is true, the bands are downloaded via a WCS endpoint (one WCS request for each band) in the dedicated job folder in the Workspace. The number of bands retrieved will be either 3 or 4 according to the required ML model.

Once all bands are locally available, they are cut into 224 x 224 pixels tiles via the Bands Slicer to be used for training the ML model. The requested ML model is loaded and run tile by tile (each tile composed by the different bands). The output is an equivalently sized black & white image showing cloud presence (in white).

When all tiles are processed and the relevant cloud masks created, the Mosaic Builder starts. The outputs from the ML model are merged to generate the overall cloud mask for the whole product. The cloud mask is then used as an alpha channel to maintain the areas being cloud-free and to have transparency (i.e. holes) for cloudy pixels. In the opposite way from the cloud mask, the alpha channel works by considering black as transparent (or 0% opacity) and white as solid (100% opacity). In order to use the cloud mask as an alpha channel the mask has to be inverted. When the alpha channel is applied, the resulting image becomes transparent in black portions of the alpha while in the pure white areas (the ones without clouds) the image is preserved. An example is shown in Table 7. Images in this table show transparent areas as red for viewing purposes only.

The newly generated image is then stacked below previous ones (if any) in order to cover the transparencies (holes) of the previous round with the newest image. This stacking order is needed to maintain the original reverse sorting designed to show most recent imagery of the area for the requested time range at the top of the stack.

Every time the Mosaic Builder terminates, it checks whether the flattened stacked images overlapping the Petawawa Forest area still contain clouds. If clouds are present, the Orchestrator performs another round handling the next product until either the area is either cloud free or no more products are available. When the job is terminated the final GeoTIFF RGB image is generated and stored in the Workspace and becomes available for download. From then on, any get_status request will return the job statistics as completed and the client can perform the get_result call to retrieve the download URL. The download is available until retention time expires. At that time the Workspace will be cleaned and the mosaic deleted.

This classification component takes the output from component D100 as the input for the deep classifier. It is represented by Landsat-8 cloud-free images (in .tif format and with three bands).

The label data, representing a ground-truth land cover classification of the Petawawa Research Forest, can be downloaded from this link. Grayscale pixel values are defined to represent specific land cover classes.

Only the overlapping part (the dashed box shown on Figure 18) of both the Landsat-8 imagery and the land cover classification dataset were used for this experiment. The data were eventually split into 375 pairs of remotely sensed and label image patches, with a size of 256*256 pixels. These paired patches were then randomly split into training set and validation set of 9:1. Specifically, the training set contains 337 image pairs, and the validation set contains 38 pairs.

For this dataset, there are a total of 21 land cover classes. The table below shows that the pixel distributions of the classes are unbalanced.

Figure 20. shows the learning curves of the model during training on the validation dataset with regards to accuracy. Accuracy indicates that, among all the pixels, how many of them are correctly classified by the model.

To help Testbed participants and the general public obtain convenient access to the trained deep learning model, the model was wrapped into a WPS and deployed using with Tomcat 8.5 and GeoServer 2.12 (official WPS extension included). If needed, the whole WPS can be easily deployed to any other desired server with some setup and configuration. Currently, the WPS can take any WCS or arbitrary web service that responses a valid GeoTIFF image as input, and provide an output image of land classification by performing preprocessing and deep learning model prediction. The figure below describes the general workflow of the WPS.

Once the user makes an Execute request, the WPS takes a 256 by 256 pixel GeoTIFF with RGB channels as input. This should be described in the request body. More commonly, the data would be provided by a WCS instance with a subsetting operation implemented. However, the WPS can be connected with any arbitrary web service as long as it returns GeoTIFFs that meet the implementation requirements. The WPS first downloads the input GeoTIFF from the given web service to the WPS server. The data will then be normalized and converted into a PNG image file. This operation not only generates the compatible input file for the trained deep learning model, but also provide users the convenience of comparing the input images and predictions. The normalization applied in each channel of the GeoTIFF linearly maps the values from an arbitrary range to 0-255. This PNG image file is cached in the server for future use and fed into a trained deep learning model. For each pixel in the input image, a corresponding land classification label is predicted by the model. With a proper colormap, the predicted land classification image is rendered. The WPS provides an optional control to decide whether to return the normalized/rendered input image or the prediction image (input image by default). Finally, the WPS prepares the response containing the specified image with some auxiliary information, such as response mimetype, and returns the result to the user.

An interface description of the implemented WPS can be obtained using the WPS DescribeProcess request: http://cici.lab.asu.edu/geoserver2.12.0/ows?service=wps&version=1.0.0&request=DescribeProcess&Identifier=gs:CNNProcessor. The DescribeProcess document is:

There are two input parameters for this WPS instance. The first one is titled “coverage”, which should be a WCS request that delegates to the WPS server. This request should respond with a 256 by 256 pixels GeoTIFF with RGB channels. The second input is titled as “output”. It takes a string as input. If the string is exactly “prediction”, the WPS returns a predicted land classification image. Otherwise the WPS returns the normalized/rendered input image. The output of the WPS is titled as “result”. Currently only PNG as output format is supported.

The prediction result can be obtained using a WPS Execute request. Users should prepare a valid WPS Execute XML body that meets the WPS interface and post the XML to http://cici.lab.asu.edu/geoserver2.12.0/wps. An example post body is:

In this request body, the input parameter “coverage” is described as a WCS post request with DomainSubset operation that subsets a 256 by 256 pixel area from the whole coverage. The post request delegates to the WPS server and sends a request to http://geoserver/wcs. This link is the short reference of the WCS service implemented by the same GeoServer as the WPS. The link is the same as http://cici.lab.asu.edu/geoserver2.12.0/wcs. Input parameter “output” is specified as “prediction”, so the response will contain the predicted land classification image. The response image and corresponding legend in the right-hand side are demonstrated as follows:

Change the input parameter “output” to an arbitrary word, we get the following normalized/rendered input image as the comparison:

There were many parameters to tune optimally for RL. Each type of agent has a single neural network to train, even if there can be several agent instances running in parallel in New Brunswick. This is the case for the harvest and transport teams: Sharing a single brain that concentrates all experiences from all the different teams, but each acts independently in their own area.

The parameters that can be tuned for each agent type are:

The following graph shows the evolution of the value of threshold; that is the probability that an agent chooses a random action vs. deciding it based on its experience. Actions will be chosen randomly as the memory is filled with experiences, and only then, when the neural network starts to learn, will the Epsilon parameters be used.

The process to train an agent is similar in all three cases. The environment provides the state to the agent, then the agent gets the best action from the neural network, modifying the state from the environment and receiving a reward. This process is stored and triggers a learning cycle, improving the neural network.

In the case of a harvest agent that has as an assigned forest block and stand, the environment provides a state containing the following information:

With this information, the agent chooses one of the following actions:

In the case of the transport agent, which has a wood stock assigned, the environment provides a state containing the following information:

With this information, the agent chooses one of the following actions:

The reward received will be the value of the transported wood (which will depend on the species, the type of mill, the stock of the mill and the wood price at that moment) minus the transportation cost.

The case of the planning agent is different, as this agent cannot be trained independently of the other two agents. The goal is to balance the harvesting of different species in order to optimize flow. This agent only takes into consideration the status of the stock in the different mills as well as current wood prices for each type (hardwood, SPF, softwood) and their trend.

With this information, the agent chooses one of the following actions:

Each priority increase for one type of species makes a small priority decrease for the rest, keeping the numbers balanced. Before the agent takes the new state and the reward, in order to see the effect of its action it waits for several steps from the other teams.

Once all agents are properly trained, they can be set to play the game harmoniously and reassess during the number of episodes set by the user. While running the episodes, the neural networks do not go through learning iterations and the threshold is set fixed to Epsilon end in order to keep a minimum randomness in the process.

The episode with the highest reward is then recorded for further manual analysis. Additionally, the top 25 percentile episodes generate aggregated statistics to find further insights that can help prepare a multi-year plan. These statistics include the forest blocks that were harvested in most of these top episodes, as well as the blocks where a road was built or the average stock in the mills.

Wood prices are available at https://www.nrcan.gc.ca/current-lumber-pulp-panel-prices/13309

Being able to forecast wood prices should have a positive impact in fine-tuning the behavior of transport agents. This is because the reward they receive will be more accurately estimated and therefore they will be able to make better decisions on what wood to transport to which mill at each moment. This approach should also have a positive impact in improving the behavior of the planning agent. Instead of just monitoring the storage status of mills, the model is also aware of the price evolution of each type of wood; it therefore has a richer set of information to base its decisions.

The wood prices data presented two main challenges: 1) they are not directly related to wood species but to a mixture of wood type and mill products, and 2) they only provide 12-month data. The first challenge was solved assuming direct relations. "Softwood lumber" refers to prices for every wood from softwood species, "Panel" refers to prices for every wood from hardwood species and "Pulp" refers to prices for every wood from SPF species. This is a gross approximation and should be refined in further revisions.

The second challenge was solved by assuming that the 12-month data is actually the seasonal component of the timeseries. Again, this approximation is objectionable but necessary, as without multi-year data it is simply not possible to find the seasonal component of a timeseries.

With these assumptions, price variations might vary up to 40% in a year. This should have an important effect on the rewards and the decisions made.

Fuel prices are available at https://www.nrcan.gc.ca/current-lumber-pulp-panel-prices/13309

As costs will be more accurately estimated, being able to forecast fuel prices should have a positive impact on fine-tuning the behavior of transport agents. Thus, transport teams should favor closer mills when fuel prices are high and high yield mills when fuel prices are low. The fuel prices data offer a very complete dataset of the monthly evolution of fuel prices (gasoline, diesel, others) since 1979 in different parts of Canada. For this model, the data included starts in year 2000, is from Saint John, New Brunswick, and represents diesel fuel at full service or self-service filling stations. In order to have consistent comparisons, prices are adjusted to their 2019-equivalent using historical Canadian inflation rates.

Analyzing the data series of fuel prices with the Dickey-Fuller test draws a p-value of 0.060783, meaning that the series is not stationary and can be decomposed.

Once is the data are decomposed into the three main components, analyzing the residual component with the Dickey-Fuller test draws a p-value of 0.000009, meaning that that component is strongly stationary as would be expected. The different components are visible in the next image:

Further analyzing the different components, three important conclusions are drawn:

Taking all this into consideration, the seasonal component of the diesel price will be included in the model, even though it is not going to have a significant impact nor is it going to help improve accuracy, as it is just a minimal part of the price forecast.

However, even if it does not seem possible to have an accurate long-term forecast of fuel prices, it might be interesting in the future to run the model with different scenarios depending on fuel (or wood) prices, such as high fuel prices - low wood prices, and evaluate how all the different agents would behave. Or, what would be the price level at which the rewards will be higher for teams to stay at home than going to work (that is, revenues are lower than costs). These applications are beyond the scope of this project.

PyTorch was used to create and train the neural networks. For each agent, the network starts with a layer that has the same number of artificial neurons as the number of state variables. Then, the agent creates as many hidden layers with as many artificial neurons as specified by the user. Finally, the last layer contains as many neurons as the number of actions the agent can take. Between artificial neuron layers, it adds a simple Relu activation function, that is, the positive part of its argument.

In order to train the model, an algorithm called Q-learning is used, which is able to find the optimal action taking into consideration not only the immediate reward received, but also the expected rewards generated by all successive steps applying the gamma parameter (or discount factor) described previously. Q is actually the name of the function that calculates the quality of an action used to provide the reinforcement.

However, when applying Q-learning in combination with a neural network (also called Deep Q-Learning), there is the risk of making the learning process unstable or divergent. This is because small updates to Q may substantially change the policy and data distribution, and thus change the correlations between Q and the target values. In order to overcome this loss of stability, two complementary techniques were used in this project: Experience Replay and Target Network.

With these techniques there will be a stable and convergent Deep Q Network. However, it is easy to for the network by adjusting the parameters, for example:

52North’s WPS2.0 compliant JavaPS was provided to extend ML processing in a networked manner. Users could use the JavaPS to conduct three actions via the WPS interface: Configure the neural network, train a given agent type, and run a specific model. These actions respectively are reachable at:

The configuration endpoint allows users to both query current model configuration and set new configuration values. All parameters except team_type are optional.

A successful POST request to the configuration endpoint returns back all current configuration settings for the specified agent, including those not changed by the user’s configuration request:

The various agents must be trained prior to execution. A user specifies which of the three agent types to train and the number of agents running in parallel. Harvest and transport agents must be trained before training the planning agent.

The WPS returns a link to a JSON collection describing the results of the training process.

When launching the execution, users may optionally specify the number of episodes to run and the length of those episodes.

Much like the training process, the WPS provides a response storing the best episode results and an aggregated summary of execution in JSON and GeoJSON. These results are best retrieved asynchronously, as long- running tasks may run into issues with connectivity being maintained across long periods of time.

Additionally, the WPS provides all compliant methods such as GetCapabilities and DescribeProcess.

The application’s workflow was fixed and configuration-driven. All dependencies were packaged in a base Docker image which was extended to include a trained model, thus becoming an application. The application then offered a single-entry point, either for training or inference. The figure below depicts the ML application and the embedded workflow.

A configuration file for the application is presented in Annex A, while some excerpts are provided in subsections below.

The two main sources of input data were a High-Resolution Digital Elevation Model (HRDEM) and a subset of the Geobase du réseau hydrographique du Québec (GRHQ) hydrographic network. The addition of satellite imagery was considered, but discarded due to potential temporal inconsistencies. The region of interest was a patch of land of about 200 square kilometers, north-west of the cities of Gatineau and Ottawa. Both data sources were downloaded from NRCan’s NFIS Geoserver at https://opendata.nfis.org/, through WCS and WFS 3.0 respectively. The figure below presents the region.

For the considered region of interest, the HRDEM has a file size of 252 GB, for a coverage of 210 000 by 300 000 pixels and a spatial resolution of 1 meter. The maximum elevation value found in this region is 609.1 meters. The hydrographic network dataset contains 36,109 differentiated features, composed of 34 678 lakes and 1431 rivers. The projection used in both was NAD 83 / UTM zone 18N. The main attributes of the hydrographic network features are presented in the following table. The type of feature is represented by TYPECE, where a value of 10 corresponds to rivers and values of 21 and over represent lakes.

The goal of the model is, given a set of hydrological features as well as HRDEM raster data, to differentiate lakes from rivers. This goal can be reformulated as the detection of lakes since these are more easily defined as entities and easier to observe as a whole. As such, using a ML model trained to recognize and localize lakes from a set of mixed hydrological features, the goal of differentiating them from rivers was also accomplished. Therefore, the object detection in this model targets only lakes.

Object detection models in the context of machine learning now come in many different forms. To simplify development, the application uses a deep convolutional network combined with a region proposal network [1]. This combination of models coined Faster R-CNN can be trained for the regression of bounding boxes around objects of interest. While such axis-aligned bounding boxes are not ideal for the fine-grained fragmentation of hydrological features, it was determined that post-processing can be used to optimize or refine the boundaries between rivers and lakes. The use of an instance-based detection and segmentation approach as in [2] would be more appropriate, as the segmentation masks would remove the need for the post-processing of region boundaries.

The train model process then uses the tensor and starts the iterations. A log output of model training can be found in Annex D. Below is the description of a default, pre-trained model as a starting point for the process. It can be noted that while custom architectures can be specified, their metadata, implementation and dependencies need to be provided separately. The implementation of D104 used mostly default architectures supported by PyTorch.

As documented in ADES and EMS best practices, a CWL descriptor is first produced for the application. Below is an excerpt of Appendix A, presenting the process description that encapsulates the ML app. The automatically generated CWL file, to be passed in the content:unit field of the owsContext, is presented in Appendix B.

CSW records were read from the Training CSW and 5 data fields were used:

The first three fields above are text-based fields which needed to be converted to numeric representations for use in the model. The Natural Language Toolkit (NLTK) library was used to normalize and extract stop word and training word tokens in the Title, and Abstract fields. Once the word tokens were extracted, the Testbed participants investigated using two algorithms for computing numeric keyword scores:

The records in the Training CSW were then used to train the various ML models.

Compusult has implemented a Multilayer Perceptron (MLP) neural network using Python and Tensorflow. The MLP contains three layers as:

The results that were generated during the MLP model training are outlined in Table 13 below. Table 12 describes the columns that make up that table.

After the model was generated the basic structure of the MLP model is as described in Table 14.

Total params: 502,017
Trainable params: 502,017
Non-trainable params: 0

During this pilot the following results were achieved:

This Arctic Discovery Catalog is available online to the sponsors at link: http://ogc.compusult.com

The output Arctic Discovery Catalog is based on the OGC CSW v2.0.2. It is available online via a CSW interface. Users can also log into WES and browse the catalog at http://ogc.compusult.com (user account required). The catalog is documented in the Testbed-15 D010 Catalog ER.

Since the Arctic Discovery Catalog is a standards-based catalog, any user/system that has implemented the OGC CSW standard can inter-operate with the Arctic Discovery Catalog using standard interfaces.

Compusult investigated implementing a multilayer CNN using Python and TensorFlow but due to the high accuracy rate of the MLP it was determined unnecessary due to the additional effort. The proposed CNN would have 7 layers as follows:

The same dataset that was used to train the MLP would be used to train the CNN.

Compusult also investigated the use of an RNN to further enhance information extraction and processing of the Title and Abstract fields. An RNN would provide additional context to the complete sentence structure but this was deemed unnecessary for a simple categorization problem such as Arctic vs non-Arctic. This might be a candidate for a future Testbed.

An evergreen-catalog harvester that uses ML to help search engines associate documents with similar thumbnails.

The harvester discovers new candidate web pages for possible inclusion in the catalog. If that page has a thumbnail in its metadata, the harvester could feed it to an ML model (almost like an image classification model) to help determine whether the candidate should be added to the catalog or not. The thumbnail similarity would be just one bit of evidence among many. However, if a minimally viable use case is being considered, the result of the thumbnail classification would determine whether to add the new page or not.

Explore unsupervised learning - The idea would be that there are some web assets (documents, datasets, etc.) that an ML model might consider worthy of including in an evergreen catalog. But we might not (yet) be able to describe the reason why in words. It could just be that the asset has the right mix of feature values to make it sufficiently similar to the assets used in model training. This is somewhat similar to the "anti-D103" approach, as it could be perceived as working against the goal of building a verbal semantic network.

There are currently two schools of thought on API structure for OGC API – Processes. These are broadly as follows:

The Type 1 pattern is the most simplistic in terms of structure as it does not mandate prefixes or operations. In the Routing API Pilot, an example operation to get a route is as follows:

The thinking behind this pattern is that the API architecture for the OGC is resource-based. Therefore all operations should be treated as resources. The purpose of the Routing Pilot work was to get a route via the interface. The method that the route is generated by is largely opaque to the user (or machine) that is calling the service.

The alternative (Type 2) describes a more typical OGC pattern that seeks to preserve some of the capabilities/patterns from WPS 2.0. This approach enables implementers to simply wrap their service according to the OGC API – Processes specification and have the calls pass through to the old service. This approach does additional work in that it preserves the concept of WPS throughout the OGC and applies some rigidity for implementers to design clients against. Which method is eventually adopted remains to be seen. Some example, potential patterns for ML using Type 1 are as follows:

The equivalent patterns for the Type 2 pattern could be represented as:

With the process ID put in the POST body, this returns a job-id GET /processes/machinelearning/jobs/<job-id>/result. Type 1 is clearly more flexible, whereas Type 2 offers structure. The actual decisions on an API pattern for the OGC could be defined in a Pilot program or future Testbed.

During the implementation of the D102 component, the input data were analyzed from an interoperability point of view, and three different types of data were found:

Based on work presented in D104 Component section of this ER, the following recommendations for future work are suggested:

Based on discussions with participants and experiences from this initiative, an approach for future work is discussed in this section.