Publication Date: 2018-01-08

Approval Date: 2017-12-07

Posted Date: 2017-10-30

Reference number of this document: OGC 17-029r1

Reference URL for this document: http://www.opengis.net/doc/PER/t13-NG009

Category: Public Engineering Report

Editor: Benjamin Pross, Christoph Stasch

Title: OGC Testbed-13: Workflows ER

OGC Engineering Report

COPYRIGHT

Copyright © 2018 Open Geospatial Consortium. To obtain additional rights of use, visit http://www.opengeospatial.org/

WARNING

This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.

LICENSE AGREEMENT

Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.

If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.

THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.

This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.

Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.

This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.

None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.

Table of Contents

1. Summary

This Engineering Report (ER) addresses the development of a consistent, flexible, adaptable workflow that will run behind the scenes. A user should be able to discover existing workflows via a catalog and execute them using their own datasets. An expert should be able to create workflows and to publish them. Previous OGC Testbed initiatives investigated workflows in the geospatial domain:

  • OWS 3 Imagery Workflow Experiments

  • OWS 4 WPS IPR Workflow descriptions and lessons learned

  • OWS 4 Topology Quality Assessment Interoperability Program Report

  • OWS 5 Data View Architecture Engineering Report

  • OWS 6 Geoprocessing Workflow Architecture Engineering Report

These initiatives mostly favored Business Processing Execution Language (BPEL) as the workflow execution language. More recent studies ([6], [7]) were performed using BPMN as a means for describing and executing workflows comprised of OGC Web services. This ER will give an overview about existing approaches to compose and execute geospatial workflows and will describe the approach taken in Testbed-13, taking into account security aspects.

Parts of this work have been funded by the COLABIS and Mudak-WRM projects funded by the German Federal Ministry of Education and Research under grant agreement numbers 03G0852C and 02WGR1431C.

1.1. Requirements

As stated above, the workflow is required to perform the following tasks:

  • Gather data

  • Check ellipsoid/projection

  • Check data quality

  • Run conflation

  • Deliver data

A planned workflow (i.e. a workflow constructed ahead of time) shall be developed combining the conflation Web Processing Service (WPS) 2.0 and data quality WPS 2.0 implemented in Testbed-12. In order to secure the workflow, the following items need to be taken into account:

  1. OGC Services and content encodings

  2. Fine grained access control implemented through an Attribute Based Access Control (ABAC) infrastructure

  3. Services and content are provided by more than one organization utilizing more than one security environment

  4. Services are not accredited to the same level of trust

  5. Some content may require a higher level of protection than some services are accredited to provide

  6. All actions must be logged and associated with the user who initiated the service chain

  7. Some of the content is sensitive (Personal Identifiable Information). Unauthorized release of this data must be avoided.

1.2. Key Findings and Prior-After Comparison

The workflow package in Testbed-13 has the goal of developing a consistent, flexible, adaptable workflow that will run behind the scenes, is described in a standardized format (BPMN) and could be shared to and executed by other users. In a nutshell, the workflow aims to automate conflation of road datasets and pre-processing steps (coordinate transformation, data quality checks). The processing steps are implemented as WPS processes and the data is passed by reference to data servers that include Web Feature Services (WFS) and Web Coverage Services (WCS) between the different processing steps.

The work and experiments that were done showed that it is possible to use OGC Web services in workflows that are protected by different security mechanisms, i.e. X.509 certificates and OAuth. Concepts of combining mainstream security with OGC Web services that have been developed during past testbeds and that are currently discussed in OGC have been proven to work in geospatial workflows. It has been shown that workflows can be created, wrapped in WPS processed and made discoverable in catalog services using current standards and technologies.

1.3. What does this ER mean for the Working Group and OGC in general

Workflows that transform geospatial datasets to derived geospatial information are usually composed of several processing steps. These steps may be provided by different WPS endpoints implementing different process profiles. Up to now, a common approach for defining, sharing and executing complex workflows (e.g. those composed of processes from different service providers) has been non-existent. This ER describes the approaches taken in OGC Testbed-13 to address these issues. Besides the general question of how to compose, share, and execute a workflow consisting of several WPS processes, different approaches for dealing with security issues such as identity mediation or delegation are also described in this ER.

Several approaches regarding workflow composition have been discussed in the Workflow DWG, [6] being the most recent document. However, no common approach has been formalized yet. This ER describes a further proof of concept of the approach described by [6], which could lead to the creation of a best practice document regarding geospatial workflow composition and execution.

1.4. Document contributor contact points

All questions regarding this document should be directed to the editor or the contributors:

Table 1. Contacts
Name Organization

Benjamin Proß (editor)

52°North GmbH

Christoph Stasch (editor)

52°North GmbH

Dimitar Misev

Rasdaman

Mark Lawrence

Compusult

1.5. Future Work

This ER shows that BPMN can be used to compose, discover and execute secured workflows in the geospatial domain. In the future, a general approach for BPMN engines to communicate with OGC Web services should be investigated, including the treatment of inputs and outputs. Also, the encoding of security aspects in BPMN should be defined. A transactional extension for WPS 2.0 and the extension of the WPS 2.0 StatusInfo document to add more information about the process status should be discussed in the WPS 2.0 SWG.

1.6. Foreword

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

2. References

The following normative documents are referenced in this document.

3. Terms and definitions

For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard OGC 06-121r9 shall apply.

4. Abbreviated terms

  • API Application Program Interface

  • AWS Amazon Web Services

  • BPEL Business Process Execution Language (also WS-BPEL)

  • BPMN Business Process Model and Notation

  • CSW Catalog Service for the Web

  • CRS Coordinate Reference System

  • ER Engineering Report

  • OGC Open Geospatial Consortium

  • OWS OGC Web Service

  • WPS Web Processing Service

  • WFS Web Feature Service

  • WCS Web Coverage Service

5. Overview

This Engineering Report describes the results of the OGC Testbed-13 Workflow thread.

In the following sections, the scenario is described, followed by an overview on workflow description formats and execution engines. The developed/used components are described in sections 8 - 11. Section 12 describes the security aspects and section 13 summarizes the recommendations.

6. Scenario

The overall goal is to create a conflation workflow consisting of several (pre-)processing steps needed to conflate road datasets of different quality. Conflation, also known as map matching or merging is the process of combining two datasets based on a set of rules to produce a more complete dataset. One example is the combination of authoritative data, e.g. authoritative road data, with crowd-sourced data, e.g. OpenStreetMap (OSM). The authoritative data is collected using certain guidelines and must follow strict requirements to ensure data quality. This leads to a smaller frequency of updates. Crowd-sourced data on the other hand, can be collected by virtually everyone. In this case, the quality of the data is monitored by the community and usually has a high volume of updates and therefore much more prone to error. As a result, the data is usually of lower quality.

The general conflation workflow consists of three major steps as shown in the figure below.

workflow overview
Figure 1. Workflow Overview

The first step is to check whether the road datasets are in the same coordinate reference system. If not, a coordinate transformation will be executed (WPS-1). The next step is to check the data quality of the road datasets (WPS-2). Thereby, the mean minimum distance between the road datasets and a set of control points is checked. The results are returned as ISO data quality. The last step in the workflow is the actual conflation (WPS-3) that is an extended version of the Conflation WPS developed in Testbed-12 that uses the Hootenanny conflation software[1] at the backend.

The workflow client can be used for two purposes:

  • Workflow composition: An expert user defines the workflow by discovering suitable WPS processes in a catalog service and composing them in a BPMN document.

  • Workflow execution: A workflow user can use the client to execute a workflow process.

The following two sequence diagrams illustrate the general sequences for both use cases.

6.1. Workflow Creation

Workflow_Creation_Sequence
Figure 2. Workflow Creation

Step 1:

  • The expert queries a catalog for processes suitable for the workflow.

Step 2:

  • The expert composes the workflow using the workflow client (BPMN editor).

Step 3:

  • The expert then selects the option in the client to send the workflow defined in the BPMN file to the workflow engine.

Step 4:

  • In order to process such a request, the workflow engine requires a security access code. The workflow client (WPS client) sends a request to the OAuth Security Service to get an access code and then includes it as part of the request.

  • The workflow engine creates a new process identifier and uploads the BPMN to the underlying BPMN engine. The BPMN engine returns an internal process identifier that can be used to start the process. A wrapper process is created in the workflow engine WPS. The WPS then returns the process identifier of the wrapper process.

Step 5:

  • The workflow client then contacts the CSW and instructs it to harvest the new process in the workflow Engine WPS.

6.2. Workflow Execution

Workflow_Execution_Sequence
Figure 3. Workflow Execution Sequence

Note: Only the security aspects of the workflow client and -engine interaction are shown here. For details about the security aspects of the workflow in general, please refer to Workflow Security

Step 1:

  • The user launches the workflow client and searches the catalog for workflows (i.e. WPS process descriptions) using keywords and data types that match their current needs.

Step 2:

  • The user finds the applicable conflation WPS process and provides the list of datasets and other input parameters for processing.

Step 3:

  • In order to process an WPS Execute request, the workflow engine WPS requires a security access code. The workflow client (WPS client) sends a request to the OAuth Security Service to get an access code.

Step 4:

  • The WPS client then constructs the applicable WPS Execute request (including the security access code) and sends it to the workflow engine WPS.

Step 5:

  • The workflow engine now invokes the first WPS by passing the references to the road datasets and a target CRS to the WPS. If the CRS is different from the target CRS, the coordinates are transformed.

Step 6:

  • The data quality is checked using the Data Quality WPS. A coverage is requested from a WCS and quality checks are performed.

Step 7:

  • If the quality is sufficient the actual conflation is executed by invoking another WPS process. The WPS then inserts the resulting dataset in a WFS-T (or WCS-T, depending on the result type) and returns a reference to it. The reference is then forwarded to the client and may be visualized in a map.

6.3. Datasets

The workflow is run for conflating road datasets in two regions: New York City in the United States of America and Za’atari in Jordan. The datasets available are described in the following two subsections.

New York City:

  • The National Map

  • OSM

  • Digital Globe satellite images

Za’atari Datasets:

  • UN

  • OSM

  • Digital Globe satellite images

6.4. Technology Integration Experiments

The following tables list the Technology Integration Experiments (TIEs) that were performed in the workflow scenario. See TIE results for additional information about the TIEs.

Involved components:

Abbreviation

Component name

Deliverable

WCL

Workflow Client

NG136

WCA

Workflow Catalog

NG135

WEN

Workflow Engine

NG131

CTP

Coordinate Transformation Process

NG130

DQP

Data Quality Check Process

NG130

COP

Conflation Process

NG130

WFS

Workflow WFS

NG132

ABU

Amazon Bucket

N/A

IBR

Internet Browser

N/A

TIEs:

TIE between

Security

Use Case

Successful

WEN - CTP

x.509, OAuth2: Client Credentials

Dominating Privileges

Y (04.10.2017)

WEN - DQP

x.509, OAuth2: Client Credentials

Tunneling Proxies

Y (04.10.2017)

WEN - COP

x.509, OAuth2: Client Credentials

N/A

Y (04.10.2017)

CTP - WFS

x.509, OAuth2: Client Credentials

Dominating Privileges

Y (04.10.2017)

DQP - WFS

x.509, OAuth2: Client Credentials

N/A

Y (04.10.2017)

COP - WFS

x.509, OAuth2: Client Credentials

N/A

Y (04.10.2017)

COP - ABU

x.509, IAM

Identity Mediation

Y (04.10.2017)

WCL - WEN InsertProcess

OAuth2: Client Credentials

N/A

Y (20.09.2017)

WCL - WEN Execute

OAuth2: Client Credentials

N/A

Y (04.10.2017)

WCL - CTP

N/A

N/A

Y (20.09.2017)

WCL - DQP

N/A

N/A

Y (20.09.2017)

WCL - COP

N/A

N/A

Y (20.09.2017)

WCL - WCA

N/A

N/A

Y (20.09.2017)

IBR - CTP

OAuth2: Authorization Code

N/A

Y (05.10.2017)

7. Overview on Workflow Description Formats and Execution Engines

This chapter will give an overview of different workflow description formats and execution engines that were used to comprise geospatial workflows.

7.1. Workflow Description Formats

In this section, two formats are identified that have been used to describe workflows in the geospatial domain, the Web Services Business Process Execution Language (WS-BPEL, BPEL in the following) and the Business Process Model and Notation (BPMN).

BPEL was released by the Organization for the Advancement of Structured Information Standards (OASIS). The current version is 2.0. It is used to orchestrate Web services, i.e. to specify executable processes and how messages are exchanged between them. It was designed to be used together with the Web Services Description Language (WSDL) 1.1 [2]. There is no standard graphical notation for BPEL, although there is a mapping from BPMN to BPEL 1.1. Former OGC Testbeds investigated the use of BPEL for the orchestration of OGC Web services. Also, a number of research papers were published dealing with this topic, see [2],[4],[5] and [10]. [8] has proposed a transactional profile for the Web Processing Service based on BPEL.

BPMN is a more recent standard for the graphical representation of business process models and also for describing the execution semantics. The use of BPMN for the orchestration of OGC Web services was investigated in recent studies (see [6], [7]). There is a growing number of engines that support BPMN, e.g. Orchestra [3], Activiti [4], Camunda [5] or jBPM[6]. BPMN was approved by the International Organization for Standardization (ISO) as ISO/IEC 19510:2013.

7.2. Workflow Execution Engines

In the following section, different workflow engines are described and approaches are shown that used them for orchestrating OGC Web services.

7.2.1. JBOSS jBPM

Open-source workflow engine released by the JBoss company. Its core is "a light-weight, extensible workflow engine written in pure Java" [7]. Several tools provide additional functionality like workflow modelling and monitoring. [6] used jBPM for their research.

Version

Programming language

BPMN 2.0 supported

License

7.3.0

Java

Yes

Apache License v2.0

7.2.2. Taverna

Open-source workflow management system started by the myGrid project [8] and currently an Apache Incubator project. Taverna is used in different scientific domains like bioinformatics or astronomy. Workflows can be created and executed using a desktop client/command line interface or in the web browser. [3] described the composition of geospatial workflows using WSDL and Taverna. [5] examined Taverna orchestration of geospatial web services.

Version

Programming language

BPMN 2.0 supported

License

3.1.0 (Command Line Tool)

Java

No (SCUFL2)

Apache License v2.0

7.2.3. Amazon Simple Workflow Service

The Amazon Simple Workflow Service (Amazon SWF) is used to coordinate work tasks that can run asynchronously in distributed systems, e.g. Amazon Elastic Compute Cloud. Geospatial workflows in Cloud environments, such as Amazon Web services, have been investigated by previous OGC Testbeds and in research (e.g. [1], [9]). However, past approaches were focused on the composition of workflows using Web services deployed in Cloud environments, not the use of Amazon SWF itself for the composition.

Version

Programming language

BPMN 2.0 supported

License

1.11.208 (Java SDK)

Java

No (Flow Framework)

Apache License v2.0

7.2.4. Camunda

Open-source platform for business process management. I.a., Camunda focuses on a more complete coverage of BPMN 2.0. Camunda offers a Java and REST API to manage workflows and a modeler to create them. Previous experiences with Camunda showed an easy to setup and robust engine.

Version

Programming language

BPMN 2.0 supported

License

7.7.0

Java

Yes

Apache License v2.0

7.2.5. Windows Workflow Foundation

Windows Workflow Foundation enables the implementation of processes in Microsoft .NET. Provides an API, a workflow engine and a designer. It is possible to access Web services with Windows Workflow Foundation. However, literature shows no evidence of orchestration of geospatial workflows using Windows Workflow Foundation.

Version

Programming language

BPMN 2.0 supported

License

4.5

.NET (C#, VB.NET)

No (XAML)

MIT

Because of its support of BPMN and the ease in setup and execution we have chosen to use Camunda in this experiment.

8. Testbed-13 Workflow Engine

The Testbed-13 Workflow Engine allows execution of workflows consisting of several WPS processes defined in a BPMN document.

8.1. Concept

The workflow engine provides two interfaces:

  • InsertProcess: This endpoint can be used to create new WPS processes for workflow compositions by posting a BPMN document to this endpoint using the HTTP POST operation.

  • WPS 2.0 interface for executing workflow: To execute the workflow, the engine will rely on the WPS 2.0 specification, i.e. for each workflow available, there will be a WPS process allowing to execute the workflow. Therefore, the execution engine supports the default operations of the WPS including:

    • GetCapabilities

    • DescribeProcess

    • Execute

    • GetStatus (for execution in asynchronous mode)

    • GetResult

8.2. Implementation

The implementation is based on the 52°North WPS framework (http://52north.org/wps) and Camunda. A transactional WPS interface (WPS-T) will be used to create new processes based upon BPMN documents.

The following image shows the components involved in the creation of a workflow WPS process:

Create Workflow Components
Figure 4. Components involved in workflow WPS process creation

The BPMN is sent to the WPS-T using the new InsertProcess operation. The current time in milliseconds is attached to the original process identifier. The BPMN is forwarded to the Camunda BPMN Engine that offers a REST interface. The engine checks the BPMN and returns a deployment identifier to the WPS-T. The WPS-T creates a process and returns the new process identifier to the client.

Below is an example of the InsertProcess request:

<?xml version="1.0" encoding="UTF-8"?>
<wps:InsertProcess xmlns:wps="http://www.opengis.net/wps/2.0"
        xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlink="http://www.w3.org/1999/xlink"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://tb12.dev.52north.org:80/workflow-wps/static/schemas/wpsInsertProcess.xsd"
        service="WPS" version="2.0.0">

        <wps:ProcessSpecification id="ConflationWorkflow">
                <wps:ProcessSpecificationAsValue
                        mimeType="application/x-bpmn">
            <bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camunda="http://camunda.org/schema/1.0/bpmn"  xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:ext="http://org.eclipse.bpmn2/ext" xmlns:xs="http://www.w3.org/2001/XMLSchema" id="Definitions_1" exporter="org.eclipse.bpmn2.modeler.core" exporterVersion="1.3.1.Final-v20161006-1425-B58" targetNamespace="http://org.eclipse.bpmn2/default/process">
            ....
            </bpmn2:definitions>
                </wps:ProcessSpecificationAsValue>
        </wps:ProcessSpecification>

</wps:InsertProcess>

(See Process specification for the full BPMN document.) The BPMN is extracted and wrapped in a multipart message:

--3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400
Content-Disposition: form-data; name="deployment-name"

ConflationWorkflow_1507125390489
--3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400
Content-Disposition: form-data; name="enable-duplicate-filtering"

true
--3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400
Content-Disposition: form-data; filename="ConflationWorkflow_1507125390489.bpmn"; name="data"

<bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camunda="http://camunda.org/schema/1.0/bpmn"  xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:ext="http://org.eclipse.bpmn2/ext" xmlns:xs="http://www.w3.org/2001/XMLSchema" id="Definitions_1" exporter="org.eclipse.bpmn2.modeler.core" exporterVersion="1.3.1.Final-v20161006-1425-B58" targetNamespace="http://org.eclipse.bpmn2/default/process">

....

</bpmn2:definitions>

--3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400--

This is sent to the REST-endpoint of the Camunda engine:

http://hostname.org/engine-rest/engine/default/deployment/create

The response contains the deployment-id and additional information:

{
    "links": [
        {
            "method": "GET",
            "href": "http://hostname.org/engine-rest/engine/default/deployment/45d60c1f-a99b-11e7-ac41-000c2988332c",
            "rel": "self"
        }
    ],
    "id": "45d60c1f-a99b-11e7-ac41-000c2988332c",
    "name": "ConflationWorkflow_1507125390489",
    "source": null,
    "deploymentTime": "2017/10/04T14:56:32",
    "tenantId": null
}

The new process will appear on the Camunda dashboard. Also, if available, the BPMN diagram can be visualized as shown in the following figure (note that some elements, e.g. inputs and outputs, are not visualized by Camunda):

camunda bpmn diagram
Figure 5. Workflow visualized in Camunda dashboard

After a successful upload to Camunda, the WPS sends a InsertProcessinfo response, as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<wps:InsertProcessInfo xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://tb12.dev.52north.org:80/workflow-wps/static/schemas/wpsInsertProcess.xsd">
    <wps:processID>testbed13.dsi.ConflationWorkflow_1507125390489</wps:processID>
</wps:InsertProcessInfo>

The process description of the newly created process is below:

<?xml version="1.0" encoding="UTF-8"?>
<wps:ProcessOfferings xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/2.0" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
  <wps:ProcessOffering processVersion="1.0.0" jobControlOptions="sync-execute async-execute" outputTransmission="value reference">
    <wps:Process>
      <ows:Title>Conflation workflow process 2017/10/04T14:56:30</ows:Title>
      <ows:Abstract>This process encapsulates a BPMN workflow that will perform data quality checks, transforms coordinates if needed and finally conflates datasets.</ows:Abstract>
      <ows:Identifier>testbed13.dsi.ConflationWorkflow_1507125390489</ows:Identifier>
      <wps:Input minOccurs="2" maxOccurs="4">
        <ows:Title>Datasets</ows:Title>
        <ows:Abstract>Vector datasets for conflation</ows:Abstract>
        <ows:Identifier>datasets</ows:Identifier>
        <wps:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
          <ns:Format default="true" mimeType="application/x-zipped-shp"/>
          <ns:Format mimeType="application/x-openstreetmap+xml"/>
          <ns:Format mimeType="application/x-zipped-shp" encoding="base64"/>
          <ns:Format mimeType="application/x-zipped-gdb"/>
          <ns:Format mimeType="application/x-zipped-gdb" encoding="base64"/>
        </wps:ComplexData>
      </wps:Input>
      <wps:Output>
        <ows:Title>Conflated dataset</ows:Title>
        <ows:Identifier>conflated-dataset</ows:Identifier>
        <wps:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
          <ns:Format default="true" mimeType="application/x-zipped-shp"/>
          <ns:Format mimeType="application/x-openstreetmap+xml"/>
          <ns:Format mimeType="application/x-zipped-shp" encoding="base64"/>
          <ns:Format mimeType="text/xml" schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd"/>
          <ns:Format mimeType="application/vnd.geo+json"/>
          <ns:Format mimeType="application/x-zipped-gdb"/>
          <ns:Format mimeType="application/x-zipped-gdb" encoding="base64"/>
        </wps:ComplexData>
      </wps:Output>
    </wps:Process>
  </wps:ProcessOffering>
</wps:ProcessOfferings>

After the WPS process was created, it can be executed by the client. The following image shows the involved components in executing a workflow WPS process:

Execute Workflow Components
Figure 6. Components involved in workflow WPS process execution

The client sends a standard execute-request to the WPS-T based on the process description, e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<wps:Execute xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlin="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd" service="WPS" version="2.0.0" response="document" mode="async">
  <ows:Identifier>testbed13.dsi.ConflationWorkflow_1507125390489</ows:Identifier>
  <wps:Input id="datasets">
    <wps:Reference mimeType="text/xml; subtype=gml/3.1.0" schema="http://schemas.opengis.net/gml/3.1.0/base/feature.xsd" xlin:href="https://tb12.dev.52north.org/security-proxy/service/wfs?service=WFS&amp;version=1.0.0&amp;request=GetFeature&amp;typeName=tb13:tnm-manhattan-streets-wgs84&amp;outputFormat=gml3"/>
  </wps:Input>
  <wps:Input id="datasets">
    <wps:Reference mimeType="text/xml; subtype=gml/3.1.0" schema="http://schemas.opengis.net/gml/3.1.0/base/feature.xsd" xlin:href="https://tb12.dev.52north.org/security-proxy/service/wfs?service=WFS&amp;version=1.0.0&amp;request=GetFeature&amp;typeName=tb13:lion-manhattan-streets&amp;outputFormat=gml3"/>
  </wps:Input>
  <wps:Input id="datasets">
    <wps:Reference xlin:href="https://tb12.dev.52north.org/aws-proxy/service/aws?service=aws&amp;url=https://s3-eu-west-1.amazonaws.com/testbed13-osm/manhattan/osm-manhattan-roads.osm" mimeType="application/x-openstreetmap+xml"/>
  </wps:Input>
  <wps:Output id="conflated-dataset" mimeType="text/xml" schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd" transmission="reference"/>
</wps:Execute>

The process identifier and inputs are extracted and forwarded to the Camunda REST-endpoint to start a new instance of the BPMN process.

http://hostname.org/engine-rest/process-definition/key/ConflationWorkflow_1507125390489/submit-form
{
  "variables": {
    "dataset1": {"value":"https://tb12.dev...","type":"String"},
    "dataset2":{"value":"https://tb12.dev...","type":"String"},
    "dataset3":{"value":"https://tb12.dev...","type":"String"}
  }
}

A WPS adapter deals with the (secured) communication with WPS internally in the Camunda BPMN Engine. After a successful execution of the BPMN process, the final output (link to resources) is sent to the WPS-T and from there forwarded to the client in a standard WPS ResultDocument.

<?xml version="1.0" encoding="UTF-8"?>
<wps:Result xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlin="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
    <wps:JobID>2206815e-4824-4b19-a790-68be5986b4c4</wps:JobID>
    <wps:Output id="conflated-dataset">
        <wps:Reference schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd" mimeType="text/xml" xlin:href="http://tb12.dev.52north.org/SecurityProxy/service/wps?request=GetOutput&amp;version=2.0.0&amp;service=WPS&amp;id=2206815e-4824-4b19-a790-68be5986b4c4conflated-dataset.cf30b672-7df2-405f-a5ee-8f6a5c558c88"/>
    </wps:Output>
</wps:Result>

The communication of the workflow engine with the different services is then logged. See Workflow engine log for an example.

8.3. Results

In the following, the final results of the workflow are shown and compared to the input datasets.

conflated result
Figure 7. The conflated result

First, this is set as a contrast to the different input datasets.

  • OpenStreetMap

osm result comparison
Figure 8. The conflated result compared to the OSM input dataset

  • The National Map

tnm result comparison
Figure 9. The conflated result compared to the TNM input dataset

  • LION (New York State data)

lion result comparison
Figure 10. The conflated result compared to the LION input dataset

From these complete views, one can see that there are slight differences between each of the input datasets and the conflated result. This means that the respective geometries were derived from the input datasets and not just blindly inserted. Next, it is worth looking at some details of the result.

  • Missing data in the result

LION result comparison
Figure 11. Missing data in the result

Parts of this segment are missing in the result dataset, although the segment is complete in the input datasets. The reason is presumably a conflict in the naming. Brooklyn Battery Tunnel vs Hugh L. Carey Tunnel. The different names are stored in either the name or alt_name attribute and only one of them is selected for the conflation.

  • Second lane added to two-lane street

The following figure shows the conflated result for Edgar Street, a street with separate one-way-lanes.