Publication Date: 2018-01-08
Approval Date: 2017-12-07
Posted Date: 2017-10-30
Reference number of this document: OGC 17-029r1
Reference URL for this document: http://www.opengis.net/doc/PER/t13-NG009
Category: Public Engineering Report
Editor: Benjamin Pross, Christoph Stasch
Title: OGC Testbed-13: Workflows ER
COPYRIGHT
Copyright © 2018 Open Geospatial Consortium. To obtain additional rights of use, visit http://www.opengeospatial.org/
WARNING
This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.
LICENSE AGREEMENT
Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.
If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.
THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.
This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.
Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.
This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.
None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.
- 1. Summary
- 2. References
- 3. Terms and definitions
- 4. Abbreviated terms
- 5. Overview
- 6. Scenario
- 7. Overview on Workflow Description Formats and Execution Engines
- 8. Testbed-13 Workflow Engine
- 9. Processing Services
- 10. Data Services
- 11. Workflow Client and Catalog
- 12. Workflow Security
- 13. Recommendations
- Appendix A: Listings
- Appendix B: XML Schema Documents
- Appendix C: TIE results
- Appendix D: Revision History
- Appendix E: Bibliography
1. Summary
This Engineering Report (ER) addresses the development of a consistent, flexible, adaptable workflow that will run behind the scenes. A user should be able to discover existing workflows via a catalog and execute them using their own datasets. An expert should be able to create workflows and to publish them. Previous OGC Testbed initiatives investigated workflows in the geospatial domain:
-
OWS 3 Imagery Workflow Experiments
-
OWS 4 WPS IPR Workflow descriptions and lessons learned
-
OWS 4 Topology Quality Assessment Interoperability Program Report
-
OWS 5 Data View Architecture Engineering Report
-
OWS 6 Geoprocessing Workflow Architecture Engineering Report
These initiatives mostly favored Business Processing Execution Language (BPEL) as the workflow execution language. More recent studies ([6], [7]) were performed using BPMN as a means for describing and executing workflows comprised of OGC Web services. This ER will give an overview about existing approaches to compose and execute geospatial workflows and will describe the approach taken in Testbed-13, taking into account security aspects.
Parts of this work have been funded by the COLABIS and Mudak-WRM projects funded by the German Federal Ministry of Education and Research under grant agreement numbers 03G0852C and 02WGR1431C.
1.1. Requirements
As stated above, the workflow is required to perform the following tasks:
-
Gather data
-
Check ellipsoid/projection
-
Check data quality
-
Run conflation
-
Deliver data
A planned workflow (i.e. a workflow constructed ahead of time) shall be developed combining the conflation Web Processing Service (WPS) 2.0 and data quality WPS 2.0 implemented in Testbed-12. In order to secure the workflow, the following items need to be taken into account:
-
OGC Services and content encodings
-
Fine grained access control implemented through an Attribute Based Access Control (ABAC) infrastructure
-
Services and content are provided by more than one organization utilizing more than one security environment
-
Services are not accredited to the same level of trust
-
Some content may require a higher level of protection than some services are accredited to provide
-
All actions must be logged and associated with the user who initiated the service chain
-
Some of the content is sensitive (Personal Identifiable Information). Unauthorized release of this data must be avoided.
1.2. Key Findings and Prior-After Comparison
The workflow package in Testbed-13 has the goal of developing a consistent, flexible, adaptable workflow that will run behind the scenes, is described in a standardized format (BPMN) and could be shared to and executed by other users. In a nutshell, the workflow aims to automate conflation of road datasets and pre-processing steps (coordinate transformation, data quality checks). The processing steps are implemented as WPS processes and the data is passed by reference to data servers that include Web Feature Services (WFS) and Web Coverage Services (WCS) between the different processing steps.
The work and experiments that were done showed that it is possible to use OGC Web services in workflows that are protected by different security mechanisms, i.e. X.509 certificates and OAuth. Concepts of combining mainstream security with OGC Web services that have been developed during past testbeds and that are currently discussed in OGC have been proven to work in geospatial workflows. It has been shown that workflows can be created, wrapped in WPS processed and made discoverable in catalog services using current standards and technologies.
1.3. What does this ER mean for the Working Group and OGC in general
Workflows that transform geospatial datasets to derived geospatial information are usually composed of several processing steps. These steps may be provided by different WPS endpoints implementing different process profiles. Up to now, a common approach for defining, sharing and executing complex workflows (e.g. those composed of processes from different service providers) has been non-existent. This ER describes the approaches taken in OGC Testbed-13 to address these issues. Besides the general question of how to compose, share, and execute a workflow consisting of several WPS processes, different approaches for dealing with security issues such as identity mediation or delegation are also described in this ER.
Several approaches regarding workflow composition have been discussed in the Workflow DWG, [6] being the most recent document. However, no common approach has been formalized yet. This ER describes a further proof of concept of the approach described by [6], which could lead to the creation of a best practice document regarding geospatial workflow composition and execution.
1.4. Document contributor contact points
All questions regarding this document should be directed to the editor or the contributors:
Name | Organization |
---|---|
Benjamin Proß (editor) |
52°North GmbH |
Christoph Stasch (editor) |
52°North GmbH |
Dimitar Misev |
Rasdaman |
Mark Lawrence |
Compusult |
1.5. Future Work
This ER shows that BPMN can be used to compose, discover and execute secured workflows in the geospatial domain. In the future, a general approach for BPMN engines to communicate with OGC Web services should be investigated, including the treatment of inputs and outputs. Also, the encoding of security aspects in BPMN should be defined. A transactional extension for WPS 2.0 and the extension of the WPS 2.0 StatusInfo document to add more information about the process status should be discussed in the WPS 2.0 SWG.
1.6. Foreword
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.
2. References
The following normative documents are referenced in this document.
3. Terms and definitions
For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard OGC 06-121r9 shall apply.
4. Abbreviated terms
-
API Application Program Interface
-
AWS Amazon Web Services
-
BPEL Business Process Execution Language (also WS-BPEL)
-
BPMN Business Process Model and Notation
-
CSW Catalog Service for the Web
-
CRS Coordinate Reference System
-
ER Engineering Report
-
OGC Open Geospatial Consortium
-
OWS OGC Web Service
-
WPS Web Processing Service
-
WFS Web Feature Service
-
WCS Web Coverage Service
5. Overview
This Engineering Report describes the results of the OGC Testbed-13 Workflow thread.
In the following sections, the scenario is described, followed by an overview on workflow description formats and execution engines. The developed/used components are described in sections 8 - 11. Section 12 describes the security aspects and section 13 summarizes the recommendations.
6. Scenario
The overall goal is to create a conflation workflow consisting of several (pre-)processing steps needed to conflate road datasets of different quality. Conflation, also known as map matching or merging is the process of combining two datasets based on a set of rules to produce a more complete dataset. One example is the combination of authoritative data, e.g. authoritative road data, with crowd-sourced data, e.g. OpenStreetMap (OSM). The authoritative data is collected using certain guidelines and must follow strict requirements to ensure data quality. This leads to a smaller frequency of updates. Crowd-sourced data on the other hand, can be collected by virtually everyone. In this case, the quality of the data is monitored by the community and usually has a high volume of updates and therefore much more prone to error. As a result, the data is usually of lower quality.
The general conflation workflow consists of three major steps as shown in the figure below.
The first step is to check whether the road datasets are in the same coordinate reference system. If not, a coordinate transformation will be executed (WPS-1). The next step is to check the data quality of the road datasets (WPS-2). Thereby, the mean minimum distance between the road datasets and a set of control points is checked. The results are returned as ISO data quality. The last step in the workflow is the actual conflation (WPS-3) that is an extended version of the Conflation WPS developed in Testbed-12 that uses the Hootenanny conflation software[1] at the backend.
The workflow client can be used for two purposes:
-
Workflow composition: An expert user defines the workflow by discovering suitable WPS processes in a catalog service and composing them in a BPMN document.
-
Workflow execution: A workflow user can use the client to execute a workflow process.
The following two sequence diagrams illustrate the general sequences for both use cases.
6.1. Workflow Creation
Step 1:
-
The expert queries a catalog for processes suitable for the workflow.
Step 2:
-
The expert composes the workflow using the workflow client (BPMN editor).
Step 3:
-
The expert then selects the option in the client to send the workflow defined in the BPMN file to the workflow engine.
Step 4:
-
In order to process such a request, the workflow engine requires a security access code. The workflow client (WPS client) sends a request to the OAuth Security Service to get an access code and then includes it as part of the request.
-
The workflow engine creates a new process identifier and uploads the BPMN to the underlying BPMN engine. The BPMN engine returns an internal process identifier that can be used to start the process. A wrapper process is created in the workflow engine WPS. The WPS then returns the process identifier of the wrapper process.
Step 5:
-
The workflow client then contacts the CSW and instructs it to harvest the new process in the workflow Engine WPS.
6.2. Workflow Execution
Note: Only the security aspects of the workflow client and -engine interaction are shown here. For details about the security aspects of the workflow in general, please refer to Workflow Security
Step 1:
-
The user launches the workflow client and searches the catalog for workflows (i.e. WPS process descriptions) using keywords and data types that match their current needs.
Step 2:
-
The user finds the applicable conflation WPS process and provides the list of datasets and other input parameters for processing.
Step 3:
-
In order to process an WPS Execute request, the workflow engine WPS requires a security access code. The workflow client (WPS client) sends a request to the OAuth Security Service to get an access code.
Step 4:
-
The WPS client then constructs the applicable WPS Execute request (including the security access code) and sends it to the workflow engine WPS.
Step 5:
-
The workflow engine now invokes the first WPS by passing the references to the road datasets and a target CRS to the WPS. If the CRS is different from the target CRS, the coordinates are transformed.
Step 6:
-
The data quality is checked using the Data Quality WPS. A coverage is requested from a WCS and quality checks are performed.
Step 7:
-
If the quality is sufficient the actual conflation is executed by invoking another WPS process. The WPS then inserts the resulting dataset in a WFS-T (or WCS-T, depending on the result type) and returns a reference to it. The reference is then forwarded to the client and may be visualized in a map.
6.3. Datasets
The workflow is run for conflating road datasets in two regions: New York City in the United States of America and Za’atari in Jordan. The datasets available are described in the following two subsections.
New York City:
-
The National Map
-
OSM
-
Digital Globe satellite images
Za’atari Datasets:
-
UN
-
OSM
-
Digital Globe satellite images
6.4. Technology Integration Experiments
The following tables list the Technology Integration Experiments (TIEs) that were performed in the workflow scenario. See TIE results for additional information about the TIEs.
Involved components:
Abbreviation |
Component name |
Deliverable |
WCL |
Workflow Client |
NG136 |
WCA |
Workflow Catalog |
NG135 |
WEN |
Workflow Engine |
NG131 |
CTP |
Coordinate Transformation Process |
NG130 |
DQP |
Data Quality Check Process |
NG130 |
COP |
Conflation Process |
NG130 |
WFS |
Workflow WFS |
NG132 |
ABU |
Amazon Bucket |
N/A |
IBR |
Internet Browser |
N/A |
TIEs:
TIE between |
Security |
Use Case |
Successful |
WEN - CTP |
x.509, OAuth2: Client Credentials |
Dominating Privileges |
Y (04.10.2017) |
WEN - DQP |
x.509, OAuth2: Client Credentials |
Tunneling Proxies |
Y (04.10.2017) |
WEN - COP |
x.509, OAuth2: Client Credentials |
N/A |
Y (04.10.2017) |
CTP - WFS |
x.509, OAuth2: Client Credentials |
Dominating Privileges |
Y (04.10.2017) |
DQP - WFS |
x.509, OAuth2: Client Credentials |
N/A |
Y (04.10.2017) |
COP - WFS |
x.509, OAuth2: Client Credentials |
N/A |
Y (04.10.2017) |
COP - ABU |
x.509, IAM |
Identity Mediation |
Y (04.10.2017) |
WCL - WEN InsertProcess |
OAuth2: Client Credentials |
N/A |
Y (20.09.2017) |
WCL - WEN Execute |
OAuth2: Client Credentials |
N/A |
Y (04.10.2017) |
WCL - CTP |
N/A |
N/A |
Y (20.09.2017) |
WCL - DQP |
N/A |
N/A |
Y (20.09.2017) |
WCL - COP |
N/A |
N/A |
Y (20.09.2017) |
WCL - WCA |
N/A |
N/A |
Y (20.09.2017) |
IBR - CTP |
OAuth2: Authorization Code |
N/A |
Y (05.10.2017) |
7. Overview on Workflow Description Formats and Execution Engines
This chapter will give an overview of different workflow description formats and execution engines that were used to comprise geospatial workflows.
7.1. Workflow Description Formats
In this section, two formats are identified that have been used to describe workflows in the geospatial domain, the Web Services Business Process Execution Language (WS-BPEL, BPEL in the following) and the Business Process Model and Notation (BPMN).
BPEL was released by the Organization for the Advancement of Structured Information Standards (OASIS). The current version is 2.0. It is used to orchestrate Web services, i.e. to specify executable processes and how messages are exchanged between them. It was designed to be used together with the Web Services Description Language (WSDL) 1.1 [2]. There is no standard graphical notation for BPEL, although there is a mapping from BPMN to BPEL 1.1. Former OGC Testbeds investigated the use of BPEL for the orchestration of OGC Web services. Also, a number of research papers were published dealing with this topic, see [2],[4],[5] and [10]. [8] has proposed a transactional profile for the Web Processing Service based on BPEL.
BPMN is a more recent standard for the graphical representation of business process models and also for describing the execution semantics. The use of BPMN for the orchestration of OGC Web services was investigated in recent studies (see [6], [7]). There is a growing number of engines that support BPMN, e.g. Orchestra [3], Activiti [4], Camunda [5] or jBPM[6]. BPMN was approved by the International Organization for Standardization (ISO) as ISO/IEC 19510:2013.
7.2. Workflow Execution Engines
In the following section, different workflow engines are described and approaches are shown that used them for orchestrating OGC Web services.
7.2.1. JBOSS jBPM
Open-source workflow engine released by the JBoss company. Its core is "a light-weight, extensible workflow engine written in pure Java" [7]. Several tools provide additional functionality like workflow modelling and monitoring. [6] used jBPM for their research.
Version |
Programming language |
BPMN 2.0 supported |
License |
7.3.0 |
Java |
Yes |
Apache License v2.0 |
7.2.2. Taverna
Open-source workflow management system started by the myGrid project [8] and currently an Apache Incubator project. Taverna is used in different scientific domains like bioinformatics or astronomy. Workflows can be created and executed using a desktop client/command line interface or in the web browser. [3] described the composition of geospatial workflows using WSDL and Taverna. [5] examined Taverna orchestration of geospatial web services.
Version |
Programming language |
BPMN 2.0 supported |
License |
3.1.0 (Command Line Tool) |
Java |
No (SCUFL2) |
Apache License v2.0 |
7.2.3. Amazon Simple Workflow Service
The Amazon Simple Workflow Service (Amazon SWF) is used to coordinate work tasks that can run asynchronously in distributed systems, e.g. Amazon Elastic Compute Cloud. Geospatial workflows in Cloud environments, such as Amazon Web services, have been investigated by previous OGC Testbeds and in research (e.g. [1], [9]). However, past approaches were focused on the composition of workflows using Web services deployed in Cloud environments, not the use of Amazon SWF itself for the composition.
Version |
Programming language |
BPMN 2.0 supported |
License |
1.11.208 (Java SDK) |
Java |
No (Flow Framework) |
Apache License v2.0 |
7.2.4. Camunda
Open-source platform for business process management. I.a., Camunda focuses on a more complete coverage of BPMN 2.0. Camunda offers a Java and REST API to manage workflows and a modeler to create them. Previous experiences with Camunda showed an easy to setup and robust engine.
Version |
Programming language |
BPMN 2.0 supported |
License |
7.7.0 |
Java |
Yes |
Apache License v2.0 |
7.2.5. Windows Workflow Foundation
Windows Workflow Foundation enables the implementation of processes in Microsoft .NET. Provides an API, a workflow engine and a designer. It is possible to access Web services with Windows Workflow Foundation. However, literature shows no evidence of orchestration of geospatial workflows using Windows Workflow Foundation.
Version |
Programming language |
BPMN 2.0 supported |
License |
4.5 |
.NET (C#, VB.NET) |
No (XAML) |
MIT |
Because of its support of BPMN and the ease in setup and execution we have chosen to use Camunda in this experiment.
8. Testbed-13 Workflow Engine
The Testbed-13 Workflow Engine allows execution of workflows consisting of several WPS processes defined in a BPMN document.
8.1. Concept
The workflow engine provides two interfaces:
-
InsertProcess: This endpoint can be used to create new WPS processes for workflow compositions by posting a BPMN document to this endpoint using the HTTP POST operation.
-
WPS 2.0 interface for executing workflow: To execute the workflow, the engine will rely on the WPS 2.0 specification, i.e. for each workflow available, there will be a WPS process allowing to execute the workflow. Therefore, the execution engine supports the default operations of the WPS including:
-
GetCapabilities
-
DescribeProcess
-
Execute
-
GetStatus (for execution in asynchronous mode)
-
GetResult
-
8.2. Implementation
The implementation is based on the 52°North WPS framework (http://52north.org/wps) and Camunda. A transactional WPS interface (WPS-T) will be used to create new processes based upon BPMN documents.
The following image shows the components involved in the creation of a workflow WPS process:
The BPMN is sent to the WPS-T using the new InsertProcess operation. The current time in milliseconds is attached to the original process identifier. The BPMN is forwarded to the Camunda BPMN Engine that offers a REST interface. The engine checks the BPMN and returns a deployment identifier to the WPS-T. The WPS-T creates a process and returns the new process identifier to the client.
Below is an example of the InsertProcess request:
<?xml version="1.0" encoding="UTF-8"?>
<wps:InsertProcess xmlns:wps="http://www.opengis.net/wps/2.0"
xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://tb12.dev.52north.org:80/workflow-wps/static/schemas/wpsInsertProcess.xsd"
service="WPS" version="2.0.0">
<wps:ProcessSpecification id="ConflationWorkflow">
<wps:ProcessSpecificationAsValue
mimeType="application/x-bpmn">
<bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:ext="http://org.eclipse.bpmn2/ext" xmlns:xs="http://www.w3.org/2001/XMLSchema" id="Definitions_1" exporter="org.eclipse.bpmn2.modeler.core" exporterVersion="1.3.1.Final-v20161006-1425-B58" targetNamespace="http://org.eclipse.bpmn2/default/process">
....
</bpmn2:definitions>
</wps:ProcessSpecificationAsValue>
</wps:ProcessSpecification>
</wps:InsertProcess>
(See Process specification for the full BPMN document.) The BPMN is extracted and wrapped in a multipart message:
--3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400 Content-Disposition: form-data; name="deployment-name" ConflationWorkflow_1507125390489 --3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400 Content-Disposition: form-data; name="enable-duplicate-filtering" true --3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400 Content-Disposition: form-data; filename="ConflationWorkflow_1507125390489.bpmn"; name="data" <bpmn2:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" xmlns:bpmn2="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:ext="http://org.eclipse.bpmn2/ext" xmlns:xs="http://www.w3.org/2001/XMLSchema" id="Definitions_1" exporter="org.eclipse.bpmn2.modeler.core" exporterVersion="1.3.1.Final-v20161006-1425-B58" targetNamespace="http://org.eclipse.bpmn2/default/process"> .... </bpmn2:definitions> --3b30bbd9-ae0e-4e3f-a6cd-fb57db0fe400--
This is sent to the REST-endpoint of the Camunda engine:
http://hostname.org/engine-rest/engine/default/deployment/create
The response contains the deployment-id and additional information:
{
"links": [
{
"method": "GET",
"href": "http://hostname.org/engine-rest/engine/default/deployment/45d60c1f-a99b-11e7-ac41-000c2988332c",
"rel": "self"
}
],
"id": "45d60c1f-a99b-11e7-ac41-000c2988332c",
"name": "ConflationWorkflow_1507125390489",
"source": null,
"deploymentTime": "2017/10/04T14:56:32",
"tenantId": null
}
The new process will appear on the Camunda dashboard. Also, if available, the BPMN diagram can be visualized as shown in the following figure (note that some elements, e.g. inputs and outputs, are not visualized by Camunda):
After a successful upload to Camunda, the WPS sends a InsertProcessinfo response, as shown below:
<?xml version="1.0" encoding="UTF-8"?>
<wps:InsertProcessInfo xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://tb12.dev.52north.org:80/workflow-wps/static/schemas/wpsInsertProcess.xsd">
<wps:processID>testbed13.dsi.ConflationWorkflow_1507125390489</wps:processID>
</wps:InsertProcessInfo>
The process description of the newly created process is below:
<?xml version="1.0" encoding="UTF-8"?>
<wps:ProcessOfferings xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/2.0" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
<wps:ProcessOffering processVersion="1.0.0" jobControlOptions="sync-execute async-execute" outputTransmission="value reference">
<wps:Process>
<ows:Title>Conflation workflow process 2017/10/04T14:56:30</ows:Title>
<ows:Abstract>This process encapsulates a BPMN workflow that will perform data quality checks, transforms coordinates if needed and finally conflates datasets.</ows:Abstract>
<ows:Identifier>testbed13.dsi.ConflationWorkflow_1507125390489</ows:Identifier>
<wps:Input minOccurs="2" maxOccurs="4">
<ows:Title>Datasets</ows:Title>
<ows:Abstract>Vector datasets for conflation</ows:Abstract>
<ows:Identifier>datasets</ows:Identifier>
<wps:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
<ns:Format default="true" mimeType="application/x-zipped-shp"/>
<ns:Format mimeType="application/x-openstreetmap+xml"/>
<ns:Format mimeType="application/x-zipped-shp" encoding="base64"/>
<ns:Format mimeType="application/x-zipped-gdb"/>
<ns:Format mimeType="application/x-zipped-gdb" encoding="base64"/>
</wps:ComplexData>
</wps:Input>
<wps:Output>
<ows:Title>Conflated dataset</ows:Title>
<ows:Identifier>conflated-dataset</ows:Identifier>
<wps:ComplexData xmlns:ns="http://www.opengis.net/wps/2.0">
<ns:Format default="true" mimeType="application/x-zipped-shp"/>
<ns:Format mimeType="application/x-openstreetmap+xml"/>
<ns:Format mimeType="application/x-zipped-shp" encoding="base64"/>
<ns:Format mimeType="text/xml" schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd"/>
<ns:Format mimeType="application/vnd.geo+json"/>
<ns:Format mimeType="application/x-zipped-gdb"/>
<ns:Format mimeType="application/x-zipped-gdb" encoding="base64"/>
</wps:ComplexData>
</wps:Output>
</wps:Process>
</wps:ProcessOffering>
</wps:ProcessOfferings>
After the WPS process was created, it can be executed by the client. The following image shows the involved components in executing a workflow WPS process:
The client sends a standard execute-request to the WPS-T based on the process description, e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<wps:Execute xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xlin="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd" service="WPS" version="2.0.0" response="document" mode="async">
<ows:Identifier>testbed13.dsi.ConflationWorkflow_1507125390489</ows:Identifier>
<wps:Input id="datasets">
<wps:Reference mimeType="text/xml; subtype=gml/3.1.0" schema="http://schemas.opengis.net/gml/3.1.0/base/feature.xsd" xlin:href="https://tb12.dev.52north.org/security-proxy/service/wfs?service=WFS&version=1.0.0&request=GetFeature&typeName=tb13:tnm-manhattan-streets-wgs84&outputFormat=gml3"/>
</wps:Input>
<wps:Input id="datasets">
<wps:Reference mimeType="text/xml; subtype=gml/3.1.0" schema="http://schemas.opengis.net/gml/3.1.0/base/feature.xsd" xlin:href="https://tb12.dev.52north.org/security-proxy/service/wfs?service=WFS&version=1.0.0&request=GetFeature&typeName=tb13:lion-manhattan-streets&outputFormat=gml3"/>
</wps:Input>
<wps:Input id="datasets">
<wps:Reference xlin:href="https://tb12.dev.52north.org/aws-proxy/service/aws?service=aws&url=https://s3-eu-west-1.amazonaws.com/testbed13-osm/manhattan/osm-manhattan-roads.osm" mimeType="application/x-openstreetmap+xml"/>
</wps:Input>
<wps:Output id="conflated-dataset" mimeType="text/xml" schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd" transmission="reference"/>
</wps:Execute>
The process identifier and inputs are extracted and forwarded to the Camunda REST-endpoint to start a new instance of the BPMN process.
http://hostname.org/engine-rest/process-definition/key/ConflationWorkflow_1507125390489/submit-form
{
"variables": {
"dataset1": {"value":"https://tb12.dev...","type":"String"},
"dataset2":{"value":"https://tb12.dev...","type":"String"},
"dataset3":{"value":"https://tb12.dev...","type":"String"}
}
}
A WPS adapter deals with the (secured) communication with WPS internally in the Camunda BPMN Engine. After a successful execution of the BPMN process, the final output (link to resources) is sent to the WPS-T and from there forwarded to the client in a standard WPS ResultDocument.
<?xml version="1.0" encoding="UTF-8"?>
<wps:Result xmlns:wps="http://www.opengis.net/wps/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlin="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/2.0 http://schemas.opengis.net/wps/2.0/wps.xsd">
<wps:JobID>2206815e-4824-4b19-a790-68be5986b4c4</wps:JobID>
<wps:Output id="conflated-dataset">
<wps:Reference schema="http://schemas.opengis.net/gml/3.1.1/base/feature.xsd" mimeType="text/xml" xlin:href="http://tb12.dev.52north.org/SecurityProxy/service/wps?request=GetOutput&version=2.0.0&service=WPS&id=2206815e-4824-4b19-a790-68be5986b4c4conflated-dataset.cf30b672-7df2-405f-a5ee-8f6a5c558c88"/>
</wps:Output>
</wps:Result>
The communication of the workflow engine with the different services is then logged. See Workflow engine log for an example.
8.3. Results
In the following, the final results of the workflow are shown and compared to the input datasets.
First, this is set as a contrast to the different input datasets.
-
OpenStreetMap
-
The National Map
-
LION (New York State data)
From these complete views, one can see that there are slight differences between each of the input datasets and the conflated result. This means that the respective geometries were derived from the input datasets and not just blindly inserted. Next, it is worth looking at some details of the result.
-
Missing data in the result
Parts of this segment are missing in the result dataset, although the segment is complete in the input datasets. The reason is presumably a conflict in the naming. Brooklyn Battery Tunnel vs Hugh L. Carey Tunnel. The different names are stored in either the name or alt_name attribute and only one of them is selected for the conflation.
-
Second lane added to two-lane street
The following figure shows the conflated result for Edgar Street, a street with separate one-way-lanes.