OGC Testbed-14: BPMN Workflow Engineering Report

Security continues to play an important role in the development of OGC standards, practice and doctrine. Implementation of security is also a requirement for all work done in the Next Generation Services thread in Testbed-14. Therefore, encoding security into a workflow request is a requirement for this work.

The three motivating use cases outlined in the Testbed-13 Security ER are:

This section briefly discusses the considerations from the ER that are relevant to the work presented here.

The Workflows ER provided an account of the work done in Testbed-13 workflows and included information on workflow construction and execution, an overview of workflow engines, transactional web processing services for accepting BPMN workflow documents, data quality services, conflation services and clients for constructing an executing the workflows. The scenario from Testbed-13 is shown in Figure 2 where the workflow client aspect stored existing workflows in a Catalog Service for the Web (CSW) and submitted the workflow to the workflow engine for execution. The workflow engine then orchestrated and executed the three WPS services to produce a result. The Web Feature Service (WFS) is utilized as a data repository within the workflow, but not formally by the workflow engine as the data is passed to the first WPS in the chain through configuration of WPS parameters.

When a user constructs a workflow for execution they currently do not know if the workflow will run or fail. This problem is exacerbated when there are workflow processes with dependencies on inputs and outputs. There are in general two types of processes that should be considered here:

If a workflow were constructed where process 1 is followed by process 2, this could potentially produce a resultant data set than if process 1 followed process 2. A likely occurrence is that if the filtering or selection criteria are two strict then the first process in the chain would produce a null result. This would then result in an error message and workflow failure.

Managing the ordering and production of results for processes is an on-going issue that is likely to remain unsolved in this Testbed as it is out of scope. However, solutions should be considered to mitigate, be it through sampling, some sort of semantic validity of workflows and/or suitable error messaging to make the user aware of process failures and a reason for the failure. Construction of some OGC specific error messaging should be considered in future Testbeds.

As mentioned previously, the data modeling aspects of this ER are key to the motivating requirements and the best practices endeavor. BPMN has the concept of data objects and data stores. The former is for modeling data in flux and the latter is for modeling data at rest. OGC services and in particular; WPS take data as parameters either by reference or as a raw type. When orchestrating OGC services, ideally this should be de-coupled and data objects passed by reference as a BPMN data object or store; rather than as an input parameter to a WPS process. The data object as a reference could then be mapped to a WPS process parameter therefore making the data object reusable throughout the workflow. Additionally, this method of mapping data objects enables users to graphically model data flows as well as process flows within the BPMN workflow diagram. Generally, the BPMN constructs for data modeling are suited to internal orchestration of data and references to data, rather than providing methods to access external data. BPMN also does not have a formal method for querying internal process variables post process completion, i.e. workflows are self-contained, therefore, getting data in and out of workflow is defined by the services that form part of the orchestration effort.

The implementation of demonstrators supporting the workflow engine work items have had to implement a helper class. In this implementation, the helper class is simplified to act as a WPS client, thus offloading the capabilities required for executing OGC services to resources that already have it.

The issue with geospatial data structures is experienced when workflows are executed synchronously, as the data are retrieved from an executed process and then passed to the next process. This could potentially be mitigated through passing of links in an asynchronous fashion. However, this would restrict the functionality of the system as it would make demands upon it that are not OGC compliant. The data inputs and outputs point relates back to the concept of data objects, which has been covered in previous sections. This helper class by default passes the results as a reference from one process to another, thus removing the requirement for the workflow engine to constantly be passing data back and forth from the engine to the next service in the chain. This approach does have a drawback in that every service in the chain is acting as both a client and a server. However, this is preferable to putting data load onto the workflow engine as it could easily form a bottleneck for processing and data.

A consideration for future work in this area is to have a system that tests for data compatibility within the workflow to make a decision on the approach to take, that is, passing by reference or passing by value. This would ensure full data compatibility throughout the workflow while making the most efficient use of computational resources.

BPMN describes Service Tasks as those done by software. Essentially the user configures a service task with parameters required by the external service which is then executed as part of the workflow on behalf of the user. A key aspect of a service task is that it is synchronous, therefore the workflow will wait for a result before continuing. This can of course cause problems if a web service times out or is non-responsive. This can be mitigated using an Event which is described in a following section.

From an OGC perspective, Service tasks are best used for orchestrating OGC web services that are synchronous, i.e. the workflow engine configures a process, executes the process and then passes the result onto the next process. This requires the workflow engine to understand spatial data structures and concepts, this is expressed in the helper class for the software package being used.

Service tasks should be used for processing rather than internal data management as these are best described in data objects and data stores which are discussed later and used for internal workflow data management. Therefore, OGC services such as WPS and Web Coverage Processing Service (WCPS) are best represented using Service tasks. Note that service tasks are different from Script tasks that execute a simple script upon execution. Script task could feasibly be used to interact with OGC services. However, the script task is designed as more of a catch-all to execute a script within the workflow rather than to interact with well-defined services.

Send tasks are used to send messages between processes and swim lanes. These are generally used to initiate another part of the workflow via sending a message to do so. This type of task does not have the ability to await a response from the service task and is completed as soon as the message is sent. A typical use case for a send task is to initiate a branch of the workflow that is separate from the primary swim lane. As with the data modeling aspects of BPMN, send tasks refer to internal workflow orchestration rather than messaging external services that are orchestrated using the service task.

Receive tasks are the Send task counter-part and will await a message sent (usually from a send task) before initiating. This task is usually used at the beginning of a swim lane to initiate tasks after instantiation. The task is considered complete as soon as the message is received. As with send tasks, these are designed for internal orchestration of workflows.

BPMN has the ability to model multiple instances of a task in a workflow as well as using loops for iterative execution of a task. These can be used as normal in OGC services, as the two examples are simply executing instances of a process with a given pattern.

Message start events are those that initiate a workflow based upon some external event. A typical example of this is that the workflow is called by an external process. This is in contrast to a standard start event where the workflow is called by a client interacting with the API interface of the workflow engine. Additionally, the Message event start does not discern between users or calling systems, it simply executes having received the message. In contrast to this is the signal event, where the workflow is awaiting a specific signal to initiate. This is likely to be a specific client or execution pattern rather than the general one that messaging is suited to. Signal events also have more flexibility for communicating aspects other than simple error messages, as is expected with the Message event. Likewise, an end event will act in the same way by throwing either a signal or message to a chosen service to end the workflow. Note that an end event is different than the email event, which is specifically setup to email results to a configured account.

Intermediate events for messages and signals are triggered in much the same way as the start and end events, however they are triggered before the workflow has completed.

Signaling functions are available from outside the workflow during execution and are often used to hold the workflow until an external task has been completed. This has crossovers with the Human Task aspect of BPMN, which relies on an external actor to provide information to allow the workflow to continue.

Error events capture and report on errors based upon a condition. For example, a workflow error might be that a process has failed for some reason, be it a coding error or unexpected behavior experienced due to the input parameters. Error messaging is a catch-all in terms of BPMN, but should be used to communicate the errors from processes or data objects. Typical uses of an Error event include; mis-configuration of processes, reporting a lack of authentication for services and data type conflicts. Error catching should be used to report back to the workflow executor in human readable language. An example of where this is useful is to watch for a null result from an external process and report back accordingly, rather than simply throwing a language specific error.

It is difficult to predict how workflows will behave with different sets of input data and parametrization of processes. For example, a workflow that executes for one set of data may fail for another set. This could be due to an Error in the typical sense (i.e. there is something wrong with one of the processes) or it could be that there is no error and the processes executed correctly but it failed for another, data dependent reason.

Considering a workflow that has two dependent processes expressed as Service tasks, with the second process configured to use the outputs from the first process. If the first process produces an output that contains zero or a null result, then it is likely that the second process would fail. Although the workflow may fail at this point, this should not be considered an error as everything in the workflow acted exactly as it should. A compensation event tells the workflow how to behave in this eventuality; it might execute a message to the user informing them of the reason for the event being triggered, alternatively it may provide an alternative execution path for results that conform to the compensation requirements.

Orchestrating services that include compensation is prudent when the workflow is complex with many branches. It may also be considered an extension of the error event instead of reporting an error, it activates an alternative path to compensate for the error.

Data objects describe data requirements for activities to be performed. The data object element can represent a singular object or a collection of objects. The life-cycle of the data object is implicitly tied to the containing process or sub-process. In fact, a data object in its raw form cannot exist without a containing process or sub-process. Data object references are related to data objects in that they enable reference to the same data object across a workflow. The advantage with data object references is that they are state-aware across the diagram where as data objects do not have a state and following the life cycle of the containing process. Data objects also have a class like hierarchy of access, much like variable scope. A process containing sub-processes may all have data objects and references. However, their accessibility is dependent on their location in the process/sub-process hierarchy. Figure 5 and 6 show the BPMN representation for a data object and data object collection.

A data store is used when data will persist beyond the life cycle of the containing process. Data stores also have the concept of a reference. However, the reference to the data store is simply a convenient way to access the data store without replicating the icon as the reference, for all intents and purposes is the same as the store. It is not appropriate to represent OGC services as data stores, as a data store refers to a traditional database. Data from OGC services is requested via a query to a standardized interface, which may refer to a set of databases or indeed, other services behind the scenes. It is more likely that access to data should be treated like any other service task in BPMN as it is essentially performing the same operation.

A data store is used for a persistent database, which is rarely the case in OGC services. Instead data are posted and accessed through services as the data architecture behind the services is opaque. Therefore, a data store should not be used to invoke an OGC service, as they are completely different concepts but maybe used to represent a local or remote traditional database storage.

The data object is used to graphically represent the data inputs and outputs from a single Task. The BPMN specification describes usage of the data object be restricted to a single Task and as such, the data object is destroyed with the completion of the Task. Data objects should be used in the following circumstances:

Additionally, data objects should be used to represent certain data types, notably, complex types that include:

This is done to enable visualization of the data flow within a workflow and between Service Tasks.

In addition to input data, processes and service tasks also often contain parameters for configuring each service. Unlike data objects, these parameters should be configured directly in the Service Task and not represented by a data object. Simple types typically consist of the following:

The nature of OGC services is to utilize complex data inputs and therefore a helper class has to be created for each implementation. This section does not describe in detail the implementation practices for creating helper classes as it is envisaged that BPMN does not support external data types natively, but instead uses a common internal method of dealing with data objects. Currently, the data parsers are defined by the processing services, as it is likely that this is the entry point for getting data. As mentioned previously, the BPMN engine has no processing capability of its own but relies on the services that it is orchestrating for processing and data parsing power. Therefore, all outputs from a WPS are represented as an Object type. This allows the output of a WPS, be it as a FeatureCollection or reference, to be successfully handled by the workflow engine. It is possible to define the input and output types as their actual types if required. The disadvantage of using Object types is that no validation is done by the workflow engine on the content of an input or output parameter.

Currently, all image types should be referred to via a reference since imagery is not supported natively within BPMN or via the helper class. But as with vector data, services that can support imagery should be used to process and parse that imagery.

This implementation does security by first testing whether the capabilities of the service requires security, it does this by getting the capabilities of a service and looking for <Constraint> tags within the advertised operations.

It is assumed that if the constraint tags are missing or are empty, then there are no security constraints on the service. If there are no security constraints, then the service execution happens as normal without the inclusion of a bearer token in the request. However, if there are found to be constraints, they are currently assumed to be OAuth2. The Bearer token is currently held in the process variables for the BPMN workflow. Originally it was considered to have a single security federation with the OAuth Bearer token inserted into the header of the request to the WPS that fronts the BPMN workflow engine. However, it quickly became apparent that this was an over-simplification of the requirements for even the simplest workflows due to:

The helper class then takes the security token as an input and inserts it into the HTTP header of the execute request for the relevant WPS service. If the token is valid then the WPS executes as normal.

The WPS is there as a simple way for a client to execute a workflow by executing a WPS process. As mentioned, jBPM uses Git as a method of remotely updating projects within the web application. Therefore, the WPS process simply commits the BPMN document to the relevant project within the jBPM instance. In the testbed, the jBPM project is already setup and the change committed is the workflow document received by the WPS. jBPM projects contain supporting files beyond the workflow document such as project configuration that includes registration of service task work items and deployment variables. Although these can be configured remotely via Git, they are not in the Testbed due to the added complexity.

This highlights an issue where jBPM requires more than a BPMN document to perform executions as there are several supporting files used in the registration process. Therefore in the testbed, projects are updated with changes to the BPMN document and these usually include updates to variable values for the WPS processes executed by the workflow engine.

jBPM offers a REST API as a method of executing workflows remotely, however this is a multi-step process. The steps are as follows:

Executing jBPM remotely requires basic authentication, which is passed in the header information. Additionally, content-type must also be set appropriately (all examples here are XML, but the system has been tested using JSON).

All of this is managed through the REST API with the following example requests.

Microservice architecture is a way of working that encourages development and deployment of small pieces of functionality in a loosely coupled and finely grained architecture. Loose coupling enables interoperability between components where small pieces of functionality can be developed and replaced without compromising the functionality of the entire system. Loose coupling of microservices also relies on lightweight protocols to perform communication between microservices and clients. Developing microservices lends itself to small teams working in parallel on different pieces of system functionality. This provides an efficient use of resources and enables that no one module of a system becomes over-complicated and ultimately un-maintainable.

Another aspect of microservice development and delivery is the use of Agile methodologies and continuous integration/delivery practices. Although not a focus of this study, continuous delivery is a software engineering process (DevOps) that has sought to utilize microservices architecture by developing capability though short cycles (sprints in Agile Scrum terminology) which means that releases are performed regularly with minimum viable products.

Containerization and micro-services have become fashionable in modern architectures, or at least are reported to have. Indeed, many large organizations still maintain legacy systems that do not implement any sort of micro-services architecture at all, therefore, these technologies are still pipeline considerations for many. Docker is a popular method of deploying capability using container technology. Containers may be thought of as very stripped-down virtual machines (VM) that generally run inside virtual machines in the loose sense. Containers package everything required for the contained service to run within the container image, enables agility, simple deployment of services and removes reliance on system wide libraries to be made available to enable software to run. Some other advantages of containers are as follows:

It should be noted that there is other container technology available including Singularity (https://singularity.lbl.gov/) and Turbo (https://turbo.net). However, Docker appears to be the technology of choice for many.

Containerization and microservices architectures are designed to make efficient use of computational resources whilst providing a stripped-down version of only the required functionality. As mentioned previously, this is in contrast to previous methods of setting up entire virtual machines for the purposes of deploying one or two applications whilst requiring only a small subset of an application’s functionality. A drawback of this approach to deploying resources is that microservice architectures can quickly become complex and unmanageable without a further software solution to orchestrate and maintain containers. Kubernetes is one such open source product that provides orchestration and management for containerized systems.

Kubernetes reports to act as a:

Although Kubernetes does container orchestration, it is marketed as a service that removes the need for container orchestration via removal of centralized control and utilization of pathways to move an entire system from its current state to the desired state.

Cloud Foundry (CF in this section) is a piece of software for managing infrastructure, services and applications. It is designed to take the work out of managing infrastructure to enable the focus to be shifted to application and service development. Essentially, CF sits at the top of the Docker→Kubernetes→CF stack and is focused on the macro-management of systems, rather than the detail of specific deployments. The software is designed to manage a system that is infrastructure and approach agnostic, working with a combination of localized VMs as well as IaaS providers such as AWS and Azure to remove the specifics of deploying applications and services on those specific platforms. This allows developers to focus on development rather than deployment.

CF is designed to manage infrastructure in a way that allows for redundancy, outages and scaling. It does this by installing applications on multiple running VMs that can be upscaled and downscaled as required. Load balancing is accomplished at via three different routes:

Geospatial data has always required a processing overhead due to the nature of the complex operations that can be performed using spatial techniques. However, with recent trends including Volunteered Geographic Information (VGI), crowdsourcing and collection of three-dimensional (3D) point clouds, data volumes have grown-significantly. Processing of these datasets is a challenge due to their size and complexity. Up until recently, OGC architectures have assumed a one-to-one relationship between server and interface, that is, one interface exposes resources on one server instance. These new forms of big data require a different approach to resource management to enable the most efficient usage.

CF and technologies like it are built to cope with processing large quantities of data across elastic architectures and resource management systems with resources spread across different cloud providers, local machines, virtual machines and containers. From an OGC perspective, many of the use cases put forward for testbeds do not require this level of behind-the-scenes resource management, as the use cases are for demonstrator deliverables. Utilization of CF and associated technologies falls into the operational sphere, where timely result return is critical to mission success. The movement to a resource-based architecture (at least in WFS 3.0) is a move that mirrors the move to REST based architectures in many of the non-geospatial (and geospatial for that matter) domains and is timely in terms of CF. The new resource-based architectures do not assert or assume a one-to-one mapping between resources and interfaces, or indeed that resources are co-located. Resource-based architectures provide an API to access the contained resources that may or may not route through to local servers.

CF has potential in the geospatial domain as it enables scalable, elastic resource management using a variety of providers. This coupled with the movement to resource-based architectures and APIs offers the OGC the opportunity to create or revise a suite of standards that support more modern, real-world operational practices.