Modernizing SDI: Enabling Data Interoperability for Regional Assessments and Cumulative Effects CDS

This CDS seeks to specifically identify standards-based solutions for SDIs that enable data interoperability of key environmental data, from multiple jurisdictions, using emerging Internet-based technology like machine-learning/reasoning, data fabrics, data lakes, cloud services, OpenAPIs, and other evolving standards, technologies and tools. This was accomplished through the use of:

In CE analysis, data is sourced from a range of jurisdictions, governments, sectors, domains, over time, and social/community contexts. The primary question or issue for a modernized SDI in this context is:

The overall objective of this CDS is to consult the community and inform federal, provincial, territorial and First Nations/Indigenous stakeholders, concerned with cumulative effects and regional assessments, how best to establish consensus and implement common, open standards-based, approaches that leverage emerging technological capabilities, leading to new levels of digital geospatial data interoperability. Consultation outcomes will provide insight into what is required to implement a modern SDI to best meet stakeholder needs in the CE context.

In addition to secondary research, an RFI, and a Modernizing SDI Workshop were used to collect the views of the SDI community to provide for future SDI modernization with a focus on CE, governance and future directions.

These key conclusions and recommendations are summarized below. A more comprehensive description and list is presented in Chapter 10, Conclusions and Recommendations.

These conclusions and recommendations listed here may provide a focus for a future modernized SDI and a possible pilot initiative following the CDS. Additional conclusions and recommendations and analysis are presented in Chapter 10.

All questions regarding this document should be directed to the editor or the contributors:

Contacts

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

Canada’s SDI, referred to as the CGDI, is the relevant base collection of standards, policies, applications, and governance that facilitate the access, use, integration, and preservation of spatial data.

GeoConnections is a national program with the mandate and responsibility to lead the CGDI through a baseline of consensus-based, internationally accepted standards-based technologies and operational policies for data sharing and integration.

The Federal Geospatial Platform (FGP) is an initiative of the Government of Canada’s Federal Committee on Geomatics and Earth Observations (FCGEO), a committee of senior executives from 21 federal departments and agencies that are producers, consumers or stakeholders in activities, requirements and infrastructure related to geomatics. In 2017, the FCGEO community acted on an opportunity for federal departments and agencies to manage geospatial information assets in a more efficient and coordinated way by using a common “platform” of technical infrastructure, policies, standards and governance. The FGP fully leverages the standards, standards-based technologies and operational policies endorsed by the CGDI.

The FGP’s primary mission is to “Geo-enable the Canadian Federal Government”. The FGP intranet site provides a collaborative online environment where federal government employees can easily share, find, view and analyze the Government of Canada’s authoritative geospatial data holdings to support informed and insightful decisions and policy-making, and ultimately provide better service for Canadians. Overall, the FGP provides an enabling infrastructure to the public service and to Canadians, for access, visualization and analysis of trusted geospatial data, services and applications.

Under the basic premise, “build it once, use it many times,” the FGP leverages coordination efforts and utilizes best practices, new technologies, and open standards to provide more accessible data and services while realizing efficiencies through shared, cloud-enabled infrastructure and economies of scale. This approach allows FGP to supply its data and services to other government initiatives. The FGP makes all federal open geospatial metadata and web services available to Canada’s Open Government Portal - Open Maps. The FGP will also underpin the Open Science and Data Platform for Cumulative Effects.

NRCan and Environment and Climate Change Canada have co-developed and launched the Open Science and Data Platform (OSDP) for Cumulative Effects. The FGP is a primary delivery partner for the OSDP initiative, with responsibility for making geospatial technologies, federal, provincial and territorial geospatial data and web services needed for CE analysis available to the OSDP.

The OSDP provides Canadians a single point of access to environmental data and scientific information that can be used to understand the CE of human activities. By looking at science, environmental data and information about development activities across the country, we can learn about potential impacts to support better decisions in the future. The scope of data and information planned for release through the OSDP highlights a critical need to collaborate with all partners towards greater data interoperability.

Figure 1 shows how the OSDP and its companions FGP and Open Maps integrate into and leverage the CGDI.

Data interoperability is generally defined as the ability for data held in one system to be compatible with other data products or systems and thus able to be integrated with other datasets across a number of different systems or analytical products. Data interoperability can be achieved by optimizing both the usability and reusability of data through the use of open standards.

This CDS will focus research and discussion on the use case of geospatial data typically required for environmental Regional Assessments (RA) and/or CE analyses. NRCan’s current commitment to providing essential geospatial data via the FGP, to support initiatives such as the OSDP, is driving this context.

Additionally, the broad scope of geospatial data requirements for RA/CE, as well as climate change studies and science, make this use case particularly and widely applicable to many stakeholders.

The scope of data needed for RA or CE analysis confirms a critical need to collaborate with all partners towards greater data interoperability. An example of the wide variety and quantity of data required for a RA and CE analysis is demonstrated by the recently completed “Regional Assessment of Offshore Oil and Gas Exploratory Drilling East of Newfoundland and Labrador”. This assessment included the following data categories:

Other geospatial data commonly used in Regional Assessments and Cumulative Effects processes can include:

A more comprehensive list is shown in Figure 2.

A total of 22 responses to the RFI were received representing contributions from 8 different countries. Figure 3 shows the geographic breakdown of respondents.

The locations of the respondents represented a cross section of the global SDI community including responses from North America, Europe, Asia and International representation.

As Figure 4 shows, almost half (45%) of the responses were from the private sector; 23% from Federal/National Governments and 14% from Academic institutions. The remaining 18% of responders are represented by Provincial, Territorial or State Governments, NGOs and Indigenous communities. Again, this is a representative cross section of organization types.

The variety of organization types and wide geographic distribution of respondents indicates that the CDS has a wide variety of stakeholder representation, increasing the validity of the study.

The RFI included a question on the role of the respondent within their organization. This question determined whether the respondent was a data provider and/or data owner, a data user (scientist/researcher or other data user), a solutions provider (e.g., a data enabler, helps provide access to the data, software company, data standards organization, app developer, etc), a policy analyst or other role.

As shown in Figure 5, there was equal representation between data providers, data users (combining scientist and data user responses) and data enablers (solutions providers).

These results indicate that these three key stakeholders have been well represented in the study.

There is no single definition of the term Spatial Data Infrastructure. However, for the purposes of this study, the following definition is appropriate:

Key components of an SDI include governance, technology, organization, and data. These components do not exist in isolation. Each component must be aware of "User Needs" and overall "Socio-economic Value". An SDI is about spatial data, but it is also about infrastructure. “Infrastructure” suggests a reliable supporting environment, in which common technologies, and policies exist. SDI supports applications using geographically-related data through a minimum set of practices, protocols, and specifications (GSDI 2012). Spatial infrastructure must provide access to spatial data in a reliable, consistent, and well-defined manner so that it can support functions addressing end-user needs.

The four key components of an SDI are shown in Figure 6:

These components are defined as follows:

In addition, an SDI includes all aspects related to spatial information including the structure of the data and all the interfaces to the systems that disseminate or present the information. At a more granular level, these fundamental components can include:

The advantages of a modernized SDI include the reduction of data duplication, preservation of data, ease of access to multiple data sources, integration of data for use in disparate tools, facilitates database maintenance, promotes institutional cooperation and general promotion of awareness of spatial data.

The number and types of stakeholders is changing with the changing environment. The ever-expanding sea of data, among other things, are causing these changes. Due to increasing issues surrounding climate change, the chances of disasters and emergencies also increases. This demonstrates the cross-cutting nature of various stakeholder groups.

The responses from the RFI have differentiated the range of stakeholders into five levels or classes. The stakeholders summarized under each level often have some level of influence on each other, illustrated by the circular arrows (Figure 7).

Using Figure 7 and beginning in the upper left, the wide class of end-users includes all consumers of products provided by the other classes, e.g., data and services, products in the form of reports and statistics, policies and regulations, etc. Following the arrows clockwise, the next class aggregates all data producers or creators, data providers, data brokers, and value-added resellers. This large group is of particular relevance, as it is responsible for one of the main products of the SDI, the data. The third class covers data processors such as GIS professionals, data scientists, data modelers, mapping experts, or others in the high-end supercomputing environment who are addressing the complexity of near or real-time analytics/forecasting geospatial products. These experts create products such as analyses, reports, statistics, or maps using data provided by the previous group. The fourth class, data handlers, somewhat intersects the previous three, and includes the hardware, storage and computing service providers that provide the necessary infrastructure for data exchange and processing. The last class, policy (decision) makers, tends to intersect the classes described before. It lays out the necessary rules and guidelines for a successful operation and governance of modernized SDIs.

Identified stakeholders from RFI responses and workshop attendees were classified in one or many of these five levels and come from a wide range of organizations. An already long, though still non-exhaustive, list is provided in Table 2.

Many of the organizations included in this list have been emphasized as particularly relevant by respondents to the Modernizing SDI CDS RFI. The editors of the ER continue to welcome the involvement and contributions of anyone involved in data management willing to support the goals and objectives of a possible future pilot.

Though the stakeholders vary considerably, there is substantial overlap in terms of needs. Generally speaking, needs for the data consumers or end users include the aspects of easy discovery, access, download and analysis of spatial data. For the data producer, provider and processor, needs include the ability to publish, integrate, aggregate and analyze geospatial data and related non-geospatial data. Focus should be on ease-of-use and effectiveness. Integrated systems, possibly in a system-of-systems or network-of-networks approach, with the ability to harvest data from existing solutions in a secure, reliable manner, should be supported.

In addition, there is a need for further requirements on real-time or archived availability, data and system Intellectual Property Rights (IPR), reuse and indemnification rules and regulations, security and privacy settings, as well as financial costs. A modernized SDI also needs to consider how to enable and respect the rights of Indigenous nations. An example are the First Nations principles of OCAP® (Ownership, Control, Access, and Possession) which assert the rights of First Nations to control data ownership and access, including how their data can be used.

On the system side, it is essential that systems are operational and reliable with clear life cycle costs to providers and users. Stakeholders require robust, but intuitive, easy-to-use tools, to access, visualize and contribute data in a manner that allows for ingestion into organizations supporting policy development and decision making. The underlying systems have to cater to various types of consumer capacities. While some of the stakeholders may have very limited internal geospatial capacity or solutions, others are far more advanced.

From an analysis of the RFI responses and workshop input, the stakeholder needs relevant to an SDI can be distilled and summarized as follows.

These four overarching needs are a simplification of the wide variety of needs facing stakeholders. However, keeping these four requirements top-of-mind during modernization will lead to a more effective, sustainable, useful and dynamic SDI for all stakeholders.

The challenge is to manage both the data/analytic contributions, and the data/analytical needs of the many stakeholders, during the present period, when an overload of new data is pouring into the community on a regular basis. Preparedness and planning phases are critical to make management and ease of access to the data a reality. The Information and Communications Technology (ICT) infrastructure must be designed to scale up in order to handle the ever-increasing demands for data and its analysis.

The engagement of stakeholders and the awareness raising of an SDI among potential stakeholders are key goals of the Modernizing SDI CDS. First and foremost, the best way to get stakeholders involved and well served is to meet their needs. This requires making data easy to find, use, and understand. It can be best summarized by the FAIR principals:

For any modernization of an SDI to be considered successful, the “FAIR” guiding principles should be adopted. The more we dive into these principles the more complex things tend to get. It must also be kept in mind that a particular SDI does not have to provide everything to everyone.

This report reflects guidelines and experiences from a significant number of data management experts to identify the best way to achieve these essential requirements. In addition, ease of use, reliability, and completeness, are further dimensions that can be actively pursued. The following subsection identifies aspects that need to be addressed in order to improve the participation and integration of stakeholders with a modernized SDI.

Coordination of SDI modernization activities and collaboration among the various organizations involved is a critical success factor for a modernized SDI. A successfully shared modernization effort would be a stepping stone to other collaboration activities that could focus on increased data collection, introduction of robust monitoring programs, and ideally reduced duplication of effort. Additional coordinating activities include fostering early coordination and planning, encouraging transparency within the public sector so that collection priorities and data requirements are clearly stated and that the most efficient approach can be applied to ensure end user needs are met. In particular, the following aspects shall be considered:

The RFI responses and workshop responses described three primary standards bodies, the ISO, the OGC and the W3C. It should be noted that these organizations do not work in isolation and significant efforts are underway to better integrate standards activities within these three organizations.

Standards and Interoperability address mechanisms and agreements to ensure that components which are part of, or that are loosely connected to, an SDI can communicate with each other. The following underlying principles govern the implementation of standards within any SDI.

Many in the environmental analysis community have adopted OGC standards. However, there were organizations responding to the RFI that were continuing to use proprietary systems. Most of the respondents, nearly 90%, noted the use of open geospatial standards in their organization to make data and maps available for inclusion in external sites and applications. Additionally, many organizations have been instructed to use OGC standards when available and develop best practices for implementation of the standards.

RFI and workshop responses described new or emerging OGC and other community technologies, standards and protocols that may be used within a modernized SDI. Figure 9 shows a percentage breakdown of these technologies.

The following sections will explore some of these emerging technologies.

There were several significant questions from the RFI that indicated SDI usage and value to respondents. As shown in Figure 10 over 80% of respondents contributed to SDIs. This indicates a broad use of SDIs for this group. In addition, close to 90% of RFI respondents noted the use of open geospatial standards in their organization representing a very high adoption of standards and emphasizing their importance.

However, when answering a question on if the SDIs meet their needs, less than 50% said that they did. Given the resources that have been employed in developing SDIs around the globe it is surprising that this number is this low. This result may be explained by examining stakeholder needs in Chapter 7.3.

Additional research into the causes of this result should be undertaken.

Data access and availability for cumulative effects and regional assessment analysis was questioned in the RFI. As shown in Figure 11, almost 70% of those creating data products released standards compliant data. This again indicates the importance of using standards in this community.

However, most respondents, almost 80%, encounter challenges when trying to integrate disparate data (different sources, times, etc.). These challenges were further explored in Chapter 9.2.

A surprising result of the respondents was the low response to the question about accessing the necessary data required. Less than 25% of RFI respondents can access the data required for their work, indicating over 75% cannot access the data they need. Obviously, any SDI modernization will have to look closely at the reasons behind this unsatisfactory result and concentrate efforts on improving the delivery of data stakeholders need.

After analyzing RFI responses and workshop input, the following challenges were described:

Addressing these challenges within any SDI modernization will improve the usefulness of data used by stakeholders.

The stakeholder needs discussed in Chapter 7.3 and results from questions answered in the previous sections 9.1 and 9.2, result in a number of requirements and constraints on a modernized SDI that will be expanded upon in this chapter. Requirements and constraints will be explored in terms of data sharing, standards and interoperability, funding and investment, integration with existing systems, agility and adaptability, and security, privacy and safety. Figure 12 illustrates these categories.

The following sections will briefly discuss more details on the various categories to ensure a robust baseline for the development of a modernized SDI architecture.

When it comes to SDI design reflected in the RFI responses, two important approaches must be differentiated. They are not mutually exclusive and a chosen approach can still be complemented by the other. In fact, both approaches represent the two extremes of a given continuum, with most implementations featuring some level of middle course. Nevertheless, the architecture design differs depending on the preferred approach. The first approach focuses on a closely architected infrastructure that provides data and apps as services. Thus, the defined architecture caters for a defined set of services (includes rehosted services) that are operated and maintained by a SDI control board, i.e., a group with control over the individual components. The second approach focuses on infrastructures, platforms, and geoportals as they currently exist and emphasizes their integration into a federation of systems. Here, emphasis is on discoverability and integration based on open standards. The first approach puts more control into the hands of the control board, whereas the second approach provides more flexibility and distributed responsibilities. Key to both approaches is the strong adherence to standards to avoid vendor lock-in with limited flexibility and extensibility. It should be emphasized that both approaches can complement each other, i.e., they do not necessarily act in isolation, but support interfaces to allow mutual usage.

The Reference Architecture must find the right balance between being prescriptive while remaining agile to allow for easy integration of upcoming technologies. This requires the modernized SDI to be implemented as a loose federation of portals and platforms, a federated system of systems, discoverable by open specifications and standards rather than being a highly architected infrastructure with data and applications being available as hosted/re-hosted services.

Independent of the chosen approach, a number of additional aspects have been repeatedly identified by RFI respondents and workshop panelists and attendees as being relevant for a successful SDI. These aspects are usually complemented with the standing request for openness as illustrated in Figure 15. Openness usually refers to a number of aspects that describe an element that is openly (in the sense of publicly and royalty free) available and reusable, developed in an open process, and accessible at minimum costs (in terms of data pure reproduction costs or even no costs).

Open science is the movement to make scientific research, data and dissemination accessible to all levels of an inquiring society, amateur or professional. Open systems include open source work and GitHub resources, choices in hardware, operating systems, Cloud, databases, developer tools, direct links to non-GIS systems such as CAD and BIM, etc. Open standards include standards as provided by OGC, IHO, CGSB, FGDC, OASIS, W3C, ASPRS, etc.) and open specifications (widely used but not yet adopted by Standards Developing Organizations (SDOs) and openly published technology such as GeoJSON, Geoservices REST API, etc.)

In addition to these general ‘Open’ requirements, many individual modernized SDI architectural requirements were described in the RFI responses and workshop. From this valuable input, an ideal modernized SDI architecture shall be designed to provide for the following:

The work carried out in this CDS and represented in this report is but a first step in both conceiving of and realizing the challenges and opportunities embodied in a future concept of modernized SDI. There is on the one hand the opportunity to be a part of re-thinking the entire idea of infrastructure, as whatever physical, social, or digital constructs can form a common, reusable basis for the full range of human pursuits. There is on the other hand the challenge of pulling together disparate tools and technologies into a coherent, interoperable whole that can accomplish this. There is no single step that can take us all the way to a modernized SDI of the future. Small and iterative steps of innovation, though, where each new design piece leads quickly to experiment and evaluation, have shown their value in achieving real progress.

Among the first steps that should be considered are a pilot implementation of the SDI capabilities outlined in this report for facilitation of CE/RA projects, followed by expansion into other domains of study where spatial information is a common and critical ingredient. The value of a modernized SDI that emerged from these initiatives would be further enhanced by research to build out the evident affinities between spatial data infrastructure, predictive modeling / AI, and dynamic digital twins. Such work would have the potential to significantly increase the types of work that modernized SDI could be used and reused to accomplish.

This report documents the need for modernized SDI to be well thought out, proven in implementation, and yet responsive and sustainable in the face of exploding data and rapidly evolving technologies. This suggests that operationalization of modernized SDI needs to be an equal part of its iterative development, an accessible capability that is at the same time ever becoming still more modern.

The workshop was presented in three parts: starting with an overview of the challenge of data interoperability in an SDI environment then presenting two panels, in which invited subject matter experts presented deep dives on the challenges, questions, ideas and possible solutions to this issue. Panel sessions followed with an audience question-and-answer period and open discussion.

This clause gives a list of the abbreviated terms and the symbols necessary for understanding this document.