Publication Date: 2020-02-13
Approval Date: 2019-12-25
Submission Date: 2019-11-18
Reference number of this document: OGC 19-083
Reference URI for this document: http://www.opengis.net/doc/PER/CitSciIE-1
Category: Public Engineering Report
Editor: Joan Masó
Title: OGC Citizen Science Interoperability Experiment Engineering Report
COPYRIGHT
Copyright © 2020 Open Geospatial Consortium. To obtain additional rights of use, visit http://www.opengeospatial.org/
WARNING
This document is not an OGC Standard. This document is an OGC Public Engineering Report created as a deliverable in an OGC Interoperability Initiative and is not an official position of the OGC membership. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an OGC Standard. Further, any OGC Engineering Report should not be referenced as required or mandatory technology in procurements. However, the discussions in this document could very well lead to the definition of an OGC Standard.
LICENSE AGREEMENT
Permission is hereby granted by the Open Geospatial Consortium, ("Licensor"), free of charge and subject to the terms set forth below, to any person obtaining a copy of this Intellectual Property and any associated documentation, to deal in the Intellectual Property without restriction (except as set forth below), including without limitation the rights to implement, use, copy, modify, merge, publish, distribute, and/or sublicense copies of the Intellectual Property, and to permit persons to whom the Intellectual Property is furnished to do so, provided that all copyright notices on the intellectual property are retained intact and that each person to whom the Intellectual Property is furnished agrees to the terms of this Agreement.
If you modify the Intellectual Property, all copies of the modified Intellectual Property must include, in addition to the above copyright notice, a notice that the Intellectual Property includes modifications that have not been approved or adopted by LICENSOR.
THIS LICENSE IS A COPYRIGHT LICENSE ONLY, AND DOES NOT CONVEY ANY RIGHTS UNDER ANY PATENTS THAT MAY BE IN FORCE ANYWHERE IN THE WORLD. THE INTELLECTUAL PROPERTY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE DO NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE INTELLECTUAL PROPERTY WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE INTELLECTUAL PROPERTY WILL BE UNINTERRUPTED OR ERROR FREE. ANY USE OF THE INTELLECTUAL PROPERTY SHALL BE MADE ENTIRELY AT THE USER’S OWN RISK. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR ANY CONTRIBUTOR OF INTELLECTUAL PROPERTY RIGHTS TO THE INTELLECTUAL PROPERTY BE LIABLE FOR ANY CLAIM, OR ANY DIRECT, SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM ANY ALLEGED INFRINGEMENT OR ANY LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR UNDER ANY OTHER LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH THE IMPLEMENTATION, USE, COMMERCIALIZATION OR PERFORMANCE OF THIS INTELLECTUAL PROPERTY.
This license is effective until terminated. You may terminate it at any time by destroying the Intellectual Property together with all copies in any form. The license will also terminate if you fail to comply with any term or condition of this Agreement. Except as provided in the following sentence, no such termination of this license shall require the termination of any third party end-user sublicense to the Intellectual Property which is in force as of the date of notice of such termination. In addition, should the Intellectual Property, or the operation of the Intellectual Property, infringe, or in LICENSOR’s sole opinion be likely to infringe, any patent, copyright, trademark or other right of a third party, you agree that LICENSOR, in its sole discretion, may terminate this license without any compensation or liability to you, your licensees or any other party. You agree upon termination of any kind to destroy or cause to be destroyed the Intellectual Property together with all copies in any form, whether held by you or by any third party.
Except as contained in this notice, the name of LICENSOR or of any other holder of a copyright in all or part of the Intellectual Property shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Intellectual Property without prior written authorization of LICENSOR or such copyright holder. LICENSOR is and shall at all times be the sole entity that may authorize you or any third party to use certification marks, trademarks or other special designations to indicate compliance with any LICENSOR standards or specifications.
This Agreement is governed by the laws of the Commonwealth of Massachusetts. The application to this Agreement of the United Nations Convention on Contracts for the International Sale of Goods is hereby expressly excluded. In the event any provision of this Agreement shall be deemed unenforceable, void or invalid, such provision shall be modified so as to make it valid and enforceable, and as so modified the entire Agreement shall remain in full force and effect. No decision, action or inaction by LICENSOR shall be construed to be a waiver of any rights or remedies available to it.
None of the Intellectual Property or underlying information or technology may be downloaded or otherwise exported or reexported in violation of U.S. export laws and regulations. In addition, you are responsible for complying with any local laws in your jurisdiction which may impact your right to import, export or use the Intellectual Property, and you represent that you have complied with any regulations or registration procedures required by applicable law to make this license enforceable.
- 1. Summary
- 2. References
- 3. Terms and definitions
- 4. Overview
- 5. O&M for Cit Sci
- 6. SOS architectures
- 7. SOS servers
- 8. SOS clients
- 9. Data quality estimations on the client side
- 9.1. Quality estimation on vector data
- 9.1.1. How data quality is presented
- 9.1.2. How to start computing data quality
- 9.1.3. Case 1: Positional accuracy of the layer from observation uncertainties
- 9.1.4. Case 2: Logical consistency of the thematic attributes
- 9.1.5. Case 3: Temporal validity of the observation date
- 9.1.6. Case 4: Validity of the positions of observations (by bounding box)
- 9.2. Quality estimation on raster data
- 9.3. Future work
- 9.1. Quality estimation on vector data
- 10. Definitions Server
- 11. User and Application Federation
- 12. Connecting Citizen Science data sets to GEOSS
- Appendix A: Integration between SCENT & LandSense
- Appendix B: Revision History
1. Summary
This Engineering report describes the first phase of the Citizen Science (CS) Interoperability Experiment (IE) organized by the EU H2020 WeObserve project under the OGC Innovation Program and supported by the four H2020 Citizen Observatories projects (SCENT, GROW, LandSense, and GroundTruth 2.0) as well as the EU H2020 NEXTGEOSS project. The activity covered aspects of data sharing architectures for Citizen Science data, data quality, data definitions and user authentication.
The final aim was to propose solutions on how Citizen Science data could be integrated in the Global Earth Observation System of Systems (GEOSS). The solution is necessarily a combination of technical and networking components, being the first ones the focus of this work. The applications of international geospatial standards in current Citizen Science and citizen observatory projects to improve interoperability and foster innovation is one of the main tasks in the IE.
The main result of the activity was to demonstrate that Sensor Observing Services can be used for Citizen Science data (as proposed in the Open Geospatial Consortium (OGC) Sensor Web Enablement for Citizen Science (SWE4CS) Discussion Paper) by implementing SWE4CS in several clients and servers that have been combined to show Citizen Science observations. In addition, an authentication server was used to create a federation between three projects. This federated approach is part of the proposed solution for GEOSS that can be found in the last chapter. Many open issues have been identified and are expected to be addressed in the second phase of the experiment, including the use of a definitions server.
1.1. Requirements & Research Motivation
This experiment was designed to demonstrate how current ICT-based tools can be applied together to allow better citizen participation in CS projects and enable better reuse of the data gathered. Citizen Science is highly transdisciplinary and heterogeneous by nature and current standardization efforts already occur in the OGC (e.g., addressing data model and sharing issues) as well as outside the OGC (primarily addressing project descriptions and dataset metadata). Citizen Science projects might benefit from concrete examples and best practices required to achieve the full benefits of interoperability. OGC is in the ideal position to develop and provide such best practice guidance to the international community. Developed solutions in this IE should be applicable to most Citizen Science projects. Findings from this IE will be generalized as practice examples and might set the basis for additional experimentation in the future.
The FP7 Citizen Observatory Web (COBWEB) project was the first to propose the use of SWE in CS. This work resulted in an OGC public Discussion Paper available on the OGC website (OGC 16-129). The Discussion Paper describes a data model for the standardized exchange of Citizen Science sampling data based on SWE standards. This Discussion Paper was the initial motivation for this IE.
Beyond the work described above, the Citizen Science Association’s International Working Group on Citizen Science Data and Metadata has developed the PPSR-CORE metadata standard and the European Citizen Science Association (ECSA) has a working group that recognizes the value of standardization in the CS activities (supported by a COST Action). However, these activities could benefit from some experimentation that would be able to suggest common best practices while recognizing the particularities and current approaches in different thematic domains, such as biodiversity monitoring. Citizen Science can complement authoritative in-situ observations and fill the information gaps in numerous scientific disciplines that could be essential for informed decision making. In that sense, the way Citizen Science can be integrated into The GEOSS (including GEOSS-Data Core as the pool to promote and share open and free data) is still under investigation.
The Ecosystem of Citizen Observatories (CO) for Environmental Monitoring WeObserve project is a Horizon 2020 funded project focused on improving the coordination between existing COs and related regional, European, and international activities. WeObserve tackles three key challenges that face COs: awareness, acceptability, and sustainability. The WeObserve Community of Practice 3 (CoP3) is about Interoperability of Citizen Science projects. The WeObserve project – via its CoP activities – has represented an opportunity to promote interoperability experimentation in collaboration with the OGC. Such collaboration addresses questions raised in the SWE4CS discussion. In addition, the work offers the possibility to directly feed the results into the relevant OGC standards and promotes their usage within GEOSS (as an important user community of OGC standards).
In anticipation of the 50th Anniversary of Earth Day in 2020, Earth Day Network, the Woodrow Wilson International Center for Scholars, and the U.S. Department of State, through the Eco-Capitals Forum, announce Earth Challenge 2020, a Citizen Science Initiative. This initiative is in collaboration with Connect4Climate – World Bank Group, Conservation X Labs, Hult Prize, National Council for Science and the Environment (NCSE), OGC, Reset, SciStarter, UN Environment, and others to be announced. Earth Challenge 2020 will help engage millions of global citizens in collecting one billion data points in areas including air quality, water quality, biodiversity, pollution, and human health. Earth Change 2020 data will be shared through the GEOSS Portal.
1.2. Prior-After Comparison
This is the first Citizen Science IE conducted by the OGC. Prior to this activity, there was a Discussion Paper on how to apply the SWE standards in Citizen Science (SWE4CS). This experiment positively tested the proposed route using Sensor Observing Services but also has opened the door to future exploration of the SensorThings API.
Also prior this activity, was a H2020 project with a Authentication Service and after the activity, three H2020 efforts formed a bigger federation demonstrating the route to federating and aggregating Citizen Science projects to contribute to GEO objectives.
This work is relevant to the OGC Citizen Science Domain Working Group.
1.3. Recommendations for Future Work
This OGC IE ended on June 2019, but a second IE is foreseen for the following year.
New possible topics for next IE to be discussed among the members include the following.
-
There is a need for clarifying how to coordinate infrastructures for Citizen Science in Europe and adopt standard procedures for data sharing and single sign on. Solving this issue will help in connecting CS to GEO. Steffen Fritz (IIASA) has proposed a side event in the next EuroGEOSS workshop to discuss this coordination with the relevant players. This is emerging as a new activity in the WeObserve Interoperability CoP that is related, but not directly connected, to the IEs.
-
The WMO (World Meteorological Organization) is concerned about the amount of different CS activities that are being organized by meteorological organizations. WMO is looking for ways to take advantage of this new data stream, but problems of standardization of what is measured and how data is being shared arise. WMO has identified the potential of these data streams and would like to harmonize the situations to make data more useful for weather predictions in the future.
-
OGC is promoting a new generation of web services based on OpenAPI. It is unclear how these new web services could impact the use of OGC standards by CS projects but it is seen as an opportunity to make OGC standards more usable and compatible with IT mainstream. A hackathon to develop OGC API specifications occurred on 20-21 June 2019 in London and subsequent hackathons and sprints continue to advance the OGC API standards.
The definition of the follow-up IE started upon completion of this IE.
1.4. Document contributor contact points
All questions regarding this document should be directed to the editor or the contributors:
Contacts
| Name | Organization |
|---|---|
Joan Maso |
UAB-CREAF |
Andy Cobley |
University of Dundee |
Valantis Tsiakos |
Institute of Communication & Computer Systems (ICCS) |
Nikolaos Tousert |
Institute of Communication & Computer Systems (ICCS) |
Theodoros Theodoropoulos |
Institute of Communication & Computer Systems (ICCS) |
Simon Jirka |
52 North |
Sven Schade |
European Commission, Joint Research Center (JRC) |
Andreas Matheus |
Secure Dimensions |
Stefano Tamascelli |
XTeam Software Solutions |
Friederike Klan |
Citizen Science Group, Institute of Data Science, DLR |
Trupti Padiya |
Citizen Science Group, Institute of Data Science, DLR, Friedrich-Schiller-University Jena |
Initiators
| Organization |
|---|
Universtat Autònoma de Barcelona - CREAF (UAB-CREAF) |
International Institute for Applied Systems Analysis (IIASA) |
Joint Research Center (JRC) |
European Space Agency (ESA) |
Woodrow Wilson International Center for Scholars (Wilson Center) |
The WeObserve project has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement No. 776740.
This presentation reflects only the editor’s views and the EU Agency is not responsible for any use that may be made of the information it contains.
1.5. Foreword
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.
1.6. Acknowledgements
This report was coordinated and developed with funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements: no 776740 (WeObserve), no 688930 (SCENT), no 689744 (Ground Truth 2.0), no 690199 (GROW Observatory), no 689812 (LandSense) as well as no 730329 (NEXTGEOSS).
2. References
The following normative documents are referenced in this document.
Although the following is an OGC Discussion Paper that is not an OGC standard and cannot be considered strictly a normative reference, it is actually the basis for several sections of this document and should be considered as important background:
3. Terms and definitions
For the purposes of this report, the definitions specified in Clause 4 of the OWS Common Implementation Standard OGC 06-121r9 shall apply. In addition, the following terms and definitions apply.
-
Citizen Observatory (CO)
Community-based environmental monitoring and information systems that invite individuals to share observations, typically via mobile phone or the web (from: https://www.weobserve.eu/about/citizen-observatories).
-
Citizen Science (CS)
The collection and analysis of data relating to the natural world by members of the general public, typically as part of a collaborative project with professional scientists (from: https://www.uen.org/crowdandcloud/citizen.shtml).
-
Citizen Science Association
A network that seeks to promote and advance citizen science in a region or around the world. Examples are the American Citizen Science Association (CSA), The European Citizen Science Association (ECSA), or the Citizen Science Global Partnership (CSGP).
-
Citizen Science Federation
A network of Citizen Science that aims to aggregate innovative Earth Observation technologies, mobile devices, community-based environmental monitoring, data collection, interpretation, and information delivery systems to empower communities to monitor and report on their environment. An example of this is the The LandSense Federation.
-
Community of Practice (CoP)
Community which works to consolidate practice-based knowledge of COs sharing information and resources as well as developing guidelines and toolkits for COs (from: https://www.weobserve.eu/cops/).
3.1. Abbreviated terms
-
CitSciIE Citizen Science Interoperability Experiment
-
CO Citizen Observatory
-
CoP Community of Practice
-
COST European Cooperation in Science and Technology
-
CS Citizen Science
-
CS DWG Citizen Science Domain Working Group
-
CSGP Citizen Science Global Partnership
-
ECSA European Association of Citizen Science
-
EO Earth Observation
-
ICT Information and Communication Technologies
-
IE Interoperability Experiment
-
O&M Observation and Measurements
-
PPSR Public Participation in Scientific Research
-
SOS Sensor Observation Service
-
SSO Single Sign On
-
SWE Sensor Web Enablement
-
SWE4CS Sensor Web Enablement for Citizen Science
-
TC Technical Committee
-
TIE Technology Integration Experiments
-
WPS Web Processing Service
4. Overview
This Engineering Report focuses on the findings of the first phase of the Citizen Science Interoperability Experiment (CitSciIE).
The primary focus of the OGC CitSciIE was to demonstrate the interoperability of Citizen Observatories and Citizen Science projects and the way OGC standards can be applied to Citizen Science, including possible relationships to other relevant standards from the community. In particular, a subset the originally proposed topics were being addressed based on the participant organizations:
-
The use of OGC standards or draft specifications (e.g., SWE or SWE4CS) to support data integration among CS projects, and with other sources, especially authoritative data;
-
The use of ISO standards, OGC publications, and community resources to document data quality aspects (e.g., UncertML, QualityML);
-
The integration of CS projects/campaigns in a Single Sign-On system (SSO) federation; and
-
The relationships between OGC standards and data and metadata standards currently used by Citizen Science projects.
The desired outcome of this experiment was the following.
-
Successfully demonstrate how OGC standards (e.g., SWE) are applicable to Citizen Science, document available supporting tools, identify the challenges of using OGC SWE standards (or Internet of Things equivalent solutions) within current Citizen Science projects, and propose a way forward. Make recommendations to the Earth Science 2020 initiative on which OGC standards should be utilized to underpin interoperable data collection and sharing.
-
Successfully demonstrate how to estimate Citizen Science data quality and make the quality indicators and conformity available in the document and in supporting tools and link them to the OGC SWE standards (or Internet of Things equivalent solutions) within current CS projects, as well as propose a way forward.
-
Determine the security considerations and the available tools to support an SSO federation that helps users in participating in several projects by using a single user account.
-
Assess the possible relationships of OGC standards (e.g., SensorML) with other existing standards in the field (e.g., PPSR - CORE, the ontology developed by the COST Action on Citizen Science, and the Citizen Science Definition Service (CS-DS) developed in the NextGEOSS project).
-
Satisfy and document the necessary requirements to integrate Citizen Science into GEOSS by using OGC standards.
This IE has been promoted by the OGC Citizen Science Domain Working Group, the WeObserve and NextGEOSS H2020 projects, and The Earth Challenge 2020 project as supported by National Geographic Society. This IE contributes not only to the interoperability and possibly standardization program of the OGC, but also to the GEOSS. This work is also relevant to the foundational objectives of the Citizen Science Global Partnership (CSGP). Regional and national Citizen Science Associations will equally benefit from the results of this OGC IE.
4.1. Structure of the activities
The official kick-off meeting for the OGC CitSciIE experiment was held on Friday 14th September 2018 at the OGC TC meeting in Stuttgart, Germany. Activities continued until March 2019.
During the kick-off meeting of the IE, the following subgroups emerged.
-
V: Vocabularies for organizing Citizen Science projects. There was a discussion on essential variables but also on other kinds of practices that can be associated to vocabularies, i.e., on how to publish vocabularies (PublishingDefs) or on defining a list of vocabularies that could be useful to experiment with (observations, project descriptions, general glossaries of terms).
-
Working item V.1: A list of the current projects that the Wilson Center knows can align with the Earth Challenge topics (air and water quality, pollution, human health, and eventually biodiversity) and extraction of a common set of variables the projects cover.
-
Working item V.2: Analysis of data models that contributors in the experiment can bring in: Air quality (HackAir), Biodiversity (Atlas of Living Australia & Natusfera), Mosquito (CREAF), Land Use (IIASA), Phenology (CREAF), Invasive Alien Species (JRC).
-
Working item V.3: Consider the COST action metadata model for inclusions as another vocabulary: this might include a set of definitions of phenomena that are being addressed by CS initiatives (based on the inventory of citizen science activities for environment policies).
-
-
D: Data sharing using OGC standards such as Observations and Measiurements (O&M) and Sensor Observation Service (SOS). A pool of services were identified for participating in an IE, including SOS services and clients and citizen science project databases and APIs.
-
Working item D.1: A set of instructions on how a CS project can easily setup an SOS service. The service could include 52North implementation and might include MiraMon SOS (with some work in the implementation). The service should address the case of a small project contributing to the Earth Challenge 2020.
-
Working item D.2: Create an SOS endpoint for HackAir data with minimum resources.
-
Working item D.3: Define the requirements for a data provider that could assist the Wilson Center in setting up the challenge database. The requirements should consider upload of data into the system. The IE preference was to go for a harvest system instead of a federated system. The working item could describe a possible architecture to allow the dialog between the central database and the small contributing projects and should impose data sharing requirements (services o APIs) on the central database.
-
-
S: Connection between LandSense federation and JRC user system.
-
Working item S.1: Interoperability test on the integration of LS-SSO and JRC-SSO.
-
-
Q: Data quality.
-
Working item Q.1: Write a document on perspectives of the different quality aspects: Quality assessment (ISO 19157-QualityML), Quality improvement, Quality plan, Data Management principles (ISO 8001), Quality documentation, Quality communication.
-
Working item Q.2: Perfect the quality measurement system based on Web Processing Service (WPS) and SOS harvest by demonstrating the concept in practice. Also include in the SOS harvesting the possibility to have a query for assessing the quality of "views"/"selections"/"fragments" of a dataset.
-
Connection with: D.2.
-
-
Working item Q.3: Refine the QualityML vocabulary with new entries considering the work done in Australia.
-
Connection with: D.3.
-
-
Working item Q.4: Add new entry point the QualityML for other common vocabulary formats like TTL, etc.
-
Connection with: V.
-
-
For each of the subgroups a chair and the main participants and contributors were identified. Responsible persons were also assigned to each of the working items.
4.2. Results detailed in the subsequent sections
These are the main activities and outcomes of the interoperability experiment detailed by activity.
Data sharing using OGC standards such as O&M and SOS
This activity has been the most active one. During the IE, the following servers have been deployed: MiraMon SOS server, Grow SOS, DLR istSOS SOS, and 52north SOS. Three clients have also been produced: MiraMon SOS browser, Grow SOS data viewer, and 52north Helgoland. In the last meeting at the EGU, the group was able to demonstrate interoperability by connecting the SOS clients to the SOS services and showing the data on clients, sometimes mixing data form different services and datasets in a single view. This is the most significant result of the experiment and is being extensively documented in this Engineering Report in sections 5, 6 and 7.
Data quality
Two quality vocabularies have been identified: Australian work done by Peter Brenton’s team (https://github.com/tdwg/bdq) and the QualityML vocabulary developed by CREAF in the GeoViQua project. The intent of this IE was to do a comparison of both approaches, but the participants were not able to do so in the timeframe of the first IE. Section 8 describes the current status of the activity. It is foreseen that the second IE will continue what was started here.
Definition server for organizing Citizen Science projects
The objective of this activity was to support the Earth Challenge 2020 research questions. The questions were defined during the first month of the experiment and now it is time to analyze the questions in terms of data needs and thematic vocabularies to be used. Because the analysis has not yet been performed, this activity has not resulted in tangible outputs and will be reintroduced in the second IE. Details of this development are described in section 9.
Connection between LandSense federation and other user systems
Secure Dimensions (Andreas Matheus) was very active in providing demonstrations and information on how the LandSense federation works and how other projects can be included in the federation and use the SSO facility. Unfortunately, no other member of the CoP had the resources to apply the SSO on their services or clients and take advantage of the LandSense offering. The activity resulted in a video demonstration that is publicly available here: https://portal.opengeospatial.org/files/?artifact_id=81550.
Section 11 Details the current status of the activity.
Other
Section 11 summarizes the lessons learned that can be applied to GEOSS.
In addition to these activities, another activity about quality annotating scientific documentation in a standard way was proposed by Lucy Bastin. A video was recorded that summarizes the idea: https://portal.opengeospatial.org/files/?artifact_id=82544.
5. O&M for Cit Sci
In a feature model all characteristics of a feature are considered properties of the feature and are not semantically separated at the abstract level.
The O&M standard ([OGC 10-004r3], Abstract Specification Topic 20: Observations and Measurements) defines a data model for observations where main concepts are separated as represented in the Figure 1.
For each observation, O&M allows us to document the following characteristics.
-
Where the observation is located: even if the observation was made remotely with a camera or a drone, it is commonly more relevant to know the position of the observed phenomena (the sensor position can also be recorded).
-
When the observation took place and what time period it represents: even if samples were collected and analyzed later, it is commonly more relevant to know the instant or period of the observed phenomenon.
-
How the observation was done: this will describe the procedure and instrument used to capture the phenomenon.
-
Who did the observation: the procedure and instrument used to capture the phenomenon was installed or used on site by someone. In citizen Science, where many observers contribute small pieces of information that together will form a dataset, it is particularly important to record at least an observer identifier.
-
What was measured: this will define the property names and units of measure of the variables observed.
-
What data was collected: this will record the actual values of the properties measured.
-
What is the expected quality of the observation: if an estimation of the quality of the observation was done, it is important to document the quality.
In the O&M data model, the above aspects are clearly separated semantically as shown in Figure 2. This is the main value of the O&M model and its usage SOS (or the SensorThingsAPI that uses a very similar approach to model the data), but it is also the main handicap in applying the standard.
| Concept | O&M | type |
|---|---|---|
Where |
featureOfInterest |
GFI_Feature |
When |
phenomenonTime, resultTime |
|
How |
procedure |
OM_Process |
Who |
procedure |
OM_Process |
What |
observedProperty |
GF_PropertyType |
Data |
result |
Any |
Quality |
resultQuality |
Even if the aspects above are separated, the O&M model gives a lot of flexibility in defining the properties and this flexibility can condition interoperability when trying to combine data from different sources. The standard give us freedom to select among the different geometries provided by GML to define the featureOfInterest. The standard gives us even more freedom on the data collected that can have any imaginable structure.
That is the reason why the data model used to represent the data gathered by a Citizen Observatory needs to be carefully considered before even starting the first data collection campaign. Data models can be designed in UML for clarity, but they are later encoded in XML. XML is the only official encoding that O&M references in the OGC website ([OGC 10-025r1] Observations and Measurements - XML Implementation v2.0). Nevertheless, there is a JSON alternative discussed in an OGC Discussion Paper ([OGC 15-100r1] OGC Observations and Measurements – JSON Implementation) that does not represent an official position of the OGC but can be implemented anyway. As it will be discussed latter, the interpretation of long XML files might be to slow in web browsers, and in this case, a JSON encoding is regarded as a good alternative either in O&M or the SensorThings API.
5.1. The GT20 examples
In the Ground Truth 2.0 project, we have been using the MiraMon implementation of O&M. This implementation assumes a simplified situation that considers that each observation can be represented by a single row in a CSV or in a single record of a database table. Coordinates are represented as a single point. In this situation, we select which column names represent the phonomenonTime, the procedure (that actually is including the user name), and the featureOfInterest (the coordinates). The rest of the columns are considered part of the data record that needs to be provided as the result.
Section 8.2.1 of the [OGC 08-094r1] OGC® SWE Common Data Model Encoding Standard v2.0 describes a way to encode a DataRecord as an array of fields that can numbers, strings, dates, etc. In our simplified assumption, this array is ideal to wrap the properties of the observations that cannot be mapped to any other O&M aspect. This practice is consistent with the section 7.2.8 of the SWE4CS discussion paper.
The following is an example of how a water quality observation is represented following the O&M model and encoded in XML.
<om:OM_Observation gml:id="vatten-fokus_2_1">
<om:type xlink:href="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_ComplexObservation"/>
<om:procedure xlink:href="http://www.opengis.uab.cat/vatten-fokus/procedure/22655"/>
<om:observedProperty xlink:href="http://www.opengis.uab.cat/vatten-fokus/observedProperty"/>
<om:featureOfInterest xlink:href="http://www.opengis.uab.cat/vatten-fokus/featureOfInterest/2"/>
<om:result xsi:type="swe:DataRecordPropertyType">
<swe:DataRecord>
<swe:field name="CREA_DATE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Creation_Date">
<swe:value>07/12/2018 17:23</swe:value>
</swe:Text>
</swe:field>
<swe:field name="SITE_NAME">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Site_name">
<swe:value>Dunkershall. V¤gtrumma uppst¤ms.</swe:value>
</swe:Text>
</swe:field>
<swe:field name="LAND_USE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Land_use_in_the_immediate_surroundings">
<swe:value>Agriculture</swe:value>
</swe:Text>
</swe:field>
<swe:field name="BANK_VEGE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Bank_vegetation">
<swe:value>Grass</swe:value>
</swe:Text>
</swe:field>
<swe:field name="NITRATE">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/NITRATE">
<swe:uom/>
<swe:value>1.50</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="PHOSPHATE">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/PHOSPHATE">
<swe:uom/>
<swe:value>0.075</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="WATER_COLOR">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Estimated_water_colour">
<swe:value>Colourless</swe:value>
</swe:Text>
</swe:field>
</swe:DataRecord>
</om:result>
</om:OM_Observation>The following is an example of how two air quality observations are represented following the O&M model and encoded in JSON.
{
"id":"meet-mee-mechelen_1_0",
"type" : "http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_ComplexObservation",
"phenomenonTime" : "2017-11-19 17:20:00+01",
"resultTime" : "2017-11-19 17:20:00+01",
"procedure" : "http://www.opengis.uab.cat/meet-mee-mechelen/procedure/5",
"observedProperty" : "http://www.opengis.uab.cat/meet-mee-mechelen/observedProperty",
"featureOfInterest" : "http://www.opengis.uab.cat/meet-mee-mechelen/featureOfInterest/1",
"result": {
"type":"DataRecord",
"field":[
{
"name" : "CAMPAIGN",
"type" : "Text",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/field/CAMPAIGN",
"value" : "Oct-Nov2017"
},
{
"name" : "bc_aggr",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr",
"value" : "3155"
},
{
"name" : "bc_aggr_mi",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_mi",
"value" : "80"
},
{
"name" : "bc_aggr_ma",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_ma",
"value" : "16413"
},
{
"name" : "bc_aggr_st",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_st",
"value" : "3398"
},
{
"name" : "uncertaint",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/uncertaint",
"value" : "0.50"
}
]
}
},
{
"id":"meet-mee-mechelen_2_1",
"type" : "http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_ComplexObservation",
"phenomenonTime" : "2017-11-19 17:20:06+01",
"resultTime" : "2017-11-19 17:20:06+01",
"procedure" : "http://www.opengis.uab.cat/meet-mee-mechelen/procedure/5",
"observedProperty" : "http://www.opengis.uab.cat/meet-mee-mechelen/observedProperty",
"featureOfInterest" : "http://www.opengis.uab.cat/meet-mee-mechelen/featureOfInterest/2",
"result": {
"type":"DataRecord",
"field":[
{
"name" : "CAMPAIGN",
"type" : "Text",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/field/CAMPAIGN",
"value" : "Oct-Nov2017"
},
{
"name" : "time_first",
"type" : "Text",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/field/time_first",
"value" : "2017-11-06 08:00:18+01"
},
{
"name" : "bc_aggr",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr",
"value" : "3382"
},
{
"name" : "bc_aggr_mi",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_mi",
"value" : "80"
},
{
"name" : "bc_aggr_ma",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_ma",
"value" : "17256"
},
{
"name" : "bc_aggr_st",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/bc_aggr_st",
"value" : "3663"
},
{
"name" : "number_of_",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/number_of_",
"value" : "25"
},
{
"name" : "number_o_1",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/number_o_1",
"value" : "13"
},
{
"name" : "mean_numbe",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/mean_numbe",
"value" : "7"
},
{
"name" : "uncertaint",
"type" : "Quantity",
"definition" :"http://www.opengis.uab.cat/meet-mee-mechelen/variable/uncertaint",
"value" : "0.50"
}
]
}
}These examples were produced by SOS requests to this URL: http://www.ogc3.uab.cat/cgi-bin/CitSci/MiraMon.cgi?. A client connecting to this service can be found here: http://www.ogc3.uab.cat/gt20/.
5.2. HackAir examples
To illustrate the flexibility of the O&M, we have included this air quality report that shows how HackAir data is presented by a 52North SOS implementation. In this case the result presents a single numerical value while the other information is provided as parameters. This approach is consistent with section 7.2.2.5 of the O&M standard.
<om:OM_Observation gml:id="o_499">
<om:type xlink:href="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement"/>
<om:phenomenonTime>
<gml:TimeInstant gml:id="phenomenonTime_499">
<gml:timePosition>2019-01-01T00:00:12.000Z</gml:timePosition>
</gml:TimeInstant>
</om:phenomenonTime>
<om:resultTime xlink:href="#phenomenonTime_499"/>
<om:procedure xlink:href="sensors_arduino_1000"/>
<om:parameter>
<om:NamedValue>
<om:name xlink:href="PM2.5_AirPollutantIndex"/>
<om:value xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">bad</om:value>
</om:NamedValue>
</om:parameter>
<om:parameter>
<om:NamedValue>
<om:name xlink:href="http://www.opengis.net/def/param-name/OGC-OM/2.0/samplingGeometry"/>
<om:value xmlns:ns="http://www.opengis.net/gml/3.2" xsi:type="ns:GeometryPropertyType">
<ns:Point ns:id="Point_sp_45C0E376C40E98E8EC0D48C05F7558C2FFD15245">
<ns:pos srsName="http://www.opengis.net/def/crs/EPSG/0/4326">52.063269625917 4.5077472925186</ns:pos>
</ns:Point>
</om:value>
</om:NamedValue>
</om:parameter>
<om:parameter>
<om:NamedValue>
<om:name xlink:href="source"/>
<om:value xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">sensors_arduino</om:value>
</om:NamedValue>
</om:parameter>
<om:parameter>
<om:NamedValue>
<om:name xlink:href="user"/>
<om:value xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">sID :1000</om:value>
</om:NamedValue>
</om:parameter>
<om:observedProperty xlink:href="PM2.5_AirPollutantValue" xlink:title="PM2.5_AirPollutantValue"/>
<om:featureOfInterest xlink:href="sensors_arduino_1000"/>
<om:result xmlns:ns="http://www.opengis.net/gml/3.2" uom="μg/m3" xsi:type="ns:MeasureType">130.67</om:result>
</om:OM_Observation>A service producing this type of results can be seen here: https://nexos.demo.52north.org/52n-sos-hackair-webapp/service.
5.3. GROW example
In the GROW project the SME Hydrologic has developed a SOS service that uses an O&M observation. In this case, a single number is provided as the result of the observation and additional parameters are transported.
<OM_Observation xmlns="http://www.opengis.net/om/2.0">
<type gml:remoteSchema="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement" />
<phenomenonTime>
<gml:TimePeriod>
<gml:beginPosition>2018-09-03T09:01:38.000Z</gml:beginPosition>
<gml:endPosition>2018-09-03T09:01:38.000Z</gml:endPosition>
</gml:TimePeriod>
</phenomenonTime>
<resultTime>
<gml:TimeInstant>
<gml:timePosition>2018-09-03T09:01:38.000Z</gml:timePosition>
</gml:TimeInstant>
</resultTime>
<procedure>Grow.Thingful.Sensors_je47sfac</procedure>
<observedProperty nilReason="Thingful.Connectors.GROWSensors.AirTemperature" />
<featureOfInterest nilReason="je47sfac" />
<result>20.64</result>
</OM_Observation>5.4. Future work
So far we have seen 3 servers using 2 different approaches to represent the result. That is not a problem for a web service (that only outputs data), but it is not the best situation to ensure interoperability at the client side where an integrated client will need to react to any possible encoding variation and deliver the best result.
5.4.1. How to encode the procedure.
The SWE4CS Discussion Paper suggest that we use an approach to encode the procedure that takes into account a recommendation extracted from section 6.18.1 of the Timeseries Profile of Observations and Measurements standard [OGC 15-042r5] that suggests an encoding for both the observation process and the operator of the sensor (the citizen doing Citizen Science) that is based on ISO metadata. This approach will ensure a uniform way to report on these two important aspects of the observation.
|
Note
|
This approach has not been implemented during the IE but it is considered something we can experiment with in the future. An example of this procedure is provided in the SWE4CS document and reproduced here for convenience. |
<om:procedure>
<tsml:ObservationProcess gml:id="op1">
<!-- processType defines observation performed by human with sensor -->
<tsml:processType
xlink:href="http://www.opengis.net/def/waterml/2.0/processType/Sensor"/>
<!-- processReference defines sampling protocol -->
<tsml:processReference
xlink:href="https://dyfi.cobwebproject.eu/skos/JapaneseKnotweedSamplingProtocol"/>
<!-- if a sensor is used, provide the link to the sensor definition here. Use
SensorML if possible -->
<tsml:parameter>
<om:NamedValue>
<om:name xlink:href="http://www.opengis.net/def/property/OGC/0/SensorType"/>
<om:value>http://www.motorola.com/XT1068</om:value>
</om:NamedValue>
</tsml:parameter>
<!-- operator defines the citizen scientist producing this observation -->
<tsml:operator>
<gmd:CI_ResponsibleParty>
<gmd:individualName>
<gco:CharacterString>Ingo Simonis</gco:CharacterString>
</gmd:individualName>
<gmd:organisationName>
<gco:CharacterString>OGC</gco:CharacterString>
</gmd:organisationName>
<gmd:role>
<gmd:CI_RoleCode
codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml"
codeListValue="resourceProvider"/>
</gmd:role>
</gmd:CI_ResponsibleParty>
</tsml:operator>
</tsml:ObservationProcess>
</om:procedure>The result is quite verbose, which might affect performance when many data is transmitted.
5.4.2. Avoiding verbosity by defining a data stream
An approach based on providing a comma-separated recordset that is described only once at the beginning should be more compact and efficient to parse.
Section 8.4.3 of the [OGC 08-094r1] OGC® SWE Common Data Model Encoding Standard v2.0 describes a way to encode a DataStream only once and then send the data directly as a CSV format using HTTP of other protocol. A similar solution could be worth to be tested in the future to increase performance.
6. SOS architectures
In this chapter, we describe three architectures tested in the IE that demonstrate end-to-end architectures as well as interoperability among servers and clients.
6.1. Architecture 1: SOS services integrated in a SOS client
In this architecture ([MiraMonSOSArchit]), the client access directly two different SOS services. It formulates a GetFeatureOfInterest to determine the positions of the individual observations and a GetObservation each time it needs to show a complete description of a single point (the user triggers this event by clicking on an icon) or if it needs to represent different icons as a function of the value of the observation. In this case, interoperability happens directly in the client. Since the SOS requests are communicated to the Internet, this client is exposing requests and response, allowing people to explore the SOS protocol with both the map browser console as well as the browser developer tools.
It is worth mentioning that this architecture is only possible if both services are declaring their willingness to be combined in the header of the responses. By default, programatically reading XML or JSON data coming from a Internet domain different from the client itself is not allowed except if the server states in the header that this is allowed. This is known as Cross-Origin Resource Sharing (CORS). The following headers will allow CORS with anybody.
Access-Control-Allow-Origin: * Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE
In the case a SOS server does not allow CORS, our client is still able to force a solution by redirecting the request to our server with an extra parameter ServerToRequest. In this case, our server will cascade the request to the specified server and return the response back to the client as if there was only one server involved in only one domain.
6.2. Architecture 2: SOS services integrated in a combined agile service
In this architecture ([GrowSOSArchit]), a common server pulls two or more SOS requests into a central tabular datastore. This datastore records only the information from the returned SOS data that is required for the final visualization and removes information that is redundant, creating a data warehouse representing only one version of the data. In this approach, the interoperability happens internally in the datastore and the SOS requests and responses are not exposed to the final client.
In the diagram below ([GRowDataFlow]), data is stored in the data warehouse and Microsoft’s PowerBI does the heavy lifting for the visualization of the combined data sources.
The common server would typically be a cloud server, but for some clients this is not necessary, in the case of PowerBI, a data bridge is created between the data source and the visualization tool before it is published to a web client.
Other visualization tools (such as Tableau) will have their own methods of connecting to the data warehouse and publishing the results to a web based client.
In this architecture, data is only as up-to-date as the latest data pull from the SOS servers; in the case of GROW, this is done nightly, but this could be made more frequent or moved towards real time using a data log pipeline in a kappa architecture.
6.3. Architecture 3: SOS service for interoperability and JSON API for fast client
This architecture was especially optimized to support the development of lightweight Sensor Web applications. This is achieved by avoiding the direct XML encoding/decoding on the client device. Instead, the interactions between the client (in this case the 52°North Helgoland Sensor Web Viewer) and the server components is achieved via a REST and JSON interface (the 52°North Sensor Web API).
This API can be directly exposed by Sensor Web servers such as the 52°North implementation. Alternatively, an available proxy component is also able to encapsulate existing OGC SOS servers behind the lightweight interface of the 52°North Sensor Web API.
The advantage of this approach is a more lightweight communication pattern to be implemented on the client side. In addition, the 52°North Sensor Web API offers further convenience methods as well as functionalities for reducing the transferred data volume (by generalizing observation data) and improving the data visualization (e.g., providing rending hints). A drawback of this approach is a less direct interaction with SOS servers, so that for integrating new SOS servers, a proxy component has to be configured/adjusted.
7. SOS servers
In this chapter we describe four SOS servers tested in the IE.
7.1. 52 North solution
The 52°North Sensor Web Server comprises several server-side modules which closely interact to provide different kinds data access functionality. In detail, server comprises the following elements.
-
Data storage: The database for storing the observation data is integrated through an object-relational mapping layer based on the Hibernate framework. This allows the flexible integration of different types of database management systems (e.g., PostgreSQL, Oracle, MS SQL Server, MySQL) and data models. For this IE, PostgreSQL was used.
-
For the access to observation data, the server offers three dedicated modules, which use the same common Sensor Web database.
-
SOS: The SOS module offers a comprehensive implementation of the OGC Sensor Observation Service 2.0 standard (including beyond the core several extended functionalities, transactional, and result handling operations). It also offers several interoperability enhancements such as a support of the INSPIRE Technical Guidance on the SOS as a Download Service.
-
SensorThings API: In addition to the SOS support, a dedicated module is available for supporting the OGC SensorThings API Part 1: Sensing (not yet evaluated as part of this IE).
-
52°North Sensor Web API: Complementary to the previous modules, the 52°North Sensor Web API is also offered. This API offers an additional, but optional, convenience layer for building client applications. While both the SOS and the SensorThings API standards are well suited for enabling the interoperable access to observation data, the Sensor Web REST-API allows to provide additional functionality that significantly facilitates the development of client applications. Typical examples of this additional functionality comprise: generalization of observation data (important for developing mobile applications), provision of rendering hints (e.g., styling information for time series), and conversion of data to mainstream formats such as CSV.
-
The URL of the instance used in this IE was: https://nexos.demo.52north.org/52n-sos-hackair-webapp/service.
More information about the initiative can be found here: https://52north.org/software/software-projects/sos/
7.2. istSOS
istSOS (Istituto Scienze della Terra Sensor Observation Service) is an OGC SOS server implementation written in Python. istSOS allows for managing and dispatch observations from monitoring sensors according to the Sensor Observation Service standard.
istSOS evolved over time from being a SOS service provider to complete data management system. But the standard does not account for a number of functionalities that were later included in the software. Some of the extending capabilities are:
-
Handle of irregular time series;
-
On-the-fly aggregation of observed measures with no-data management;
-
Capability to filter observations based on partial observed property names (LIKE filtering support); and
-
Native support for data validation and data quality index associated with each observation.
The project also provides also a Graphical user Interface that allows for easing the daily operations and a RESTFul Web API for automatizing administration procedures.
The URL of the instance used in this IE was: http://artemis.geogr.uni-jena.de/istsos_ie/soil?service=SOS&request=GetCapabilities&version=1.0.0 More information about the initiative can be found here: http://istsos.org/
7.3. Comparison of 52 North SOS and istSOS implementations
52 North and istSOS are both RESTful implementations of OGC SOS standard. We provide a short comparison of these tools, which might help end-users to choose a tool based on their requirements.
52 North and istSOS both provide support for all core operations defined in the OGC SOS specification. 52 North SOS is a Java based implementation whereas istSOS is a Python based implementation. As they are based on different programming languages, their supported hosting application server differs. In order to deploy 52 North SOS, the end-user can either use Tomcat, Jetty, or Glassfish. In order to deploy istSOS, Apache mod_wsgi is required. Both implementations have a restful web interface and support JSON, XML, and plain text for data encoding. 52 North offers bindings like KVP, SOAP, POX, and EXI. istSOS offers bindings like KVP and SOAP. They both run on major OSs like Windows, Mac, and Linux. They both support Postgres/PostGIS as underlying database management system. 52 North implementation also supports database like Oracle, Microsoft SQL Server and MySQL. 52 North implementation provides multilingual support for querying data. istSOS offers automatic notifications via email, Twitter, or other social media. Both SOS implementations provide a user-friendly graphical interface which includes a built-in client, data viewer, and data manager. Both tools come with a detailed documentation with proper examples. They both have a friendly support community which is easily reachable via email and have a supporting emailing list where users can post questions.
7.4. MiraMon SOS Server
MiraMon Server is a stand alone CGI application that runs on Windows operating systems that can be used in combination with a web server such as Internet Information Server or Apache for Windows. It is the ideal solution for people that already uses MiraMon professional on desktop because it uses the same MiraMon formats in the back-end. MiraMon server is based on the same libraries that are used by MiraMon professional and has the same capabilities in terms of CRS support, interpolation algorithms, MMZX compression, etc. One particularity of the software is the internal tiled schema required to serve maps and tiles in a fast an scalable way. MiraMon Server uses OGC web services as a baseline for the interaction to the client. Currently, MiraMon server provides support for the following standards:
-
Web Map Service (all versions)
-
Web Map Tile Service (all versions)
-
Web Coverage Service (version 1.0)
-
Web Feature Service (version 2.0)
-
Web Processing Service (version 1.0)
-
Sensor Observing Service (version 2.0)
The SOS capacity is used in tandem with the Web Feature Service and uses the same MiraMon topologically-structured formats in the back-end. It has been developed in the Ground Truth 2.0 project to serve interoperable data from the Citizen Observatories created during that project. The current implementation is incomplete and only supports GetFeatureOfInterest and GetOBservation operation with limited capabilities. The objective of the minimum capabilities developed was to report the requirements of a viewer client needed to represent a map of the features of interest provided by the service and to allow for a query in a point to get more information about the observations at that point. Each dataset in MiraMon becomes a observedProperty in the SOS service. Each observation is a position in a PNT file that has a DBF record associated that is automatically transformed to a O&M DataRecord. Internally, it is possible to mark field names in the DBF that are associated to concepts in the O&M, such as the phenomenon time and the user name.
Below is the internal format for the small REL5 document necessary to include the extra information that the server requires.
file name: C:\inetpub\SIWeb\gt20\VattenFokus\VattenFokusT.dbf Last update on: 24-01-2019 Number of records: 254 Number of fields per record: 32 Character set: Windows ANSI (88, 0x58) Field characteristics: ------------------------------------------------------------------------------------------- NUM | NAME | DESCRIPTOR | T | SIZ | REL ------------------------------------------------------------------------------------------- 1 | ID_GRAFIC | Identificador Gràfic ID | N | 3 | 2 | USER_ID | User ID | N | 5 | 3 | SAMPLE_ID | Sample ID | N | 5 | 4 | CREA_DATE | Creation Date | C | 16 | 5 | CHAN_DATE | Modification date | C | 16 | 6 | SAMPLEDATE | Sample date | C | 16 | 7 | GROUP_ID | Group ID | C | 34 | 8 | SITE_NAME | Site name | C | 111 | 9 | Sample_date_time | Sample date/time | C | 16 | 10 | N_PARTICIPANT | Total number of participants | N | 3 | 11 | NOTES | Notes | C | 276 | 12 | WATER_TYPE | Freshwater body type | C | 7 | 13 | OTHER_WATER_TYPE | Other freshwater body type | C | 50 | 14 | LAND_USE | Land use in surroundings | C | 17 | 15 | OTHER_LAND_USE | Other land use in surrounding | C | 84 | 16 | BANK_VEGE | Bank vegetation | C | 36 | 17 | OTHER_BANK_VEGE | Other bank vegetation | C | 74 | 18 | ON_WATER | On the water surface | C | 30 | 19 | POPUT_SOURCES | Pollution in surroundings | C | 46 | 20 | WaterUses | Evidence of water uses | C | 33 | 21 | OTHER_WATER_USE | Other evidence of water uses | C | 10 | 22 | AQUATIC_LIVE | Evidence of aquatic life | C | 69 | 23 | OTHER_AQUATIC_LIVE | Other evidence of aq. life | C | 38 | 24 | ALGUE | Algae presence | C | 16 | 25 | WATER_FLOW | Estimated the water flow | C | 7 | 26 | WATER_LEVEL | Estimated water level | C | 7 | 27 | NITRATE | Nitrate | N | 4 | 28 | PHOSPHATE | Phosphate | N | 5 | 29 | TURBIDITY | Water Quality Turbidity | C | 7 | 30 | RESULT | Result | N | 3 | 31 | WATER_COLOR | Estimated water colour | C | 10 | 32 | OTHER_WATER_COLOR | Other estimated water colour | C | 43 |
[VERSIO]
Vers=5
SubVers=0
[GetObservation]
GetObsservation_Vers=5
GetOBservation_SubVers=0
Fitxer=MeetMeeMechelenT.rel
CampDataHoraFenomen=time_last
CampNomSensor=street_namFinal representation as XML O&M of the same structure:
<?xml version="1.0" encoding="ISO-8859-1"?>
<sos:GetObservationResponse xmlns:sos="http://www.opengis.net/sos/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:om="http://www.opengis.net/om/2.0" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:swe="http://www.opengis.net/swe/2.0">
<sos:observationData>
<om:OM_Observation gml:id="vatten-fokus_2_1">
<om:type xlink:href="http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_ComplexObservation"/>
<om:procedure xlink:href="http://www.opengis.uab.cat/vatten-fokus/procedure/22655"/>
<om:observedProperty xlink:href="http://www.opengis.uab.cat/vatten-fokus/observedProperty"/>
<om:featureOfInterest xlink:href="http://www.opengis.uab.cat/vatten-fokus/featureOfInterest/2"/>
<om:result xsi:type="swe:DataRecordPropertyType">
<swe:DataRecord>
<swe:field name="SAMPLE_ID">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/SAMPLE_ID">
<swe:uom/>
<swe:value>45821</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="CREA_DATE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Creation_Date">
<swe:value>07/12/2018 17:23</swe:value>
</swe:Text>
</swe:field>
<swe:field name="CHAN_DATE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Modification_date">
<swe:value>07/12/2018 17:23</swe:value>
</swe:Text>
</swe:field>
<swe:field name="SAMPLEDATE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Sample_date">
<swe:value>07/12/2018 15:00</swe:value>
</swe:Text>
</swe:field>
<swe:field name="GROUP_ID">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Group_ID">
<swe:value>Dunkern, Group ID: 38438</swe:value>
</swe:Text>
</swe:field>
<swe:field name="SITE_NAME">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Site_name">
<swe:value>Dunkershall. V¤gtrumma uppst¤ms.</swe:value>
</swe:Text>
</swe:field>
<swe:field name="Sample_date_time">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Sample_date/time">
<swe:value>07/12/2018 15:00</swe:value>
</swe:Text>
</swe:field>
<swe:field name="N_PARTICIPANT">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/N_PARTICIPANT">
<swe:uom/>
<swe:value>1</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="NOTES">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Notes">
<swe:value>+2 grader C.</swe:value>
</swe:Text>
</swe:field>
<swe:field name="WATER_TYPE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Freshwater_body_type">
<swe:value>Other</swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_WATER_TYPE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_freshwater_body_type">
<swe:value>Dike</swe:value>
</swe:Text>
</swe:field>
<swe:field name="LAND_USE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Land_use_in_the_immediate_surroundings">
<swe:value>Agriculture</swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_LAND_USE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_the_land_use_in_the_immediate_surroundings">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="BANK_VEGE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Bank_vegetation">
<swe:value>Grass</swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_BANK_VEGE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_bank_vegetation">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="ON_WATER">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/On_the_water_surface">
<swe:value>None</swe:value>
</swe:Text>
</swe:field>
<swe:field name="POPUT_SOURCES">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Pollution_sources_in_the_immediate_surroundings">
<swe:value>Other</swe:value>
</swe:Text>
</swe:field>
<swe:field name="WaterUses">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Evidence_of_water_uses">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_WATER_USE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_evidence_of_water_uses">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="AQUATIC_LIVE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Evidence_of_aquatic_life">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_AQUATIC_LIVE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_evidence_of_aquatic_life">
<swe:value></swe:value>
</swe:Text>
</swe:field>
<swe:field name="ALGUE">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Algae_presence">
<swe:value>No algae</swe:value>
</swe:Text>
</swe:field>
<swe:field name="WATER_FLOW">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Estimated_the_water_flow">
<swe:value>Surging</swe:value>
</swe:Text>
</swe:field>
<swe:field name="WATER_LEVEL">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Estimated_water_level">
<swe:value>Average</swe:value>
</swe:Text>
</swe:field>
<swe:field name="NITRATE">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/NITRATE">
<swe:uom/>
<swe:value>1.50</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="PHOSPHATE">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/PHOSPHATE">
<swe:uom/>
<swe:value>0.075</swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="TURBIDITY">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Water_Quality_Secchi_Tube_(Turbidity).">
<swe:value><14</swe:value>
</swe:Text>
</swe:field>
<swe:field name="RESULT">
<swe:Quantity definition="http://www.opengis.uab.cat/vatten-fokus/variable/RESULT">
<swe:uom/>
<swe:value></swe:value>
</swe:Quantity>
</swe:field>
<swe:field name="WATER_COLOR">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Estimated_water_colour">
<swe:value>Colourless</swe:value>
</swe:Text>
</swe:field>
<swe:field name="OTHER_WATER_COLOR">
<swe:Text definition="http://www.opengis.uab.cat/vatten-fokus/field/Other_estimated_water_colour">
<swe:value></swe:value>
</swe:Text>
</swe:field>
</swe:DataRecord>
</om:result>
</om:OM_Observation>
</sos:observationData>
...
</sos:GetObservationResponse>7.5. GROW SOS server implementation
The GROW SOS service is tightly integrated into the GROW platform based around Hydrologic’s existing Hydronet 4 platform. The SOS 2.0 service runs concurrently with the GROW standards API in the HydroNET GROW Server.
The service implements two .net packages that disseminate GROW data to the SOS 2.0 standard within the GROW instance of HydroNET 4 Server. A SOS package covers the mapping of SOS 2.0 requests to GROW requests and the mapping of GROW data structures to SOS 2.0 standards. A second package Ogc.Wrapper.Entities contains the SOS 2.0 entity definitions.
The Base URL of the GROW SOS 2.0 service: http://grow-beta-api.hydronet.com/api/service/sos
SOS 2.0 knows four core operations: GetCapabilities, DescribeSensor, GetObservation, and GetFeatureOfInterest.
7.5.1. GetCapabilities
This operation lists all available metadata in the service and provides a detailed list of all other operations that are provided in the service itself. It provides the information you need to execute other operations within the SOS 2.0 effectively.
7.5.2. GetObservation
This operation gives the client access to observation data from sensors. What data is returned is dependent on the parameters you give as a client.
The GetObservation operation requires the following parameters.
-
Procedure: The identifier of the sensor. The procedure can be found in the GetCapabilities response.
-
ObservedProperty: The parameter that you want to query data for. A sensor can provide data for multiple parameters (e.g., soil moisture, temperature). Which ObservedProperies are available for this sensor can be found in the GetCapabilities response or the DescribeSensor response of the relevant sensor.
-
TemporalFilter: The timespan for which you want to query data. Datetimes follow the ISO 8601 standard. The full timespan of data that the sensor provides can be found in the GetCapabilities or DescribeSensor (for the relevant sensor) response.
7.5.3. GetFeatureOfInterest
This operation provides information about features of interest (name, description, coordinates, etc.) of a sensor or an observation.
The GetFeatureOfInterest operation requires the following parameter:
-
procedure: The identifier of the sensor. The procedure can be found in the GetCapabilities response.
The SOS 2.0 service in GROW provides two response types: XML and JSON. The client can provide ResponseFormat in the parameters to define which response is desired. If no ResponseFormat is given, the service returns XML by default.
8. SOS clients
In this chapter we describe three SOS integrated clients tested in the IE.
8.1. Grow Client
The GROW client is a demonstrator application showing the how data sources can be combined in a tabular data warehouse and off-the-shelf visualization tools can be used to create rich visualizations. In the case of this demonstrator, Microsoft’s PowerBi was used, although commercial tools such as Tableau or Spotfire can be used. In addition, free tools such as Grafana, Rawgraphs, or Apache Superset can be employed.
The diagram below shows a typical output using PowerBi, showing GROW sensor locations combined with Airhack sensor locations.
In this application, data is pulled from two SOS sources (GROW and AirHack) via a python script, usually overnight. This data is stored in a tabular data warehouse, in this prototype this is just flat files, however a full scale data warehouse such as Microsoft’s Analysis services or a tabular database such as Cassandra could be used.
Data from the warehouse is pulled nightly into Microsoft’s cloud and then published to a web page that makes heavy use of JavaScript to provide an rich exporable interface.
This approach can be extended to mix different types of data into one visualization. In the figure below, gridded data (locations of sensors) is combined with time series data so that the user can explore the data more fully. In this application, when a user click on a sensor, the time series data from that sensor is displayed to the user. In this case, the tool takes the time series data and displays the average, minimum, and maximum of the time series data.
8.2. MiraMon Client
The MiraMon map Browser is a long-term developing effort to create a visualization, analysis, and download tool that runs in modern web browsers. Based on HTML5 and JavaScript, it uses OGC web service protocols to connect to web services and show the information to the user. The objective of the development of this tool is to assign to web browser and the JavaScript engine as much work as possible, limiting interactions to the server to the minimum possible and the transfer of information to a format that is as raw as possible. This approach can be surprising: these days are many application which prefer to perform processing functionalities in the cloud and not by the client machine. Most of the time, the MiraMon map Browser is directly responsible for creating the visualization on-the-fly based on the raw data, allowing the user to change visualization properties, perform analysis, statistics, or build time series in the client side directly. In the Data quality estimations on the client side section, we will show how the same principle can be used to compute overall data set quality can be computed on-the-fly, as well.
Below are the main functionalities and standards used to achieve that functionality.
-
Raster visualization and query by location is possible using OGC Web Map Service and Web Map Tile Service.
-
Raster analytics is possible by using the OGC Web Map Service in an special way that transmits binary arrays of values (raw or RLE encoded) instead of pictorial representations. Pixel based analysis is performed directly in JavaScript in the client side and visualized on-the-fly.
-
Vector visualization and query by location is possible using OGC Web Feature Service and Sensor Observation Service. The client accepts both XML an JSON formats.
-
Data download is limited to the use of Web Coverage Service v.1.0.
Several layers of data coming from different servers and using different protocols can be overlaid simultaneously. Some layers represent data from a single dataset while other can be a virtual datasets computed on-the-fly for each zoom and pan and created by combining data from more than one dataset and server.
Both WFS and SOS visualization are currently limited to points that are represented in a HTML5 canvas as icons, circles, and texts and could be easily extended to lines and polygons in the near future. These functionalities were particularly useful to show occasional observations made by citizens in different places. We were able to show together Ground Truth 2.0 observations with HackAir observations using the SOS protocol directly in XML and in JSON. Some of the visualization functionalities were actually improved during the IE, such as the capability to condition the color of a circle by an attribute (or an observed result) of the feature. This way, it was possible to represent the level of concentration of a pollutant as a colored circle using a color code that was represented in the legend.
Representing the positions of the observations will require only the GetFeatureOfInterest operation unless the visualization of the feature should depend on the result of the observation. In the later case, a GetObservation is performed and all the information on the features is loaded in the client, making query by location non-dependent on the server.