OGC Testbed-16: Earth Observation Application Packages with Jupyter Notebooks

The first scenario explored the handling of Jupyter Notebooks and the interaction with data through the Data Access and Processing API (DAPA). The scenario was refined as follows:

The second scenario explored interactive and batch mode execution on hosted Jupyter. This scenario is described as follows:

The third scenario explored:

The architecture of a Jupyter Notebook consists of a server rendering a Notebook document and an interacting programming language kernel:

Jupyter kernels are the programming-language specific processes that run the Notebook cells code. IPython is the reference Jupyter kernel, providing a powerful environment for interactive computing in Python. A large set of kernels for other languages are available.

With administrator rights, kernels can be added to the Jupyter host using python or conda. Conda is an open-source cross-platform package and environment manager system for any language (e.g. Python, R, Java). The conda command easily creates, saves, loads and switches between environments.

The following commands create a conda environment, install a Python2 kernel in the environment, then register the conda environment as a kernel in Jupyter:

A Notebook might depend on specific software (e.g. GDAL, Orfeo Toolboox, geopandas) used from the Jupyter kernel. Depending on the configured permissions, those dependencies can be installed either from the Jupyter terminal, or using system commands directly from the Notebook (enabling others users to repeat the installation commands). The environment is persisted in the user environment of the Jupyter host.

The usual means for installing dependencies from a Notebook are through pip commands (python libraries) and conda (virtual environment).

Conda environments can be registered as a Jupyter kernel. The environment is specified in a configuration file (see example below) listing a set of channels (repositories hosting the packages - potentially custom software -) and dependencies.

To create an environment from an environment.yml file, the following command should be used:

The kernel maintains the state of a Notebook’s computations. When restarting the kernel, the Notebook reset its states and therefore looses all results of Notebook cells. However, the dependencies installed using system commands generally remains in the user environment depending on the specific implementation.

From any Jupyter session, it is possible to start a terminal and execute commands in the kernel. In particular, git commands can be used to pull a repository and push changes.

As illustrated below, from the terminal, the Notebook documents and the environment can be saved and retrieved from Git repositories (1). Then the environment can be provisioned using pip or conda (2).

In order to push back changes on the Notebook, the user can execute a git command using JupyterLab terminal, and potentially push the change to a user private repository. Therefore, as illustrated below, multiple users might collaborate on a shared Notebook repository.

Security is a big challenge with sharing Notebooks. The potential for threat actors to attempt to exploit the Notebook for nefarious purposes is increasing with the growing popularity of Jupyter Notebook.

From the users' perspective, the Notebook server shall prevent untrusted code to be executed on user’s behalf when the Notebook is opened. In particular, the security model of Jupyter is based on a trust attribute (a signature computed from a digest of the Notebook’s contents plus a secret key) that prevents executing any untrusted code and sanitize the HTML source code. The Notebook’s trust is updated when the user explicitly decides to execute a specific cell.

From the platform point of view, the security challenge implies offering the interactivity of a Notebook without allowing arbitrary code execution on the system by the end-user. In particular, restricting the access to the Notebook server is important. As the Notebook user might execute local system commands, the vulnerability of the system should be carefully reviewed - in particular regarding all the write permissions granted on the various offered resources.

In its simplest form, a Jupyter Notebook is simply executed by a kernel managed by a Jupyter server such as JupyterLab. JupyterLab provides the familiar building blocks of the classic Jupyter Notebook (Notebook, terminal, text editor, file browser, rich outputs, etc.) in a more flexible user interface and enabling plugin extensions.

The JupyterLab interface consists of a main work area containing tabs of documents and activities, a collapsible left sidebar, and a menu bar. The left sidebar contains a file browser, the list of running kernels and terminals, the command palette, the Notebook cell tools inspector, and the tabs list.

Sharing a Notebook server such as JupyterLab would make the concurrent users' commands collide and overwrite each other as the Notebook has exactly one interactive session connected to a kernel. In order to provision instances of a Jupyter Notebook and share the Notebook, a hub serving multiple users is required to spawn, manage, and proxy multiple instances of the single-user Jupyter Notebook server.

JupyterHub allows creating Jupyter single-user servers that guarantee the isolation of the user Notebook servers. The Notebook is copied to ensure the original deployed Notebook is preserved. JupyterHub includes JupyterLab and thus also supports the provisioning of the environment using Conda.

Although nothing prevents a side component managing the sharing of Notebook documents, this support goes beyond the provided capabilities. Indeed, only extensions of JupyterHub provides Notebook sharing support.

Full reproducibility requires the possibility of recreating the system that was originally used to generate the results. This can, to a large extent, be accomplished by using Conda to make a project environment with specific versions of the packages that are needed in the project. The limitations of Conda are reached when complex software installation is required (e.g. a specific configuration of a tool).

For the support of sophisticated systems, repo2Docker can be used to build a reproducible container that can be executed anywhere by a Docker engine. repo2Docker can build a computational environment for any repository that follows the Reproducible Execution Environment Specification (https://repo2Docker.readthedocs.io/en/latest/specification.html#specification).

As illustrated on the figure below, for building a Jupyter Notebook container, repo2Docker must be used from a repository that consists of the Notebook resources and a Dockerfile that builds the environment and install the Jupyter kernel.

Assuming that the Jupyter kernel is installed in the container, the JupyterLab interface can be launched from within a user #session without additional configuration by simply appending /lab to the end of the URL like so:

Note that, the DockerSpawner (https://jupyterhub-Dockerspawner.readthedocs.io/en/latest/) from JupyterHub allows spawning Notebooks provisioned from a Docker image.

BinderHub is an open-source tool that enables deploying a custom computing environment described on a Git repository and make it available (from an associated URL) by many remote users. A common use of Binder is the sharing of Jupyter Notebooks.

As illustrated on figure below, BinderHub combines the tools described above to generate the Notebook: Repo2Docker generates a Docker image from the Git repository (potentially setting a conda environment), and the JupyterLab container is provisioned online based on JupyterHub running on Kubernetes.

The Binder project also provides mybinder.org, a pre-existing and free to use BinderHub deployment used during this project. However, only a private BinderHub deployment might support accessing private Git repositories.

In order to share a Notebook developed locally with other developers, the Git repository can simply be linked with Binder for provisioning an online Notebook. On mybinder.org, the repository is turned to a Notebook by filling the form shown below.

Binder is meant for interactive and ephemeral interactive coding. This means that the code is ideally suited for relatively short sessions that will automatically shut down after 10 minutes of inactivity (default configuration). The creation of a Binder session might take a long time as the following chain of events happens when a user clicks a Binder link:

A set of Jupyter Notebook hosting solutions are available such as Microsoft Azure Notebooks, Amazon EMR Notebook, and OnlineNotebook.net. The online Jupyter hosting solutions enable Notebook collaboration and support considerable capability to extend the environment with additional libraries (e.g. pip, conda) but is, however, limited without the Dockerfile support. Moreover, free solutions are of course restricted by the availability of hardware resources.

In particular, Google Colaboratory (or "Colab") supports the import of libraries using pip and apt-get, the synchronization with Git repositories, and the access control management. Note also that Colab introduces new concepts and deviations from the original Jupyter interface.

Cocalc stands out from other hosting solutions with the support of real-time collaboration. However, Dockerfiles are also not supported.

nbviewer is a web application that lets you enter the URL of a Jupyter Notebook file, renders that Notebook as a static HTML web page, and gives you a stable link to that page which you can share with others. nbviewer also supports browsing collections of Notebooks (e.g., in a GitHub repository) and rendering Notebooks in other formats (e.g., slides, scripts).

In contrast to JupyterLab, nbviewer does not execute Notebooks. It only renders in a web page the inputs and outputs saved in a Notebook document.

Papermill was considered for enabling the execution of a Notebook in batch mode. An alternative solution based on nbconvert and Python dictionaries was also investigated.

Notebooks can be deployed and executed by the ADES component using the Docker Application Package which is basically a Docker Image referenced in an Application Descriptor (e.g. typically based on CWL).

Independent of the command-line approaches described above, repo2Docker can be used for building the container embedding the Notebook application. Defining the container built from either a Conda environment or a Dockerfile offers enough flexibility for encapsulating any tool or library including papermill or the nbconvert application.

In particular, using the papermill approach, the following steps are required to prepare the Docker Application Package:

On the other side, using the nbconvert approach, the following steps are required to prepare the Docker Application Package:

As mentioned above, a dedicated tool can be implemented to generate the command-line application and the Application Descriptor both from the Python dictionary. Note also that the possibility of combining a dictionary definition with the papermill approach was not investigated but would be feasible.

Since the ADES application package is based on containers, the interoperability for deploying the application is ensured whatever command-line approach is used (i.e. papermill or nbconvert).

However, in addition to the deployment packaging format, the execution concerns on the target platforms should be considered on different levels:

The coupling of the Notebook libraries with the platform offered services is illustrated on figure below.

As proposed in the Master System Design Document (MSDD) of the EO Exploitation Platform Common Architecture, the Platform might declare its capabilities (e.g. support APIs, file formats, etc.) in a well-known 'bootstrap' URL. Additionally, the Application Descriptor may be enhanced for including all capabilities requirements, thus enabling the platform to detect conflicting deployments.

As in the OGC EO Apps Pilot project, support for a manifest with inputs and outputs based on the STAC specification was provided.

The SpatioTemporal Asset Catalog (STAC) specification provides a common language to describe a range of geospatial information and standardizes the way geospatial assets are exposed online and discovered. A SpatioTemporal Asset is a document representing information about the earth captured in a certain space and time including imagery (satellite, aerial, drone, etc.), derived data products and alternative geospatial captures. However, compared to a classic catalogue such as OpenSearch, STAC does not support specifying any search operation.

STAC is actually composed of three component specifications that together make up the core STAC specification but can be used independently in other contexts:

STAC defines only a minimal core and is designed for extensions. Extensions define the domain-specific fields that can be easily added to any STAC Item. This approach is reflected by the already available extensions such as: Data Cube, Electro-Optical, Project, SAR, Satellite, Tiled Assets, etc.

STAC is used as a solution to handle the discrepancy between platforms related to the format of EO products. For example, the Sentinel-2 SAFE format, illustrated on figure below, might be provided in a compressed archive on some platforms, with a different structure, or even a different format. Moreover, STAC can be used to provide additional information (spatial footprint, masks, bands, orbit direction, etc.) about both submitted inputs and generated outputs.

In summary, STAC facilitates the data stage-in and stage-out by conveying useful information between ADES and the Applications. The downside is the need to support the STAC profile in the application.

Several EO platforms already offer a Jupyter Notebook environment with built-in tools and API for operating the supported service. Some examples are Mundi (DIAS), CREODIAS, Open Data Cube, and Sentinel-Hub.

Like the platforms mentioned above, a Jupyter Notebook environment can act as a platform frontend and be provisioned with a set of tools and libraries.

As shown on figure below, the environment provides some tools and a set of Python API libraries for interacting with the platform services such as the catalogue, data access and processing services.

Such a Notebook environment is a user-friendly place to easily develop applications close to the data. User-friendly Python libraries (or any other support kernel language) can be easily generated from the recent OGC APIs expressed in OpenAPI (such as the draft OGC Processes API or the draft OGC Data Access and Processing API). Indeed, the Swagger technologies enable automatically creating client libraries in a wide set of languages.

The libraries may facilitate a very intuitive composition of operations. In particular, a very simple data access call can be passed as a variable to a processing function, and thus build in few lines a complex workflow. The example below illustrates how a Jupyter Notebook (with a Java kernel) can perform the chaining sequence of a two-processes execution based on the client library implemented during OGC Testbed-14.

As a possible solution, we recommend a careful attention to OpenEO, an open application programming interface (API) that connects clients in languages such as R, Python and JavaScript to Earth observation cloud back-ends in a simple and unified way. Many backend implementations are already provided (e.g. OGC WCPS).

The following scenario reflects the collaboration activities on Notebooks that are considered for extending the EO XP Platform with Jupyter support:

As mentioned earlier, Jupyter Notebook would offer an ideal development environment for building applications that are executed on the ADES.

Developing the Jupyter-based application within the standard environment of the platform removes the need to install such environment. Moreover, the platforms can offer a set of predefined tools and include the relevant client libraries for accessing the services.

Finally, the platform might automate the creation of the Application Package by implementing the following sequence illustrated below:

From the many implementation approaches, no single solution appears to be obvious compared to others. However, BinderHub and JupyterHub provide valuable starting blocks for managing the Jupyter environment.

BinderHub enables defining containers but also supports alternatives. In particular, if only a Conda environment is defined in the repository, then the standard EO XP Platform container is provisioned. However, the Jupyter servers provisioned by BinderHub might be considered as stateless as the user loses any changes each time the session expires.

Unlike BinderHub which relies on repositories, JupyterHub based platforms (such as Mundi DIAS) handle Jupyter sessions that enable the user to manage files in a private user workspace. This aspect should be considered when designing a solution to develop and share Notebook on a EO platform.

The mapping of water masks in flooding situations (e.g. for river gauges affected by severe weather events) is a valuable application of Sentinel-1 SAR data. Emergency response organizations are highly interested in the intensity and duration of floods so as to address flood-related damage events. In addition, the flooding history needs to be documented so that detailed information about the occurrence, frequency, and duration of flooding events for the affected areas is available. This information can be used, for example, to validate flood risk assessments and planning.

The overall process is divided into three individual steps, all represented by a dedicated Notebook:

The chaining of the Notebooks was established using the following technologies:

The CWL definition that was developed for the orchestration of the three Notebooks is:

The ADES implementations by the other participants support the execution of CWL definitions. The integration therefore is straightforward. In addition to the above presented CWL definition, each Notebook was accompanied with an individual CWL definition, making the Notebook individually executable.

An example CWL for Notebook #1 is:

Note that for demo purposes, required API credentials have been defined as inputs for the Notebook and CWL. In a production environment, the application platform APIs (e.g. S3 buckets for accessing data) shall be available without explicit authentication required.

The Testbed-16 scenarios selected by Terradue explored the packaging of Jupyter Notebooks following the recommendations of previous OGC Testbeds proposing. This was to provide a straightforward workflow from Jupyter Notebooks development to their deployment as a service in an Exploitation Platform supporting the ADES/EMS approach.

These scenarios demonstrate how a Jupyter Notebook can be used to discover, access and process (locally or remotely) Earth Observation products following three preliminary scenarios with specific steps and exploring the corresponding elements (e.g. Python Packages, OGC Services, Earth Observation data).

The platform environment must provide developers with a well-defined set of operational processes and procedures to allow a single robust packaging of applications and allow their testing, validation and deployment in production in different execution scenarios. This section covers the different execution scenarios and environments for the use cases defined.

For the validation and deployment of the different execution scenarios, the starting elements are the assets created by Alice in Scenario 2, Step 2:

The contents of the software repositories include the elements listed and described in the table below:

Alice create a Notebook that requests a DAPA service (https://t16.ldproxy.net/ghcnd/), serving station-based measurements from land-based stations worldwide. This Notebooks takes Area of Interest (AoI) and Time of Interest (ToI) input parameters and return a JSON file containing the matching stations.

The first cell of the Notebook contains the variables AoI and ToI to fill. The cell has a metadata tag "parameters" that enables papermill batch execution.

The next cellules performs a search query on the DAPA service and builds the JSON result file with the matching stations.

Alice must produce the following items :

Alice builds her Notebook and installs all the dependencies needed using Conda.

Once the Notebook is ready to be shared, she exports all the installed dependencies by using:

In order to execute the Jupyter Notebook in batch mode, a generic method using papermill, CWL and a bash script that need to be incorporated into the Docker image was developed.

The following generic bash script run.sh transforms the input parameters into a papermill command line :

This script transforms the inputs sent to the script from a command line, into a YAML file used by papermill to override the input parameters in the Jupyter Notebook and then call papermill to execute the Notebook in batch mode.

The supported parameters are limited to literal inputs : Numeric, character string, files and URL. The single parameters are prefixed with "-S" and the multiple parameters by "-M"

The CWL file calls this script within the Docker image and must map the parameters using the same prefix:

Finally the WPS process description specifies the inputs and outputs, with their types and cardinality. The process description also specifies the CWL location and docker image. The ADES needs this information in order to pull and execute the Jupyter Notebook.

NB: This json file will be later referred to as dapa-station-wps-description.json.

Alice then hosts those files on a GitHub repository with the following structure:

NB: The CWL does not need to be in this repository. The CWL can be hosted anywhere.

Alice needs now to generate a Docker container and host it on a Docker repository.

To do this she can use jupyter-repo2docker. This tool creates a Docker image from a GitHub repository.

Eric wants to retrieve Alice’s Notebook for modification and local execution.

Bob want to deploy the dapa-stations application package on an XP platform.

During previous OGC Testbeds, process workflows where orchestrated and executed through various workflow languages such as CWL/BPMN. In TB-16 Geomatys experimented with the orchestration of a workflow directly inside a Jupyter Notebook.

We built two Notebooks:

This two Jupyter Notebooks are packaged with the method described above and deployed to an EMS/ADES platform. Geomatys then built a third Notebook taking sentinel-2/Deimos/PROBA-V data as input, called the first NDVIMultiSensor process through the ADES WPS API, retrieved the results, and then called the second NDVIStacker process with the previous results. The workflow chaining the two process is achieved within the Notebook.

To perform WPS call, Geomatys wrote a single simple Python cell that sent an execute request, looped on the getStatus method, and finally called the getResult method.

This workflow Notebook can be packaged and deployed as any other Juptyer Notebook, making it available in batch mode for anyone.