OGC Testbed-14: Federated Clouds Engineering Report

The European Open Science Cloud (EOSC), and specifically the EOSC-hub project [8], has the charter to use the European Grid Infrastructure (EGI) [9], the EUDAT Collaborative Data Infrastructure (EUDAT CDI) [10], and the INDIGO-DataCloud [11] to jointly offer data, services, and software for research across the European Union (EU). EOSC-hub will be a federated integration of these systems for EOSC, as a whole. Identifying how this integration can be done and achieving it will be a challenge.

The Enabling Grid for E-sciencE (EGEE) project received three rounds of funding from the European Commission (EC) before the creation of the EGI Foundation in 2010 to operate EGI in a sustainable manner. At its beginning EGI supported a number of grid computing software stacks to enable distributed e-science. However, as cloud computing and the on-demand provisioning of resources became an established business model, EGI has increasingly provided a cloud e-infrastructure. As EGI was already a collaboration of different resource providers, it was clear from early on that an EGI Federated Cloud was needed.

Rather than requiring all EGI cloud providers to run the same cloud software stack, EGI defined a Cloud Management Stack (CMS). CMS provides standardized interfaces to common cloud infrastructure operations which the cloud providers must support. The Open Cloud Computing Interface (OCCI) [12] is used for Virtual Machine (VM) management, and the Cloud Data Management Interface (CDMI) [13] is used for object storage. Additional services are available for identity federation, accounting, information discovery, monitoring, etc. Identity federation is done by a Virtual Organization Management Service (VOMS) that operates as an Attribute Authority. The VOMS can issue augmented X.509 proxy certificates that are signed with the user’s original certificate. Different Virtual Organizations can be set-up to support different research communities. Each Virtual Organization (VO), however, is expected to support a core set of VO services, e.g., a VOMS service for membership and authorization, a MyProxy service for managing proxy certs, and a Dashboard service, among others. While the EGI Federated Cloud primarily provides IaaS, other PaaS and SaaS-level tools are also available.

In contrast to EGI, EUDAT [10] has always focused on data management and access, and is technically not cloud-based. (It does not offer on-demand provisioning of any resources.) EUDAT provides a set of services whereby sites and users can make large data sets more discoverable, accessible, and manageable. Sites and research organizations can either use or join EUDAT. EUDAT users have a core set of services available:

Organizations can join the EUDAT CDI by installing and operating a core software stack. A key part of this stack is iRODS [14], a rule-based data management tool. Different iRODS installations can peer with one another and exchange data based on defined policies. Deploying the EUDAT CDI requires making three principal federation connections with other CDI installations:

EUDAT users are authenticated through B2ACCESS which can ingest a variety of credentials. B2ACCESS maintains different service groups of which users can be a member. Attributes are also maintained which can be granted to users. Through the attribute namespace, different attributes can be associated with specific EUDAT services. For example, if a user has an "eudat:b2stage:role=admin" attribute, they have administrative rights for the b2stage service.

While there is a B2ACCESS administration portal, the B2ACCESS service itself is based on Unity ID Management [15]. Unity ID is a logically centralized ID management site where users have a Unity ID. That ID is associated with an organization and a project within that organization. Unity organizations have a fixed set of roles: owner, manager, user and guest.

In contrast to EUDAT, the goal of the INDIGO-Datacloud (INtegrating Distributed data Infrastructures for Global ExplOitation) was to provide a Platform-as-a-Service layer for computing and data analysis. Based on an apparently known set of resource providers, TOSCA (Topology and Orchestration Specification for Cloud Applications) [16] specifications can be used to instantiate a set of computing resources across different OpenStack and OpenNebula providers. This can include the replication of data and the scheduling of data analytic containers in proximity to the data. All of these types of services can use the INDIGO Identity and Access Management (IAM) service. IAM provides an identity hub integrated with a Token Translation Service (TTS). The INDIGO Login Service can take SAML, X.509 (and OpenID Connect) credential information, and provide this on standard OpenID-Connect flows and endpoints. The TTS can convert this information into service-specific credentials for external services, such as Amazon’s S3. This kind of data cloud PaaS layer can be used to support many different types of Science Gateways.

Usefully combining these systems will require extracting the best features of each of them and integrating them through a fundamental model that naturally supports the needed and desired integration. At a very top-level, we could say that the desired integration is to provide any number of federated environments that are managed separately but can share resources. Each of these federated environments could be considered a science gateway, much like the model for the INDIGO-Datacloud. Members of such environments will have access to data and computing resources appropriate for that science gateway community. Each of these environments could be deployed with access to a set of data management functions, much like EUDAT. Access to iRODS services for different purposes could be managed as a part of each environment. These services could be considered a platform and deployed on-demand. The allocation of compute resources to run the platform services and the access to the various data sets (owned by different owners) could all be managed as part of a virtual administrative domain. Actually achieving this kind of integration will certainly not be trivial, but the NIST federation model appears to be to support this goal. It is also general and extensible enough allow additional environments, platforms data sets, policies, governance, etc.

InCommon [17] is a federation-based collaboration systems that is run as part of Internet2 [18]. The core capability of InCommon is maintaining a set of Identity Providers and Service Providers across all institutions that are using InCommon to facilitate federation-based collaboration. These IdPs and SPs are maintained in what is simply called the metadata file. The metadata file is a flat, eXtensible Markup Language (XML) listing of Entity_Descriptors, where an Entity_Descriptor is either an IdP or SP from any institution. (There are currently >6000 metadata entities.)

Several versions of the metadata file (called aggregates) are maintained for operational purposes. These include the preview, main and fallback aggregates. Clearly the main aggregate is considered the current operational metadata. The preview aggregate allows institutions to test new IdPs and SPs, while the fallback aggregate allows institutions to avoid using the main aggregate if that is causing any issues. A fourth aggregate is also available that is IdP-only.

Participating institutions can download the metadata file at any time. It is recommended that the file be downloaded once a day. The metadata files are not encrypted, but are all signed using the same metadata signing key and the SHA-256 digest algorithm. When downloading a metadata file, an institution must obtain an authentic copy of the InCommon Metadata Signing Certificate, whereby the signature on any downloaded aggregate can be verified.

InCommon maintains a strict process for adding new IdPs or SPs to the metadata. Submissions from institutions are vetted by a Registration Authority. This is done usually within one business day, but could take longer. InCommon also provides a set of traditional "human organizational" services, e.g., for resolving disputes between participating institutions regarding the operation or use of an IdP or SP.

By enabling institutions to share a flat file of IdP and SP information that is trusted, InCommon provides a valuable service that enables these institutions to collaborate in any way they choose. However, the use of a single metadata file where all participants can see and use all IdPs and SPs cannot provide key governance capabilities.

InCommon provides no way for participants to define any kind of discovery policies for the IdPs and SPs. There is no way for participants to share their endpoints (IdPs and SPs) with just a select subset of participants. There is no way for current federation participants to vet new members that will be allowed to discover and use the federation’s resources. Essentially providing one centralized repository of metadata that must be periodically replicated also means that the responsiveness to change is quite slow, e.g., one business day or so.

While participating institutions may provide IdPs that they trust and will issue credentials to other institutions, there is no inherent commonality or uniformity of the identity credentials and attributes that are issued. That is to say, there is no inherent commonality or uniformity in how resource access decisions are made for institutions that wish to share similar resources for common purposes and goals.

Having a specific set of shared information for each specific federation enables all these governance issues to be addressed. Having a known set of identity credentials and attributes enables a known set of discovery and access policies to defined and agreed upon by the current federation partners. This directly facilitates the participant’s common goals for the federation.

While InCommon does not directly provide such capabilities, there is nothing to prevent the participants from "rolling their own". This is precisely what has happened. An ad hoc ecosystem of discovery services, etc., have been built and deployed by individual institutions to provide some of the desired governance capabilities. While this represents a valuable data point, they are not built to any standards and may or may not be generalizable to other application domains.

InCommon is also part of a global interfederation service called eduGain [19] that is operated by the GÉANT network project in the EU. eduGAIN works by consolidating all of the national metadata registries into an international metadata registry. While it can be said that eduGAIN represents a truly global interfederation, it nonetheless has the same, basic business model — and drawbacks — as InCommon. It is noted that eduGAIN does maintain additional metadata whereby IdPs and SPs can be filtered according to their home federation.

eduRoam [20] is another service operated by Internet2. eduRoam essentially operates a specific type of federated identity management for one specific type of resource: wifi access at academic institutions around the world. As such, it is an important example of a limited type of federation for a very narrow purpose.

Each individual academic institution can manage wifi access for their students, faculty and staff using established, centralized authentication and authorization mechanisms. The goal of eduRoam is to enable users from one academic institution to travel to another eduRoam-enabled institution and get immediate wifi access. This is done by using a tree of RADIUS servers [21] that connects all eduRoam-enabled institutions. When a user from a remote institution connects a device to the local eduRoam service, the user is prompted for their home institution. A secure TLS session is then initiated with the home institution’s wifi IdP over the tree of RADIUS servers. During this TLS session, the remote user authenticates to their home IdP. If this authentication home IdP is successful, the local institution grants local wifi access. It is noted that while the local institution sees the result of the authentication, at no time does the local institution see the remote user’s identity credentials. These are handled directly between the user and the home IdP over the TLS session.

The eduRoam business model implies trust relationships between all institutions for the sole purpose of wifi access. Each institution must trust the IdPs at remote institutions to authenticate users. Since exactly one type of resource is being shared in the eduRoam use case, there is no need for a resource catalog or discovery policies. Also, since one type of resource is being managed, there is no need to have a richer set of authorization attributes whereby a richer set of access policies can be defined. While the tree of RADIUS servers can generally follow geospatial boundaries (e.g., nations and continents), eduRoam does not have a general mechanism whereby specific subsets of institutions can agree to grant wifi access among themselves, but not others. Nonetheless, eduRoam does represent one fundamental approach to the federated identity issue.

CILogon [22] is one of those services that live within the InCommon ecosystem. CILogon enables InCommon users to obtain federated X.509 certificates based on their existing identifies at their home institutions. These certificates can then be used to access services across a much wider environment of research services at different institutions. CILogon currently supports 88 IdPs across different institutions. Initially when authenticating, a user would have to pick from a drop-down list of 88 IdPs to identify their desired IdP. This was avoided by using a text box supporting incremental search, along with cookies to remember prior selections.

The CILogon web front-end uses SAML and OpenID user authentication, whereby a standard PKCS12 file is downloaded containing their credentials. This certificate can be downloaded completely outside the web browser by using the SAML Enhanced Client or Proxy (ECP) profile. CILogon operates three different Certification Authorities (CAs) that each support a different Level of Assurance (LOA) concerning identity. These LOAs are termed Silver, Basic and OpenID. (See [22] for details.) This arrangement enables relying parties to choose which types of certificates to accept based on their required LOA.

CILogon also uses OAuth to integrate external web applications. That is to say, a user can delegate a CILogon certificate to a web portal whereby it can act on the user’s behalf. As an example, Globus (also known as Globus Online) can rely on specific CILogon certificate attributes to authorize large-scale GridFTP data transfers between research institutions.

CILogon works because the research institutions and projects involved have chosen to trust the CILogon certificates, and the specific attributes these certificates carry. This enables these organizations to manage geographically distributed users. As examples, XSEDE, the Open Science Grid (OSG), the Laser Interferometer Gravitational Wave Observatory (LIGO), and the Ocean Observatories Initiative (OOI) all use CILogon to make data and services available to their users, regardless of where their home institutions may be.

CILogon provides a valuable federated identity service that can be used to manage user bases that span different institutions and geographic areas. However, it is noted that this is all on the identity side, and not on the resource side. While different institutions that are participating in the same project could make services available to the same user base, there is no coordination about how this should be done. There is no discussion of resource discovery policy. There is also no discussion of the coordination of certificate attributes and their semantics across organizations. Attributes may need to be added to CILogon certificates to support what a specific project may need to do. This is evidently done on a case-by-case basis, rather than having a uniform way of managing different attribute sets for different "federated" environments.

REFEDS [23] (the Research and Education FEDerations group) is operated by GÉANT, the pan-European Research and Education Network. In a way similar to InCommon and eduGAIN, REFEDS enables federations of national federations. That is to say, national federations can make the IdPs and SPs available to the larger interfederation.

While enabling sharing and collaboration, this has led to the Entity Type problem. Different national federations will have different ways of expressing (or not) different services or policies. This is the semantic interoperability problem. To address this, REFEDS has added the notion of Entity Categories. The purpose of Entity Categories is to group entities that share common properties. Entity categories can also be used by IdPs to manage their Attribute Release policies. A set of attributes can be specified that are released to all SPs in a given category.

Each national federation must provide a Federation Policy. These Federation Policies define — at the human organization level — the Obligations and Rights of the Federation Operator and the Federation Members, where the members could be IdPs or SPs. These policies define things like how members may join or leave, what level of assurance identity credentials must have, what attributes must be released to SPs, and for what purposes SPs use the retrieved information.

An interesting aspect of REFEDS is the Hide from Discovery entity category. This is a special category of IdPs that should not be shown on discovery searches. There are several reasons for this: (1) the IdP may be in test, and not be ready for production, (2) there may be a name collision or near-name collision with another IdP, or (3) the IdP may be limited in its network accessibility and is thus unsuitable to general use.

REFEDS has also looked at how to manage the distribution of federation metadata across the entire environment. (REFEDS calls this the Metadata Flow problem.) This can be broadly partitioned in upstream — how federation operators can publish their local entities to REFEDS and GÉANT — and downstream — how REFEDS and GÉANT can publish metadata to different members. This is managed allowing IdPs and SPs to either Opt-In or Opt-Out of pushing their metadata to REFEDS/GÉANT. This is a basic 2x2 decision matrix with four combinations: 1) Full Opt-In: entities are pushed to REFEDS/GÉANT only on specific request, 2) Full Opt-Out: entities are not pushed to REFEDS/GÉANT only on specific request, 3) IdP Opt-In, SP Opt-Out, and 4) IdP Opt-Out, SP Opt-In.

The Entity Categories in REFEDS are a step in the right direction, but are still not very general. Entity Categories are an addition to the REFEDS model. Discovery policies should be general, and not limited to Opt-in or Opt-Out. While federations could operate with disparate IdPs, enabling some degree of commonality in identity credentials could greatly improve semantic interoperability. The federation descriptions and policies should be formalized and automated as much as possible, so as to be manageable by Federation Managers. This should essentially define the business model for any given federation or virtual organization. As such, it should be possible to instantiate any given federation type on-demand.

As a world-class, high-energy physics research institution, CERN (the European Organization for Nuclear Research) has enormous and varied computational requirements. To more flexibly support those requirements, CERN has been pursuing cloud computing. This enables CERN to allocate compute resources when they are needed. However, like many organizations, CERN has periods when resource demand exceeds capacity. In these cases, CERN will augment their own resources with commercial cloud resources.

CERN describes one approach to address this need [24], [25]. The Kubernetes container management system offers a basic Kubernetes cluster federation capability. Different Kubernetes clusters can be manually deployed on various commercial cloud providers. Using Kubernetes federation, these clusters can be jointly managed as a single entity. Once the initial cluster is deployed, the kubefed init fed command can be issued to start a kubernetes federation. When other clusters are started — potentially on other cloud providers — the kubefed join command can be issued with the appropriate arguments to join the previously initialized clusters. kubernetes ensures that the entire multi-cloud federation appears as a unified whole. Once the desired clusters have been created and connected, HTCondor [26] is used to run production workloads across the assembled infrastructure.

While this represents an interesting example of using multiple clouds to support world-class science, it is nonetheless a very simple, manually managed form of combining just compute resources that are then managed as a single administrative domain, using the usual kubernetes management tools. When a kubefed join command completes, there is an implicit trust relationship between the established multi-cluster and the joining cluster. The only services that are being managed are the kubernetes cluster services. Aside from the usual kubernetes authorizations, there is no differential authorizations based on who is providing the resources, and who is asking to use them.

OpenStack is a well-funded, open source cloud software project [27]. While there are many teams working on various cloud-related services, OpenStack has a set of core services that any cloud software stack would have. This includes compute (Nova), storage (Swift) and networking (Neutron). Keystone is the security service that all other OpenStack services use to authenticate users and authorize what they can do, e.g., instantiate a VM, or create/read/write a storage container.

In Keystone v3, the Keystone team has built-out basic support for cloud federation. The goal of this work is specifically to support the hybrid cloud business model. This would enable one OpenStack installation to cloud-burst to another OpenStack installation.

To do this, the Keystone Application Programming Interface (API) was augmented with the OS-FEDERATION extension. This extension essentially added a few calls whereby a local Keystone administrator could explicitly configure their Keystone service to federate-in or federate-out with a specific remote OpenStack installation. That is to say, this establishes an explicit trust relationship between two Keystone.

Federation-out enables a local Keystone to act as an IdP for a remote Keystone. When doing so, Keystone issues PySAML2 assertions about the used owned by that Keystone. Federate-in allows users from a remote OpenStack to invoke services on the local OpenStack. When remote services are requesting identity assertions, SAML 2 or OpenID Connect can be used.

As part of OS-FEDERATION, the local Keystone administrator can define mapping rules. These mapping rules are used to map the attribute assertions issued by the remote user’s IdP to a specific project and group within a domain in the local Keystone. This approach enables remote users to be granted the roles and authorizations of the group they were mapped to. These local group authorizations are then used to make local service access decisions. In this way, Keystone does not have to maintain guest accounts or do any type of dynamic account creation.

OpenStack represents an important example of cloud federation, but with some clear simplifying limitations. First, it is intended to manage a small, fixed set of common cloud infrastructure services, i.e., OpenStack services. It is not intended to manage arbitrary, application-level services. While Keystone v3 domains are semantically very close to virtual administrative domains or virtual organizations, and roles can be domain-specific, Keystone does not provide or enforce the common understanding or common state that is necessary to manage a distributed federation. As a case in point, the mapping rules that each Keystone administrator uses to map remote users to local groups are independently decided and implemented. Finally, it is noted that these Keystone federations are all manually managed, pair-wise federations.

KeyVOMS is another example of a centralized, third-party federation provider [28, 29]. KeyVOMS simply repurposed a stand-alone deployment of the OpenStack Keystone v3 server to function as a Keystone-based Virtual Organization Management Systems (KeyVOMS). Here, the concept of a Virtual Organization (VO) developed in the grid computing era [30] was adapted to be essentially a federation instance.

Keystone v3 introduced the domain to its object model for managing users, projects, and access to all other OpenStack services. Domain could "own" users and projects. Users could be granted project membership by the domain administrator. The domain administrator could also grant different roles to users. Keystone maintains a service catalog for the entire installation, e.g., Nova, Swift, Cinder, etc. These service endpoints could be associated with projects, and thus made available to those project members. Whenever a user authenticated to Keystone for a specific domain and project, they would receive a cryptographically signed AUTH_TOKEN that included a filtered service catalog of just the services they were authorized to use.

Two key insights enabled the development of KeyVOMS: (1) a Keystone domain was tantamount to a VO, and (2) the Keystone service catalog could include arbitrary, application-level services — in addition to the other cloud-infrastructure services. Hence, any remote service owner that wished to participate in a VO could register a service endpoint with Keystone, and then associate it with a specific domain and project.

Whenever a user invoked one of these services (after authenticating and receiving their filtered service catalog), they needed to provide their AUTH_TOKEN, which included their roles, and also their domain (VO) and project memberships. Keystone services are built using Web Service Gateway Interfaces (WSGI). Hence, each service has a configurable pipeline of stages that can be used to condition service requests before actually handing them to the service for execution. For KeyVOMS, a VO_AUTH pipeline stage was added to any service that needed to be VO-enabled. Hence, whenever a request arrived from a VO member, the VO_AUTH stage would validate the user’s token with KeyVOMS. The user’s VO roles could then be used to make an access decision.

To make all this work, only the Keystone rule file was modified to add three roles. A VOMS_Admin role was added that was actually just a synonym for a Keystone Admin. A VO_Admin role enabled a user to manage anything within a single domain/VO federation. Finally, a VO_Site_Admin role was also added. These members were authorized to register services in KeyVOMS within a specific domain/VO.

A number of demonstrations were done using KeyVOMS. This involved VO-enabled services for managing RSS feeds, map data servers, and file servers. The demonstrations include international disaster response scenarios and managing access to resident space object tracking data from four continents.

KeyVOMS is a good example of a centralized, third-party, federation manager. Keystone was designed and built to be a service whose purpose is to manage access to service endpoints. As an open source project, it was straight-forward to leverage the Keystone object model for a much larger purpose, i.e., managing federations based on sharing service endpoints.

As a centralized system, member vetting is up to the discretion of the VOMS_Admin and VO_Admins. (On-boarding new FMs is a moot issue since KeyVOMS is a single, centralized FM.) Federated resource discovery policy was accomplished by using Keystone’s endpoint filtering capability.

Initially Keystone v3 did not have domain-specific roles. All roles were visible across all domains. This is sufficient if only a small, fixed set of services are being managed across all domains — such as a set of cloud infrastructure services. However, if arbitrary, application-level services are being managed, then each domain/VO should have its own set of roles. Domain-specific roles were finally implemented in the Pike release. Resource owners had unilateral authority to set the access policy for their resources. However, to participate in a VO, these policies would have to be based on the roles defined in a domain/VO. The Resource owner also had to explicitly trust the KeyVOMS server and VO_Admin to grant roles only to truly authorized users.

No specific accounting or auditing was done — other than the usual Keystone logging. Nonetheless, accounting and auditing could be supported. Since accounting and auditing is not really federation-specific, this could be addressed in the larger context of the OpenStack open source project.

Federation discovery was supported in a simple way — this only involved listing domains/VOs. Currently only the VOMS_Admin has authorization to list all domains/VOs. However, it would be straight-forward to allow domains/VOs to be discoverable by arbitrary users based on some set of attributes.

An important observation is that Keystone v3 could be generalized to a general-purpose, distributed P2P Federation Manager. The Keystone object model could be extended to include virtual domains that have additional semantics. Projects and service endpoints would be replicated among every Keystone installation that is a member of a federation, i.e., member of a virtual domain. Hence, service endpoints would eventually be discoverable by authorized domain/VO/project members. When a service invocation is made, the user’s credentials could be routed back to their home Keystone for validation — in a manner similar to that used by eduRoam.

While performance and scalability issues would certainly exist, this approach is certainly feasible. It also means that internal FM capability could be realized as an optional part of a Keystone deployment. This could also be an external FM capability if the Keystone operator wished to make the federation services available to a wider set of users.

The Interoperable Global Trust Federation (IGTF) and the GÉANT Trusted Certificate Service both provide a valuable and necessary service for making federations work. Simply put, IGTF and the GÉANT TCS provide a trusted set of identity credential providers. This is done on a global scale by operating through three different regional Policy Management Authorities: (1) Asia-Pacific, (2) Europe, Middle East and Africa, and (3) the Americas. (By industrial standards, though, the actual resource demands by international science collaborations may be modest).

IGTF defines a minimum set of requirements and recommendations for the operation of PKI Certificate Authorities, attribute assertions, and attribute release. IGTF also maintains a set of authentication profiles. These specify the policies and technical requirements for classes of identity assertions, and the assertion providers that provide them. These authentication profiles are associated with different Levels of Assurance that balance cost and feasibility for IGTF identity providers. Details on these issues can be found at [31]. It is noted that IGTF is a peer organization to many other organizations, including the REFEDS.

The GÉANT Trusted Certificate Service (TCS) performs a similar function, but within the realm of GÉANT services. This is done by relying on a commercial Certificate Authority operator (DigiCert). Five different types of certificates are available for a variety of purposes, e.g., authenticating servers and establishing secure sessions, identifying users, and signing code and documents. This is run in conjunction with the Trusted Academic CA Repository (TACAR). TACAR hosts the trust anchors of the Public Key Infrastructures (PKI) needed for GÉANT services such as eduRoam, eduGAIN and perfSONAR. The IGTF has accredited most of the CA root certificates hosted by TACAR.

IGTF and the GÉANT TCS both illustrate a critical function for federations to work: a set of trusted identity providers that are coupled with known governance. Organizations that wish to engage in federations must establish those common roots of trust. This could (and should) also include the common, known set of identity attributes that are integral to a federation’s "business model". For federations to be truly instantiated on-demand, such trust infrastructures should be established prior to need.

GENI (Global Environment for Network Innovation) [32] provides a network environment that can support many different types of experimental networks at the same time. This capability has direct relevance to supporting federations.

GENI uses the notions of projects and slices to manage experimental environments. Every project has a project lead that can allocate one or more slices within the project. A slice is a set of virtualized resources that are used in isolation from other experiments and experimental communities. These resources (bare metal machines, virtual machines, small clouds) are owned and operated by different institutions. The resources at each institution are called an aggregate. Aggregates provide resources to the different slices and projects based on GENI-issued user and slice credentials.

GENI is organized with a separate control plane and data plane. When instantiating a slice, the control plane will create different Virtual Local Area Networks (VLANs) over the same physical links. The project that uses the slice cannot see any network traffic from any other experiment.

The allocation of aggregate resources is managed at each site through the Aggregate Manager API. This enables aggregates to advertise and allocate resources. The authorization to use these APIs is managed through three different GENI clearinghouses. These are operated by the GENI Program Office, EmuLab, and Planetlab. Clearinghouses issue GENI user and slice credentials, and manage project and slice membership. It is interesting to note that GENI users can authenticate to GENI using their InCommon credentials.

Aggregate owners must federate with one or more clearinghouses. In the GENI context, this means that aggregate owners must trust the credentials issued by these clearinghouses. It is evidently common for aggregate owners to federate with all three clearinghouses such that experimenters can get resources through any clearinghouse.

A key governance property here is that the GENI clearinghouses act as a federation root of trust for all entities within a GENI federation. That is to say, any member of a GENI project must trust the credentials signed by the clearinghouses. While this trust model may not work for all application domains, it does reduce the required number of trust relationships from O(N²) to O(N).

GENI uses a federated environment to manage the networked aggregation of computing resources from different institutions for different experimental purposes. This is clearly not a general federation since only a narrow, fixed type of resources are being managed, i.e., networked resources. Once a site makes aggregate resources available to GENI, there is no selective resource discovery policies. Any GENI user can look at the available resources and request their allocation.

Nonetheless, GENI represents a fundamental capability that has wide applicability. CloudLab [33] uses GENI to connect and access resources from different sites to entire, experimental clouds to be allocated on-demand. PlanetLab [34] has also looked at using GENI for similar purposes. Chameleon Cloud [35] is looking at using GENI for Identity Federation whereby users can login to either GENI or Chameleon environments. To do so, a GENI project association is necessary. Hence, the GENI/Chameleon Federation project was created that Chameleon users must be a member of [36].

Jetstream [37] is a geographically distributed cloud that is hosted by Indiana University (IU), the Texas Advanced Computing Center (TACC), and the University of Arizona (UA). These sites have 100Gps connections provided by Internet2. While the operational Jetstream environment is based on OpenStack, Jetstream offers the Atmosphere browser-based interface. Atmosphere was originally developed to support scientific applications and workloads. Jetstream is not intended to be a high-performance computing environment, but rather to support interactive research and prototyping. Through this interface, users can launch resources (e.g., VMs) at IU or TACC. (UA provides a test environment.)

XSEDE (Extreme Science and Engineering Discovery Environment) is a distributed infrastructure, for large-scale scientific applications, that is funded by the National Science Foundation (NSF). While XSEDE nominally integrates resources across seven US institutions, it acts as a service broker across many more institutions than that. The XSEDE User Portal (XUP) provides users with a convenient, web-based interface to the XSEDE resources.

The important connection between these two systems is that Jetstream is accessible through XSEDE credentials issued by Globus Auth [38]. Globus Auth can broker authentication and authorization between users, identity providers and resource providers. This includes identity providers such as XSEDE, InCommon, and other commercial web service providers. A key concept is that a Globus Auth account can be a set of identities — a primary identity with one or more linked identities. Using this possible set of identities, Globus Auth can then act as an authorization server by issuing attribute assertions that enable a legitimate user to access a desired resource.

Globus Auth is implemented as an application on top of an AWS Reliable Data Service PostgreSQL database, with a web interface and API [39]. As such, Globus Auth is essentially a centralized, third-party Federation Manager. Hence, all identity providers and resource services that are managed by Globus Auth are registered in this central location. This makes it straight-forward for Globus Auth to manage Globus groups [40], which can even be hierarchical. Group membership can be used to manage access to data and services. A resource server can define different scopes for itself. This means that clients wishing to use that resource must request a properly scoped access token from Globus Auth.

Globus Auth relies on OpenID Connect and OAuth 2 to achieve these capabilities. This enables Globus Auth to exchange identity assertions with multiple Identity Providers, and also to delegate authorizations to third parties. As an example, Globus Auth can delegate tokens to Globus Transfer to accomplish third-party data transfers on behalf of a user.

Globus Auth is, again, a system that comes close to the NIST Conceptual Model. It is essentially an external federation provider with a centralized, third-party deployment model. It provides the group concept to manage sets of users and their authorizations. This is similar to a virtual domain or virtual organization, yet there are still differences. A group is something like a federation instance. A group can be created by any authenticated user. The group owner can then grant membership in the group, and determine the visibility of their group and subgroups.

Resource servers can also register with Globus Auth to use Globus Auth as an authorization server. Resources can define different scopes that are tantamount to different access policies, and they can also require that only specific IdPs can be used to grant access tokens. Groups can also be used to manage access to specific services, such as group wikis, but it is unclear how general this approach is. Globus Auth does support scientific data discovery but apparently not general resource discovery, i.e., to include arbitrary application-level services. There appears to be no general policy mechanism for managing resource discovery.

All in all, Globus Auth comes close to a general federation management tool. Its centralized implementation greatly simplifies many of its management requirements. While the group concept is close to a federation instance in the NIST Conceptual Model, it is not presented as a coherent, unified approach to managing federation instances, as in the NIST model. Also, Globus Auth is a centralized third-party federation provider. Generalizing it to a distributed implementation that can be deployed on-demand will be a major undertaking.

The Authentication and Authorisation framework for Research and Collaboration communities (AARC) [41] extends the capabilities of eduGAIN. A key function of eduGAIN is the maintenance of the metadata — a large, undifferentiated file of IdP and SP descriptors. AARC adds the notion of groups and actually uses the terminology of Virtual Organizations. The authorization to use a given service can be based on a user’s specific group memberships and the group roles or attributes that have been granted to them. Recognizing the importance of semantic interoperability, work has been done to ensure that consistent syntax and semantics are maintained throughout a VO.

While a significant number of pilot projects have been done, a large part of AARC has been focused on high-level design work. A key aspect of AARC has been the choice and design of a Token Translation Service. Rather than taking the approach of converting "native" identity credentials into one common VO credential that is understood everywhere in a VO, the Token Translation Service can translate any member’s credential type into any other type that is understood by a service the member wishes to invoke. That is to say, it can translate OpenID Connect into SAML assertions, or OpenID Connect into SSH keys. Even when no token type translation needs to be done, AARC defines User Attribute Services whereby attribute enrichment can be done, i.e., attributes can be added to a member’s identity credential thereby enabling them to access desired services in the VO.

As a system built on top of eduGAIN and GÉANT, AARC adds key functionality for the flexible management of virtual domains — or VOs. AARC (and eduGAIN) are still an external federation provider, but with the added management flexibility of groups, roles and attributes, i.e., VOs. While services can base their access policies on group membership and granted attributes, there is no discussion of resource discovery policies associated with VOs. eduGAIN is based on maintaining an essentially flat metadata file of IdPs and SPs that is visible to all member institutions. In many situations, resource owners that want to make their resource available to a specific VO for a specific purpose, may wish to define a discovery policy, such that only specific VO members with a genuine need to know can discover and access their resource.

The many AARC federation support services are presented in the conceptual AARC Blueprint Architecture. As a conceptual architecture, the functional capabilities can be allocated or colocated in different ways. The Token Translation Service (TTS), for example, could be (a) embedded in the AARC Proxy or with the End Services themselves, or (b) used in a standalone configuration. An Embedded TTS would only issue credentials for a single service or service invocation. A Standalone TTS, however, could issue credentials for many services across many organizations. In all cases, though, it appears that all member requests are proxied by AARC. This has performance latency implications but does mean that services can be made part of a VO w/o having to make any changes to the front-end of the services themselves.

While AARC does add valuable flexibility to the eduGAIN model, it continues to have some of the governance baked-in. While VOs could have one or more VO Admins, all VO member sites must trust these admins. As a centralized federation provider, AARC and eduGAIN can have no notion of a site admin that can manage their local users or resources. eduGAIN also manages the metadata, i.e., the admission of all new IdPs and SPs. Nonetheless, AARC has identified and demonstrates key capabilities in support of more general federations.

Fogbow [42] is a joint EU/Brazilian cloud infrastructure federation project. Rather than taking a multi-cloud or hybrid cloud approach, Fogbow is providing infrastructure federation across different cloud providers. Fogbow is cloud federation middleware implemented as a suite of microservices. This middleware is composed of three major components. The Membership Service controls which cloud providers belong to the federation, and manages network communication among them. The Allocation Manager at each cloud provides a well-defined common API to manage resources anywhere in the federation environment. Finally, the Allocation Manager uses an Orchestration Manager at each cloud provider to interact directly with that provider’s API. Identity and authorization are handled by issuing federation tokens. When a user makes a service request that is routed to a remote provider, that federation token is translated into a local access token. Local token translation is unilaterally controlled by each cloud administrator. A reverse tunneling service is provided to enable users, user applications and Allocation Managers to communicate either locally or remotely, anywhere within the federated environment.

Fogbow demonstrates another interesting combination of possible federation capabilities. Using a network service to enable (and isolate) a federation’s network traffic is an important part of providing communication security for a federation. For this reason, such network support will eventually be an integral part of federation management. It is not a coincidence that this network service is part of the Membership Service.

It is also important to note that the Fogbow project choose to use a common federation token that has meaning anywhere in the federation. Only when a service request gets to a local provider is this token translated into something that is locally understood. As federations become more widely used, it may be the case that federation tokens become more widely understood and trusted. Hence, such token translation may be become less of an issue.

Finally, it is noted that currently Fogbow is just an infrastructure federation. It is managing a small set of core cloud services. However, a service catalog is under development. There may be no inherent reason why Fogbow could not be generalized to arbitrary, application-level services.

The FICAM (Federal Identity, Credential and Access Management) initiative [43] is a program run by the Federal Chief Information Officers (CIO) Council. This initiative has defined a general, Federal FICAM Enterprise Architecture for managing identity and credentials to make authentication and authorization decisions at the enterprise scale. This includes a wide range of topics, such as identify proofing, credential distribution, and enterprise governance.

FICAM also identifies some federation use cases, where users from one security domain can access resources in another domain. These cases make the same assumptions as the standards they are built on. Namely that the user already knows about the service they are attempting to use. The assumption is that resource discovery has already been done by out-of-band methods. This assumes that the user and resource are operating in an open environment. Only after the initial user request is made does the system try to verify who is making the request, who might vouch for their identity, determine which authorizations the user may have that are relevant to the resource being requested. This also assumes that the SP and IdP involved understand the same identity and authorization attributes.

The NIST Reference Architecture has been developed to address all of these issues. This is done by establishing a virtual environment with a known boundary, i.e., a closed environment, wherein the necessary and desired resource management can be done by the participants. Resource discovery and discovery policies within the context of this virtual administrative domain can be customized, and are an integral part of the model. This also enables having a common understanding of identity and authorization attributes.

Aside from these federation-related issues, all of the enterprise-level identity and credential management practices defined by FICAM are nonetheless applicable for the NIST Reference Architecture. These FICAM issues should be systematically reviewed to determine how they should be used in federated environments. Unfortunately, this is out of scope for this document.

Ping Federate [44] is a commercial product of Ping Identity [45] that enables one company to provide specific corporate services to other companies. Such corporate services include things like email and travel arrangements. The key concept is that a client company continues to maintain the IdP that issues credentials for their employees (users) to use the corporate services being provided by the external company.

For example, when an employee starts their email client, that email client connects to the external email provider. The external email provider, however, redirects the employee to authenticate against their employer’s IdP, e.g., an LDAP server. After successful authentication, the employee gets access to their corporate email, all of which is being managed by the external provider.

This arrangement means there is an explicit trust relationship between the external provider and a corporation’s IdP. The external provider is trusting the corporate IdP to make authentication decisions. While this is a key example of a specific aspect of federation, it is noted that Ping Federate must be configured and deployed to manage one fixed service between two fixed entities, e.g., email between a corporation and its email provider. Its main function is (a) to enable authentication decisions to be made, but without divulging employee’s identity credentials to the provider, and (b) to secure the network communications between the two.

As such, Ping Federate is far from a general federation. An arbitrary number of arbitrary application-level services cannot be handled. As a result, a rich set of resource discovery and access policies are not needed. Since only one static type of service is being handled per installation, any necessary accounting or auditing becomes a much simpler issue. Nonetheless, Ping Federate is an important example of a commercially established product that addresses a key aspect of general federation.

There are many other systems and projects that could be covered. With the efforts surveyed so far, however, we will be able to support our analysis and conclusions. Nonetheless, we will end this subsection by making at least a brief reference to some of these other systems, projects and organizations. Many others are certainly possible.

Existing standards like OpenID, OAuth, OpenID Connect and UMA all do not assume that there is any pre-existing relationship between the Relying Party/client making a request and the service/resource provider. The sequence diagrams of all these standards essentially start with the client making a request to a service. This assumes that the client has already discovered the service, by some means which is unspecified or out-of-scope for the standard. Hence, these standards do not address the issue of resource discovery policy, i.e., how a resource provider can control who can discover and access their resource/service.

As an example, OpenID Connect (OIDC) essentially combines OpenID authentication with OAuth authorization. When a User’s Browser makes a request to a website (Relying Party, RP), the RP immediately makes a request to an OIDC Provider (OP). The OP provides an authorization endpoint to which the User’s browser is redirected. The User is challenged to authenticate through the OP using an appropriate back-end IdP which returns a set of user attributes (on successful authentication). The User is again redirected to the RP which uses the authenticated user’s attributes to request an access token from the OP. After the access token has been validated, the User reissues the original request to the RP through a third redirection, which can finally be honored.

It is noted that OAuth (and OpenID Connect) utilize a token exchange protocol [56] whereby a client can exchange some representation of resource authorization for an access token. This can also include token exchange to support delegation and impersonation. Since this interaction takes place between a client and a Security Token Service, this interaction can be called a token exchange. This can be contrasted with systems such as AARC and the INDIGO-DataCloud that use what can be more aptly called token translation. Since these systems proxy all communication, e.g., service invocation, the client never sees or uses the translated token. In all cases, though, a new token is derived from an existing token. The primary difference is who gets to use the new token once it is created.

While User-Managed Access (UMA) also does not assume any pre-existing relationship, it does enable (a) a Resource Owner to register a set of resource-specific policy conditions with an UMA Authorization Server ahead of time, and (b) a client to pre-register a set of RegisteredScopes. When doing an authorization assessment as part of a service request from a client, the UMA Auth Server must compare the client’s RegisteredScopes with the RequestedScopes, along with the TicketScopes() associated with the resource. This is essentially testing if the client has the authorization to execute the requested operation on the given resource. After this, any applicable policies are evaluated, which is out-of-scope. The UMA standard is also silent on how such pre-registration is managed and by whom.

In reviewing the functional behavior of these standards, it is clear that a key capability that the NIST Federation Model provides is establishing and managing a set of relationships prior to any federation member attempting to invoke a federation resource. That is to say, a federation is a set of pre-existing relationships. This pre-established context/relationship enables resource discovery, scope, and policy issues to be managed prior to use. It allows the definition of governance, i.e., who can manage what within any given federation. Such a set of pre-established relationships can be called virtual domain or a virtual organization, and can be managed by a Federation Manager. By explicitly managing the relationship, and doing so more completely, we can (a) address issues such as resource discovery, and (b) simplify resource invocation by avoiding numerous redirections to collect information about a user. Also, the resource owner still defines local access policy and makes final access decisions.

The Security Assertion Markup Language (SAML) is an XML-based standard for communicating user information concerning authentication, entitlement and attributes. As such, SAML has been used as the implementation vehicle for other tools and standards, such as Internet2 Shibboleth and OASIS WS-Security. SAML enables Single Sign-on (SSO) by enabling the communication of an authentication assertion from one site to another site. SAML can also use the same secure mechanism to communicate additional attribute information about a user. This also enables a user’s attributes to be selectively released. That is to say, if a particular user attribute is not important for the Service Provider to make an authorization decision, it can be withheld.

SAML consists of assertions, protocols, bindings and profiles. Assertions can concern authentication, attributes or access decisions. SAML has several request/response protocols for communications with SPs and IdPs. This includes interactions such as authenticating a user, requesting specific attributes assertions, and terminating login sessions. When originally defined, SAML used a SOAP binding, but the HTTP Redirect binding is used far more commonly today. Several profiles are available for different usage scenarios. For example, the Web Browser SSO Profile manages communication between a SP and IdP to enable SSO from a browser. There is also a X.500/LDAP Profile for how X.500/LDAP attributes are carried within SAML assertions.

It is also useful to point out the distinctions between using SAML assertions and PKI Certificates for doing authentications. SAML can retrieve attributes on each invocation, while attributes become part of a signed PKI certificate. This can be an issue if a User’s attributes can change with some frequency. It is possible to use PKI for authentication and then collect authorization information using SAML. SAML assertions can also be protected by signing them with a PKI key.

While SAML has been a very successful standard, it nonetheless does not address the issue of pre-established relationships, including resource discovery. It is assumed that the User already knows about the service that is being invoked. The primary function that SAML is used for is to provide a way for a Service Provider to redirect a User’s request to a SAML Provider that can authenticate the User with an external IdP. While SAML was not intended to address authorization, it is noted that in the absence of any pre-established relationships, there is also no agreement on which attributes SAML should be used to retrieve to support access to a given service. SAML may very well have a role in supporting general federation, but it will have to be integrated with a system of one or more Federation Managers.

The eXtensible Access Control Markup Language (XACML) standard actually defines far more than just a markup language. The authorization architecture of Policy Enforcement Points, Policy Decision Points, Policy Administration Points, and Policy Information Points provides a nice separation of concerns. This avoids the issues encountered when things like separate Access Control Lists are hard-coded into each application. XACML provides Attribute-Based Access Control (ABAC) whereby resource access control decisions are made based on the attributes associated with users. It is noted that Role-Based Access Control (RBAC) can be also implemented as a special case of ABAC. Clearly XACML can be used with the attributes carried in PKI certificates.

Not surprisingly, since both SAML and XACML have been developed by OASIS, there is a profile for integrating SAML2 with all versions of XACML. SAML can be used to carry XACML policies, policy queries, and policy query responses. SAML attribute assertions can be consumed by XACML PDPs, and authorization decisions can be returned.

Even though XACML was designed to be a general-purpose access control mechanism with a general-purpose Policy Language, it could not easily express geospatial policy constraints. Hence, OGC defined the GeoXACML extension to XACML. In essence, this extension defined a geometry model whereby geometric data types could be included in access policies, along with functions to test for topological relationships between geometries.

As an access control architecture, XACML (and GeoXACML) could very well be used in federated environments. If being used in a distributed deployment, the policy stores would have to be kept consistent. The policies themselves would also have to be managed in a way that consistently captures the desired governance of any given federation instance. Nonetheless, the general XACML design approach could possibly be integrated into the design and implementations of Federation Managers and how they are used by federation participants.