Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 92 additions & 90 deletions content/v1.20/guides/multi-tenant.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@ weight: 240
---

This guide describes how to use Crossplane effectively in multi-tenant
environments by utilizing Kubernetes primitives and compatible policy
environments by using Kubernetes primitives and compatible policy
enforcement projects in the cloud native ecosystem.

## Summary

Infrastructure operators in multi-tenant Crossplane environments typically
utilize composition and Kubernetes RBAC to define lightweight, standardized
policies that dictate what level of self-service developers are given when
requesting infrastructure. This is primarily achieved through exposing abstract
resource types at the namespace scope, defining `Roles` for teams and
individuals within that namespace, and patching the `spec.providerConfigRef` of
use composition and Kubernetes RBAC to define lightweight, standardized
policies that dictate what level of self-service developers have when
requesting infrastructure. Crossplane achieves this self-service by exposing
abstract resource types at the namespace scope, defining `Roles` for teams and
individuals in that namespace, and patching the `spec.providerConfigRef` of
the underlying managed resources so that they use a specific `ProviderConfig`
and credentials when provisioned from each namespace. Larger organizations, or
those with more complex environments, may choose to incorporate third-party
Expand All @@ -23,30 +23,29 @@ describe each of these scenarios in greater detail.

- [Summary](#summary)
- [Background](#background)
- [Cluster-Scoped Managed Resources](#cluster-scoped-managed-resources)
- [Namespace Scoped Claims](#namespace-scoped-claims)
- [Single Cluster Multi-Tenancy](#single-cluster-multi-tenancy)
- [Cluster-scoped managed resources](#cluster-scoped-managed-resources)
- [Namespace-scoped claims](#namespace-scoped-claims)
- [Single cluster multi-tenancy](#single-cluster-multi-tenancy)
- [Composition as an Isolation Mechanism](#composition-as-an-isolation-mechanism)
- [Namespaces as an Isolation Mechanism](#namespaces-as-an-isolation-mechanism)
- [Policy Enforcement with Open Policy Agent](#policy-enforcement-with-open-policy-agent)
- [Multi-Cluster Multi-Tenancy](#multi-cluster-multi-tenancy)
- [Reproducible Platforms with Configuration Packages](#reproducible-platforms-with-configuration-packages)
- [Control Plane of Control Planes](#control-plane-of-control-planes)
- [Policy enforcement with Open Policy Agent](#policy-enforcement-with-open-policy-agent)
- [Multi-cluster multi-tenancy](#multi-cluster-multi-tenancy)
- [Reproducible platforms with configuration packages](#reproducible-platforms-with-configuration-packages)
- [Control plane of control planes](#control-plane-of-control-planes)

## Background

Crossplane is designed to run in multi-tenant environments where many teams are
consuming the services and abstractions provided by infrastructure operators in
the cluster. This functionality is facilitated by two major design patterns in
the Crossplane ecosystem.
Crossplane runs in multi-tenant environments where multiple teams consume the
services and abstractions that infrastructure operators provide in the cluster.
Two major design patterns in the Crossplane ecosystem enable this capability.

### Cluster-Scoped Managed Resources
### Cluster-scoped managed resources

Typically, Crossplane providers, which supply granular [managed resources] that
reflect an external API, authenticate by using a `ProviderConfig` object that
points to a credentials source (such as a Kubernetes `Secret`, the `Pod`
filesystem, or an environment variable). Then, every managed resource references
a `ProviderConfig` that points to credentials with sufficient permissions to
file system, or an environment variable). Then, every managed resource references
a `ProviderConfig` that points to credentials with permissions to
manage that resource type.

For example, the following `ProviderConfig` for `provider-aws` points to a
Expand All @@ -66,8 +65,8 @@ spec:
key: creds
```

If a user desired for these credentials to be used to provision an
`RDSInstance`, they would reference the `ProviderConfig` in the object manifest:
If a user wants these credentials to provision an
`RDSInstance`, they reference the `ProviderConfig` in the object manifest:

```yaml
apiVersion: database.aws.crossplane.io/v1beta1
Expand All @@ -90,26 +89,27 @@ spec:
name: aws-rdsmysql-conn
```

Since both the `ProviderConfig` and all managed resources are cluster-scoped,
the RDS controller in `provider-aws` will resolve this reference by fetching the
Because both the `ProviderConfig` and all managed resources are cluster-scoped,
the RDS controller in `provider-aws` resolves this reference by fetching the
`ProviderConfig`, obtaining the credentials it points to, and using those
credentials to reconcile the `RDSInstance`. This means that anyone who has been
given [RBAC] to manage `RDSInstance` objects can use any credentials to do so.
credentials to reconcile the `RDSInstance`. This means that anyone with
[RBAC] to manage `RDSInstance` objects can use any credentials to do so.
In practice, Crossplane assumes that only folks acting as infrastructure
administrators or platform builders will interact directly with cluster-scoped
administrators or platform builders interact directly with cluster-scoped
resources.

### Namespace Scoped Claims
### Namespace-scoped claims

While managed resources exist at the cluster scope, composite resources, which
are defined using a **CompositeResourceDefinition (XRD)** may exist at either
a **CompositeResourceDefinition (XRD)** defines, may exist at either
the cluster or namespace scope. Platform builders define XRDs and
**Compositions** that specify what granular managed resources should be created
in response to the creation of an instance of the XRD. More information about
this architecture can be found in the [Composition] documentation.
**Compositions** that specify what granular managed resources Crossplane
creates in response to creating an instance of the XRD. The
[Composition] documentation has more information about
this architecture.

Every XRD is exposed at the cluster scope, but only those with `spec.claimNames`
defined will have a namespace scoped variant.
Every XRD exists at the cluster scope, but only those with `spec.claimNames`
defined have a namespace scoped variant.

```yaml
apiVersion: apiextensions.crossplane.io/v1
Expand All @@ -127,77 +127,77 @@ spec:
...
```

When the example above is created, Crossplane will produce two
When you create the preceding example, Crossplane produces two
[CustomResourceDefinitions]:
1. A cluster-scoped type with `kind: XMySQLInstance`. This is referred to as a
1. A cluster-scoped type with `kind: XMySQLInstance`. Crossplane calls this a
**Composite Resource (XR)**.
2. A namespace scoped type with `kind: MySQLInstance`. This is referred to as a
2. A namespace scoped type with `kind: MySQLInstance`. Crossplane calls this a
**Claim (XRC)**.

Platform builders may choose to define an arbitrary number of Compositions that
map to these types, meaning that creating a `MySQLInstance` in a given namespace
can result in the creations of any set of managed resources at the cluster
scope. For instance, creating a `MySQLInstance` could result in the creation of
the `RDSInstance` defined above.
the `RDSInstance` defined preceding.

## Single Cluster Multi-Tenancy
## Single cluster multi-tenancy

Depending on the size and scope of an organization, platform teams may choose to
run one central Crossplane control plane, or many different ones for each team
or business unit. This section will focus on servicing multiple teams within a
single cluster, which may or may not be one of many other Crossplane clusters in
run one central Crossplane control plane, or one for each team
or business unit. This section focuses on servicing multiple teams in
a single cluster, which may or may not be one of multiple other Crossplane clusters in
the organization.

### Composition as an Isolation Mechanism

While managed resources always reflect every field that the underlying provider
API exposes, XRDs can have any schema that a platform builder chooses. The
fields in the XRD schema can then be patched onto fields in the underlying
fields in the XRD schema can then map onto fields in the underlying
managed resource defined in a Composition, essentially exposing those fields as
configurable to the consumer of the XR or XRC.

This feature serves as a lightweight policy mechanism by only giving the
consumer the ability to customize the underlying resources to the extent the
platform builder desires. For instance, in the examples above, a platform
platform builder desires. For instance, in the preceding examples, a platform
builder may choose to define a `spec.location` field in the schema of the
`XMySQLInstance` that's an enum with options `east` and `west`. In the
Composition, those fields could map to the `RDSInstance` `spec.region` field,
making the value either `us-east-1` or `us-west-1`. If no other patches were
defined for the `RDSInstance`, giving a user the ability (using RBAC) to create
a `XMySQLInstance` / `MySQLInstance` would be akin to giving the ability to
create a specifically configured `RDSInstance`, where they can only decide
the region where it lives and they're restricted to two options.

This model is in contrast to many infrastructure as code tools where the end
user must have provider credentials to create the underlying resources that are
rendered from the abstraction. Crossplane takes a different approach, defining
making the value either `us-east-1` or `us-west-1`. If the Composition defined
no other patches for the `RDSInstance`, giving a user the ability (using RBAC)
to create a `XMySQLInstance` / `MySQLInstance` would be akin to giving the
ability to create a specifically configured `RDSInstance`, where they can only
decide the region where it lives and they're restricted to two options.

This model is in contrast to most infrastructure as code tools where the end
user must have provider credentials to create the underlying resources that
the abstraction renders. Crossplane takes a different approach, defining
various credentials in the cluster (using the `ProviderConfig`), then giving
only the provider controllers the ability to utilize those credentials and
only the provider controllers the ability to use those credentials and
provision infrastructure on the users behalf. This creates a consistent
permission model, even when using many providers with differing IAM models, by
standardizing on Kubernetes RBAC.
permission model, even when using different providers with differing IAM
models, by standardizing on Kubernetes RBAC.

### Namespaces as an Isolation Mechanism

While the ability to define abstract schemas and patches to concrete resource
types using composition is powerful, the ability to define Claim types at the
namespace scope enhances the functionality further by enabling RBAC to be
applied with namespace restrictions. Most users in a cluster don't have access
types using composition is powerful, support for defining Claim types at the
namespace scope enhances this capability further by enabling RBAC to apply
with namespace restrictions. Most users in a cluster don't have access
to cluster-scoped resources as they're considered only relevant to
infrastructure admins by both Kubernetes and Crossplane.

Building on our `XMySQLInstance` / `MySQLInstance` example, a platform
Building on the `XMySQLInstance` / `MySQLInstance` example, a platform
builder may choose to define permissions on `MySQLInstance` at the namespace
scope using a `Role`. This allows for giving users the ability to create and
manage `MySQLInstances` in their given namespace, but not the ability to see
those defined in other namespaces.

Furthermore, because the `metadata.namespace` is a field on the XRC, patching can
be utilized to configure managed resources based on the namespace in which the
corresponding XRC was defined. This is especially useful if a platform builder
wants to designate specific credentials or a set of credentials that users in a
given namespace can utilize when provisioning infrastructure using an XRC. This
can be accomplished today by creating one or more `ProviderConfig` objects that
Furthermore, because the `metadata.namespace` is a field on the XRC, patching
configures managed resources based on the namespace in which the
corresponding XRC exists. Namespace-scoped patching is useful if a platform
builder wants to set specific credentials or a set of credentials that users
in a given namespace can use when provisioning infrastructure using an XRC.
You can do this today by creating one or more `ProviderConfig` objects that
include the name of the namespace in the `ProviderConfig` name. For example, if
any `MySQLInstance` created in the `team-1` namespace should use specific AWS
credentials when the provider controller creates the underlying `RDSInstance`,
Expand Down Expand Up @@ -238,46 +238,48 @@ resources:
fromFieldPath: Required
```

This would result in the `RDSInstance` using the `ProviderConfig` of whatever
namespace the corresponding `MySQLInstance` was created in.
This results in the `RDSInstance` using the `ProviderConfig` of whatever
namespace held the corresponding `MySQLInstance` at creation time.

> Note that this model currently only allows for a single `ProviderConfig` per
> namespace. However, future Crossplane releases should allow for defining a set
> of `ProviderConfig` that can be selected from using [Multiple Source Field
> patching].
> Note that this model only allows for a single `ProviderConfig` per
> namespace. Future Crossplane releases should allow for defining a set
> of `ProviderConfig` that you can pick from using [Multiple Source Field
> patching].

### Policy Enforcement with Open Policy Agent
<!-- vale Google.Headings = NO -->
### Policy enforcement with Open Policy Agent
<!-- vale Google.Headings = YES -->

In some Crossplane deployment models, only using composition and RBAC to define
policy won't be flexible enough. However, because Crossplane brings
policy isn't flexible enough. Because Crossplane brings
management of external infrastructure to the Kubernetes API, it's well suited
to integrate with other projects in the cloud native ecosystem. Organizations
and individuals that need a more robust policy engine, or just prefer a more
general language for defining policy, often turn to [Open Policy Agent] (OPA).
general language for defining policy, often turn to [Open Policy Agent].
OPA allows platform builders to write custom logic in [Rego], a domain specific
language. Writing policy in this manner allows for not only incorporating the
information available in the specific resource being evaluated, but also using
information available in the specific resource that OPA evaluates, but also using
other state represented in the cluster. Crossplane users typically install OPA
[Gatekeeper] to make policy management as streamlined as possible.

> A live demo of using OPA with Crossplane can be viewed [here].
> Watch a live demo of using OPA with Crossplane [here].

## Multi-Cluster Multi-Tenancy
## Multi-cluster multi-tenancy

Organizations that deploy Crossplane across many clusters typically take
Organizations that deploy Crossplane across multiple clusters typically take
advantage of two major features that make managing multiple control planes much
simpler.

### Reproducible Platforms with Configuration Packages
### Reproducible platforms with configuration packages

[Configuration packages] allow platform builders to package their XRDs and
Compositions into [OCI images] that can be distributed via any OCI compliant
Compositions into [OCI images] that distribute via any OCI compliant
image registry. These packages can also declare dependencies on providers,
meaning that a single package can declare all of the granular managed resources,
the controllers that must be deployed to reconcile them, and the abstract types
meaning that a single package can declare all the granular managed resources,
the controllers that must deploy to reconcile them, and the abstract types
that expose the underlying resources using composition.

Organizations with many Crossplane deployments utilize Configuration packages to
Organizations with several Crossplane deployments use Configuration packages to
<!-- vale alex.Condescending = NO -->
reproduce their platform in each cluster. This can be as simple as installing
<!-- vale alex.Condescending = YES -->
Expand All @@ -288,19 +290,19 @@ alongside it.
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --set configuration.packages='{"registry.upbound.io/xp/getting-started-with-aws:latest"}'
```

### Control Plane of Control Planes
### Control plane of control planes

Taking the multi-cluster multi-tenancy model one step further, some
organizations opt to manage their many Crossplane clusters using a single
organizations opt to manage their multiple Crossplane clusters using a single
central Crossplane control plane. This requires setting up the central cluster,
then using a provider to spin up new clusters (such as an [EKS Cluster] using
[provider-aws]), then using [provider-helm] to install Crossplane into the new
remote cluster, potentially bundling a common Configuration package into each
install using the method described above.
remote cluster, bundling a common Configuration package into each
install using the preceding method.

This advanced pattern allows for full management of Crossplane clusters using
Crossplane itself, and when done properly, is a scalable solution to providing
dedicated control planes to many tenants within a single organization.
Crossplane itself, and is a scalable solution to providing
dedicated control planes to multiple tenants in a single organization.


<!-- Named Links -->
Expand Down
Loading