diff --git a/.gitleaksignore b/.gitleaksignore index d34dc1da4b..b3465feeaf 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -103,3 +103,6 @@ fd60bdc4fdfe8b66925db07865cb530eab4978df:docs/docs-content/integrations/kubernet 511e735952ff4babb08c522a7febdd856740c3f9:docs/docs-content/vertex/system-management/reverse-proxy.md:private-key:141 511e735952ff4babb08c522a7febdd856740c3f9:docs/docs-content/vertex/system-management/reverse-proxy.md:private-key:167 511e735952ff4babb08c522a7febdd856740c3f9:docs/docs-content/vertex/system-management/reverse-proxy.md:private-key:239 +9e62b4b635976b0ab93d4dddcf29d33365664091:docs/docs-content/integrations/kubernetes.md:generic-api-key:391 +9e62b4b635976b0ab93d4dddcf29d33365664091:docs/docs-content/integrations/kubernetes.md:generic-api-key:759 +9e62b4b635976b0ab93d4dddcf29d33365664091:docs/docs-content/integrations/kubernetes.md:generic-api-key:1125 diff --git a/docs/docs-content/integrations/k3s.md b/docs/docs-content/integrations/k3s.md index 6333b17511..9ed71d8d8d 100644 --- a/docs/docs-content/integrations/k3s.md +++ b/docs/docs-content/integrations/k3s.md @@ -25,6 +25,151 @@ the respective owner. Once we stop supporting the minor version, we initiate the + + +### Prerequisites + +- An edge device with AMD64 processor architecture or a Palette Virtual Cluster. +- A minimum of 2 CPU cores and 1 GB memory. + +### Parameters + +Since you can deploy both virtual clusters and Edge clusters using K3s, you have different configuration options +depending on the cluster type. + + + + + +| **Parameter** | **Description** | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `cluster.config.cluster-cidr` | Specifies the CIDR range that can be used by pods in the cluster. | +| `cluster.config.service-cidr` | Specifies the CIDR range that can be used by services in the cluster. | +| `kube-apiserver-arg` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port. | +| `kube-controller-manager-arg` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `kubelet-arg` | This parameter contains extra arguments for Kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the K3s pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-oidc-identity-provider-for-edge). | + +You can add cloud-init stages, which allow you to customize your instances declaratively. The cloud-init stages are +exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open source project. For more information, +check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) reference. + + + + + +Since you are setting up a virtual cluster inside another Kubernetes cluster, you can configure its pods and services +differently than the host cluster. The default configuration file you get includes parameters that offer you a higher +degree of customization. These configuration parameters are exposed in the cluster group settings page. + +| **Parameter** | **Description** | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enableHA` | Determines whether the control plane is deployed in high availability mode. If you set this parameter to true, make sure to adjust the number of replicas and use an external datastore. | +| `defaultImageRegistry` | Specifies the default registry from which images are pulled. The value of this parameter will be prepended to all deployed virtual cluster images. If an image has already been deployed as part of the virtual cluster, the deployed images within the virtual cluster will not be rewritten. | +| `sync` | Specifies which Kubernetes resources are synced between the virtual and host clusters. | +| `storage` | Specifies storage settings such as persistence and PVC size. By default, storage of the virtual cluster uses the same storage class as the host cluster, but you can also optionally specify a different storage class. | +| `ingress` | Configures the ingress resource that allows you to access the virtual cluster. | + + + + + +### Usage + +K3s is available for Edge host deployments as well as virtual clusters that you can create from cluster groups. Refer to +the +[Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) +guide and the [Create and Manage Cluster Groups](../clusters/cluster-groups/create-cluster-group.md) guide for more +information. + +:::info + +In order to use K3s as part of an Edge deployment, you need to go through the EdgeForge process and specify K3s as your +intended Kubernetes distribution when you build your OS image. For more information, refer to the +[EdgeForge Workflow](../clusters/edge/edgeforge-workflow/) guide. + +::: + +#### Configure OIDC Identity Provider for Edge + +You can modify the configuration file to configure your Edge cluster to use an OpenID Connect (OIDC) Identity Provider +(IDP) for authentication. You can use a custom third-party IDP, such as Okta, or use Palette as your IDP. + +When you add the K3s pack to a cluster profile, Palette displays the OIDC IDP options listed below: + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to + specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in + [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper + permissions to view and access the project's resources is able to use kubectl CLI to access cluster. This setting + displays in the YAML file as `palette`. When you select **Palette**, all you have to do to enable OIDC for your + cluster is create role bindings to configure authorization. You do not need to provide extra parameters such as + `oidc-issuer-url` as you need to when you configure a custom OIDC provider. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure + OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose + **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more + information, check out the [SSO Setup](../user-management/saml-sso/saml-sso.md) guide. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes +role to users and groups, refer to +[Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +:::warning + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will +not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC +authentication and not SAML authentication. + +::: + +To configure a custom OIDC IDP, choose **Custom** when adding the K3s pack to your profile, and then follow these steps: + +1. Add the following OIDC parameters to the `kube-apiserver-arg` section of your configuration file for your Kubernetes + layer when creating a cluster profile. + + ```yaml + cluster: + config: + kube-apiserver-arg: + - oidc-issuer-url="provider URL" + - oidc-client-id="client-id" + - oidc-groups-claim="groups" + - oidc-username-claim="email" + ``` + +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes YAML file and replace the + placeholders with your third-party OIDC IDP details. The `clientConfig` section must be placed at the root level of + the YAML file. + ```yaml + clientConfig: + oidc-issuer-url: "OIDC-ISSUER-URL" + oidc-client-id: "OIDC-CLIENT-ID" + oidc-client-secret: "OIDC-CLIENT-SECRET" + oidc-extra-scope: profile,email,openid + ``` + +After you have configured the IDP for authentication, you can proceed to create role bindings to configure authorization +in your cluster. Refer to [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings) +for more guidance. + +#### Configure OIDC Identity Provider for Palette Virtual Clusters + +If you are using K3s in a virtual cluster inside of a cluster group, you can also configure OIDC for your cluster. Refer +to [Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md) for +more guidance. + +#### Add a Certificate for Reverse Proxy + +You can use a reverse proxy with a K3s Kubernetes cluster. The reverse proxy allows you to connect to the cluster API of +a Palette-managed Kubernetes cluster in private networks or clusters configured with private API endpoints. For more +information, refer to the [Spectro Proxy](frp.md) pack guide. + + + ### Prerequisites diff --git a/docs/docs-content/integrations/kubernetes-edge.md b/docs/docs-content/integrations/kubernetes-edge.md index 3de7cce271..f7dce32322 100644 --- a/docs/docs-content/integrations/kubernetes-edge.md +++ b/docs/docs-content/integrations/kubernetes-edge.md @@ -82,11 +82,12 @@ four months. Once we stop supporting the minor version, we initiate the deprecat # Versions Supported - + + ## Prerequisites -- A minimum of 2 CPU and 4GB Memory. +- A minimum of 2 CPU and 4 GB Memory. ## Parameters @@ -428,27 +429,412 @@ Follow these steps to configure a third-party OIDC IDP. 1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. + ```yaml + cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` + +2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes configuration file. + + ```yaml + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` + +3. Provide third-party OIDC IDP details. + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map +many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To +learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes +pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + + + + + +## Prerequisites + +- A minimum of 2 CPU and 4 GB Memory. + +## Parameters + +| Parameter | Description | +| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `cluster.config.clusterConfiguration.apiServer.extraArgs` | This parameter contains extra arguments for the Kubernetes API server, such as enabling audit logging, enabling certain authorization modes, and setting profiling and secure-port. | +| `cluster.config.clusterConfiguration.apiServer.extraVolumes` | This parameter describes extra volumes for the Kubernetes API server, such as `audit-log` and `audit-policy`. | +| `cluster.config.clusterConfiguration.controllerManager.extraArgs` | This parameter describes extra arguments for the Kubernetes Controller Manager, such as enabling certain feature gates and setting profiling. | +| `cluster.config.clusterConfiguration.etcd.local.dataDir` | This parameter specifies the data directory for etcd, the distributed key-value store that Kubernetes uses to persist cluster state. | +| `cluster.config.clusterConfiguration.networking.podSubnet` | The IP subnet range to assign to pods. Default: 192.168.0.0/16 | +| `cluster.config.clusterConfiguration.networking.serviceSubnet` | The IP subnet range to assign to services. Default: 192.169.0.0/16 | +| `cluster.config.clusterConfiguration.scheduler.extraArgs` | This parameter contains extra arguments for the Kubernetes scheduler, such as disabling profiling. | +| `cluster.config.initConfiguration.nodeRegistration.kubeletExtraArgs` | This parameter contains extra arguments for kubelet during node registration, such as setting feature gates, protecting kernel defaults, and disabling the read-only port. | +| `pack.palette.config.oidc.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK-E pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](#configure-custom-oidc). | + +You can add cloud-init stages exposed by [Kairos](https://kairos.io/docs/architecture/cloud-init/), an open-source +project. For more information, check out the [Cloud Init Stages](../clusters/edge/edge-configuration/cloud-init.md) +reference. + +You can also use pack settings described in the [Palette eXtended Kubernetes](kubernetes.md) guide. + +## Usage + +The Kubernetes configuration file is where you can do the following: + +- Manually configure a third-party OIDC IDP. For more information, check out + [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more + information, refer to the [Spectro Proxy](frp.md) guide. + +#### Configuration Changes + +The PXK-E Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the +`identityProvider` parameter. + +```yaml +pack: + palette: + config: + oidc: + identityProvider: +``` + +#### Example Kubernetes Configuration File + ```yaml cluster: config: | clusterConfiguration: apiServer: extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" + advertise-address: "0.0.0.0" + anonymous-auth: "true" + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + authorization-mode: RBAC,Node + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + disable-admission-plugins: AlwaysAdmit + enable-admission-plugins: AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction + profiling: "false" + secure-port: "6443" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + extraVolumes: + - hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + name: audit-log + pathType: DirectoryOrCreate + - hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + name: audit-policy + pathType: File + readOnly: true + timeoutForControlPlane: 10m0s + controllerManager: + extraArgs: + feature-gates: RotateKubeletServerCertificate=true + pod-eviction-timeout: 1m0s + profiling: "false" + terminated-pod-gc-threshold: "25" + use-service-account-credentials: "true" + dns: {} + kubernetesVersion: v1.26.4 + etcd: + local: + dataDir: "/etc/kubernetes/etcd" + extraArgs: + listen-client-urls: "https://0.0.0.0:2379" + networking: + podSubnet: 192.168.0.0/16 + serviceSubnet: 192.169.0.0/16 + scheduler: + extraArgs: + profiling: "false" + initConfiguration: + localAPIEndpoint: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + joinConfiguration: + discovery: {} + nodeRegistration: + kubeletExtraArgs: + event-qps: "0" + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: "true" + read-only-port: "0" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +stages: + initramfs: + - sysctl: + vm.overcommit_memory: 1 + kernel.panic: 10 + kernel.panic_on_oops: 1 + commands: + - ln -s /etc/kubernetes/admin.conf /run/kubeconfig + files: + - path: /etc/hosts + permission: "0644" + content: | + 127.0.0.1 localhost + - path: "/etc/kubernetes/audit-policy.yaml" + owner_string: "root" + permission: 0600 + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: None + users: ["system:kube-proxy"] + verbs: ["watch"] + resources: + - group: "" # core + resources: ["endpoints", "services", "services/status"] + - level: None + users: ["system:unsecured"] + namespaces: ["kube-system"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["configmaps"] + - level: None + users: ["kubelet"] # legacy kubelet identity + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + userGroups: ["system:nodes"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["nodes", "nodes/status"] + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["endpoints"] + - level: None + users: ["system:apiserver"] + verbs: ["get"] + resources: + - group: "" # core + resources: ["namespaces", "namespaces/status", "namespaces/finalize"] + - level: None + users: ["cluster-autoscaler"] + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" # core + resources: ["configmaps", "endpoints"] + # Don't log HPA fetching metrics. + - level: None + users: + - system:kube-controller-manager + verbs: ["get", "list"] + resources: + - group: "metrics.k8s.io" + # Don't log these read-only URLs. + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + # Don't log events requests. + - level: None + resources: + - group: "" # core + resources: ["events"] + # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes + - level: Request + users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + - level: Request + userGroups: ["system:nodes"] + verbs: ["update","patch"] + resources: + - group: "" # core + resources: ["nodes/status", "pods/status"] + omitStages: + - "RequestReceived" + # deletecollection calls can be large, don't log responses for expected namespace deletions + - level: Request + users: ["system:serviceaccount:kube-system:namespace-controller"] + verbs: ["deletecollection"] + omitStages: + - "RequestReceived" + # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, + # so only log at the Metadata level. + - level: Metadata + resources: + - group: "" # core + resources: ["secrets", "configmaps"] + - group: authentication.k8s.io + resources: ["tokenreviews"] + omitStages: + - "RequestReceived" + # Get repsonses can be large; skip them. + - level: Request + verbs: ["get", "list", "watch"] + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for known APIs + - level: RequestResponse + resources: + - group: "" # core + - group: "admissionregistration.k8s.io" + - group: "apiextensions.k8s.io" + - group: "apiregistration.k8s.io" + - group: "apps" + - group: "authentication.k8s.io" + - group: "authorization.k8s.io" + - group: "autoscaling" + - group: "batch" + - group: "certificates.k8s.io" + - group: "extensions" + - group: "metrics.k8s.io" + - group: "networking.k8s.io" + - group: "policy" + - group: "rbac.authorization.k8s.io" + - group: "settings.k8s.io" + - group: "storage.k8s.io" + omitStages: + - "RequestReceived" + # Default level for all other requests. + - level: Metadata + omitStages: + - "RequestReceived" +pack: + palette: + config: + oidc: + identityProvider: palette ``` +### Configure OIDC Identity Provider + +The OIDC IDP feature offers the convenience of managing OIDC at the Kubernetes layer. The OIDC IDP feature is +particularly useful for environments that do not have their own IDP configured. In this scenario, you can leverage +Palette as an IDP without having to configure a third-party IDP. We also support the ability to take advantage of other +OIDC providers by making it possible for you to configure OIDC at the tenant level. For additional flexibility, if you +wish to use a different IDP than the one configured at the tenant level, you can select a different IDP by adding the +OIDC configuration to your cluster profile. + +When you add the PXK-E pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes +role to users and groups, refer to +[Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +You can create a role binding that maps individual users or groups assigned within the OIDC provider's configuration to +a role. To learn more, review [Use RBAC with OIDC](kubernetes-edge.md#use-rbac-with-oidc). You can also configure OIDC +for virtual clusters. For guidance, refer to +[Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to + specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in + [Configure Custom OIDC](kubernetes-edge.md#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper + permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting + displays in the YAML file as `palette`. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure + OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose + **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more + information, check out the [SSO Setup](../user-management/saml-sso/saml-sso.md) guide. + +:::info + +If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option will +not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC +authentication and not SAML authentication. + +::: + +### Configure Custom OIDC + +Follow these steps to configure a third-party OIDC IDP. + +1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a + cluster profile. + + ```yaml + cluster: + config: | + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` + 2. Add the following `clientConfig` section that contains OIDC parameters to your Kubernetes configuration file. -```yaml -clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` 3. Provide third-party OIDC IDP details. @@ -471,7 +857,7 @@ In this example, Palette is used as the IDP, and all users in the `dev-east-2` w ## Prerequisites -- A minimum of 2 CPU and 4GB Memory. +- A minimum of 2 CPU and 4 GB Memory. ## Parameters @@ -813,28 +1199,28 @@ Follow these steps to configure a third-party OIDC IDP. 1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. -```yaml -cluster: - config: - clusterConfiguration: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml + cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. -```yaml -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` 3. Provide third-party OIDC IDP details. @@ -857,7 +1243,7 @@ In this example, Palette is used as the IDP, and all users in the `dev-east-2` w ## Prerequisites -- A minimum of 2 CPU and 4GB Memory. +- A minimum of 2 CPU and 4 GB Memory. ## Parameters @@ -1199,28 +1585,28 @@ Follow these steps to configure a third-party OIDC IDP. 1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. -```yaml -cluster: - config: - clusterConfiguration: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml + cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. -```yaml -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` 3. Provide third-party OIDC IDP details. @@ -1243,7 +1629,7 @@ In this example, Palette is used as the IDP, and all users in the `dev-east-2` w ## Prerequisites -- A minimum of 2 CPU and 4GB Memory. +- A minimum of 2 CPU and 4 GB Memory. ## Parameters @@ -1586,28 +1972,28 @@ Follow these steps to configure a third-party OIDC IDP. 1. Add the following OIDC parameters to the `apiServer.extraArgs` section of your Kubernetes YAML file when creating a cluster profile. -```yaml -cluster: - config: - clusterConfiguration: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml + cluster: + config: + clusterConfiguration: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Add the following `kubeadmconfig.clientConfig` section that contains OIDC parameters to your Kubernetes YAML file. -```yaml -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` 3. Provide third-party OIDC IDP details. @@ -1641,8 +2027,6 @@ All versions less than v1.25.x are considered deprecated. Upgrade to a newer ver You can reference Kubernetes in Terraform with the following code snippet. -
- ```hcl data "spectrocloud_registry" "public_registry" { name = "Public Repo" diff --git a/docs/docs-content/integrations/kubernetes.md b/docs/docs-content/integrations/kubernetes.md index f1444ded1f..aa60fa36dc 100644 --- a/docs/docs-content/integrations/kubernetes.md +++ b/docs/docs-content/integrations/kubernetes.md @@ -71,20 +71,20 @@ four months. Once we stop supporting the minor version, we initiate the deprecat - + ## Prerequisites -- A minimum of 4 CPU and 4GB Memory. +- A minimum of 4 CPU and 4 GB Memory. - Operating System (OS) dependencies as listed in the table. -| OS Distribution | OS Version | Supports Kubernetes 1.27.x | -| --------------- | ---------- | -------------------------- | -| CentOS | 7.7 | ✅ | -| Ubuntu | 22.04 | ✅ | -| Ubuntu | 20.04 | ❌ | -| Ubuntu | 18.04 | ❌ | + | OS Distribution | OS Version | Supports Kubernetes 1.27.x | + | --------------- | ---------- | -------------------------- | + | CentOS | 7.7 | ✅ | + | Ubuntu | 22.04 | ✅ | + | Ubuntu | 20.04 | ❌ | + | Ubuntu | 18.04 | ❌ | ## Parameters @@ -340,26 +340,394 @@ to learn more. Click the **Amazon EKS** tab for steps to configure OIDC for EKS 1. Add the following parameters to your Kubernetes YAML file when creating a cluster profile. + ```yaml + kubeadmconfig: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` + +2. Under the `clientConfig` parameter section of Kubernetes YAML file, uncomment the `oidc-` configuration lines. + + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` + + + + + +Follow these steps to configure OIDC for managed EKS clusters. + +1. In the Kubernetes pack, uncomment the lines in the `oidcIdentityProvider` parameter section of the Kubernetes pack, + and enter your third-party provider details. + + ```yaml + oidcIdentityProvider: + identityProviderConfigName: "Spectro-docs" + issuerUrl: "issuer-url" + clientId: "user-client-id-from-Palette" + usernameClaim: "email" + usernamePrefix: "-" + groupsClaim: "groups" + groupsPrefix: "" + requiredClaims: + ``` + +2. Under the `clientConfig` parameter section of Kubernetes pack, uncomment the `oidc-` configuration lines. + + ```yaml + clientConfig: + oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" + oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" + oidc-client-secret: 1gsranjjmdgahm10j8r6m47ejokm9kafvcbhi3d48jlc3rfpprhv + oidc-extra-scope: profile,email + ``` + +3. Provide third-party OIDC IDP details. + + + + + +### Use RBAC with OIDC + +You can create a role binding that uses individual users as the subject or specify a group name as the subject to map +many users to a role. The group name is the group assigned in the OIDC provider's configuration. Below is an example. To +learn more, review [Create Role Bindings](../clusters/cluster-management/cluster-rbac.md#create-role-bindings). + +Assume you created a group named `dev-east-2` within an OIDC provider. If you configure the host cluster's Kubernetes +pack with all the correct OIDC settings, you could then create a role binding for the `dev-east-2` group. + +In this example, Palette is used as the IDP, and all users in the `dev-east-2` would inherit the `cluster-admin` role. + +![A subject of the type group is assigned as the subject in a RoleBinding](/clusters_cluster-management_cluster-rbac_cluster-subject-group.png) + +### Custom MAAS Endpoint + +You can specify a custom MAAS endpoint and port that instructs Palette to direct all MAAS API requests to the provided +endpoint URL. Use the `cloud.maas.customEndpoint` and `cloud.maas.customEndpointPort` parameters to specify the custom +MAAS API URL and port. This is useful in scenarios where the MAAS API endpoint is not resolvable outside of the MAAS +network. + +The following example shows how to specify a custom MAAS endpoint and port in the Kubernetes YAML file. Make sure the +`cloud.maas` section is at the same level as the `pack` section. + +```yaml hideClipboard {10-14} +pack: + k8sHardening: True + podCIDR: "192.168.0.0/16" + serviceClusterIpRange: "10.96.0.0/12" + palette: + config: + dashboard: + identityProvider: palette + +cloud: + maas: + customEndpoint: "maas-api.example.maas.org" + customEndpointPort: "6443" +``` + + + + + +## Prerequisites + +- A minimum of 4 CPU and 4 GB Memory. + +- Operating System (OS) dependencies as listed in the table. + + | OS Distribution | OS Version | Supports Kubernetes 1.27.x | + | --------------- | ---------- | -------------------------- | + | CentOS | 7.7 | ✅ | + | Ubuntu | 22.04 | ✅ | + | Ubuntu | 20.04 | ❌ | + | Ubuntu | 18.04 | ❌ | + +## Parameters + +| Parameter | Description | +| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `pack.podCIDR` | The CIDR range for Pods in the cluster. This should match the networking layer property. Default: `192.168.0.0/16` | +| `pack.serviceClusterIpRange` | The CIDR range for services in the cluster. This should not overlap with any IP ranges assigned to nodes or pods. Default: `10.96.0.0/12` | +| `pack.serviceDomain` | The cluster DNS service domain. Default: `cluster.local`. To change the default, you must add this parameter to the Kubernetes YAML file at cluster creation and specify the cluster DNS service domain to use. This value cannot be changed after cluster creation is complete. Refer to the [Change Cluster DNS Service Domain](kubernetes.md?versions=k8s_v1.27#change-cluster-dns-service-domain) section. | +| `pack.palette.config.dashboard.identityProvider` | Dynamically enabled OpenID Connect (OIDC) Identity Provider (IDP) setting based on your UI selection when you add the PXK pack to your profile. This parameter appears in the YAML file after you make a selection. Refer to [Configure OIDC Identity Provider](kubernetes.md#configure-oidc-identity-provider). | +| `kubeadmconfig.apiServer.extraArgs` | A list of additional apiServer flags you can set. | +| `kubeadmconfig.apiServer.extraVolumes` | A list of additional volumes to mount on the apiServer. | +| `kubeadmconfig.controllerManager.extraArgs` | A list of additional ControllerManager flags to set. | +| `kubeadmconfig.scheduler.extraArgs` | A list of additional Kube scheduler flags to set. | +| `kubeadmconfig.kubeletExtraArgs` | A list of kubelet arguments to set and copy to the nodes. | +| `kubeadmconfig.files` | A list of additional files to copy to the nodes. | +| `kubeadmconfig.preKubeadmCommands` | A list of additional commands to invoke **before** running kubeadm commands. | +| `kubeadmconfig.postKubeadmCommands` | A list of additional commands to invoke **after** running kubeadm commands. | +| `kubeadmconfig.clientConfig` | Settings to manually configure OIDC-based authentication when you choose a third-party (Custom) IDP. Refer to [Configure Custom OIDC](#configure-custom-oidc). | +| `cloud.maas.customEndpoint` | The custom MAAS API or DNS endpoint URL to use for the PXK cluster. This parameter is only available for MAAS. | +| `cloud.maas.customEndpointPort` | The custom MAAS API or DNS endpoint port to use for the PXK cluster. This parameter is only available for MAAS. Default value is `6443`. | + +## Usage + +The Kubeadm configuration file is where you can do the following: + +- Change the default `podCIDR` and `serviceClusterIpRange` values. CIDR IPs specified in the configuration file take + precedence over other defined CIDR IPs in your environment. + + As you build your cluster, check that the `podCIDR` value does not overlap with any hosts or with the service network + and the `serviceClusterIpRange` value does not overlap with any IP ranges assigned to nodes or pods. For more + information, refer to the [Clusters](../clusters/clusters.md) guide and + [Cluster Deployment Errors](../troubleshooting/cluster-deployment.md). + +- Change the default cluster DNS service domain from `cluster.local` to a DNS domain that you specify. You can only + change the DNS domain during cluster creation. For more information, refer to + [Change Cluster DNS Service Domain](kubernetes.md?versions=k8s_v1.27#change-cluster-dns-service-domain). + +- Manually configure a third-party OpenID Connect (OIDC) Identity Provider (IDP). For more information, check out + [Configure Custom OIDC](#configure-custom-oidc). + +- Add a certificate for the Spectro Proxy pack if you want to use a reverse proxy with a Kubernetes cluster. For more + information, refer to the [Spectro Proxy](frp.md) guide. + +### Change Cluster DNS Service Domain + +The `pack.serviceDomain` parameter with default value `cluster.local` is not visible in the Kubernetes YAML file, and +its value can only be changed at cluster creation. To change the value, you must add `serviceDomain: "cluster.local"` to +the Kubernetes YAML file when you create a cluster, and specify the service domain you want to use. + +```yaml hideClipboard +pack: + k8sHardening: True + podCIDR: "172.16.0.0/16" + serviceClusterIPRange: "10.96.0.0/12" + serviceDomain: "" +``` + +:::warning + +You can only specify the service domain at cluster creation. After cluster creation completes, you cannot update the +value. Attempting to update it results in the error `serviceDomain update is forbidden for existing cluster`. + +::: + +For more information about networking configuration with DNS domains, refer to the Kubernetes +[Networking](https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-Networking) +API documentation. + +### Configuration Changes + +The PXK Kubeadm configuration is updated to dynamically enable OIDC based on your IDP selection by adding the +`identityProvider` parameter. + ```yaml +palette: + config: + dashboard: + identityProvider: +``` + +### Example Kubeadm Configuration File + +```yaml hideClipboard +pack: + k8sHardening: True + podCIDR: "192.168.0.0/16" + serviceClusterIpRange: "10.96.0.0/12" + palette: + config: + dashboard: + identityProvider: palette kubeadmconfig: apiServer: extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" + secure-port: "6443" + anonymous-auth: "true" + profiling: "false" + disable-admission-plugins: "AlwaysAdmit" + default-not-ready-toleration-seconds: "60" + default-unreachable-toleration-seconds: "60" + enable-admission-plugins: "AlwaysPullImages,NamespaceLifecycle,ServiceAccount,NodeRestriction,PodSecurity" + admission-control-config-file: "/etc/kubernetes/pod-security-standard.yaml" + audit-log-path: /var/log/apiserver/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + authorization-mode: RBAC,Node + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" + extraVolumes: + - name: audit-log + hostPath: /var/log/apiserver + mountPath: /var/log/apiserver + pathType: DirectoryOrCreate + - name: audit-policy + hostPath: /etc/kubernetes/audit-policy.yaml + mountPath: /etc/kubernetes/audit-policy.yaml + readOnly: true + pathType: File + - name: pod-security-standard + hostPath: /etc/kubernetes/pod-security-standard.yaml + mountPath: /etc/kubernetes/pod-security-standard.yaml + readOnly: true + pathType: File + controllerManager: + extraArgs: + profiling: "false" + terminated-pod-gc-threshold: "25" + pod-eviction-timeout: "1m0s" + use-service-account-credentials: "true" + feature-gates: "RotateKubeletServerCertificate=true" + scheduler: + extraArgs: + profiling: "false" + kubeletExtraArgs: + read-only-port: "0" + event-qps: "0" + feature-gates: "RotateKubeletServerCertificate=true" + protect-kernel-defaults: "true" + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256" + files: + - path: hardening/audit-policy.yaml + targetPath: /etc/kubernetes/audit-policy.yaml + targetOwner: "root:root" + targetPermissions: "0600" + - path: hardening/90-kubelet.conf + targetPath: /etc/sysctl.d/90-kubelet.conf + targetOwner: "root:root" + targetPermissions: "0600" + - targetPath: /etc/kubernetes/pod-security-standard.yaml + targetOwner: "root:root" + targetPermissions: "0600" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "baseline" + enforce-version: "v1.26" + audit: "baseline" + audit-version: "v1.26" + warn: "restricted" + warn-version: "v1.26" + audit: "restricted" + audit-version: "v1.26" + exemptions: + # Array of authenticated usernames to exempt. + usernames: [] + # Array of runtime class names to exempt. + runtimeClasses: [] + # Array of namespaces to exempt. + namespaces: [kube-system] + + preKubeadmCommands: + - 'echo "====> Applying kernel parameters for Kubelet"' + - 'sysctl -p /etc/sysctl.d/90-kubelet.conf' + postKubeadmCommands: + - 'echo "List of post kubeadm commands to be executed"' + + # Client configuration to add OIDC based authentication flags in kubeconfig + #clientConfig: + #oidc-issuer-url: "{{ .spectro.pack.kubernetes.kubeadmconfig.apiServer.extraArgs.oidc-issuer-url }}" + #oidc-client-id: "{{ .spectro.pack.kubernetes.kubeadmconfig.apiServer.extraArgs.oidc-client-id }}" + #oidc-client-secret: yourSecretClientSecretGoesHere + #oidc-extra-scope: profile,email ``` +### Configure OIDC Identity Provider + +Platforms that use PXK can use the OIDC IDP feature, which offers the convenience of managing OIDC at the Kubernetes +layer. The OIDC IDP feature is particularly useful for environments that do not have their own IDP configured. In this +scenario, you can leverage Palette as an IDP without having to configure a third-party IDP. We also support the ability +to take advantage of other OIDC providers by making it possible for you to configure OIDC at the tenant level. For +additional flexibility, if you wish to use a different IDP than the one configured at the tenant level, you can select a +different IDP by adding the OIDC configuration to your cluster profile. + +When you add the PXK pack to a cluster profile, Palette displays the OIDC IDP options listed below. + +All the options require you to map a set of users or groups to a Kubernetes RBAC role. To learn how to map a Kubernetes +role to users and groups, refer to +[Create Role Bindings](/clusters/cluster-management/cluster-rbac#create-role-bindings). You can also configure OIDC for +virtual clusters. For guidance, refer to +[Configure OIDC for a Virtual Cluster](../clusters/palette-virtual-clusters/configure-oidc-virtual-cluster.md). + +- **None**: This setting does not require OIDC configuration for the cluster. It displays in the YAML file as `noauth`. + + :::warning + + We do not recommend choosing **None** in a production environment, as it may disable authentication for add-ons that + rely on OIDC. + + ::: + +- **Custom**: This is the default setting and does not require OIDC configuration. However, if desired, it allows you to + specify a third-party OIDC provider by configuring OIDC statements in the YAML file as described in + [Configure Custom OIDC](#configure-custom-oidc). This setting displays in the YAML file as `none`. + +- **Palette**: This setting makes Palette the IDP. Any user with a Palette account in the tenant and the proper + permissions to view and access the project's resources is able to log into the Kubernetes dashboard. This setting + displays in the YAML file as `palette`. + +- **Inherit from Tenant**: This setting allows you to apply RBAC to multiple clusters and requires you to configure + OpenID Connect (OIDC) in **Tenant Settings**. In Tenant Admin scope, navigate to **Tenant Settings** > **SSO**, choose + **OIDC**, and provide your third-party IDP details. This setting displays in the YAML file as `tenant`. For more + information, check out the [SSO Setup](../user-management/saml-sso/saml-sso.md) guide. + + :::info + + If your IDP uses Security Assertion Markup Language (SAML) authentication, then the **Inherit from Tenant** option + will not work, and you will need to use the **Custom** option instead. This is because Kubernetes supports only OIDC + authentication and not SAML authentication. + + ::: + +### Configure Custom OIDC + +The custom method to configure OIDC and apply RBAC for an OIDC provider can be used for all cloud services except Amazon +Elastic Kubernetes Service (EKS) and [Azure-AKS](../clusters/public-cloud/azure/aks.md). + + + + + +Follow these steps to configure a third-party OIDC IDP. You can apply these steps to all the public cloud providers +except Azure AKS and Amazon EKS clusters. Azure AKS and Amazon EKS require different configurations. AKS requires you to +use Azure Active Directory (AAD) to enable OIDC integration. Refer to +[Enable OIDC in Kubernetes Clusters With Entra ID](../user-management/saml-sso/palette-sso-with-entra-id.md#enable-oidc-in-kubernetes-clusters-with-entra-id) +to learn more. Click the **Amazon EKS** tab for steps to configure OIDC for EKS clusters. + +1. Add the following parameters to your Kubernetes YAML file when creating a cluster profile. + + ```yaml + kubeadmconfig: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` + 2. Under the `clientConfig` parameter section of Kubernetes YAML file, uncomment the `oidc-` configuration lines. -```yaml -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` @@ -370,27 +738,27 @@ Follow these steps to configure OIDC for managed EKS clusters. 1. In the Kubernetes pack, uncomment the lines in the `oidcIdentityProvider` parameter section of the Kubernetes pack, and enter your third-party provider details. -```yaml -oidcIdentityProvider: - identityProviderConfigName: "Spectro-docs" - issuerUrl: "issuer-url" - clientId: "user-client-id-from-Palette" - usernameClaim: "email" - usernamePrefix: "-" - groupsClaim: "groups" - groupsPrefix: "" - requiredClaims: -``` + ```yaml + oidcIdentityProvider: + identityProviderConfigName: "Spectro-docs" + issuerUrl: "issuer-url" + clientId: "user-client-id-from-Palette" + usernameClaim: "email" + usernamePrefix: "-" + groupsClaim: "groups" + groupsPrefix: "" + requiredClaims: + ``` 2. Under the `clientConfig` parameter section of Kubernetes pack, uncomment the `oidc-` configuration lines. -```yaml -clientConfig: - oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" - oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" - oidc-client-secret: 1gsranjjmdgahm10j8r6m47ejokm9kafvcbhi3d48jlc3rfpprhv - oidc-extra-scope: profile,email -``` + ```yaml + clientConfig: + oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" + oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" + oidc-client-secret: 1gsranjjmdgahm10j8r6m47ejokm9kafvcbhi3d48jlc3rfpprhv + oidc-extra-scope: profile,email + ``` 3. Provide third-party OIDC IDP details. @@ -443,16 +811,16 @@ cloud: ## Prerequisites -- A minimum of 4 CPU and 4GB Memory. +- A minimum of 4 CPU and 4 GB Memory. - Operating System (OS) dependencies as listed in the table. -| OS Distribution | OS Version | Supports Kubernetes 1.27.x | -| --------------- | ---------- | -------------------------- | -| CentOS | 7.7 | ✅ | -| Ubuntu | 22.04 | ✅ | -| Ubuntu | 20.04 | ❌ | -| Ubuntu | 18.04 | ❌ | + | OS Distribution | OS Version | Supports Kubernetes 1.27.x | + | --------------- | ---------- | -------------------------- | + | CentOS | 7.7 | ✅ | + | Ubuntu | 22.04 | ✅ | + | Ubuntu | 20.04 | ❌ | + | Ubuntu | 18.04 | ❌ | ## Parameters @@ -706,26 +1074,26 @@ to learn more. Click the **Amazon EKS** tab for steps to configure OIDC for EKS 1. Add the following parameters to your Kubernetes YAML file when creating a cluster profile. -```yaml -kubeadmconfig: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml + kubeadmconfig: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Under the `clientConfig` parameter section of Kubernetes YAML file, uncomment the `oidc-` configuration lines. -```yaml -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` @@ -736,27 +1104,27 @@ Follow these steps to configure OIDC for managed EKS clusters. 1. In the Kubernetes pack, uncomment the lines in the `oidcIdentityProvider` parameter section of the Kubernetes pack, and enter your third-party provider details. -```yaml -oidcIdentityProvider: - identityProviderConfigName: "Spectro-docs" - issuerUrl: "issuer-url" - clientId: "user-client-id-from-Palette" - usernameClaim: "email" - usernamePrefix: "-" - groupsClaim: "groups" - groupsPrefix: "" - requiredClaims: -``` + ```yaml + oidcIdentityProvider: + identityProviderConfigName: "Spectro-docs" + issuerUrl: "issuer-url" + clientId: "user-client-id-from-Palette" + usernameClaim: "email" + usernamePrefix: "-" + groupsClaim: "groups" + groupsPrefix: "" + requiredClaims: + ``` 2. Under the `clientConfig` parameter section of Kubernetes pack, uncomment the `oidc-` configuration lines. -```yaml -clientConfig: - oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" - oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" - oidc-client-secret: 1gsranjjmdgahm10j8r6m47ejokm9kafvcbhi3d48jlc3rfpprhv - oidc-extra-scope: profile,email -``` + ```yaml + clientConfig: + oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" + oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" + oidc-client-secret: 1gsranjjmdgahm10j8r6m47ejokm9kafvcbhi3d48jlc3rfpprhv + oidc-extra-scope: profile,email + ``` 3. Provide third-party OIDC IDP details. @@ -783,16 +1151,16 @@ In this example, Palette is used as the IDP, and all users in the `dev-east-2` w ## Prerequisites -- A minimum of 4 CPU and 4GB Memory. +- A minimum of 4 CPU and 4 GB Memory. - Operating System (OS) dependencies as listed in the table. -| OS Distribution | OS Version | Supports Kubernetes 1.26.x | -| --------------- | ---------- | -------------------------- | -| CentOS | 7.7 | ✅ | -| Ubuntu | 22.04 | ✅ | -| Ubuntu | 20.04 | ❌ | -| Ubuntu | 18.04 | ❌ | + | OS Distribution | OS Version | Supports Kubernetes 1.26.x | + | --------------- | ---------- | -------------------------- | + | CentOS | 7.7 | ✅ | + | Ubuntu | 22.04 | ✅ | + | Ubuntu | 20.04 | ❌ | + | Ubuntu | 18.04 | ❌ | ## Parameters @@ -871,8 +1239,6 @@ palette: identityProvider: ``` -
- ### Example Kubeadm Configuration File ```yaml @@ -972,7 +1338,7 @@ kubeadmconfig: preKubeadmCommands: - 'echo "====> Applying kernel parameters for Kubelet"' - - 'sysctl -p /etc/sysctl.d/90-kubelet.conf' + - 'sysctl --load /etc/sysctl.d/90-kubelet.conf' postKubeadmCommands: - 'echo "List of post kubeadm commands to be executed"' @@ -1048,26 +1414,26 @@ to learn more. Click the **Amazon EKS** tab for steps to configure OIDC for EKS 1. Add the following parameters to your Kubernetes YAML file when creating a cluster profile. -```yaml -kubeadmconfig: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml + kubeadmconfig: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Under the `clientConfig` parameter section of Kubernetes YAML file, uncomment the `oidc-` configuration lines. -```yaml -ubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml + ubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` @@ -1078,27 +1444,27 @@ Follow these steps to configure OIDC for managed EKS clusters. 1. In the Kubernetes pack, uncomment the lines in the `oidcIdentityProvider` parameter section of the Kubernetes pack, and enter your third-party provider details. -```yaml hideClipboard -oidcIdentityProvider: - identityProviderConfigName: "Spectro-docs" - issuerUrl: "issuer-url" - clientId: "user-client-id-from-Palette" - usernameClaim: "email" - usernamePrefix: "-" - groupsClaim: "groups" - groupsPrefix: "" - requiredClaims: -``` + ```yaml hideClipboard + oidcIdentityProvider: + identityProviderConfigName: "Spectro-docs" + issuerUrl: "issuer-url" + clientId: "user-client-id-from-Palette" + usernameClaim: "email" + usernamePrefix: "-" + groupsClaim: "groups" + groupsPrefix: "" + requiredClaims: + ``` 2. Under the `clientConfig` parameter section of Kubernetes pack, uncomment the `oidc-` configuration lines. -```yaml hideClipboard -clientConfig: - oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" - oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" - oidc-client-secret: yourSecretClientSecretGoesHere - oidc-extra-scope: profile,email -``` + ```yaml hideClipboard + clientConfig: + oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" + oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" + oidc-client-secret: yourSecretClientSecretGoesHere + oidc-extra-scope: profile,email + ``` 3. Provide third-party OIDC IDP details. @@ -1125,16 +1491,16 @@ In this example, Palette is used as the IDP, and all users in the `dev-east-2` w ## Prerequisites -- A minimum of 4 CPU and 4GB Memory. +- A minimum of 4 CPU and 4 GB Memory. - Operating System (OS) dependencies as listed in the table. -| OS Distribution | OS Version | Supports Kubernetes 1.25.x | -| --------------- | ---------- | -------------------------- | -| CentOS | 7.7 | ✅ | -| Ubuntu | 22.04 | ✅ | -| Ubuntu | 20.04 | ❌ | -| Ubuntu | 18.04 | ❌ | + | OS Distribution | OS Version | Supports Kubernetes 1.25.x | + | --------------- | ---------- | -------------------------- | + | CentOS | 7.7 | ✅ | + | Ubuntu | 22.04 | ✅ | + | Ubuntu | 20.04 | ❌ | + | Ubuntu | 18.04 | ❌ | ## Parameters @@ -1387,26 +1753,26 @@ to learn more. Click the **Amazon EKS** tab for steps to configure OIDC for EKS 1. Add the following parameters to your Kubernetes YAML file when creating a cluster profile. -```yaml hideClipboard -kubeadmconfig: - apiServer: - extraArgs: - oidc-issuer-url: "provider URL" - oidc-client-id: "client-id" - oidc-groups-claim: "groups" - oidc-username-claim: "email" -``` + ```yaml hideClipboard + kubeadmconfig: + apiServer: + extraArgs: + oidc-issuer-url: "provider URL" + oidc-client-id: "client-id" + oidc-groups-claim: "groups" + oidc-username-claim: "email" + ``` 2. Under the `clientConfig` parameter section of Kubernetes YAML file, uncomment the `oidc-` configuration lines. -```yaml hideClipboard -kubeadmconfig: - clientConfig: - oidc-issuer-url: "" - oidc-client-id: "" - oidc-client-secret: "" - oidc-extra-scope: profile,email,openid -``` + ```yaml hideClipboard + kubeadmconfig: + clientConfig: + oidc-issuer-url: "" + oidc-client-id: "" + oidc-client-secret: "" + oidc-extra-scope: profile,email,openid + ``` 3. Provide third-party OIDC IDP details. Refer to the [SAML & SSO Setup](/user-management/saml-sso) for guidance on configuring a third party IDP with Palette. @@ -1420,27 +1786,27 @@ Follow these steps to configure OIDC for managed EKS clusters. 1. In the Kubernetes pack, uncomment the lines in the `oidcIdentityProvider` parameter section of the Kubernetes pack, and enter your third-party provider details. -```yaml - oidcIdentityProvider: hideClipboard - identityProviderConfigName: 'Spectro-docs' - issuerUrl: 'issuer-url' - clientId: 'user-client-id-from-Palette' - usernameClaim: "email" - usernamePrefix: "-" - groupsClaim: "groups" - groupsPrefix: "" - requiredClaims: -``` + ```yaml + oidcIdentityProvider: hideClipboard + identityProviderConfigName: 'Spectro-docs' + issuerUrl: 'issuer-url' + clientId: 'user-client-id-from-Palette' + usernameClaim: "email" + usernamePrefix: "-" + groupsClaim: "groups" + groupsPrefix: "" + requiredClaims: + ``` 2. Under the `clientConfig` parameter section of Kubernetes pack, uncomment the `oidc-` configuration lines. -```yaml -clientConfig: - oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" - oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" - oidc-client-secret: yourSecretClientSecretGoesHere - oidc-extra-scope: profile,email -``` + ```yaml + clientConfig: + oidc-issuer-url: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.issuerUrl }}" + oidc-client-id: "{{ .spectro.pack.kubernetes-eks.managedControlPlane.oidcIdentityProvider.clientId }}" + oidc-client-secret: yourSecretClientSecretGoesHere + oidc-extra-scope: profile,email + ``` diff --git a/docs/docs-content/integrations/rke2.md b/docs/docs-content/integrations/rke2.md index 2c78528522..594980ce1d 100644 --- a/docs/docs-content/integrations/rke2.md +++ b/docs/docs-content/integrations/rke2.md @@ -13,8 +13,6 @@ logoUrl: "https://registry.dev.spectrocloud.com/v1/kubernetes-rke2/blobs/sha256: the U.S. Federal Government sector. To meet the Kubernetes security and compliance goals required by the U.S. Federal Government, RKE2 establishes the following: -
- 1. Provides defaults and configuration options that allow clusters to pass the CIS Kubernetes Benchmark v1.6 with minimal operator intervention. @@ -38,10 +36,9 @@ the respective owner. Once we stop supporting the minor version, we initiate the The following RKE2 versions are supported to work with Palette. -
- - + + ## Prerequisites @@ -63,7 +60,41 @@ guide to learn more. RKE2 offers several customization options, ranging from networking to security. We recommend you review the following RKE2 documentation: -
+- [Configuration Options](https://docs.rke2.io/install/configuration) + +- [Inbound Network Rules](https://docs.rke2.io/install/requirements#inbound-network-rules) + +- [Registries Configuration](https://docs.rke2.io/install/containerd_registry_configuration) + +- [Advanced Options](https://docs.rke2.io/advanced) + +Many of the Day-2 cluster management responsibilities are handled by Palette. Review the +[Cluster Management](../clusters/cluster-management/cluster-management.md) reference resource to learn more about +Palette and Day-2 operations. + + + + + +## Prerequisites + +- A Linux operating system. Refer to the official [RKE2 requirements](https://docs.rke2.io/install/requirements) for + more details on supported Linux distributions and versions. + +- 8 GB Memory + +- 4 CPU + +- An Edge host. Refer to the [Edge](../clusters/edge/edge.md) documentation to learn more about Edge. + +## Usage + +You can add RKE2 to an Edge cluster profile as the Kubernetes layer. Refer to the +[Create an Infrastructure Profile](../profiles/cluster-profiles/create-cluster-profiles/create-infrastructure-profile.md) +guide to learn more. + +RKE2 offers several customization options, ranging from networking to security. We recommend you review the following +RKE2 documentation: - [Configuration Options](https://docs.rke2.io/install/configuration) @@ -100,8 +131,6 @@ guide to learn more. RKE2 offers several customization options, ranging from networking to security. We recommend you review the following RKE2 documentation: -
- - [Configuration Options](https://docs.rke2.io/install/configuration) - [Inbound Network Rules](https://docs.rke2.io/install/requirements#inbound-network-rules) @@ -137,8 +166,6 @@ guide to learn more. RKE2 offers several customization options, ranging from networking to security. We recommend you review the following RKE2 documentation: -
- - [Configuration Options](https://docs.rke2.io/install/configuration) - [Inbound Network Rules](https://docs.rke2.io/install/requirements#inbound-network-rules) @@ -173,8 +200,6 @@ You can add RKE2 to an Edge cluster profile as the Kubernetes layer. To learn mo RKE2 offers several customization options, ranging from networking to security. We recommend you review the following RKE2 documentation: -
- - [Configuration Options](https://docs.rke2.io/install/configuration) - [Inbound Network Rules](https://docs.rke2.io/install/requirements#inbound-network-rules)