From 5ead7b4ab5e2881105d06eba08711e7a1514221b Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Mon, 22 Jan 2024 15:48:06 -0800 Subject: [PATCH] Add instructions to install Antrea in encap mode in AKS AKS has supported a "Bring Your Own CNI" mode for a while now: https://learn.microsoft.com/en-us/azure/aks/use-byo-cni?tabs=azure-cli With this support, Antrea can easily be installed in encap mode in AKS clusters, with no limitation (unlike for EKS where there is no more connectivity from the control-plane to the Nodes). The only requirement is to enable Antrea NodeIPAM. This support relies on the [Konnectivity Service](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/), which is deployed by default in "recent" AKS versions, and guarantees connectivity from the control-plane to the Nodes, regardless of which CNI is used. Fixes #5817 Signed-off-by: Antonin Bas --- docs/aks-installation.md | 133 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 123 insertions(+), 10 deletions(-) diff --git a/docs/aks-installation.md b/docs/aks-installation.md index fb001f867cb..93e06505576 100644 --- a/docs/aks-installation.md +++ b/docs/aks-installation.md @@ -5,20 +5,36 @@ Engine cluster. ## Deploy Antrea to an AKS cluster -Antrea can be deployed to an AKS cluster in `networkPolicyOnly` mode, in which -Antrea enforces NetworkPolicies and implements other services for the AKS -cluster, while Azure CNI takes care of Pod IPAM and traffic routing across Nodes. -For more information about `networkPolicyOnly` mode, refer to [this design document](design/policy-only.md). +Antrea can be deployed to an AKS cluster either in `networkPolicyOnly` mode or +in `encap` mode. + +In `networkPolicyOnly` mode, Antrea enforces NetworkPolicies and implements +other services for the AKS cluster, while the Azure CNI takes care of Pod IPAM +and traffic routing across Nodes. For more information about `networkPolicyOnly` +mode, refer to [this design document](design/policy-only.md). + +In `encap` mode, Antrea is in charge of Pod IPAM and of all the networking +functions on the Nodes. Using `encap` mode provides access to additional Antrea +features, such as Multicast, as inter-Node Pod traffic is encapsulated, and is +not handled directly by the Azure Virtual Network. Note that the [caveats](eks-installation.md#deploying-antrea-in-encap-mode) +which apply when deploying Antrea in `encap` mode on EKS do *not* apply for AKS. + +We recommend `encap` mode, as it will give you access to the most Antrea +features. ### AKS Prerequisites Install the Azure Cloud CLI. Refer to [Azure CLI installation guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) -### Creating the cluster +We recommend using the latest version available (use at least version 2.39.0). + +### Deploying Antrea in `networkPolicyOnly` mode + +#### Creating the cluster You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI. -1. Create an AKS Cluster +1. Create an AKS cluster ```bash export RESOURCE_GROUP_NAME=aks-antrea-cluster @@ -35,13 +51,13 @@ You can use any method to create an AKS cluster. The example given here is using **Note** Do not specify network-policy option. -2. Get AKS Cluster Credentials +2. Get AKS cluster credentials ```bash az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME ``` -3. Access your Cluster +3. Access your cluster ```bash kubectl get nodes @@ -50,9 +66,9 @@ You can use any method to create an AKS cluster. The example given here is using aks-nodepool1-84330359-vmss000001 Ready agent 6m25s v1.16.10 ``` -### Deploying Antrea +#### Deploying Antrea -1. Prepare the Cluster Nodes +1. Prepare the cluster Nodes Deploy ``antrea-node-init`` DaemonSet to enable ``azure cni`` to operate in transparent mode. @@ -110,6 +126,103 @@ you should be able to see these Pods running in your cluster: pod "tunnelfront-649ff5fb55-5lxg7" deleted ``` +### Deploying Antrea in `encap` mode + +AKS now officially supports [Bring your own Container Network Interface (BYOCNI)](https://learn.microsoft.com/en-us/azure/aks/use-byo-cni). +Thanks to this, you can deploy Antrea on AKS in `encap` mode, and you will not +lose access to any functionality. Check the AKS BYOCNI documentation for +prerequisites, in particular for AKS version requirements. + +#### Creating the cluster + +You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI. + +1. Create an AKS cluster + + ```bash + export RESOURCE_GROUP_NAME=aks-antrea-cluster + export CLUSTER_NAME=aks-antrea-cluster + export LOCATION=westus + + az group create --name $RESOURCE_GROUP_NAME --location $LOCATION + az aks create \ + --resource-group $RESOURCE_GROUP_NAME \ + --name $CLUSTER_NAME \ + --node-count 2 \ + --network-plugin none + ``` + + Notice `--network-plugin none`, which tells AKS not to install any CNI plugin. + +2. Get AKS cluster credentials + + ```bash + az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME + ``` + +3. Access your cluster + + ```bash + kubectl get nodes + NAME STATUS ROLES AGE VERSION + aks-nodepool1-40948307-vmss000000 NotReady agent 18m v1.27.7 + aks-nodepool1-40948307-vmss000001 NotReady agent 17m v1.27.7 + ``` + + The Nodes are supposed to report a `NotReady` Status, since no CNI plugin is + installed yet. + +#### Deploying Antrea + +You can use Helm to easily install Antrea (or any other supported installation +method). Just make sure that you configure Antrea NodeIPAM: + +```bash +# you may not need this: +helm repo add antrea https://charts.antrea.io +helm repo update + +cat <> values-aks.yml +nodeIPAM: + enable: true + clusterCIDRs: ["10.10.0.0/16"] +EOF + +helm install -n kube-system -f values-aks.yml antrea antrea/antrea +``` + +For more information about how to configure Antrea Node IPAM, please refer to +[Antrea Node IPAM guide](antrea-ipam.md#running-nodeipam-within-antrea-controller). + +After a while, make sure that all your Nodes report a `Ready` Status and that +all your Pods are running correctly. Some Pods, and in particular the +`metrics-server` Pods, may restart once after installing Antrea; this is not an +issue. + +After a successful installation, Pods should look like this: + +```bash +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system antrea-agent-bpskv 2/2 Running 0 7m34s +kube-system antrea-agent-pfqrn 2/2 Running 0 7m34s +kube-system antrea-controller-555b8c799d-wk8zz 1/1 Running 0 7m34s +kube-system cloud-node-manager-2nszz 1/1 Running 0 31m +kube-system cloud-node-manager-wj68q 1/1 Running 0 31m +kube-system coredns-789789675-2nwd7 1/1 Running 0 6m48s +kube-system coredns-789789675-lbkfn 1/1 Running 0 31m +kube-system coredns-autoscaler-649b947bbd-j5wqc 1/1 Running 0 31m +kube-system csi-azuredisk-node-4bnnl 3/3 Running 0 31m +kube-system csi-azuredisk-node-52nwd 3/3 Running 0 31m +kube-system csi-azurefile-node-2h66l 3/3 Running 0 31m +kube-system csi-azurefile-node-dhrf2 3/3 Running 0 31m +kube-system konnectivity-agent-5fc7989878-6nhwl 1/1 Running 0 31m +kube-system konnectivity-agent-5fc7989878-t2n6h 1/1 Running 0 30m +kube-system kube-proxy-96c9p 1/1 Running 0 31m +kube-system kube-proxy-x8g8s 1/1 Running 0 31m +kube-system metrics-server-5955767688-2hjvn 2/2 Running 0 3m45s +kube-system metrics-server-5955767688-vmcq7 2/2 Running 0 3m45s +``` + ## Deploy Antrea to an AKS Engine cluster Antrea is an integrated CNI of AKS Engine, and can be installed in