Skip to content

Commit

Permalink
Add instructions to install Antrea in encap mode in AKS
Browse files Browse the repository at this point in the history 
AKS has supported a "Bring Your Own CNI" mode for a while now:
https://learn.microsoft.com/en-us/azure/aks/use-byo-cni?tabs=azure-cli

With this support, Antrea can easily be installed in encap mode in AKS
clusters, with no limitation (unlike for EKS where there is no more
connectivity from the control-plane to the Nodes). The only requirement
is to enable Antrea NodeIPAM.

This support relies on the
[Konnectivity Service](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/),
which is deployed by default in "recent" AKS versions, and guarantees
connectivity from the control-plane to the Nodes, regardless of which
CNI is used.

Fixes #5817

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
  • Loading branch information
@antoninbas
antoninbas committed Jan 23, 2024
1 parent c0c3e25 commit 5ead7b4
Showing 1 changed file with 123 additions and 10 deletions.
133 changes: 123 additions & 10 deletions docs/aks-installation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,36 @@ Engine cluster.

## Deploy Antrea to an AKS cluster

Antrea can be deployed to an AKS cluster in `networkPolicyOnly` mode, in which
Antrea enforces NetworkPolicies and implements other services for the AKS
cluster, while Azure CNI takes care of Pod IPAM and traffic routing across Nodes.
For more information about `networkPolicyOnly` mode, refer to [this design document](design/policy-only.md).
Antrea can be deployed to an AKS cluster either in `networkPolicyOnly` mode or
in `encap` mode.

In `networkPolicyOnly` mode, Antrea enforces NetworkPolicies and implements
other services for the AKS cluster, while the Azure CNI takes care of Pod IPAM
and traffic routing across Nodes. For more information about `networkPolicyOnly`
mode, refer to [this design document](design/policy-only.md).

In `encap` mode, Antrea is in charge of Pod IPAM and of all the networking
functions on the Nodes. Using `encap` mode provides access to additional Antrea
features, such as Multicast, as inter-Node Pod traffic is encapsulated, and is
not handled directly by the Azure Virtual Network. Note that the [caveats](eks-installation.md#deploying-antrea-in-encap-mode)
which apply when deploying Antrea in `encap` mode on EKS do *not* apply for AKS.

We recommend `encap` mode, as it will give you access to the most Antrea
features.

### AKS Prerequisites

Install the Azure Cloud CLI. Refer to [Azure CLI installation guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)

### Creating the cluster
We recommend using the latest version available (use at least version 2.39.0).

### Deploying Antrea in `networkPolicyOnly` mode

#### Creating the cluster

You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.

1. Create an AKS Cluster
1. Create an AKS cluster

```bash
export RESOURCE_GROUP_NAME=aks-antrea-cluster
Expand All @@ -35,13 +51,13 @@ You can use any method to create an AKS cluster. The example given here is using

**Note** Do not specify network-policy option.

2. Get AKS Cluster Credentials
2. Get AKS cluster credentials

```bash
az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME
```

3. Access your Cluster
3. Access your cluster

```bash
kubectl get nodes
Expand All @@ -50,9 +66,9 @@ You can use any method to create an AKS cluster. The example given here is using
aks-nodepool1-84330359-vmss000001 Ready agent 6m25s v1.16.10
```

### Deploying Antrea
#### Deploying Antrea

1. Prepare the Cluster Nodes
1. Prepare the cluster Nodes

Deploy ``antrea-node-init`` DaemonSet to enable ``azure cni`` to operate in transparent mode.

Expand Down Expand Up @@ -110,6 +126,103 @@ you should be able to see these Pods running in your cluster:
pod "tunnelfront-649ff5fb55-5lxg7" deleted
```

### Deploying Antrea in `encap` mode

AKS now officially supports [Bring your own Container Network Interface (BYOCNI)](https://learn.microsoft.com/en-us/azure/aks/use-byo-cni).
Thanks to this, you can deploy Antrea on AKS in `encap` mode, and you will not
lose access to any functionality. Check the AKS BYOCNI documentation for
prerequisites, in particular for AKS version requirements.

#### Creating the cluster

You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.

1. Create an AKS cluster

```bash
export RESOURCE_GROUP_NAME=aks-antrea-cluster
export CLUSTER_NAME=aks-antrea-cluster
export LOCATION=westus
az group create --name $RESOURCE_GROUP_NAME --location $LOCATION
az aks create \
--resource-group $RESOURCE_GROUP_NAME \
--name $CLUSTER_NAME \
--node-count 2 \
--network-plugin none
```

Notice `--network-plugin none`, which tells AKS not to install any CNI plugin.

2. Get AKS cluster credentials

```bash
az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME
```

3. Access your cluster

```bash
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-40948307-vmss000000 NotReady agent 18m v1.27.7
aks-nodepool1-40948307-vmss000001 NotReady agent 17m v1.27.7
```

The Nodes are supposed to report a `NotReady` Status, since no CNI plugin is
installed yet.

#### Deploying Antrea

You can use Helm to easily install Antrea (or any other supported installation
method). Just make sure that you configure Antrea NodeIPAM:

```bash
# you may not need this:
helm repo add antrea https://charts.antrea.io
helm repo update
cat <<EOF >> values-aks.yml
nodeIPAM:
enable: true
clusterCIDRs: ["10.10.0.0/16"]
EOF
helm install -n kube-system -f values-aks.yml antrea antrea/antrea
```

For more information about how to configure Antrea Node IPAM, please refer to
[Antrea Node IPAM guide](antrea-ipam.md#running-nodeipam-within-antrea-controller).

After a while, make sure that all your Nodes report a `Ready` Status and that
all your Pods are running correctly. Some Pods, and in particular the
`metrics-server` Pods, may restart once after installing Antrea; this is not an
issue.

After a successful installation, Pods should look like this:

```bash
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system antrea-agent-bpskv 2/2 Running 0 7m34s
kube-system antrea-agent-pfqrn 2/2 Running 0 7m34s
kube-system antrea-controller-555b8c799d-wk8zz 1/1 Running 0 7m34s
kube-system cloud-node-manager-2nszz 1/1 Running 0 31m
kube-system cloud-node-manager-wj68q 1/1 Running 0 31m
kube-system coredns-789789675-2nwd7 1/1 Running 0 6m48s
kube-system coredns-789789675-lbkfn 1/1 Running 0 31m
kube-system coredns-autoscaler-649b947bbd-j5wqc 1/1 Running 0 31m
kube-system csi-azuredisk-node-4bnnl 3/3 Running 0 31m
kube-system csi-azuredisk-node-52nwd 3/3 Running 0 31m
kube-system csi-azurefile-node-2h66l 3/3 Running 0 31m
kube-system csi-azurefile-node-dhrf2 3/3 Running 0 31m
kube-system konnectivity-agent-5fc7989878-6nhwl 1/1 Running 0 31m
kube-system konnectivity-agent-5fc7989878-t2n6h 1/1 Running 0 30m
kube-system kube-proxy-96c9p 1/1 Running 0 31m
kube-system kube-proxy-x8g8s 1/1 Running 0 31m
kube-system metrics-server-5955767688-2hjvn 2/2 Running 0 3m45s
kube-system metrics-server-5955767688-vmcq7 2/2 Running 0 3m45s
```

## Deploy Antrea to an AKS Engine cluster

Antrea is an integrated CNI of AKS Engine, and can be installed in
Expand Down

0 comments on commit 5ead7b4

Please sign in to comment.