Add support for short-circuiting in AntreaProxy #4815

hongliangl · 2023-04-06T10:33:15Z

Short-circuiting is used to ensure that the traffic from Pod/Node clients to
external addresses behaves the same way as the traffic from external clients to
external addresses.

External clients do not need to consider which Nodes have local Endpoints, as the
load balancer handles this for them. However, for Pod/Node clients, when the
externalTrafficPolicy of the Service is set to "Local", it will not work on Nodes
without an Endpoint. With this PR, even when the externalTrafficPolicy is set
to "Local", Pod/Node clients without local Endpoints can still work by selecting
Endpoints from the cluster.

hongliangl · 2023-04-06T10:38:47Z

@tnqn @wenyingd Could you help review this roughly first? I modify method InstallServiceFlows in this draft PR to implement short-circuiting in AntreaProxy, and I'm not sure if it is a good way. Could you give some ideas? Thanks.

tnqn

@hongliangl I think you should first have a clear description to explain what it tries to do for better understanding. The term short-circuiting is too obscure. Even a few weeks ago we had totally different understandings about it, and I'm not sure if we reach a consensus yet before I take a deep look at the code, not to mention other reviewers.

pkg/agent/openflow/client.go

jianjuns · 2023-05-05T04:15:30Z

@hongliangl : the commit is "Add support for ExternalIP in AntreaProxy"? Have you pushed the right commit?

hongliangl · 2023-05-05T10:53:24Z

@hongliangl : the commit is "Add support for ExternalIP in AntreaProxy"? Have you pushed the right commit?

I pushed wrong git commit. I just updated it.

third_party/proxy/service.go

pkg/agent/openflow/client.go

pkg/agent/openflow/pipeline.go

luolanzone · 2023-05-08T09:54:56Z

@hongliangl I would suggest to remove See kubernetes/kubernetes#108526 for more information. in the commit message. Every force pushed commits will be showed up in the issue history.

tnqn · 2023-05-11T08:29:20Z

@hongliangl I think NodePort should respect ExternalTrafficPolicy according to https://github.com/kubernetes/kubernetes/blob/122a459dcbf7b2317ac5bd3793ed94a400bd4a77/staging/src/k8s.io/api/core/v1/types.go#L4746-L4749

hongliangl · 2023-05-11T08:34:32Z

@hongliangl I think NodePort should respect ExternalTrafficPolicy according to https://github.com/kubernetes/kubernetes/blob/122a459dcbf7b2317ac5bd3793ed94a400bd4a77/staging/src/k8s.io/api/core/v1/types.go#L4746-L4749

Since it has the definition in API comment, we should implement proxy as it describes.

pkg/agent/openflow/client.go

tnqn · 2023-05-16T05:51:15Z

The dependency has been merged, could you rebase?

hongliangl · 2023-05-16T05:52:05Z

The dependency has been merged, could you rebase?

Sure, I'll rebase it right now.

tnqn

LGTM overall

pkg/agent/openflow/pipeline.go

pkg/agent/proxy/proxier.go

pkg/agent/proxy/proxier_test.go

tnqn · 2023-05-16T15:26:43Z

test/e2e/proxy_test.go

-
-		clientIP, err := probeClientIPFromPod(data, pod, busyboxContainerName, url)
-		require.NoError(t, err, errMsg)
-		require.Equal(t, clientIP, expectedClientIPs[idx])


Is the clientIP expected to change?

For non-host-network Endpoint, the clientIP should be always the same (source Pod IP), but for host-network Endpoint, if the Endpoint is on local host-network, the clientIP should be the source Pod IP; if the Endpoint is on a remote Node, the clientIP will be the transparent interface IP. In this test, we have two host-network Endpoint, and both of them can be selected even when ExternalTrafficPolicy is Local for client from Pod.

what's transparent interface IP?

It should be Node IP. For example, if the Endpoint is a host network on another Node, the packet will be forwarded to local Node host network, then will be SNATed to remote Node. The SNATed IP should be the local Node IP.

OK, it seems the SNAT is not necessary but I understand it's not related to the PR.

For example, assuming that there are two Nodes:
Node A: 192.168.77.100, Pod CIDR 10.10.0.0/24
Node B: 192.168.77.101, Pod CIDR 10.10.1.0/24
Service: 10.96.0.9:8080, two Endpoints, 192.168.77.100:8080, 192.168.77.101:8080
Pod on Node A: 10.10.0.10

Previously, if externalTrafficPolicy is Local, for Pod -> Service, only Endpoint 192.168.77.100:8080 will be selected, and the clientIP is 10.10.0.10. Now, Endpoint 192.168.77.101:8080 could be also selected. If so:

on Pod network: 10.10.0.10 -> 10.96.0.9

on host network: 10.10.0.10 -> 192.168.77.101

to make sure that the reply packets can be returned correctly, on network between Nodes: 192.168.77.100 -> 192.168.77.101, then the clientIP is 192.168.77.100. As a result, the clientIP is not certain. It could be 10.10.0.10 or 192.168.77.100.

tnqn

LGTM

tnqn · 2023-05-17T02:05:20Z

/test-all

Short-circuiting is used to ensure that the traffic from Pod/Node clients to external addresses behaves the same way as the traffic from external clients to external addresses. External clients do not need to consider which Nodes have local Endpoints, as the load balancer handles this for them. However, for Pod/Node clients, when the externalTrafficPolicy of the Service is set to "Local", it will not work on Nodes without an Endpoint. With this PR, even when the externalTrafficPolicy is set to "Local", Pod/Node clients without local Endpoints can still work by selecting Endpoints from the cluster. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

hongliangl · 2023-05-17T02:46:26Z

LGTM

@tnqn only renamed a variable, no other changes.

hongliangl · 2023-05-17T05:38:33Z

/test-all

tnqn · 2023-05-18T02:27:06Z

/test-windows-proxyall-e2e
/test-windows-conformance
/test-windows-e2e

tnqn · 2023-05-18T09:18:05Z

/test-windows-conformance

hongliangl · 2023-05-18T12:34:09Z

Thanks for merging this PR.

Short-circuiting is used to ensure that the traffic from Pod/Node clients to external addresses behaves the same way as the traffic from external clients to external addresses. External clients do not need to consider which Nodes have local Endpoints, as the load balancer handles this for them. However, for Pod/Node clients, when the externalTrafficPolicy of the Service is set to "Local", it will not work on Nodes without an Endpoint. With this PR, even when the externalTrafficPolicy is set to "Local", Pod/Node clients without local Endpoints can still work by selecting Endpoints from the cluster. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

hongliangl requested review from tnqn and wenyingd April 6, 2023 10:33

tnqn reviewed Apr 13, 2023

View reviewed changes

pkg/agent/openflow/client.go Outdated Show resolved Hide resolved

pkg/agent/openflow/client.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20230328-short-circuit branch 2 times, most recently from 988c0a5 to e6b33de Compare April 16, 2023 12:00

hongliangl marked this pull request as ready for review April 16, 2023 12:00

hongliangl force-pushed the 20230328-short-circuit branch from e6b33de to 29a1a29 Compare April 16, 2023 14:08

hongliangl added area/proxy Issues or PRs related to proxy functions in Antrea action/release-note Indicates a PR that should be included in release notes. labels Apr 16, 2023

hongliangl force-pushed the 20230328-short-circuit branch from 29a1a29 to 8675f47 Compare May 4, 2023 01:47

vicky-liu added this to the Antrea v1.12 release milestone May 4, 2023

hongliangl force-pushed the 20230328-short-circuit branch from 8675f47 to 8af63ce Compare May 5, 2023 10:51

jianjuns reviewed May 5, 2023

View reviewed changes

third_party/proxy/service.go Outdated Show resolved Hide resolved

jianjuns reviewed May 5, 2023

View reviewed changes

pkg/agent/openflow/client.go Outdated Show resolved Hide resolved

pkg/agent/openflow/client.go Outdated Show resolved Hide resolved

pkg/agent/openflow/pipeline.go Outdated Show resolved Hide resolved

pkg/agent/openflow/pipeline.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20230328-short-circuit branch 2 times, most recently from 1326571 to 0f3fc2b Compare May 6, 2023 02:07

hongliangl requested review from jianjuns and tnqn May 6, 2023 02:08

hongliangl force-pushed the 20230328-short-circuit branch 4 times, most recently from 194f5db to 133d264 Compare May 11, 2023 08:18

hongliangl force-pushed the 20230328-short-circuit branch from 133d264 to 34f157d Compare May 11, 2023 09:38

tnqn reviewed May 11, 2023

View reviewed changes

pkg/agent/openflow/client.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20230328-short-circuit branch 2 times, most recently from cff9001 to 3f0a45c Compare May 12, 2023 03:12

hongliangl force-pushed the 20230328-short-circuit branch from 3f0a45c to 4ac17e1 Compare May 16, 2023 06:19

hongliangl requested a review from tnqn May 16, 2023 06:20

tnqn reviewed May 16, 2023

View reviewed changes

pkg/agent/openflow/pipeline.go Show resolved Hide resolved

pkg/agent/proxy/proxier.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20230328-short-circuit branch from 4ac17e1 to 29a8b10 Compare May 16, 2023 07:49

tnqn reviewed May 16, 2023

View reviewed changes

hongliangl force-pushed the 20230328-short-circuit branch 2 times, most recently from 3184f14 to 6eaf6b0 Compare May 16, 2023 12:23

hongliangl requested a review from tnqn May 16, 2023 12:26

tnqn reviewed May 16, 2023

View reviewed changes

pkg/agent/proxy/proxier_test.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20230328-short-circuit branch from 6eaf6b0 to fff3c66 Compare May 16, 2023 14:43

tnqn reviewed May 16, 2023

View reviewed changes

tnqn previously approved these changes May 17, 2023

View reviewed changes

hongliangl dismissed tnqn’s stale review via 8caa6ec May 17, 2023 02:45

hongliangl force-pushed the 20230328-short-circuit branch from fff3c66 to 8caa6ec Compare May 17, 2023 02:45

tnqn approved these changes May 17, 2023

View reviewed changes

tnqn merged commit 46d2a3c into antrea-io:main May 18, 2023

hongliangl deleted the 20230328-short-circuit branch May 18, 2023 12:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for short-circuiting in AntreaProxy #4815

Add support for short-circuiting in AntreaProxy #4815

hongliangl commented Apr 6, 2023 •

edited

Loading

hongliangl commented Apr 6, 2023

tnqn left a comment

jianjuns commented May 5, 2023

hongliangl commented May 5, 2023

luolanzone commented May 8, 2023

tnqn commented May 11, 2023

hongliangl commented May 11, 2023

tnqn commented May 16, 2023

hongliangl commented May 16, 2023

tnqn left a comment

tnqn May 16, 2023

hongliangl May 16, 2023

tnqn May 16, 2023

hongliangl May 16, 2023

tnqn May 17, 2023

hongliangl May 17, 2023

tnqn left a comment

tnqn commented May 17, 2023

hongliangl commented May 17, 2023

hongliangl commented May 17, 2023

tnqn commented May 18, 2023

tnqn commented May 18, 2023

hongliangl commented May 18, 2023

Add support for short-circuiting in AntreaProxy #4815

Add support for short-circuiting in AntreaProxy #4815

Conversation

hongliangl commented Apr 6, 2023 • edited Loading

hongliangl commented Apr 6, 2023

tnqn left a comment

Choose a reason for hiding this comment

jianjuns commented May 5, 2023

hongliangl commented May 5, 2023

luolanzone commented May 8, 2023

tnqn commented May 11, 2023

hongliangl commented May 11, 2023

tnqn commented May 16, 2023

hongliangl commented May 16, 2023

tnqn left a comment

Choose a reason for hiding this comment

tnqn May 16, 2023

Choose a reason for hiding this comment

hongliangl May 16, 2023

Choose a reason for hiding this comment

tnqn May 16, 2023

Choose a reason for hiding this comment

hongliangl May 16, 2023

Choose a reason for hiding this comment

tnqn May 17, 2023

Choose a reason for hiding this comment

hongliangl May 17, 2023

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

tnqn commented May 17, 2023

hongliangl commented May 17, 2023

hongliangl commented May 17, 2023

tnqn commented May 18, 2023

tnqn commented May 18, 2023

hongliangl commented May 18, 2023

hongliangl commented Apr 6, 2023 •

edited

Loading