About healthchecking

[PeanutSkinEr]

HI， VIP gets announced using OSPF(quagga) and is set at the interface lo. ipip tunnels are present and up.
Because VIP was set at lo interface, healthcheck packets were not transmitted outside from enp0s3.
but, healthchecking_ipip.o work in enp0s3. What are the ways to solve this problem? Thanks.

[tehnerd]

you can specify interface for healthchecking (where vips are defined) w/ -hc_intf=lo (if we are talking about grpc example). don't forget to add clsact qdisk there for things to work (see EXAMPLE.md on how to)

[tehnerd]

Also jfyi. There are some issues with so_mark and Linux kernel which were been fixed in upstream around 5.0 kernel. The issue is that sometimes TCP fin packet could be sent with out so_mark set

[PeanutSkinEr]

It's done, You helped. Thank you！

[kgs]

Hi @tehnerd,

I have exactly the same issue and I do not understand how setting -hc_intf=lo where VIPs are defined can help.

Let's say my VIP is 1.2.3.4 and it is added to lo. I also have some other interfaces, mainly egress one where there is some other IP, let's say 4.4.4.4.

When I do this change I see that hc ipip packets have correct external ip structure (src = artifical ip, dst = ip of real) but incorrect (I think) internal ip structure:

• dst ip = 1.2.3.4 correct (which is VIP)
• src ip = 1.2.3.4 not correct and causes packets that reach real server to stay there (they have src=dst).

Am I missing something?

[tehnerd]

There is src option in ip route command. So you need to add somethibg like ip route 1.2.3.4/32 src x.x.x.x via dev lo

'''
src ADDRESS
the source address to prefer when sending to the
destinations covered by the route prefix.
'''

[kgs]

Hi,
thanks @tehnerd for fast response!

Is it standard requirement when Katran and L7 LB are on the same host? Or my configuration is just weird?

Because it seems that this do not help. Here is my setup (a little bit obfuscated):

# ip route add 100.100.2.36/32 src 100.100.3.9 dev lo

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 100.100.3.64/32 brd 100.100.3.64 scope host lo
       valid_lft forever preferred_lft forever
    inet 100.100.2.36/32 scope global lo
       valid_lft forever preferred_lft forever
 ...
 14: bond0.904@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 xdp qdisc noqueue state UP group default qlen 1000
    link/ether SO:ME:TH:IN:G brd ff:ff:ff:ff:ff:ff
    inet 100.100.3.9/28 brd 100.100.3.15 scope global bond0.904
       valid_lft forever preferred_lft forever

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         100.100.3.1      0.0.0.0         UG    0      0        0 bond0.904
10.0.0.0        10.1.70.1       255.0.0.0       UG    0      0        0 bond0.170
10.1.70.0       0.0.0.0         255.255.255.0   U     0      0        0 bond0.170
100.100.2.36     0.0.0.0         255.255.255.255 UH    0      0        0 lo
100.100.3.0      0.0.0.0         255.255.255.240 U     0      0        0 bond0.904
169.254.0.0     0.0.0.0         255.255.0.0     U     1008   0        0 bond0
169.254.0.0     0.0.0.0         255.255.0.0     U     1013   0        0 bond0.170
169.254.0.0     0.0.0.0         255.255.0.0     U     1014   0        0 bond0.904

# ip route show
default via 100.100.3.1 dev bond0.904
10.0.0.0/8 via 10.1.70.1 dev bond0.170
10.1.70.0/24 dev bond0.170 proto kernel scope link src 10.1.70.69
100.100.2.36 dev lo scope link src 100.100.3.9
100.100.3.0/28 dev bond0.904 proto kernel scope link src 100.100.3.9
169.254.0.0/16 dev bond0 scope link metric 1008
169.254.0.0/16 dev bond0.170 scope link metric 1013
169.254.0.0/16 dev bond0.904 scope link metric 1014

# ip route get 100.100.2.36
local 100.100.2.36 dev lo src 100.100.2.36 uid 0
    cache <local>

# ./send_hc_request.py 100.100.2.36 80 631046920
Connecting...
Traceback (most recent call last):
  File "./send_hc_request.py", line 49, in <module>
    main()
  File "./send_hc_request.py", line 45, in main
    send_packet(sys.argv[1], int(sys.argv[2]), sys.argv[3])
  File "./send_hc_request.py", line 24, in send_packet
    s.connect((host, port))
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.timeout: timed out

Also on other host (100.100.3.8, where healthcheck is targetted) I see:
16:27:11.614810 ethertype IPv4, IP 100.100.3.9 > 100.100.3.8: IP 100.100.2.36.44586 > 100.100.36.80: Flags [S], seq 3847640263, win 65495, options [mss 65495,sackOK,TS val 3631283526 ecr 0,nop,wscale 8], length 0 (ipip-proto-4)

which confirms that this is not working and packet is stuck on other host and does not go back :/

Any ideas?

[tehnerd]

route w/ src should be added in local table (ip route ... table local)

[tehnerd]

Also please open a new issue. So the FB folks (the ones who own this repo) would be able to see it and add what is missing in configuration

[kgs]

@tehnerd awesome, it works, thanks! Didn't know about existence of table local...

Do you think I still need to create separate issue if this is solved now?

[tehnerd]

as you wish. you could even improve documentation and add this snippet there if you want