Post Fast reboot static route configured is missing #6225

sudhanshukumar22 · 2020-12-16T17:18:31Z

- Why I did it
Post Fast reboot static route configured is missing.

Log:
Prior To fast reboot:
2020-11-03 07:48:28,726 T0000: INFO [D1-H3] FCMD: show ip route
2020-11-03 07:48:28,828 T0000: INFO [D1-H3] Codes: K - kernel route, C - connected, S - static, R - RIP,
2020-11-03 07:48:28,828 T0000: INFO [D1-H3] O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
2020-11-03 07:48:28,828 T0000: INFO [D1-H3] T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
2020-11-03 07:48:28,828 T0000: INFO [D1-H3] F - PBR, f - OpenFabric,
2020-11-03 07:48:28,828 T0000: INFO [D1-H3] > - selected route, * - FIB route, q - queued route, r - rejected route
2020-11-03 07:48:28,829 T0000: INFO [D1-H3]
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] K>* 0.0.0.0/0 [0/202] via 10.59.128.1, eth0, 00:02:04
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] C>* 10.10.10.0/24 is directly connected, Ethernet120, 00:01:34
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] C>* 10.59.128.0/20 is directly connected, eth0, 00:02:04
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] S>* 20.20.20.0/24 [1/0] via 10.10.10.2, Ethernet120, 00:01:23
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] C>* 192.168.12.0/24 is directly connected, Ethernet121, 00:00:29
2020-11-03 07:48:28,829 T0000: INFO [D1-H3] sonic#

Post Fast Reboot:
2020-11-03 07:51:40,228 T0000: INFO [D1-H3] FCMD: show ip route
2020-11-03 07:51:40,329 T0000: INFO [D1-H3] Codes: K - kernel route, C - connected, S - static, R - RIP,
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] F - PBR, f - OpenFabric,
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] > - selected route, * - FIB route, q - queued route, r - rejected route
2020-11-03 07:51:40,330 T0000: INFO [D1-H3]
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] K>* 0.0.0.0/0 [0/202] via 10.59.128.1, eth0, 00:02:08
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] C>* 10.10.10.0/24 is directly connected, Ethernet120, 00:00:58
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] C>* 10.59.128.0/20 is directly connected, eth0, 00:02:08
2020-11-03 07:51:40,330 T0000: INFO [D1-H3] C>* 192.168.12.0/24 is directly connected, Ethernet121, 00:00:58
2020-11-03 07:51:40,331 T0000: INFO [D1-H3] sonic#
- How I did it
Allow zebra socket to come up first before static daemon can start. This is because if static daemon comes up before or at the same time as zebra, zebra misses the static route add message from static daemon. This causes static route to be not present in zebra. This is easily seen during reboot (after config save in FRR).
- How to verify it

- Which release branch to backport (provide reason below if selected)

201811
201911
202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

ben-gale · 2020-12-22T21:50:15Z

@pavel-shirshov , @lguohan - This one has been inactive for a week - how do we move towards merge pls?

sudhanshukumar22 · 2021-01-11T04:59:44Z

Hi all,
There are no NTP changes in my defect. can anyone suggest why NTP testcase is failing here ?

ben-gale · 2021-01-20T22:36:35Z

retest vsimage please

lguohan · 2021-01-20T22:37:38Z

does it also happen on the normal reboot? from the description it seems it can happen on normal reboot as well.

lguohan · 2021-01-20T22:41:01Z

@shi-su , we have solve a race condition between zebra and staticd in #6478 , @sudhanshukumar22, can you look at that pr and see if we can the same approach to solve the problem.

sudhanshukumar22 · 2021-01-21T06:00:32Z

does it also happen on the normal reboot? from the description it seems it can happen on normal reboot as well.

@lguohan , yes, it is a basic testcase and will happen with normal reboot as well. here, we have to create the static route, save the FRR config and reboot. After reboot, the static route present in FRR config will not be replayed.

lguohan · 2021-01-21T06:07:18Z

can you look the other people and use the same approach

lguohan · 2021-01-21T06:35:07Z

dockers/docker-fpm-frr/docker_init.sh

+    -t /usr/share/sonic/templates/unisolate.j2,/usr/sbin/bgp-unisolate \
+"
+
+CFGGEN_PARAMS_UNIFIED=" \


this is unrelated change, should be move to a different pr.

OK, I will move it to another PR. This change was done to support split mode along with separate mode. The default is still separate mode.

lguohan · 2021-01-21T06:35:32Z

dockers/docker-fpm-frr/docker_init.sh

+    rm -f /etc/frr/bgpd.conf /etc/frr/zebra.conf /etc/frr/staticd.conf
+elif [ "$CONFIG_TYPE" == "split" ]; then
+    echo "service integrated-vtysh-config" > /etc/frr/vtysh.conf
+    sonic-cfggen $CFGGEN_PARAMS_SPLIT


unrelated changes. can you pull to a different pr?

lguohan · 2021-01-21T06:36:56Z

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2

@@ -39,6 +39,17 @@ stderr_logfile=syslog
 dependent_startup=true
 dependent_startup_wait_for=rsyslogd:running

+[program:zsocket]
+command=/usr/bin/zsocket.sh


@shi-su , i think the problem you discovered is a generic problem. i think it is better that you pull you script as a delicated script and wait for zebra socket available, and then both bgpd, staticd or other daemons shoud wait for that script to exit.

Agree. I will rewrite my script as a separate one to wait for zebra to be available and then let the daemons wait for it to exit similar to the code changes here.

I drafted a PR to pull the script #6519. It took some ideas in this PR, thanks to @sudhanshukumar22. Maybe we can stick to that script. There are some other changes in this PR, maybe they can be rebased onto #6519.

@ben-gale to comment

I will review that code.

Since @shi-su has done the changes for wait script, I will remove this. I will put my changes on top of them.

lguohan · 2021-01-21T08:23:00Z

@sudhanshukumar22, can you take a look at #6519 ?

lguohan · 2021-01-21T08:24:24Z

dockers/docker-fpm-frr/zsocket.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+secs=30


do we really need to wait this long?

I had put max wait. It will not matter since the script will exit as soon as zebra socket is available.

shi-su · 2021-02-04T17:27:37Z

dockers/docker-fpm-frr/Dockerfile.j2

@@ -18,7 +18,8 @@ RUN apt-get update   && \
        iproute2        \
        libjson-c3      \
        logrotate       \
-        libunwind8
+        libunwind8      \
+	net-tools


Is net-tools still necessary here?

I have one doubt. I am deleting the /etc/sonic/config_db.json and rebooting the system. After reboot, I expect that config db and vtysh config should be empty. But I see that we still have BGP entries in redis DB. e.g.",
router bgp 65100
neighbor 10.0.0.1 remote-as 65200"
Can you suggest where from these entries are coming after reboot. It is expected that when we delete config_db.json and rebooting, the redis DB should also become empty.

Could you check if /etc/sonic/minigraph.xml exists? I think if config_db.json does not exist, it will try to recover the configuration from minigraph.

I have checked this file. It is not present in /etc/sonic, but at some other place. But even after deleting the file and deleting /etc/sonic/config_db.json and reboot, I still see that all BGP configurations are retained. I don't know from where this config is getting replayed. I am using default mode (separated).

Is there anything (e.g., config_db.json, minigraph.xml) in /host/old_config?

shi-su · 2021-02-04T17:28:22Z

dockers/docker-fpm-frr/Dockerfile.j2

@@ -49,6 +50,7 @@ RUN apt-get clean -y      && \

 COPY ["frr", "/usr/share/sonic/templates"]
 COPY ["docker_init.sh", "/usr/bin/"]
+COPY ["zsocket.sh", "/usr/bin/"]


This line is already added a few lines later.

shi-su · 2021-02-04T17:29:35Z

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2

@@ -29,7 +29,7 @@ stderr_logfile=syslog
 dependent_startup=true

 [program:zebra]
-command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M fpm -M snmp
+command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -K 300 -r -M fpm -M snmp --v6-rr-semantics


Could you please help clarify why these changes are necessary on this topic? What will happen without these changes?

shi-su · 2021-02-04T17:31:40Z

src/sonic-bgpcfgd/bgpcfgd/main.py

@@ -25,7 +25,7 @@
 def do_work():
    """ Main function """
    frr = FRR(["bgpd", "zebra", "staticd"])
-    frr.wait_for_daemons(seconds=20)
+    frr.wait_for_daemons(seconds=60)


Do we really need to wait for this long? Is there a scenario that the 20-second is not enough?

sudhanshukumar22 force-pushed the docker_frr_changes branch 2 times, most recently from 855e573 to 7ba63c0 Compare December 16, 2020 17:30

ben-gale added the brcm-upstream label Dec 22, 2020

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare December 25, 2020 18:34

sudhanshukumar22 force-pushed the docker_frr_changes branch from 7ba63c0 to bb30c50 Compare January 9, 2021 14:33

lguohan requested a review from shi-su January 20, 2021 22:36

sudhanshukumar22 force-pushed the docker_frr_changes branch from bb30c50 to 0925d7b Compare January 21, 2021 06:03

lguohan reviewed Jan 21, 2021

View reviewed changes

lguohan added Bug 🐛 Master Branch Quality labels Jan 21, 2021

lguohan reviewed Jan 21, 2021

View reviewed changes

sudhanshukumar22 force-pushed the docker_frr_changes branch from 0925d7b to 938c035 Compare February 4, 2021 04:48

shi-su suggested changes Feb 4, 2021

View reviewed changes

sudhanshukumar22 force-pushed the docker_frr_changes branch from 938c035 to 07fc2d8 Compare February 7, 2021 18:16

Post Fast reboot static route configured is missing

5cd97d0

sudhanshukumar22 force-pushed the docker_frr_changes branch from 07fc2d8 to 5cd97d0 Compare February 15, 2021 09:26

sudhanshukumar22 closed this Feb 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Post Fast reboot static route configured is missing #6225

Post Fast reboot static route configured is missing #6225

sudhanshukumar22 commented Dec 16, 2020

ben-gale commented Dec 22, 2020

sudhanshukumar22 commented Jan 11, 2021

ben-gale commented Jan 20, 2021

lguohan commented Jan 20, 2021

lguohan commented Jan 20, 2021

sudhanshukumar22 commented Jan 21, 2021

lguohan commented Jan 21, 2021

lguohan Jan 21, 2021

sudhanshukumar22 Jan 22, 2021

lguohan Jan 21, 2021

sudhanshukumar22 Jan 22, 2021

lguohan Jan 21, 2021

shi-su Jan 21, 2021

shi-su Jan 21, 2021

lguohan Jan 21, 2021

sudhanshukumar22 Jan 22, 2021

sudhanshukumar22 Jan 22, 2021

lguohan commented Jan 21, 2021

lguohan Jan 21, 2021

sudhanshukumar22 Jan 22, 2021

shi-su Feb 4, 2021

sudhanshukumar22 Feb 7, 2021

shi-su Feb 10, 2021

sudhanshukumar22 Feb 11, 2021

shi-su Feb 13, 2021

shi-su Feb 4, 2021

shi-su Feb 4, 2021

shi-su Feb 4, 2021

Post Fast reboot static route configured is missing #6225

Post Fast reboot static route configured is missing #6225

Conversation

sudhanshukumar22 commented Dec 16, 2020

ben-gale commented Dec 22, 2020

sudhanshukumar22 commented Jan 11, 2021

ben-gale commented Jan 20, 2021

lguohan commented Jan 20, 2021

lguohan commented Jan 20, 2021

sudhanshukumar22 commented Jan 21, 2021

lguohan commented Jan 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Jan 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment