Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the log messages in addKernelNeigh/Route from ERROR to INFO #2437

Merged
merged 8 commits into from
Sep 1, 2022

Conversation

judyjoseph
Copy link
Contributor

What I did
Change the Error reporting for API's addKernelRoute, addKernelNeigh from ERR to INFO as the code logic already deletes and retries the creation of Neigh/Route

For the test tests/test_virtual_chassis.py::test_chassis_system_neigh the following change was made

  1. Grouped the neighbor creation, validation of the DB entry present in local and remote LC into a new function chassis_system_neigh_create()
  2. Call this function first with mac_address = "00:01:02:03:04:77", action as "add" to create a new neighbor entry.
  3. Call this function again with mac_address = "00:01:02:03:04:05", action as "change" to update the mac_addres of that neighbor entry already created before.

Why I did it
There was sonic-mgmt test failures in multi-asic devices where the Neighbor and routes learnt in in one ASIC is synced with the nearby ASIC, and in case of mac move there is possibility for an existing entry to be updated with a new mac address ( using the command /sbin/ip neigh add lladdr dev ) . This causes error as ip neigh/route add command will fail (error : nbrmgrd RTNETLINK answers: File exists)

E               Failed: Processes "['analyze_logs--<MultiAsicSonicHost str2--lc1-1>']" failed with exit code "1"
E               Exception:
E               expected_match: 0
E               expected_missing_match: 0
E               match: 1
E               
E               Match Messages:
E               Aug 16 16:39:19.939498 str2--lc1-1 ERR swss1#nbrmgrd: :- doStateSystemNeighTask: Route entry add on dev Ethernet-IB1 failed for 'str2--lc1-1|asic0|PortChannel102|29.0.0.2'
E               
E               Traceback:
E               Traceback (most recent call last):
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/helpers/parallel.py", line 31, in run
E                   Process.run(self)
E                 File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
E                   self._target(*self._args, **self._kwargs)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/helpers/parallel.py", line 226, in wrapper
E                   target(*args, **kwargs)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/__init__.py", line 39, in analyze_logs
E                   dut_analyzer.analyze(markers[node.hostname])
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/loganalyzer.py", line 369, in analyze
E                   self._verify_log(analyzer_summary)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/loganalyzer.py", line 133, in _verify_log
E                   raise LogAnalyzerError(result_str)
E               LogAnalyzerError: expected_match: 0
E               expected_missing_match: 0
E               match: 1
E               
E               Match Messages:
E               Aug 16 16:39:19.939498 str2--lc1-1 ERR swss1#nbrmgrd: :- doStateSystemNeighTask: Route entry add on dev Ethernet-IB1 failed for 'str2--lc1-1|asic0|PortChannel102|29.0.0.2'

---------------------------------------------------- generated xml file: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/logs/tr.xml ------------------------------------------------------
=================================================================================== short test summary info ====================================================================================
ERROR arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost str2--lc1-1>']" failed wit...
============================================================================= 1 passed, 1 error in 316.96 seconds ==============================================================================

How I verified it
With this fix the tests passed

jujoseph@c3e1277334f8:~/CHASSIS/CHASSIS/sonic-mgmt-int/tests$ ./run_tests.sh -c arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0]  -n vms29-t2--1 -d str2--lc1-1 -i "../ansible/str2,../ansible/veos" -t "t2,any" -S "dualtor_io" -e "--skip_sanity" -u
=== Running tests in groups ===
/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
===================================================================================== test session starts ======================================================================================
platform linux2 -- Python 2.7.17, pytest-4.6.5, py-1.11.0, pluggy-0.13.1
ansible: 2.8.12
rootdir: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests, inifile: pytest.ini
plugins: forked-1.3.0, metadata-1.11.0, xdist-1.28.0, html-1.22.1, allure-pytest-2.8.22, repeat-0.9.1, ansible-2.2.2
collected 1 item                                                                                                                                                                               

arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0] PASSED                                                                                   [100%]

----------------------------------------------------- generated xml file: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/logs/tr.xml ------------------------------------------------------
================================================================================== 1 passed in 317.49 seconds ==================================================================================
INFO:root:Can not get Allure report URL. Please check logs
jujoseph@c3e1277334f8:~/CHASSIS/CHASSIS/sonic-mgmt-int/tests$

Details if related

Copy link
Contributor

@abdosi abdosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@judyjoseph judyjoseph merged commit 83a186a into sonic-net:master Sep 1, 2022
@judyjoseph judyjoseph deleted the arp_neighbor_new branch September 7, 2022 17:12
yxieca pushed a commit that referenced this pull request Sep 8, 2022
…2437)

* Change the log messages from ERROR to INFO.
* Update the test_chassis_system_neigh test to check the mac address change of a neighbor.
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this pull request Sep 15, 2022
Update sonic-swss submodule pointer to include the following:
* f0138a2 [portmgr] Fixed the orchagent crash due to late arrival of notif (sonic-net#2431) ([sonic-net#2451](sonic-net/sonic-swss#2451))
* 7cfde48 Change the log messages in addKernelNeigh/Route from ERROR to INFO ([sonic-net#2437](sonic-net/sonic-swss#2437))
* 2c5116e [202205][counters] Improve performance by polling only configured ports buffer queue/pg counters ([sonic-net#2432](sonic-net/sonic-swss#2432))

Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this pull request Sep 16, 2022
Update sonic-swss submodule pointer to include the following:
* 04ce7be check state_db for po before sending ARP/ND pkts (sonic-net#2444) ([sonic-net#2450](sonic-net/sonic-swss#2450))
* f0138a2 [portmgr] Fixed the orchagent crash due to late arrival of notif (sonic-net#2431) ([sonic-net#2451](sonic-net/sonic-swss#2451))
* 7cfde48 Change the log messages in addKernelNeigh/Route from ERROR to INFO ([sonic-net#2437](sonic-net/sonic-swss#2437))
* 2c5116e [202205][counters] Improve performance by polling only configured ports buffer queue/pg counters ([sonic-net#2432](sonic-net/sonic-swss#2432))

Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this pull request Sep 21, 2022
Update sonic-swss submodule pointer to include the following:
* 8eea92e [202205][counters] Revert PR sonic-net#2432 for the buffer queue/pg counters improvement ([sonic-net#2462](sonic-net/sonic-swss#2462))
* 5d8636a [202205] Enhance orchagent and buffer manager in error handling (sonic-net#2414) ([sonic-net#2449](sonic-net/sonic-swss#2449))
* aa22237 [Everflow/ERSPAN] Set correct destination port and mac address when the nexthop is updated for ERSPAN mirror destination (sonic-net#2392) ([sonic-net#2455](sonic-net/sonic-swss#2455))
* 04ce7be check state_db for po before sending ARP/ND pkts (sonic-net#2444) ([sonic-net#2450](sonic-net/sonic-swss#2450))
* f0138a2 [portmgr] Fixed the orchagent crash due to late arrival of notif (sonic-net#2431) ([sonic-net#2451](sonic-net/sonic-swss#2451))
* 7cfde48 Change the log messages in addKernelNeigh/Route from ERROR to INFO ([sonic-net#2437](sonic-net/sonic-swss#2437))
* 2c5116e [202205][counters] Improve performance by polling only configured ports buffer queue/pg counters ([sonic-net#2432](sonic-net/sonic-swss#2432))

Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
prsunny pushed a commit to sonic-net/sonic-buildimage that referenced this pull request Sep 21, 2022
Update sonic-swss submodule pointer to include the following:
* 8eea92e [202205][counters] Revert PR #2432 for the buffer queue/pg counters improvement ([#2462](sonic-net/sonic-swss#2462))
* 5d8636a [202205] Enhance orchagent and buffer manager in error handling (#2414) ([#2449](sonic-net/sonic-swss#2449))
* aa22237 [Everflow/ERSPAN] Set correct destination port and mac address when the nexthop is updated for ERSPAN mirror destination (#2392) ([#2455](sonic-net/sonic-swss#2455))
* 04ce7be check state_db for po before sending ARP/ND pkts (#2444) ([#2450](sonic-net/sonic-swss#2450))
* f0138a2 [portmgr] Fixed the orchagent crash due to late arrival of notif (#2431) ([#2451](sonic-net/sonic-swss#2451))
* 7cfde48 Change the log messages in addKernelNeigh/Route from ERROR to INFO ([#2437](sonic-net/sonic-swss#2437))
* 2c5116e [202205][counters] Improve performance by polling only configured ports buffer queue/pg counters ([#2432](sonic-net/sonic-swss#2432))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants