Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Debian9-Z9100] config save is hanging indefinitely #1566

Closed
amitabhja opened this issue Apr 6, 2018 · 2 comments
Closed

[Debian9-Z9100] config save is hanging indefinitely #1566

amitabhja opened this issue Apr 6, 2018 · 2 comments
Assignees
@taoyl-ms
Labels
Platform: DellEMC

Comments

@amitabhja
Copy link

Description

config save is hangs indefinitely after deploying t0 topology minigraph from server
cannot take o/p of "show techsupport" as that cmd also hangs indefinitely
using the label 161
https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-stretch-all/lastSuccessfulBuild/artifact/target/sonic-broadcom.bin

Steps to reproduce the issue:

  1. deploy minigraph from server for t0 topology
  2. perform cmd "config save" from console
  3. the cli hangs indefintely

Describe the results you received:
<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>
root@sonic-z9100-02:~# config save
Existing file will be overwritten, continue? [y/N]: y
Running command: sonic-cfggen -d --print-data > /etc/sonic/config_db.json
<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>

the cmd hangs indefinitely

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>
dmin@sonic-z9100-02:~$ show ver
SONiC Software Version: SONiC.HEAD.161-7c8a635
Distribution: Debian 9.4
Kernel: 4.9.0-5-amd64
Build commit: 7c8a635
Build date: Tue Apr 3 11:25:06 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.161-7c8a635 263bcbed3b14 358.1 MB
docker-syncd-brcm latest 263bcbed3b14 358.1 MB
docker-orchagent-brcm HEAD.161-7c8a635 f5326eae6970 287 MB
docker-orchagent-brcm latest f5326eae6970 287 MB
docker-lldp-sv2 HEAD.161-7c8a635 d77ff6995334 297.1 MB
docker-lldp-sv2 latest d77ff6995334 297.1 MB
docker-dhcp-relay HEAD.161-7c8a635 92ada5e29818 280.1 MB
docker-dhcp-relay latest 92ada5e29818 280.1 MB
docker-database HEAD.161-7c8a635 746e6563e61f 278.7 MB
docker-database latest 746e6563e61f 278.7 MB
docker-teamd HEAD.161-7c8a635 b7ac2067fd06 284.1 MB
docker-teamd latest b7ac2067fd06 284.1 MB
docker-snmp-sv2 HEAD.161-7c8a635 7096ad720bf0 319.2 MB
docker-snmp-sv2 latest 7096ad720bf0 319.2 MB
docker-router-advertiser HEAD.161-7c8a635 a78cf51c9c00 276.4 MB
docker-router-advertiser latest a78cf51c9c00 276.4 MB
docker-platform-monitor HEAD.161-7c8a635 072d237209c4 298.3 MB
docker-platform-monitor latest 072d237209c4 298.3 MB
docker-fpm-quagga HEAD.161-7c8a635 02f60ab8b7b5 290.6 MB
docker-fpm-quagga latest 02f60ab8b7b5 290.6 MB

admin@sonic-z9100-02:~$
<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>

**Attach debug file `sudo generate_dump`:**

"show techsupport" cli also hangs indefinitely
@lguohan
Copy link
Collaborator

lguohan commented Aug 24, 2018

taoyl will introduce timeout and then you can see if you can repro.

@amitabhja
Copy link
Author

Verification Logs:
root@sonic:# config save
Existing file will be overwritten, continue? [y/N]: y
Running command: sonic-cfggen -d --print-data > /etc/sonic/config_db.json
root@sonic:#

root@sonic:~# show ver
SONiC Software Version: SONiC.HEAD.693-3f37b96
Distribution: Debian 9.5
Kernel: 4.9.0-7-amd64
Build commit: 3f37b96
Build date: Sun Aug 26 09:13:46 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.693-3f37b96 e934afcb65a3 336.6 MB
docker-syncd-brcm latest e934afcb65a3 336.6 MB
docker-orchagent-brcm HEAD.693-3f37b96 38a356f0df7a 257.6 MB
docker-orchagent-brcm latest 38a356f0df7a 257.6 MB
docker-lldp-sv2 HEAD.693-3f37b96 9b54765a5987 270.8 MB
docker-lldp-sv2 latest 9b54765a5987 270.8 MB
docker-dhcp-relay HEAD.693-3f37b96 0758dc72b19f 253.7 MB
docker-dhcp-relay latest 0758dc72b19f 253.7 MB
docker-database HEAD.693-3f37b96 33f38670a603 252.3 MB
docker-database latest 33f38670a603 252.3 MB
docker-teamd HEAD.693-3f37b96 d1df4e88a6a0 257.2 MB
docker-teamd latest d1df4e88a6a0 257.2 MB
docker-snmp-sv2 HEAD.693-3f37b96 791154a0e7cf 291.2 MB
docker-snmp-sv2 latest 791154a0e7cf 291.2 MB
docker-router-advertiser HEAD.693-3f37b96 0b3ab8f3fca2 250 MB
docker-router-advertiser latest 0b3ab8f3fca2 250 MB
docker-platform-monitor HEAD.693-3f37b96 5bf09433f1f1 283.2 MB
docker-platform-monitor latest 5bf09433f1f1 283.2 MB
docker-fpm-quagga HEAD.693-3f37b96 cd3a0dc65949 264.1 MB
docker-fpm-quagga latest cd3a0dc65949 264.1 MB

@dgsudharsan dgsudharsan mentioned this issue May 5, 2021
Closed
4 tasks
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this issue May 5, 2021
@dgsudharsan
sonic-utilities submodule update
8eaa652 
fff4051 Fixing serial number read to get from DB if it is populated (sonic-net#1580)
be974bf [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (sonic-net#1589)
9492eab Use swsscommon instead of swsssdk (sonic-net#1510)
0f4988b Add pg-drop script to sonic filesystem (sonic-net#1583)
cbe2159 [vnet] Add "vnet_route_check" script (sonic-net#1300)
9120766 Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (sonic-net#1530)
2e09b22 Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)
daall added a commit to daall/sonic-buildimage that referenced this issue May 5, 2021
@daall
[202012][utilities] Update submodule
9e13ee7 
* Add pg-drop script to sonic filesystem (sonic-net#1583)
* Fixing serial number read to get from DB if it is populated (sonic-net#1580)
* Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)
* Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (sonic-net#1530) (sonic-net#1592)

Signed-off-by: Danny Allen <daall@microsoft.com>
@daall daall mentioned this issue May 5, 2021
Closed
4 tasks
@dgsudharsan dgsudharsan mentioned this issue May 5, 2021
Merged
4 tasks
stephenxs added a commit to stephenxs/sonic-buildimage that referenced this issue May 6, 2021
@stephenxs
Advance submodule head for sonic-utilities
0e1bb1d 
68ea9efc Add pg-drop script to sonic filesystem (sonic-net#1583)
b216bf0a Fixing serial number read to get from DB if it is populated (sonic-net#1580)
fa7230c6 Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)

Signed-off-by: Stephen Sun <stephens@nvidia.com>
@stephenxs stephenxs mentioned this issue May 6, 2021
Closed
4 tasks
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this issue May 6, 2021
@dgsudharsan
sonic-utilities submodule update
d908f27 
9a88cb6 [sonic_installer] dont fail package migration (sonic-net#1591)
615e531 [show][config] Add new snmp commands (sonic-net#1347)
fff4051 Fixing serial number read to get from DB if it is populated (sonic-net#1580)
be974bf [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (sonic-net#1589)
9492eab Use swsscommon instead of swsssdk (sonic-net#1510)
0f4988b Add pg-drop script to sonic filesystem (sonic-net#1583)
cbe2159 [vnet] Add "vnet_route_check" script (sonic-net#1300)
9120766 Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (sonic-net#1530)
2e09b22 Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)
liat-grozovik pushed a commit that referenced this issue May 6, 2021
@stephenxs
[202012][utilities] Update submodule (#7542)
2fc748e 
68ea9efc Add pg-drop script to sonic filesystem (#1583)
b216bf0a Fixing serial number read to get from DB if it is populated (#1580)
fa7230c6 Handle the new db version which mellanox_buffer_migrator isn't interested (#1566)

Signed-off-by: Stephen Sun <stephens@nvidia.com>
yxieca pushed a commit that referenced this issue May 6, 2021
@dgsudharsan
sonic-utilities submodule update (#7532)
06377f0 
9a88cb6 [sonic_installer] dont fail package migration (#1591)
615e531 [show][config] Add new snmp commands (#1347)
fff4051 Fixing serial number read to get from DB if it is populated (#1580)
be974bf [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (#1589)
9492eab Use swsscommon instead of swsssdk (#1510)
0f4988b Add pg-drop script to sonic filesystem (#1583)
cbe2159 [vnet] Add "vnet_route_check" script (#1300)
9120766 Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (#1530)
2e09b22 Handle the new db version which mellanox_buffer_migrator isn't interested (#1566)
lguohan pushed a commit that referenced this issue May 7, 2021
@qiluo-msft
[sonic-utilities] submodule update (#7546)
dee7d8f 
Includes below commits

9a88cb6 2021-05-06 | [sonic_installer] dont fail package migration (#1591) [Stepan Blyshchak]
615e531 2021-05-05 | [show][config] Add new snmp commands (#1347) [Travis Van Duyn]
fff4051 2021-05-05 | Fixing serial number read to get from DB if it is populated (#1580) [Sudharsan Dhamal Gopalarathnam]
be974bf 2021-05-05 | [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (#1589) [Sumukha Tumkur Vani]
9492eab 2021-05-04 | Use swsscommon instead of swsssdk (#1510) [Andriy Yurkiv]
0f4988b 2021-05-04 | Add pg-drop script to sonic filesystem (#1583) [Andriy Yurkiv]
cbe2159 2021-05-04 | [vnet] Add "vnet_route_check" script (#1300) [Volodymyr Samotiy]
9120766 2021-05-03 | Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (#1530) [Qi Luo]
2e09b22 2021-05-03 | Handle the new db version which mellanox_buffer_migrator isn't interested (#1566) [Stephen Sun]
stepanblyschak pushed a commit to stepanblyschak/sonic-buildimage that referenced this issue May 10, 2021
@stephenxs
Handle the new db version which mellanox_buffer_migrator isn't intere…
2e09b22 
…sted (sonic-net#1566)

Enhancement: handle the case that no buffer change in the latest database version
Current, the following two versions are the same:
- The latest version changed by mellanox_buffer_migrator
- The latest version in CONFIG_DB

That won't be true if another part in CONFIG_DB is updated. In that case, the latest version in CONFIG_DB will be greater than the latest version in mellanox_buffer_migrator.
However, this can break the buffer migrator unit test:
- The db_migrator will always migrate the database to the latest version
- The config database version check will fail in case the latest version in the config database doesn't match that defined in the buffer migrator.

This is to support this case.

Signed-off-by: Stephen Sun <stephens@nvidia.com>
raphaelt-nvidia pushed a commit to raphaelt-nvidia/sonic-buildimage that referenced this issue May 23, 2021
@dgsudharsan @raphaelt-nvidia
sonic-utilities submodule update (sonic-net#7532)
bddd3a1 
9a88cb6 [sonic_installer] dont fail package migration (sonic-net#1591)
615e531 [show][config] Add new snmp commands (sonic-net#1347)
fff4051 Fixing serial number read to get from DB if it is populated (sonic-net#1580)
be974bf [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (sonic-net#1589)
9492eab Use swsscommon instead of swsssdk (sonic-net#1510)
0f4988b Add pg-drop script to sonic filesystem (sonic-net#1583)
cbe2159 [vnet] Add "vnet_route_check" script (sonic-net#1300)
9120766 Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (sonic-net#1530)
2e09b22 Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)
carl-nokia pushed a commit to carl-nokia/sonic-buildimage that referenced this issue Aug 7, 2021
@dgsudharsan
sonic-utilities submodule update (sonic-net#7532)
28537ad 
9a88cb6 [sonic_installer] dont fail package migration (sonic-net#1591)
615e531 [show][config] Add new snmp commands (sonic-net#1347)
fff4051 Fixing serial number read to get from DB if it is populated (sonic-net#1580)
be974bf [neighbor_advertiser] Use existing tunnel if present for creating tunnel mappings (sonic-net#1589)
9492eab Use swsscommon instead of swsssdk (sonic-net#1510)
0f4988b Add pg-drop script to sonic filesystem (sonic-net#1583)
cbe2159 [vnet] Add "vnet_route_check" script (sonic-net#1300)
9120766 Relax the install_requires, no need to exact version as long as there are no broken changes with future versions (sonic-net#1530)
2e09b22 Handle the new db version which mellanox_buffer_migrator isn't interested (sonic-net#1566)
taras-keryk pushed a commit to taras-keryk/sonic-buildimage that referenced this issue Apr 28, 2022
@stephenxs
Make advance_version_for_expected_database available for other db mig…
f1726fe 
…rator test cases as well (sonic-net#1614)

- What I did
Originally, the method advance_version_for_expected_database was introduced (in sonic-net#1566) to handle the case the latest version in CONFIG_DB is greater than the latest version in mellanox_buffer_migrator.
Now there are other database migrators whose test cases can also encounter this situation, like port auto-negotiation (sonic-net#1568) and port-channel for LACP key (sonic-net#1473).
So I would like to make the method public, available for all database migrators.
Related database migrator test cases have been updated accordingly.

- How to verify it
Run the unit test.

Signed-off-by: Stephen Sun <stephens@nvidia.com>
taras-keryk pushed a commit to taras-keryk/sonic-buildimage that referenced this issue Apr 28, 2022
@stephenxs
Fix the target db version of portchannel-key test (sonic-net#1842)
00948d0 
What I did
The port-channel key migrator was introduced in version 2_0_2 so the expected database version of the test case should be 2_0_2.
It was modified to 2_0_3 when the new version was introduced by mistake. This won't fail the test but disable the require its database version to be updated every time a new version is introduced. (Refer sonic-net#1566 and sonic-net#1614 for details)
This is to correct it by changing it back to 2_0_2.

Signed-off-by: Stephen Sun <stephens@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@taoyl-ms taoyl-ms

Labels
Platform: DellEMC
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lguohan @taoyl-ms @amitabhja