Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Debian9-Z9100]device crashed after issuing reboot cmd #1567

Closed
amitabhja opened this issue Apr 6, 2018 · 1 comment
Closed

[Debian9-Z9100]device crashed after issuing reboot cmd #1567

amitabhja opened this issue Apr 6, 2018 · 1 comment
Assignees

Comments

@amitabhja
Copy link

amitabhja commented Apr 6, 2018

Description

Device crashed after issuing reboot cmd
The device was in the process of rebooting when crash was seen.
We are using latest label 161:
https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-stretch-all/161/artifact/target/sonic-broadcom.bin

We are seeing this for a second time in image 154 also.
steps to repro:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

  1. load minigraph of t0 topology. it will hang indefinitely
  2. initiate reboot of device
  3. device crash and reboot
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Corefiles: image 161
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
neighsyncd.1523018483.64.core.gz
orchagent.1523018483.46.core.gz
python.1523018483.30.core.gz

teamsyncd.1523018487.33.core.gz
buffermgrd.1523018483.95.core.gz
fpmsyncd.1523018484.42.core.gz
intfsyncd.1523018483.61.core.gz
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Corefiles: image 154
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
orchagent.1523342750.45.core.gz
neighsyncd.1523342750.63.core.gz
intfsyncd.1523342750.60.core.gz
fpmsyncd.1523342750.42.core.gz
teamsyncd.1523342750.32.core.gz
syncd.1523342756.26.core.gz
python.1523342750.31.core.gz
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

**Output of `show version`:**

Show Version: image 161
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
root@sonic:/var/core# timed out waiting for input: auto-logout
admin@sonic:$
admin@sonic:
$ show ver
SONiC Software Version: SONiC.HEAD.161-7c8a635
Distribution: Debian 9.4
Kernel: 4.9.0-5-amd64
Build commit: 7c8a635
Build date: Tue Apr 3 11:25:06 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.161-7c8a635 263bcbed3b14 358.1 MB
docker-syncd-brcm latest 263bcbed3b14 358.1 MB
docker-orchagent-brcm HEAD.161-7c8a635 f5326eae6970 287 MB
docker-orchagent-brcm latest f5326eae6970 287 MB
docker-lldp-sv2 HEAD.161-7c8a635 d77ff6995334 297.1 MB
docker-lldp-sv2 latest d77ff6995334 297.1 MB
docker-dhcp-relay HEAD.161-7c8a635 92ada5e29818 280.1 MB
docker-dhcp-relay latest 92ada5e29818 280.1 MB
docker-database HEAD.161-7c8a635 746e6563e61f 278.7 MB
docker-database latest 746e6563e61f 278.7 MB
docker-teamd HEAD.161-7c8a635 b7ac2067fd06 284.1 MB
docker-teamd latest b7ac2067fd06 284.1 MB
docker-snmp-sv2 HEAD.161-7c8a635 7096ad720bf0 319.2 MB
docker-snmp-sv2 latest 7096ad720bf0 319.2 MB
docker-router-advertiser HEAD.161-7c8a635 a78cf51c9c00 276.4 MB
docker-router-advertiser latest a78cf51c9c00 276.4 MB
docker-platform-monitor HEAD.161-7c8a635 072d237209c4 298.3 MB
docker-platform-monitor latest 072d237209c4 298.3 MB
docker-fpm-quagga HEAD.161-7c8a635 02f60ab8b7b5 290.6 MB
docker-fpm-quagga latest 02f60ab8b7b5 290.6 MB

admin@sonic:~$

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Show version: image 154
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
root@sonic:~# show ver
SONiC Software Version: SONiC.HEAD.154-e69ce25
Distribution: Debian 9.4
Kernel: 4.9.0-5-amd64
Build commit: e69ce25
Build date: Wed Mar 28 00:28:39 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.154-e69ce25 808e4bcc594f 358.1 MB
docker-syncd-brcm latest 808e4bcc594f 358.1 MB
docker-orchagent-brcm HEAD.154-e69ce25 1c46dd25a7e7 287 MB
docker-orchagent-brcm latest 1c46dd25a7e7 287 MB
docker-lldp-sv2 HEAD.154-e69ce25 645690db4632 297.1 MB
docker-lldp-sv2 latest 645690db4632 297.1 MB
docker-dhcp-relay HEAD.154-e69ce25 426ad89eeae7 280.1 MB
docker-dhcp-relay latest 426ad89eeae7 280.1 MB
docker-database HEAD.154-e69ce25 17edc49cb2d3 278.8 MB
docker-database latest 17edc49cb2d3 278.8 MB
docker-teamd HEAD.154-e69ce25 6faa0ba0dde5 284.1 MB
docker-teamd latest 6faa0ba0dde5 284.1 MB
docker-snmp-sv2 HEAD.154-e69ce25 17ef37ef7e39 319.3 MB
docker-snmp-sv2 latest 17ef37ef7e39 319.3 MB
docker-router-advertiser HEAD.154-e69ce25 3a813b8ffad0 276.4 MB
docker-router-advertiser latest 3a813b8ffad0 276.4 MB
docker-platform-monitor HEAD.154-e69ce25 f9215595cd58 298.4 MB
docker-platform-monitor latest f9215595cd58 298.4 MB
docker-fpm-quagga HEAD.154-e69ce25 24d35e3cdc11 290.6 MB
docker-fpm-quagga latest 24d35e3cdc11 290.6 MB

root@sonic:~#
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Describe the results you received:
Console Logs: image 161
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
root@sonic:~# reboot
[ OK ] Stopped Platform monitor container.
[ OK ] Stopped SNMP container.
[ OK ] Stopped LLDP container.
[ OK ] Stopped BGP container.
[ OK ] Stopped DHCP relay container.
[ OK ] Stopped Router advertiser container.
Stopping TEAMD container...
Stopping switch state service...
[ OK ] Stopped TEAMD container.
[ OK ] Stopped switch state service.
Stopping Database container...
Stopping Opennsl kernel modules init...
[ OK ] Stopped Opennsl kernel modules init.
[ 4190.937214] BUG: unable to handle kernel paging request at ffffffffc069d0a8
[ 4191.020861] IP: [] do_vfs_ioctl+0x8a/0x600
[ 4191.088783] PGD 16140d067 [ 4191.119093] PUD 16140f067
PMD 4278f4067 [ 4191.166181] PTE 0
[ 4191.189207]
[ 4191.207021] Oops: 0000 [#1] SMP
[ 4191.244623] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4192.157817] CPU: 1 PID: 2651 Comm: CtrEvict.0 Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4192.278756] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4192.363237] task: ffff912169634f00 task.stack: ffffa071c3c4c000
[ 4192.434171] RIP: 0010:[] [] do_vfs_ioctl+0x8a/0x600
[ 4192.531251] RSP: 0018:ffffa071c3c4fea0 EFLAGS: 00010a87
[ 4192.594895] RAX: ffffffffc069d060 RBX: ffff9121a90b5001 RCX: 00007f11b0557c60
[ 4192.680415] RDX: 0000000000004c16 RSI: 0000000000004c16 RDI: 0000000000000035
[ 4192.765938] RBP: ffff9121a90b5000 R08: 0000000000000003 R09: 00000000000001a8
[ 4192.851462] R10: ffff9121a90b5038 R11: ffff9121694bf9c0 R12: 00007f11b0557c60
[ 4192.936986] R13: ffff91216afc7918 R14: 00007f11b0557c60 R15: 0000000000000010
[ 4193.022512] FS: 00007f11b0558700(0000) GS:ffff9121bfc80000(0000) knlGS:0000000000000000
[ 4193.119498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4193.188357] CR2: ffffffffc069d0a8 CR3: 00000003e9678000 CR4: 0000000000100670
[ 4193.273882] Stack:
[ 4193.297949] ffff91216a134c00 c2bc5e190b1e8b2e 0000000000000000 0000000000000009
[ 4193.387012] 0000000000000189 00000000014b03d0 0000000000000000 00000000ffffffff
[ 4193.476078] ffffffffb7ef7bff c2bc5e190b1e8b2e ffff9121a90b5001 ffff9121a90b5000
[ 4193.565143] Call Trace:
[ 4193.594428] [] ? SyS_futex+0x7f/0x160
[ 4193.658081] [] ? SyS_ioctl+0x74/0x80
[ 4193.720690] [] ? system_call_fast_compare_end+0xc/0xb7
[ 4193.802051] Code: 20 c0 0f 84 2d 03 00 00 81 fa 78 58 04 c0 0f 84 ee 02 00 00 41 0f b7 45 00 66 25 00 f0 66 3d 00 80 0f 84 a3 04 00 00 48 8b 45 28 <48> 8b 40 48 48 85 c0 0f 84 56 02 00 00 4c 89 e2 48 89 ef ff d0
[ 4194.034961] RIP [] do_vfs_ioctl+0x8a/0x600
[ 4194.103925] RSP
[ 4194.145700] CR2: ffffffffc069d0a8
[ 4194.185397] ---[ end trace 1e4ef867d447dd0b ]---
[ 4194.999878] BUG: unable to handle kernel paging request at ffffffffc069af67
[ 4195.083525] IP: [] 0xffffffffc069af67
[ 4195.146232] PGD 16140d067 [ 4195.176545] PUD 16140f067
PMD 4278f4067 [ 4195.223632] PTE 0
[ 4195.246655]
[ 4195.264468] Oops: 0010 [#2] SMP
[ 4195.302072] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4196.214432] CPU: 3 PID: 2630 Comm: bcmINTR Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4196.332247] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4196.416726] task: ffff912169b2ef80 task.stack: ffffa071c3bcc000
[ 4196.487661] RIP: 0010:[] [] 0xffffffffc069af67
[ 4196.579534] RSP: 0018:ffffa071c3bcfda0 EFLAGS: 00010246
[ 4196.643182] RAX: 0000000000000004 RBX: ffffffffc069d5f0 RCX: 0000000000000000
[ 4196.728699] RDX: 0000000000000001 RSI: ffff912169b2f9c0 RDI: ffff9121bfd98940
[ 4196.814217] RBP: 00007f11c2388ce0 R08: ffff912169b2f330 R09: 0000000000000000
[ 4196.899734] R10: ffff9121a8d34800 R11: 0000000000000000 R12: 0000000000000028
[ 4196.985253] R13: ffffffffc069d608 R14: 00007f11c2388ce0 R15: 00007f11c2389680
[ 4197.070777] FS: 00007f11c2389680(0000) GS:ffff9121bfd80000(0000) knlGS:0000000000000000
[ 4197.167759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4197.236615] CR2: ffffffffc069af67 CR3: 00000003e9678000 CR4: 0000000000100670
[ 4197.322132] Stack:
[ 4197.346195] 0000000001412000 ffff91216a006400 0000000000000e70 d94e8693c28fed5b
[ 4197.435251] 0000000000000000 0000000000000000 ffff912169b2ef80 ffffffffb7eb7b10
[ 4197.524310] ffffffffc069d5f8 ffffffffc069d5f8 0000000000000000 0000000000000000
[ 4197.613265] Call Trace:
[ 4197.642543] [] ? prepare_to_wait_event+0xf0/0xf0
[ 4197.717655] [] ? do_vfs_ioctl+0x9f/0x600
[ 4197.784427] [] ? SyS_futex+0x7f/0x160
[ 4197.848075] [] ? SyS_ioctl+0x74/0x80
[ 4197.910685] [] ? system_call_fast_compare_end+0xc/0xb7
[ 4197.992040] Code: Bad RIP value.
[ 4198.031940] RIP [] 0xffffffffc069af67
[ 4198.095687] RSP
[ 4198.137457] CR2: ffffffffc069af67
[ 4198.177145] ---[ end trace 1e4ef867d447dd0c ]---
[ 4198.177152] BUG: unable to handle kernel paging request at ffffffffc0795723
[ 4198.177157] IP: [] 0xffffffffc0795723
[ 4198.177159] PGD 16140d067
[ 4198.177160] PUD 16140f067
[ 4198.177160] PMD 4278f4067
[ 4198.177161] PTE 0
[ 4198.177161]
[ 4198.177163] Oops: 0010 [#3] SMP
[ 4198.177206] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca
[ 4198.177209] ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4198.177212] CPU: 0 PID: 2632 Comm: SOC KNET RX Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4198.177213] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4198.177215] task: ffff912168a40280 task.stack: ffffa071c3c14000
[ 4198.177217] RIP: 0010:[] [] 0xffffffffc0795723
[ 4198.177218] RSP: 0018:ffffa071c3c17b30 EFLAGS: 00010246
[ 4198.177219] RAX: 0000000000000004 RBX: 0000000000000000 RCX: 0000000000000000
[ 4198.177220] RDX: 0000000000000001 RSI: ffff912168a40cc0 RDI: ffff9121bfc18940
[ 4198.177221] RBP: 0000000000000000 R08: 000003d0b9b29533 R09: 0000000000000000
[ 4198.177222] R10: 0000000000000000 R11: ffff9121a82ca000 R12: 0000000000000000
[ 4198.177223] R13: ffffffffc079fcc0 R14: ffffa071c3c17c08 R15: ffff9121a877a000
[ 4198.177225] FS: 00007f11c2383700(0000) GS:ffff9121bfc00000(0000) knlGS:0000000000000000
[ 4198.177226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4198.177227] CR2: ffffffffc0795723 CR3: 00000003e9678000 CR4: 0000000000100670
[ 4198.177228] Stack:
[ 4198.177232] ffffffffc079fcc0 0000000100000000 ffffffffc079fcc0 ffff912168a40280
[ 4198.177234] 0000000000000000 0000000000000000 ffff912168a40280 ffffffffb7eb7b10
[ 4198.177236] ffffffffc079fcc8 ffffffffc079fcc8 0000000000000000 ffffa071c3c17c08
[ 4198.177237] Call Trace:
[ 4198.177244] [] ? prepare_to_wait_event+0xf0/0xf0
[ 4198.177250] [] ? do_vfs_ioctl+0x9f/0x600
[ 4198.177253] [] ? SyS_futex+0x7f/0x160
[ 4198.177255] [] ? SyS_ioctl+0x74/0x80
[ 4198.177258] [] ? system_call_fast_compare_end+0xc/0xb7
[ 4198.177267] Code: Bad RIP value.
[ 4198.177269] RIP [] 0xffffffffc0795723
[ 4198.177270] RSP
[ 4198.177270] CR2: ffffffffc0795723
[ 4198.177273] ---[ end trace 1e4ef867d447dd0d ]---
[ 4201.503605] sched: RT throttling activated
[ OK ] Stopped Database container.
Stopping Docker Application Container E[ 4201.770135] BUG: unable to handle kernel paging request at ffffffffc069d0c8
[ 4201.859325] IP: [] filp_close+0x1f/0x70
[ 4201.924121] PGD 16140d067 [ 4201.954430] PUD 16140f067
PMD 4278f4067 [ 4202.001516] PTE 0
[ 4202.024538]
[ 4202.042353] Oops: 0000 [#4] SMP
[ 4202.079961] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4202.993051] CPU: 1 PID: 2670 Comm: syncd Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4203.108783] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4203.193262] task: ffff9121694fe2c0 task.stack: ffffa071c3c34000
[ 4203.264197] RIP: 0010:[] [] filp_close+0x1f/0x70
[ 4203.358161] RSP: 0018:ffffa071c3c37cb8 EFLAGS: 00010246
[ 4203.421809] RAX: ffffffffc069d060 RBX: ffff9121a90b5000 RCX: ffff9121694be710
[ 4203.507332] RDX: ffff9121694be720 RSI: ffff9121694be680 RDI: ffff9121a90b5000
[ 4203.592856] RBP: ffff9121694be680 R08: fffff73b0fa26c20 R09: 00000000000000a8
[ 4203.678373] R10: 0000000000000000 R11: 0000000000000500 R12: 0000000000000000
[ 4203.763891] R13: 0000000000000001 R14: 0000000000000035 R15: ffff9121694be6a8
[ 4203.849416] FS: 00007f11c2571740(0000) GS:ffff9121bfc80000(0000) knlGS:0000000000000000
[ 4203.946400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4204.015254] CR2: ffffffffc069d0c8 CR3: 00000003eadac000 CR4: 0000000000100670
[ 4204.100771] Stack:
[ 4204.124834] 00000000000001ff 0000000000000000 ffff9121694be680 ffffffffb802123c
[ 4204.213896] ffff9121694fe2c0 ffff9121694fe9d8 ffff91216a134c00 0000000000000000
[ 4204.302956] 00007f11bc839601 ffff91216a134c68 ffffffffb7e79e37 00000000000000c3
[ 4204.392015] Call Trace:
[ 4204.421291] [] ? put_files_struct+0x6c/0xc0
[ 4204.491188] [] ? do_exit+0x2b7/0xae0
[ 4204.553794] [] ? do_group_exit+0x3a/0xa0
[ 4204.620570] [] ? get_signal+0x297/0x640
[ 4204.686304] [] ? do_signal+0x36/0x6a0
[ngine...
4204.749953] [] ? group_send_sig_info+0x16/0x40
[ 4204.833393] [] ? poll_select_copy_remaining+0xdf/0x150
[ 4204.914752] [] ? exit_to_usermode_loop+0x71/0xb0
[ 4204.989858] [] ? syscall_return_slowpath+0x54/0x60
[ 4205.067051] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 4205.149448] Code: 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48
[ 4205.379026] RIP [] filp_close+0x1f/0x70
[ 4205.444857] RSP
[ 4205.486626] CR2: ffffffffc069d0c8
[ 4205.526314] ---[ end trace 1e4ef867d447dd0e ]---
[ 4205.587472] Fixing recursive fault but reboot is needed!
[ 4207.104527] BUG: unable to handle kernel paging request at ffffffffc069d0c8
[ 4207.188190] IP: [] filp_close+0x1f/0x70
[ 4207.252992] PGD 16140d067 [ 4207.283304] PUD 16140f067
PMD 4278f4067 [ 4207.330399] PTE 0
[ 4207.353428]
[ 4207.371248] Oops: 0000 [#5] SMP
[ 4207.408858] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4208.323028] CPU: 2 PID: 2650 Comm: bcmCNTR.0 Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4208.442930] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4208.527413] task: ffff9121696341c0 task.stack: ffffa071c3c44000
[ 4208.598354] RIP: 0010:[] [] filp_close+0x1f/0x70
[ 4208.692326] RSP: 0018:ffffa071c3c47cb8 EFLAGS: 00010246
[ 4208.755978] RAX: ffffffffc069d060 RBX: ffff9121a90b5000 RCX: ffff9121694bfa50
[ 4208.841503] RDX: ffff9121694bfa60 RSI: ffff9121694bf9c0 RDI: ffff9121a90b5000
[ 4208.927026] RBP: ffff9121694bf9c0 R08: fffff73b0fa5c820 R09: ffffffffb8b18c80
[ 4209.012551] R10: ffffffffb8b18c40 R11: ffffffffffffffe4 R12: 0000000000000000
[ 4209.098074] R13: 0000000000000001 R14: 0000000000000035 R15: ffff9121694bf9e8
[ 4209.183600] FS: 00007f11b0560700(0000) GS:ffff9121bfd00000(0000) knlGS:0000000000000000
[ 4209.280592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4209.349451] CR2: ffffffffc069d0c8 CR3: 00000003e9678000 CR4: 0000000000100670
[ 4209.434975] Stack:
[ 4209.459042] 00000000000001ff 0000000000000000 ffff9121694bf9c0 ffffffffb802123c
[ 4209.548112] ffff9121696341c0 ffff9121696348d8 ffff91216a006400 0000000000000000
[ 4209.637177] 00000000ffffff00 ffff91216a006468 ffffffffb7e79e37 0000000000000000
[ 4209.726249] Call Trace:
[ 4209.755534] [] ? put_files_struct+0x6c/0xc0
[ 4209.825437] [] ? do_exit+0x2b7/0xae0
[ 4209.888052] [] ? do_group_exit+0x3a/0xa0
[ 4209.954833] [] ? get_signal+0x297/0x640
[ 4210.020573] [] ? do_signal+0x36/0x6a0
[ 4210.084229] [] ? do_vfs_ioctl+0x9f/0x600
[ 4210.151010] [] ? exit_to_usermode_loop+0x71/0xb0
[ 4210.226125] [] ? syscall_return_slowpath+0x54/0x60
[ 4210.303322] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 4210.385724] Code: 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48
[ 4210.618648] RIP [] filp_close+0x1f/0x70
[ 4210.684486] RSP
[ 4210.726261] CR2: ffffffffc069d0c8
[ 4210.765956] ---[ end trace 1e4ef867d447dd0f ]---
[ 4210.831914] Fixing recursive fault but reboot is needed!
[ 4210.911384] BUG: unable to handle kernel paging request at ffffffffc069d0c8
[ 4210.995030] IP: [] filp_close+0x1f/0x70
[ 4211.059819] PGD 16140d067 [ 4211.090128] PUD 16140f067
PMD 4278f4067 [ 4211.137216] PTE 0
[ 4211.160237]
[ 4211.178050] Oops: 0000 [#6] SMP
[ 4211.215653] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc iTCO_wdt iTCO_vendor_support evdev intel_powerclamp coretemp kvm_intel dcdbas kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr intel_cstate sg lpc_ich mfd_core ipmi_msghandler shpchp button acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ehci_pci i2c_i801 ahci ehci_hcd i2c_smbus libahci usbcore usb_common libata igb scsi_mod i2c_algo_bit dca ptp i2c_ismt pps_core [last unloaded: linux_kernel_bde]
[ 4212.128224] CPU: 0 PID: 2537 Comm: syncd Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 4212.243956] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 4212.328435] task: ffff91216a236200 task.stack: ffffa071c3a54000
[ 4212.399375] RIP: 0010:[] [] filp_close+0x1f/0x70
[ 4212.493333] RSP: 0018:ffffa071c3a57cb8 EFLAGS: 00010246
[ 4212.556980] RAX: ffffffffc069d060 RBX: ffff9121a90b5000 RCX: ffff91216a247800
[ 4212.642504] RDX: ffff9121694be9e0 RSI: ffff9121694be940 RDI: ffff9121a90b5000
[ 4212.728029] RBP: ffff9121694be940 R08: ffff9121a7188000 R09: 0000000000000083
[ 4212.813551] R10: 0000000000000000 R11: 0000000000000440 R12: 0000000000000000
[ 4212.899069] R13: 0000000000000001 R14: 0000000000000035 R15: ffff9121694be968
[ 4212.984594] FS: 00007f11c2571740(0000) GS:ffff9121bfc00000(0000) knlGS:0000000000000000
[ 4213.081577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4213.150431] CR2: ffffffffc069d0c8 CR3: 000000042cbe2000 CR4: 0000000000100670
[ 4213.235949] Stack:
[ 4213.260010] 00000000000001ff 0000000000000000 ffff9121694be940 ffffffffb802123c
[ 4213.348964] ffff91216a236200 ffff91216a236918 ffff91216a006400 0000000000000000
[ 4213.438022] 0000000000000001 ffff91216a006468 ffffffffb7e79e37 0000000000000000
[ 4213.527082] Call Trace:
[ 4213.556357] [] ? put_files_struct+0x6c/0xc0
[ 4213.626255] [] ? do_exit+0x2b7/0xae0
[ 4213.688861] [] ? do_group_exit+0x3a/0xa0
[ 4213.755636] [] ? get_signal+0x297/0x640
[ 4213.821370] [] ? do_signal+0x36/0x6a0
[ 4213.885019] [] ? do_send_sig_info+0x65/0x90
[ 4213.954921] [] ? exit_to_usermode_loop+0x71/0xb0
[ 4214.030027] [] ? syscall_return_slowpath+0x54/0x60
[ 4214.107219] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 4214.189617] Code: 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48
[ 4214.418882] RIP [] filp_close+0x1f/0x70
[ 4214.484712] RSP
[ 4214.526483] CR2: ffffffffc069d0c8
[ 4214.566170] ---[ end trace 1e4ef867d447dd10 ]---
[ 4214.627403] Fixing recursive fault but reboot is needed!
[ OK ] Stopped Docker Application Container Engine.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Slices.
[ OK ] Removed slice User and Session Slice.
[ OK ] Stopped target Paths.
[ OK ] Stopped target Sockets.
[ OK ] Closed Syslog Socket.
[ OK ] Closed D-Bus System Message Bus Socket.
[ OK ] Stopped target Network.
Stopping ifup for Ethernet76...
Stopping ifup for Ethernet80...
Stopping ifup for Ethernet40...
Stopping ifup for Ethernet28...
Stopping ifup for Ethernet36...
Stopping ifup for Ethernet112...
Stopping ifup for Ethernet52...
Stopping ifup for Ethernet12...
Stopping ifup for Ethernet84...
Stopping ifup for Ethernet116...
Stopping ifup for Ethernet92...
Stopping ifup for Ethernet20...
Stopping ifup for Ethernet68...
Stopping ifup for Ethernet56...
Stopping ifup for Ethernet8...
Stopping ifup for Ethernet120...
Stopping ifup for Ethernet4...
Stopping ifup for Ethernet96...
Stopping ifup for Ethernet100...
Stopping ifup for Ethernet60...
Stopping ifup for Ethernet24...
Stopping ifup for eth0...
Stopping ifup for Ethernet88...
Stopping ifup for Ethernet44...
Stopping ifup for Ethernet32...
Stopping Raise network interfaces...
Stopping ifup for Ethernet108...
Stopping ifup for Ethernet16...
Stopping ifup for Ethernet72...
Stopping ifup for Ethernet104...
Stopping ifup for Ethernet0...
Stopping ifup for Ethernet124...
Stopping ifup for Ethernet48...
Stopping ifup for Ethernet64...
[ OK ] Closed Docker Socket for the API.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped target Swap.
Stopping Update UTMP about System Boot/Shutdown...
[ OK ] Stopped target Encrypted Volumes.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
[ OK ] Stopped Forward Password Requests to Wall Directory Watch.
Stopping Load/Save Random Seed...
[ OK ] Stopped Load/Save Random Seed.
[ OK ] Stopped ifup for Ethernet80.
[ OK ] Stopped ifup for Ethernet76.
[ OK ] Stopped ifup for Ethernet40.
[ OK ] Stopped ifup for Ethernet28.
[ OK ] Stopped ifup for Ethernet36.
[ OK ] Stopped ifup for Ethernet112.
[ OK ] Stopped ifup for Ethernet52.
[ OK ] Stopped ifup for Ethernet12.
[ OK ] Stopped ifup for Ethernet84.
[ OK ] Stopped ifup for Ethernet116.
[ OK ] Stopped ifup for Ethernet92.
[ OK ] Stopped ifup for Ethernet20.
[ OK ] Stopped ifup for Ethernet68.
[ OK ] Stopped ifup for Ethernet56.
[ OK ] Stopped ifup for Ethernet8.
[ OK ] Stopped ifup for Ethernet120.
[ OK ] Stopped ifup for Ethernet4.
[ OK ] Stopped ifup for Ethernet96.
[ OK ] Stopped ifup for Ethernet100.
[ OK ] Stopped ifup for Ethernet60.
[ OK ] Stopped ifup for Ethernet24.
[ OK ] Stopped Update UTMP about System Boot/Shutdown.
[ OK ] Stopped ifup for Ethernet88.
[ OK ] Stopped ifup for Ethernet44.
[ OK ] Stopped ifup for Ethernet32.
[ OK ] Stopped ifup for Ethernet108.
[ OK ] Stopped ifup for Ethernet16.
[ OK ] Stopped ifup for Ethernet72.
[ OK ] Stopped ifup for Ethernet104.
[ OK ] Stopped ifup for Ethernet0.
Unmounting /var/log...
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Stopped Raise network interfaces.
[ OK ] Stopped ifup for Ethernet124.
[ OK ] Stopped ifup for Ethernet48.
[ OK ] Stopped ifup for Ethernet64.
[ OK ] Unmounted /var/log.
[ OK ] Stopped ifup for eth0.
[ OK ] Stopped target Network (Pre).
Stopping netfilter persistent configuration...
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Stopped netfilter persistent configuration.
[ OK ] Stopped target Local File Systems.
Unmounting /host...
Unmounting /var/lib/docker...
Unmounting /boot...
[ OK ] Stopped Load Kernel Modules.
[FAILED] Failed unmounting /host.
[ OK ] Unmounted /var/lib/docker.
[ OK ] Unmounted /boot.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Stopped Remount Root and Kernel File Systems.
[ OK ] Reached target Shutdown.
[ 4446.916400] watchdog: watchdog0: watchdog did not stop!
[ 4446.981305] systemd-shutdow: 38 output lines suppressed due to ratelimiting
[ 4447.084282] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 4447.175080] systemd-journald[282]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 4447.274004] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 4447.362733] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
[ 4447.444028] systemd-shutdown[1]: Unmounting file systems.
[ 4447.509118] systemd-shutdown[1]: Remounting '/host' read-only with options 'data=ordered'.
[ 4447.637749] systemd-shutdown[1]: Unmounting /host.
[ 4447.700104] systemd-shutdown[1]: Remounting '/' read-only with options 'lowerdir=/root,upperdir=/root/host/image-HEAD.161-7c8a635/rw,workdir=/root/host/image-HEAD.161-7c8a635/work'.
[ 4447.911335] systemd-shutdown[1]: Remounting '/host' read-only with options 'data=ordered'.
[ 4448.010656] systemd-shutdown[1]: Unmounting /host.
[ 4448.072192] systemd-shutdown[1]: Could not unmount /host: Device or resource busy
[ 4448.231587] kvm: exiting hardware virtualization
[ 4448.287258] sd 4:0:0:0: [sda] Synchronizing SCSI cache
[ 4448.773730] reboot: Restarting system
[ 4448.817621] reboot: machine restart
[ 4448.859705] ACPI MEMORY or I/O RESET_REG.

BIOS Boot Selector for Z9100
Primary BIOS Version 3.23.0.2-6

SMF Version: MSS 2.1, FPGA 2.1
Last POR=0x11, Reset Cause=0x55

CPGC Memtest ................................ PASS

CPGC Memtest ................................ PASS

Version 2.17.1245. Copyright (C) 2016 American Megatrends, Inc.
BIOS Date: 12/16/2016 19:05:20 Ver: 0ACBZ028
Press or to enter setup.

Booting `SONiC-OS-HEAD.161-7c8a635'

                                                        Loading SONiC-OS OS initial ramdisk ...


                                         [*     ] A start job is running for Raise network interfaces (5s / 5mi[  OK  ] Started Raise network interfaces.

[ OK ] Reached target Network.
Starting Permit User Sessions...
Starting Docker Application Container Engine...
Starting OpenBSD Secure Shell server...
[ OK ] Reached target Network is Online.
Starting /etc/rc.local Compatibility...
[ 65.978761] rc.local[596]: + [ ! -e /host/machine.conf ]
Starting LSB: Start NTP daemon...
[ 66.110522] rc.local[596]: + . /host/machine.conf
[ OK ] Started Permit User Sessions.
[ 66.224632] rc.local[596]: + onie_version=3.23.1.2-5
[ OK ] Started /etc/rc.local Compatibility.
[ 66.344569] rc.local[596]: + onie_vendor_id=674
[ OK ] Started Getty on tty1.
[ 66.476406] rc.local[596]: + onie_platform=x86_64-dell_z9100_c2538-r0
[ OK ] Started Serial Getty on ttyS1.
[ 66.620364] rc.local[596]: + onie_machine=dell_z9100_c2538
[ OK ] Reached target Login Prompts.
[ 66.752365] rc.local[596]: + onie_machine_rev=0
[ OK ] Started LSB: Start NTP daemon.
[ 66.872402] rc.local[596]: + onie_arch=x86_64
[ 66.992380] rc.local[596]: + onie_config_version=1
[ 67.052299] rc.local[596]: + onie_build_date=2017-01-03T11:44-0800
[ 67.132539] rc.local[596]: + onie_partition_type=gpt
[ OK ] Started OpenBSD Secure Shell server.
[ 67.208383] rc.local[596]: + onie_kernel_version=4.1.28
[ 67.356455] rc.local[596]: + onie_firmware=bios
[ 67.416420] rc.local[596]: + onie_switch_asic=bcm
[ 67.476625] rc.local[596]: + onie_skip_ethmgmt_macs=no
[ 67.548411] rc.local[596]: + echo install platform dependent packages at the first boot time
[ 67.656422] rc.local[596]: install platform dependent packages at the first boot time
[ 67.755662] rc.local[596]: + cat /etc/sonic/sonic_version.yml
[ 67.847619] rc.local[596]: + grep build_version
[ 67.910202] rc.local[596]: + cut -f2 -d
[ 67.966498] rc.local[596]: + eval sonic_version='HEAD.161-7c8a635'
[ 68.044448] rc.local[596]: + sonic_version=HEAD.161-7c8a635
[ 68.120495] rc.local[596]: + [ -f /host/image-HEAD.161-7c8a635/platform/firsttime ]
[ OK ] Started Docker Application Container Engine.
[ 68.224388] rc.local[596]: + exit 0
Starting Database container...

Debian GNU/Linux 9 sonic ttyS1

sonic login:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Corefiles:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
neighsyncd.1523018483.64.core.gz
orchagent.1523018483.46.core.gz
python.1523018483.30.core.gz

teamsyncd.1523018487.33.core.gz
buffermgrd.1523018483.95.core.gz
fpmsyncd.1523018484.42.core.gz
intfsyncd.1523018483.61.core.gz
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Console Logs: image 154
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>
root@sonic:~# reboot
[ OK ] Stopped SNMP container.
[ OK ] Stopped BGP container.
[ OK ] Stopped LLDP container.
[ OK ] Stopped Router advertiser container.
[ OK ] Stopped Platform monitor container.
[ OK ] Stopped DHCP relay container.
Stopping TEAMD container...
Stopping switch state service...
[ OK ] Stopped TEAMD container.
[ OK ] Stopped switch state service.
Stopping Database container...
Stopping Opennsl kernel modules init...
[ OK ] Stopped Opennsl kernel module[ 1257.201219] BUG: unable to handle kernel paging request at ffffffffc05970a8
[ 1257.296011] IP: [] do_vfs_ioctl+0x8a/0x600
[ 1257.363933] PGD 28000d067 [ 1257.394243] PUD 28000f067
PMD 427896067 [ 1257.441330] PTE 0
[ 1257.464356]
[ 1257.482170] Oops: 0000 [#1] SMP
[ 1257.519780] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mods init.
igb usbcore i2c_algo_bit dca usb_common ptp pps_core [last unloaded: linux_kernel_bde]
[ 1258.442770] CPU: 0 PID: 2646 Comm: bcmL2X.0 Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1258.561623] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1258.646100] task: ffff97cc2a7e43c0 task.stack: ffffa75643bdc000
[ 1258.717034] RIP: 0010:[] [] do_vfs_ioctl+0x8a/0x600
[ 1258.814121] RSP: 0018:ffffa75643bdfea0 EFLAGS: 00010a87
[ 1258.877772] RAX: ffffffffc0597060 RBX: ffff97cc680d7c01 RCX: 00007fc3c400a9e0
[ 1258.963301] RDX: 0000000000004c16 RSI: 0000000000004c16 RDI: 0000000000000035
[ 1259.048831] RBP: ffff97cc680d7c00 R08: 0000000000000003 R09: 00000000000001a8
[ 1259.134361] R10: ffff97cc680d7c38 R11: ffff97cc2be35c00 R12: 00007fc3c400a9e0
[ 1259.219886] R13: ffff97cc2bca6ed8 R14: 00007fc3c400a9e0 R15: 0000000000000002
[ 1259.305417] FS: 00007fc3c400b700(0000) GS:ffff97cc7fc00000(0000) knlGS:0000000000000000
[ 1259.402410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1259.471268] CR2: ffffffffc05970a8 CR3: 000000042949a000 CR4: 0000000000100670
[ 1259.556792] Stack:
[ 1259.580859] 0000000000000000 ac20ad88ec181d98 0000000000000000 0000000000000009
[ 1259.669929] 0000000000000189 0000000001daa2b0 0000000000000000 00000000ffffffff
[ 1259.759001] ffffffffad6f7bff ac20ad88ec181d98 ffff97cc680d7c01 ffff97cc680d7c00
[ 1259.848063] Call Trace:
[ 1259.877341] [] ? SyS_futex+0x7f/0x160
[ 1259.940993] [] ? SyS_ioctl+0x74/0x80
[ 1260.003603] [] ? system_call_fast_compare_end+0xc/0xb7
[ 1260.084958] Code: 20 c0 0f 84 2d 03 00 00 81 fa 78 58 04 c0 0f 84 ee 02 00 00 41 0f b7 45 00 66 25 00 f0 66 3d 00 80 0f 84 a3 04 00 00 48 8b 45 28 <48> 8b 40 48 48 85 c0 0f 84 56 02 00 00 4c 89 e2 48 89 ef ff d0
[ 1260.317149] RIP [] do_vfs_ioctl+0x8a/0x600
[ 1260.386113] RSP
[ 1260.427888] CR2: ffffffffc05970a8
[ 1260.467586] ---[ end trace f5b3563022838163 ]---
[ 1261.334252] BUG: unable to handle kernel paging request at ffffffffc0594f67
[ 1261.417904] IP: [] 0xffffffffc0594f67
[ 1261.480623] PGD 28000d067 [ 1261.510937] PUD 28000f067
PMD 427896067 [ 1261.558024] PTE 0
[ 1261.581046]
[ 1261.598861] Oops: 0010 [#2] SMP
[ 1261.636469] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mod igb usbcore i2c_algo_bit dca usb_common ptp pps_core [last unloaded: linux_kernel_bde]
[ 1262.548620] CPU: 1 PID: 2604 Comm: bcmINTR Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1262.666436] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1262.750916] task: ffff97cc2a526fc0 task.stack: ffffa75643b1c000
[ 1262.821857] RIP: 0010:[] [] 0xffffffffc0594f67
[ 1262.913730] RSP: 0000:ffffa75643b1fda0 EFLAGS: 00010246
[ 1262.977378] RAX: 0000000000000004 RBX: ffffffffc05975f0 RCX: 0000000000000000
[ 1263.062896] RDX: 0000000000000001 RSI: ffff97cc2a527a00 RDI: ffff97cc7fc98940
[ 1263.148419] RBP: 00007fc3ce5c2ce0 R08: ffff97cc2a527370 R09: 0000000000000000
[ 1263.233938] R10: 0000000003de8b05 R11: 0000000003f43bae R12: 0000000000000028
[ 1263.319460] R13: ffffffffc0597608 R14: 00007fc3ce5c2ce0 R15: 00007fc3ce5c3680
[ 1263.404979] FS: 00007fc3ce5c3680(0000) GS:ffff97cc7fc80000(0000) knlGS:0000000000000000
[ 1263.501963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1263.570817] CR2: ffffffffc0594f67 CR3: 000000042949a000 CR4: 0000000000100670
[ 1263.656335] Stack:
[ 1263.680396] 0000000001d0e000 ffff97cc2bd9dc00 0000000000000190 5ddc3084d0947987
[ 1263.769453] 0000000000000000 0000000000000000 ffff97cc2a526fc0 ffffffffad6b7b10
[ 1263.858513] ffffffffc05975f8 ffffffffc05975f8 0000000000000000 0000000000000000
[ 1263.947571] Call Trace:
[ 1263.976849] [] ? prepare_to_wait_event+0xf0/0xf0
[ 1264.051954] [] ? do_vfs_ioctl+0x9f/0x600
[ 1264.118728] [] ? SyS_futex+0x7f/0x160
[ 1264.182375] [] ? SyS_ioctl+0x74/0x80
[ 1264.244979] [] ? system_call_fast_compare_end+0xc/0xb7
[ 1264.326333] Code: Bad RIP value.
[ 1264.366233] RIP [] 0xffffffffc0594f67
[ 1264.429981] RSP
[ 1264.471751] CR2: ffffffffc0594f67
[ 1264.511438] ---[ end trace f5b3563022838164 ]---
[ 1264.511445] BUG: unable to handle kernel paging request at ffffffffc059c723
[ 1264.511449] IP: [] 0xffffffffc059c723
[ 1264.511451] PGD 28000d067
[ 1264.511452] PUD 28000f067
[ 1264.511453] PMD 427896067
[ 1264.511453] PTE 0
[ 1264.511454]
[ 1264.511456] Oops: 0010 [#3] SMP
[ 1264.511497] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mod igb usbcore i2c_algo_bit dca
[ 1264.511500] usb_common ptp pps_core [last unloaded: linux_kernel_bde]
[ 1264.511503] CPU: 3 PID: 2613 Comm: SOC KNET RX Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1264.511504] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1264.511505] task: ffff97cc2bf66e00 task.stack: ffffa75643b5c000
[ 1264.511508] RIP: 0010:[] [] 0xffffffffc059c723
[ 1264.511509] RSP: 0018:ffffa75643b5fb30 EFLAGS: 00010246
[ 1264.511510] RAX: 0000000000000004 RBX: 0000000000000000 RCX: 0000000000000000
[ 1264.511511] RDX: 0000000000000001 RSI: ffff97cc2bf67840 RDI: ffff97cc7fd98940
[ 1264.511512] RBP: 0000000000000000 R08: 00000125ad616731 R09: 0000000000000000
[ 1264.511513] R10: 00000000964fda00 R11: 0000000000000000 R12: 0000000000000000
[ 1264.511514] R13: ffffffffc05a6cc0 R14: ffffa75643b5fc08 R15: ffff97cc6cbf0000
[ 1264.511516] FS: 00007fc3ce5bd700(0000) GS:ffff97cc7fd80000(0000) knlGS:0000000000000000
[ 1264.511517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1264.511518] CR2: ffffffffc059c723 CR3: 000000042949a000 CR4: 0000000000100670
[ 1264.511518] Stack:
[ 1264.511522] ffffffffc05a6cc0 0000000100000000 ffffffffc05a6cc0 ffff97cc2bf66e00
[ 1264.511524] 00007fc3ce5bcb70 ffff97cc00000000 ffff97cc2bf66e00 ffffffffad6b7b10
[ 1264.511527] ffffffffc05a6cc8 ffffffffc05a6cc8 ffffffffffffff10 ffffa75643b5fc08
[ 1264.511528] Call Trace:
[ 1264.511535] [] ? prepare_to_wait_event+0xf0/0xf0
[ 1264.511540] [] ? do_vfs_ioctl+0x9f/0x600
[ 1264.511543] [] ? SyS_futex+0x7f/0x160
[ 1264.511545] [] ? SyS_ioctl+0x74/0x80
[ 1264.511549] [] ? system_call_fast_compare_end+0xc/0xb7
[ 1264.511558] Code: Bad RIP value.
[ 1264.511560] RIP [] 0xffffffffc059c723
[ 1264.511561] RSP
[ 1264.511561] CR2: ffffffffc059c723
[ 1264.511564] ---[ end trace f5b3563022838165 ]---
[ 1267.837916] sched: RT throttling activated
[ 1269.984434] BUG: unable to handle kernel paging request at ffffffffc05970c8
[ 1270.068082] IP: [] filp_close+0x1f/0x70
[ 1270.132877] PGD 28000d067 [ OK 1270.163191] PUD 28000f067
PMD 427896067 [ 1270.226938] PTE 0
[ 1270.249960]
[ 1270.267774] Oops: 0000 [#4] SMP
0m] Stopped Data[ 1270.305383] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel
sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mod igb usbcore i2c_algo_bit dca[ 1270.633736] usb_common ptp pps_core [last unloaded: linux_kernel_bde][ 1270.633743] CPU: 0 PID: 2577 Comm: syncd Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1270.633744] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1270.633746] task: ffff97cc2be10080 task.stack: ffffa75643aa4000
[ 1270.633747] RIP: 0010:[]
[ 1270.633752] [] filp_close+0x1f/0x70
[ 1270.633753] RSP: 0018:ffffa75643aa7cb8 EFLAGS: 00010246
[ 1270.633754] RAX: ffffffffc0597060 RBX: ffff97cc680d7c00 RCX: ffff97cc2be35c90
[ 1270.633755] RDX: ffff97cc2be35ca0 RSI: ffff97cc2be35c00 RDI: ffff97cc680d7c00
[ 1270.633756] RBP: ffff97cc2be35c00 R08: ffffe87a0fa940a0 R09: ffffffffae318c80
[ 1270.633757] R10: ffffffffae318c40 R11: 00000000000007ba R12: 0000000000000000
[ 1270.633758] R13: 0000000000000001 R14: 0000000000000035 R15: ffff97cc2be35c28
[ 1270.633760] FS: 00007fc3c6938700(0000) GS:ffff97cc7fc00000(0000) knlGS:0000000000000000
[ 1270.633761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1270.633762] CR2: ffffffffc05970c8 CR3: 000000042949a000 CR4: 0000000000100670
[ 1270.633763] Stack:
[ 1270.633764] 00000000000001ff
[ 1270.633765] 0000000000000000 ffff97cc2be35c00 ffffffffad82123c[ 1270.633767] ffff97cc2be10080
[ 1270.633768] ffff97cc2be10798 ffff97cc2bd9dc00 0000000000000000[ 1270.633770] 0000000000665e00
[ 1270.633770] ffff97cc2bd9dc68 ffffffffad679e37 000100cc00000000[ 1270.633773] Call Trace:
[ 1270.633779] [] ? put_files_struct+0x6c/0xc0
[ 1270.633782] [] ? do_exit+0x2b7/0xae0
[ 1270.633784] [] ? do_group_exit+0x3a/0xa0
[ 1270.633787] [] ? get_signal+0x297/0x640
[ 1270.633790] [] ? do_signal+0x36/0x6a0
[ 1270.633793] [] ? exit_to_usermode_loop+0x71/0xb0
[ 1270.633795] [] ? syscall_return_slowpath+0x54/0x60
[ 1270.633799] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 1270.633801] Code:
[ 1270.633802] 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48 [ 1270.633833] RIP
[ 1270.633835] [] filp_close+0x1f/0x70
RSP
[ 1270.633836] CR2: ffffffffc05970c8
[ 1270.633839] ---[ end trace f5b3563022838166 ]---
[ 1270.639608] Fixing recursive fault but reboot is needed!
[ 1270.646013] BUG: unable to handle kernel paging request at ffffffffc05970c8
[ 1270.646019] IP: [] filp_close+0x1f/0x70
[ 1270.646023] PGD 28000d067
[ 1270.646023] PUD 28000f067
[ 1270.646024] PMD 427896067
[ 1270.646025] PTE 0
[ 1270.646025]
[ 1270.646027] Oops: 0000 [#5] SMP
[ 1270.646065] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mod igb usbcore i2c_algo_bit dca
[ 1270.646067] usb_common ptp pps_core [last unloaded: linux_kernel_bde]
[ 1270.646071] CPU: 3 PID: 2550 Comm: syncd Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1270.646072] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1270.646073] task: ffff97cc32f76f00 task.stack: ffffa756439d4000
[ 1270.646076] RIP: 0010:[] [] filp_close+0x1f/0x70
[ 1270.646077] RSP: 0018:ffffa756439d7cb8 EFLAGS: 00010246
[ 1270.646078] RAX: ffffffffc0597060 RBX: ffff97cc680d7c00 RCX: ffff97cc2be35190
[ 1270.646079] RDX: ffff97cc2be351a0 RSI: ffff97cc2be35100 RDI: ffff97cc680d7c00
[ 1270.646080] RBP: ffff97cc2be35100 R08: ffffe87a0fadf020 R09: 0000000000000042
[ 1270.646081] R10: 0000000000000000 R11: 00000000000005cc R12: 0000000000000000
[ 1270.646082] R13: 0000000000000001 R14: 0000000000000035 R15: ffff97cc2be35128
[ 1270.646084] FS: 00007fc3ce7ab740(0000) GS:ffff97cc7fd80000(0000) knlGS:0000000000000000
[ 1270.646085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1270.646086] CR2: ffffffffc05970c8 CR3: 000000042c610000 CR4: 0000000000100670
[ 1270.646087] Stack:
[ 1270.646090] 00000000000001ff 0000000000000000 ffff97cc2be35100 ffffffffad82123c
[ 1270.646093] ffff97cc32f76f00 ffff97cc32f77618 ffff97cc2bd9dc00 0000000000000000
[ 1270.646095] 0000000000000001 ffff97cc2bd9dc68 ffffffffad679e37 0000000000000000
[ 1270.646096] Call Trace:
[ 1270.646101] [] ? put_files_struct+0x6c/0xc0
[ 1270.646104] [] ? do_exit+0x2b7/0xae0
[ 1270.646106] [] ? do_group_exit+0x3a/0xa0
[ 1270.646109] [] ? get_signal+0x297/0x640
[ 1270.646112] [] ? do_signal+0x36/0x6a0
[ 1270.646114] [] ? do_send_sig_info+0x65/0x90
[ 1270.646117] [] ? exit_to_usermode_loop+0x71/0xb0
[ 1270.646121] [] ? syscall_return_slowpath+0x54/0x60
[ 1270.646124] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 1270.646151] Code: 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48
[ 1270.646153] RIP [] filp_close+0x1f/0x70
[ 1270.646154] RSP
[ 1270.646155] CR2: ffffffffc05970c8
[ 1270.646158] ---[ end trace f5b3563022838167 ]---
[ 1270.651853] Fixing recursive fault but reboot is needed!
[ 1271.234808] BUG: unable to handle kernel paging request at ffffffffc05970c8
[ 1271.234814] IP: [] filp_close+0x1f/0x70
[ 1271.234816] PGD 28000d067
[ 1271.234817] PUD 28000f067
[ 1271.234818] PMD 427896067
[ 1271.234818] PTE 0
[ 1271.234818]
[ 1271.234821] Oops: 0000 [#6] SMP
[ 1271.234859] Modules linked in: optoe sff_8436_eeprom at24 nvmem_core br_netfilter bridge stp llc evdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass crct10dif_pclmul pcspkr crc32_pclmul ghash_clmulni_intel sg intel_cstate ipmi_msghandler button shpchp lpc_ich mfd_core acpi_cpufreq xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci libahci i2c_i801 i2c_smbus i2c_ismt ehci_pci libata ehci_hcd scsi_mod igb usbcore i2c_algo_bit dca
[ 1271.234862] usb_common ptp pps_core [last unloaded: linux_kernel_bde]
[ 1271.234866] CPU: 0 PID: 2648 Comm: syncd Tainted: G D O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
[ 1271.234867] Hardware name: Dell, Inc. Z9100/Z9100-CPU, BIOS 5.6.5 12/16/2016
[ 1271.234868] task: ffff97cc2a5ae300 task.stack: ffffa75643bec000
[ 1271.234871] RIP: 0010:[] [] filp_close+0x1f/0x70
[ 1271.234872] RSP: 0018:ffffa75643befcb8 EFLAGS: 00010246
[ 1271.234873] RAX: ffffffffc0597060 RBX: ffff97cc680d7c00 RCX: ffff97cc2bd64500
[ 1271.234874] RDX: ffff97cc2bdf1160 RSI: ffff97cc2bdf10c0 RDI: ffff97cc680d7c00
[ 1271.234875] RBP: ffff97cc2bdf10c0 R08: ffffe87a0fa9f820 R09: 00000000000000a7
[ 1271.234876] R10: 0000000000000000 R11: 0000000000000238 R12: 0000000000000000
[ 1271.234877] R13: 0000000000000001 R14: 0000000000000035 R15: ffff97cc2bdf10e8
[ 1271.234879] FS: 00007fc3ce7ab740(0000) GS:ffff97cc7fc00000(0000) knlGS:0000000000000000
[ 1271.234880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1271.234881] CR2: ffffffffc05970c8 CR3: 000000042ad80000 CR4: 0000000000100670
[ 1271.234882] Stack:
[ 1271.234886] 00000000000001ff 0000000000000000 ffff97cc2bdf10c0 ffffffffad82123c
[ 1271.234888] ffff97cc2a5ae300 ffff97cc2a5aea18 ffff97cc2bded800 0000000000000000
[ 1271.234891] 00007fc3c8a73601 ffff97cc2bded868 ffffffffad679e37 00000000000000c3
[ 1271.234891] Call Trace:
[ 1271.234897] [] ? put_files_struct+0x6c/0xc0
[ 1271.234901] [] ? do_exit+0x2b7/0xae0
[ 1271.234903] [] ? do_group_exit+0x3a/0xa0
[ 1271.234905] [] ? get_signal+0x297/0x640
[ 1271.234909] [] ? do_signal+0x36/0x6a0
[ 1271.234911] [] ? group_send_sig_info+0x16/0x40
[ 1271.234913] [] ? poll_select_copy_remaining+0xdf/0x150
[ 1271.234916] [] ? exit_to_usermode_loop+0x71/0xb0
[ 1271.234918] [] ? syscall_return_slowpath+0x54/0x60
[ 1271.234922] [] ? system_call_fast_compare_end+0xb5/0xb7
[ 1271.234950] Code: 00 48 83 c7 10 e9 12 b7 01 00 66 90 0f 1f 44 00 00 41 54 55 53 48 8b 47 38 48 85 c0 74 47 48 8b 47 28 45 31 e4 48 89 fb 48 89 f5 <48> 8b 40 68 48 85 c0 74 05 ff d0 41 89 c4 f6 43 45 40 75 16 48
[ 1271.234952] RIP [] filp_close+0x1f/0x70
[ 1271.234952] RSP
[ 1271.234953] CR2: ffffffffc05970c8
[ 1271.234956] ---[ end trace f5b3563022838168 ]---
[ 1271.240891] Fixing recursive fault but reboot is needed!
Stopping Docker Application Container Engine...
[ OK ] Stopped Docker Application Container Engine.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Sockets.
[ OK ] Closed Syslog Socket.
[ OK ] Closed D-Bus System Message Bus Socket.
[ OK ] Stopped target Slices.
[ OK ] Removed slice User and Session Slice.
[ OK ] Stopped target Paths.
[ OK ] Stopped target Network.
Stopping ifup for Ethernet40...
Stopping ifup for Ethernet52...
Stopping ifup for Ethernet72...
Stopping ifup for Ethernet28...
Stopping ifup for Ethernet44...
Stopping ifup for Ethernet68...
Stopping ifup for Ethernet16...
Stopping ifup for Ethernet36...
Stopping ifup for Ethernet96...
Stopping ifup for Ethernet104...
Stopping ifup for Ethernet88...
Stopping ifup for Ethernet24...
Stopping ifup for Ethernet76...
Stopping ifup for Ethernet60...
Stopping ifup for Ethernet92...
Stopping ifup for Ethernet108...
Stopping ifup for Ethernet100...
Stopping ifup for Ethernet64...
Stopping ifup for Ethernet32...
Stopping Raise network interfaces...
Stopping ifup for Ethernet120...
Stopping ifup for Ethernet48...
Stopping ifup for Ethernet20...
Stopping ifup for Ethernet124...
Stopping ifup for Ethernet56...
Stopping ifup for Ethernet0...
Stopping ifup for Ethernet112...
Stopping ifup for Ethernet84...
Stopping ifup for eth0...
Stopping ifup for Ethernet12...
Stopping ifup for Ethernet116...
Stopping ifup for Ethernet80...
Stopping ifup for Ethernet4...
Stopping ifup for Ethernet8...
[ OK ] Closed Docker Socket for the API.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped target Encrypted Volumes.
[ OK ] Stopped Forward Password Requests to Wall Directory Watch.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
Stopping Load/Save Random Seed...
[ OK ] Stopped target Swap.
Stopping Update UTMP about System Boot/Shutdown...
[ OK ] Stopped ifup for Ethernet40.
[ OK ] Stopped ifup for Ethernet52.
[ OK ] Stopped ifup for Ethernet72.
[ OK ] Stopped ifup for Ethernet28.
[ OK ] Stopped ifup for Ethernet44.
[ OK ] Stopped ifup for Ethernet68.
[ OK ] Stopped ifup for Ethernet16.
[ OK ] Stopped ifup for Ethernet36.
[ OK ] Stopped ifup for Ethernet96.
[ OK ] Stopped ifup for Ethernet104.
[ OK ] Stopped ifup for Ethernet88.
[ OK ] Stopped ifup for Ethernet24.
[ OK ] Stopped ifup for Ethernet76.
[ OK ] Stopped Load/Save Random Seed.
[ OK ] Stopped ifup for Ethernet60.
[ OK ] Stopped ifup for Ethernet92.
[ OK ] Stopped ifup for Ethernet108.
[ OK ] Stopped ifup for Ethernet100.
[ OK ] Stopped ifup for Ethernet64.
[ OK ] Stopped ifup for Ethernet32.
[ OK ] Stopped ifup for Ethernet20.
[ OK ] Stopped ifup for Ethernet120.
[ OK ] Stopped ifup for Ethernet48.
[ OK ] Stopped ifup for Ethernet124.
[ OK ] Stopped ifup for Ethernet56.
[ OK ] Stopped ifup for Ethernet0.
[ OK ] Stopped ifup for Ethernet112.
[ OK ] Stopped ifup for Ethernet84.
[ OK ] Stopped ifup for Ethernet12.
[ OK ] Stopped Update UTMP about System Boot/Shutdown.
[ OK ] Stopped ifup for Ethernet116.
[ OK ] Stopped ifup for Ethernet80.
Unmounting /var/log...
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Stopped ifup for Ethernet4.
[ OK ] Stopped ifup for Ethernet8.
[ OK ] Unmounted /var/log.
[ OK ] Stopped ifup for eth0.
[ OK ] Stopped Raise network interfaces.
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Stopped target Network (Pre).
Stopping netfilter persistent configuration...
[ OK ] Stopped netfilter persistent configuration.
[ OK ] Stopped target Local File Systems.
Unmounting /host...
Unmounting /var/lib/docker...
Unmounting /boot...
[ OK ] Stopped Load Kernel Modules.
[FAILED] Failed unmounting /host.
[ OK ] Unmounted /var/lib/docker.
[ OK ] Unmounted /boot.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped Remount Root and Kernel File Systems.
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Reached target Shutdown.
[ 2105.319165] watchdog: watchdog0: watchdog did not stop!
[ 2105.385026] systemd-shutdow: 38 output lines suppressed due to ratelimiting
[ 2105.490006] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 2105.582729] systemd-journald[283]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 2105.683863] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 2105.772652] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
[ 2105.853952] systemd-shutdown[1]: Unmounting file systems.
[ 2105.919052] systemd-shutdown[1]: Remounting '/host' read-only with options 'data=ordered'.
[ 2106.047797] systemd-shutdown[1]: Unmounting /host.
[ 2106.110100] systemd-shutdown[1]: Remounting '/' read-only with options 'lowerdir=/root,upperdir=/root/host/image-HEAD.154-e69ce25/rw,workdir=/root/host/image-HEAD.154-e69ce25/work'.
[ 2106.321374] systemd-shutdown[1]: Remounting '/host' read-only with options 'data=ordered'.
[ 2106.420683] systemd-shutdown[1]: Unmounting /host.
[ 2106.482167] systemd-shutdown[1]: Could not unmount /host: Device or resource busy
[ 2106.621592] kvm: exiting hardware virtualization
[ 2106.677345] sd 4:0:0:0: [sda] Synchronizing SCSI cache
[ 2107.163582] reboot: Restarting system
[ 2107.207477] reboot: machine restart
[ 2107.249562] ACPI MEMORY or I/O RESET_REG.

BIOS Boot Selector for Z9100
Primary BIOS Version 3.23.0.2-6

SMF Version: MSS 2.1, FPGA 2.1
Last POR=0x11, Reset Cause=0x55

CPGC Memtest ................................ PASS

CPGC Memtest ................................ PASS

Version 2.17.1245. Copyright (C) 2016 American Megatrends, Inc.
BIOS Date: 12/16/2016 19:05:20 Ver: 0ACBZ028
Press or to enter setup.

Loading SONiC-OS OS kernel ...
Loading SONiC-OS OS initial ramdisk ...
[ OK ] Started Raise network interfaces.
[ OK ] Reached target Network.
Starting OpenBSD Secure Shell server...
Starting Docker Application Container Engine...
Starting Permit User Sessions...
[ OK ] Reached target Network is Online.
Starting LSB: Start NTP daemon...
Starting /etc/rc.local Compatibility...
[ OK ] Started Permit User Sessions.
[ 66.329957] rc.local[589]: + [ ! -e /host/machine.conf ]
[ OK ] Started /etc/rc.local Compatibility.
[ 66.534746] rc.local[589]: + . /host/machine.conf
[ OK ] Started Getty on tty1.
[ 66.656385] rc.local[589]: + onie_version=3.23.1.2-5
[ OK ] Started Serial Getty on ttyS1.
[ 66.772381] rc.local[589]: + onie_vendor_id=674
[ OK ] Reached target Login Prompts.
[ 66.892297] rc.local[589]: + onie_platform=x86_64-dell_z9100_c2538-r0
[ OK ] Started LSB: Start NTP daemon.
[ 67.040436] rc.local[589]: + onie_machine=dell_z9100_c2538
[ OK ] Started OpenBSD Secure Shell server.
[ 67.172366] rc.local[589]: + onie_machine_rev=0
[ 67.304312] rc.local[589]: + onie_arch=x86_64
[ 67.364535] rc.local[589]: + onie_config_version=1
[ 67.424604] rc.local[589]: + onie_build_date=2017-01-03T11:44-0800
[ 67.500436] rc.local[589]: + onie_partition_type=gpt
[ 67.560401] rc.local[589]: + onie_kernel_version=4.1.28
[ 67.624461] rc.local[589]: + onie_firmware=bios
[ 67.684492] rc.local[589]: + onie_switch_asic=bcm
[ 67.744543] rc.local[589]: + onie_skip_ethmgmt_macs=no
[ 67.808520] rc.local[589]: + echo install platform dependent packages at the first boot time
[ 67.916617] rc.local[589]: install platform dependent packages at the first boot time
[ 68.042683] rc.local[589]: + + cat /etc/sonic/sonic_version.ymlgrep
[ 68.125671] rc.local[589]: build_version
[ 68.181893] rc.local[589]: + cut -f2 -d
[ 68.237498] rc.local[589]: + eval sonic_version='HEAD.154-e69ce25'
[ 68.312448] rc.local[589]: + sonic_version=HEAD.154-e69ce25
[ OK ] Started Docker Application Container Engine.
[ 68.388408] rc.local[589]: + [ -f /host/image-HEAD.154-e69ce25/platform/firsttime ]
Starting Database container...
[ 68.556425] rc.local[589]: + exit 0

Debian GNU/Linux 9 sonic ttyS1

sonic login: admin
Password:
Last login: Tue Apr 10 06:26:59 UTC 2018 on ttyS1
Linux sonic 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64
You are on


/ | / _ | \ | ()/ |
_
| | | | | | | |
) | || | |\ | | |

|
/ ___/|| _||____|

-- Software for Open Networking in the Cloud --

Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.

Help: http://azure.github.io/SONiC/

admin@sonic:~$

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
@amitabhja
Copy link
Author

Verification Logs:
root@sonic:~# reboot
Stopping syncd...
requested COLD shutdown
[ OK ] Stopped SNMP container.
Stopping switch state service...
[ OK ] Stopped Delays snmp container until SONiC has started.
[ OK ] Stopped LLDP container.
[ OK ] Stopped switch state service.
Stopping Opennsl kernel modules init...
[ OK ] Stopped /etc/rc.local Compatibility.
[ OK ] Stopped target Network is Online.
[ OK ] Stopped Opennsl kernel modules init.
[ OK ] Stopped Database container.
Stopping Docker Application Container Engine...
[ OK ] Stopped Docker Application Container Engine.
[ OK ] Stopped target Network.
Stopping Raise network interfaces...
Stopping ifup for eth0...
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Paths.
[ OK ] Stopped target Sockets.
[ OK ] Closed D-Bus System Message Bus Socket.
[ OK ] Closed Syslog Socket.
[ OK ] Stopped target Slices.
[ OK ] Removed slice User and Session Slice.
[ OK ] Closed Docker Socket for the API.
[ OK ] Stopped target System Initialization.
Stopping Update UTMP about System Boot/Shutdown...
[ OK ] Stopped target Swap.
[ OK ] Stopped target Encrypted Volumes.
[ OK ] Stopped Forward Password Requests to Wall Directory Watch.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
Stopping Load/Save Random Seed...
[ OK ] Stopped Load/Save Random Seed.
[ OK ] Stopped Update UTMP about System Boot/Shutdown.
Unmounting /var/log...
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Unmounted /var/log.
[ OK ] Stopped ifup for eth0.
[ OK ] Stopped Raise network interfaces.
[ OK ] Stopped target Network (Pre).
Stopping netfilter persistent configuration...
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Stopped netfilter persistent configuration.
[ OK ] Stopped Load Kernel Modules.
[ OK ] Stopped target Local File Systems.
Unmounting /boot...
Unmounting /host...
Unmounting /var/lib/docker...
[ OK ] Unmounted /boot.
[FAILED] Failed unmounting /host.
[ OK ] Unmounted /var/lib/docker.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped Remount Root and Kernel File Systems.
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Reached target Shutdown.
[15879.067117] watchdog: watchdog0: watchdog did not stop!
[15879.741612] reboot: Restarting system

BIOS Boot Selector for Z9100
Primary BIOS Version 3.23.0.2-6

SMF Version: MSS 2.1, FPGA 2.1
Last POR=0x11, Reset Cause=0x55

CPGC Memtest ................................ PASS

==========================================

root@sonic:~# show ver
SONiC Software Version: SONiC.HEAD.693-3f37b96
Distribution: Debian 9.5
Kernel: 4.9.0-7-amd64
Build commit: 3f37b96
Build date: Sun Aug 26 09:13:46 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm HEAD.693-3f37b96 e934afcb65a3 336.6 MB
docker-syncd-brcm latest e934afcb65a3 336.6 MB
docker-orchagent-brcm HEAD.693-3f37b96 38a356f0df7a 257.6 MB
docker-orchagent-brcm latest 38a356f0df7a 257.6 MB
docker-lldp-sv2 HEAD.693-3f37b96 9b54765a5987 270.8 MB
docker-lldp-sv2 latest 9b54765a5987 270.8 MB
docker-dhcp-relay HEAD.693-3f37b96 0758dc72b19f 253.7 MB
docker-dhcp-relay latest 0758dc72b19f 253.7 MB
docker-database HEAD.693-3f37b96 33f38670a603 252.3 MB
docker-database latest 33f38670a603 252.3 MB
docker-teamd HEAD.693-3f37b96 d1df4e88a6a0 257.2 MB
docker-teamd latest d1df4e88a6a0 257.2 MB
docker-snmp-sv2 HEAD.693-3f37b96 791154a0e7cf 291.2 MB
docker-snmp-sv2 latest 791154a0e7cf 291.2 MB
docker-router-advertiser HEAD.693-3f37b96 0b3ab8f3fca2 250 MB
docker-router-advertiser latest 0b3ab8f3fca2 250 MB
docker-platform-monitor HEAD.693-3f37b96 5bf09433f1f1 283.2 MB
docker-platform-monitor latest 5bf09433f1f1 283.2 MB
docker-fpm-quagga HEAD.693-3f37b96 cd3a0dc65949 264.1 MB
docker-fpm-quagga latest cd3a0dc65949 264.1 MB

root@sonic:~#

@prsunny prsunny mentioned this issue Dec 24, 2020
3 tasks
lguohan pushed a commit that referenced this issue Dec 26, 2020
Introduce tunnel manager daemon. Start the process as part of swss container

Submodule update for swss:
9ed3026 - 2020-12-24 : [NAT] ACL Rule with DO_NOT_NAT action is getting failed. (#1502) [Akhilesh Samineni]
c39a4b1 - 2020-12-23 : Mux/IPTunnel orchagent changes (#1497) [Prince Sunny]
bc8df0e - 2020-12-23 : Add support for headroom pool watermark (#1567) [Neetha John]
theasianpianist pushed a commit to theasianpianist/sonic-buildimage that referenced this issue Feb 5, 2022
Signed-off-by: Neetha John <nejo@microsoft.com>

Includes all the changes in sonic-net#1453 along with fix for error msgs seen in syslog - Got invalid response type from redis 5
The error msg was due to incorrect return type in lua script.

What I did
Added 'SAI_BUFFER_POOL_STAT_XOFF_ROOM_WATERMARK_BYTES' in the interested counters to be queried
Updated the the buffer lua script to update the headroom pool watermark counters
Updated watermark orch to act on headroom pool clear request

Why I did it
To expose the headroom pool watermark counters in SONiC

How I verified it
On platforms with headroom pool support, verified that headroom pool watermark counters are getting updated as expected

admin@sonic:~$ show headroom-pool persistent-watermark 
Headroom pool maximum occupancy:
                 Pool    Bytes
---------------------  -------
ingress_lossless_pool   863616
On platforms without headroom pool support, headroom pool watermark counters show as N/A

admin@sonic:~$ show headroom-pool persistent-watermark 
Headroom pool maximum occupancy:
                 Pool    Bytes
---------------------  -------
ingress_lossless_pool      N/A
New sonic mgmt test added and it passed. sonic-net/sonic-mgmt#2614

Verified the error msg seen in syslog by enabling buffer pool watermark on vs docker. With the fix, build a new vs docker and verified that the msgs are no longer seen when buffer pool watermark is enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants