Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Celestica][E1031] USB tty devices not listed under \dev after SONiC installed #6603

Closed
Blueve opened this issue Jan 29, 2021 · 9 comments · Fixed by #6715
Closed

[Celestica][E1031] USB tty devices not listed under \dev after SONiC installed #6603

Blueve opened this issue Jan 29, 2021 · 9 comments · Fixed by #6715
Assignees
@Blueve
Labels
Issue for 202012 Master Branch Quality Platform: Celestica

Comments

@Blueve
Copy link
Contributor

Blueve commented Jan 29, 2021 •
edited
Loading

Description

USB tty devices not listed under \dev folder after SONiC installed, however the USB devices already been recognized by system.

Steps to reproduce the issue:

  1. Connect Console Switch and E1031 with USB link
  2. Install SONiC image from ONIE
  3. Run show version
  4. Run lsusb -t
  5. Run 'ls \dev\ttyUSB*'

Run above test for SONiC.202012.13-c8b3a709 and SONiC.master.565-e616a329

Describe the results you received:
Test Result for SONiC.202012.13-c8b3a709

SONiC Software Version: SONiC.202012.13-c8b3a709
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: c8b3a709
Build date: Sun Jan 24 18:15:04 UTC 2021
Built by: johnar@jenkins-worker-4

Platform: x86_64-cel_e1031-r0
HwSKU: Celestica-E1031-T48S4
ASIC: broadcom
ASIC Count: 1
/usr/local/bin/decode-syseeprom : ERROR : Failed to read eeprom : [Errno 2] No such file or directory: '/sys/class/i2c-adapter/i2c-2/2-0050/eeprom'
Serial Number: 
Uptime: 08:33:04 up 6 min,  1 user,  load average: 6.68, 2.22, 0.82

Docker images:
REPOSITORY                    TAG                  IMAGE ID            SIZE
docker-syncd-brcm             202012.13-c8b3a709   45efbad0189a        642MB
docker-syncd-brcm             latest               45efbad0189a        642MB
docker-snmp                   202012.13-c8b3a709   ecf3c6d7184e        436MB
docker-snmp                   latest               ecf3c6d7184e        436MB
docker-teamd                  202012.13-c8b3a709   04735115bac6        406MB
docker-teamd                  latest               04735115bac6        406MB
docker-nat                    202012.13-c8b3a709   eb1946647f23        408MB
docker-nat                    latest               eb1946647f23        408MB
docker-router-advertiser      202012.13-c8b3a709   e447b03b927a        395MB
docker-router-advertiser      latest               e447b03b927a        395MB
docker-platform-monitor       202012.13-c8b3a709   6e9e777e59fb        604MB
docker-platform-monitor       latest               6e9e777e59fb        604MB
docker-lldp                   202012.13-c8b3a709   7b2ddb771ca0        435MB
docker-lldp                   latest               7b2ddb771ca0        435MB
docker-dhcp-relay             202012.13-c8b3a709   a013afe4444b        402MB
docker-dhcp-relay             latest               a013afe4444b        402MB
docker-sonic-mgmt-framework   202012.13-c8b3a709   aae6f04cbb50        613MB
docker-sonic-mgmt-framework   latest               aae6f04cbb50        613MB
docker-orchagent              202012.13-c8b3a709   12a3223cf4e1        424MB
docker-orchagent              latest               12a3223cf4e1        424MB
docker-sonic-telemetry        202012.13-c8b3a709   147f83d96865        470MB
docker-sonic-telemetry        latest               147f83d96865        470MB
docker-fpm-frr                202012.13-c8b3a709   a10a24092af0        422MB
docker-fpm-frr                latest               a10a24092af0        422MB
docker-sflow                  202012.13-c8b3a709   5e51bc3bcb30        406MB
docker-sflow                  latest               5e51bc3bcb30        406MB
docker-database               202012.13-c8b3a709   0d5b77bc9804        394MB
docker-database               latest               0d5b77bc9804        394MB

admin@sonic:~$ lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/7p, 480M
            |__ Port 1: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 7: Dev 10, If 0, Class=Hub, Driver=hub/7p, 480M
                |__ Port 5: Dev 15, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 3, Class=Vendor Specific Class, Driver=, 12M

Test Result for SONiC.master.565-e616a329

admin@sonic:~$ show version

SONiC Software Version: SONiC.master.565-e616a329
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: e616a329
Build date: Wed Jan 27 09:04:10 UTC 2021
Built by: johnar@jenkins-worker-8

Platform: x86_64-cel_e1031-r0
HwSKU: Celestica-E1031-T48S4
ASIC: broadcom
ASIC Count: 1
/usr/local/bin/decode-syseeprom : ERROR : Failed to read eeprom : [Errno 2] No such file or directory: '/sys/class/i2c-adapter/i2c-2/2-0050/eeprom'
Serial Number: 
Uptime: 08:53:13 up 5 min,  1 user,  load average: 1.30, 0.56, 0.23

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-syncd-brcm             latest                aed8504c10f7        642MB
docker-syncd-brcm             master.565-e616a329   aed8504c10f7        642MB
docker-snmp                   latest                6022d6181cd4        436MB
docker-snmp                   master.565-e616a329   6022d6181cd4        436MB
docker-teamd                  latest                a91e58f252e5        406MB
docker-teamd                  master.565-e616a329   a91e58f252e5        406MB
docker-router-advertiser      latest                1d06a5a10073        395MB
docker-router-advertiser      master.565-e616a329   1d06a5a10073        395MB
docker-platform-monitor       latest                99dee506feef        603MB
docker-platform-monitor       master.565-e616a329   99dee506feef        603MB
docker-macsec                 latest                a7fe60aca839        409MB
docker-macsec                 master.565-e616a329   a7fe60aca839        409MB
docker-lldp                   latest                717acfba87df        435MB
docker-lldp                   master.565-e616a329   717acfba87df        435MB
docker-dhcp-relay             latest                b32c3f6f5e62        402MB
docker-dhcp-relay             master.565-e616a329   b32c3f6f5e62        402MB
docker-database               latest                5700e811eebe        395MB
docker-database               master.565-e616a329   5700e811eebe        395MB
docker-sonic-mgmt-framework   latest                eda2647815b7        613MB
docker-sonic-mgmt-framework   master.565-e616a329   eda2647815b7        613MB
docker-orchagent              latest                4a57a6e350df        424MB
docker-orchagent              master.565-e616a329   4a57a6e350df        424MB
docker-nat                    latest                48503087eeed        408MB
docker-nat                    master.565-e616a329   48503087eeed        408MB
docker-sonic-telemetry        latest                281caf9c8ec9        470MB
docker-sonic-telemetry        master.565-e616a329   281caf9c8ec9        470MB
docker-fpm-frr                latest                3308944dc040        424MB
docker-fpm-frr                master.565-e616a329   3308944dc040        424MB
docker-sflow                  latest                55299e85bef7        406MB
docker-sflow                  master.565-e616a329   55299e85bef7        406MB

admin@sonic:~$ lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/7p, 480M
            |__ Port 1: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 1: Dev 4, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 2: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 3: Dev 6, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 4: Dev 7, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 5: Dev 8, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 0, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 1, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 2, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 6: Dev 9, If 3, Class=Vendor Specific Class, Driver=, 12M
            |__ Port 7: Dev 10, If 0, Class=Hub, Driver=hub/7p, 480M
                |__ Port 5: Dev 15, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 5: Dev 15, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 3: Dev 13, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 1: Dev 11, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 6: Dev 16, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 3, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 4: Dev 14, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 1, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 2, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 0, Class=Vendor Specific Class, Driver=, 12M
                |__ Port 2: Dev 12, If 3, Class=Vendor Specific Class, Driver=, 12M

admin@sonic:~$ ls /dev/ttyUSB*
ls: cannot access '/dev/ttyUSB*': No such file or directory

Describe the results you expected:

ttyUSB0 ~ ttyUSB47 should list under \dev folder.

Additional information you deem important (e.g. issue happens only occasionally):

The driver missing caused this issue. Re-plug can bring the driver back. Below command can also bring the driver back without touch the box physically:

sudo sh -c "echo '1-1.1' > /sys/bus/usb/drivers/usb/unbind"
sudo sh -c "echo '1-1.1' > /sys/bus/usb/drivers/usb/bind"

Before

admin@sonic:~$ lsmod | grep 210x

admin@sonic:~$ lsmod | grep usb
usbcore               294912  2 ehci_pci,ehci_hcd
usb_common             16384  1 usbcore

admin@sonic:~$ ls /sys/bus/usb/drivers
hub  usb  usbfs

After

admin@sonic:~$ lsmod | grep 210x
cp210x                 36864  0
usbserial              53248  1 cp210x
usbcore               294912  4 usbserial,ehci_pci,cp210x,ehci_hcd

admin@sonic:~$ lsmod | grep usb
usbserial              53248  1 cp210x
usbcore               294912  4 usbserial,ehci_pci,cp210x,ehci_hcd
usb_common             16384  1 usbcore

admin@sonic:~$ ls /sys/bus/usb/drivers
cp210x  hub  usb  usbfs  usbserial_generic
@lguohan
Copy link
Collaborator

lguohan commented Jan 29, 2021

can you debug this with celestica?

@lguohan
Copy link
Collaborator

lguohan commented Jan 29, 2021

looks like their platform driver is not installed properly.

@Blueve
Copy link
Contributor Author

Blueve commented Jan 29, 2021

looks like their platform driver is not installed properly.

Yes, the driver is missing. I've looped Celestica in an email thread.

@Blueve
Copy link
Contributor Author

Blueve commented Jan 29, 2021

The driver missing caused this issue. Re-plug can bring the driver back. Below command can also bring the driver back without touch the box physically:

sudo sh -c "echo '1-1.1' > /sys/bus/usb/drivers/usb/unbind"
sudo sh -c "echo '1-1.1' > /sys/bus/usb/drivers/usb/bind"

Before

admin@sonic:~$ lsmod | grep 210x

admin@sonic:~$ lsmod | grep usb
usbcore               294912  2 ehci_pci,ehci_hcd
usb_common             16384  1 usbcore

admin@sonic:~$ ls /sys/bus/usb/drivers
hub  usb  usbfs

After

admin@sonic:~$ lsmod | grep 210x
cp210x                 36864  0
usbserial              53248  1 cp210x
usbcore               294912  4 usbserial,ehci_pci,cp210x,ehci_hcd

admin@sonic:~$ lsmod | grep usb
usbserial              53248  1 cp210x
usbcore               294912  4 usbserial,ehci_pci,cp210x,ehci_hcd
usb_common             16384  1 usbcore

admin@sonic:~$ ls /sys/bus/usb/drivers
cp210x  hub  usb  usbfs  usbserial_generic

But not sure why the usb uart bridge couldn't binding the cp210x driver automatically. Maybe a driver bug?

@Blueve
Copy link
Contributor Author

Blueve commented Jan 29, 2021

First SONiC boot after installed

[    0.000000] Command line: BOOT_IMAGE=/image-master.565-e616a329/boot/vmlinuz-4.19.0-9-2-amd64 root=UUID=a5efd993-0a35-48a2-891e-eae4db4e358f rw console=tty0 console=ttyS1,9600n8 quiet intel_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-master.565-e616a329/fs.squashfs loopfstype=squashfs apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 module_blacklist=gpio_ich
[    0.294743] Kernel command line: BOOT_IMAGE=/image-master.565-e616a329/boot/vmlinuz-4.19.0-9-2-amd64 root=UUID=a5efd993-0a35-48a2-891e-eae4db4e358f rw console=tty0 console=ttyS1,9600n8 quiet intel_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-master.565-e616a329/fs.squashfs loopfstype=squashfs apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 module_blacklist=gpio_ich
[    0.466781] pci 0000:00:16.0: quirk_usb_early_handoff+0x0/0x6c1 took 20109 usecs
[    2.280424] usbcore: registered new interface driver usbfs
[    2.280461] usbcore: registered new interface driver hub
[    2.280512] usbcore: registered new device driver usb
[    2.310874] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[    2.310878] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.310881] usb usb1: Product: EHCI Host Controller
[    2.310883] usb usb1: Manufacturer: Linux 4.19.0-9-2-amd64 ehci_hcd
[    2.310886] usb usb1: SerialNumber: 0000:00:16.0
[    2.646893] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    2.803085] usb 1-1: New USB device found, idVendor=8087, idProduct=07db, bcdDevice= 0.03
[    2.803090] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.090660] usb 1-1.1: new high-speed USB device number 3 using ehci-pci
[    3.199583] usb 1-1.1: New USB device found, idVendor=0424, idProduct=2517, bcdDevice= 0.00
[    3.199588] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.199591] usb 1-1.1: Product: C1010
[    3.199593] usb 1-1.1: Manufacturer: Celestica
[    3.199596] usb 1-1.1: SerialNumber: C1010NNN010A22BY200022
[    3.486674] usb 1-1.1.1: new full-speed USB device number 4 using ehci-pci
[    3.685205] usb 1-1.1.1: New USB device found, idVendor=10c4, idProduct=ea71, bcdDevice= 1.00
[    3.685210] usb 1-1.1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.685213] usb 1-1.1.1: Product: CP2108 Quad USB to UART Bridge
[    3.685215] usb 1-1.1.1: Manufacturer: Silicon Labs
[    3.685218] usb 1-1.1.1: SerialNumber: 0001
...

unbind USB Hub 1-1.1

[18835.706965] usb 1-1.1.1: USB disconnect, device number 4
[18835.707527] usb 1-1.1.2: USB disconnect, device number 5
[18835.708085] usb 1-1.1.3: USB disconnect, device number 6
[18835.708656] usb 1-1.1.4: USB disconnect, device number 7
[18835.709212] usb 1-1.1.5: USB disconnect, device number 8
[18835.709824] usb 1-1.1.6: USB disconnect, device number 9
[18835.710399] usb 1-1.1.7: USB disconnect, device number 10
[18835.710417] usb 1-1.1.7.1: USB disconnect, device number 11
[18835.714029] usb 1-1.1.7.2: USB disconnect, device number 12
[18835.714616] usb 1-1.1.7.3: USB disconnect, device number 13
[18835.715183] usb 1-1.1.7.4: USB disconnect, device number 14
[18835.715752] usb 1-1.1.7.5: USB disconnect, device number 15
[18835.716304] usb 1-1.1.7.6: USB disconnect, device number 16

bind USB Hub 1-1.1 again

[18854.361823] usb 1-1.1.1: new full-speed USB device number 17 using ehci-pci
[18854.472331] usb 1-1.1.1: New USB device found, idVendor=10c4, idProduct=ea71, bcdDevice= 1.00
[18854.472338] usb 1-1.1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[18854.472341] usb 1-1.1.1: Product: CP2108 Quad USB to UART Bridge
[18854.472343] usb 1-1.1.1: Manufacturer: Silicon Labs
[18854.472346] usb 1-1.1.1: SerialNumber: 0001
[18854.491621] usbcore: registered new interface driver usbserial_generic
[18854.498074] usbserial: USB Serial support registered for generic
[18854.499926] usbcore: registered new interface driver cp210x
[18854.500754] usbserial: USB Serial support registered for cp210x
[18854.513999] usb 1-1.1.1: cp210x converter now attached to ttyUSB0
[18854.515048] usb 1-1.1.1: cp210x converter now attached to ttyUSB1
[18854.516320] usb 1-1.1.1: cp210x converter now attached to ttyUSB2
[18854.517579] usb 1-1.1.1: cp210x converter now attached to ttyUSB3

Seems like the system not register USB interface during first boot except hub's interface.

@Blueve
Copy link
Contributor Author

Blueve commented Jan 29, 2021

Possible relating topics:

https://www.linuxquestions.org/questions/linux-newbie-8/usbcore-only-registers-new-interface-driver-cdc_wdm-when-i-hot-plug-my-device-after-bootup-4175638603/
https://bugzilla.redhat.com/show_bug.cgi?id=253823
https://askubuntu.com/questions/1015207/udev-rule-at-boot-time
https://www.linuxquestions.org/questions/mandriva-30/how-to-have-udev-start-at-boot-432287/

Keywords: udev, usbcore, coldplug

@Blueve
Copy link
Contributor Author

Blueve commented Jan 29, 2021

@lguohan The driver is embedded in kernel, usually we don't need to install it. Consider the issue only happens with coldplug device, it more likely a timing issue, I guess the usbcore/udev are not fully initialize during boot.

@Blueve
Copy link
Contributor Author

Blueve commented Feb 2, 2021

Additional testing with 201911, no issue observed.

  • In that branch, the C0 udev rule has not been merged yet.
  • The Kernel version is lower than 202012
admin@sonic:~$ show version

SONiC Software Version: SONiC.201911.115-fcf93dda
Distribution: Debian 9.13
Kernel: 4.9.0-14-2-amd64
Build commit: fcf93dda
Build date: Sat Jan 30 05:52:21 UTC 2021
Built by: johnar@worker-s2e0cb0

admin@sonic:~$ sudo dmesg | grep usb
[    0.000000] Command line: BOOT_IMAGE=/image-201911.115-fcf93dda/boot/vmlinuz-4.9.0-14-2-amd64 root=UUID=d5708d60-8f8e-4fd3-abc5-bab251322c5a rw console=tty0 console=ttyS1,9600n8 quiet intel_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-201911.115-fcf93dda/fs.squashfs loopfstype=squashfs apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 module_blacklist=gpio_ich
[    0.000000] Kernel command line: BOOT_IMAGE=/image-201911.115-fcf93dda/boot/vmlinuz-4.9.0-14-2-amd64 root=UUID=d5708d60-8f8e-4fd3-abc5-bab251322c5a rw console=tty0 console=ttyS1,9600n8 quiet intel_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-201911.115-fcf93dda/fs.squashfs loopfstype=squashfs apparmor=1 security=apparmor varlog_size=4096 usbcore.autosuspend=-1 module_blacklist=gpio_ich
[    1.481208] usbcore: registered new interface driver usbfs
[    1.481234] usbcore: registered new interface driver hub
[    1.481295] usbcore: registered new device driver usb
[    1.911739] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    1.911743] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.911746] usb usb1: Product: EHCI Host Controller
[    1.911748] usb usb1: Manufacturer: Linux 4.9.0-14-2-amd64 ehci_hcd
[    1.911751] usb usb1: SerialNumber: 0000:00:16.0
[    2.239590] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    2.395967] usb 1-1: New USB device found, idVendor=8087, idProduct=07db
[    2.395971] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.687586] usb 1-1.1: new high-speed USB device number 3 using ehci-pci
[    2.796448] usb 1-1.1: New USB device found, idVendor=0424, idProduct=2517
[    2.796452] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.796455] usb 1-1.1: Product: C1010
[    2.796457] usb 1-1.1: Manufacturer: Celestica
[    2.796460] usb 1-1.1: SerialNumber: C1010NNN010A22BY200022
[    3.084362] usb 1-1.1.1: new full-speed USB device number 4 using ehci-pci
[    3.197915] usb 1-1.1.1: New USB device found, idVendor=10c4, idProduct=ea71
[    3.197919] usb 1-1.1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.197922] usb 1-1.1.1: Product: CP2108 Quad USB to UART Bridge
[    3.197925] usb 1-1.1.1: Manufacturer: Silicon Labs
[    3.197927] usb 1-1.1.1: SerialNumber: 0001
[    3.279549] usb 1-1.1.2: new full-speed USB device number 5 using ehci-pci
[    3.389910] usb 1-1.1.2: New USB device found, idVendor=10c4, idProduct=ea71
[    3.389915] usb 1-1.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.389918] usb 1-1.1.2: Product: CP2108 Quad USB to UART Bridge
[    3.389921] usb 1-1.1.2: Manufacturer: Silicon Labs
[    3.389923] usb 1-1.1.2: SerialNumber: 0001
[    3.471527] usb 1-1.1.3: new full-speed USB device number 6 using ehci-pci
[    3.581899] usb 1-1.1.3: New USB device found, idVendor=10c4, idProduct=ea71
[    3.581904] usb 1-1.1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.581907] usb 1-1.1.3: Product: CP2108 Quad USB to UART Bridge
[    3.581910] usb 1-1.1.3: Manufacturer: Silicon Labs
[    3.581912] usb 1-1.1.3: SerialNumber: 0001
[    3.664997] usb 1-1.1.4: new full-speed USB device number 7 using ehci-pci
[    3.773889] usb 1-1.1.4: New USB device found, idVendor=10c4, idProduct=ea71
[    3.773894] usb 1-1.1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.773897] usb 1-1.1.4: Product: CP2108 Quad USB to UART Bridge
[    3.773899] usb 1-1.1.4: Manufacturer: Silicon Labs
[    3.773902] usb 1-1.1.4: SerialNumber: 0001
[    3.855517] usb 1-1.1.5: new full-speed USB device number 8 using ehci-pci
[    3.966260] usb 1-1.1.5: New USB device found, idVendor=10c4, idProduct=ea71
[    3.966265] usb 1-1.1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    3.966268] usb 1-1.1.5: Product: CP2108 Quad USB to UART Bridge
[    3.966270] usb 1-1.1.5: Manufacturer: Silicon Labs
[    3.966273] usb 1-1.1.5: SerialNumber: 0001
[    3.997753] usbcore: registered new interface driver usbserial
[    3.997787] usbcore: registered new interface driver usbserial_generic
[    3.997809] usbserial: USB Serial support registered for generic
[    4.006410] usbcore: registered new interface driver cp210x
[    4.006438] usbserial: USB Serial support registered for cp210x
[    4.043314] usb 1-1.1.1: cp210x converter now attached to ttyUSB0
[    4.048403] usb 1-1.1.6: new full-speed USB device number 9 using ehci-pci
[    4.059733] usb 1-1.1.1: cp210x converter now attached to ttyUSB1
[    4.061846] usb 1-1.1.1: cp210x converter now attached to ttyUSB2
[    4.069612] usb 1-1.1.1: cp210x converter now attached to ttyUSB3
[    4.072782] usb 1-1.1.2: cp210x converter now attached to ttyUSB4
[    4.085546] usb 1-1.1.2: cp210x converter now attached to ttyUSB5
[    4.099097] usb 1-1.1.2: cp210x converter now attached to ttyUSB6
[    4.104635] usb 1-1.1.2: cp210x converter now attached to ttyUSB7
[    4.109143] usb 1-1.1.3: cp210x converter now attached to ttyUSB8
[    4.110870] usb 1-1.1.3: cp210x converter now attached to ttyUSB9
[    4.118318] usb 1-1.1.3: cp210x converter now attached to ttyUSB10
[    4.127541] usb 1-1.1.3: cp210x converter now attached to ttyUSB11
[    4.128376] usb 1-1.1.4: cp210x converter now attached to ttyUSB12
[    4.131253] usb 1-1.1.4: cp210x converter now attached to ttyUSB13
[    4.143563] usb 1-1.1.4: cp210x converter now attached to ttyUSB14
[    4.144375] usb 1-1.1.4: cp210x converter now attached to ttyUSB15
[    4.150198] usb 1-1.1.5: cp210x converter now attached to ttyUSB16
[    4.154836] usb 1-1.1.5: cp210x converter now attached to ttyUSB17
[    4.159532] usb 1-1.1.6: New USB device found, idVendor=10c4, idProduct=ea71
[    4.159537] usb 1-1.1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    4.159540] usb 1-1.1.6: Product: CP2108 Quad USB to UART Bridge
[    4.159542] usb 1-1.1.6: Manufacturer: Silicon Labs
[    4.159545] usb 1-1.1.6: SerialNumber: 0001
[    4.160858] usb 1-1.1.6: cp210x converter now attached to ttyUSB19
[    4.161846] usb 1-1.1.6: cp210x converter now attached to ttyUSB20
[    4.162848] usb 1-1.1.6: cp210x converter now attached to ttyUSB21
[    4.163984] usb 1-1.1.6: cp210x converter now attached to ttyUSB22
[    4.171395] usb 1-1.1.5: cp210x converter now attached to ttyUSB18
[    4.173171] usb 1-1.1.5: cp210x converter now attached to ttyUSB23
[    4.244345] usb 1-1.1.7: new high-speed USB device number 10 using ehci-pci
[    4.352748] usb 1-1.1.7: New USB device found, idVendor=0424, idProduct=2517
[    4.352753] usb 1-1.1.7: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.643509] usb 1-1.1.7.1: new full-speed USB device number 11 using ehci-pci
[    4.753855] usb 1-1.1.7.1: New USB device found, idVendor=10c4, idProduct=ea71
[    4.753860] usb 1-1.1.7.1: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    4.753863] usb 1-1.1.7.1: Product: CP2108 Quad USB to UART Bridge
[    4.753865] usb 1-1.1.7.1: Manufacturer: Silicon Labs
[    4.753868] usb 1-1.1.7.1: SerialNumber: 0001
[    4.755227] usb 1-1.1.7.1: cp210x converter now attached to ttyUSB24
[    4.756345] usb 1-1.1.7.1: cp210x converter now attached to ttyUSB25
[    4.757478] usb 1-1.1.7.1: cp210x converter now attached to ttyUSB26
[    4.758873] usb 1-1.1.7.1: cp210x converter now attached to ttyUSB27
[    4.836785] usb 1-1.1.7.2: new full-speed USB device number 12 using ehci-pci
[    4.947182] usb 1-1.1.7.2: New USB device found, idVendor=10c4, idProduct=ea71
[    4.947187] usb 1-1.1.7.2: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    4.947190] usb 1-1.1.7.2: Product: CP2108 Quad USB to UART Bridge
[    4.947193] usb 1-1.1.7.2: Manufacturer: Silicon Labs
[    4.947195] usb 1-1.1.7.2: SerialNumber: 0001
[    4.948634] usb 1-1.1.7.2: cp210x converter now attached to ttyUSB28
[    4.951496] usb 1-1.1.7.2: cp210x converter now attached to ttyUSB29
[    4.953146] usb 1-1.1.7.2: cp210x converter now attached to ttyUSB30
[    4.954188] usb 1-1.1.7.2: cp210x converter now attached to ttyUSB31
[    5.031468] usb 1-1.1.7.3: new full-speed USB device number 13 using ehci-pci
[    5.141963] usb 1-1.1.7.3: New USB device found, idVendor=10c4, idProduct=ea71
[    5.141968] usb 1-1.1.7.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    5.141971] usb 1-1.1.7.3: Product: CP2108 Quad USB to UART Bridge
[    5.141973] usb 1-1.1.7.3: Manufacturer: Silicon Labs
[    5.141976] usb 1-1.1.7.3: SerialNumber: 0001
[    5.143319] usb 1-1.1.7.3: cp210x converter now attached to ttyUSB32
[    5.144438] usb 1-1.1.7.3: cp210x converter now attached to ttyUSB33
[    5.145450] usb 1-1.1.7.3: cp210x converter now attached to ttyUSB34
[    5.146454] usb 1-1.1.7.3: cp210x converter now attached to ttyUSB35
[    5.224458] usb 1-1.1.7.4: new full-speed USB device number 14 using ehci-pci
[    5.333830] usb 1-1.1.7.4: New USB device found, idVendor=10c4, idProduct=ea71
[    5.333836] usb 1-1.1.7.4: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    5.333839] usb 1-1.1.7.4: Product: CP2108 Quad USB to UART Bridge
[    5.333841] usb 1-1.1.7.4: Manufacturer: Silicon Labs
[    5.333844] usb 1-1.1.7.4: SerialNumber: 0001
[    5.335699] usb 1-1.1.7.4: cp210x converter now attached to ttyUSB36
[    5.336818] usb 1-1.1.7.4: cp210x converter now attached to ttyUSB37
[    5.338070] usb 1-1.1.7.4: cp210x converter now attached to ttyUSB38
[    5.339074] usb 1-1.1.7.4: cp210x converter now attached to ttyUSB39
[    5.419465] usb 1-1.1.7.5: new full-speed USB device number 15 using ehci-pci
[    5.536446] usb 1-1.1.7.5: New USB device found, idVendor=10c4, idProduct=ea71
[    5.536452] usb 1-1.1.7.5: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    5.536454] usb 1-1.1.7.5: Product: CP2108 Quad USB to UART Bridge
[    5.536457] usb 1-1.1.7.5: Manufacturer: Silicon Labs
[    5.536459] usb 1-1.1.7.5: SerialNumber: 0001
[    5.538809] usb 1-1.1.7.5: cp210x converter now attached to ttyUSB40
[    5.539940] usb 1-1.1.7.5: cp210x converter now attached to ttyUSB41
[    5.543000] usb 1-1.1.7.5: cp210x converter now attached to ttyUSB42
[    5.545369] usb 1-1.1.7.5: cp210x converter now attached to ttyUSB43
[    5.623444] usb 1-1.1.7.6: new full-speed USB device number 16 using ehci-pci
[    5.734819] usb 1-1.1.7.6: New USB device found, idVendor=10c4, idProduct=ea71
[    5.734824] usb 1-1.1.7.6: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[    5.734827] usb 1-1.1.7.6: Product: CP2108 Quad USB to UART Bridge
[    5.734830] usb 1-1.1.7.6: Manufacturer: Silicon Labs
[    5.734832] usb 1-1.1.7.6: SerialNumber: 0001
[    5.736182] usb 1-1.1.7.6: cp210x converter now attached to ttyUSB44
[    5.737178] usb 1-1.1.7.6: cp210x converter now attached to ttyUSB45
[    5.740060] usb 1-1.1.7.6: cp210x converter now attached to ttyUSB46
[    5.742219] usb 1-1.1.7.6: cp210x converter now attached to ttyUSB47
[   18.588709] Modules linked in: i2c_mux_pca954x i2c_mux dps200(O) pmbus_core hlx_gpio_ich(O) smc(O) bridge stp llc nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm linux_knet_cb(O) linux_bcm_knet(O) psample(O) linux_user_bde(O) linux_kernel_bde(O) irqbypass evdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate bonding serio_raw pcspkr lpc_ich mfd_core sg shpchp button acpi_cpufreq cp210x usbserial xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache nvme nvme_core nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs sd_mod crc32c_intel
[   19.447527]  aesni_intel ahci libahci aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd libata i2c_i801 i2c_smbus scsi_mod ehci_pci ehci_hcd igb i2c_algo_bit dca usbcore ptp usb_common pps_core i2c_ismt

@Blueve
Copy link
Contributor Author

Blueve commented Feb 2, 2021 •
edited
Loading

@lguohan

Base on current investigation, consider there are two possibility:

  1. The new Kernel has bug which cause the cp210x couldn't attach to usb device at first boot
  2. The C0 udev rule blocked the normal process

Here is my plan:
For 1: I am not familiar how the kernel work with usb driver...But if this is the reason, I think we might not able to update kernel easily. Looking forward your suggestion @lguohan
For 2: I will work on 202012 and build an image without C0 udev rule to see if the usb driver can attached after first boot

No matter which one is the root cause, if we can't fix itself, the plan B will be:

  1. Create/Edit a linux service (OneShot)
  2. Run a script (unbind usb 1-1.1 and bind usb 1-1.1) after some SONiC component started, the code can be inject here https://github.com/Azure/sonic-buildimage/blob/d475b96aa314621c0c73d9f1c7e190448631f1ee/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.postinst

Another possible fix is add modprobe cp210x in LN21: https://github.com/Azure/sonic-buildimage/blob/d475b96aa314621c0c73d9f1c7e190448631f1ee/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.init

@Blueve Blueve self-assigned this Feb 2, 2021
@Blueve Blueve linked a pull request Feb 8, 2021 that will close this issue
[Celestica][haliburton][cp210x] modprobe cp210x to ensure the driver can be load properly #6715
Merged
6 tasks
@Blueve Blueve mentioned this issue Feb 8, 2021
Merged
6 tasks
lguohan pushed a commit that referenced this issue Feb 10, 2021
@Blueve
[Celestica][haliburton][cp210x] modprobe cp210x to ensure the driver …
d8fe592 
…loaded properly (#6715)

This PR is to fix issue: #6603
The CP210x driver not attached properly after first login (cold-plug), lsusb and dmesg shown that the usb devices has been recognized but driver is missing.
daall pushed a commit that referenced this issue Feb 16, 2021
@Blueve @daall
[Celestica][haliburton][cp210x] modprobe cp210x to ensure the driver …
b92dfb1 
…loaded properly (#6715)

This PR is to fix issue: #6603
The CP210x driver not attached properly after first login (cold-plug), lsusb and dmesg shown that the usb devices has been recognized but driver is missing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Blueve Blueve

Labels
Issue for 202012 Master Branch Quality Platform: Celestica
Projects
None yet
Milestone
No milestone
2 participants
@lguohan @Blueve