CLI support for SmartSwitch PMON #3271

rameshraghupathy · 2024-04-14T23:30:23Z

What I did

Enhanced the following CLIs to support SmartSwitch PMON as described in the PMON HLD documentation "https://github.com/sonic-net/SONiC/blob/d19d8933a43d0a31a4f3b2310f4336f289bca340/doc/smart-switch/pmon/smartswitch-pmon.md"

CLIs:
Added new module "DPUX" support for 1 and 2 below
1. "config chassis module startup DPUX" , where X could be 0, to the maximum number of DPUs-1 in the SmartSwitch chassis
2. "config chassis module shutdown DPUX"

Extended the following CLIs to support the new module "DPUX" and also proved a "all" option to display the "SWITCH" and all "DPUX" modules
1. "show reboot-cause" will remain the same and added "show reboot-cause all"
2. "show reboot-cause history" will remain the same and added "show reboot-cause history ", where module name could be DPUX, SWITCH and all.

Extended the following CLIs to support the new module "DPUX" and also proved a "all" option to display the "SWITCH" and all "DPUX" modules
1. "show system-health summary" will remain the same and added sub-command "show system-health summary ", where module name could be DPUX, SWITCH and all.
2. "show system-health monitor-list" will remain the same and added sub-command "show system-health monitor-list ", where module name could be DPUX, SWITCH and all.
3. "show system-health summary" will remain the same and added sub-command "show system-health summary ", where module name could be DPUX, SWITCH and all.etail" will remain the same and added sub-command "show system-health detail ", where module name could be DPUX, SWITCH and all.
4. Added a new sub command "show system-health dpu ", where module name could be DPUX, and all. This new subcommand will provide additional DPU state details as mentioned in the HLD

How I did it

Kept the original CLI output unaltered
Added sub command to support SmartSwitch "DPUs"
Added additional code in chassisd, and in platform modules.py, chassis.py to support it
Updated the DB tables as mentioned in the PMON HLD

How to verify it

Build an image with the required files (refer to the other upstream PRs and the platform PRs)
Require files:
- This PR including reboot_cause.py, chassis_modules.py, system_health.py)
- The other PR including module_base.py, chassis_base.py, docker-pmon.supervisord.conf.j2, chassisd, mock_module_base.py, and the appropriate database_config.json
- Platform "platform-cisco-8000" supporting PMON (module.py, chassis.py, inventory.py, pmon_daemon_control.json, and the required grpc and DB changes)
Run the CLIs and see the new output

Previous command output (if the output of a command-line utility has changed)

root@sonic:~# show reboot-cause
Unknown

root@sonic:~# show reboot-cause history
Name Cause Time User Comment

2023_06_19_11_00_24 Power Loss N/A N/A Unknown (First boot of SONiC version 202311.10869-dirty-2024044)

New command output (if the output of a command-line utility has changed)

root@sonic:~# show reboot-cause history all
Device Name Cause Time User Comment

SWITCH 2023_06_19_11_00_24 Power Loss N/A N/A Unknown (First boot of SONiC version 202311.10869-dirty-2024044)

root@sonic:~# show reboot-cause history SWITCH
Device Name Cause Time User Comment

SWITCH 2023_06_19_11_00_24 Power Loss N/A N/A Unknown (First boot of SONiC version 202311.10869-dirty-2024044)

root@sonic:~# show reboot-cause history DPU0
Device Name Cause Time User Comment

config/chassis_modules.py

show/reboot_cause.py

oleksandrivantsiv · 2024-04-26T01:00:49Z

Can you please add UT for the new functions?

show/reboot_cause.py

addressed. 2. The DPU reboot-cause data is fetched directly fromn the chassis_state_db now

now

temporarily bypassing the check

implementation

show/chassis_modules.py

is differed for now.

prgeor · 2024-09-16T21:04:41Z

@rameshraghupathy Could you please add sample output for the DPUs/Smartswitch case?

prgeor · 2024-09-16T21:14:48Z

@rameshraghupathy can you update the CLI in this doc https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md . There is a "reboot-cause" section

prgeor · 2024-09-16T21:21:51Z

show/system_health.py

+
+'''
+#
+# TBD: Uncomment this code in phase:2


@rameshraghupathy can you remove these comment code? You can raise a PR in your personal PR for future reference.

prgeor · 2024-09-16T21:22:28Z

tests/system_health_test.py

+
+'''
+#
+# TBD: Uncomment this code in phase:2 when system-health is supported


@rameshraghupathy can you please remove these commented test code. you can raise a PR in your personal repo for future reference

@rameshraghupathy can you please remove these commented test code. you can raise a PR in your personal repo for future reference

Done

@prgeor

rameshraghupathy · 2024-09-24T21:35:32Z

@rameshraghupathy Could you please add sample output for the DPUs/Smartswitch case?

These CLIs were executed on the Switch:

root@MtFuji:/opt/cisco/bin# show reboot-cause
Power Loss
root@MtFuji:/opt/cisco/bin# show reboot-cause all
Device Name Cause Time User

SWITCH 2024_07_24_20_43_22 Power Loss N/A N/A
DPU7 DPU7 Unknown N/A N/A
DPU3 DPU3 Unknown N/A N/A
DPU2 DPU2 Unknown N/A N/A
DPU1 DPU1 Unknown N/A N/A
root@MtFuji:/opt/cisco/bin# show reboot-cause history
Name Cause Time User Comment

2024_07_24_20_43_22 Power Loss N/A N/A Unknown
2024_07_24_06_21_32 Power Loss N/A N/A Unknown
2024_07_23_23_06_57 Kernel Panic Tue Jul 23 11:02:27 PM UTC 2024 N/A N/A
2024_07_23_12_07_57 Power Loss N/A N/A Unknown
2024_07_23_11_21_32 Power Loss N/A N/A Unknown
2024_07_23_09_51_35 Power Loss N/A N/A Unknown (First boot of SONiC version azure_cisco_master.15585-dirty-20240907.225359)
root@MtFuji:/opt/cisco/bin#
root@MtFuji:/opt/cisco/bin# show reboot-cause history DPU7
Device Name Cause Time User Comment

DPU7 DPU7 Unknown N/A N/A N/A
DPU7 DPU7 Unknown N/A N/A N/A
DPU7 DPU7 Software causes (Hardware watchdog reset) N/A N/A N/A
root@MtFuji:/opt/cisco/bin# show reboot-cause history all
Device Name Cause Time User Comment

SWITCH 2024_07_24_20_43_22 Power Loss N/A N/A Unknown
SWITCH 2024_07_24_06_21_32 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_23_06_57 Kernel Panic Tue Jul 23 11:02:27 PM UTC 2024 N/A N/A
SWITCH 2024_07_23_12_07_57 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_11_21_32 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_09_51_35 Power Loss N/A N/A Unknown (First boot of SONiC version azure_cisco_master.15585-dirty-20240907.225359)
DPU7 DPU7 Unknown N/A N/A N/A
DPU7 DPU7 Unknown N/A N/A N/A
DPU7 DPU7 Software causes (Hardware watchdog reset) N/A N/A N/A
DPU3 DPU3 Unknown N/A N/A N/A
DPU3 DPU3 Unknown N/A N/A N/A
DPU2 DPU2 Unknown N/A N/A N/A
DPU2 DPU2 Software causes (Reboot) N/A N/A N/A
DPU2 DPU2 Unknown N/A N/A N/A
DPU2 DPU2 Unknown N/A N/A N/A
DPU1 DPU1 Unknown N/A N/A N/A
DPU1 DPU1 Software causes (Reboot) N/A N/A Unknown

root@MtFuji:/opt/cisco/bin# show reboot-cause history SWITCH
Device Name Cause Time User Comment

SWITCH 2024_07_24_20_43_22 Power Loss N/A N/A Unknown
SWITCH 2024_07_24_06_21_32 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_23_06_57 Kernel Panic Tue Jul 23 11:02:27 PM UTC 2024 N/A N/A
SWITCH 2024_07_23_12_07_57 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_11_21_32 Power Loss N/A N/A Unknown
SWITCH 2024_07_23_09_51_35 Power Loss N/A N/A Unknown (First boot of SONiC version azure_cisco_master.15585-dirty-20240907.225359)

@prgeor

rameshraghupathy · 2024-09-24T21:39:59Z

@rameshraghupathy Could you please add sample output for the DPUs/Smartswitch case?

Done. BTW, the platform work is in progress to update the N/A fields in the output

rameshraghupathy · 2024-09-24T22:14:06Z

@rameshraghupathy can you update the CLI in this doc https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md . There is a "reboot-cause" section

Done
@prgeor

extensions

PR will be raised eventually for phase:2

prgeor · 2024-09-25T14:59:40Z

doc/Command-Reference.md

+
+- Example:
+  ```
+  root@MtFuji:~$  show reboot-cause all


@rameshraghupathy This will list the DPUs that are in admin shut?

prgeor · 2024-09-25T15:00:28Z

doc/Command-Reference.md

+  SWITCH    2024_07_24_20_43_22  Power Loss                 N/A     N/A
+  DPU2      2024_07_24_20_43_22  Software causes (Reboot)   N/A     N/A
+  DPU1      2024_07_24_20_43_22  Software causes (Reboot)   N/A     N/A
+  ```


@rameshraghupathy What will be the output for DPU that is in admin shut?

prgeor · 2024-09-25T15:02:56Z

scripts/intfutil

@@ -221,12 +223,16 @@ def port_optics_get(db, intf_name, type):
    Get optic type info for port
    """
    full_table_id = PORT_TRANSCEIVER_TABLE_PREFIX + intf_name
+    port_id = PORT_TABLE_PREFIX + intf_name
+    port_role = db.get(db.CONFIG_DB, port_id, PORT_ROLE)


@rameshraghupathy I am not sure why we need the PORT_ROLE in config DB. Can you point me to the HLD section?

prgeor · 2024-09-25T15:04:47Z

show/chassis_modules.py

+    state_db = SonicV2Connector(host="127.0.0.1", port="6379")
    state_db.connect(state_db.STATE_DB)

-    key_pattern = '*'
+    key_pattern = CHASSIS_MODULE_INFO_TABLE + '|*'
    if chassis_module_name:
-        key_pattern = '|' + chassis_module_name
+        key_pattern = CHASSIS_MODULE_INFO_TABLE + '|' + chassis_module_name

-    keys = state_db.keys(state_db.STATE_DB, CHASSIS_MODULE_INFO_TABLE + key_pattern)
+    keys = state_db.keys(state_db.STATE_DB, key_pattern)
    if not keys:
        print('Key {} not found in {} table'.format(key_pattern, CHASSIS_MODULE_INFO_TABLE))


@rameshraghupathy Have you tested this CLI change in T2 chassis?

prgeor · 2024-09-25T15:07:16Z

show/reboot_cause.py

+@reboot_cause.command()
+@click.argument('module_name', required=False)
+def history(module_name):
+    """Show history of reboot-cause"""
+    reboot_cause_history = fetch_reboot_cause_history_from_db(module_name)
+    if module_name is not None:
+        header = ['Device', 'Name', 'Cause', 'Time', 'User', 'Comment']
+        click.echo(tabulate(reboot_cause_history, header, numalign="left"))
    else:
-        click.echo("Reboot-cause history is not yet available in StateDB")
-        sys.exit(1)
+        header = ['Name', 'Cause', 'Time', 'User', 'Comment']
+        click.echo(tabulate(reboot_cause_history, header, numalign="left"))


@rameshraghupathy please test this CLI on non-smartswitch

prgeor · 2024-09-25T15:08:07Z

show/system_health.py

+            row[6] = value
+
+
+@system_health.command()


@rameshraghupathy Seems this command is not mentioned in command-reference.md?

prgeor · 2024-09-25T15:09:23Z

show/system_health.py

+        elif up_cnt == 3:
+            oper_status = "Online"
+        else:
+            oper_status = "Partial Online"


@rameshraghupathy What is partial online? Is this defined in HLD?

prgeor · 2024-09-25T15:11:18Z

show/system_health.py

+        for key, value in state_info.items():
+            if key.endswith('_state'):
+                if value.lower() == 'up':
+                    up_cnt = up_cnt + 1
+                if 'midplane' in key and value.lower() == 'down':
+                    midplanedown = True


@rameshraghupathy can you explain whats going on here? Either the midplane is down or up when the user executed the CLI. ?

prgeor · 2024-09-25T15:12:52Z

show/reboot_cause.py

@@ -24,6 +23,100 @@ def read_reboot_cause_file():
    return reboot_cause_dict


+# Function to fetch reboot cause data from database
+def fetch_data_from_db(module_name, fetch_history=False, use_chassis_db=False):


@rameshraghupathy how are these functions prevented from executing on non-smartswitch platforms?

prgeor · 2024-09-25T15:17:43Z

show/reboot_cause.py

+            entry = rdb.get_all(rdb.CHASSIS_STATE_DB, tk)
+        else:
+            entry = rdb.get_all(rdb.STATE_DB, tk)
+
+        if module_name is not None:
+            if 'device' in entry:
+                if module_name != entry['device'] and module_name != "all":
+                    continue
+                if entry['device'] in d and not history:
+                    append = False
+                    continue
+                elif not entry['device'] in d or entry['device'] in d and history:
+                    if not entry['device'] in d:
+                        d.append(entry['device'])
+                        append = True
+            r.append(entry['device'] if 'device' in entry else "SWITCH")
+
+        name = tk.replace(prefix, "")
+        if "|" in name:
+            name = name[:name.rindex('|')] + ''
+        r.append(name)
+        r.append(entry['cause'] if 'cause' in entry else "")
+        r.append(entry['time'] if 'time' in entry else "")
+        r.append(entry['user'] if 'user' in entry else "")
+        if append and not fetch_history:
+            table.append(r)
+        elif fetch_history:
+            r.append(entry['comment'] if 'comment' in entry else "")
+            if module_name is None or module_name == 'all' or module_name.startswith('SWITCH') or \
+               'device' in entry and module_name == entry['device']:


@rameshraghupathy can you test if these CLIs crash if the DB is empty? are we handling those corner case gracefully?

prgeor · 2024-09-25T15:18:49Z

show/system_health.py

+def show_module_state(module_name):
+    chassis_state_db = SonicV2Connector(host=CHASSIS_SERVER, port=CHASSIS_SERVER_PORT)
+    chassis_state_db.connect(chassis_state_db.CHASSIS_STATE_DB)
+    key = 'DPU_STATE|'


@rameshraghupathy I don't see any check here that prevents this function being called on non-smartswitch platform

rameshraghupathy added 2 commits April 14, 2024 15:45

CLI support for SmartSwitch PMON

11cc04d

imad minor fixes

02df0ea

oleksandrivantsiv reviewed Apr 26, 2024

View reviewed changes

config/chassis_modules.py Outdated Show resolved Hide resolved

oleksandrivantsiv reviewed Apr 26, 2024

View reviewed changes

show/reboot_cause.py Outdated Show resolved Hide resolved

oleksandrivantsiv reviewed Apr 26, 2024

View reviewed changes

show/reboot_cause.py Outdated Show resolved Hide resolved

oleksandrivantsiv reviewed Apr 26, 2024

View reviewed changes

show/reboot_cause.py Show resolved Hide resolved

rameshraghupathy added 3 commits April 27, 2024 08:21

Did some cleanup for backward compatibility

e0e4700

removed the column wrapping

0a8fc5a

Made it backward compatible and removed textwrap and added ut to PR

6d61faa

rameshraghupathy mentioned this pull request Apr 29, 2024

PMON High Level Design for SmartSwitch (Replaces the PR #1538) sonic-net/SONiC#1584

Merged

rameshraghupathy added 10 commits April 30, 2024 19:46

1. There was a duplication of part of a function and that has been

8d95dae

addressed. 2. The DPU reboot-cause data is fetched directly fromn the chassis_state_db now

reboot_cause and system_health are obtained directly from chassisStateDB

5c1b666

now

The expected and result are the same but the test is throwing an error,

fe4a8cf

temporarily bypassing the check

Let us get the build going and then look into the test mockup

f896438

Implemented as per the pmon hld, also made some improvements in the

9d6c093

implementation

Fixed the key for CHASSIS_MODULE_INFO_TABLE entries

0904515

Fixed "show reboot-cause all" and "show reboot-cause history all"

ccc380b

Addressing review comments

a8fa81d

Checking if the test issue still exists

1cf96a0

Resolving SA errors triggered due to reboot_cause_test

64fd559

vvolam reviewed Jun 5, 2024

View reviewed changes

show/chassis_modules.py Outdated Show resolved Hide resolved

rameshraghupathy added 8 commits June 6, 2024 20:17

Resolved pre-commit issues

d202e1c

Resolved pre-commit issues

b8c92ae

Improving coverage

9986f7b

Fixed SA related warnings

0dc52f6

Did some cleanup

93df26d

Minor improvements and fixes

7a2aaf4

Adding tests for system health

26f9b8a

Adding more system health related tests

3a592f8

rameshraghupathy and others added 9 commits August 31, 2024 19:09

Testing SA and UT

6d7d539

Added 2 spaces before inline comment

433bc50

Merge branch 'sonic-net:master' into master

3ddcc9c

Enabling "show system-health dpu" cli alone. The rest of the dpu health

95da5c0

is differed for now.

Fixed SA issues

627dd5e

Adde new line at EOF

934e6ef

Enabling the UT for the CLI "show system-health dpu"

64d06ec

Resolved SA issues

4870a86

Resolved a SA issue

fed3f67

prgeor reviewed Sep 16, 2024

View reviewed changes

rameshraghupathy added 2 commits September 24, 2024 15:17

Added smartswitch specific "reboot-cause" and "reboot-cause history" CLI

68b6416

extensions

Removed the phase:2 related system-health cli extensions as a seperate

d229307

PR will be raised eventually for phase:2

rameshraghupathy force-pushed the master branch from f818c3c to d229307 Compare September 24, 2024 22:26

prgeor reviewed Sep 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI support for SmartSwitch PMON #3271

CLI support for SmartSwitch PMON #3271

rameshraghupathy commented Apr 14, 2024 •

edited

Loading

oleksandrivantsiv commented Apr 26, 2024

prgeor commented Sep 16, 2024

prgeor commented Sep 16, 2024

prgeor Sep 16, 2024

prgeor Sep 16, 2024

rameshraghupathy Sep 24, 2024

rameshraghupathy commented Sep 24, 2024 •

edited

Loading

rameshraghupathy commented Sep 24, 2024

rameshraghupathy commented Sep 24, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

prgeor Sep 25, 2024

CLI support for SmartSwitch PMON #3271

Are you sure you want to change the base?

CLI support for SmartSwitch PMON #3271

Conversation

rameshraghupathy commented Apr 14, 2024 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

oleksandrivantsiv commented Apr 26, 2024

prgeor commented Sep 16, 2024

prgeor commented Sep 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rameshraghupathy commented Sep 24, 2024 • edited Loading

rameshraghupathy commented Sep 24, 2024

rameshraghupathy commented Sep 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rameshraghupathy commented Apr 14, 2024 •

edited

Loading

rameshraghupathy commented Sep 24, 2024 •

edited

Loading