PCP: PMDA: Introduce new PMDA for RDS #2230

mohith-kumar-thummaluru · 2025-07-02T12:07:08Z

This commit adds a new PMDA (Performance Metrics Domain Agent) for Reliable Datagram Sockets (RDS). It exports key metrics including connection information, socket and connection statistics, and details of send, receive, and retransmit queues for performance analysis using Performance Co-Pilot (PCP).

This PMDA is intended to aid in diagnosing network-related issues on systems using RDS over Infiniband or TCP.

This commit adds a new PMDA (Performance Metrics Domain Agent) for Reliable Datagram Sockets (RDS). It exports key metrics including connection information, socket and connection statistics, and details of send, receive, and retransmit queues for performance analysis using Performance Co-Pilot (PCP). This PMDA is intended to aid in diagnosing network-related issues on systems using RDS over Infiniband or TCP. Signed-off-by: Mohith Kumar Thummaluru <mohith.k.kumar.thummaluru@oracle.com>

natoscott

Distinct lack of QA tests for the PMDA but also for the helper scripts - please add. And
if we can use a standard module or command rather than using libc directly and adding the python ping implementation(s), that'd be better. Other comments inline.

natoscott · 2025-07-03T07:26:47Z

src/pmns/stdpmid.pcp

@@ -161,7 +161,8 @@ QA_TEST_PERL	252
 SIMPLE		253
 MEMORY_PYTHON	254
 # end QA section
-### MORE FREE SLOTS 255..510 ###
+RDS		256
+### MORE FREE SLOTS 257..510 ###


What happened to 255?

natoscott · 2025-07-03T07:27:32Z

src/pmdas/rds/pyprep

+            os.symlink(filename, pyf)
+    os.chdir(PMDADIR)
+os.chdir(CWD)
+


Hmm, what is this script doing? It definitely shouldn't be needed.

let me recheck this

natoscott · 2025-07-03T07:28:27Z

src/pmdas/rds/pmdards.python

+
+sys.path.append("/var/lib/pcp/pmdas/rds")
+sys.path.append("/var/lib/pcp/pmdas/rds")
+from modules.rds_ping import rds_ping_all_avlbl_dest


Need to remove the hard-coded use of /var/lib/pcp/pmdas - this should be extracted via pmGetConfig

in my dev env, i was facing trouble without these lines, can you point me to some examples on how to use pmGetConfig

Here's an example:

pcp/src/pmdas/openvswitch/pmdaopenvswitch.python

Line 34 in f7aaf6a

PMDA_DIR = PCP.pmGetConfig('PCP_PMDAS_DIR')

natoscott · 2025-07-03T07:36:06Z

src/pmdas/rds/Install

+daemon_opt=false
+
+# Prepare PMDA Python files (needed for easier packaging)
+$PCP_PYTHON_PROG $PCP_PMDAS_DIR/rds/pyprep


Remove this - see notes later regarding this script.

natoscott · 2025-07-03T07:43:08Z

src/pmdas/rds/pmdards.python

+    """
+    # Centralized metric mapping
+    METRICS = {
+        # CLUSTER_CONN


General notes here apply to the metric specifications below. All of the units seem to be set to zero - seems unlikely? All of the metrics seem to be hard-coded as PM_SEM_INSTANT - are there really no counters? That'd be very unusual and based on the names, alot of them look like counters. There's no help text for any of the metrics, so I can't tell what they are. The vast majority seem to be specified as 32 bit values too - this is also very unusual these days, most things would be 64bit. The metrics specified with "8k" and "1m" are duplicated - these should be specified once and have instances of 8k/1m instead of a null instance domain.

natoscott · 2025-07-03T08:00:42Z

src/pmdas/rds/modules/rds_ping.py

+import time
+import json
+from time import perf_counter
+sys.path.append("/var/lib/pcp/pmdas/rds/modules")


Hard-coded PCP path here - use pmGetConfig.

natoscott · 2025-07-03T08:01:15Z

src/pmdas/rds/modules/rds_ping.py

+        sock_fd = socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0)
+        return sock_fd
+    except socket.error as err:
+        print("socket creation failed with error %s" % (err))


Lots of print (-to-stdout) - these look like errors for stderr

i'll fix it

natoscott · 2025-07-03T08:03:06Z

src/pmdas/rds/modules/rds_ping.py

+    print()
+
+def main(argv):
+    parser = argparse.ArgumentParser(description='python version of rds-ping utility.',


Where did this code come from? All authored for this PR (and GPL2+) or is there another author/license involved?

this is authored for this PR

but there a C implementation of it available - https://github.com/oracle/rds-tools/blob/rdma-vos/rds-tools-2/rds-ping.c

I think its OK to stick with python over C, particularly since the rest of the PMDA is python code. How do we know this code works though? (there's no tests, not even indirect tests using the agent to run the code, nor even pylint static checks yet - let's get these in-place asap). Its a lot of code to add to the project for everyone to maintain.

Agreed. Let me work on the test scripts.

natoscott · 2025-07-03T08:04:48Z

src/pmdas/rds/modules/rds_info.py

+
+        # If the user requests help or an invalid option is passed
+        if option == "--help" or option not in infos:
+            print("Usage: python script.py <option>")


good catch, let me fix it

natoscott · 2025-07-03T08:06:34Z

src/pmdas/rds/modules/rds_info.py

+if __name__ == "__main__":
+    # Ensure an argument is passed
+    if len(sys.argv) != 2:
+        print("Usage: python rds_info.py <option>")


Looks like this could share a usage message with the above script.py code?

mohith-kumar-thummaluru · 2025-07-03T10:14:30Z

if we can use a standard module or command rather than using libc directly and adding the python ping implementation(s), that'd be better.

There were multiple reasons behind choosing the libc-based approach instead of relying on standard tools.

We wanted to eliminate dependencies on native utilities like ping and rds-ping, especially since our use case required functionality beyond what these tools offer. Our implementation provides an enhanced version of rds-ping, which supports sending pings to multiple connections using a single socket. This socket can be bound to a specific source address and ToS (Type of Service) value, significantly reducing overhead and simplifying connection management.

Additionally, the Python implementation of getsockopt allocates a fixed 1024-byte buffer when retrieving connection information from the kernel. While this is typically sufficient for generic socket information, it may be inadequate for extracting RDS protocol-specific data. Although CPython internally invokes the C getsockopt system call, it ignores the return value, which is critical for parsing the data bytes returned by the kernel. This omission can lead to incorrect parsing or missed information. In contrast, the direct C implementation using libc respects the return value and imposes no such buffer limitation, allowing for more accurate and flexible data extraction.

Considering all these factors, we implemented the required tools directly using libc to gain better control, accuracy, and efficiency for our specific needs.

natoscott requested changes Jul 3, 2025

View reviewed changes

Uh oh!

PCP: PMDA: Introduce new PMDA for RDS #2230

Are you sure you want to change the base?

PCP: PMDA: Introduce new PMDA for RDS #2230

Uh oh!

Conversation

mohith-kumar-thummaluru commented Jul 2, 2025

Uh oh!

natoscott left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohith-kumar-thummaluru commented Jul 3, 2025

Uh oh!

Uh oh!