GPU memory related issues

[lijh5]

1. Can UCX recognize which specific GPU's address space the current user data buffer belongs to? For example, I called cudaMalloc on gpu3 to apply for 4M of space, called ucx_tag_nbx_send() for data transfer, and ucx can detect that it is gpu memory through cuda_copy. Can it be detected that it is gpu3 memory?
2. How does cuda_copy or rocm_copy detect that it belongs to GPU memory?
3. Does UCX support node hardware topology detection? For example, I want to know which GPU is closest to the current network card.

Thank you.

[yosefe]

1. yes, by using cuPointerGetAttributes in uct_cuda_copy_md_query_attributes
2. cuda memory is detected in same function, uct_cuda_copy_md_query_attributes, and the result is cached in the memtype cache
3. yes, by the code in ucs/sys/topo