Do not error on NCCL library load failure during CUDA driver creation

[sogartar]

If a library symbol is not found or the NCCL library version does not match the requirements do not fail during CUDA driver creation.

[sogartar]

I started a discussion here #13400

[benvanik]

let's return UNAVAILABLE here instead so that we have a consistent status code to look for

[stellaraccident]

I think this needs more of a re-organization. See my notes here: #13422

[okkwon]

I agree. I thought about making the nccl loading more compatible with the old version, which does not have ncclCommInitRankConfig, but I remembered that there were some perf improvement by tweaking the config. So let's keep the version as is.

@sogartar Do you want to make a stab? Let me know.

Note that 2.17 is not backward compatible to 2.15 because it extended the config structure, and also has a bug that makes the program hang. @trevor-m reported the issue to the NCCL team.

[trevor-m]

I agree. I thought about making the nccl loading more compatible with the old version, which does not have ncclCommInitRankConfig, but I remembered that there were some perf improvement by tweaking the config. So let's keep the version as is.

@sogartar Do you want to make a stab? Let me know.

Note that 2.17 is not backward compatible to 2.15 because it extended the config structure, and also has a bug that makes the program hang. @trevor-m reported the issue to the NCCL team.

I realized the config structure change is safe since nccl checks the version field before trying to read those new fields.
The hang issue is still remaining.

[okkwon]

This is done at #13422. Closing.