Allowing dynamic linking of NCCL

[jakirkham]

Appears that NCCL's static library is used when building xgboost. Am curious if it would be reasonable to allow users to dynamically link instead or as an alternative to static linking.

[hcho3]

@jakirkham Static linking is used here so that multi-GPU training works "out of the box". Many users will install XGBoost by running pip install xgboost, without necessarily installing NCCL first. Is there a scenario where dynamic linking would be beneficial?

[trivialfis]

@hcho3 Actually I would like to do that. We stay with static by default, but add an option for dynamic linking. To me, the reason is the same with dmlc-core, see #4360 .

[hcho3]

@trivialfis I see that dynamic linking of NCCL may be desirable when XGBoost is used as a dependency to another application (via CMake exported target). Right now, if an application using NCCL imports XGBoost, the NCCL library will be duplicated and cause code bloat.

[trivialfis]

Code bloat is fine, it's nothing close to DL library. The problem is it leads to undefined behaviour when we have functions with same signature but different implementation. The linker can fail to detect that, as illustrated in above issue and related reproducible code.

[hcho3]

@trivialfis Meaning two versions of NCCL co-existing?

[trivialfis]

@hcho3 See https://github.com/trivialfis/xgbissues/tree/master/%234360

[hcho3]

@trivialfis I don't think that link is exactly applicable to the case of NCCL, since we are not including the NCCL source code. We are importing the static library pre-compiled by NVIDIA.

[trivialfis]

The example above imports statically compiled version of lib.h in middle user, lib.h can be nccl, middleuser.cc can be XGBoost. And finaluser.cc can be users of XGBoost. It's not completely the same situation here since nccl is more than just a header. But the general idea is that having duplicated library is dangerous.

[hcho3]

Got it. So to prevent undefined behavior, we want to de-duplicate dependencies, including dmlc-core and NCCL.

[trivialfis]

@hcho3

[trivialfis]

Yes. That would be my reason.

[jakirkham]

Another use case for shared libraries is in package management. Namely we are able to create smaller packages and better track relationships between them by building shared libraries and creating explicit dependencies between upstream and downstream packages that need these shared libraries.