Impact of new PyPI dependency.

[trivialfis]

Comment:

Hi all,

We want to add NCCL as a pyproject dependency to avoid static linking in PyPI binary release: dmlc/xgboost#9796 , this can help us reduce two-thirds of the weight on PyPI. Others (conda-forge, for instance) can continue to link nccl at compile time without any change. (non-breaking for C++)

dependencies = [
    "numpy",
    "scipy",
    "nvidia-nccl-cu12 ; platform_system == 'Linux' and platform_machine != 'aarch64'"
]

However, we are not sure what would be the potential impact on the conda Python package during installation. After the change, is it still possible for XGBoost on conda-forge to use nccl from conda channels instead of PyPI? It would be great if we don't need to fetch nccl using pip when XGBoost is installed using conda/mamba. (like pip --no-dependencies) If not, is there anything we can do to ensure the Python package is compatible with both distribution methods, like adding a pyproject template or inserting predicates somewhere in the build/install process?

[xhochy]

This sounds like we need to maintain a patch on the conda-forge side that removes this entry from the pyproject.toml as part of the recipe for XGBoost.

[trivialfis]

If there's indeed no way to declare a specific dependency being "PyPI only", we can probably add a template for generating the toml file based on target environments and run it as part the the packaging steps.

[jakirkham]

Currently we are using setup.py as we are on 1.7.6. When we upgrade to 2.0.1, we will move to pip

Normally conda-build disables installing dependencies from pip. Unfortunately this doesn't work with multiple outputs (as used here) due to a conda-build bug ( conda/conda-build#3993 )

As a workaround to the issue previously noted, we typically either set the same environment variables conda-build would or pass equivalent flags to pip

Don't think we will need to do more than that for this case (so no patch hopefully)

[trivialfis]

That's good news! I was worrying about how to make specialization.