Implement declarative ext-modules in pyproject.toml ("experimental")

[abravalheri]

Summary of changes

Implement declarative definition of the equivalent of setup(ext_modules=[...]) via pyproject.toml:

• Modified JSON schema and associated validation
• Modified setuptools/config/_apply_pyprojecttoml.py to transform configs from pyproject.toml into setuptools.Extension object.
• Added temporary experimental warning to collect feedback.
• Added initial/basic documentation for the feature.

Closes #2220 (the original issue suggests setup.cfg, this PR implements the capability in pyproject.toml, but that should be fine).

/crossref cython/cython#6338
/cc @webknjaz

Pull Request Checklist

• Changes have tests - well just very simple ones
• News fragment added in newsfragments/.
  (See documentation for details)

[abravalheri]

What this PR does not do is automatic detection as suggested by @jaraco in #2220 (comment).

I feel that this would be the next step with a separated discussion/implementation. See #4569.

[abravalheri]

@jaraco, would it be OK if we experiment with allowing binary extensions to be defined in pyproject.toml as proposed in this PR for the next release?

[jaraco]

@jaraco, would it be OK if we experiment with allowing binary extensions to be defined in pyproject.toml as proposed in this PR for the next release?

SGTM

[webknjaz]

(the original issue suggests setup.cfg, this PR implements the capability in pyproject.toml, but that should be fine)

@abravalheri personally, I still like using setup.cfg to unclutter pyproject.toml. But up to you.

[webknjaz]

@da-woods ^

[webknjaz]

@abravalheri have you looked into interfaced suggested by https://ofek.dev/extensionlib/? Also, there's an interesting structure https://github.com/joshua-auchincloss/hatch-cython#usage.

If some of those bits look good, perhaps it's worth gravitating towards those. OTOH, bear in mind that the PyPA adoption has been postponed: https://mail.python.org/archives/list/pypa-committers@python.org/thread/H76Q6OO5TFHLR2IICXRRJ5ZHEPBPUVR7/.

[webknjaz]

How would this work with SWIG?
#4619

[abravalheri]

abravalheri have you looked into interfaced suggested by https://ofek.dev/extensionlib/? Also, there's an interesting structure https://github.com/joshua-auchincloss/hatch-cython#usage.

I don't want to do anything fancy here, simply translate the configuration that already exists in setuptools and that people are already used to. I had a look on extensionlib, but that seems to have a different target (interaction on backends and backend plugins), which would not be the case. Also, I don't think they should be using the [project] table without passing a PEP, as the table is reserved for approved standards.

The approach I am following is very similar to the same one implemented in setuptools-rust.

The interface implemented in this PR will be friendly to existing setuptools users and help with porting. Users will be able to write the equivalent of static setuptools.Extension objects in pyproject.toml in an almost 1-to-1 direct translation.

[abravalheri]

I still like using setup.cfg to unclutter pyproject.toml

In previous discussions in the setuptools repository, it has been established that the y vision is to not maintain 2 different formats. That is why I think we should not implement new features on setup.cfg.

Regarding g the cluttering of pyproject.toml, I do feel that this problem exists. However, the primary purpose for the existence of the file is to be used for building Python packages, so I believe it is the right place of building configs. It might be worth considering moving non-packaging information out of pyproject.toml.

In the future, we might explore other convention-over-configuration approaches, but this PR can establish the foundation that we can add those on top.

[webknjaz]

🎉

[webknjaz]

@abravalheri I was thinking whether I could apply this to https://github.com/aio-libs/multidict somehow (the C-extensions are written in pure C, not Cython) and realized that one dynamic bit present is OS-dependent CFLAGS: https://github.com/aio-libs/multidict/blob/f3876fd/setup.py#L12-L30.

Do you think a future setuptools version would be able to accommodate that?

[abravalheri]

We possibly can implement some bits of dynamic functionality on top of the existing static configuration. But we have to think about a proposal that makes sense.

I saw the link you sent on https://github.com/joshua-auchincloss/hatch-cython#usage, and my first impressions where:

• The include_<somelib> functionality seem to be something very useful to incorporate, so that we can provide support for libraries such as numpy. I would not necessarily use the same syntax, but the overall idea sounds interesting.
• I did not like the "conditionals" on the platform matching:

compile_args = [
  # single string
  "-v",
  # by platform
  { platforms = ["linux", "darwin"], arg = "-Wcpp" },
  # by platform & arch
  { platforms = "darwin", arch = "x86_64", arg = "-arch x86_64" },
  { platforms = ["darwin"], arch = "arm64", arg = "-arch arm64" },
  # with pep508 markers
  { platforms = ["darwin"], arch = "x86_64", arg = "-I/usr/local/opt/llvm/include", depends_path = true, marker = "python_version <= '3.10'"  },
]

If feels pretty much a re-implementation of a classical switch...case/if...elif..else statement on top of TOML¹. If someone reaches the stage that they need to implement something almost procedural like this, they might just be better off to write a couple of lines in Python code. Implementing something like this would feel like "re-inventing the wheel".

What I want to avoid is people writing TOML just for the sake of it or just because it is more fashionable nowadays.

I think adding dynamic capability to integrate better with plugins and 3rd-party libraries makes a lot of sense. But hand-picked customisations may just make more sense written in Python.

Footnotes

1. It is not because a language was created to be primarily descriptive, that someone cannot implement a somehow procedural DSL on top of it... Historical examples include XSLT. ↩