gh-76535: Make PyUnicode_ToLowerFull and friends public

[lysnikolaou]

Make _PyUnicode_ToLowerFull, _PyUnicode_ToUpperFull, _PyUnicode_ToTitleFull and _PyUnicode_ToFoldedFull public and rename them to PyUnicode_ToLower etc.

• Issue: Unclear intention of deprecating Py_UNICODE_TOLOWER / Py_UNICODE_TOUPPER #76535

---

📚 Documentation preview 📚: https://cpython-previews--136176.org.readthedocs.build/

[lysnikolaou]

Thanks for taking a look @vstinner! Feedback addressed.

[serhiy-storchaka]

In #76535 (comment) @vstinner suggested to provide a constant which is the minimum buffer size.

If this is indeed a hard constant which will never be changed in future Unicode standards, then I prefer this way. It is too expensive to allocate the output buffer dynamically.

cc @ezio-melotti, our Unicode expert.

[lysnikolaou]

Another question I have is whether we want to expose something like the following to handle the Greek letter sigma edge case:

int PyUnicode_ToLowerHandleSigma(Py_UCS4 *str, Py_UCS4 ch, Py_UCS4 *buffer, int size)

[vstinner]

Thanks, I prefer this API which is more future-proof, it doesn't depend on a specific Unicode version.

[vstinner]

PyUnicode_ToLowerHandleSigma

Would you mind to elaborate? I'm not aware of this special case.

[vstinner]

If this is indeed a hard constant which will never be changed in future Unicode standards

Even if it's a constant which will never (!) change, IMO it's better to request a size as an argument to make the caller responsible to check the buffer size. APIs which accept a pointer with no size are a bad pattern, like the deprecated gets() function.

[lysnikolaou]

Further feedback addressed.

Would you mind to elaborate? I'm not aware of this special case.

There's one special case, the Greek letter sigma, where the result of lower casing is context-specific. More specifically, Σ gets lower-cased to ς if it's at the end of the word or to σ otherwise. This is handled in lower_ucs4 right now.

[vstinner]

Can you try to add tests to Modules/_testcapi/unicode.c and Lib/test/test_capi/test_unicode.py?

[vstinner]

There's one special case, the Greek letter sigma, where the result of lower casing is context-specific. More specifically, Σ gets lower-cased to ς if it's at the end of the word or to σ otherwise.

Oh, that's a tricky case. Proposed API takes a single character, so we don't know if Σ is at the end of a word or not. I don't think that it's worth it to handle this special case in proposed API.

[lysnikolaou]

Can you try to add tests to Modules/_testcapi/unicode.c and Lib/test/test_capi/test_unicode.py?

Done.

[serhiy-storchaka]

If we add too many parameters and runtime checks, this will make the API slower and more difficult to use.