Improving test tools and strategies for multi-threading

[rgommers]

The suggested plan of attack and documented test strategies on https://py-free-threading.github.io are up-to-date given what we know and the tools we have available. However, the testing strategies for uncovering thread-safety issues are falling short of what we need. Improving this should be a priority.

Ideas that need implementing or protyping:

• create a decorator to easily run a pytest test case in parallel (can be based on the test patterns for threading and concurrent.futures in this doc section, and look something like the run_in_parallel from scikit-image)
• a pytest plugin to all tests in a full test suite in parallel (e.g. using the decorator from the task above)
• use ThreadSanitizer on a Python package (it has proven useful for CPython)
  • set up for NumPy and figure out how useful it is (what does it catch, what's the false positive rate)
  • document how to do this with Meson and CMake

[ngoldbaum]

For TSAN, I found some meson configuration in the QEMU repo that might be something to start with. It looks like all that's needed is making sure -fsanitize=thread gets into CFLAGS and LDFLAGS.

[ngoldbaum]

Oh never mind, meson has built-in support for it via b_sanitize. For some reason I thought there wasn't support.

[h-vetinari]

https://py-free-threading.github.io looks like a cool resource! Are you planning to announce it at some point?

[ngoldbaum]

We did, via a blog post. Is there a spot you would have more easily heard the announcement?

[ngoldbaum]

I spent a little time looking at setting up TSAN for NumPy. Unfortunately it's not quite as easy to get working as ASAN. I got it working on my Mac, but because you need to set DYLD_INSERT_LIBRARIES and system integrity protection means that subprocesses don't inherit that environment variable, many tests fail because of that. Eventually I hit a seg fault in test_dtype.py::TestMonsterType::test_dict_recursion which I suspect is causing a stack overflow under TSAN. I'll have to look closer.

I tried also on my x86 linux laptop and unfortunately in Linux TSAN has issues with ASLR on newer kernels: google/sanitizers#1716 (comment)

I might be able to get this working with clang 18 on Linux, but that might be annoying to get working on CI since it looks like that version hasn't been packaged yet for Ubuntu.

[h-vetinari]

We did, via a blog post. Is there a spot you would have more easily heard the announcement?

I think an announcement on discuss.python.org would make sense...? 🤔

[h-vetinari]

I might be able to get this working with clang 18 on Linux, but that might be annoying to get working on CI since it looks like that version hasn't been packaged yet for Ubuntu.

It's packaged (also for linux) in conda-forge. ;)

conda install clang_linux-64=18

[h-vetinari]

Or plain clang if you want the bare binaries without any default flags etc. (which you can of course still override after the environment has been activated)