Create and use test harness for the Windows kernel (freestanding)

[ben-craig]

It would be awesome if the applicable parts of the STL were tested for use in the Windows kernel. The author of this issue (Ben Craig) has tinkered with such a thing in the past, and I'm willing to help. This could be considered preparatory work for https://wg21.link/p1642, but it has value even with the status quo of the standard.

Low level theory of operation for tests:

• Have a "regular" user mode executable that takes the name of a kernel binary on the command line. The user mode executable will load and run the test in that kernel binary.
• The kernel binary will have a non-main entry point, and it will call main like a not-special function
• assert isn't freestanding, so the test harness (and not the production code) will have it's own, non-conforming implementation of assert that logs to a global buffer and continues. This has worked out surprisingly well for me, though in the general case, you could obviously corrupt kernel mode. Assert messages are output to stderr, and still cause the user mode executable to return non-zero.
• Running the kernel mode tests requires administrator privileges
• As part of the machine setup, test signing needs to be enabled on the machine / VM under test, and a self-signed certificate needs to be installed.
• This will likely require using the lit "Executor" machinery from the libcxx test suites. That machinery allows running tests through emulators, or ssh, or, in this case, through a user mode executable.
• If a test messes up bad enough, it will crash / blue screen your machine. So test in user mode first.

Open questions:

• What cmake and/or lit invocations should users need to run to cause these tests to be run? I suspect that we shouldn't make this test mode be part of a "default" x64 run.
• Should there be a distinct cmake configuration for kernel mode?
• Is installing certificates and enabling test driver signing permissible on a continuous integration run?

[cbezault]

I believe all of the above is possible but will require effort. The only part I'm unsure of is the right way to get elevated permissions.

I believe the executor lit machinery is deprecated and we plan on moving our testing closer to that of upstream libcxx (see the discussion in #1090).

I am 100% for making this a separate test run and all of these tests should run a separate machine from, or after all of, the existing tests. We would probably also want to recycle vms after every use after this change. I think hiding the tests behind another stl-lit.py-like script (maybe kernel-mode-lit.py) would be fine.

[ldionne]

The executor machinery isn't deprecated per se -- it just changed. An executor is now just a plain script that runs your executable as desired (before it was a Python class). You can still have custom executors by specifying the %{exec} substitution when configuring your test suite.

[DBJDBJ]

This text is outdated. Please see: MS STL works with /kernel builds and it raises SEH exceptions

For starters, it might be enough not to allow ms stl compilation in /kernel builds

// <vector>
#ifdef _KERNEL_MODE
#error kernel builds are not yet supported
#else

[sylveon]

Compiling with /kernel will already make including an error, because exceptions are off and there is no global new by default.

[ben-craig]

@DBJDBJ that would be a significant regression from the status quo. Large portions of the MS STL currently work in kernel mode, whether they are "supported" or not. Note that some effort was put into MSVC 2017 to make driver development more palatable: https://devblogs.microsoft.com/cppblog/stl-features-and-fixes-in-vs-2017-15-8/

@sylveon There are large portions of the STL that are still useful without exceptions and without global new. See https://wg21.link/p0829 for a (slightly dated) idea of what kinds of things could be supported in /kernel.

Two years ago, I did a hacky run of the libcxx test suite with a prototype of the harness I'm planning on adding. So this isn't entirely blue-sky development.

[sylveon]

Yes, I'm aware that the STL works and is used in kernel mode, was just suggesting that the guard is unneeded.

[ben-craig]

For now, I will require that the user's cmake invocation will provide a little more information. There will be something along the lines of a KERNEL_MODE=ON flag, and a WINDOWS_DDK_PATH flag

[DBJDBJ]

This text is outdated. Please see: MS STL works with /kernel builds and it raises SEH exceptions

@ben-craig Thanks -- I am not attacking thus there is no need for advocating. :)

Please see this. for some trivial C++ std lib misuse in presence of MS STL. /kernel build.

    std::vector< std::string > vs = { "Element zero." } ;
    auto s = vs.at(22);

Contrary to some firm beliefs that indeed compiles and runs. And it throws std::out_of_range exception too, caught with SEH in that source. (Try it on windows and then open the minidump generated with VS, then go to "Native debug", to learn which exception is thrown) .

Perhaps I misunderstood the /kernel text ... ? For me "no exceptions" means "no exceptions". Also, that code has no replacement new() and delete(), as it should. Is that significant in some way?

One might think until "a lot of things" are resolved MS STL is not fit for /kernel builds. Currently, the situation is very moot. It will significantly clarify the status if it is simply and temporarily: "No /kernel builds". In my mind, that is more clear than: "Large portions of the MS STL currently work". Or " a (slightly dated) idea of what kinds of things could be supported in /kernel.".

This is not going to be an easy task. Again: I am not attacking ...

[ben-craig]

I agree that vector, string, and any other class or function that does a runtime heap allocation or originates a C++ exception are out of scope for default /kernel builds. tuple, move, sort, variant, and many others are fine though. The papers I linked to (and authored) audited the spec for features that do things that are not suitable in kernel, microcontroller, and GPU environments.

[DBJDBJ]

This text is outdated. Please see: MS STL works with /kernel builds and it raises SEH exceptions

@ben-craig thanks -- it is a moot point and unofficial texts, on what can be used and what can not be used from MS STL for /kernel builds. Thus I think the simple #error sorry no can do /kernel builds for now, will work for people trying to use the MS STL in /kernel builds. Who at the same time have not researched the issues or are even completely unaware there are any issues.

And, (I was avoiding this until now): There are insurmountable issues. But that is for some other thread perhaps.

ps: I know your work and am in favour of it.

[DBJDBJ]

In /kernel builds MS MSTL "seems to work" and raises the SEH exceptions.

Here is the code I am using: https://godbolt.org/z/9fr9fY
In case you would like to try, you need to copy that to your windows to e.g. main.cpp. CL command-line I am using, to build is: cl /DNDEBUG /kernel /Fe: your_app_name_here.exe main.cpp. ( start from "Visual Studio 2019 Developer Command Prompt..." )

• Is MS STL raising SEH exceptions "suitable in the kernel, microcontroller, and GPU environments"?
• I might think SEH raising is in compliance with the /kernel switch builds? I am not sure.
• Is it ISO C++ compliant for an STL implementation to raise non-C++ exceptions?

That (and more) is for the good team to document. We (whoever "we" are) need an authoritative document(ation) on these and adjacent subjects. (Apologies if that already exists somewhere)

[ben-craig]

SEH is not a substitute for C++ exceptions. It's not about the throw and the catch, but the destructors called along the way. SEH normally doesn't cause destructors to get called during unwinding. Note that if you put a throw 0; directly in the code, you get error C2980: C++ exception handling is not supported with /kernel.

[DBJDBJ]

@ben-craig Thank you for mentioning destructors which I did not. SEH is very different: please look into the Godbolt link I have supplied. There is no __finaly in there which should be ... but that shape of main or WinMain might be the mandate for MS STL /kernel builds. All just a distraction. The first key question is architectural in nature.

Is (MS STL) raising SEH exceptions "suitable in the kernel, microcontroller, and GPU environments"? That is: is it suitable for "Free Standing" ISO C++?

And there is this elephant question in the corner:

Is it ISO C++ compliant for an STL implementation to raise non-C++ exceptions?

These are the questions for which answers I am looking for. I do enjoy being able to have a "conversation" in here, but I shall leave it to the experts.

[ben-craig]

Is it ISO C++ compliant for an STL implementation to raise non-C++ exceptions?

Short answer: No.
Longer answer: You might be able to use SEH as the implementation of undefined behavior in some places. Implementations are allowed to do anything when the behavior is undefined. Using SEH as a replacement for std::exception based exceptions is not conforming though. Destructors and catch blocks need to work.

Is (MS STL) raising SEH exceptions "suitable in the kernel, microcontroller, and GPU environments"? That is: is it suitable for "Free Standing" ISO C++?

Someone would need to propose them for standardization and discuss their tradeoffs. If they aren't in the standard, then they aren't suitable for use by facilities in the standard. Should such a proposal materialize, I would be against it. SEH relies on some state that is local to a thread (not to be confused with thread_local). State that is local to a thread is very expensive in GPU environments, as there are lots and lots of threads.

Note that this is specifically in regard to the STL raising SEH exceptions. Code that isn't the STL can raise SEH exceptions all they want.

[DBJDBJ]

@ben-craig thank you for this answer/comment/opinion.

Perhaps it might be very interesting to produce, (single OS) P1640R1 with SEH included. One can safely predict MS STL SEH variant will be faster than C++ exceptions.

As far as I can see destructors are also being called in the /kernel mode and with SE being raised. Update: of course not entirely true. If longjmp aka SE is raised, destructors that happen to be in the same scope will not be executed.

[ben-craig]

I'm having trouble in a couple different areas.

1. Test variant management
   My current work-in-progress has an is_kernel flag that gets sets on the CMake command line, and only lit looks at it. So far so good. When I run, I don't know of a good way to stop building with clang. In addition, /EHsc is set in lots and lots of .lst files. So this is making it difficult to build kernel mode instead of user mode.

An alternative is to build kernel mode in addition to user mode and to have the executor skip running the tests unless some extra flag is set. This would require everyone to install the WDK in order to build the repo. From a selfish standpoint, this sounds fantastic, but I can see how it might irritate others. The other issue is that I suspect it would require changing a whole lot of .lst files.

2. Include path management
   The Visual Studio developer command prompts put a bunch of include and link paths in the environment, and those paths are intended for user mode development. The cmake build of the STL relies on those paths to find C-runtime headers, amongst other things. This is a little dirty though, as the C-runtime headers are mixed in with the installed C++ headers. If I accidentally deleted (for example) the header in git, then the tests may not pick up on that, as it would find the header in the installed include directory.

This all ties back to kernel mode because the include path in the environment is not suitable for kernel mode development, and the correct include path is not at all obvious. If we go down the path of always building kernel mode tests, then we'll need to do interesting things to the environment so that the user mode include path doesn't wreck things. Even if we don't always build kernel mode tests, we'll need to be super careful that we pick up the "right" versions of various duplicated internal headers. For example, vadefs.h:
C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29304\include\vadefs.h
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\km\crt\vadefs.h

If I pick up the km\crt version of vadefs.h, then the Visual Studio vcruntime.h gets very angry. But maybe I want the km\crt version of string.h in order to get the kernel memcpy? This unclear line of who owns what headers makes this hard.

[DBJDBJ]

That "std" in the middle is MS STL. With no c++ exceptions enabled.

[StephanTLavavej]

This is out of scope now, and #2277 removed our initial attempts to add support for this in the CI.