ArgIteratorWindows: Match post-2008 C runtime rather than CommandLineToArgvW
#19655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
squeek502
force-pushed
the
windows-argv-post-2008
branch
from
651fee2
to
633f758
Compare
castholm
reviewed
Apr 14, 2024
There was a problem hiding this comment.
Great catch! I didn't fully realize CommandLineToArgvW and the C/C++ runtime algorithm were two different algorithms and had assumed the "pre-/post-2008" part meant that CommandLineToArgvW was silently updated at that time, not that C/C++ runtime split off from it.
Juggling arguments on Windows is a bit of a wild west, as I'm sure we will soon see in the follow-up BatBadBut mitigation PR 😅
squeek502
force-pushed
the
windows-argv-post-2008
branch
3 times, most recently
from
0716dfb
to
e0fa937
Compare
…ToArgvW On Windows, the command line arguments of a program are a single WTF-16 encoded string and it's up to the program to split it into an array of strings. In C/C++, the entry point of the C runtime takes care of splitting the command line and passing argc/argv to the main function. ziglang#18309 updated ArgIteratorWindows to match the behavior of CommandLineToArgvW, but it turns out that CommandLineToArgvW's behavior does not match the behavior of the C runtime post-2008. In 2008, the C runtime argv splitting changed how it handles consecutive double quotes within a quoted argument (it's now considered an escaped quote, e.g. `"foo""bar"` post-2008 would get parsed into `foo"bar`), and the rules around argv[0] were also changed. This commit makes ArgIteratorWindows match the behavior of the post-2008 C runtime, and adds a standalone test that verifies the behavior matches both the MSVC and MinGW argv splitting exactly in all cases (it checks that randomly generated command line strings get split the same way). The motivation here is roughly the same as when the same change was made in Rust (rust-lang/rust#87580), that is (paraphrased): - Consistent behavior between Zig and modern C/C++ programs - Allows users to escape double quotes in a way that can be more straightforward Additionally, the suggested mitigation for BatBadBut (https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/) relies on the post-2008 argv splitting behavior for roundtripping of the arguments given to `cmd.exe`. Note: it's not necessary for the suggested mitigation to work, but it is necessary for the suggested escaping to be parsed back into the intended argv by ArgIteratorWindows after being run through a `.bat` file.
squeek502
force-pushed
the
windows-argv-post-2008
branch
from
e0fa937
to
cffe199
Compare
|
Thanks @squeek502. |
andrewrk
added
breaking
squeek502
added a commit
to squeek502/zig
that referenced
this pull request
Apr 19, 2024
… on Windows (BatBadBut) > Note: This first part is mostly a rephrasing of https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/ > See that article for more details On Windows, it is possible to execute `.bat`/`.cmd` scripts via CreateProcessW. When this happens, `CreateProcessW` will (under-the-hood) spawn a `cmd.exe` process with the path to the script and the args like so: cmd.exe /c script.bat arg1 arg2 This is a problem because: - `cmd.exe` has its own, separate, parsing/escaping rules for arguments - Environment variables in arguments will be expanded before the `cmd.exe` parsing rules are applied Together, this means that (1) maliciously constructed arguments can lead to arbitrary command execution via the APIs in `std.process.Child` and (2) escaping according to the rules of `cmd.exe` is not enough on its own. A basic example argv field that reproduces the vulnerability (this will erroneously spawn `calc.exe`): .argv = &.{ "test.bat", "\"&calc.exe" }, And one that takes advantage of environment variable expansion to still spawn calc.exe even if the args are properly escaped for `cmd.exe`: .argv = &.{ "test.bat", "%CMDCMDLINE:~-1%&calc.exe" }, (note: if these spawned e.g. `test.exe` instead of `test.bat`, they wouldn't be vulnerable; it's only `.bat`/`.cmd` scripts that are vulnerable since they go through `cmd.exe`) Zig allows passing `.bat`/`.cmd` scripts as `argv[0]` via `std.process.Child`, so the Zig API is affected by this vulnerability. Note also that Zig will search `PATH` for `.bat`/`.cmd` scripts, so spawning something like `foo` may end up executing `foo.bat` somewhere in the PATH (the PATH searching of Zig matches the behavior of cmd.exe). > Side note to keep in mind: On Windows, the extension is significant in terms of how Windows will try to execute the command. If the extension is not `.bat`/`.cmd`, we know that it will not attempt to be executed as a `.bat`/`.cmd` script (and vice versa). This means that we can just look at the extension to know if we are trying to execute a `.bat`/`.cmd` script. --- This general class of problem has been documented before in 2011 here: https://learn.microsoft.com/en-us/archive/blogs/twistylittlepassagesallalike/everyone-quotes-command-line-arguments-the-wrong-way and the course of action it suggests for escaping when executing .bat/.cmd files is: - Escape first using the non-cmd.exe rules - Then escape all cmd.exe 'metacharacters' (`(`, `)`, `%`, `!`, `^`, `"`, `<`, `>`, `&`, and `|`) with `^` However, escaping with ^ on its own is insufficient because it does not stop cmd.exe from expanding environment variables. For example: ``` args.bat %PATH% ``` escaped with ^ (and wrapped in quotes that are also escaped), it *will* stop cmd.exe from expanding `%PATH%`: ``` > args.bat ^"^%PATH^%^" "%PATH%" ``` but it will still try to expand `%PATH^%`: ``` set PATH^^=123 > args.bat ^"^%PATH^%^" "123" ``` The goal is to stop *all* environment variable expansion, so this won't work. Another problem with the ^ approach is that it does not seem to allow all possible command lines to round trip through cmd.exe (as far as I can tell at least). One known example: ``` args.bat ^"\^"key^=value\^"^" ``` where args.bat is: ``` @echo %1 %2 %3 %4 %5 %6 %7 %8 %9 ``` will print ``` "\"key value\"" ``` (it will turn the `=` into a space for an unknown reason; other minor variations do roundtrip, e.g. `\^"key^=value\^"`, `^"key^=value^"`, so it's unclear what's going on) It may actually be possible to escape with ^ such that every possible command line round trips correctly, but it's probably not worth the effort to figure it out, since the suggested mitigation for BatBadBut has better roundtripping and leads to less garbled command lines overall. --- Ultimately, the mitigation used here is the same as the one suggested in: https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/ The mitigation steps are reproduced here, noted with one deviation that Zig makes (following Rust's lead): 1. Replace percent sign (%) with %%cd:~,%. 2. Replace the backslash (\) in front of the double quote (") with two backslashes (\\). 3. Replace the double quote (") with two double quotes (""). 4. ~~Remove newline characters (\n).~~ - Instead, `\n`, `\r`, and NUL are disallowed and will trigger `error.InvalidBatchScriptArg` if they are found in `argv`. These three characters do not roundtrip through a `.bat` file and therefore are of dubious/no use. It's unclear to me if `\n` in particular is relevant to the BatBadBut vulnerability (I wasn't able to find a reproduction with \n and the post doesn't mention anything about it except in the suggested mitigation steps); it just seems to act as a 'end of arguments' marker and therefore anything after the `\n` is lost (and same with NUL). `\r` seems to be stripped from the command line arguments when passed through a `.bat`/`.cmd`, so that is also disallowed to ensure that `argv` can always fully roundtrip through `.bat`/`.cmd`. 5. Enclose the argument with double quotes ("). The escaped command line is then run as something like: cmd.exe /d /e:ON /v:OFF /c "foo.bat arg1 arg2" Note: Previously, we would pass `foo.bat arg1 arg2` as the command line and the path to `foo.bat` as the app name and let CreateProcessW handle the `cmd.exe` spawning for us, but because we need to pass `/e:ON` and `/v:OFF` to cmd.exe to ensure the mitigation is effective, that is no longer tenable. Instead, we now get the full path to `cmd.exe` and use that as the app name when executing `.bat`/`.cmd` files. --- A standalone test has also been added that tests two things: 1. Known reproductions of the vulnerability are tested to ensure that they do not reproduce the vulnerability 2. Randomly generated command line arguments roundtrip when passed to a `.bat` file and then are passed from the `.bat` file to a `.exe`. This fuzz test is as thorough as possible--it tests that things like arbitrary Unicode codepoints and unpaired surrogates roundtrip successfully. Note: In order for the `CreateProcessW` -> `.bat` -> `.exe` roundtripping to succeed, the .exe must split the arguments using the post-2008 C runtime argv splitting implementation, see ziglang#19655 for details on when that change was made in Zig.
squeek502
added a commit
to squeek502/zig
that referenced
this pull request
Apr 23, 2024
… on Windows (BatBadBut) > Note: This first part is mostly a rephrasing of https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/ > See that article for more details On Windows, it is possible to execute `.bat`/`.cmd` scripts via CreateProcessW. When this happens, `CreateProcessW` will (under-the-hood) spawn a `cmd.exe` process with the path to the script and the args like so: cmd.exe /c script.bat arg1 arg2 This is a problem because: - `cmd.exe` has its own, separate, parsing/escaping rules for arguments - Environment variables in arguments will be expanded before the `cmd.exe` parsing rules are applied Together, this means that (1) maliciously constructed arguments can lead to arbitrary command execution via the APIs in `std.process.Child` and (2) escaping according to the rules of `cmd.exe` is not enough on its own. A basic example argv field that reproduces the vulnerability (this will erroneously spawn `calc.exe`): .argv = &.{ "test.bat", "\"&calc.exe" }, And one that takes advantage of environment variable expansion to still spawn calc.exe even if the args are properly escaped for `cmd.exe`: .argv = &.{ "test.bat", "%CMDCMDLINE:~-1%&calc.exe" }, (note: if these spawned e.g. `test.exe` instead of `test.bat`, they wouldn't be vulnerable; it's only `.bat`/`.cmd` scripts that are vulnerable since they go through `cmd.exe`) Zig allows passing `.bat`/`.cmd` scripts as `argv[0]` via `std.process.Child`, so the Zig API is affected by this vulnerability. Note also that Zig will search `PATH` for `.bat`/`.cmd` scripts, so spawning something like `foo` may end up executing `foo.bat` somewhere in the PATH (the PATH searching of Zig matches the behavior of cmd.exe). > Side note to keep in mind: On Windows, the extension is significant in terms of how Windows will try to execute the command. If the extension is not `.bat`/`.cmd`, we know that it will not attempt to be executed as a `.bat`/`.cmd` script (and vice versa). This means that we can just look at the extension to know if we are trying to execute a `.bat`/`.cmd` script. --- This general class of problem has been documented before in 2011 here: https://learn.microsoft.com/en-us/archive/blogs/twistylittlepassagesallalike/everyone-quotes-command-line-arguments-the-wrong-way and the course of action it suggests for escaping when executing .bat/.cmd files is: - Escape first using the non-cmd.exe rules - Then escape all cmd.exe 'metacharacters' (`(`, `)`, `%`, `!`, `^`, `"`, `<`, `>`, `&`, and `|`) with `^` However, escaping with ^ on its own is insufficient because it does not stop cmd.exe from expanding environment variables. For example: ``` args.bat %PATH% ``` escaped with ^ (and wrapped in quotes that are also escaped), it *will* stop cmd.exe from expanding `%PATH%`: ``` > args.bat ^"^%PATH^%^" "%PATH%" ``` but it will still try to expand `%PATH^%`: ``` set PATH^^=123 > args.bat ^"^%PATH^%^" "123" ``` The goal is to stop *all* environment variable expansion, so this won't work. Another problem with the ^ approach is that it does not seem to allow all possible command lines to round trip through cmd.exe (as far as I can tell at least). One known example: ``` args.bat ^"\^"key^=value\^"^" ``` where args.bat is: ``` @echo %1 %2 %3 %4 %5 %6 %7 %8 %9 ``` will print ``` "\"key value\"" ``` (it will turn the `=` into a space for an unknown reason; other minor variations do roundtrip, e.g. `\^"key^=value\^"`, `^"key^=value^"`, so it's unclear what's going on) It may actually be possible to escape with ^ such that every possible command line round trips correctly, but it's probably not worth the effort to figure it out, since the suggested mitigation for BatBadBut has better roundtripping and leads to less garbled command lines overall. --- Ultimately, the mitigation used here is the same as the one suggested in: https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/ The mitigation steps are reproduced here, noted with one deviation that Zig makes (following Rust's lead): 1. Replace percent sign (%) with %%cd:~,%. 2. Replace the backslash (\) in front of the double quote (") with two backslashes (\\). 3. Replace the double quote (") with two double quotes (""). 4. ~~Remove newline characters (\n).~~ - Instead, `\n`, `\r`, and NUL are disallowed and will trigger `error.InvalidBatchScriptArg` if they are found in `argv`. These three characters do not roundtrip through a `.bat` file and therefore are of dubious/no use. It's unclear to me if `\n` in particular is relevant to the BatBadBut vulnerability (I wasn't able to find a reproduction with \n and the post doesn't mention anything about it except in the suggested mitigation steps); it just seems to act as a 'end of arguments' marker and therefore anything after the `\n` is lost (and same with NUL). `\r` seems to be stripped from the command line arguments when passed through a `.bat`/`.cmd`, so that is also disallowed to ensure that `argv` can always fully roundtrip through `.bat`/`.cmd`. 5. Enclose the argument with double quotes ("). The escaped command line is then run as something like: cmd.exe /d /e:ON /v:OFF /c "foo.bat arg1 arg2" Note: Previously, we would pass `foo.bat arg1 arg2` as the command line and the path to `foo.bat` as the app name and let CreateProcessW handle the `cmd.exe` spawning for us, but because we need to pass `/e:ON` and `/v:OFF` to cmd.exe to ensure the mitigation is effective, that is no longer tenable. Instead, we now get the full path to `cmd.exe` and use that as the app name when executing `.bat`/`.cmd` files. --- A standalone test has also been added that tests two things: 1. Known reproductions of the vulnerability are tested to ensure that they do not reproduce the vulnerability 2. Randomly generated command line arguments roundtrip when passed to a `.bat` file and then are passed from the `.bat` file to a `.exe`. This fuzz test is as thorough as possible--it tests that things like arbitrary Unicode codepoints and unpaired surrogates roundtrip successfully. Note: In order for the `CreateProcessW` -> `.bat` -> `.exe` roundtripping to succeed, the .exe must split the arguments using the post-2008 C runtime argv splitting implementation, see ziglang#19655 for details on when that change was made in Zig.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On Windows, the command line arguments of a program are a single WTF-16 encoded string and it's up to the program to split it into an array of strings. In C/C++, the entry point of the C runtime takes care of splitting the command line and passing argc/argv to the main function.
#18309 (cc @castholm) updated ArgIteratorWindows to match the behavior of CommandLineToArgvW, but it turns out that CommandLineToArgvW's behavior does not match the behavior of the C runtime post-2008. In 2008, the C runtime argv splitting changed how it handles consecutive double quotes within a quoted argument (it's now considered an escaped quote, e.g.
"foo""bar"post-2008 would get parsed intofoo"bar), and the rules around argv[0] were also changed.This commit makes ArgIteratorWindows match the behavior of the post-2008 C runtime, and adds a standalone test that verifies the behavior matches both the MSVC and MinGW argv splitting exactly in all cases (it checks that randomly generated command line strings get split the same way).
The motivation here is roughly the same as when the same change was made in Rust (rust-lang/rust#87580), that is (paraphrased):
Additionally, the suggested mitigation for BatBadBut (https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/) relies on the post-2008 argv splitting behavior for roundtripping of the arguments given to
cmd.exe. Note: post-2008 argv splitting is not necessary for the suggested mitigation to work, but it is necessary for the suggested escaping to be parsed back into the intended argv by ArgIteratorWindows after being run through a.batfile.Notes: