-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NTLM interop problems with MacOS #887
Comments
triage: please upvote this if you're seeing this issue. |
Hi @davidsh , I am trying to do windows authentication in linux according to this: https://docs.microsoft.com/cs-cz/aspnet/core/security/authentication/windowsauth?view=aspnetcore-3.1&tabs=visual-studio#kestrel What I have observed, .net core 3.1 connects to System.Net.NTAuthentication which calls System.Net.Security.NegotiateStreamPal which calls unix Interop.NetSecurityNative.InitSecContext which invokes somehow the gss method in linux without gss-ntlmssp library it throws error "An unsupported mechanism was requested." because there is no mechanism defined in /etc/gss/mech.d/ folder. when i install gss-ntlmssp library, it throws error: "No credentials were supplied, or the credentials were unavailable or inaccessible" i have tried to compile my own gss-ntlmssp, and found out that it does not even trigger any gss_ methods. i tested this on centos7 vm, ubuntu windows subsytem (18.04) and debian docker image (customized mcr.microsoft.com/dotnet/core/sdk:3.1-buster) - all behave the same please see my full config in here: https://stackoverflow.com/questions/60296237/how-can-i-enable-debug-log-with-microsoft-aspnetcore-authentication-negotiate-pa also note, that klist shows my spn, and i am able to connect to mssql using integrated security what am i missing?? does this work for anybody? |
@scholtz The problem you are describing is not related to this particular issue. This issue is only about NTLM problems (client-side) on MacOS. So, it would be best for you to open a new issue to discuss your authentication issues on Linux. That said, the problem you are describing is a server-side problem with asp.net core. Kestrel server does NOT support NTLM protocol. There is no way to tell it to send back 'Www-Authenticate: NTLM'. It only supports 'Www-Authenticate: Negotiate' protocol. 'Negotiate' protocol is usually Kerberos but can fall back to using NTLM packets (over SPNEGO protocol) in certain cases. NTLM fallback with Negotiate is something that isn't really supported either in Kestrel although I was able to make it work better in .NET 5. See: dotnet/corefx#42522 So, are you really trying to get NTLM working? You mention 'klist shows my spn and mssql works with integrated security'. So, that probably means you want to use Kerberos anyways. The 'gss-ntlmssp' component is not going to help you here since it is only for NTLM and not Kerberos. You might want to open an issue in aspnetcore repo to get help on getting Kestrel working with Kerberos. I also recommend you set the following environment variable on Linux to get GSS-API traces, i.e export KRB5_TRACE=/dev/stdout |
Hi @davidsh Great work with the dotnet/corefx#42522 .. Is there any chance how i can test it in my environment? How can I compile .net5 source code? Would it take too much effort for you to port it to .net core 3 :) ? I believe I am missing some very simple configuration thing to make it work. Could you plase take a look at here, and suggest me how to proceed? The kerberos kinit thing and mssql authentication does not necessary needs to be setup the same as negotiate auth in kestrel. I believe that Kestrel does the auth negotiate, which goes to gssapi, which goes to gssntlmssp lib and somehow it should authenticate to AD. The thing is that the first negotiate request goes well, and i get the response to the user, so it fails at the challange verification somehow because the winbind library does not verify the user token. So the real question is what do you have setup on your linux machine to make it work. According to your advise i have started the issue in aspnetcore: dotnet/aspnetcore#19397 . I do not want to spam this thread, so if you will be so kind, please reply there. Thanks |
@scholtz we do not backport fixes unless there is strong demand for them. You can try .NET 5 daily builds (or public previews coming soon) - see installers and binaries |
Hi @karelz , thanks for that.. I am trying to extend docker image mcr.microsoft.com/dotnet/core/sdk:3.1-buster with .net5, with no success yet. I was able to install runtime using this:
but i am not able to find dependencies for the sdk:
Any ideas where i can find
? or is there any .net5 sdk docker image available? in here https://hub.docker.com/_/microsoft-dotnet-core-sdk/ i dont see any.. thanks |
@scholtz I would expect to see .NET 5 Docker image. At minimum with Preview builds. @richlander @MichaelSimons do you know more about it? (or someone who does) |
dotnet/core Docker images will be published for all preview releases of .NET 5 (e.g. preview 1). There are no .NET 5 images in the dotnet/core repositories yet because preview 1 hasn't been released. Don't worry though we have you covered 😸, you can always get the last-known-good (LKG) builds for the next release of .NET Core by going to the dotnet/core-nightly repositories. @scholtz, take a look at the |
@MichaelSimons Yes, exactly. Thanks for that :) I was able to to create project from tamplate, added the negotiate library, build and run the project however at the end it did not solved my issue :( |
It would be nice if significant defects like this would be back ported to the LTS version of the SDK. |
For others running into this issue, I ran into an identical problem during
|
Is this ever going to get fixed? It's been busted for over a year!!! |
yes, it is broken for long time @InsanelyOne. But the core team is small and there are many issues to address. I have plan for 6.0 and we can hopefully close this gap. It would be great if interested uses can participate in early verification and provide feedback. |
@wfurt What's the current status of your attempts? No rush, just surveying what are our options. I verified that we have problem connecting to HTTP endpoints of MS Exchange servers that have only NTLM and Negotiate authentication enabled (macOS 11.3 on the client). The managed NTLM implementation does work against those servers and all the other ones we have tested so far. One thing that I noticed was that with the managed implementation it was necessary to ensure that the HTTP content of the first negotiation response had to be read and discarded. If that was not done the authentication failed despite looking perfectly correct in Fiddler. We also have our own managed NTLM implementation that we use for non-HTTP protocols and we also tested it successfully against the problematic servers. It implements few more of the NTLM options not present in yours but probably nothing too important. |
The progress is not so good @filipnavara. I'm stuck on other priorities and there was no progress in last few month.
If anybody can help with either one it would increase changes to get it to 6. |
I am willing to invest some time into it. I have a local branch with MD4 implementation (backed by OpenSSL, CommonCrypto and Java API). RC4 is readily available on macOS through CommonCrypto and easy to add. On OpenSSL it's generally available but deprecated from version 3.0, not sure about Android. I can test that. We do have a managed implementation as part of our NTLM code which I can open source under MIT license. However, due to patent situation and long standing rule of not shipping crypto algorithm implementations in .NET I didn't consider that an option. |
Let's start with the macOS and get it functional. Using CommonCrypto or load from system ressl seems fine. |
WIP branch with MD4 and RC4 implementation: https://github.com/filipnavara/runtime/tree/md4 |
Thanks @filipnavara. The md4 is interesting, Im not convinced (yet?) we need rc4. Is there anything else I should look at besides the crypto support? |
RC4 would be needed for session key encryption but it may not be necessary for minimal implementation. You were right that MD4 is missing on Android so that's something that would have to be solved eventually. Only the crypto support is in the branch linked above. I decided to dig a bit deeper to figure out why the macOS native NTLM provider is actually failing. I've some test infrastructure running but I didn't find a way to get much logging information from Windows yet. Notably the values sent by macOS seem quite reasonable in the context of NTLMv2 with all the advanced protections (target information, channel binding, MIC). Neither your implementation, our internal implementation or the one in MailKit supports all these features but the MailKit one has at least few of the missing bits. One thing I could not figure out was why the macOS implementation sends 24 bytes of zeros in the LM challenge and the specification was not much of a help but curiously MailKit had an answer. So far, the only bug in Apple implementation I found was fixed upstream (heimdal/heimdal@622c4de) but remains broken in Apple code. The NTLM version number was incorrectly encoded. |
@wfurt @filipnavara if it's of any use, I converted the C reference implementation of MD4 to C#. You can find it here and works on all .NET platforms: https://github.com/JeroenBer/Ntlm/blob/main/NtlmHttp/MD4.cs |
Yes I know @JeroenBer. However, pushing new crypto code to runtime is different level of scrutiny. It would be easier if we can avoid it and depend on the platform. |
@wfurt 👍👍 |
@wfurt Do you by any chance have an insight how to get more info about NTLM failures on the Windows server side? I do get an event in the Event Viewer that is quite short on the information on why it failed. Event tracing didn't seem to produce anything useful either. I probably don't have an access to checked versions of Windows builds but I'd be fine installing it if there was a chance of getting more information. Otherwise I'll probably try to implement flag by flag what macOS is sending and try to see when it breaks. That may take a while though since it means implementing all the target information logic and MIC. |
I can ask around. Is there specific scenarios you are trying to solve? |
I am just generally peeking to see why the macOS NTLM implementation (ie. heavily modified Heimdal code) doesn't work. So far the only bug in the implementation I found is the incorrect OS version encoding which is documented as debug-only and to be ignored by the implementations. The event on server lists the authentication info properly but fails with pretty generic error 0x80090308 ( While I still think that it's worth pursuing the managed NTLM implementation for multiple reasons I also believe it would be useful to understand why the built-in implementation actually fails to authenticate. I have Wireshark and Fiddler logs with the dissected authentication packets but there's nothing obviously wrong. |
Progress report: I found a couple of minor issues in https://github.com/wfurt/Ntlm:
I've implemented various different features:
Now the exchange looked quite similar to what macOS was sending. The last difference was the missing MIC and If it turned out to be only some incompatibility with the way MIC is calculated it would likely be possible to make a very targeted fix. It's possible to use the GSSAPI to generate all the messages. Then we can ask for the session key (GSS_NTLM_GET_SESSION_KEY_X) and rewrite the last message and its MIC to our liking. At this point this is purely a speculation though. |
So, funny story, there's a one line fix: --- a/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs
+++ b/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs
@@ -119,7 +119,7 @@ private static async Task<HttpResponseMessage> SendWithNtAuthAsync(HttpRequestMe
}
ChannelBinding? channelBinding = connection.TransportContext?.GetChannelBinding(ChannelBindingKind.Endpoint);
- NTAuthentication authContext = new NTAuthentication(isServer: false, challenge.SchemeName, challenge.Credential, spn, ContextFlagsPal.Connection, channelBinding);
+ NTAuthentication authContext = new NTAuthentication(isServer: false, challenge.SchemeName, challenge.Credential, spn, ContextFlagsPal.Connection | ContextFlagsPal.InitIntegrity, channelBinding);
string? challengeData = challenge.ChallengeData;
try
{ This causes the
It generates the MIC and key exchange fields but not all the flags necessary for the key exchange. |
could you push the changes/fixes you have so far back to the managed implementation @filipnavara (in case we will use it later) |
@wfurt I'll clean them up and submit a PR to your repository. It may just take me a few days to do it while I deal with some higher priority issues at work. |
I'm glad to see the problem could be fixed for .NET 6! Thanks a lot @filipnavara |
not at the moment @Michriko. It would need to meet servicing bar and get approval from servicing committee. |
The bar for servicing would be:
I didn't see signs of either on this issue yet ... |
I was seriously considering to submit a backport to Mono (which is stuck on a post-.NET Core 2.1 snapshot of CoreFX) because it's blocking us on Xamarin workloads (macOS at the moment, possibly iOS and Android depending on our release timing). JFYI as a data point. At the moment we have very crude fallback to (We specifically don't need a backport to .NET Core 3.1 or .NET 5.) |
.NET Core NTLM support is broken on MacOS. It has probably been broken for a while. But due to lack of test coverage we didn't notice. I did my testing on MacOS X 10.15.2 (Catalina).
The root cause is complex and due to possible interop issues with the NTLM support in the Heimdal library that ships with the Mac. Prior to MacOS 10.7, the Mac used MIT Kerberos open-source libraries. In 10.7, it switched to Heimdal open-source libraries. You can see the source code of the various open-source libraries used in the Mac here.
We have had some reports by customers of problems with Mac .NET Core clients using NTLM. At first, we thought that some NTLM support package was missing on the Mac. On Linux, we require the installation of the 'gss-ntlmssp' package which by default is not present on Linux distros nor on our .NET Core docker images. But the Mac already has NTLM support installed in the GSS-API mechanisms.
We have some simple NTLM tests in our repo:
runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs
Lines 588 to 593 in 57df202
And these were running and passing on the Mac. However, the tests only validate that an initial NTLM base64 token is generated and sent to the server. Those tests don't do end-to-end validation of a username/password.
We have other tests for HttpClient that do end-to-end validation:
runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs
Lines 568 to 570 in 57df202
But those are not turned on right now. They require environment variables to be enabled to point to a remote NTLM server. After turning on those tests on my local Mac, I discovered that NTLM is broken for Mac .NET Core.
More NTLM (and Kerberos) tests will be enabled for CI as part of follow-up PRs to #463
If I use Safari on the Mac, it works fine against the NTLM test server. But the Safari app on the Mac uses its own NTLM library and not the GSS-API Heimdal library that .NET Core uses.
Looking at several WireShark traces of working and non-working scenarios shows differences in various NTLM protocol fields. Although the Mac is sending NTLM packets to the server, the end result is that the server (Windows in this case) denies the authentication. We'll need to work with the Windows protocol team to help enable advanced diagnostics to see why the Windows server denied the authentication.
I will continue looking into the problem by trying out different NTLM servers to test against (such as Linux Apache) to get more information on working versus non-working scenarios. My current hypothesis is that there is a problem with Heimdal's NTLM v2 support. The NTLM protocol has many differences between NTLMv1 versus NTLMv2.
The text was updated successfully, but these errors were encountered: