Use managed NTLM/SPNEGO on Apple platforms by default

[filipnavara]

Apple implementation of NTLM has two major compatibility issues:

• It always splits user name by @ into name and domain. Unfortunately this has different semantics on the wire. It may work in some cases, but it fails when the NTLM is relayed to directory controller with different name, and for long user names. Issue NTLM 401 macOS (Apple sillicon) #82547.
• NTLM signatures are generated incorrectly and can cause buffer overflows. Issue NTAuthentication.MakeSignature produces different thing on macOS, Windows and Linux #65678.

Let's try to use the managed NTLM implementation on Apple platforms now that we can support it with Kerberos at the same time. There's still an opt-out through setting UseManagedNtlm app context switch to false.

Fixes #65678
Fixes #82547

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Apple implementation of NTLM has two major compatibility issues:

• It always splits user name by @ into name and domain. Unfortunately this has different semantics on the wire. It may work in some cases, but it fails when the NTLM is relayed to directory controller with different name, and for long user names. Issue NTLM 401 macOS (Apple sillicon) #82547.
• NTLM signatures are generated incorrectly and can cause buffer overflows. Issue NTAuthentication.MakeSignature produces different thing on macOS, Windows and Linux #65678.

Let's try to use the managed NTLM implementation on Apple platforms now that we can support it with Kerberos at the same time. There's still an opt-out through setting UseManagedNtlm app context switch to false.

Fixes #65678
Fixes #82547

Author:	filipnavara
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[karelz]

@filipnavara we discussed it and we fear that it might cause regressions. What is your confidence?
Given that we are close to fork of 8.0 into RC branch (around 8/14), perhaps we might wait with this for 9.0 (when main becomes 9.0 branch). Would you be ok with it?

[filipnavara]

@filipnavara we discussed it and we fear that it might cause regressions. What is your confidence? Given that we are close to fork of 8.0 into RC branch (around 8/14), perhaps we might wait with this for 9.0 (when main becomes 9.0 branch). Would you be ok with it?

Sure. There's already opt-in for .NET 8 so I am fine with doing this in .NET 9 with a bit more of a leeway.

[wfurt]

we can always pull it into servicing if we get decent verification and there is need.

[filipnavara]

What is your confidence?

There are few things to balance here.

Apple's NTLM implementation has both compatibility issues and known buffer overflows that are relatively easy to trigger. Apple has shown no interest in fixing the buffer overflows. Since they don't happen in the core authentication flow in HTTP and/or SMTP I don't expect people to run into it though. The compatibility issues are difficult to diagnose and several people run into it already. So, as far as NTLM itself is concerned I am quite confident that the managed implementation is better choice.

Unfortunately, to offer consistent experience we have to use managed SPNEGO as well, and that received very little testing so far. I expect us to switch our application to it in the opt-in mode once .NET 8 is released, and we should get enough exposure throughout the .NET 9 timeline.

[wfurt]

LGTM