Proxy authentication not recognized in macOS

[olivepedro]

Description

I'm using HttpClient to open an URI, like this:
HttpResponseMessage response = httpClient.GetAsync("https://www.microsoft.com/").Result;

I setup a proxy using environment variables http_proxy/https_proxy=http://user:pass@server:port
In Windows and Ubuntu, I can successful connect to the proxy. In macOS, I get the error:

Reproduction Steps

Sample app:

Console.WriteLine("Starting");

HttpClientHandler handler = new();

HttpClient httpClient = new(handler);

try
{
    HttpResponseMessage response = httpClient.GetAsync("https://www.microsoft.com/").Result;

    response.EnsureSuccessStatusCode();
}
catch (Exception e)
{
    Console.WriteLine("\nException Caught!");
    Console.WriteLine($"Message :{e.Message}");
    Console.WriteLine($"Inner :{e.InnerException}");
}
finally
{
    handler.Dispose();
    httpClient.Dispose();
}

Console.WriteLine("Ending");

Set environment variables:

export http_proxy=http://user:pass@server:port
export https_proxy=http://user:pass@server:port

Expected behavior

Connection is successful and we get the console output:

Actual behavior

an exception is thrown:

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm using HttpClient to open an URI, like this:
HttpResponseMessage response = httpClient.GetAsync("https://www.microsoft.com/").Result;

I setup a proxy using environment variables http_proxy/https_proxy=http://user:pass@server:port
In Windows and Ubuntu, I can successful connect to the proxy. In macOS, I get the error:

Reproduction Steps

Sample app:

Console.WriteLine("Starting");

HttpClientHandler handler = new();

HttpClient httpClient = new(handler);

try
{
    HttpResponseMessage response = httpClient.GetAsync("https://www.microsoft.com/").Result;

    response.EnsureSuccessStatusCode();
}
catch (Exception e)
{
    Console.WriteLine("\nException Caught!");
    Console.WriteLine($"Message :{e.Message}");
    Console.WriteLine($"Inner :{e.InnerException}");
}
finally
{
    handler.Dispose();
    httpClient.Dispose();
}

Console.WriteLine("Ending");

Set environment variables:

export http_proxy=http://user:pass@server:port
export https_proxy=http://user:pass@server:port

Expected behavior

Connection is successful and we get the console output:

Actual behavior

an exception is thrown:

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	olivepedro
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[wfurt]

Can you post packet captures @olivepedro? And would it work if you set the proxy explicitly using https://learn.microsoft.com/en-us/dotnet/api/system.net.webproxy?view=net-7.0?

That code is same as on Linux so I don't see reason why this should behave any different.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[grazius]

Hello
We finally found the root cause of our problem.
It's not related to the Mac implementation but to the authentication through our http proxy when we call https urls.

To find the root cause we switch in java to make the same test and we also get a 407 return code.
To resolve this in java we referrer to this url https://www.oracle.com/java/technologies/javase/8u111-relnotes.html

Disable Basic authentication for HTTPS tunneling

In some environments, certain authentication schemes may be undesirable when proxying HTTPS. Accordingly, the Basic authentication scheme has been deactivated, by default, in the Oracle Java Runtime, by adding Basic to the jdk.http.auth.tunneling.disabledSchemes networking property. Now, proxies requiring Basic authentication when setting up a tunnel for HTTPS will no longer succeed by default. If required, this authentication scheme can be reactivated by removing Basic from the jdk.http.auth.tunneling.disabledSchemes networking property, or by setting a system property of the same name to "" ( empty ) on the command line.

Additionally, the jdk.http.auth.tunneling.disabledSchemes and jdk.http.auth.proxying.disabledSchemes networking properties, and system properties of the same name, can be used to disable other authentication schemes that may be active when setting up a tunnel for HTTPS, or proxying plain HTTP, respectively.

Have we got the same restriction in dotnet ?
An environment variable exists to allow this use case ?

Best regards

[wfurt]

Do you know what scheme is used in successful case? You can create CredentialCache and using it instead of credentials. But that does not work with the environment - you would need to set WebProxy explicitly and pass the cache to is as credentials.

[grazius]

using System.Net;
using System.Collections;

String targetUrl = "http://dev.azure.com";
String proxyUrl = "http://myproxyserver:3128";
String proxyUser = "user";
String proxyPassword = "pass!";

// Console.WriteLine("GetEnvironmentVariables: ");
// foreach (DictionaryEntry de in Environment.GetEnvironmentVariables())
//     Console.WriteLine("  {0} = {1}", de.Key, de.Value);

// Non System proxy # DOES NOT WORK
// --------------------------
var creds = new NetworkCredential(userName: proxyUser, password: proxyPassword);
var proxy = new WebProxy
{
    Address = new Uri(proxyUrl),
    Credentials = creds
};
var httpClientHandler = new HttpClientHandler
{
    UseProxy = true,
    Proxy = proxy
};

// System proxy # WORK
// --------------------------
// var httpClientHandler = new HttpClientHandler
// {
//     UseProxy = true,
//     Proxy = null, // use system proxy
//     DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
// };

// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
// client.Timeout = TimeSpan.FromSeconds(5);

if(httpClientHandler.Proxy != null){
    Console.WriteLine("Custom Proxy");
    Console.WriteLine("targetUrl: " + targetUrl);
    Console.WriteLine("type: " + httpClientHandler.Proxy.GetType());
    Console.WriteLine("proxy: " + httpClientHandler.Proxy.GetProxy(new Uri(targetUrl)));
} else {
    Console.WriteLine("Default Proxy");
    Console.WriteLine("targetUrl: " + targetUrl);
    Console.WriteLine("type: " + HttpClient.DefaultProxy.GetType());
    Console.WriteLine("proxy: " + HttpClient.DefaultProxy.GetProxy(new Uri(targetUrl)));
}

var response = await client.GetAsync(targetUrl);
Console.WriteLine(response.StatusCode);

The code used for testing
Does not work when we try to not use System Proxy to call http://dev.azure.com or https://dev.azure.com
Is there something we do wrong on the Credential binding ?

[wfurt]

right. so instead of NetworkCredential use CredentialCache. That allows you to do credentials only for given authentication scheme. That assumes you would know what that is. But that should be easy since you have working setup.

[grazius]

Now that works also with the Non System Proxy, thanks !

But finally we try to use the proxy implementation provided by Environment variables by switching on the System proxy and we make :

export HTTPS_PROXY='http://user:password@proxyurl:3128'
dotnet run
##################
Default Proxy
targetUrl: https://dev.azure.com
type: System.Net.Http.HttpEnvironmentProxy
proxy: http://user:password@proxyurl:3128/
Unhandled exception. System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://user:password@proxyurl:3128/' failed with status code '407'."
   at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in /Users/a24309/Workspace/git/fork-resilience/dotnet-console/Program.cs:line 56
   at Program.<Main>(String[] args)

[grazius]

what i see in source code
https://github.com/dotnet/runtime/blob/8a5ae8bbd0e111a13446187fe64ca7b2f68285a9/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpEnvironmentProxy.cs#LL91C13-L91C13

It's a NetworkCredential that is used !

[wfurt]

I don't understand the last comment @grazius. The environment syntax does not allow to specify authentication scheme. So either the proxy needs to clearly indicate what it supports or it is going to fail. It is difficult to make guesses without actually seeing the exchange.

[grazius]

Hello

I don't understand the last comment @grazius.

It's just to say that the HttpEnvironmentProxy code use a NetworkCredential where in my "Non System proxy" implementation i need to use a CredentialCache... So i'm not surprised that the return code is 407 when we try to use environment variable to specify the proxy.

So either the proxy needs to clearly indicate what it supports or it is going to fail.

# define proxy without creds
export HTTPS_PROXY='http://my-corporate-proxy:3128'
# make a simple curl (over the proxy)
curl -vvv https://www.google.com                                       

* Uses proxy env variable HTTPS_PROXY == 'http://my-corporate-proxy:3128'
*   Trying 1.1.1.1:3128...
* Connected to my-corporate-proxy (1.1.1.1) port 3128 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.87.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 407 authenticationrequired
< Via: x.x.x.x (McAfee Web Gateway version)
< Date: Wed, 07 Jun 2023 07:47:27 GMT
< Content-Type: text/html
< Cache-Control: no-cache
< Content-Length: 3785
< X-Frame-Options: deny
< Proxy-Connection: Keep-Alive
< Proxy-Authenticate: Negotiate
< Proxy-Authenticate: Basic realm="Directory identifier (LDAP)"
< 
* Ignore 3785 bytes of response-body
* CONNECT tunnel failed, response 407
* Closing connection 0
curl: (56) CONNECT tunnel failed, response 407

What our corporate proxy does not indicate clearly that make the authentication fail with proxy define as environment variable ?

Best regards

[wfurt]

The Proxy-Authenticate: Negotiate should be sufficient and it should take precedence over Basic. It is not clear to me why Basic would be advertised when not functional but that is irrelevant. When you connect do you see Negotiate scheme in follow-up request or do you Basic or would it just stop there?

And what .NET version do you use? #887 was fixed in 6.0 so it should work if you use recent versions.

[grazius]

With credentials provided in the environment variable
Curl seem's to use Basic

export HTTPS_PROXY='http://user-proxy:password-proxy@host-proxy:3128'
curl -vvvvv https://www.google.com

* Uses proxy env variable HTTPS_PROXY == 'http://user-proxy:password-proxy@host-proxy:3128'
*   Trying 1.1.1.1:3128...
* Connected to host-proxy (1.1.1.1) port 3128 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to www.google.com:443
* Proxy auth using Basic with user 'user-proxy'
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> Proxy-Authorization: Basic XXXXXXXXXXXXXXXXX==
> User-Agent: curl/7.87.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.0 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Unknown (8):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=www.google.com
*  start date: May 25 21:56:32 2023 GMT
*  expire date: Jun 24 21:56:32 2023 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: O=MITM Root Interception Externe; OU=MITM IPS Interception Externe; CN=Service_Browsing
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: www.google.com]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x14a810a00)
> GET / HTTP/2
> Host: www.google.com
> user-agent: curl/7.87.0
> accept: */*
> 
< HTTP/2 200 
...

I use net7.0 locally

[wfurt]

and HttpClient?

[grazius]

Yes regarding the code previously provided

var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

[wfurt]

The question was not about code but what does it sends on wire....

[grazius]

Hello
i make tcpdump to understand what is done.

-------------
REQ 1
-------------

Hypertext Transfer Protocol
    CONNECT dev.azure.com:443 HTTP/1.1\r\n
    Host: dev.azure.com:443\r\n
    \r\n
    [Full request URI: dev.azure.com:443]
    [HTTP request 1/2]
    [Response in frame: 183]
    [Next request in frame: 206]

Hypertext Transfer Protocol
    HTTP/1.1 407 authenticationrequired\r\n
    Via: 1.1 1.1.1.1 (McAfee Web Gateway xxxx)\r\n
    Date: Thu, 08 Jun 2023 11:06:55 GMT\r\n
    Content-Type: text/html\r\n
    Cache-Control: no-cache\r\n
    Content-Length: 3783\r\n
    X-Frame-Options: deny\r\n
    Proxy-Connection: Keep-Alive\r\n
    Proxy-Authenticate: Negotiate\r\n
    Proxy-Authenticate: Basic realm="Directory identifier (LDAP)"\r\n
    \r\n
    [HTTP response 1/2]
    [Time since request: 0.005718000 seconds]
    [Request in frame: 179]
    [Next request in frame: 206]
    [Next response in frame: 209]
    [Request URI: dev.azure.com:443]
    File Data: 3783 bytes

-------------
REQ 2
-------------

Hypertext Transfer Protocol
    CONNECT dev.azure.com:443 HTTP/1.1\r\n
    Host: dev.azure.com:443\r\n
    Proxy-Authorization: Negotiate XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=\r\n
    \r\n
    [Full request URI: dev.azure.com:443]
    [HTTP request 2/2]
    [Prev request in frame: 179]
    [Response in frame: 209]

Hypertext Transfer Protocol
    HTTP/1.1 407 authenticationrequired\r\n
    Via: 1.1 1.1.1.1 (McAfee Web Gateway xxxx)\r\n
    Date: Thu, 08 Jun 2023 11:06:55 GMT\r\n
    Content-Type: text/html\r\n
    Cache-Control: no-cache\r\n
    Content-Length: 3783\r\n
    X-Frame-Options: deny\r\n
    Proxy-Connection: Keep-Alive\r\n
    Proxy-Authenticate: Negotiate\r\n
    Proxy-Authenticate: Basic realm="Directory identifier (LDAP)"\r\n
    \r\n
    [HTTP response 2/2]
    [Time since request: 0.005641000 seconds]
    [Prev request in frame: 179]
    [Prev response in frame: 183]
    [Request in frame: 206]
    [Request URI: dev.azure.com:443]
    File Data: 3783 bytes

So dotnet try to use the Negociate only

I also added try catch

Exception Caught!
Message :The proxy tunnel request to proxy 'http://user:pass@proxy:3128/' failed with status code '407'."
Inner :

[grazius]

To complete, on our Mac Workstation we've got Kerberos to interact with our Active Directory.

When using HttpEnvironmentProxy:

• They're is a way to force the authentication scheme to Basic ?
• Or to disable the NTLM support (in order to ignore Negociate) ?

Best regards

[wfurt]

The HttpClient will try to find strongest authentication scheme and use it. If that fails, it would not try different scheme with same credentials. Aside from the CredentialCache where the schemes are explicit there is no good way how to change it. If server does not offer Negotiate, HttpClient will use Basic.

Note that Negotiate does not necessarily mean Kerberos. It can still be simple NTLM and there may be some variations on domain name and form.

There was some discussion and insight in #82547

[grazius]

Our corporate proxy is used for internet access so i think it's normal that it present the Negociate Auth scheme before the Basic.

But here i use a dotnet application (this application is ArtifactTool developed internally by Microsoft).
This application is provided through the azure-cli (developped in Python) on we have no problem to use Basic auth.

The only way today i found to make this tool working is to had a mitmproxy without authentication that call our corporate proxy with the Basic auth ... But it's really a workaround.

What i don't understand here is our proxy work with java, python, a simple curl ... but not with dotnet core :(

So they're is a no way to force the authentication scheme to Basic ?
Or disable the NTLM support (in order to ignore Negociate) ?

Best regards

[wfurt]

There is way but you need to do it explicitly and not via environment.

[grazius]

No evolution can be planned to allow this case ?

[wfurt]

I feel this is corner case - this did not come up so far e.g. this is first instance. If anything you should figure out what the Negotiate does not work. Did you look at the issue I linked?

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[ghost]

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.