Error code 139 when attempting Microsoft.Data.SqlClient connection on Linux with net6.0

[rytido]

I get error code 139 when attempting Microsoft.Data.SqlClient connection on Linux with net6.0 (6.0.100). I see the error when using dotnet watch run, otherwise it exits silently. The same code works on net5.0.

watch : Exited with error code 139

To reproduce

app.fsproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Program.fs" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Data.SqlClient" Version="3.0.1" />
  </ItemGroup>

</Project>

Program.fs

open System
open Microsoft.Data.SqlClient

let server = Environment.GetEnvironmentVariable("SERVER")
let connectionString = $"Server={server};Trusted_Connection=True;"

let cnxn = new SqlConnection(connectionString)

cnxn.Open() // program silently exits here (or with code 139 if using watch)
printf "connection is open"
cnxn.Close()

[JRahnama]

@rytido thanks for bringing this up. Using UID and PWD will solve the issue.

open System
open Microsoft.Data.SqlClient

let connectionString = "Data Source=127.0.0.1;UID=<user name>;PWD=<password>;"
let cnxn = new SqlConnection(connectionString)
exception Error1 of string
try
cnxn.Open()
Console.WriteLine(cnxn.State)
cnxn.Close()
Console.WriteLine(cnxn.State)
with
| Error1(str) -> printfn "Error1 %s" str

[rytido]

Did Kerberos auth support get dropped? I cannot necessarily use sql server auth. This worked in net5.0.

[JRahnama]

@rytido I will test with Kerberos and will update you soon.

[JRahnama]

something to mention here about a breaking change on net 6 and kerberos . Port has been removed from SPN for Kerberos and Negotiate. I made a C# application with net 6 and kerberos and got the same results as yours. F# acted the same. When I used user name and password everything worked fine.

[rytido]

That's not the issue for two reasons:

1. If I revert to netcoreapp3.1 my app works, and that doc indicates 3.x behavior should be the same as 6.
2. I set the env var DOTNET_SYSTEM_NET_HTTP_USEPORTINSPN=1, and just to be certain also set the context switch AppContext.SetSwitch("System.Net.Http.UsePortInSpn", true). The net6 version of this still crashes.

[stcpottdav]

I am having the same issue. After running a test, I used dmesg to show where the segmentation fault happened:
dotnet[659]: segfault at 0 ip 00007fdc7e8a63bc sp 00007ffe5b10db70 error 4 in libSystem.Net.Security.Native.so[7fdc7e8a6000+1000]

[JRahnama]

@roji have you seen similar issue with net 6 and Kerberos? I have tested with Microsoft.Data.SqlClient and System.Data.SqlClient using Kerberos authentication. Tested with C# and F# no exception is thrown. It just does not work, no results come back and it gets terminated. The documentation suggests adding a switch will fix the issue. I tested with the switch and environment variable, no luck so far. Just wanted to check with you.

Thank you.

[rytido]

May also be worth noting that in an AspNetCore app this doesn't just make a single request fail, it silently kills the whole service.

[wfurt]

I think this is dup of dotnet/runtime#60906. The linked breaking change only impact HttpClient in cases it tries to generate SPN on it's own.

[wfurt]

BTW it may be best (fastest) to update SqlClient to react to dotnet/runtime#55037.

[JRahnama]

thanks @wfurt. Although the issue seems external according to the link you posted. I'll do more debugging to see why no exception is thrown

[wfurt]

for runtime, we call EnsureGssInitialized via static constructor. That will load all the function pointers. AFAIK SqlClient does not do it so it would crash (dereferencing NULL in gssapi code) when trying to use the functions. If you call NetSecurityNative_EnsureGssInitialized once, the crash should go away. Let me know if that make sense @JRahnama.

[JRahnama]

@wfurt thanks for the update. I added the EnsureGssInitialized to SqlClient and it worked for net 6 and failed for net 5. Just to confirm this is expected correct? I mean that part of the code is only for net 6 and after if I am not mistaken.

[wfurt]

yes, this was done late (unfortunately) in 6.0. So either wrap it In try/catch or even better

if (Environment.Version.Major >= 6)
{
    EnsureGssInitialized();
}

Long run, I would like to add enough public surface so SqlClient (and others) do not need to depend on private functions.
We missed 6.0.1 maintenance and next one will be early next year. If you can do anything faster it would be great @JRahnama.
If there is new package available, please ping both threads so people can try it.

[JRahnama]

Unfortunately we are releasing GA version 4.0 today (more than half of the process is done) and most of the documents are prepared without this fix included, but we can arrange a hotfix release for near future. It needs our PM, @David-Engel , approval.

[wfurt]

ok. That would still probably be much faster than runtime.
Adding @karelz and @danmoseley for visibility.

[thestonehead]

If someone didn't think of this, I cloned the repo and added

        [DllImport(Interop.Libraries.NetSecurityNative, EntryPoint = "NetSecurityNative_EnsureGssInitialized")]
        internal static extern int EnsureGssInitialized();


        static NetSecurityNative()
        {
            EnsureGssInitialized();
        }

to src/Microsoft.Data.SqlClient/netcore/src/Common/src/Interop/Unix/System.Net.Security.Native/Interop.NetSecurityNative.cs

and packaged homemade nuget and put it in our local repository to use until official fix is available.

[danmoseley]

As noted above, we just missed 6.0.1, and with the holiday period, 6.0.2 is not expected to ship until the 2nd week of Feb.

[JRahnama]

We are planning on a hotfix from our side sometime soon. Time will be announced soon.

[JRahnama]

@rytido, @thestonehead can you try with this build and let me know if that works for you.

[aletsche]

We face the same issue when using the container image "ubi8/dotnet-60" from Red Hat and trying to connect to a MSSQL database using Kerberos with the code shown below. Using "ubi8/dotnet-50" everthing works as expected.

sh-4.4$ cat Program.cs
using System;
using System.Data.SqlClient;
using System.Text;

namespace sqltest
{
    class Program
    {
        static void Main(string[] args)
        {
            try 
            {
                while (true) 
                {
                    String connectionString = Environment.GetEnvironmentVariable("DATABASE_URL");
                    String sql = Environment.GetEnvironmentVariable("DATABASE_QUERY");
                    
                    using (SqlConnection connection = new System.Data.SqlClient.SqlConnection(connectionString))
                    {
                        Console.WriteLine("\nQuery data example:");
                        Console.WriteLine("=========================================\n");

                        connection.Open();       

                        using (SqlCommand command = new SqlCommand(sql, connection))
                        {
                            using (SqlDataReader reader = command.ExecuteReader())
                            {
                                while (reader.Read())
                                {
                                    for (int i = 0; i < reader.FieldCount; i++)
                                    {
                                        if (i > 0) { Console.Write(", "); }
                                        Console.Write(reader.GetValue(i));
                                        
                                    }
                                    Console.WriteLine();
                                }
                            }
                        }                    

                        connection.Close();
                    }
                    System.Threading.Thread.Sleep(30000); 
                }
            }
            catch (SqlException e)
            {
                Console.WriteLine(e.ToString());
            }
        }
    }
}
sh-4.4$

[JRahnama]

@aletsche the issue exists on Microsoft.Data.SqlClient and System.Data.SqlClient when using net 6. The proposed fixed is for Microsoft.Data.SqlClient. Is it possible for you to test with MDS and let us know if it works for you.

[aletsche]

@JRahnama unfortunately I don't have any possibility to test with MDS personally. Will ask a colleague tomorrow .. he might have the necessary environment to test

[danmoseley]

BTW, this would be for 6.0.4 to release in April. Meantime perhaps someone would like to explicitly paste in the code necessary to workaround, that was mentioned above, for the ease of others.

[JRahnama]

@danmoseley we are working on a PR to back port the fix to SDS.

[JRahnama]

I am reopening this issue to keep a track on all other interconnected issues.

[ladeak]

I have run into the same issue. I have tested the suggested nuget package: Microsoft.Data.SqlClient version 4.1.0, and in .net6 + RHEL 8 I don't get segfault anymore with EntityFrameworkCore.

However, using EntityFramework (the non-core version) version 6.4.4, I still face the issue. The workaround by @clbiggs seems to work there.

Should there be a bugfix in System.Data.SqlClient package too? (I believe that is being used by the classic EntityFramework)

[ErikEJ]

@ladeak as an alternative you can use this package with MDS 4

https://www.nuget.org/packages/ErikEJ.EntityFramework.SqlServer/

[cli00004]

Hi Just wondering if this issue is fixed? I'm still experiencing the same issue with latest Microsoft.Data.SqlClient 4.1.0, also tried with System.Data.SqlClient 4.8.3

[duyn9uyen]

I've been struggling trying to upgrade our .NET 5 API to .NET 6 with for days! We host our API on a Linux containers in Kubernetes. The container kept crashing and I saw that it threw a Segmentation fault (core dumped). error. Explicitly referencing Microsoft.Data.SqlClient 4.1.0 worked for me!

[JRahnama]

@duyn9uyen the problem was addressed in v4.1.

[Palsternakka]

I'm having the same issue with a .NET Core app on Xubuntu - as soon as a connection is attempted the app throws a segmentation fault (connection string Data Source=10.1.3.7;Initial Catalog=CreditsafeProcessing;Integrated Security=true)

Project is using Microsoft.EntityFrameworkCore.SqlServer version 6.0.7, run using .NET Core version 6.0.302

I've run kinit successfully so based on all my reading it should work? Further details here

[JRahnama]

@Palsternakka Microsoft.EntityFrameworkCore.SqlServe is using M.D.SqlClient v2.1.4. Can you overwrite the version in the csproj file by adding packageReference to version 4.1? The issue was addressed first in v4.0.1 and is not back ported to previous versions yet.

<ItemGroup>
      <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
</ItemGroup>

[hameedshk]

After lot of debugging , below code fix works for me in .net 6.0 framework.

Environment - openshift red hat OS and slq server authentication in Kerberos mode

.csproj change
<PackageReference Include="System.Data.SqlClient" Version="4.8.2" />

Program.cs change

public class Program
    {
         [DllImport("System.Net.Security.Native", EntryPoint = "NetSecurityNative_EnsureGssInitialized")]
        internal static extern int EnsureGssInitialized();
        /// <summary>
        /// Main method
        /// </summary>
        /// <param name="args"></param>
        public static void Main(string[] args)
        {
          if (OperatingSystem.IsLinux())
                {
                    System.Console.WriteLine("Ensuring Gss is Initialized ({ApplicationContext})...");
                    EnsureGssInitialized();
                }
            CreateHostBuilder(args).Build().Run();
        }

Startup.cs change

public void ConfigureServices(IServiceCollection services)
{
  services.AddDataProtection()
 .PersistKeysToFileSystem(new DirectoryInfo(@"/opt/app-root/.aspnet/DataProtection-Keys"));       
    

[Palsternakka]

@Palsternakka Microsoft.EntityFrameworkCore.SqlServe is using M.D.SqlClient v2.1.4. Can you overwrite the version in the csproj file by adding packageReference to version 4.1? The issue was addressed first in v4.0.1 and is not back ported to previous versions yet.

<ItemGroup>
      <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
</ItemGroup>

This worked for me, thank you!

[hameedshk]

@Palsternakka Microsoft.EntityFrameworkCore.SqlServe is using M.D.SqlClient v2.1.4. Can you overwrite the version in the csproj file by adding packageReference to version 4.1? The issue was addressed first in v4.0.1 and is not back ported to previous versions yet.

<ItemGroup>
      <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
</ItemGroup>

This worked for me, thank you!

I did try this as I had seen this fix in one github .net 6.0 know issues and this was the one of the first thing that I tried and unfortunately this didn’t work for me and not only 4.1.0 ,I have tried next and its previous as well from nuget official website.

[IOnlyFetchBranches]

Just wanna mention

@Palsternakka Microsoft.EntityFrameworkCore.SqlServe is using M.D.SqlClient v2.1.4. Can you overwrite the version in the csproj file by adding packageReference to version 4.1? The issue was addressed first in v4.0.1 and is not back ported to previous versions yet.

<ItemGroup>
      <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
</ItemGroup>

This worked for me, thank you!

This worked for us, running the dotnet6.0 runtime RHEL images. It is unfortunate this is not fixed though, any word on this from the devs?

[JRahnama]

@IOnlyFetchBranches the fix has been back ported to 2.1 version. You can overwrite the version in the csproj to 2.1.5 and probably that will fix the problem for you. You can see the release notes here.
For System.Data.SqlClient the fix has been ported and will be available on the next release.

[carlossanlop]

All - FYI the fix introduced by this PR has been published to the System.Data.SqlClient package in nuget: https://www.nuget.org/packages/System.Data.SqlClient/4.8.4

[JRahnama]

Closing the issue as the fix has been ported to System.Data.SqlClient v4.8.4 latest release.