Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value #25074

frankbuckley · 2021-06-10T13:09:12Z

Upgrading to Microsoft.Data.SqlClient 3.0.0 results in InvalidCastException ("Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'") - that does not occur with Microsoft.Data.SqlClient 2.1.3 - when async enumerating over the results of a query that includes null rowversion values and when sqlOptions.EnableRetryOnFailure().

I thought this might be something to do with dotnet/SqlClient#998. However, enabling the LegacyRowVersionNullBehaviour switch does not fix the problem.

In trying to narrow down a repro, it became clear the error only occurs if sqlOptions.EnableRetryOnFailure() is called when configuring the context. This, plus the fact that non-async enumeration of the same query works seems to suggest problem in EfCore.

Versions

Observed with:

5.0.7
6.0.0-preview.4.21253.1

Repro:

Repro project at: https://github.com/frankbuckley/efcore-sqldata3

Database:

drop table if exists dbo.Price;
go

drop table if exists dbo.Occurrence;
go

create table dbo.Occurrence
(
    Id        int          not null identity,
    Title     nvarchar(80) not null,
    Timestamp rowversion   not null,
    constraint pk_Occurrence
        primary key clustered (Id)
);
create table dbo.Price
(
    OccurrenceId int        not null,
    Currency     char(3)    not null,
    Value        decimal    not null,
    Timestamp    rowversion not null,
    constraint pk_Price
        primary key clustered (OccurrenceId, Currency),
    constraint fk_Price_Occurrence
        foreign key (OccurrenceId)
        references dbo.Occurrence (Id)
);
go

Program:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;

namespace EfCoreMsSqlData3
{
    internal class Program
    {
        private static async Task Main(string[] args)
        {
            // Makes no difference

            // AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.LegacyRowVersionNullBehaviour", true);

            using (EventsDbContext db = new())
            {
                if ((await db.Occurrences.CountAsync()) == 0)
                {
                    // Note: no prices, therefore LEFT JOIN when included in query of occurrences will return nulls

                    for (int i = 0; i < 10; i++)
                    {
                        db.Occurrences.Add(new Occurrence { Title = "Test " + i });
                    }

                    await db.SaveChangesAsync();
                }
            }

            // This works

            using (EventsDbContext db = new())
            {
                foreach (Occurrence? o in db.Occurrences.Include(o => o.Prices))
                {
                    Console.WriteLine(o.Title + " (" + o.Timestamp + ")");
                }
            }

            // This fails

            using (EventsDbContext db = new())
            {
                await foreach (Occurrence? o in db.Occurrences.Include(o => o.Prices).AsAsyncEnumerable())
                {
                    Console.WriteLine(o.Title + " (" + o.Timestamp + ")");
                }
            }
        }
    }

    public class EventsDbContext : DbContext
    {
        private const string Connection = "Data Source=(local);Initial Catalog=EfCoreMsSqlData3;" +
            "Integrated Security=True;Connect Timeout=60;Encrypt=False;TrustServerCertificate=False;" +
            "ApplicationIntent=ReadWrite;MultiSubnetFailover=False";

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder
                .EnableDetailedErrors()
                .EnableSensitiveDataLogging()
                .UseSqlServer(Connection, options =>
                {
                    // Remove this and it works...

                    options.EnableRetryOnFailure();
                })
                .LogTo(Console.WriteLine);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Occurrence>()
                .ToTable("Occurrence")
                .HasKey(o => o.Id);

            modelBuilder.Entity<Occurrence>()
                .Property(o => o.Timestamp)
                .IsRowVersion();

            modelBuilder.Entity<Occurrence>()
                .HasMany(o => o.Prices)
                .WithOne(o => o.Occurrence)
                .HasForeignKey(p => p.OccurrenceId);

            modelBuilder.Entity<Price>()
                .ToTable("Price")
                .HasKey(p => new { p.OccurrenceId, p.Currency });

            modelBuilder.Entity<Price>()
                .Property(o => o.Timestamp)
                .IsRowVersion();
        }

        public DbSet<Occurrence> Occurrences { get; set; }
    }


    public abstract class PersistedObject
    {
        public byte[] Timestamp { get; set; }
    }

    public abstract class Entity<TId> : PersistedObject
        where TId : IEquatable<TId>
    {
        public TId Id { get; set; }
    }

    public class Occurrence : Entity<int>
    {
        public string Title { get; set; }

        public List<Price> Prices { get; set; }
    }

    public class Price : PersistedObject
    {
        public int OccurrenceId { get; set; }

        public string Currency { get; set; }

        public Occurrence Occurrence { get; set; }

        public decimal Value { get; set; }
    }
}

Stacktrace:

System.InvalidOperationException: An error occurred while reading a database value for property 'Price.Timestamp'. The expected type was 'System.Byte[]' but the actual value was null.
       ---> System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'.
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueFromSqlBufferInternal[T](SqlBuffer data, _SqlMetaData metaData)
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueInternal[T](Int32 i)
         at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValue[T](Int32 i)
         at lambda_method58(Closure , DbDataReader , Int32[] )
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
         --- End of inner exception stack trace ---
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadRow()
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Environment

Originally discovered in integration tests running on Ubuntu 20.04 with SDK 5.0.301 and Azure SQL Database.

Repro tested on Windows with local SQL Server 15.0.2080.9:

dotnet --info

.NET SDK (reflecting any global.json):
 Version:   5.0.301
 Commit:    ef17233f86

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19043
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.301\

Host (useful for support):
  Version: 5.0.7
  Commit:  556582d964

.NET SDKs installed:
  3.1.410 [C:\Program Files\dotnet\sdk]
  5.0.100 [C:\Program Files\dotnet\sdk]
  5.0.202 [C:\Program Files\dotnet\sdk]
  5.0.204 [C:\Program Files\dotnet\sdk]
  5.0.300 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

The text was updated successfully, but these errors were encountered:

roji · 2021-06-10T13:36:06Z

Note that EF is reverting back to SqlClient 2.1 (see #25057), since SqlClient 3.0 isn't an LTS release.

frankbuckley · 2021-06-10T13:49:45Z

Does that imply EF does not intend to maintain compatibility with the latest stable SqlClient?

We don't tend to worry about LTS versions as we deploy continuously. For .NET projects, we usually take dependencies on the latest stable releases on the assumption we stay up to date with latest bug fixes and enhancements.

roji · 2021-06-10T13:56:53Z

Does that imply EF does not intend to maintain compatibility with the latest stable SqlClient?

It means that LTS versions of EF should only depend on LTS versions of SqlClient, otherwise you'll have LTS EF relying on an unsupported version at some point.

Of course, assuming SqlClient itself is backwards-compatible, you can still decide to run with 3.0.0 in your particular program. I also think we should investigate this issue (i.e. is it actually an issue an EF) regardless of whether we actually reference 3.0.0.

ErikEJ · 2021-06-10T14:10:37Z

@frankbuckley are you using EF Core 5 in both cases, I see both 5 and 6 mentioned in your .csproj file?

So, can you repro with EF Core 5 with the switch enabled?

frankbuckley · 2021-06-10T14:21:12Z

@ErikEJ - yes, repro with 5.0.7 and daily build - running with .NET 5

frankbuckley · 2021-06-10T14:23:31Z

From what I can see, non-async query iteration results in no attempt to read joined Price.Timestamp:

whereas async query iteration results in attempt to read that field:

frankbuckley · 2021-06-10T14:25:26Z

Of course, assuming SqlClient itself is backwards-compatible, you can still decide to run with 3.0.0 in your particular program. I also think we should investigate this issue (i.e. is it actually an issue an EF) regardless of whether we actually reference 3.0.0.

@roji Thanks for clarification

frankbuckley · 2021-06-10T18:54:46Z

Also seeing this in another test - yet, query results do not contain any bigints. This is not IAsyncEnumerable.

Oddly (compared to above), exception only arises if I do not call options.EnableRetryOnFailure().

System.InvalidOperationException: An error occurred while reading a database value. The expected type was 'System.Nullable`1[System.Int32]' but the actual value was of type 'System.Int64'.
---> System.InvalidCastException: Unable to cast object of type 'System.Int64' to type 'System.Int32'.
 at Microsoft.Data.SqlClient.SqlBuffer.get_Int32()
 at Microsoft.Data.SqlClient.SqlDataReader.GetInt32(Int32 i)
 at lambda_method661(Closure , QueryContext , DbDataReader )
 --- End of inner exception stack trace ---
 at lambda_method661(Closure , QueryContext , DbDataReader )
 at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.PopulateIncludeCollection[TIncludingEntity,TIncludedEntity](Int32 collectionId, QueryContext queryContext, DbDataReader dbDataReader, SingleQueryResultCoordinator resultCoordinator, Func`3 parentIdentifier, Func`3 outerIdentifier, Func`3 selfIdentifier, IReadOnlyList`1 parentIdentifierValueComparers, IReadOnlyList`1 outerIdentifierValueComparers, IReadOnlyList`1 selfIdentifierValueComparers, Func`5 innerShaper, INavigationBase inverseNavigation, Action`2 fixup, Boolean trackingQuery)
 at lambda_method664(Closure , QueryContext , DbDataReader , ResultContext , SingleQueryResultCoordinator )
 at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()
---> System.InvalidCastException: Unable to cast object of type 'System.Int64' to type 'System.Int32'.
 at Microsoft.Data.SqlClient.SqlBuffer.get_Int32()
 at Microsoft.Data.SqlClient.SqlDataReader.GetInt32(Int32 i)
 at lambda_method661(Closure , QueryContext , DbDataReader )
 --- End of inner exception stack trace ---
 at lambda_method661(Closure , QueryContext , DbDataReader )
 at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.PopulateIncludeCollection[TIncludingEntity,TIncludedEntity](Int32 collectionId, QueryContext queryContext, DbDataReader dbDataReader, SingleQueryResultCoordinator resultCoordinator, Func`3 parentIdentifier, Func`3 outerIdentifier, Func`3 selfIdentifier, IReadOnlyList`1 parentIdentifierValueComparers, IReadOnlyList`1 outerIdentifierValueComparers, IReadOnlyList`1 selfIdentifierValueComparers, Func`5 innerShaper, INavigationBase inverseNavigation, Action`2 fixup, Boolean trackingQuery)
 at lambda_method664(Closure , QueryContext , DbDataReader , ResultContext , SingleQueryResultCoordinator )
 at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()

SimonCropp · 2021-06-10T23:07:08Z

we get the same issue but we are not using EnableRetryOnFailure


System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'.
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueFromSqlBufferInternal[T](SqlBuffer data, _SqlMetaData metaData)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueInternal[T](Int32 i)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValue[T](Int32 i)
   at lambda_method1558(Closure , DbDataReader , Int32[] )
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadRow()
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'.
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueFromSqlBufferInternal[T](SqlBuffer data, _SqlMetaData metaData)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueInternal[T](Int32 i)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValue[T](Int32 i)
   at lambda_method1558(Closure , DbDataReader , Int32[] )
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadRow()
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

smitpatel · 2021-06-11T00:43:30Z

@frankbuckley - For issue with long to int, that seems to be separate issue than the other issues reported, perhaps we should start a separate issue thread for it. Stacktrace is trying to read Int32 value but server returned Int64. You can get SQL for EF Core query and try investigating which column returns Int64 and why. Generally that happens when there is some mismatch in configuration somewhere. Also if you don't have enabled Sensitive data logging, it can also give more details sometimes in terms of which column/property caused error. See https://docs.microsoft.com/en-us/ef/core/logging-events-diagnostics/simple-logging#getting-detailed-messages

smitpatel · 2021-06-11T00:48:51Z

For original issue with byte[],
my understanding is that new SqlClient returns DbNull rather than empty byte[] as before. I believe this may have some special processing in EF Core.
Specifically looking at code below

efcore/src/EFCore.SqlServer/Storage/Internal/SqlServerTypeMappingSource.cs

Lines 48 to 56 in 7105f84

    
           private readonly SqlServerByteArrayTypeMapping _rowversion 
        
               = new( 
        
                   "rowversion", 
        
                   size: 8, 
        
                   comparer: new ValueComparer<byte[]>( 
        
                       (v1, v2) => StructuralComparisons.StructuralEqualityComparer.Equals(v1, v2), 
        
                       v => StructuralComparisons.StructuralEqualityComparer.GetHashCode(v), 
        
                       v => v == null ? null : v.ToArray()), 
        
                   storeTypePostfix: StoreTypePostfix.None);

We specify a custom comparer for byte[] in SqlServer, perhaps that comparer was working before as we had empty array and now it fails because we need the usual DbNull check. I would expect this to cause issue in all the cases and not just the case with EnableRetryOnFailure.

EnableRetryOnFailure in above cases is just red herring. Hence @SimonCropp is seeing error without it. The other possibility of error (though less likely) is in BufferedDataReader code. Which buffers a DbDataReader for re-trying strategies. EF Core may buffer queries in certain scenarios even when EnableRetryOnFailure is not enabled. Specifically when doing split query and MARS being off (what I am seeing in stacktrace of @SimonCropp's report).

cc: @ajcvickers for value comparer in type mapping source
cc: @AndriySvyryd if we have some odd quirks in buffered data reader

SimonCropp · 2021-06-11T00:57:01Z

new SqlClient returns DbNull rather than empty byte[] as before

@smitpatel is that documented anywhere?

smitpatel · 2021-06-11T00:59:00Z

This is the PR dotnet/SqlClient#998 for the change. Breaking change noted here https://github.com/dotnet/SqlClient/blob/main/release-notes/3.0/3.0.0-preview2.md

SimonCropp · 2021-06-11T01:00:17Z

shouldnt the final release notes for SqlClient include all the changes? i would not expect people would review all the preview release notes when going from one stable to another stable

SimonCropp · 2021-06-11T01:01:57Z

sigh i only read this https://github.com/dotnet/SqlClient/releases/tag/v3.0.0 not this https://github.com/dotnet/SqlClient/blob/main/release-notes/3.0/3.0.0.md

frankbuckley · 2021-06-11T12:39:07Z

@frankbuckley - For issue with long to int, that seems to be separate issue than the other issues reported, perhaps we should start a separate issue thread for it.

#25082

jonsagara · 2021-06-11T21:01:03Z

I also started seeing this issue after upgrading to Microsoft.Data.SqlClient 3.0.0. Reverting to 2.1.3 fixed the issue.

System.InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Byte[]'.
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueFromSqlBufferInternal[T](SqlBuffer data, _SqlMetaData metaData)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValueInternal[T](Int32 i)
   at Microsoft.Data.SqlClient.SqlDataReader.GetFieldValue[T](Int32 i)
   at lambda_method229(Closure , DbDataReader , Int32[] )
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadObject(DbDataReader reader, Int32 ordinal, ReaderColumn column)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.ReadRow()
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)

ErikEJ · 2021-06-12T06:59:59Z

@jonsagara can you provide a complete repro?
Also, have you tried the AppContext switch?

jonsagara · 2021-06-13T19:06:15Z

Unfortunately, I haven't been able to reproduce it in a test app that I can share, but I can reproduce it at will in my application. :(

The application is a .NET 5 console app using EF Core 5.0.7 with a SQL Server database. The code that generates the failing query looks like this:

var query = _context.UploadedDocuments
    .Include(ud => ud.RequestedBy)
    .Include(ud => ud.LastDownloadedByUser)
    .Where(ud => ud.DocumentType == UploadedDocumentType.PDF && ud.UploadedUtc != null);

if (noTracking)
{
    query = query.AsNoTracking();
}

return await query
    .OrderByDescending(ud => ud.UploadedUtc)
    .FirstOrDefaultAsync();

An UploadedDocument represents a file uploaded by a User. The LastDownloadedByUser navigation property is also a User, and will be null if no one has ever downloaded the document. This is what causes the exception: the null LastDownloadedByUser columns contain a [Timestamp] byte[] that is also null, and Microsoft.Data.SqlClient tries to cast that from DbNull to byte[].

I hope that helps.

SimonCropp · 2021-06-16T02:30:12Z

I note this has been assigned to v6, which i assume is in November. IMO this is a significant enough issue that perhaps it justifies a patch?

ErikEJ · 2021-06-16T04:43:46Z

@SimonCropp A complete repro is needed first. 😀

SimonCropp · 2021-06-16T05:18:04Z

@ErikEJ does the repro provided in the description not work?

ie

Repro project at: https://github.com/frankbuckley/efcore-sqldata3

SimonCropp · 2021-06-25T04:25:16Z

so does this justify a patch?

SimonCropp · 2021-07-05T03:30:14Z

just wondering if there is any answer for my question from 19 days ago?

smitpatel · 2021-07-06T20:17:07Z

@SimonCropp - This issue is put in 6.0 milestone so we plan to fix it in 6.0 release. We will look into the repro to investigate and fix the issue when we get around to it. We are currently working on other things. We will circle back if we are not able to repro the issue when we investigate.

We will discuss about patching this once we have identified the fix (the complexity and risk associated with the fix determine if it can be patched or not). Since this behavior only happens when user manually upgrade to a higher major version of SqlClient being used rather than the one shipped with EF Core in past, it lowers priority. Though we intend to make sure that future releases of EF Core works well with SqlClient 3.0

SimonCropp · 2021-07-07T22:05:56Z

@smitpatel thanks so much for the clarification

andygjp · 2021-07-19T16:55:31Z

I have the same issue.

Enabling the Switch.Microsoft.Data.SqlClient.LegacyRowVersionNullBehavior AppContext switch does not help. It does not restore previous behaviour - it returns DBNull and not an empty byte array.

I recreated the issue here: dotnet/SqlClient#1175

ssteiner · 2021-07-28T07:02:11Z

Ran into the same thing. All of a sudden, a query like this

existingTeam = await context.PmgrTeams.Include(x => x.PmgrTeamMembers).Include(x => x.Location) .Include(x => x.Location.Cluster).Include(x => x.Location.Cluster.InternalNumberDialPlan).Include(x => x.Location.Cluster.ClusterServers) .FirstOrDefaultAsync(t => t.Id == teamId).ConfigureAwait(false);

The code was unchanged for a while, but I recently enabled sqlOptions.EnableRetryOnFailure(). the Table Location has a byte[] RowVersion colum, but this query existingTeam = await context.PmgrTeams.Include(x => x.PmgrTeamMembers).Include(x => x.Location) works, so what trips things up is probably the RowVersion on InternalNumberDialPlan

Things work if I go sync again. it also works if I just run the query with two includes, and then follow the table links (so access existingTeam.Location.Cluster)

As my byte arrays are version stamps, they're not empty, and there's no other byte arrays in my model.

ErikEJ · 2021-07-28T07:07:02Z

A fix to make the AppContext switch work properly is in progress dotnet/SqlClient#1182

roji · 2021-08-25T12:41:56Z

I've tracked this down to what looks like a bug in SqlClient 3.0.0, opened dotnet/SqlClient#1228 to track. In a nutshell, SqlDataReader.IsDbNull returns wrong results for null timestamp when used after ReadAsync.

Note also that The fix for LegacyRowVersionNullBehaviour has been merged for 4.0.0-preview1 (hopefully also to be backported to 3.0.1), though if dotnet/SqlClient#1228 is fixed that AppContext switch shouldn't be necessary for EF Core to work properly.

roji · 2021-08-25T18:38:16Z

According to dotnet/SqlClient#1228 (comment), the SqlClient bug was also already fixed as part of dotnet/SqlClient#1228.

roji · 2021-08-25T21:19:12Z

Checked with the just-released 4.0.0-preview1.21237.2, and the bug no longer repro's there.

frankbuckley added the customer-reported label Jun 10, 2021

smitpatel added the type-bug label Jun 11, 2021

frankbuckley mentioned this issue Jun 11, 2021

Query fails with InvalidCastException without options.EnableRetryOnFailure() #25082

Closed

AndriySvyryd added this to the 6.0.0 milestone Jun 11, 2021

AndriySvyryd assigned smitpatel and roji Jun 11, 2021

AndriySvyryd added the area-query label Jun 11, 2021

roji mentioned this issue Aug 25, 2021

Incorrect IsDBNull results on null rowversion when using ReadAsync dotnet/SqlClient#1228

Closed

roji added the closed-external label Aug 25, 2021

roji removed this from the 6.0.0 milestone Aug 25, 2021

roji unassigned smitpatel Aug 25, 2021

ajcvickers closed this as completed Aug 28, 2021

ajcvickers removed type-bug area-query labels Aug 28, 2021

Mike-E-angelo added a commit to DragonSpark/Framework that referenced this issue Jan 21, 2022

Fixed EFCore SqlClient issue dotnet/efcore#25074

8950ee7

ajcvickers reopened this Oct 16, 2022

ajcvickers closed this as not planned Won't fix, can't repro, duplicate, stale Oct 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value #25074

Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value #25074

frankbuckley commented Jun 10, 2021 •

edited

Loading

roji commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

roji commented Jun 10, 2021

ErikEJ commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

SimonCropp commented Jun 10, 2021

smitpatel commented Jun 11, 2021

smitpatel commented Jun 11, 2021

SimonCropp commented Jun 11, 2021

smitpatel commented Jun 11, 2021

SimonCropp commented Jun 11, 2021 •

edited

Loading

SimonCropp commented Jun 11, 2021

frankbuckley commented Jun 11, 2021

jonsagara commented Jun 11, 2021

ErikEJ commented Jun 12, 2021

jonsagara commented Jun 13, 2021

SimonCropp commented Jun 16, 2021

ErikEJ commented Jun 16, 2021

SimonCropp commented Jun 16, 2021

SimonCropp commented Jun 25, 2021

SimonCropp commented Jul 5, 2021

smitpatel commented Jul 6, 2021

SimonCropp commented Jul 7, 2021

andygjp commented Jul 19, 2021

ssteiner commented Jul 28, 2021

ErikEJ commented Jul 28, 2021

roji commented Aug 25, 2021 •

edited

Loading

roji commented Aug 25, 2021

roji commented Aug 25, 2021

Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value #25074

Microsoft.Data.SqlClient 3.0.0 breaks async enumeration of results of SQL Server query including null rowversion value #25074

Comments

frankbuckley commented Jun 10, 2021 • edited Loading

Versions

Repro:

Stacktrace:

Environment

roji commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

roji commented Jun 10, 2021

ErikEJ commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

frankbuckley commented Jun 10, 2021

SimonCropp commented Jun 10, 2021

smitpatel commented Jun 11, 2021

smitpatel commented Jun 11, 2021

SimonCropp commented Jun 11, 2021

smitpatel commented Jun 11, 2021

SimonCropp commented Jun 11, 2021 • edited Loading

SimonCropp commented Jun 11, 2021

frankbuckley commented Jun 11, 2021

jonsagara commented Jun 11, 2021

ErikEJ commented Jun 12, 2021

jonsagara commented Jun 13, 2021

SimonCropp commented Jun 16, 2021

ErikEJ commented Jun 16, 2021

SimonCropp commented Jun 16, 2021

SimonCropp commented Jun 25, 2021

SimonCropp commented Jul 5, 2021

smitpatel commented Jul 6, 2021

SimonCropp commented Jul 7, 2021

andygjp commented Jul 19, 2021

ssteiner commented Jul 28, 2021

ErikEJ commented Jul 28, 2021

roji commented Aug 25, 2021 • edited Loading

roji commented Aug 25, 2021

roji commented Aug 25, 2021

frankbuckley commented Jun 10, 2021 •

edited

Loading

SimonCropp commented Jun 11, 2021 •

edited

Loading

roji commented Aug 25, 2021 •

edited

Loading