[CUDA][1.17.1][Regression] Access Violation in an inference session Run

[claeyzre]

Describe the issue

I have an old ONNX model translated from tensorflow with tf2onnx a while ago.
It always used to run fine in the previous version (<1.17) but I now hit an Access Violation during the inference.

In 1.16.3 everything works fine. The package is onnxruntime-gpu.
You can find the model here

The error happens during the inference and not at the creation of the InferenceSession. While creating the inference session the models seems to be correctly put on GPU memory and the error happens when calling .Run

Also the error does not happen when I specify CPUExecutionProvider in the providers list.

To reproduce

The following code runs fine in 1.16.3 but not in 1.17.1

from onnxruntime import InferenceSession
import numpy as np
model_path = r"path_to_the_given_model.onnx"
session = InferenceSession(model_path, providers=["CUDAExecutionProvider"])
inputs = {"input_ids": np.ones((1, 200), dtype=np.int32)}
outputs = session.run(input_feed=inputs, output_names=["output_0", "output_1"])

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.17.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.8

[snnn]

Found a bug in #17200

[snnn]

I removed the "platform:windows" label since it is not Windows specific.

[snnn]

Fixed in #19889