Training Error (CUDA error: CUBLAS_STATUS_EXECUTION_FAILED )

[gkuberreddy]

Hi, thanks for your great work. I am planning on training your model with custom dataset. I encounter following error:

""""CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)"""""

I tried multiple ways to solve this issue such as reducing batch size to 1, reducing number of gpus to 1 and reducing resolution of images to 64X64. I am training on NVIDIA Titan Xp GPUs with 12GB RAM. I didn't find any luck yet!

Can you help me resolve this issue?