per volume secret tokens not recognized

[jmccormick2001]

What happened:
I'm trying to specify 'per-volume' secret templates in a StorageClass but the external-provisioner is not recognizing them for the CreateVolume 'provisioner-secret-name' secret. Instead it shows this error in the provisioner log:

Warning ProvisioningFailed 1s (x4 over 8s) infinibox-csi-driver_infinidat-csi-driver-driver-0_e943e557-81ae-4050-bb17-d212ae1faed6 failed to provision volume with StorageClass "ibox-nfs-anno-secret": error resolving value "${pvc.annotations['infinidat.com/ibox_secret']}": invalid tokens: ["pvc.annotations['infinidat.com/ibox_secret']"]

What you expected to happen:
I expected the provisioner to understand the syntax of the tokens as described in this document:
https://kubernetes-csi.github.io/docs/secrets-and-credentials-storage-class.html

How to reproduce it:
I created a StorageClass like this:
parameters:
# reference secret with InfiniBox credentials
csi.storage.k8s.io/controller-expand-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
csi.storage.k8s.io/controller-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
csi.storage.k8s.io/node-stage-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
csi.storage.k8s.io/provisioner-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi

Anything else we need to know?:
I guess my question is why doesn't the provisioner-secret accept the token template like the other secrets?

Environment:

• Driver version: 3.6 and 4.0

• Kubernetes version (use kubectl version):

• Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"9b1e0d27df3cf7b2ea878cd668ce709cc4e4c41a", GitTreeState:"clean", BuildDate:"2023-11-22T02:40:30Z", GoVersion:"go1.20.10 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v5.0.1
  Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4+k3s1", GitCommit:"36645e7311e9bdbbf2adb79ecd8bd68556bc86f6", GitTreeState:"clean", BuildDate:"2023-07-28T09:46:04Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

• NAME="Rocky Linux"
  VERSION="8.7 (Green Obsidian)"
  ID="rocky"
  ID_LIKE="rhel centos fedora"
  VERSION_ID="8.7"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
  ANSI_COLOR="0;32"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:rocky:rocky:8:GA"
  HOME_URL="https://rockylinux.org/"
  BUG_REPORT_URL="https://bugs.rockylinux.org/"
  ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
  ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
  REDHAT_SUPPORT_PRODUCT="Rocky Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

• Kernel (e.g. uname -a): Linux jeff-test 4.18.0-425.19.2.el8_7.x86_64 [WIP] provision PV #1 SMP Tue Apr 4 22:38:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: k3s

• Others:

[hoyho]

IMO, This is a historical issue. In early implementation here #69
The last parameter is a pvc pointer which is nil here,

	// Resolve provision secret credentials.
	// No PVC is provided when resolving provision/delete secret names, since the PVC may or may not exist at delete time.
	provisionerSecretRef, err := getSecretReference(provisionerSecretNameKey, provisionerSecretNamespaceKey, options.Parameters, pvName, nil)
	if err != nil {
		return nil, err
	}
	

that will not match any pvc.annotations token

  func getSecretReference(secretParams secretParamsMap, storageClassParams map[string]string, pvName string, pvc *v1.PersistentVolumeClaim) (*v1.SecretReference, error) {
	...
	if pvc != nil {
			nameParams[tokenPVCNameKey] = pvc.Name
			nameParams[tokenPVCNameSpaceKey] = pvc.Namespace
			for k, v := range pvc.Annotations {
				nameParams["pvc.annotations['"+k+"']"] = v
			}
		}
	}
	...

The reason why it does not support token ${pvc.annotations['<ANNOTATION_KEY>']} was already explained in the comment:
No PVC is provided when resolving provision/delete secret names, since the PVC may or may not exist at delete time.

This caused inconsistency but still make sense, because the PVC may not exist at delete time, so it is not possible to get the annotation value from the PVC （If we reference to it).

However, With this PR got merged,
the deletion secret will be stored in PV's metadata(annotations) and it can be resolved on deletion time without relying on PVC.
So, I think It is possible to support templated secret in pvc annoations.