(custom-resources): (custom resource python 3.11 runtime deployment fails for certain new regions)

[GavinZZ]

Describe the bug

The following error happens when deploying a stack to new isolated reegions that uses custom resources with python runtime. The default runtime we use for these custom resources are python 3.9 and after investigation, the default boto3 version has incorrect configuration with runtime 3.9.

However, customers don't have an easy way to change the runtime version for custom resources. We should update all python runtime custom resources to runtime 3.11 as this is the latest version that have full global support in all regions.

{
            "timestamp": 1724109383373,
            "message": "fatal error: Could not connect to the endpoint URL: \"https://amzn-assets-5638490-ale-767472097353.s3.us-isof-south-1.amazonaws.com/pipelines_aggregate_transform_5638490/13fa89dc-1f7a-4769-a624-eb2025dc397d_FeatureConfigSrcAsset/artifact\"\n",
            "ingestionTime": 1724109392391
        },
        {
            "timestamp": 1724109384913,
            "message": "[ERROR]\t2024-08-19T23:16:24.875Z\t9d7c7bf9-feae-4dd0-874f-eff29fe1eed4\tCommand '['/opt/awscli/aws', 's3', 'cp', 's3://amzn-assets-5638490-ale-767472097353/pipelines_aggregate_transform_5638490/13fa89dc-1f7a-4769-a624-eb2025dc397d_FeatureConfigSrcAsset/artifact', '/tmp/tmpninvahq0/46cf5471-d369-483e-918f-d52f6763b70f']' returned non-zero exit status 1.\nTraceback (most recent call last):\n  File \"/var/task/index.py\", line 128, in handler\n    s3_deploy(s3_source_zips, s3_dest, user_metadata, system_metadata, prune, exclude, include, source_markers, extract)\n  File \"/var/task/index.py\", line 173, in s3_deploy\n    aws_command(\"s3\", \"cp\", s3_source_zip, archive)\n  File \"/var/task/index.py\", line 243, in aws_command\n    subprocess.check_call([aws] + list(args))\n  File \"/var/lang/lib/python3.9/subprocess.py\", line 373, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'cp', 's3://amzn-assets-5638490-ale-767472097353/pipelines_aggregate_transform_5638490/13fa89dc-1f7a-4769-a624-eb2025dc397d_FeatureConfigSrcAsset/artifact', '/tmp/tmpninvahq0/46cf5471-d369-483e-918f-d52f6763b70f']' returned non-zero exit status 1.",
            "ingestionTime": 1724109392391
        },
        {
            "timestamp": 1724109384913,
            "message": "[ERROR]\t2024-08-19T23:16:24.913Z\t9d7c7bf9-feae-4dd0-874f-eff29fe1eed4\t| cfn_error: Command '['/opt/awscli/aws', 's3', 'cp', 's3://amzn-assets-5638490-ale-767472097353/pipelines_aggregate_transform_5638490/13fa89dc-1f7a-4769-a624-eb2025dc397d_FeatureConfigSrcAsset/artifact', '/tmp/tmpninvahq0/46cf5471-d369-483e-918f-d52f6763b70f']' returned non-zero exit status 1.\n",
            "ingestionTime": 1724109392391
        },
        {
            "timestamp": 1724109384913,
            "message": "[INFO]\t2024-08-19T23:16:24.913Z\t9d7c7bf9-feae-4dd0-874f-eff29fe1eed4\t| response body:\n{\"Status\": \"FAILED\", \"Reason\": \"Command '['/opt/awscli/aws', 's3', 'cp', 's3://amzn-assets-5638490-ale-767472097353/pipelines_aggregate_transform_5638490/13fa89dc-1f7a-4769-a624-eb2025dc397d_FeatureConfigSrcAsset/artifact', '/tmp/tmpninvahq0/46cf5471-d369-483e-918f-d52f6763b70f']' returned non-zero exit status 1.\", \"PhysicalResourceId\": \"2024/08/19/[$LATEST]d6f1b26fcfd34fbfafd2621cbb94e523\", \"StackId\": \"arn:aws-iso-f:cloudformation:us-isof-south-1:767472097353:stack/GlueStudioFeatures-us-isof-south-1-gamma-Stack/b5f7b770-5e80-11ef-b1d1-0a0cd33e71e4\", \"RequestId\": \"1509bf72-d85b-448c-9b6b-fc07101361b7\", \"LogicalResourceId\": \"FeatureConfigDeployerFeatureConfigDeployment1724101448513CustomResourceAD59A91B\", \"NoEcho\": false, \"Data\": {}}\n",
            "ingestionTime": 1724109392391
        },

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Able to deploy successfully to all regions

Current Behavior

Unable to deploy to new isolated regions due to the errors above.

Reproduction Steps

Any stack that uses python runtime custom resource like S3 BucketDeploymentSingletonFunction

Possible Solution

Upgrade for all these custom resources.

Additional Information/Context

No response

CDK CLI Version

all

Framework Version

No response

Node.js Version

20

OS

MacOs

Language

TypeScript, Python, .NET, Java, Go

Language Version

No response

Other information

No response

[awslukeguan]

As an alternative, please try to set lambda runtime Custom Resources with CustomResourceConfig.

The following will set the AWS-vended custom resource in the app to PYTHON_3_11:
import { CustomResourceConfig } from 'aws-cdk-lib/custom-resources';
CustomResourceConfig.of(app).addLambdaRuntime(lambda.Runtime.PYTHON_3_11);

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources-readme.html

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.