Add Serverless Cache Resource #1425

blakeromano · 2024-07-29T03:08:21Z

Description of your changes

Adds the Serverless Cache Elasticache Resource

Fixes #1117

I have:

Read and followed Crossplane's contribution process.
Run make reviewable to ensure this PR is ready for review.
Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Ran make e2e UPTEST_EXAMPLE_LIST=examples/elasticache/v1beta1/serverlesscache.yaml

Some output:

    logger.go:42: 20:03:00 | case/3-delete | - apiVersion: elasticache.aws.upbound.io/v1beta1
    logger.go:42: 20:03:00 | case/3-delete |   kind: ServerlessCache
    logger.go:42: 20:03:00 | case/3-delete |   metadata:
    logger.go:42: 20:03:00 | case/3-delete |     annotations:
    logger.go:42: 20:03:00 | case/3-delete |       crossplane.io/external-create-pending: "2024-07-29T02:56:53Z"
    logger.go:42: 20:03:00 | case/3-delete |       crossplane.io/external-create-succeeded: "2024-07-29T02:56:53Z"
    logger.go:42: 20:03:00 | case/3-delete |       crossplane.io/external-name: example
    logger.go:42: 20:03:00 | case/3-delete |       crossplane.io/paused: "false"
    logger.go:42: 20:03:00 | case/3-delete |       meta.upbound.io/example-id: elasticache/v1beta1/serverlesscache
    logger.go:42: 20:03:00 | case/3-delete |       upjet.upbound.io/test: "true"
    logger.go:42: 20:03:00 | case/3-delete |       uptest-old-id: example
    logger.go:42: 20:03:00 | case/3-delete |     creationTimestamp: "2024-07-29T02:56:28Z"
    logger.go:42: 20:03:00 | case/3-delete |     deletionGracePeriodSeconds: 0
    logger.go:42: 20:03:00 | case/3-delete |     deletionTimestamp: "2024-07-29T02:59:26Z"
    logger.go:42: 20:03:00 | case/3-delete |     finalizers:
    logger.go:42: 20:03:00 | case/3-delete |     - finalizer.managedresource.crossplane.io
    logger.go:42: 20:03:00 | case/3-delete |     generation: 5
    logger.go:42: 20:03:00 | case/3-delete |     labels:
    logger.go:42: 20:03:00 | case/3-delete |       testing.upbound.io/example-name: example
    logger.go:42: 20:03:00 | case/3-delete |     name: example
    logger.go:42: 20:03:00 | case/3-delete |     resourceVersion: "3465"
    logger.go:42: 20:03:00 | case/3-delete |     uid: c19c75d6-f06c-4df3-ab39-36224e2e5233
    logger.go:42: 20:03:00 | case/3-delete |   spec:
    logger.go:42: 20:03:00 | case/3-delete |     deletionPolicy: Delete
    logger.go:42: 20:03:00 | case/3-delete |     forProvider:
    logger.go:42: 20:03:00 | case/3-delete |       cacheUsageLimits:
    logger.go:42: 20:03:00 | case/3-delete |       - dataStorage:
    logger.go:42: 20:03:00 | case/3-delete |         - maximum: 10
    logger.go:42: 20:03:00 | case/3-delete |           unit: GB
    logger.go:42: 20:03:00 | case/3-delete |         ecpuPerSecond:
    logger.go:42: 20:03:00 | case/3-delete |         - maximum: 5000
    logger.go:42: 20:03:00 | case/3-delete |       dailySnapshotTime: "08:00"
    logger.go:42: 20:03:00 | case/3-delete |       description: Test Server
    logger.go:42: 20:03:00 | case/3-delete |       engine: memcached
    logger.go:42: 20:03:00 | case/3-delete |       majorEngineVersion: "1.6"
    logger.go:42: 20:03:00 | case/3-delete |       region: us-west-1
    logger.go:42: 20:03:00 | case/3-delete |       securityGroupIdRefs:
    logger.go:42: 20:03:00 | case/3-delete |       - name: example
    logger.go:42: 20:03:00 | case/3-delete |       securityGroupIdSelector:
    logger.go:42: 20:03:00 | case/3-delete |         matchLabels:
    logger.go:42: 20:03:00 | case/3-delete |           testing.upbound.io/example-name: example
    logger.go:42: 20:03:00 | case/3-delete |       securityGroupIds:
    logger.go:42: 20:03:00 | case/3-delete |       - sg-02a24b56292c9f6bb
    logger.go:42: 20:03:00 | case/3-delete |       subnetIdRefs:
    logger.go:42: 20:03:00 | case/3-delete |       - name: sample-subnet1
    logger.go:42: 20:03:00 | case/3-delete |       - name: sample-subnet2
    logger.go:42: 20:03:00 | case/3-delete |       subnetIdSelector:
    logger.go:42: 20:03:00 | case/3-delete |         matchLabels:
    logger.go:42: 20:03:00 | case/3-delete |           testing.upbound.io/example-name: example
    logger.go:42: 20:03:00 | case/3-delete |       subnetIds:
    logger.go:42: 20:03:00 | case/3-delete |       - subnet-04b38356a93a6e5d9
    logger.go:42: 20:03:00 | case/3-delete |       - subnet-053d9784dce180737
    logger.go:42: 20:03:00 | case/3-delete |       tags:
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-kind: serverlesscache.elasticache.aws.upbound.io
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-name: example
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-providerconfig: default
    logger.go:42: 20:03:00 | case/3-delete |     initProvider: {}
    logger.go:42: 20:03:00 | case/3-delete |     managementPolicies:
    logger.go:42: 20:03:00 | case/3-delete |     - '*'
    logger.go:42: 20:03:00 | case/3-delete |     providerConfigRef:
    logger.go:42: 20:03:00 | case/3-delete |       name: default
    logger.go:42: 20:03:00 | case/3-delete |   status:
    logger.go:42: 20:03:00 | case/3-delete |     atProvider:
    logger.go:42: 20:03:00 | case/3-delete |       arn: arn:aws:elasticache:us-west-1:ACCOUNT_ID:serverlesscache:example
    logger.go:42: 20:03:00 | case/3-delete |       cacheUsageLimits:
    logger.go:42: 20:03:00 | case/3-delete |       - dataStorage:
    logger.go:42: 20:03:00 | case/3-delete |         - maximum: 10
    logger.go:42: 20:03:00 | case/3-delete |           unit: GB
    logger.go:42: 20:03:00 | case/3-delete |         ecpuPerSecond:
    logger.go:42: 20:03:00 | case/3-delete |         - maximum: 5000
    logger.go:42: 20:03:00 | case/3-delete |       createTime: "2024-07-29T02:56:54Z"
    logger.go:42: 20:03:00 | case/3-delete |       dailySnapshotTime: "08:00"
    logger.go:42: 20:03:00 | case/3-delete |       description: Test Server
    logger.go:42: 20:03:00 | case/3-delete |       endpoint:
    logger.go:42: 20:03:00 | case/3-delete |       - address: example-obzlvu.serverless.usw1.cache.amazonaws.com
    logger.go:42: 20:03:00 | case/3-delete |         port: 11211
    logger.go:42: 20:03:00 | case/3-delete |       engine: memcached
    logger.go:42: 20:03:00 | case/3-delete |       fullEngineVersion: 1.6.22
    logger.go:42: 20:03:00 | case/3-delete |       id: example
    logger.go:42: 20:03:00 | case/3-delete |       majorEngineVersion: "1.6"
    logger.go:42: 20:03:00 | case/3-delete |       readerEndpoint:
    logger.go:42: 20:03:00 | case/3-delete |       - address: example-obzlvu.serverless.usw1.cache.amazonaws.com
    logger.go:42: 20:03:00 | case/3-delete |         port: 11212
    logger.go:42: 20:03:00 | case/3-delete |       securityGroupIds:
    logger.go:42: 20:03:00 | case/3-delete |       - sg-02a24b56292c9f6bb
    logger.go:42: 20:03:00 | case/3-delete |       snapshotRetentionLimit: 0
    logger.go:42: 20:03:00 | case/3-delete |       status: available
    logger.go:42: 20:03:00 | case/3-delete |       subnetIds:
    logger.go:42: 20:03:00 | case/3-delete |       - subnet-04b38356a93a6e5d9
    logger.go:42: 20:03:00 | case/3-delete |       - subnet-053d9784dce180737
    logger.go:42: 20:03:00 | case/3-delete |       tags:
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-kind: serverlesscache.elasticache.aws.upbound.io
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-name: example
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-providerconfig: default
    logger.go:42: 20:03:00 | case/3-delete |       tagsAll:
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-kind: serverlesscache.elasticache.aws.upbound.io
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-name: example
    logger.go:42: 20:03:00 | case/3-delete |         crossplane-providerconfig: default
    logger.go:42: 20:03:00 | case/3-delete |     conditions:
    logger.go:42: 20:03:00 | case/3-delete |     - lastTransitionTime: "2024-07-29T02:59:18Z"
    logger.go:42: 20:03:00 | case/3-delete |       reason: ReconcileSuccess
    logger.go:42: 20:03:00 | case/3-delete |       status: "True"
    logger.go:42: 20:03:00 | case/3-delete |       type: Synced
    logger.go:42: 20:03:00 | case/3-delete |     - lastTransitionTime: "2024-07-29T02:59:26Z"
    logger.go:42: 20:03:00 | case/3-delete |       reason: Deleting
    logger.go:42: 20:03:00 | case/3-delete |       status: "False"
    logger.go:42: 20:03:00 | case/3-delete |       type: Ready
    logger.go:42: 20:03:00 | case/3-delete |     - lastTransitionTime: "2024-07-29T02:59:18Z"
    logger.go:42: 20:03:00 | case/3-delete |       reason: UpToDate
    logger.go:42: 20:03:00 | case/3-delete |       status: "True"
    logger.go:42: 20:03:00 | case/3-delete |       type: Test
    logger.go:42: 20:03:00 | case/3-delete |     - lastTransitionTime: "2024-07-29T02:59:18Z"
    logger.go:42: 20:03:00 | case/3-delete |       reason: Success
    logger.go:42: 20:03:00 | case/3-delete |       status: "True"
    logger.go:42: 20:03:00 | case/3-delete |       type: LastAsyncOperation
    logger.go:42: 20:03:00 | case/3-delete | kind: List
    logger.go:42: 20:03:00 | case/3-delete | metadata:
    logger.go:42: 20:03:00 | case/3-delete |   resourceVersion: ""
    logger.go:42: 20:03:00 | case/3-delete | running command: [sh -c echo "Dump Claim manifests for the delete assertion step:" || ${KUBECTL} get claim --all-namespaces -o yaml]
    logger.go:42: 20:03:00 | case/3-delete | Dump Claim manifests for the delete assertion step:
    logger.go:42: 20:03:00 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait serverlesscache.elasticache.aws.upbound.io/example --for=delete --timeout 10s]
    logger.go:42: 20:03:02 | case/3-delete | serverlesscache.elasticache.aws.upbound.io/example condition met
    logger.go:42: 20:03:02 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait securitygroup.ec2.aws.upbound.io/example --for=delete --timeout 10s]
    logger.go:42: 20:03:02 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait vpc.ec2.aws.upbound.io/sample-vpc --for=delete --timeout 10s]
    logger.go:42: 20:03:03 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait subnet.ec2.aws.upbound.io/sample-subnet1 --for=delete --timeout 10s]
    logger.go:42: 20:03:03 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait subnet.ec2.aws.upbound.io/sample-subnet2 --for=delete --timeout 10s]
    logger.go:42: 20:03:04 | case/3-delete | running command: [/Users/blake.romano/code/provider-aws/.cache/tools/darwin_arm64/kubectl-v1.24.3 wait managed --all --for=delete --timeout 10s]
    logger.go:42: 20:03:04 | case/3-delete | test step completed 3-delete
    logger.go:42: 20:03:05 | case | Failed to collect events for case in ns kuttl-test-neat-chicken: no matches for kind "Event" in version "events.k8s.io/v1beta1"
    logger.go:42: 20:03:05 | case | Trying with events eventsv1 API...
    logger.go:42: 20:03:05 | case | case events from ns kuttl-test-neat-chicken:
    logger.go:42: 20:03:05 | case | Deleting namespace: kuttl-test-neat-chicken
=== CONT  kuttl
    harness.go:402: run tests finished
    harness.go:511: cleaning up
    harness.go:553: skipping cluster tear down
    harness.go:554: to connect to the cluster, run: export KUBECONFIG="/Users/blake.romano/code/provider-aws/kubeconfig"
--- PASS: kuttl (398.89s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/case (397.80s)
PASS
20:03:05 [ OK ] running automated tests

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

turkenf · 2024-07-29T08:42:55Z

Thank you, @blakeromano 🙏 Since you are now a contributor, you should be able to trigger uptest.

blakeromano · 2024-07-29T14:39:13Z

/test-examples="examples/elasticache/v1beta1/serverlesscache.yaml"

blakeromano · 2024-07-29T15:14:46Z

@turkenf just something to note... I am seeing a lot of flakiness on being able to spin up subnets which I was able to observe locally on my own AWS Account but with the issue coming from us-west-1b instead of us-west-1c in the failure seen above.

For example:

async create failed: failed to create the resource: [{0 creating EC2
    logger.go:42: 15:07:19 | case/0-apply |         Subnet: InvalidParameterValue: Value (us-west-1c) for parameter availabilityZone
    logger.go:42: 15:07:19 | case/0-apply |         is invalid. Subnets can currently only be created in the following availability
    logger.go:42: 15:07:19 | case/0-apply |         zones: us-west-1a, us-west-1b.\n\tstatus code: 400, request id: 69e6d782-aad4-4dbd-9a53-0e0df2cfe407
    logger.go:42: 15:07:19 | case/0-apply |         \ []}]

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

…vider-aws into serverless-cache

blakeromano · 2024-07-29T15:50:01Z

/test-examples="examples/elasticache/v1beta1/serverlesscache.yaml"

blakeromano · 2024-07-29T16:14:37Z

I think that the tests may be stuck now 😢

async create failed: resource creation call returned error diags:
    logger.go:42: 15:59:19 | case/0-apply |         creating ElastiCache Serverless Cache: operation error ElastiCache: CreateServerlessCache,
    logger.go:42: 15:59:19 | case/0-apply |         https response error StatusCode: 400, RequestID: ab37d9bd-c75c-4a58-8842-86aeed710895,
    logger.go:42: 15:59:19 | case/0-apply |         ServerlessCacheAlreadyExistsFault: Serverless Cache already exists

turkenf · 2024-07-29T16:31:57Z

Please use metadata.name: example-${Rand.RFC1123Subdomain} for ServerlessCache resource. This will generate a random name for the uptest. It would also be a good idea to edit the other example names in the YAML, such as name: example-sg-serverlesscache for SecurityGroup and the others.

mergenci · 2024-07-30T05:10:25Z

@blakeromano The error message is somewhat misleading. The flakiness is because us-west-1c doesn't have enough resources to serve as an availability zone at that time. The same could happen to us-west-1a and us-west-1b.

turkenf

Many thanks for your effort @blakeromano, I left a few comments for you to consider. Most of them are simple, name-related comments.

turkenf · 2024-08-01T18:29:57Z

config/elasticache/config.go

+		r.TerraformCustomDiff = func(diff *terraform.InstanceDiff, _ *terraform.InstanceState, _ *terraform.ResourceConfig) (*terraform.InstanceDiff, error) {
+			if diff != nil && diff.Attributes != nil {
+				delete(diff.Attributes, "security_group_names.#")
+			}
+			return diff, nil
+		}


Why do we add this func?

This was present in the replicationgroup resource. This seemed to have been added during a TF Update recently and I maybe "assumed" that serverlesscache would have the same error as we saw during TF Provider Update.

I see; however, we usually avoid adding this kind of configuration unless they are really required. My suggestion is please remove it, test it, and add it if really required. What do you think?

Since the uptest passed successfully, we can proceed without the TerraformCustomDiff function, thank you.

examples/elasticache/v1beta1/serverlesscache.yaml

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

blakeromano · 2024-08-01T22:19:40Z

/test-examples="examples/elasticache/v1beta1/serverlesscache.yaml"

turkenf · 2024-08-06T08:21:58Z

Hey @blakeromano, if you are interested we can include it in the planned release this week #1425 (comment)

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

…vider-aws into serverless-cache

blakeromano · 2024-08-06T20:59:47Z

/test-examples="examples/elasticache/v1beta1/serverlesscache.yaml"

turkenf

LGTM 🙌

blakeromano added 2 commits July 28, 2024 20:04

Add Serverless Cache Resource

a6ae507

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

add file header

acb6a77

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

blakeromano added 3 commits July 29, 2024 08:14

Merge branch 'main' into serverless-cache

0daec4b

update to us-west-1b

4696592

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

Merge branch 'serverless-cache' of https://github.com/blakeromano/pro…

b9c4a04

…vider-aws into serverless-cache

turkenf reviewed Aug 1, 2024

View reviewed changes

update naming

a87018c

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

Merge branch 'main' into serverless-cache

740edad

blakeromano marked this pull request as ready for review August 1, 2024 23:44

blakeromano requested review from ulucinar, sergenyalcin and mbbush as code owners August 1, 2024 23:44

blakeromano added 3 commits August 6, 2024 13:58

remove ignorediff

4d4dded

Signed-off-by: Blake R <85771645+blakeromano@users.noreply.github.com>

Merge branch 'serverless-cache' of https://github.com/blakeromano/pro…

75600bf

…vider-aws into serverless-cache

Merge branch 'main' into serverless-cache

38b315d

turkenf approved these changes Aug 7, 2024

View reviewed changes

turkenf merged commit de03a38 into crossplane-contrib:main Aug 7, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Serverless Cache Resource #1425

Add Serverless Cache Resource #1425

blakeromano commented Jul 29, 2024

turkenf commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

turkenf commented Jul 29, 2024

mergenci commented Jul 30, 2024

turkenf left a comment

turkenf Aug 1, 2024

blakeromano Aug 1, 2024

turkenf Aug 2, 2024

turkenf Aug 7, 2024

blakeromano commented Aug 1, 2024

turkenf commented Aug 6, 2024

blakeromano commented Aug 6, 2024

turkenf left a comment

Add Serverless Cache Resource #1425

Add Serverless Cache Resource #1425

Conversation

blakeromano commented Jul 29, 2024

Description of your changes

How has this code been tested

turkenf commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

blakeromano commented Jul 29, 2024

turkenf commented Jul 29, 2024

mergenci commented Jul 30, 2024

turkenf left a comment

Choose a reason for hiding this comment

turkenf Aug 1, 2024

Choose a reason for hiding this comment

blakeromano Aug 1, 2024

Choose a reason for hiding this comment

turkenf Aug 2, 2024

Choose a reason for hiding this comment

turkenf Aug 7, 2024

Choose a reason for hiding this comment

blakeromano commented Aug 1, 2024

turkenf commented Aug 6, 2024

blakeromano commented Aug 6, 2024

turkenf left a comment

Choose a reason for hiding this comment