You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Terraform Version & Prov:
Terraform 1.5.5, open-source
AFT Version: 1.12.1
Terraform Version & Provider Versions
Please provide the outputs of terraform version and terraform providers from within your AFT environment
I can provide these if needed, but leaving them out because it's non-trivial to run the AFT terraform locally, and I don't think this issue is related to Terraform.
Bug Description
High-level: Sometimes, an AFT request for an account is syntactically correct, but Control Tower fails to actually create the account. Today, we had an account request where this happened, but the only error we got from AFT was about a failed call to DescribeAccount, even though the true error was from Control Tower and visible in Service Catalog in our management account.
More specifically: The aft-invoke-aft-account-provisioning-framework Lambda does not proactively catch when the CreateManagedAccount call failed in Control Tower, and as a result the event data sent to the Lambda lists the account ID as Not Available. aft-invoke-aft-account-provisioning-framework then tries to call AWS Organization's DescribeAccount method with the account ID Not Available, which causes boto3 to throw an exception (An error occurred (InvalidInputException) when calling the DescribeAccount operation: You provided a string that exceeds that maximum length.).
This does cause a failure to propagate in AFT, which is good, but proactively catching and erroring with a more descriptive message than the DescribeAccount error would make this easier to debug. Even saying that the account creation request failed would have sent my debugging in a more productive direction.
To Reproduce
Steps to reproduce the behavior:
Create an AFT account request where the account email is already used as the root account email for another AWS account.
Commit the change so AFT's pipelines will apply the Terraform and AFT will try to create the account.
The aft-failure-notifications SNS topic should receive a message with the DescribeAccount error mentioned above.
Expected behavior
If the account fails to create in Control Tower, AFT should error with that information rather than trying to continue with an invalid account ID.
If I have to trace through the entire AFT pipeline step by step, it makes debugging AFT failures take longer than it should.
Related Logs
I think I shared everything that's relevant elsewhere, can grab more logs if that would be useful.
Additional context
We had an AFT account request fail on initial creation due to this error, which I found in Service Catalog's provisioned products list:
AWS Control Tower cannot create an account using email user@company.com because an AWS account with that email already exists, but it is not part of your AWS Control Tower organization.
However, the error sent from the AFT failures SNS topic just had this less useful error:
AFT account request failed
An error occurred in the 'aft-invoke-aft-account-provisioning-framework' Lambda function.
For more information, search AWS Request ID 'c3e55225-a997-47a6-b3d7-6be2e2eea65d' in CloudWatch log group '/aws/lambda/aft-invoke-aft-account-provisioning-framework'
Error Message: An error occurred (InvalidInputException) when calling the DescribeAccount operation: You provided a string that exceeds that maximum length.
After looking at the source code for the aft-invoke-aft-account-provisioning-framework Lambda, I found that it calls the DescribeAccount operation on a Control Tower event, but I couldn't figure out what the actual problematic event content was.
Tracing back, I noted that the aft-controltower-event-logger EventBridge rule triggers the aft-invoke-aft-account-provisioning-framework Lambda. I went to look at the EventBridge rule and saw it also triggers the aft-controltower-event-logger Lambda, which I noted writes to the aft-controltower-events Dynamo table, so I went to that table. Finally, I found the problematic event:
Terraform Version & Prov:
Terraform 1.5.5, open-source
AFT Version: 1.12.1
Terraform Version & Provider Versions
Please provide the outputs of
terraform version
andterraform providers
from within your AFT environmentI can provide these if needed, but leaving them out because it's non-trivial to run the AFT terraform locally, and I don't think this issue is related to Terraform.
Bug Description
High-level: Sometimes, an AFT request for an account is syntactically correct, but Control Tower fails to actually create the account. Today, we had an account request where this happened, but the only error we got from AFT was about a failed call to
DescribeAccount
, even though the true error was from Control Tower and visible in Service Catalog in our management account.More specifically: The aft-invoke-aft-account-provisioning-framework Lambda does not proactively catch when the
CreateManagedAccount
call failed in Control Tower, and as a result the event data sent to the Lambda lists the account ID asNot Available
. aft-invoke-aft-account-provisioning-framework then tries to call AWS Organization's DescribeAccount method with the account IDNot Available
, which causes boto3 to throw an exception (An error occurred (InvalidInputException) when calling the DescribeAccount operation: You provided a string that exceeds that maximum length.
).This does cause a failure to propagate in AFT, which is good, but proactively catching and erroring with a more descriptive message than the DescribeAccount error would make this easier to debug. Even saying that the account creation request failed would have sent my debugging in a more productive direction.
To Reproduce
Steps to reproduce the behavior:
aft-failure-notifications
SNS topic should receive a message with the DescribeAccount error mentioned above.Expected behavior
If the account fails to create in Control Tower, AFT should error with that information rather than trying to continue with an invalid account ID.
Related Logs
I think I shared everything that's relevant elsewhere, can grab more logs if that would be useful.
Additional context
We had an AFT account request fail on initial creation due to this error, which I found in Service Catalog's provisioned products list:
However, the error sent from the AFT failures SNS topic just had this less useful error:
After looking at the source code for the aft-invoke-aft-account-provisioning-framework Lambda, I found that it calls the DescribeAccount operation on a Control Tower event, but I couldn't figure out what the actual problematic event content was.
Tracing back, I noted that the
aft-controltower-event-logger
EventBridge rule triggers theaft-invoke-aft-account-provisioning-framework
Lambda. I went to look at the EventBridge rule and saw it also triggers theaft-controltower-event-logger
Lambda, which I noted writes to theaft-controltower-events
Dynamo table, so I went to that table. Finally, I found the problematic event:Partially redacted event JSON
The text was updated successfully, but these errors were encountered: