possible bug - unauthenticated state downloads

[hpatel826]

The DevOps team at my company has run into issues deploying applications using tfstateremote through helmfile in the last 24ish hours. No tooling changes in our pipelines.

We suspect that the download code located here was able to download state in an unauthenticated manner up until yesterday. The go-tfe client does not provide download methods for the state file, just links to the download URLs. The HTTP client in the above file does not configure any sort of authenticated context via cookies or headers.

The error we are seeing via helmfile is (modified for brevity):

expand tfstateremote://app.terraform.io/org/workspace/resource: reading tfstate for resource: unsupported state version 0

And with tfstate-lookup directly:

tfstate-lookup -state remote://app.terraform.io/org/workspace/resource
unsupported state version 0

After modifying the source and adding some logging statements:

go run main.go -state remote://app.terraform.io/org/workspace/resource
{"errors":[{"status":"401","title":"unauthorized"}]}
https://app.terraform.io/api/state-versions/redacted/hosted_state
unsupported state version 0
exit status 1

We suspect that TFE patched this silently as a security issue.

Are we crazy? Anybody else experiencing or able to reproduce this issue?

[joeybenamy]

Also experiencing the same issue.

[fujiwara]

Thank you for reporting!
Fixed in v1.1.4, Please try it!

https://github.com/fujiwara/tfstate-lookup/releases/tag/v1.1.4

[joeybenamy]

@fujiwara FYI, this is the statement from Hashicorp on this particular issue. Basically the download URL endpoint used for state files changed and it used to be pre-signed and the new endpoint is not and requires a bearer token.

Thanks for reaching out to HashiCorp Support!

My name is Yordan and I have taken ownership of this ticket.

I have tracked down the change made to Terraform Cloud. And have spoken to the engineer implementing this change.

They have assured me that the hosted_state endpoint is new. Can you share details to confirm you were able to access it before without any credentials?

Any security issues found are communicated to our users. We notify the impacted users of any change due to security reasons so the lack of notification here is because we believe there isn't a security problem here.

The implementation details of the API did change, but that's not something we document. The API documentation just says that the hosted-state-download-url attribute contains the URL for downloading the state; it doesn't provide any guarantee that the URL will be a pre-signed URL or a 200 response (rather than a 3xx). The authentication section of the API states that "All requests must be authenticated with a bearer token."

I hope this information helps to get a better understanding of what has changed.

[fujiwara]

@joeybenamy Thank you for the sharing! I understood perfectly.

[joeybenamy]

@joeybenamy Thank you for the sharing! I understood perfectly.

Another explanation from Hashicorp:

I apologize for the delayed response. Our team is working on implementing some improvements to object lifecycle management. To support those improvements, we've made a change to the state version response object. Specifically, the hosted-json-state-download-url and the hosted-state-download-url attributes return a different type of URL. Previously, they returned pre-signed URLs (i.e. archivist.terraform.io/v1/object/*). Now, they return a URL (i.e. app.terraform.io/api/state-versions/*/hosted_state) that redirects to a pre-signed URL (i.e. archivist.terraform.io/v1/object/*). This undocumented behavior was an implementation detail we needed to change to support this work. Going forward, to download the state version, clients will need to supply the authentication token to the hosted_state endpoint and follow the HTTP redirect to the pre-signed URL.