Fargate instances incorrectly detected as OnPrem

[rs-garrick]

My ECS tasks that use cloudwatch agent on EC2 instances work great. The exact same container run on Fargate detect themselves as OnPrem and then break instance role permissions by adding these lines to the toml config:
profile = "AmazonCloudWatchAgent"
shared_credential_file = "/root/.aws/credentials"

As a temporary measure, is there a way to tell cloudwatch agent not to add those 2 lines?

Thanks in advance!

[ThisIsQasim]

@hdj630 Same issue happens when running Cloudwatch agent on EKS Fargate. The metadata service isn't available so the agent decides that it is running On-prem and starts to look for credentials in .aws/credentials even though it has an IAM role attached via a serviceAccount. Running the same pod on EC2 with the same serviceAccount works fine.

[ThisIsQasim]

I have pushed an image to thisisqasim/cloudwatch-agent that allows an env FORCE_MODE_EC2=True which forces it to use ec2 mode and not look for credentials at .aws/credetials

[rs-garrick]

@ThisIsQasim That's great! Your behaviour should be the default.

But really, the config system of amazon-cloudwatch-agent is just bizarre. Why must it act differently than every other AWS client app? It's trying to be "smart" and just being inflexible. It should just use the auth defaults of the aws library and be done with it.

[ThisIsQasim]

I think it’s being done to validate the config for EC2 specific features e.g. {instance_id}. It should however let the sdk handle the credentials chain and not manually handle it.

[ThisIsQasim]

I have changed the flag for forcing EC2 mode from FORCE_MODE_EC2=True to RUN_IN_AWS=True as per instructions from the maintainers on #122

[rs-garrick]

Just to be clear, your patch does not change the default behaviour, correct? Fargate instances will continue to be detected as OnPrem.

While I appreciate having a work-around, the patch seems strange to me. You've added an undocumented env var to fix behaviour that should be a straight-forward command-line option or automatic. Why not look at one of the existing variables like AWS_EXECUTION_ENV? It can be either AWS_ECS_FARGATE or AWS_ECS_EC2`

[ThisIsQasim]

You are right in assuming this to be a workaround and not the ideal solution. Ideally, it should detect Fargate automatically without extra flags.

However, as per my limited knowledge, there is no metadata service of any kind on EKS Fargate nor are any AWS specific Env Vars injected. This makes it very difficult to automatically determine if the agent is running on EKS Fargate or OnPrem k8s. So the only solution, again as per my limited knowledge, was to add a flag that forces EC2 mode.

Fargate on ECS is different but I have never used it. Feel free to do it properly and make a PR. The maintainers sound like helpful people and I am sure they'll take up your contributions.

[github-actions]

This issue was marked stale due to lack of activity.

[jhnlsn]

I would like to revisit this issue and look at our credentials provider implementation. The default credentials provider for the aws sdk is able to handle being on Fargate vs k8s native cluster and move through different modes of credentials in order. I believe we can also take this approach with the agent.

[github-actions]

This issue was marked stale due to lack of activity.

[ThisIsQasim]

This is still a pending issue

[github-actions]

This issue was marked stale due to lack of activity.

[ThisIsQasim]

I believe this still needs attention

[khanhntd]

Hey @rs-garrick, this would be a short notice from me. However, would you able to build the image from source by using make dockerized-build and confirm this image works on your ECS Fargate. Moreover, for @ThisIsQasim, please help me in creating a separate issue even though its still addresses the same problem but on different platform. Thanks in advance for this