Description
The amazon-codeguru-profiler-python-agent fails to get EC2 instance metadata from the IMDSv2 api when running on EC2 instances due to using a GET request to fetch the token when a PUT request is required. I don't really know what the impact is because as the logs note: Unable to get Ec2 instance metadata, this is normal when running in a different environment (e.g. Fargate), profiler will still work
. So the profiler seems to still work on EC2 instances but just doesn't have the EC2 instance metadata.
Simplified program to reproduce issue:
import logging
import time
from codeguru_profiler_agent import Profiler
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG)
Profiler(profiling_group_name='MyProfilingGroup').start()
time.sleep(30)
Relevant logs from running the program on an EC2 instance with IMDSv2 with python 3.7 and codeguru-profiler-agent 1.2.4:
...
DEBUG:codeguru_profiler_agent.agent_metadata.fleet_info:Making a request to http://169.254.169.254/latest/api/token with headers set for these keys: dict_keys(['X-aws-ec2-metadata-token-ttl-seconds'])
INFO:codeguru_profiler_agent.agent_metadata.aws_ec2_instance:Unable to get Ec2 instance metadata, this is normal when running in a different environment (e.g. Fargate), profiler will still work
DEBUG:codeguru_profiler_agent.agent_metadata.aws_ec2_instance:Caught exception:
Traceback (most recent call last):
File "/opt/env/lib/python3.7/site-packages/codeguru_profiler_agent/agent_metadata/aws_ec2_instance.py", line 68, in look_up_metadata
token = cls.__look_up_ec2_api_token()
File "/opt/env/lib/python3.7/site-packages/codeguru_profiler_agent/agent_metadata/aws_ec2_instance.py", line 62, in __look_up_ec2_api_token
headers={EC2_METADATA_TOKEN_TTL_HEADER_KEY: EC2_METADATA_TOKEN_TTL_HEADER_VALUE}) \
File "/opt/env/lib/python3.7/site-packages/codeguru_profiler_agent/agent_metadata/fleet_info.py", line 26, in http_get
return request.urlopen(req, timeout=METADATA_URI_TIMEOUT_SECONDS) # nosec
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/local/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 405: Not Allowed
...
Looks like the issue was introduced in this PR here: #24 and the test didn't catch it because the mock doesn't doesn't match the IMDSv2 API responses.
For the instance metadata docs see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#instance-metadata-v2-how-it-works and https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html where it notes only a PUT request is allowed for fetching the token while the agent uses a GET request: https://github.com/aws/amazon-codeguru-profiler-python-agent/blob/main/codeguru_profiler_agent/agent_metadata/aws_ec2_instance.py#L61 which is why the request fails and returns HTTP Error 405: Not Allowed
.
So the fix looks to be change the agent to use a PUT request to get the token and fix the corresponding test.