AWS Node Termination Handler has the ability to emit a Kubernetes event every time an interruption signal is sent from AWS and also every time an operation is attempted on a node. More information on how to get events can be found here.
There are two relevant parameters:
-
emit-kubernetes-events
:If true, Kubernetes events will be emitted when interruption events are received and when actions are taken on Kubernetes nodes. Defaults to
false
. -
kubernetes-events-extra-annotations
:A comma-separated list of
key=value
extra annotations to attach to all emitted Kubernetes events. Example:"first=annotation,sample.annotation/number=two"
There are a number of events that can be emitted, each one with a reason that can be used to quickly identify the event nature and for filtering. Each event will also have a message with extended information. Here's a reasons summary:
AWS interruption event reasons:
RebalanceRecommendation
ScheduledEvent
SQSTermination
SpotInterruption
Node action reasons:
Cordon
CordonError
CordonAndDrain
CordonAndDrainError
PreDrain
PreDrainError
PostDrain
PostDrainError
Uncordon
UncordonError
MonitorError
If emit-kubernetes-events
is enabled and enable-sqs-termination-draining
is disabled (meaning we're operating in IMDS mode), AWS Node Termination Handler will automatically inject a set of annotations to each event it emits. Such annotations are gathered from the underlying host's IMDS endpoint and enrich each event with information about the host that emitted it.
NOTE: In Queue Processor mode, the default IMDS mode annotations will be disabled but you can still define a set of extra annotations.
The default IMDS mode annotations are:
Name | Example value |
---|---|
account-id |
123456789012 |
availability-zone |
us-west-2a |
instance-id |
i-abcdef12345678901 |
instance-life-cycle |
spot |
instance-type |
m5.8xlarge |
local-hostname |
ip-10-1-2-3.us-west-2.compute.internal |
local-ipv4 |
10.1.2.3 |
public-hostname |
my-example.host.net |
public-ipv4 |
42.42.42.42 |
region |
us-west-2 |
If kubernetes-events-extra-annotations
are specified they will be appended to the above. In case of collision, the user-defined annotation wins.
All events are about Kubernetes Node
objects so they belong in the default
namespace. The event source is aws-node-termination-handler
. From command line, use kubectl
to get the events as follows:
kubectl get events --field-selector "source=aws-node-termination-handler"
To narrow down the search you can use multiple field selectors, like:
kubectl get events --field-selector "reason=SpotInterruption,involvedObject.name=ip-10-1-2-3.us-west-2.compute.internal"
Results can also be printed out in JSON or YAML format and piped to processors like jq
or yq
. Then, the above annotations can also be used for discovery and filtering.