Not able to tag autoscaling group created by EKS Managed Node Group

[yasinlachiny]

Description

I have an issue like #2128. The provided document does not exist in the master branch and actually, it doesn't work.

I'm trying to add tag for ASG , it is working when all the resources are already created. but when I am trying to run it from scratch I am getting this error

│ Error: Invalid for_each argument
│ 
│   on main.tf line 62, in resource "aws_autoscaling_group_tag" "this":
│   62:   for_each = { for tag in flatten([
│   63:     for asg in flatten([
│   64:       # Unpack autoscaling group name from EKS node group
│   65:       for resources in module.eks.eks_managed_node_groups : resources.node_group_resources[0].autoscaling_groups[0]
│   66:       ]) : [
│   67:       # Map each tag in `tags` to each autoscaling group in EKS node group => returns list of maps
│   68:       for k, v in module.tags.tags : { asg = asg.name, key = k, val = v }
│   69:     ]
│   70:   ]) : "${tag.asg}-${tag.key}" => { asg = tag.asg, key = tag.key, val = tag.val } }
│     ├────────────────
│     │ module.eks.eks_managed_node_groups is object with 4 attributes
│     │ module.tags.tags is map of string with 19 elements
│ 
│ The "for_each" map includes keys derived from resource attributes that
│ cannot be determined until apply, and so Terraform cannot determine the
│ full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map
│ keys statically in your configuration and place apply-time results only in
│ the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply
│ only the resources that the for_each value depends on, and then apply a
│ second time to fully converge.

and this is my code:

module "eks" {
...
}

resource "aws_autoscaling_group_tag" "this" {
  # Build map of maps to iterate over = `for_each` won't take a list of maps here
  for_each = { for tag in flatten([
    for asg in flatten([
      # Unpack autoscaling group name from EKS node group
      for resources in module.eks.eks_managed_node_groups : resources.node_group_resources[0].autoscaling_groups[0]
      ]) : [
      # Map each tag in `tags` to each autoscaling group in EKS node group => returns list of maps
      for k, v in module.tags.tags : { asg = asg.name, key = k, val = v }
    ]
  ]) : "${tag.asg}-${tag.key}" => { asg = tag.asg, key = tag.key, val = tag.val } }

  autoscaling_group_name = each.value.asg

  tag {
    key                 = each.value.key
    value               = each.value.val
    propagate_at_launch = true
  }
  depends_on = [
    module.eks
  ]
 }

It's working well for the currently deployed node_group but for the new node group it throughs an error.
It may because of the well known issue of unknown values used in a for_each loop.

If your request is for a new feature, please use the Feature request template.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

I tested it with 18.29 and 19.6.0

• Module version [Required]:

• Terraform version:

• Provider version(s):

Expected behavior

It's working fine for deployed node_group and I expected that it works fine for new node group

Actual behavior

I got an error

│ The "for_each" map includes keys derived from resource attributes that
│ cannot be determined until apply, and so Terraform cannot determine the
│ full set of keys that will identify the instances of this resource.
│ 

[bryantbiggs]

this is a lifecycle issue - you cannot add this tag logic until the node groups exist. this is why the module does not offer this logic natively; it has to be added after the nodegroups are created

[wcarlsen]

@yasinlachiny you should be able to solve your issue if you where to not use "${tag.asg}-${tag.key}" as key replaced the unknown tag.asg with something node group name related. We do simple node group labels and node group taints propagation to the ASG's as tag and we made it work in this way.

[yasinlachiny]

@wcarlsen
Thank you for your response. that makes sense and I'll look at it.
meanwhile is it possible to share your code related to this issue?
It helps me a lot:)

[wcarlsen]

The sha256 stuff is totally unnecessary and can be skipped though

locals {
  node_label_tag_prefix = "k8s.io/cluster-autoscaler/node-template/label/"
  node_taint_tag_prefix = "k8s.io/cluster-autoscaler/node-template/taint/"

  node_labels = flatten([for k, v in module.eks.eks_managed_node_groups : [
    for l, w in v.node_group_labels : {
      sha : sha256("${k}${l}")
      name : v.node_group_autoscaling_group_names[0]
      key : l
      value : w
    }
  ] if length(v.node_group_labels) > 0])

  node_taints = flatten([for k, v in module.eks.eks_managed_node_groups : [
    for l, w in v.node_group_taints : {
      sha : sha256("${k}${w.key}")
      name : v.node_group_autoscaling_group_names[0]
      key : w.key
      value : w.value
      effect : w.effect
    }
  ] if length(v.node_group_taints) > 0])
}

resource "aws_autoscaling_group_tag" "labels" {
  for_each = {
    for k, v in local.node_labels :
    v.sha => v
  }
  autoscaling_group_name = each.value.name
  tag {
    key                 = "${local.node_label_tag_prefix}${each.value.key}"
    propagate_at_launch = true
    value               = each.value.value
  }
}

resource "aws_autoscaling_group_tag" "taints" {
  for_each = {
    for k, v in local.node_taints :
    v.sha => v
  }
  autoscaling_group_name = each.value.name
  tag {
    key                 = "${local.node_taint_tag_prefix}${each.value.key}"
    propagate_at_launch = true
    value               = "${each.value.value}:${each.value.effect}"
  }
}

[wcarlsen]

@bryantbiggs I know above example is super specific for the cluster-autoscaler, but would the above approach make sense in some form to include for managed node groups. I would be happy to make an attempt, if you'd like?

[bryantbiggs]

This has already been tried and exhausted - its not possible at the moment #1558 (comment)

[bryantbiggs]

if you want tags on managed nodegroups, use the module as its defined with the custom launch template and the tags will be added correctly

[wcarlsen]

@bryantbiggs sorry for being a bit persistent, but did you notice how the key in the loop above is 100% predictable, because it uses a combination of node group name and label key instead of use an asg related key? Doesn't that make a difference?

[bryantbiggs]

Nodegroup names are not a known entity, users can utilize the prefix option which is randomly generated by Terraform

[wcarlsen]

Sorry I meant the key in module.eks.eks_managed_node_groups which is known, since it is user defined.

[bryantbiggs]

I mean, you are free to experiment and try out solutions but I fear we've already exhausted them.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.