Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Undesired metrics added #1197

Open
ecerulm opened this issue Jun 4, 2024 · 2 comments
Open

Undesired metrics added #1197

ecerulm opened this issue Jun 4, 2024 · 2 comments
Labels
Stale

Comments

@ecerulm
Copy link
Contributor

ecerulm commented Jun 4, 2024

Describe the bug

I set amazon-cloudwatch-agent to only grab a single prometheus metric java_lang_memory_heapmemoryusage_used

              "metric_selectors": [
                "^java_lang_memory_heapmemoryusage_used"
              ]

but amazon-cloudwatch-agent is adding the following undesired metrics to the PutLogEvents

  • jmx_scrape_cached_beans
  • jmx_scrape_duration_seconds
  • jmx_scrape_error

CloudWatch log event:

{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent/Prometheus",
            "Dimensions": [
                [
                    "service",
                    "service_instance",
                    "ClusterName",
                    "host",
                    "job",
                    "prom_metric_type",
                    "instance",
                    "node",
                    "application"
                ]
            ],
            "Metrics": [
                {
                    "Name": "java_lang_memory_heapmemoryusage_used",
                    "Unit": "Bytes"
                },
                {
                    "Name": "jmx_scrape_cached_beans"
                },
                {
                    "Name": "jmx_scrape_duration_seconds"
                },
                {
                    "Name": "jmx_scrape_error"
                }
            ]
        }
    ],
    "ClusterName": "tableau-dp2",
    "Timestamp": "1717502587825",
    "Version": "0",
    "application": "Tableau",
    "host": "ip-10-26-128-105",
    "instance": "127.0.0.1:12302",
    "job": "jmx",
    "node": "node1",
    "prom_metric_type": "gauge",
    "service": "vizqlservice",
    "service_instance": "2",
    "java_lang_memory_heapmemoryusage_used": 506484968,
    "jmx_scrape_cached_beans": 0,
    "jmx_scrape_duration_seconds": 0.057368237,
    "jmx_scrape_error": 0
}

the .CloudWatchMetrics.Metrics contains 4 elements where I only expected one.

I in principle don't want jmx_scrape_cached_beans, jmx_scrape_duration_seconds or jmx_scrape_error as a cloudwatch metric at all (specially I don't want to pay for those) .

I believe it's a bug to be sending those since I explicitly set only java_lang_memory_heapmemoryusage_used unless I misunderstood the configuration , if so , is there any way to prevent those jmx_xxx metrics to be included?

Steps to reproduce
If possible, provide a recipe for reproducing the error.

What did you expect to see?
A clear and concise description of what you expected to see.

What did you see instead?
A clear and concise description of what you saw instead.

What version did you use?
Version: Version: CWAgent/1.300039.0b612 (go1.22.2; linux; amd64)

What config did you use?
config.json


{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root",
    "debug": true
  },
  "metrics": {
    "aggregation_dimensions": [
      [
        "InstanceId"
      ]
    ],
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "collectd": {
        "metrics_aggregation_interval": 60
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "metrics_collection_interval": 60,
        "totalcpu": true
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ]
      },
      "diskio": {
        "measurement": [
          "io_time",
          "write_bytes",
          "read_bytes",
          "writes",
          "reads"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "netstat": {
        "measurement": [
          "tcp_established",
          "tcp_time_wait"
        ],
        "metrics_collection_interval": 60
      },
      "statsd": {
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ],
        "metrics_collection_interval": 60
      }
    }
  },
  "logs": {
    "metrics_collected": {
      "prometheus": {
        "cluster_name": "tableau-dp2",
        "log_group_name": "tableau-dp2",
        "prometheus_config_path": "/opt/aws/amazon-cloudwatch-agent/etc/prometheus.yaml",
        "emf_processor": {
          "metric_declaration_dedup": true,
          "metric_namespace": "CWAgent/Prometheus",
          "metric_unit": {
            "java_lang_memory_heapmemoryusage_used": "Bytes"
          },
          "metric_declaration": [
            {
              "source_labels": ["node"],
              "label_matcher": "*",
              "dimensions": [
                [
                  "ClusterName",
                  "node",
                  "application",
                  "service",
                  "service_instance"
                ]
              ],
              "metric_selectors": [
                "^java_lang_memory_heapmemoryusage_used"
              ]
            }
          ]
        }
      }
    },
    "force_flush_interval": 5
  }
}

prometheus.yaml

global:
  scrape_interval: 1m
  scrape_timeout: 10s
scrape_configs:
  - job_name: jmx
    sample_limit: 10000
    file_sd_configs:
      - files: ["/opt/aws/amazon-cloudwatch-agent/etc/prometheus_sd_jmx.yaml"]

prometheus_sd_jmx.yaml

- targets:
  - 127.0.0.1:12300
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "0"
    node: node1
- targets:
  - 127.0.0.1:12301
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "1"
    node: node1
- targets:
  - 127.0.0.1:12302
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "2"
    node: node1
- targets:
  - 127.0.0.1:12303
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "3"
    node: node1

Environment
OS: (e.g., "Ubuntu 20.04")

Additional context
Add any other context about the problem here.
@ecerulm
Copy link
Contributor Author

ecerulm commented Jun 14, 2024 

curl -s localhost:12300/metrics
# HELP catalina_globalrequestprocessor_bytesreceived Amount of data received, in bytes Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=bytesReceived
# TYPE catalina_globalrequestprocessor_bytesreceived untyped
catalina_globalrequestprocessor_bytesreceived{name="\"http-nio-8033\""} 1280106.0
# HELP catalina_globalrequestprocessor_bytessent Amount of data sent, in bytes Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=bytesSent
# TYPE catalina_globalrequestprocessor_bytessent untyped
catalina_globalrequestprocessor_bytessent{name="\"http-nio-8033\""} 7.28018507E8
# HELP catalina_globalrequestprocessor_errorcount Number of errors Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=errorCount
# TYPE catalina_globalrequestprocessor_errorcount untyped
catalina_globalrequestprocessor_errorcount{name="\"http-nio-8033\""} 58.0
# HELP catalina_globalrequestprocessor_maxtime Maximum time to process a request Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=maxTime
# TYPE catalina_globalrequestprocessor_maxtime untyped
catalina_globalrequestprocessor_maxtime{name="\"http-nio-8033\""} 31332.0
# HELP catalina_globalrequestprocessor_processingtime Total time to process the requests Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=processingTime
# TYPE catalina_globalrequestprocessor_processingtime untyped
catalina_globalrequestprocessor_processingtime{name="\"http-nio-8033\""} 2083689.0
# HELP catalina_globalrequestprocessor_requestcount Number of requests processed Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=requestCount
# TYPE catalina_globalrequestprocessor_requestcount untyped
catalina_globalrequestprocessor_requestcount{name="\"http-nio-8033\""} 5729.0
# HELP catalina_manager_activesessions Number of active sessions at this moment Catalina:name=null,type=Manager,attribute=activeSessions
# TYPE catalina_manager_activesessions untyped
catalina_manager_activesessions{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_duplicates Number of duplicated session ids generated Catalina:name=null,type=Manager,attribute=duplicates
# TYPE catalina_manager_duplicates untyped
catalina_manager_duplicates{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_expiredsessions Number of sessions that expired ( doesn't include explicit invalidations ) Catalina:name=null,type=Manager,attribute=expiredSessions
# TYPE catalina_manager_expiredsessions untyped
catalina_manager_expiredsessions{context="/vizql",host="localhost"} 203.0
# HELP catalina_manager_maxactive Maximum number of active sessions so far Catalina:name=null,type=Manager,attribute=maxActive
# TYPE catalina_manager_maxactive untyped
catalina_manager_maxactive{context="/vizql",host="localhost"} 59.0
# HELP catalina_manager_maxactivesessions The maximum number of active Sessions allowed, or -1                        for no limit Catalina:name=null,type=Manager,attribute=maxActiveSessions
# TYPE catalina_manager_maxactivesessions untyped
catalina_manager_maxactivesessions{context="/vizql",host="localhost"} -1.0
# HELP catalina_manager_persistauthentication Indicates whether sessions shall persist authentication information when being persisted (e.g. across application restarts). Catalina:name=null,type=Manager,attribute=persistAuthentication
# TYPE catalina_manager_persistauthentication untyped
catalina_manager_persistauthentication{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_processexpiresfrequency The frequency of the manager checks (expiration and passivation) Catalina:name=null,type=Manager,attribute=processExpiresFrequency
# TYPE catalina_manager_processexpiresfrequency untyped
catalina_manager_processexpiresfrequency{context="/vizql",host="localhost"} 6.0
# HELP catalina_manager_processingtime Time spent doing housekeeping and expiration Catalina:name=null,type=Manager,attribute=processingTime
# TYPE catalina_manager_processingtime untyped
catalina_manager_processingtime{context="/vizql",host="localhost"} 21.0
# HELP catalina_manager_rejectedsessions Number of sessions we rejected due to maxActive being reached Catalina:name=null,type=Manager,attribute=rejectedSessions
# TYPE catalina_manager_rejectedsessions untyped
catalina_manager_rejectedsessions{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionaveragealivetime Average time an expired session had been alive Catalina:name=null,type=Manager,attribute=sessionAverageAliveTime
# TYPE catalina_manager_sessionaveragealivetime untyped
catalina_manager_sessionaveragealivetime{context="/vizql",host="localhost"} 18.0
# HELP catalina_manager_sessioncounter Total number of sessions created by this manager Catalina:name=null,type=Manager,attribute=sessionCounter
# TYPE catalina_manager_sessioncounter untyped
catalina_manager_sessioncounter{context="/vizql",host="localhost"} 202.0
# HELP catalina_manager_sessioncreaterate Session creation rate in sessions per minute Catalina:name=null,type=Manager,attribute=sessionCreateRate
# TYPE catalina_manager_sessioncreaterate untyped
catalina_manager_sessioncreaterate{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionexpirerate Session expiration rate in sessions per minute Catalina:name=null,type=Manager,attribute=sessionExpireRate
# TYPE catalina_manager_sessionexpirerate untyped
catalina_manager_sessionexpirerate{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionmaxalivetime Longest time an expired session had been alive Catalina:name=null,type=Manager,attribute=sessionMaxAliveTime
# TYPE catalina_manager_sessionmaxalivetime untyped
catalina_manager_sessionmaxalivetime{context="/vizql",host="localhost"} 2213.0
# HELP catalina_manager_warnonsessionattributefilterfailure Should a WARN level log message be generated if a session attribute fails to match sessionAttributeNameFilter or sessionAttributeClassNameFilter? Catalina:name=null,type=Manager,attribute=warnOnSessionAttributeFilterFailure
# TYPE catalina_manager_warnonsessionattributefilterfailure untyped
catalina_manager_warnonsessionattributefilterfailure{context="/vizql",host="localhost"} 0.0
# HELP java_lang_classloading_loadedclasscount java.lang:name=null,type=ClassLoading,attribute=LoadedClassCount
# TYPE java_lang_classloading_loadedclasscount untyped
java_lang_classloading_loadedclasscount 40878.0
# HELP java_lang_classloading_totalloadedclasscount java.lang:name=null,type=ClassLoading,attribute=TotalLoadedClassCount
# TYPE java_lang_classloading_totalloadedclasscount untyped
java_lang_classloading_totalloadedclasscount 43298.0
# HELP java_lang_classloading_unloadedclasscount java.lang:name=null,type=ClassLoading,attribute=UnloadedClassCount
# TYPE java_lang_classloading_unloadedclasscount untyped
java_lang_classloading_unloadedclasscount 2420.0
# HELP java_lang_classloading_verbose java.lang:name=null,type=ClassLoading,attribute=Verbose
# TYPE java_lang_classloading_verbose untyped
java_lang_classloading_verbose 0.0
# HELP java_lang_g1_old_generation_collectioncount java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=CollectionCount
# TYPE java_lang_g1_old_generation_collectioncount untyped
java_lang_g1_old_generation_collectioncount{type="GarbageCollector"} 0.0
# HELP java_lang_g1_old_generation_collectiontime java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=CollectionTime
# TYPE java_lang_g1_old_generation_collectiontime untyped
java_lang_g1_old_generation_collectiontime{type="GarbageCollector"} 0.0
# HELP java_lang_g1_old_generation_valid java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=Valid
# TYPE java_lang_g1_old_generation_valid untyped
java_lang_g1_old_generation_valid{type="GarbageCollector"} 1.0
# HELP java_lang_g1_young_generation_collectioncount java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=CollectionCount
# TYPE java_lang_g1_young_generation_collectioncount untyped
java_lang_g1_young_generation_collectioncount{type="GarbageCollector"} 139.0
# HELP java_lang_g1_young_generation_collectiontime java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=CollectionTime
# TYPE java_lang_g1_young_generation_collectiontime untyped
java_lang_g1_young_generation_collectiontime{type="GarbageCollector"} 4233.0
# HELP java_lang_g1_young_generation_lastgcinfo_duration CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=duration
# TYPE java_lang_g1_young_generation_lastgcinfo_duration untyped
java_lang_g1_young_generation_lastgcinfo_duration{type="GarbageCollector"} 5.0
# HELP java_lang_g1_young_generation_lastgcinfo_endtime CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=endTime
# TYPE java_lang_g1_young_generation_lastgcinfo_endtime untyped
java_lang_g1_young_generation_lastgcinfo_endtime{type="GarbageCollector"} 8.6729938E7
# HELP java_lang_g1_young_generation_lastgcinfo_gcthreadcount CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=GcThreadCount
# TYPE java_lang_g1_young_generation_lastgcinfo_gcthreadcount untyped
java_lang_g1_young_generation_lastgcinfo_gcthreadcount{type="GarbageCollector"} 5.0
# HELP java_lang_g1_young_generation_lastgcinfo_id CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=id
# TYPE java_lang_g1_young_generation_lastgcinfo_id untyped
java_lang_g1_young_generation_lastgcinfo_id{type="GarbageCollector"} 139.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=committed
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 4849664.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.161536E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 8.0412672E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="Compressed Class Space",type="GarbageCollector"} 3.2243712E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Eden Space",type="GarbageCollector"} 7.56023296E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Old Gen",type="GarbageCollector"} 6.6060288E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Survivor Space",type="GarbageCollector"} 5242880.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="Metaspace",type="GarbageCollector"} 2.42782208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=init
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="Compressed Class Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Eden Space",type="GarbageCollector"} 2.7262976E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Old Gen",type="GarbageCollector"} 3.9845888E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Survivor Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="Metaspace",type="GarbageCollector"} 0.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=max
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 8183808.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="Compressed Class Space",type="GarbageCollector"} 1.073741824E9
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Eden Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Old Gen",type="GarbageCollector"} 8.355053568E9
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Survivor Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="Metaspace",type="GarbageCollector"} -1.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=used
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 1952128.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.1590016E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 7.8974208E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="Compressed Class Space",type="GarbageCollector"} 2.8333752E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Eden Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Old Gen",type="GarbageCollector"} 5.0559672E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Survivor Space",type="GarbageCollector"} 5242880.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="Metaspace",type="GarbageCollector"} 2.30296208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=committed
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 4849664.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.161536E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 8.0412672E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="Compressed Class Space",type="GarbageCollector"} 3.2243712E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Eden Space",type="GarbageCollector"} 7.66509056E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Old Gen",type="GarbageCollector"} 6.53262848E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Survivor Space",type="GarbageCollector"} 2097152.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="Metaspace",type="GarbageCollector"} 2.42782208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=init
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="Compressed Class Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Eden Space",type="GarbageCollector"} 2.7262976E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Old Gen",type="GarbageCollector"} 3.9845888E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Survivor Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="Metaspace",type="GarbageCollector"} 0.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=max
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 8183808.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="Compressed Class Space",type="GarbageCollector"} 1.073741824E9
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Eden Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Old Gen",type="GarbageCollector"} 8.355053568E9
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Survivor Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="Metaspace",type="GarbageCollector"} -1.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=used
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 1952128.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.1590016E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 7.8974208E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="Compressed Class Space",type="GarbageCollector"} 2.8333752E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Eden Space",type="GarbageCollector"} 2.81018368E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Old Gen",type="GarbageCollector"} 5.055452E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Survivor Space",type="GarbageCollector"} 2097152.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="Metaspace",type="GarbageCollector"} 2.30296208E8
# HELP java_lang_g1_young_generation_lastgcinfo_starttime CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=startTime
# TYPE java_lang_g1_young_generation_lastgcinfo_starttime untyped
java_lang_g1_young_generation_lastgcinfo_starttime{type="GarbageCollector"} 8.6729933E7
# HELP java_lang_g1_young_generation_valid java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=Valid
# TYPE java_lang_g1_young_generation_valid untyped
java_lang_g1_young_generation_valid{type="GarbageCollector"} 1.0
# HELP java_lang_memory_heapmemoryusage_committed java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=committed
# TYPE java_lang_memory_heapmemoryusage_committed untyped
java_lang_memory_heapmemoryusage_committed 1.421869056E9
# HELP java_lang_memory_heapmemoryusage_init java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=init
# TYPE java_lang_memory_heapmemoryusage_init untyped
java_lang_memory_heapmemoryusage_init 6.7108864E7
# HELP java_lang_memory_heapmemoryusage_max java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=max
# TYPE java_lang_memory_heapmemoryusage_max untyped
java_lang_memory_heapmemoryusage_max 8.355053568E9
# HELP java_lang_memory_heapmemoryusage_used java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=used
# TYPE java_lang_memory_heapmemoryusage_used untyped
java_lang_memory_heapmemoryusage_used 9.10347056E8
# HELP java_lang_threading_currentthreadallocatedbytes java.lang:name=null,type=Threading,attribute=CurrentThreadAllocatedBytes
# TYPE java_lang_threading_currentthreadallocatedbytes untyped
java_lang_threading_currentthreadallocatedbytes 8136312.0
# HELP java_lang_threading_currentthreadcputime java.lang:name=null,type=Threading,attribute=CurrentThreadCpuTime
# TYPE java_lang_threading_currentthreadcputime untyped
java_lang_threading_currentthreadcputime 7.6256619E7
# HELP java_lang_threading_currentthreadcputimesupported java.lang:name=null,type=Threading,attribute=CurrentThreadCpuTimeSupported
# TYPE java_lang_threading_currentthreadcputimesupported untyped
java_lang_threading_currentthreadcputimesupported 1.0
# HELP java_lang_threading_currentthreadusertime java.lang:name=null,type=Threading,attribute=CurrentThreadUserTime
# TYPE java_lang_threading_currentthreadusertime untyped
java_lang_threading_currentthreadusertime 6.0E7
# HELP java_lang_threading_daemonthreadcount java.lang:name=null,type=Threading,attribute=DaemonThreadCount
# TYPE java_lang_threading_daemonthreadcount untyped
java_lang_threading_daemonthreadcount 399.0
# HELP java_lang_threading_objectmonitorusagesupported java.lang:name=null,type=Threading,attribute=ObjectMonitorUsageSupported
# TYPE java_lang_threading_objectmonitorusagesupported untyped
java_lang_threading_objectmonitorusagesupported 1.0
# HELP java_lang_threading_peakthreadcount java.lang:name=null,type=Threading,attribute=PeakThreadCount
# TYPE java_lang_threading_peakthreadcount untyped
java_lang_threading_peakthreadcount 679.0
# HELP java_lang_threading_synchronizerusagesupported java.lang:name=null,type=Threading,attribute=SynchronizerUsageSupported
# TYPE java_lang_threading_synchronizerusagesupported untyped
java_lang_threading_synchronizerusagesupported 1.0
# HELP java_lang_threading_threadallocatedmemoryenabled java.lang:name=null,type=Threading,attribute=ThreadAllocatedMemoryEnabled
# TYPE java_lang_threading_threadallocatedmemoryenabled untyped
java_lang_threading_threadallocatedmemoryenabled 1.0
# HELP java_lang_threading_threadallocatedmemorysupported java.lang:name=null,type=Threading,attribute=ThreadAllocatedMemorySupported
# TYPE java_lang_threading_threadallocatedmemorysupported untyped
java_lang_threading_threadallocatedmemorysupported 1.0
# HELP java_lang_threading_threadcontentionmonitoringenabled java.lang:name=null,type=Threading,attribute=ThreadContentionMonitoringEnabled
# TYPE java_lang_threading_threadcontentionmonitoringenabled untyped
java_lang_threading_threadcontentionmonitoringenabled 0.0
# HELP java_lang_threading_threadcontentionmonitoringsupported java.lang:name=null,type=Threading,attribute=ThreadContentionMonitoringSupported
# TYPE java_lang_threading_threadcontentionmonitoringsupported untyped
java_lang_threading_threadcontentionmonitoringsupported 1.0
# HELP java_lang_threading_threadcount java.lang:name=null,type=Threading,attribute=ThreadCount
# TYPE java_lang_threading_threadcount untyped
java_lang_threading_threadcount 670.0
# HELP java_lang_threading_threadcputimeenabled java.lang:name=null,type=Threading,attribute=ThreadCpuTimeEnabled
# TYPE java_lang_threading_threadcputimeenabled untyped
java_lang_threading_threadcputimeenabled 1.0
# HELP java_lang_threading_threadcputimesupported java.lang:name=null,type=Threading,attribute=ThreadCpuTimeSupported
# TYPE java_lang_threading_threadcputimesupported untyped
java_lang_threading_threadcputimesupported 1.0
# HELP java_lang_threading_totalstartedthreadcount java.lang:name=null,type=Threading,attribute=TotalStartedThreadCount
# TYPE java_lang_threading_totalstartedthreadcount untyped
java_lang_threading_totalstartedthreadcount 5614.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 1.0
# HELP jmx_exporter_build_info JMX Exporter build information
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{name="jmx_prometheus_httpserver",version="1.0.1"} 1
# HELP jmx_scrape_cached_beans Number of beans with their matching rule cached
# TYPE jmx_scrape_cached_beans gauge
jmx_scrape_cached_beans 0.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.008187466
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0

as you can see there is a lot of jmx_* metrics but for some reason amazon-cloudwatch-agent is grabbing only

  • jmx_scrape_cached_beans
  • jmx_scrape_duration_seconds
  • jmx_scrape_error

although I explicitly said I don't want any of those jmx_ since I used "^java_lang_memory_heapmemoryusage_used" as metric_selectors

I principle I expected only to see java_lang_memory_heapmemoryusage_used as the only metric exported to CW.

@github-actions GitHub Actions
Copy link
Contributor

This issue was marked stale due to lack of activity.

@github-actions github-actions bot added the Stale label Sep 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ecerulm