Add EMS message

[wooyoungAhn]

Hello
good morning

I changed the settings as below to add EMS alert and restarted harvest, but EMS alert could not be imported.
However, there are no errors left in the poller log, making it difficult to troubleshoot.
Could you share how to find trouble?

poller log

[root@ ~]# cat /var/log/harvest/poller_nas0304.log | grep ems
{"level":"info","Poller":"nas0304","collector":"Ems:Ems","path":"conf/ems/9.6.0/custom_ems.yaml","v":"9.11.1","caller":"collector/helpers.go:139","time":"2023-07-19T06:15:26+09:00","message":"best-fit template"}
{"level":"info","Poller":"nas0304","collector":"Ems:Ems","total instances":0,"caller":"ems/ems.go:353","time":"2023-07-19T06:15:29+09:00"}
{"level":"info","Poller":"nas0304","collector":"Ems:Ems","path":"conf/ems/9.6.0/custom_ems.yaml","v":"9.11.1","caller":"collector/helpers.go:139","time":"2023-07-19T06:16:24+09:00","message":"best-fit template"}
{"level":"info","Poller":"nas0304","collector":"Ems:Ems","total instances":0,"caller":"ems/ems.go:353","time":"2023-07-19T06:16:27+09:00"}

I share my config.

Add message catalog

Ontap CLI

event catalog show -severity EMERGENCY,ALERT,ERROR -message-name *callhome*

Message                          Severity         SNMP Trap Type
-------------------------------- ---------------- -----------------
callhome.sfo.takeover.failed     ALERT            Severity-based
...
callhome.wafl.inconsistent.user.block 
                                 ALERT            Severity-based
227 entries were displayed.

dir

[root@localhost ems]# ls -al
drwxr-xr-x.  3 harvest harvest 4096  7.19 06:09 .
drwxr-xr-x. 10 harvest harvest 4096  7.18 12:35 ..
drwxr-xr-x.  2 harvest harvest 4096  7.19 06:10 9.6.0
-rw-r--r--.  1 harvest harvest  207  7.19 06:10 custom.yaml
-rw-r--r--.  1 harvest harvest  200  5.3 21:08 default.yaml

custom.yaml

[root@localhost ems]# cat custom.yaml 
collector: Ems

client_timeout: 1m
schedule:
  - instance: 24h
  - data:     3m

# More information https://github.com/NetApp/harvest/blob/main/cmd/collectors/ems/README.md

objects:
  Ems: custom_ems.yaml

custom_ems.yaml

name:   EMS
query:  api/support/ems/events
object: ems

# default list of exports attached to all EMS event
exports:
  - message.name                            => message
  - node.name                               => node
  - node.uuid                               => node_uuid
  - message.severity                        => severity
  - index                                   => index
  - time                                     => time
 
events:
  - name: LUN.destroy
    exports:
      - parameters.lun_path    => lun_path
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.volume_dsid => volume_ds_id
      - parameters.volume_name => volume

  - name: LUN.offline
    exports:
      - ^^parameters.object_uuid => object_uuid
      - parameters.lun_path      => lun_path
      - parameters.object_type   => object_type
      - parameters.volume_dsid   => volume_ds_id
      - parameters.volume_name   => volume
    resolve_when_ems:
      - name: LUN.online
        resolve_after: 672h

  - name: NVMeNS.destroy
    exports:
      - parameters.NVMeNS_path => path
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.volume_dsid => volume_ds_id
      - parameters.volume_name => volume

  - name: NVMeNS.offline
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.path        => path
      - parameters.volume_dsid => volume_ds_id
      - parameters.volume_name => volume

  - name: NVMeNS.online
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.path        => path
      - parameters.volume_dsid => volume_ds_id
      - parameters.volume_name => volume

  - name: Nblade.cifsManyAuths
    exports:
      - parameters.object_type     => object_type
      - parameters.object_uuid     => object_uuid
      - parameters.remoteIpAddress => remote_ip_address
      - parameters.vsId            => vs_id

  - name: Nblade.cifsMaxOpenSameFile
    exports:
      - parameters.IpAddress   => ip_address
      - parameters.filePath    => file_path
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.port        => port
      - parameters.shareName   => share
      - parameters.vserverName => svm

  - name: Nblade.cifsMaxSessPerUsrConn
    exports:
      - parameters.IpAddress   => ip_address
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.port        => port
      - parameters.userName    => user
      - parameters.vserverName => svm

  - name: Nblade.cifsNbNameConflict
    exports:
      - parameters.IpAddress   => ip_address
      - parameters.nbName      => nb
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid

  - name: Nblade.cifsNoPrivShare
    exports:
      - parameters.clientIp    => client_ip
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.userName    => user
      - parameters.vserverId   => svm_uuid

  - name: Nblade.nfsV4PoolExhaust
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.poolname    => pool

  - name: Nblade.vscanBadUserPrivAccess
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.scannerIp   => scanner_ip
      - parameters.userName    => user
      - parameters.vserverName => svm

  - name: Nblade.vscanConnBackPressure
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.scannerIp   => scanner_ip
      - parameters.vserverName => svm

  - name: Nblade.vscanConnInactive
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.scannerIp   => scanner_ip
      - parameters.vserverName => svm

  - name: Nblade.vscanNoRegdScanner
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.scannerIp   => scanner_ip
      - parameters.vserverName => svm

  - name: Nblade.vscanNoScannerConn
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid
      - parameters.vserverName => svm

  - name: Nblade.vscanVirusDetected
    exports:
      - parameters.SID           => sid
      - parameters.clientIp      => client_ip
      - parameters.filePath      => file_path
      - parameters.object_type   => object_type
      - parameters.object_uuid   => object_uuid
      - parameters.vscanServerIp => vscan_server_ip
      - parameters.vserverName   => svm

  - name: arl.netra.ca.check.failed
    exports:
      - parameters.aggr_uuid => aggr_uuid
      - parameters.vol       => volume

  - name: arw.volume.state
    matches:
      - name: op
        value: disable-in-progress
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.volume.state
    matches:
      - name: op
        value: disabled
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.volume.state
    matches:
      - name: op
        value: dry-run
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.volume.state
    matches:
      - name: op
        value: dry-run-paused
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.volume.state
    matches:
      - name: op
        value: enable-paused
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.volume.state
    matches:
      - name: op
        value: enabled
    exports:
      - parameters.op          => op
      - parameters.volumeName  => volume
      - parameters.volumeUuid  => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.vserver.state
    matches:
      - name: op
        value: disabled
    exports:
      - parameters.op          => op
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: arw.vserver.state
    matches:
      - name: op
        value: dry-run
    exports:
      - parameters.op          => op
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: callhome.arw.activity.seen
    exports:
      - parameters.volName     => volume
      - parameters.volUuid     => volume_uuid
      - parameters.vserverName => svm
      - parameters.vserverUuid => svm_uuid

  - name: callhome.battery.low
    exports:
      - ^^node.name => node
    resolve_when_ems:
      - name: nvram.battery.charging.normal
        resolve_after: 672h

  - name: callhome.hainterconnect.down

  - name: callhome.sp.hbt.missed

  - name: callhome.sp.hbt.stopped

  - name: cifs.shadowcopy.failure
    exports:
      - parameters.clientShadowCopyId    => client_shadow_copy_id
      - parameters.clientShadowCopySetId => client_shadow_copy_set_id
      - parameters.filerShadowCopyId     => filer_shadow_copy_id
      - parameters.filerShadowCopySetId  => filer_shadow_copy_set_id
      - parameters.object_type           => object_type
      - parameters.object_uuid           => object_uuid
      - parameters.operation             => operation
      - parameters.shareName             => share

  - name: cloud.aws.iamNotInitialized
    exports:
      - parameters.nodeUuid => node_uuid

  - name: cluster.switch.pwr.fail
    exports:
      - parameters.pwr_supply_name => pwr_supply
      - parameters.status          => status
      - parameters.switch_name     => switch

  - name: disk.outOfService
    exports:
      - parameters.diskName     => disk
      - parameters.powerOnHours => power_on_hours
      - parameters.serialno     => serial_no

  - name: diskShelf.psu.added
    exports:
      - parameters.channelName  => channel
      - parameters.location     => location
      - parameters.serialNumber => serial_no
      - parameters.shelfIdent   => shelf_ident

  - name: diskShelf.psu.removed
    exports:
      - parameters.channelName  => channel
      - parameters.location     => location
      - parameters.serialNumber => serial_no
      - parameters.shelfIdent   => shelf_ident

  - name: fabricpool.full
    exports:
      - parameters.cluster_uuid      => cluster_uuid
      - parameters.licensed_capacity => licensed_capacity
      - parameters.used_capacity     => used_capacity

  - name: fabricpool.nearly.full
    exports:
      - parameters.cluster_uuid      => cluster_uuid
      - parameters.licensed_capacity => licensed_capacity
      - parameters.used_capacity     => used_capacity

  - name: gb.netra.ca.check.failed
    exports:
      - parameters.aggr_uuid => aggr_uuid
      - parameters.vol       => volume

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: RaidLeftBehindAggrAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: RaidLeftBehindSpareAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: NoISLPresentAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: ClusterSeveredAllLinksAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: StorageBridgePortDown_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: StorageBridgeTempAboveCritical_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: StorageBridgeTempBelowCritical_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: StorageBridgeUnreachable_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: StorageFCAdapterFault_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: InterconnectAdapterOfflineAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: RaidDegradedMirrorAggrAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: InterclusterBrokenConnectionAlert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: SASAdapterOffline_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: FabricSwitchFanFail_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: FabricSwitchPowerFail_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: FabricSwitchTempCritical_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: FabricSwitchTempSensorFailed_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: hm.alert.raised
    matches:
      - name: alert_id
        value: FabricSwitchUnreachable_Alert
    exports:
      - ^^parameters.alert_id              => alert_id
      - parameters.acknowledge             => acknowledge
      - parameters.acknowledger            => acknowledger
      - parameters.alert_corrective_action => alert_corrective_action
      - parameters.alerting_resource       => alerting_resource
      - parameters.asup_enable             => asup_enable
      - parameters.monitor                 => monitor
      - parameters.possible_effect         => possible_effect
      - parameters.suppress                => suppress
      - parameters.suppressor              => suppressor
    resolve_when_ems:
      - name: hm.alert.cleared
        resolve_after: 672h

  - name: mcc.config.auso.stDisabled
    exports:
      - ^^parameters.alertId        => alert_id
      - parameters.alertingResource => alerting_resource
    resolve_when_ems:
      - name: mcc.config.auso.stEnabled
        resolve_after: 672h

  - name: mgmtgwd.rootvolrec.low.space
    exports:
      - parameters.node            => node
      - parameters.threshold_in_mb => threshold_in_mb

  - name: monitor.fan.critical
    exports:
      - ^^parameters.report => report
    resolve_when_ems:
      - name: monitor.fan.ok
        resolve_after: 672h

  - name: monitor.fan.failed
    exports:
      - ^^parameters.report => report
    resolve_when_ems:
      - name: monitor.fan.ok
        resolve_after: 672h

  - name: monitor.fan.warning
    exports:
      - ^^parameters.report => report
    resolve_when_ems:
      - name: monitor.fan.ok
        resolve_after: 672h

  - name: nvmf.graceperiod.active
    exports:
      - parameters.days_remaining  => days_remaining
      - parameters.expiration_date => expiration_date
      - parameters.object_type     => object_type
      - parameters.object_uuid     => object_uuid

  - name: nvmf.graceperiod.expired
    exports:
      - parameters.object_type => object_type
      - parameters.object_uuid => object_uuid

  - name: nvmf.graceperiod.start
    exports:
      - parameters.days_remaining  => days_remaining
      - parameters.expiration_date => expiration_date
      - parameters.object_type     => object_type
      - parameters.object_uuid     => object_uuid

  - name: object.store.unavailable
    exports:
      - ^^parameters.node_uuid => node_uuid
      - parameters.configname  => config
    resolve_when_ems:
      - name: object.store.available
        resolve_after: 672h

  - name: objstore.host.unresolvable
    exports:
      - parameters.hostname => host
      - parameters.nodeUuid => node_uuid

  - name: objstore.interclusterlifDown
    exports:
      - parameters.ipspaceID => ipspace_id
      - parameters.nodeUuid  => node_uuid

  - name: osc.signatureMismatch
    exports:
      - parameters.bucket         => bucket
      - parameters.nodeUuid       => node_uuid
      - parameters.operation      => operation
      - parameters.serverHostname => server_host

  - name: qos.monitor.memory.maxed
    exports:
      - ^^parameters.object_uuid => object_uuid
      - parameters.count         => count
      - parameters.object_type   => object_type
    resolve_when_ems:
      - name: qos.monitor.memory.abated
        resolve_after: 672h

  - name: scsiblade.san.config.active
    exports:
      - parameters.num_luns => num_luns
      - parameters.state    => state

  - name: scsitarget.fct.port.full
    exports:
      - parameters.active_commands => active_commands
      - parameters.max_commands    => max_commands
      - parameters.portname        => port

  - name: scsitarget.fct.sfpRxPowerLow
    exports:
      - parameters.adapter         => adapter
      - parameters.operating_value => operating_value
      - parameters.threshold_value => threshold_value

  - name: scsitarget.fct.sfpTxPowerLow
    exports:
      - parameters.adapter         => adapter
      - parameters.operating_value => operating_value
      - parameters.threshold_value => threshold_value

  - name: ses.status.fanError
    exports:
      - ^^parameters.prodChannel => prod_channel
      - parameters.fanNumber     => fan_number
    resolve_when_ems:
      - name: ses.status.fanInfo
        resolve_after: 672h

  - name: sk.panic

  - name: sm.mediator.added
    exports:
      - parameters.cluster     => cluster
      - parameters.ipAddress   => ip_address
      - parameters.peerCluster => peer_cluster
      - parameters.version     => version

  - name: sm.mediator.cacert.expired
    exports:
      - ^^parameters.ipAddress => ip_address
      - parameters.expiryDate  => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.cacert.expiring
    exports:
      - ^^parameters.ipAddress  => ip_address
      - parameters.daysToExpire => days_to_expire
      - parameters.expiryDate   => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.clientc.expired
    exports:
      - ^^parameters.ipAddress => ip_address
      - parameters.expiryDate  => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.clientc.expiring
    exports:
      - ^^parameters.ipAddress  => ip_address
      - parameters.daysToExpire => days_to_expire
      - parameters.expiryDate   => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.misconfigured
    exports:
      - ^^parameters.ipAddress => ip_address
      - parameters.cluster     => cluster
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.removed
    exports:
      - parameters.cluster     => cluster
      - parameters.ipAddress   => ip_address
      - parameters.peerCluster => peer_cluster
      - parameters.version     => version

  - name: sm.mediator.serverc.expired
    exports:
      - ^^parameters.ipAddress => ip_address
      - parameters.expiryDate  => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.serverc.expiring
    exports:
      - ^^parameters.ipAddress  => ip_address
      - parameters.daysToExpire => days_to_expire
      - parameters.expiryDate   => expiry_date
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sm.mediator.unreachable
    exports:
      - ^^parameters.ipAddress => ip_address
      - parameters.cluster     => cluster
    resolve_when_ems:
      - name: sm.mediator.in.quorum
        resolve_after: 672h

  - name: sms.status.out.of.sync
    exports:
      - ^^parameters.relationship_id => relationship_id
      - parameters.dstpath           => dst_path
      - parameters.srcpath           => src_path
    resolve_when_ems:
      - name: sms.status.in.sync
        resolve_after: 672h

  - name: sms.status.out.of.sync.cg
    exports:
      - parameters.cg_relationship_id => cg_relationship_id
      - parameters.dstcgpath          => dst_cg_path
      - parameters.srccgpath          => src_cg_path

  - name: sp.ipmi.lost.shutdown
    exports:
      - ^^node.name            => node
      - parameters.num_minutes => num_minutes
    resolve_when_ems:
      - name: sp.heartbeat.resumed
        resolve_after: 672h

  - name: sp.notConfigured
    exports:
      - ^^node.name => node
    resolve_when_ems:
      - name: sp.heartbeat.resumed
        resolve_after: 672h

  - name: unowned.disk.reminder
    exports:
      - parameters.count => count

  - name: vserver.stop.succeeded
    exports:
      - ^^parameters.vserver_name => svm
      - parameters.vserver_uuid   => svm_uuid
    resolve_when_ems:
      - name: vserver.start.succeeded
        resolve_after: 672h

  - name: wafl.ca.resync.complete
    exports:
      - parameters.aggr              => aggr
      - parameters.aggr_uuid         => aggr_uuid
      - parameters.mirror_config_id  => mirror_config_id
      - parameters.primary_config_id => primary_config_id

  - name: wafl.readdir.expired
    exports:
      - parameters.app            => app
      - parameters.directory_inum => directory_inum
      - parameters.object_uuid    => object_uuid
      - parameters.snapid         => snap_id
      - parameters.volident       => vol_ident
      - parameters.volume         => volume

  - name: wafl.vol.autoSize.done
    exports:
      - parameters.app         => app
      - parameters.event_type  => event_type
      - parameters.object_uuid => object_uuid
      - parameters.size        => size
      - parameters.vol         => volume
      - parameters.volident    => vol_ident

  - name: wafl.vvol.offline
    exports:
      - ^^parameters.volident => vol_ident
      - parameters.app        => app
      - parameters.instuuid   => inst_uuid
      - parameters.name       => volume
    resolve_when_ems:
      - name: wafl.vvol.online
        resolve_after: 672h

  - name: wafl.vvol.restrict
    exports:
      - ^^parameters.volident => vol_ident
      - parameters.app        => app
      - parameters.instuuid   => inst_uuid
      - parameters.vol        => volume
    resolve_when_ems:
      - name: wafl.vvol.online
        resolve_after: 672h
  - name: callhome.aggr.meta.repair
  - name: callhome.aggr.offline
  - name: callhome.aggr.restricted
  - name: callhome.aggr.wafliron
  - name: callhome.andu.pausederr
  - name: callhome.appdm.snaprest.fail
  - name: callhome.arw.activity.seen
  - name: callhome.bad.ram
  - name: callhome.battery.failure
  - name: callhome.battery.low
  - name: callhome.battery.overchg
  - name: callhome.battery.overtemp
  - name: callhome.battery.warning
  - name: callhome.bcom.vsinit.failed
  - name: callhome.bdfu.suspended
  - name: callhome.bootfs.chkdsk
  - name: callhome.bridge.reboot.unexp
  - name: callhome.c.fan
  - name: callhome.c.fan.fru.degraded
  - name: callhome.c.fan.fru.fault
  - name: callhome.c.fan.fru.rm
  - name: callhome.c.fan.fru.shut
  - name: callhome.carrier.fault
  - name: callhome.ch.ps.degraded
  - name: callhome.ch.ps.fan.bad.xmin
  - name: callhome.ch.ps.fault
  - name: callhome.ch.ps.wrong.in
  - name: callhome.chassis.hitemp
  - name: callhome.chassis.overtemp
  - name: callhome.chassis.power
  - name: callhome.chassis.power.down
  - name: callhome.chassis.ps.degraded
  - name: callhome.chassis.ps.fault
  - name: callhome.chassis.ps.off
  - name: callhome.chassis.ps.replace
  - name: callhome.chassis.ps.rm
  - name: callhome.chassis.undertemp
  - name: callhome.checksum
  - name: callhome.checksum.multiple
  - name: callhome.clam.node.ooq
  - name: callhome.client.app.alert
  - name: callhome.client.app.emerg
  - name: callhome.client.app.err
  - name: callhome.clus.net.degraded
  - name: callhome.clus.vol.cre.fail
  - name: callhome.clus.vol.del.fail
  - name: callhome.config
  - name: callhome.cpu.fan
  - name: callhome.cpu.fan.fault
  - name: callhome.css.toc.repair
  - name: callhome.ctran.gcror.timeout
  - name: callhome.ctran.jpc.splitbrain
  - name: callhome.dafs.nic.stop
  - name: callhome.dbs.cluster.cannot.sync.blocks
  - name: callhome.dbs.cluster.capacity.full
  - name: callhome.dbs.cluster.capacity.nearly.full
  - name: callhome.dbs.data.unavailable
  - name: callhome.dbs.generic.cluster.fault
  - name: callhome.dbs.invalid.block.data
  - name: callhome.dbs.secondary.cache.full
  - name: callhome.dbs.service.capacity.full
  - name: callhome.dbs.service.capacity.nearly.full
  - name: callhome.device.qual
  - name: callhome.disk.carrier.evacuation.notpossible
  - name: callhome.diskown.AssignExceeded
  - name: callhome.disks.missing
  - name: callhome.dr.apply.failed
  - name: callhome.dsk.bypass
  - name: callhome.dsk.config
  - name: callhome.dsk.disable
  - name: callhome.dsk.fault
  - name: callhome.dsk.fw.needed
  - name: callhome.dsk.init
  - name: callhome.dsk.io.degraded
  - name: callhome.dsk.label
  - name: callhome.dsk.label.broken
  - name: callhome.dsk.label.v
  - name: callhome.dsk.missing
  - name: callhome.dsk.no.spin
  - name: callhome.dsk.noio
  - name: callhome.dsk.predict
  - name: callhome.dsk.redun.fault
  - name: callhome.dsk.scrub
  - name: callhome.dsk.scrub.done
  - name: callhome.dsk.scrub.parity
  - name: callhome.dsk.sense
  - name: callhome.dsk.shrank
  - name: callhome.encl.access
  - name: callhome.ext.cache.bad
  - name: callhome.fans.failed
  - name: callhome.fc.al.link
  - name: callhome.fc.al.poor
  - name: callhome.fcp.adapter.failed
  - name: callhome.fcp.sram.dump
  - name: callhome.fdsk.fault
  - name: callhome.fdsk.label
  - name: callhome.fdsk.label.v
  - name: callhome.fdsk.missing
  - name: callhome.fdsk.noio
  - name: callhome.fdsk.shrank
  - name: callhome.flash.cache.degraded
  - name: callhome.flash.cache.failed
  - name: callhome.flash.cache.offlined
  - name: callhome.fs.lun.resized
  - name: callhome.hainterconnect.down
  - name: callhome.hba.failed
  - name: callhome.hm.alert.critical
  - name: callhome.hm.alert.major
  - name: callhome.hm.alert.minor
  - name: callhome.hostport.bypass
  - name: callhome.hostport.disable
  - name: callhome.ibretimerprog.fail
  - name: callhome.iscsi.hba.crash
  - name: callhome.isp.adapter.bad
  - name: callhome.jpc.selection.fail
  - name: callhome.link.change.stuck
  - name: callhome.mcc.auso.2min
  - name: callhome.mcc.auso.complete
  - name: callhome.mcc.auso.trig.fail
  - name: callhome.mcc.clst.dm.wd.gvup
  - name: callhome.mcc.cluster_dm.stream_in_error
  - name: callhome.mcc.heal_aggr.failed
  - name: callhome.mcc.heal_root.failed
  - name: callhome.mcc.personality
  - name: callhome.mcc.switchback.failed
  - name: callhome.mcc.switchover.failed
  - name: callhome.mccip.auso.disabled
  - name: callhome.mdb.recovery.unsuccessful
  - name: callhome.mlm.array.portOneController
  - name: callhome.mlm.lun.OnePath
  - name: callhome.multi.dsk.cksum.err
  - name: callhome.ndo.cr.nso.paused.err
  - name: callhome.ndo.cr.paused.err
  - name: callhome.netif.fatal.error
  - name: callhome.netif.init.failed
  - name: callhome.netif.uncorEccError
  - name: callhome.netinet.dup.clustIP
  - name: callhome.no.inodes
  - name: callhome.nse.ak.check.failed
  - name: callhome.nvdimm.cecc
  - name: callhome.nvdimm.failure
  - name: callhome.nvme.exhausted
  - name: callhome.nvme.offline
  - name: callhome.nvmem.volt.hi
  - name: callhome.nvmem.volt.lo
  - name: callhome.nvram.cecc
  - name: callhome.nvram.failure
  - name: callhome.nvram.iofs.fail
  - name: callhome.nvram.openfs.fail
  - name: callhome.partner.down
  - name: callhome.pool.aggr.noncomp
  - name: callhome.raid.no.recover
  - name: callhome.raid.over.max
  - name: callhome.raidtree.assim
  - name: callhome.reboot.agentwatchdog
  - name: callhome.reboot.panic
  - name: callhome.reboot.watchdog
  - name: callhome.root.vol.recovery.reqd
  - name: callhome.rtc.battery.low
  - name: callhome.sas.init.failed
  - name: callhome.sb.pp.fcp.failo.err
  - name: callhome.sb.pp.fcp.init.err
  - name: callhome.sblade.import.susp
  - name: callhome.sblade.lu.resync.to
  - name: callhome.sblade.lu.rst.hung
  - name: callhome.sblade.mgmt.wedged
  - name: callhome.sblade.prop.fail
  - name: callhome.sblade.unavailable
  - name: callhome.sblade.vol.ev.lost
  - name: callhome.sblade.vs.op.tt
  - name: callhome.scsi.tgt.misconfig
  - name: callhome.sdsk.fault
  - name: callhome.sdsk.label
  - name: callhome.sdsk.label.v
  - name: callhome.sdsk.missing
  - name: callhome.sdsk.noio
  - name: callhome.sdsk.shrank
  - name: callhome.ses.drive.missing
  - name: callhome.sfo.downrev
  - name: callhome.sfo.miscount
  - name: callhome.sfo.personality
  - name: callhome.sfo.takeover
  - name: callhome.sfo.takeover.failed
  - name: callhome.sfo.takeover.m.dr
  - name: callhome.sfo.takeover.panic
  - name: callhome.sfo.to.comm.loss
  - name: callhome.shlf.fan
  - name: callhome.shlf.fan.warn
  - name: callhome.shlf.fault
  - name: callhome.shlf.overtemp
  - name: callhome.shlf.power.intr
  - name: callhome.shlf.ps.fault
  - name: callhome.shutdown.pending
  - name: callhome.sm.vsdr.break.fail
  - name: callhome.snapmirror.oos.pers
  - name: callhome.sp.hbt.stopped
  - name: callhome.spares.low
  - name: callhome.spm.process.maxexit
  - name: callhome.svm.mig.abort.fail
  - name: callhome.svm.mig.clean.fail
  - name: callhome.svm.mig.fail
  - name: callhome.svm.mig.pause.fail
  - name: callhome.svm.mig.pstCut.fail
  - name: callhome.svm.mig.setCfg.fail
  - name: callhome.svo.prevented
  - name: callhome.sync.mirror.nospc
  - name: callhome.syncm.isolation
  - name: callhome.syncm.plex
  - name: callhome.syncm.plex.offlined
  - name: callhome.syncm.verify
  - name: callhome.syncsm.exception
  - name: callhome.syncsm.oos.fatal
  - name: callhome.sys.config
  - name: callhome.sys.config.crit
  - name: callhome.tgt.lun.nospc
  - name: callhome.unsupported.snaplock
  - name: callhome.ups.low.battery
  - name: callhome.ups.on.battery
  - name: callhome.ups.shutting.down
  - name: callhome.vdom.reconcile
  - name: callhome.vfiler.move.to.dst
  - name: callhome.vol.offline
  - name: callhome.vol.space.crit
  - name: callhome.wafl.check
  - name: callhome.wafl.inconsistent
  - name: callhome.wafl.inconsistent.block
  - name: callhome.wafl.inconsistent.user.block

[rahulguptajss]

@wooyoungAhn I have tried this template and tested an event callhome.aggr.meta.repair and it works as expected. Below are the steps I have tried for testing.

1. Generate an event from ONTAP CLI

set diag
event generate -message-name callhome.aggr.meta.repair -values Test

2. Wait for Harvest to do Next Polling and check prometheus for data.

   Harvest logs

2023-07-19T11:36:38+05:30 INF collector/collector.go:483 > Collected Poller=AFF250 apiMs=298 calcMs=0 collector=EMS:Ems instances=1 metrics=6 parseMs=0 pluginMs=0

Prometheus Data.

[wooyoungAhn]

Thank you for your reply
could you share the harvest version?
I tried the 23.05.0-1 linux/amd64

[rahulguptajss]

@wooyoungAhn The steps shared in this GitHub discussion are from version 23.05 only. Can you please try following these steps and let us know if you are able to get the ems? If issue persists, please send us the full logs at ng-harvest-files@netapp.com

[wooyoungAhn]

Hello
Thank you for your help.
I already tried generate ems message test.
I will more tried It my self before I will sending the logs.

What I found out today after trying
ems is not received from the point when something is added to the corresponding yaml.

cp ems.yaml custom_ems.yaml
echo -e "  - name: callhome.aggr.meta.repair\n""  - name: callhome.aggr.offline\n""  - name: callhome.aggr.restricted\n" >> custom_ems.yaml

For example, if only 3 items are added to the existing ems, it works fine, but if all items are entered, the ems event is not fetched.

I can't seem to get duplicates ems message even when they are specified.
I don't know if I couldn't find it, but there are no logs related to ems.

[rahulguptajss]

@wooyoungAhn Have you been able to resolve the issue? Can you check if the following query returns any records for the missing EMS events in Prometheus?

last_over_time(ems_events[48h])

We can also try running EMS collector in debug mode as below and collect more logs to understand better.

bin/harvest start POLLERNAME -c EMS --config harvest.yml --loglevel=0 -f

Replace POLLERNAME with the name defined in harvest configuration.

[wooyoungAhn]

Hello @rahulguptajss
I found error, but i don't understand why failed to fetch data.

Log error="failed to fetch data: error making request 0

2023-07-23T19:55:22+09:00 DBG ems/ems.go:371 > starting data poll Poller=tnas0304 collector=EMS:Ems
2023-07-23T19:55:22+09:00 DBG collectors/commonutils.go:74 > Poller=tnas0304 cluster time="2023-07-23 19:53:25 +0900 KST" collector=EMS:Ems
2023-07-23T19:55:22+09:00 DBG rest/rest.go:548 > Poller=tnas0304 collector=EMS:Ems href=api/support/ems/events?return_records=true&fields=*&message.severity=alert|emergency|error|informational|notice&time=>=1690109425&message.name=LUN.destroy,LUN.offline,LUN.online,NVMeNS.destroy,NVMeNS.offline,NVMeNS.online,Nblade.cifsManyAuths,Nblade.cifsMaxOpenSameFile,Nblade.cifsMaxSessPerUsrConn,Nblade.cifsNbNameConflict,Nblade.cifsNoPrivShare,Nblade.nfsV4PoolExhaust,Nblade.vscanBadUserPrivAccess,Nblade.vscanConnBackPressure,Nblade.vscanConnInactive,Nblade.vscanNoRegdScanner,Nblade.vscanNoScannerConn,Nblade.vscanVirusDetected,arl.netra.ca.check.failed,arw.volume.state,arw.vserver.state,callhome.aggr.meta.repair,callhome.aggr.offline,callhome.aggr.restricted,callhome.aggr.wafliron,callhome.andu.pausederr,callhome.appdm.snaprest.fail,callhome.arw.activity.seen,callhome.bad.ram,callhome.battery.failure,callhome.battery.low,callhome.battery.overchg,callhome.battery.overtemp,callhome.battery.warning,callhome.bcom.vsinit.failed,callhome.bdfu.suspended,callhome.bootfs.chkdsk,callhome.bridge.reboot.unexp,callhome.c.fan,callhome.c.fan.fru.degraded,callhome.c.fan.fru.fault,callhome.c.fan.fru.rm,callhome.c.fan.fru.shut,callhome.carrier.fault,callhome.ch.ps.degraded,callhome.ch.ps.fan.bad.xmin,callhome.ch.ps.fault,callhome.ch.ps.wrong.in,callhome.chassis.hitemp,callhome.chassis.overtemp,callhome.chassis.power,callhome.chassis.power.down,callhome.chassis.ps.degraded,callhome.chassis.ps.fault,callhome.chassis.ps.off,callhome.chassis.ps.replace,callhome.chassis.ps.rm,callhome.chassis.undertemp,callhome.checksum,callhome.checksum.multiple,callhome.clam.node.ooq,callhome.client.app.alert,callhome.client.app.emerg,callhome.client.app.err,callhome.clus.net.degraded,callhome.clus.vol.cre.fail,callhome.clus.vol.del.fail,callhome.config,callhome.cpu.fan,callhome.cpu.fan.fault,callhome.css.toc.repair,callhome.ctran.gcror.timeout,callhome.ctran.jpc.splitbrain,callhome.dafs.nic.stop,callhome.dbs.cluster.cannot.sync.blocks,callhome.dbs.cluster.capacity.full,callhome.dbs.cluster.capacity.nearly.full,callhome.dbs.data.unavailable,callhome.dbs.generic.cluster.fault,callhome.dbs.invalid.block.data,callhome.dbs.secondary.cache.full,callhome.dbs.service.capacity.full,callhome.dbs.service.capacity.nearly.full,callhome.device.qual,callhome.disk.carrier.evacuation.notpossible,callhome.diskown.AssignExceeded,callhome.disks.missing,callhome.dr.apply.failed,callhome.dsk.bypass,callhome.dsk.config,callhome.dsk.disable,callhome.dsk.fault,callhome.dsk.fw.needed,callhome.dsk.init,callhome.dsk.io.degraded,callhome.dsk.label,callhome.dsk.label.broken,callhome.dsk.label.v,callhome.dsk.missing,callhome.dsk.no.spin,callhome.dsk.noio,callhome.dsk.predict,callhome.dsk.redun.fault,callhome.dsk.scrub,callhome.dsk.scrub.done,callhome.dsk.scrub.parity,callhome.dsk.sense,callhome.dsk.shrank,callhome.encl.access,callhome.ext.cache.bad,callhome.fans.failed,callhome.fc.al.link,callhome.fc.al.poor,callhome.fcp.adapter.failed,callhome.fcp.sram.dump,callhome.fdsk.fault,callhome.fdsk.label,callhome.fdsk.label.v,callhome.fdsk.missing,callhome.fdsk.noio,callhome.fdsk.shrank,callhome.flash.cache.degraded,callhome.flash.cache.failed,callhome.flash.cache.offlined,callhome.fs.lun.resized,callhome.hainterconnect.down,callhome.hba.failed,callhome.hm.alert.critical,callhome.hm.alert.major,callhome.hm.alert.minor,callhome.hostport.bypass,callhome.hostport.disable,callhome.ibretimerprog.fail,callhome.iscsi.hba.crash,callhome.isp.adapter.bad,callhome.jpc.selection.fail,callhome.link.change.stuck,callhome.mcc.auso.2min,callhome.mcc.auso.complete,callhome.mcc.auso.trig.fail,callhome.mcc.clst.dm.wd.gvup,callhome.mcc.cluster_dm.stream_in_error,callhome.mcc.heal_aggr.failed,callhome.mcc.heal_root.failed,callhome.mcc.personality,callhome.mcc.switchback.failed,callhome.mcc.switchover.failed,callhome.mccip.auso.disabled,callhome.mdb.recovery.unsuccessful,callhome.mlm.array.portOneController,callhome.mlm.lun.OnePath,callhome.multi.dsk.cksum.err,callhome.ndo.cr.nso.paused.err,callhome.ndo.cr.paused.err,callhome.netif.fatal.error,callhome.netif.init.failed,callhome.netif.uncorEccError,callhome.netinet.dup.clustIP,callhome.no.inodes,callhome.nse.ak.check.failed,callhome.nvdimm.cecc,callhome.nvdimm.failure,callhome.nvme.exhausted,callhome.nvme.offline,callhome.nvmem.volt.hi,callhome.nvmem.volt.lo,callhome.nvram.cecc,callhome.nvram.failure,callhome.nvram.iofs.fail,callhome.nvram.openfs.fail,callhome.partner.down,callhome.pool.aggr.noncomp,callhome.raid.no.recover,callhome.raid.over.max,callhome.raidtree.assim,callhome.reboot.agentwatchdog,callhome.reboot.panic,callhome.reboot.watchdog,callhome.root.vol.recovery.reqd,callhome.rtc.battery.low,callhome.sas.init.failed,callhome.sb.pp.fcp.failo.err,callhome.sb.pp.fcp.init.err,callhome.sblade.import.susp,callhome.sblade.lu.resync.to,callhome.sblade.lu.rst.hung,callhome.sblade.mgmt.wedged,callhome.sblade.prop.fail,callhome.sblade.unavailable,callhome.sblade.vol.ev.lost,callhome.sblade.vs.op.tt,callhome.scsi.tgt.misconfig,callhome.sdsk.fault,callhome.sdsk.label,callhome.sdsk.label.v,callhome.sdsk.missing,callhome.sdsk.noio,callhome.sdsk.shrank,callhome.ses.drive.missing,callhome.sfo.downrev,callhome.sfo.miscount,callhome.sfo.personality,callhome.sfo.takeover,callhome.sfo.takeover.failed,callhome.sfo.takeover.m.dr,callhome.sfo.takeover.panic,callhome.sfo.to.comm.loss,callhome.shlf.fan,callhome.shlf.fan.warn,callhome.shlf.fault,callhome.shlf.overtemp,callhome.shlf.power.intr,callhome.shlf.ps.fault,callhome.shutdown.pending,callhome.sm.vsdr.break.fail,callhome.snapmirror.oos.pers,callhome.sp.hbt.missed,callhome.sp.hbt.stopped,callhome.spares.low,callhome.spm.process.maxexit,callhome.svm.mig.abort.fail,callhome.svm.mig.clean.fail,callhome.svm.mig.fail,callhome.svm.mig.pause.fail,callhome.svm.mig.pstCut.fail,callhome.svm.mig.setCfg.fail,callhome.svo.prevented,callhome.sync.mirror.nospc,callhome.syncm.isolation,callhome.syncm.plex,callhome.syncm.plex.offlined,callhome.syncm.verify,callhome.syncsm.exception,callhome.syncsm.oos.fatal,callhome.sys.config,callhome.sys.config.crit,callhome.tgt.lun.nospc,callhome.unsupported.snaplock,callhome.ups.low.battery,callhome.ups.on.battery,callhome.ups.shutting.down,callhome.vdom.reconcile,callhome.vfiler.move.to.dst,callhome.vol.offline,callhome.vol.space.crit,callhome.wafl.check,callhome.wafl.inconsistent,callhome.wafl.inconsistent.block,callhome.wafl.inconsistent.user.block,cifs.shadowcopy.failure,cloud.aws.iamNotInitialized,cluster.switch.pwr.fail,disk.outOfService,diskShelf.psu.added,diskShelf.psu.removed,fabricpool.full,fabricpool.nearly.full,gb.netra.ca.check.failed,hm.alert.cleared,hm.alert.raised,mcc.config.auso.stDisabled,mcc.config.auso.stEnabled,mgmtgwd.rootvolrec.low.space,monitor.fan.critical,monitor.fan.failed,monitor.fan.ok,monitor.fan.warning,nvmf.graceperiod.active,nvmf.graceperiod.expired,nvmf.graceperiod.start,nvram.battery.charging.normal,object.store.available,object.store.unavailable,objstore.host.unresolvable,objstore.interclusterlifDown,osc.signatureMismatch,qos.monitor.memory.abated,qos.monitor.memory.maxed,scsiblade.san.config.active,scsitarget.fct.port.full,scsitarget.fct.sfpRxPowerLow,scsitarget.fct.sfpTxPowerLow,ses.status.fanError,ses.status.fanInfo,sk.panic,sm.mediator.added,sm.mediator.cacert.expired,sm.mediator.cacert.expiring,sm.mediator.clientc.expired,sm.mediator.clientc.expiring,sm.mediator.in.quorum,sm.mediator.misconfigured,sm.mediator.removed,sm.mediator.serverc.expired,sm.mediator.serverc.expiring,sm.mediator.unreachable,sms.status.in.sync,sms.status.out.of.sync,sms.status.out.of.sync.cg,sp.heartbeat.resumed,sp.ipmi.lost.shutdown,sp.notConfigured,unowned.disk.reminder,vserver.start.succeeded,vserver.stop.succeeded,wafl.ca.resync.complete,wafl.readdir.expired,wafl.vol.autoSize.done,wafl.vvol.offline,wafl.vvol.online,wafl.vvol.restrict&order_by=index%20desc
2023-07-23T19:55:22+09:00 DBG collector/collector.go:314 > handling error during [data] poll... Poller=tnas0304 collector=EMS:Ems
2023-07-23T19:55:22+09:00 ERR collector/collector.go:367 > error="failed to fetch data: error making request 0: " Poller=tnas0304 collector=EMS:Ems task=data
2023-07-23T19:55:22+09:00 DBG collector/collector.go:422 > exporting collected (0) data Poller=tnas0304 collector=EMS:Ems

**2023-07-23T19:55:22+09:00 ERR collector/collector.go:367 > error="failed to fetch data: error making request 0: " Poller=tnas0304 collector=EMS:Ems task=data**

PromQL ems_event query

Test to get data

event generate -message-name callhome.battery.low -values test_events

bin/harvest rest show data -p tnas0304 --api support/ems/events --field message.name=callhome.battery.low

curl --user admin --insecure 'https://10.50.10.223/api/support/ems/events?return_records=true&fields=*&message.name=callhome.battery.low' # tnas0304
[
 {
  "poller": "tnas0304",
  "addr": "10.50.10.223",
  "api": "support/ems/events",
  "version": "9.11.1",
  "cluster_name": "tnas0304",
  "records": [
   {
    "index": 2252205,
    "log_message": "callhome.battery.low: Call home for BATTERY_LOW.",
    "message": {
     "name": "callhome.battery.low",
     "severity": "alert"
    },
    "node": {
     "name": "tnas04",
     "uuid": "f9adb85a-3d73-11ed-a608-d039ea9136fa"
    },
    "parameters": [
     {
      "name": "subject",
      "value": "test_events"
     }
    ],
    "source": "notifyd",
    "time": "2023-07-23T20:09:17+09:00"
   },
   {
    "index": 2251911,
    "log_message": "callhome.battery.low: Call home for BATTERY_LOW.",
    "message": {
     "name": "callhome.battery.low",
     "severity": "alert"
    },
    "node": {
     "name": "tnas04",
     "uuid": "f9adb85a-3d73-11ed-a608-d039ea9136fa"
    },
    "parameters": [
     {
      "name": "subject",
      "value": "test_events"
     }
    ],
    "source": "notifyd",
    "time": "2023-07-23T19:52:51+09:00"
   },
   {
    "index": 2305061,
    "log_message": "callhome.battery.low: Call home for BATTERY_LOW.",
    "message": {
     "name": "callhome.battery.low",
     "severity": "alert"
    },
    "node": {
     "name": "tnas03",
     "uuid": "4f27e156-3d73-11ed-93a7-d039ea3d0f35"
    },
    "parameters": [
     {
      "name": "subject",
      "value": "test"
     }
    ],
    "source": "notifyd",
    "time": "2023-07-23T19:47:05+09:00"
   },
   {
    "index": 2305036,
    "log_message": "callhome.battery.low: Call home for BATTERY_LOW.",
    "message": {
     "name": "callhome.battery.low",
     "severity": "alert"
    },
    "node": {
     "name": "tnas03",
     "uuid": "4f27e156-3d73-11ed-93a7-d039ea3d0f35"
    },
    "parameters": [
     {
      "name": "subject",
      "value": "test"
     }
    ],
    "source": "notifyd",
    "time": "2023-07-23T19:43:32+09:00"
   }
  ],
  "num_records": 4,
  "poll_ms": 134
 }
]

[rahulguptajss]

Thanks @wooyoungAhn . Let's attempt a direct call to ONTAP and observe the response to determine if we encounter the same error. Please replace USER, PASS, and CLUSTER_IP with the appropriate values for your cluster.

curl -s -k -u USER:PASS 'https://CLUSTER_IP/api/support/ems/events?return_records=true&fields=*&message.severity=alert|emergency|error|informational|notice&time=>=1690109425&message.name=LUN.destroy,LUN.offline,LUN.online,NVMeNS.destroy,NVMeNS.offline,NVMeNS.online,Nblade.cifsManyAuths,Nblade.cifsMaxOpenSameFile,Nblade.cifsMaxSessPerUsrConn,Nblade.cifsNbNameConflict,Nblade.cifsNoPrivShare,Nblade.nfsV4PoolExhaust,Nblade.vscanBadUserPrivAccess,Nblade.vscanConnBackPressure,Nblade.vscanConnInactive,Nblade.vscanNoRegdScanner,Nblade.vscanNoScannerConn,Nblade.vscanVirusDetected,arl.netra.ca.check.failed,arw.volume.state,arw.vserver.state,callhome.aggr.meta.repair,callhome.aggr.offline,callhome.aggr.restricted,callhome.aggr.wafliron,callhome.andu.pausederr,callhome.appdm.snaprest.fail,callhome.arw.activity.seen,callhome.bad.ram,callhome.battery.failure,callhome.battery.low,callhome.battery.overchg,callhome.battery.overtemp,callhome.battery.warning,callhome.bcom.vsinit.failed,callhome.bdfu.suspended,callhome.bootfs.chkdsk,callhome.bridge.reboot.unexp,callhome.c.fan,callhome.c.fan.fru.degraded,callhome.c.fan.fru.fault,callhome.c.fan.fru.rm,callhome.c.fan.fru.shut,callhome.carrier.fault,callhome.ch.ps.degraded,callhome.ch.ps.fan.bad.xmin,callhome.ch.ps.fault,callhome.ch.ps.wrong.in,callhome.chassis.hitemp,callhome.chassis.overtemp,callhome.chassis.power,callhome.chassis.power.down,callhome.chassis.ps.degraded,callhome.chassis.ps.fault,callhome.chassis.ps.off,callhome.chassis.ps.replace,callhome.chassis.ps.rm,callhome.chassis.undertemp,callhome.checksum,callhome.checksum.multiple,callhome.clam.node.ooq,callhome.client.app.alert,callhome.client.app.emerg,callhome.client.app.err,callhome.clus.net.degraded,callhome.clus.vol.cre.fail,callhome.clus.vol.del.fail,callhome.config,callhome.cpu.fan,callhome.cpu.fan.fault,callhome.css.toc.repair,callhome.ctran.gcror.timeout,callhome.ctran.jpc.splitbrain,callhome.dafs.nic.stop,callhome.dbs.cluster.cannot.sync.blocks,callhome.dbs.cluster.capacity.full,callhome.dbs.cluster.capacity.nearly.full,callhome.dbs.data.unavailable,callhome.dbs.generic.cluster.fault,callhome.dbs.invalid.block.data,callhome.dbs.secondary.cache.full,callhome.dbs.service.capacity.full,callhome.dbs.service.capacity.nearly.full,callhome.device.qual,callhome.disk.carrier.evacuation.notpossible,callhome.diskown.AssignExceeded,callhome.disks.missing,callhome.dr.apply.failed,callhome.dsk.bypass,callhome.dsk.config,callhome.dsk.disable,callhome.dsk.fault,callhome.dsk.fw.needed,callhome.dsk.init,callhome.dsk.io.degraded,callhome.dsk.label,callhome.dsk.label.broken,callhome.dsk.label.v,callhome.dsk.missing,callhome.dsk.no.spin,callhome.dsk.noio,callhome.dsk.predict,callhome.dsk.redun.fault,callhome.dsk.scrub,callhome.dsk.scrub.done,callhome.dsk.scrub.parity,callhome.dsk.sense,callhome.dsk.shrank,callhome.encl.access,callhome.ext.cache.bad,callhome.fans.failed,callhome.fc.al.link,callhome.fc.al.poor,callhome.fcp.adapter.failed,callhome.fcp.sram.dump,callhome.fdsk.fault,callhome.fdsk.label,callhome.fdsk.label.v,callhome.fdsk.missing,callhome.fdsk.noio,callhome.fdsk.shrank,callhome.flash.cache.degraded,callhome.flash.cache.failed,callhome.flash.cache.offlined,callhome.fs.lun.resized,callhome.hainterconnect.down,callhome.hba.failed,callhome.hm.alert.critical,callhome.hm.alert.major,callhome.hm.alert.minor,callhome.hostport.bypass,callhome.hostport.disable,callhome.ibretimerprog.fail,callhome.iscsi.hba.crash,callhome.isp.adapter.bad,callhome.jpc.selection.fail,callhome.link.change.stuck,callhome.mcc.auso.2min,callhome.mcc.auso.complete,callhome.mcc.auso.trig.fail,callhome.mcc.clst.dm.wd.gvup,callhome.mcc.cluster_dm.stream_in_error,callhome.mcc.heal_aggr.failed,callhome.mcc.heal_root.failed,callhome.mcc.personality,callhome.mcc.switchback.failed,callhome.mcc.switchover.failed,callhome.mccip.auso.disabled,callhome.mdb.recovery.unsuccessful,callhome.mlm.array.portOneController,callhome.mlm.lun.OnePath,callhome.multi.dsk.cksum.err,callhome.ndo.cr.nso.paused.err,callhome.ndo.cr.paused.err,callhome.netif.fatal.error,callhome.netif.init.failed,callhome.netif.uncorEccError,callhome.netinet.dup.clustIP,callhome.no.inodes,callhome.nse.ak.check.failed,callhome.nvdimm.cecc,callhome.nvdimm.failure,callhome.nvme.exhausted,callhome.nvme.offline,callhome.nvmem.volt.hi,callhome.nvmem.volt.lo,callhome.nvram.cecc,callhome.nvram.failure,callhome.nvram.iofs.fail,callhome.nvram.openfs.fail,callhome.partner.down,callhome.pool.aggr.noncomp,callhome.raid.no.recover,callhome.raid.over.max,callhome.raidtree.assim,callhome.reboot.agentwatchdog,callhome.reboot.panic,callhome.reboot.watchdog,callhome.root.vol.recovery.reqd,callhome.rtc.battery.low,callhome.sas.init.failed,callhome.sb.pp.fcp.failo.err,callhome.sb.pp.fcp.init.err,callhome.sblade.import.susp,callhome.sblade.lu.resync.to,callhome.sblade.lu.rst.hung,callhome.sblade.mgmt.wedged,callhome.sblade.prop.fail,callhome.sblade.unavailable,callhome.sblade.vol.ev.lost,callhome.sblade.vs.op.tt,callhome.scsi.tgt.misconfig,callhome.sdsk.fault,callhome.sdsk.label,callhome.sdsk.label.v,callhome.sdsk.missing,callhome.sdsk.noio,callhome.sdsk.shrank,callhome.ses.drive.missing,callhome.sfo.downrev,callhome.sfo.miscount,callhome.sfo.personality,callhome.sfo.takeover,callhome.sfo.takeover.failed,callhome.sfo.takeover.m.dr,callhome.sfo.takeover.panic,callhome.sfo.to.comm.loss,callhome.shlf.fan,callhome.shlf.fan.warn,callhome.shlf.fault,callhome.shlf.overtemp,callhome.shlf.power.intr,callhome.shlf.ps.fault,callhome.shutdown.pending,callhome.sm.vsdr.break.fail,callhome.snapmirror.oos.pers,callhome.sp.hbt.missed,callhome.sp.hbt.stopped,callhome.spares.low,callhome.spm.process.maxexit,callhome.svm.mig.abort.fail,callhome.svm.mig.clean.fail,callhome.svm.mig.fail,callhome.svm.mig.pause.fail,callhome.svm.mig.pstCut.fail,callhome.svm.mig.setCfg.fail,callhome.svo.prevented,callhome.sync.mirror.nospc,callhome.syncm.isolation,callhome.syncm.plex,callhome.syncm.plex.offlined,callhome.syncm.verify,callhome.syncsm.exception,callhome.syncsm.oos.fatal,callhome.sys.config,callhome.sys.config.crit,callhome.tgt.lun.nospc,callhome.unsupported.snaplock,callhome.ups.low.battery,callhome.ups.on.battery,callhome.ups.shutting.down,callhome.vdom.reconcile,callhome.vfiler.move.to.dst,callhome.vol.offline,callhome.vol.space.crit,callhome.wafl.check,callhome.wafl.inconsistent,callhome.wafl.inconsistent.block,callhome.wafl.inconsistent.user.block,cifs.shadowcopy.failure,cloud.aws.iamNotInitialized,cluster.switch.pwr.fail,disk.outOfService,diskShelf.psu.added,diskShelf.psu.removed,fabricpool.full,fabricpool.nearly.full,gb.netra.ca.check.failed,hm.alert.cleared,hm.alert.raised,mcc.config.auso.stDisabled,mcc.config.auso.stEnabled,mgmtgwd.rootvolrec.low.space,monitor.fan.critical,monitor.fan.failed,monitor.fan.ok,monitor.fan.warning,nvmf.graceperiod.active,nvmf.graceperiod.expired,nvmf.graceperiod.start,nvram.battery.charging.normal,object.store.available,object.store.unavailable,objstore.host.unresolvable,objstore.interclusterlifDown,osc.signatureMismatch,qos.monitor.memory.abated,qos.monitor.memory.maxed,scsiblade.san.config.active,scsitarget.fct.port.full,scsitarget.fct.sfpRxPowerLow,scsitarget.fct.sfpTxPowerLow,ses.status.fanError,ses.status.fanInfo,sk.panic,sm.mediator.added,sm.mediator.cacert.expired,sm.mediator.cacert.expiring,sm.mediator.clientc.expired,sm.mediator.clientc.expiring,sm.mediator.in.quorum,sm.mediator.misconfigured,sm.mediator.removed,sm.mediator.serverc.expired,sm.mediator.serverc.expiring,sm.mediator.unreachable,sms.status.in.sync,sms.status.out.of.sync,sms.status.out.of.sync.cg,sp.heartbeat.resumed,sp.ipmi.lost.shutdown,sp.notConfigured,unowned.disk.reminder,vserver.start.succeeded,vserver.stop.succeeded,wafl.ca.resync.complete,wafl.readdir.expired,wafl.vol.autoSize.done,wafl.vvol.offline,wafl.vvol.online,wafl.vvol.restrict&order_by=index%20desc' 

[wooyoungAhn]

Hello
I couldn't find the cause of the previous problem, but now the collection works fine.
I appreciate your help.