panic: runtime error: invalid memory address or nil pointer dereference

[elsgaard]

Current Behavior

unexpected fault address 0xffffffffffffffff
fatal error: fault
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x1 pc=0xa62e57]

Expected Behavior

Windows Exporter should not crash

Steps To Reproduce

Running for some time with below config.

collectors:
  enabled: os,cpu,cpu_info,diskdrive,license,logical_disk,memory,net,tcp,service,system,terminal_services,process,time,iis

Environment

• windows_exporter Version: 0.30.8
• Windows Server Version: Windows Server 2016 standard

windows_exporter logs

Nothing suspicious in the exporter log with debug enabled
`
time=2025-06-29T10:51:36.755Z level=INFO source=main.go:150 msg="using configuration file: config.yaml"
time=2025-06-29T10:51:36.756Z level=DEBUG source=main.go:153 msg="logging has Started"
time=2025-06-29T10:51:36.803Z level=WARN source=os.go:109 msg="The os collector holds a number of deprecated metrics and will be removed mid 2025. See https://github.com/prometheus-community/windows_exporter/pull/1596 for more information." collector=os
time=2025-06-29T10:51:36.806Z level=DEBUG source=iis.go:233 msg="Detected IIS 10.0\n" collector=iis
time=2025-06-29T10:51:37.891Z level=INFO source=net.go:272 msg="nic/addresses collector is in an experimental state! The configuration and metrics may change in future. Please report any issues." collector=net
time=2025-06-29T10:51:38.323Z level=DEBUG source=terminal_services.go:243 msg="host is not a connection broker skipping Connection Broker performance metrics."
time=2025-06-29T10:51:38.567Z level=INFO source=main.go:265 msg="Running as EMEATC\\thomas.elsgaard"
time=2025-06-29T10:51:38.570Z level=INFO source=main.go:185 msg="Enabled collectors: os, cpu, cpu_info, diskdrive, license, logical_disk, memory, net, tcp, service, system, terminal_services, process, time, iis"
time=2025-06-29T10:51:38.572Z level=INFO source=main.go:203 msg="starting windows_exporter in 1.8217936s" version=0.30.8 branch=HEAD revision=d451acbd63ccdbad48e0c2266e99983e727bf4cc goversion=go1.23.4 builddate=20250621-09:31:57 maxprocs=6
time=2025-06-29T10:51:38.575Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9182
time=2025-06-29T10:51:38.575Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9182
time=2025-06-29T12:41:44.986Z level=DEBUG source=collect.go:220 msg="collector license succeeded after 0s, resulting in 5 metrics"
time=2025-06-29T12:41:44.987Z level=DEBUG source=collect.go:220 msg="collector memory succeeded after 1.0123ms, resulting in 35 metrics"
time=2025-06-29T12:41:44.992Z level=DEBUG source=collect.go:220 msg="collector os succeeded after 5.9959ms, resulting in 11 metrics"
time=2025-06-29T12:41:44.993Z level=DEBUG source=collect.go:220 msg="collector os succeeded after 7.0016ms, resulting in 11 metrics"
time=2025-06-29T12:41:44.995Z level=DEBUG source=collect.go:220 msg="collector license succeeded after 9.002ms, resulting in 5 metrics"
time=2025-06-29T12:41:45.010Z level=DEBUG source=collect.go:220 msg="collector time succeeded after 23.9981ms, resulting in 8 metrics"
time=2025-06-29T12:41:45.010Z level=DEBUG source=collect.go:220 msg="collector logical_disk succeeded after 23.9981ms, resulting in 34 metrics"
time=2025-06-29T12:41:45.015Z level=DEBUG source=collect.go:220 msg="collector diskdrive succeeded after 28.9993ms, resulting in 36 metrics"
time=2025-06-29T12:41:45.036Z level=DEBUG source=collect.go:220 msg="collector system succeeded after 50.0001ms, resulting in 9 metrics"
time=2025-06-29T12:41:45.038Z level=DEBUG source=collect.go:220 msg="collector diskdrive succeeded after 51.9994ms, resulting in 36 metrics"
time=2025-06-29T12:41:45.038Z level=DEBUG source=collect.go:220 msg="collector tcp succeeded after 51.9994ms, resulting in 22 metrics"
time=2025-06-29T12:41:45.059Z level=DEBUG source=collect.go:220 msg="collector net succeeded after 72.9995ms, resulting in 27 metrics"
time=2025-06-29T12:41:45.061Z level=DEBUG source=collect.go:220 msg="collector cpu_info succeeded after 74.955ms, resulting in 42 metrics"
time=2025-06-29T12:41:45.059Z level=DEBUG source=collect.go:220 msg="collector cpu_info succeeded after 72.9995ms, resulting in 42 metrics"
time=2025-06-29T12:41:45.070Z level=DEBUG source=collect.go:220 msg="collector process succeeded after 83.9519ms, resulting in 2001 metrics"
time=2025-06-29T12:41:45.071Z level=DEBUG source=collect.go:220 msg="collector process succeeded after 82.9528ms, resulting in 2001 metrics"
time=2025-06-29T12:41:45.171Z level=DEBUG source=collect.go:220 msg="collector service succeeded after 184.9589ms, resulting in 2786 metrics"
time=2025-06-29T12:41:45.176Z level=DEBUG source=collect.go:220 msg="collector service succeeded after 189.959ms, resulting in 2786 metrics"
time=2025-06-29T12:41:45.291Z level=DEBUG source=collect.go:220 msg="collector cpu succeeded after 305.0672ms, resulting in 115 metrics"
time=2025-06-29T12:41:45.300Z level=DEBUG source=collect.go:220 msg="collector terminal_services succeeded after 314.0049ms, resulting in 35 metrics"
time=2025-06-29T12:41:45.301Z level=DEBUG source=collect.go:220 msg="collector memory succeeded after 314.9858ms, resulting in 35 metrics"
time=2025-06-29T12:41:45.302Z level=DEBUG source=collect.go:220 msg="collector time succeeded after 315.9809ms, resulting in 8 metrics"
time=2025-06-29T12:41:45.303Z level=DEBUG source=collect.go:220 msg="collector logical_disk succeeded after 316.9965ms, resulting in 34 metrics"
time=2025-06-29T12:41:45.309Z level=DEBUG source=collect.go:220 msg="collector system succeeded after 323.2034ms, resulting in 9 metrics"
time=2025-06-29T12:41:45.316Z level=DEBUG source=collect.go:220 msg="collector net succeeded after 329.8294ms, resulting in 27 metrics"
time=2025-06-29T12:41:45.318Z level=DEBUG source=collect.go:220 msg="collector cpu succeeded after 331.8203ms, resulting in 115 metrics"
time=2025-06-29T12:41:45.319Z level=DEBUG source=collect.go:220 msg="collector tcp succeeded after 332.8202ms, resulting in 22 metrics"
time=2025-06-29T12:41:45.329Z level=DEBUG source=collect.go:220 msg="collector terminal_services succeeded after 342.896ms, resulting in 35 metrics"
time=2025-06-29T12:41:45.333Z level=DEBUG source=collect.go:220 msg="collector iis succeeded after 345.8206ms, resulting in 207 metrics"
time=2025-06-29T12:41:45.333Z level=DEBUG source=collect.go:220 msg="collector iis succeeded after 346.8201ms, resulting in 207 metrics"
time=2025-06-29T12:42:44.989Z level=DEBUG source=collect.go:220 msg="collector license succeeded after 2.9941ms, resulting in 5 metrics"

`




`
unexpected fault address 0xffffffffffffffff
fatal error: fault
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x1 pc=0xa62e57]

goroutine 14102 [running]:
golang.org/x/sys/windows.UTF16PtrToString(0x1211009?)
        C:/Users/runneradmin/go/pkg/mod/golang.org/x/sys@v0.31.0/windows/syscall_windows.go:121 +0x17
github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).collectWorker(0xc00015c480, 0xc000686770, {0x1, 0x435, {0x0, 0x0, 0x0, 0x0, 0x18f62, 0x18f50, ...}})
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:243 +0x3a
github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).Collect.func1(0xc000686770, 0xc000396000)
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:225 +0xfc
created by github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).Collect in goroutine 14012
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:223 +0xcb
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0xa62e57]

goroutine 14035 gp=0xc00028ddc0 m=14 mp=0xc00041c808 [running]:
runtime.throw({0x12112bf?, 0x8ef976?})
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/panic.go:1067 +0x4d fp=0xc000983d78 sp=0xc000983d48 pc=0x95614d
runtime.sigpanic()
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/signal_windows.go:414 +0xd0 fp=0xc000983dc0 sp=0xc000983d78 pc=0x935e30
golang.org/x/sys/windows.UTF16PtrToString(0x95cb57?)
        C:/Users/runneradmin/go/pkg/mod/golang.org/x/sys@v0.31.0/windows/syscall_windows.go:121 +0x17 fp=0xc000983de8 sp=0xc000983dc0 pc=0xa62e57
github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).collectWorker(0xc00015c480, 0xc000686770, {0x18fc000018fee, 0x100000020, {0x0, 0x435, 0x0, 0x0, 0x0, 0x0, ...}})
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:243 +0x3a fp=0xc000983ee8 sp=0xc000983de8 pc=0xfa6efa
github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).Collect.func1(0xc000686770, 0xc000396000)
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:225 +0xfc fp=0xc000983fc0 sp=0xc000983ee8 pc=0xfa6e5c
github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).Collect.gowrap1()
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:228 +0x28 fp=0xc000983fe0 sp=0xc000983fc0 pc=0xfa6d28
runtime.goexit({})
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000983fe8 sp=0xc000983fe0 pc=0x95eb21
created by github.com/prometheus-community/windows_exporter/internal/collector/service.(*Collector).Collect in goroutine 14012
        D:/a/windows_exporter/windows_exporter/internal/collector/service/service.go:223 +0xcb

goroutine 1 gp=0xc00005c000 m=nil [select, 111 minutes]:
runtime.gopark(0xc0005d9cc0?, 0x3?, 0x20?, 0x1c?, 0xc0005d99e2?)
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/proc.go:424 +0xce fp=0xc0003f5868 sp=0xc0003f5848 pc=0x95626e
runtime.selectgo(0xc0003f5cc0, 0xc0005d99dc, 0xc0000796c0?, 0x0, 0xc0004f9260?, 0x1)
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/select.go:335 +0x7a5 fp=0xc0003f5990 sp=0xc0003f5868 pc=0x933485
main.run({0x136ca38, 0xc0000796c0}, {0xc00006a3b0, 0x1, 0x1})
        D:/a/windows_exporter/windows_exporter/cmd/windows_exporter/main.go:230 +0x201b fp=0xc0003f5f00 sp=0xc0003f5990 pc=0x105915b
main.main()
        D:/a/windows_exporter/windows_exporter/cmd/windows_exporter/main.go:52 +0xcb fp=0xc0003f5f50 sp=0xc0003f5f00 pc=0x10570cb
runtime.main()
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/proc.go:272 +0x27d fp=0xc0003f5fe0 sp=0xc0003f5f50 pc=0x92151d
runtime.goexit({})
        C:/Users/runneradmin/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.windows-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003f5fe8 sp=0xc0003f5fe0 pc=0x95eb21
`

Anything else?

We only see this crash on 1 server out of many.

[jkroepke]

It seems that the one specific server has a service with a suspicious name.

[jkroepke]

please check, if the following fix resolves your problem

https://github.com/prometheus-community/windows_exporter/actions/runs/15957240693/artifacts/3427502716

[elsgaard]

Unfortunately i still see the crash. I have put the complete crash and debug logs here (they are too big to include here): https://github.com/elsgaard/exporter_logs/tree/main

[jkroepke]

It's very hard to reproduce the issue for me. So I have to guess.

Could you test that one?

https://github.com/prometheus-community/windows_exporter/actions/runs/15980686796/artifacts/3434648011

[elsgaard]

@jkroepke I have been testing on it, and i have seen an interesting behavior, that might / might no be related. When creating the issue, i was seeing the panics, but i was also seeing a lot of gaps in the service metrics collected like below (but the gaps did not match the panics 100% timewise).

For redundancy, we have 2 Prometheus instances scraping same targets, and the problem with gaps in the service data and the panics stopped as soon as i stopped the redundant Prometheus instance. Not sure if it's a separate issue, or linked to these panics, but there seems to be a problem in the service collector when 2 Prometheus instances are scraping at the same time.
Below you can see, that as soon as i stopped the second Prometheus instance from scraping the exporter, it looks correct again.

maybe be that this scraping with two Prometheus nodes at the same time could be linked to the panics ?

[jkroepke]

do you just using the metrics endpoint?

oder any additional parameters?

Generally, I would say yes, parallel requests is a issue for the collectors. But there ware builtin windows_exporter technics to prevents parallel requests.

In case the builtin technics wont work, it would explain it.

What is the scrape interval of prometheus instances?

[elsgaard]

Just using the /metrics endpoint, nothing special or special parameters.
Yeah i have seen other exporters having problems with parallel requests.

I think the fix you made was good for the panics. And lets for now just close this issue. I will investigate the parallel requests a little deeper, and create a new issue if i can get some more info. For now i resolved my problem to stop one of the Prometheus instances scraping the exporter, and then re-enable it again.

Thanks for the assistance!

[jkroepke]

And lets for now just close this issue.

The orginal issue still remains. So from my point of view, the bug still remains. It's correct I implement a catch for the panic, but the panic should not be appear.

[jkroepke]

maybe be that this scraping with two Prometheus nodes at the same time could be linked to the panics ?

I went through the source code and noticed that the concurrency lock was indeed missing. It looks like I accidentally removed it. Only the related variable declarations were still there.

I’ve now re-added the logic. Could you please test it again in your environment, using the full load from two Prometheus instances?

https://github.com/prometheus-community/windows_exporter/actions/runs/16035426302/artifacts/3453077335

[elsgaard]

maybe be that this scraping with two Prometheus nodes at the same time could be linked to the panics ?

I went through the source code and noticed that the concurrency lock was indeed missing. It looks like I accidentally removed it. Only the related variable declarations were still there.

I’ve now re-added the logic. Could you please test it again in your environment, using the full load from two Prometheus instances?

https://github.com/prometheus-community/windows_exporter/actions/runs/16035426302/artifacts/3453077335

I have rolled the version out on 10 servers, that is getting scraped by two Prometheus instances. Will keep you updated on the testing. Thanks for looking into this problem!