Docker plugin stopped starting Docker containers

[andreasgalek]

Jenkins and plugins versions report

Environment

Jenkins: 2.440.2
OS: Linux - 5.15.0-206.153.7.1.el9uek.x86_64
Java: 11.0.23 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.21.0
Parameterized-Remote-Trigger:3.2.0
ace-editor:1.1
allure-jenkins-plugin:2.31.1
analysis-model-api:12.1.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
appaloosa-plugin:1.4.12
asm-api:9.7-33.v4d23ef79fcc8
atlassian-bitbucket-server-integration:4.0.0
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
azure-credentials:312.v0f3973cd1e59
azure-sdk:157.v855da_0b_eb_dc2
badge:1.9.1
basic-branch-build-strategies:81.v05e333931c7d
bitbucket:241.v6d24a_57f9359
blackduck-detect:9.0.0
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jira:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-blocker-plugin:1.7.9
build-failure-analyzer:2.5.0
build-name-setter:2.4.2
build-timeout:1.32
build-timestamp:1.0.3
build-user-vars-plugin:1.9
buildtriggerbadge:251.vdf6ef853f3f5
built-on-column:1.4
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
clover:4.14.2.596.vb_4d6475e990b_
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
compress-buildlog:1.2
conditional-buildstep:1.4.3
config-file-provider:968.ve1ca_eb_913f8c
copyartifact:722.v0662a_9b_e22a_c
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:677.vdc9d38cb_254d
cucumber-reports:5.8.1
custom-checkbox-parameter:1.4
dark-theme:439.vdef09f81f85e
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.3-1
description-setter:239.vd0a_6b_785f92d
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.6
docker-workflow:572.v950f58993843
dtkit-api:3.0.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
email-ext:2.105
embeddable-build-status:467.v4a_954796e45d
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
extensible-choice-parameter:1.8.1
external-monitor-job:215.v2e88e894db_f8
extra-columns:1.26
favorite:2.208.v91d65b_7792a_c
file-operations:214.v2e7dc7f25757
file-parameters:316.va_83a_1221db_a_7
folder-properties:1.2.1
font-awesome-api:6.5.1-3
forensics-api:2.4.0
generic-webhook-trigger:2.1.1
git:5.2.1
git-changelog:3.38
git-client:4.7.0
git-forensics:2.1.0
git-parameter:0.9.19
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
golang:1.4
google-oauth-plugin:1.330.vf5e86021cb_ec
gradle:2.10
groovy:457.v99900cb_85593
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.10.1-15.v0d99f670e0a_7
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hidden-parameter:237.v4b_df26c7a_f0e
htmlpublisher:1.33
http_request:1.18
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jenkins-multijob-plugin:627.v7c23cef20a_6a
jersey2-api:2.42-147.va_28a_44603b_d5
jira:3.13
jira-steps:2.0.165.v8846cf59f3db
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
job-dsl:1.87
job-import-plugin:3.6
jobConfigHistory:1229.v3039470161a_d
jobcopy-builder:1.5.0
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:719.vcb_d039b_77d0d
leastload:3.0.0
lockable-resources:1246.v28b_e4cc6fa_16
log-parser:2.3.3
mail-watcher-plugin:1.19
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-40.v58107308b_7a_7
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
maven-artifact-choicelistprovider:1.17
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
momentjs:1.1.1
msbuild:1.31
mstest:1.0.5
multibranch-build-strategy-extension:48.v3dc306525d0c
multibranch-scan-webhook-trigger:1.0.11
naginator:1.436.vb_e769dcb_cdf6
nant:248.vcc8a_3eec8db_a
next-build-number:1.8
nexus-artifact-uploader:2.14
nexus-jenkins-plugin:3.16.510.v4d23e22cf563
node-iterator-api:55.v3b_77d4032326
nodelabelparameter:1.12.0
nunit:485.ve8a_85357320d
oauth-credentials:0.646.v02b_66dc03d2e
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
parameter-separator:166.vd0120849b_386
parameterized-scheduler:262.v00f3d90585cc
parameterized-trigger:787.v665fcf2a_830b_
performance:957.v658a_7065b_92a_
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:179.vc5cb_98f6db_38
plugin-usage-plugin:4.4
plugin-util-api:4.1.0
popper-api:1.16.1-3
popper2-api:2.11.6-4
powershell:2.1
prism-api:1.29.0-13
prometheus:2.5.3
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
remote-file:1.24
resource-disposer:0.23
run-condition:1.7
saml:4.464.vea_cb_75d7f5e0
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
signtool-jenkins-plugin:0.3
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
subversion:1256.vee91953217b_6
theme-manager:215.vc1ff18d67920
thinBackup:1.19
throttle-concurrents:2.14
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
uno-choice:2.8.3
urltrigger:1.02
validating-string-parameter:183.v3748e79b_9737
variant:60.v7290fc0eb_b_cd
versioncolumn:243.vda_c20eea_a_8a_f
view-job-filters:369.ve0513a_a_f5524
vsphere-cloud:2.27
warnings-ng:11.2.2
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-cps-global-lib:612.v55f2f80781ef
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-remote-loader:1.6
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45
xtrigger-api:1.0
xunit:3.1.4

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins master: Oracle Linux 9
Jenkins agents: Oracle Linux 9 + Windows Server 2019

Reproduction steps

The issue cannot be reproduced. In the last couple of months, we had it twice.
Symptoms:

1. Jobs using labels defined in Docker agent templates do NOT start!
2. No corresponding entries in the log of com.nirima.jenkins.plugins.docker!

Expected Results

The Docker plugin should react and

1. Pick a Docker cloud in which a Docker image template with the given label is defined.
2. Create Docker containers in the selected Docker host.
3. Start the job in the container.

Actual Results

No reaction from the plugin. Triggered jobs wait endlessly for their execution as their labels cannot be resolved.

Anything else?

Current workaround: Restarting the complete Jenkins application solves the problem.

Are you interested in contributing a fix?

No response

[MarkEWaite]

I can't duplicate this issue. The agents start reliably for me. If more details become available that make the problem consistently visible, please share that information.