Shepherd updates back and forth between sha version and latest #110
Comments
|
may i ask what you are using for notifications? |
Telegram |
|
@GuyKh Could you please verify whether this issue still persists with the latest shepherd version? If yes, please run shepherd with Make sure to update your image specifier to |
|
@moschlar latest update occurred 2 months ago, so is your question relates to the last 2 months - in this case - the answer is definitely YES I can tell that since Jan 17th, things have been quiet on this front |
|
It's here again :)
|
|
From that log and screenshot I can't see that any service is "flapping"... Can you share your Docker Swarm stack files? |
|
Looking at the swarm files - I was using Retrying this with |
|
Well... this still occurs:
|
|
The updating image of What I can't really tell is From sha to non-sha. Is it because same sha? I'm not sure. |
|
I'm really haven't looked in, but I think the issue is the resolving of |
|
Like I said yesterday, from your recent reports, I can not see the issue that you have been describing in your first post and the title of this issue. Please try to reproduce this with the latest official image and show us the logging output. |
|
I'm experiencing this issue as well. Here are the messages from Apprise: [Shepherd] Service backend_file-proxy updated on 5d4000a267bd
Wed Apr 3 02:49:03 CEST 2024 Service backend_file-proxy was updated from [ghcr.io/REDACTED/file-proxy:1.0@sha256:d9a3268c22892f7272773cd9b6caabe3630c6877f8b28373f7dbb57822f6d342](http://ghcr.io/REDACTED/file-proxy:1.0@sha256:d9a3268c22892f7272773cd9b6caabe3630c6877f8b28373f7dbb57822f6d342) to [ghcr.io/REDACTED/file-proxy:1.0](http://ghcr.io/REDACTED/file-proxy:1.0)
AppriseApprise | Today at 2:49 AM
[Shepherd] Service backend_file-proxy updated on 5d4000a267bd
Wed Apr 3 03:00:26 CEST 2024 Service backend_file-proxy was updated from [ghcr.io/REDACTED/file-proxy:1.0](http://ghcr.io/REDACTED/file-proxy:1.0) to [ghcr.io/REDACTED/file-proxy:1.0@sha256:d9a3268c22892f7272773cd9b6caabe3630c6877f8b28373f7dbb57822f6d342](http://ghcr.io/REDACTED/file-proxy:1.0@sha256:d9a3268c22892f7272773cd9b6caabe3630c6877f8b28373f7dbb57822f6d342)
AppriseApprise | Today at 3:00 AMWe're experiencing this service using either Docker Hub or GitHub container registry, so the problem is probably not the registries. Here are the logs from shepherd (verbose): Wed Apr 3 02:48:57 CEST 2024 Trying to update service backend_file-proxy with image ghcr.io/REDACTED/file-proxy:1.0
image ghcr.io/REDACTED/file-proxy:1.0 could not be accessed on a registry to record
its digest. Each node will access ghcr.io/REDACTED/file-proxy:1.0 independently,
possibly leading to different nodes running different
versions of the image.
Wed Apr 3 02:49:03 CEST 2024 Service backend_file-proxy was updated!
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 323 100 2 100 321 9 1446 --:--:-- --:--:-- --:--:-- 1461
Wed Apr 3 03:00:25 CEST 2024 Trying to update service backend_file-proxy with image ghcr.io/REDACTED/file-proxy:1.0
Wed Apr 3 03:00:26 CEST 2024 Service backend_file-proxy was updated!
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 323 100 2 100 321 10 1664 --:--:-- --:--:-- --:--:-- 1682The example has a defined version tag, but this is also happening to services using images with a latest tag. shepherd:
image: containrrr/shepherd
environment:
SLEEP_TIME: '5m'
FILTER_SERVICES: 'label=shepherd.autodeploy'
ROLLBACK_ON_FAILURE: 'true'
REGISTRIES_FILE: /var/run/secrets/shepherd-registries-auth
WITH_REGISTRY_AUTH: 'true'
APPRISE_SIDECAR_URL: 'notify:5000'
TZ: Europe/Berlin
secrets:
- shepherd-registries-auth
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- notification
deploy:
placement:
constraints:
- node.role == managerbtw. thanks for a great service. Shepherd has really improved our deployment strategy. |
Just reproduced this with latest version: Logs: |
|
I guess this might be connected to the docker version. Can you tell us the version @GuyKh ? It would be good to know if these commands both return sha-hashed image ids, in your cluster: docker service inspect general_ouroboros -f '{{.PreviousSpec.TaskTemplate.ContainerSpec.Image}}'
docker service inspect general_ouroboros -f '{{.Spec.TaskTemplate.ContainerSpec.Image}}' |
|
Last message was: Here are my stats: Looks like all latest messages include |
|
Mhh, not sure what to make out of this. I have to say I did not test shepherd with docker 26 yet myself. |
|
I was running into this issue on docker 20, but it still exists in docker 26. What I also notice is the sha doesn’t match between what shepherd pulls and what |
|
I am not sure this is the root cause, but I am able to create a service with an image without the digest by doing the following:
From this post: https://stackoverflow.com/questions/39811230/why-doesnt-my-newly-created-docker-have-a-digest |
|
Based on this comment, run |
|
So based on @shizunge's comments, this sounds like a docker / usability problem rather than a shepherd bug. |
See pic:
Very often I'm getting two updates, one from latest (for example, or a versioned image) to a version with sha and then back to non-sha one.
e.g.
The text was updated successfully, but these errors were encountered: