Auto reconnect or quickly exit when cassandra cluster down and then up. #1821
Labels
stale
A stale issue or PR that will automatically be closed.
Comments
|
I'm experiencing the same issue. I run Cassandra as a Statefullset within my K8s cluster. For some reason my Cassandra cluster became unhealthy. I initiated a restarted of the Cassandra pods and the cluster was healthy again, but i got the same error as you mention. I was able to get it working again by restarting the Loki pod, but this is not preferred. |
|
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
|
I also have this error on online loki |
|
|
4 tasks
MichelHollands
pushed a commit
that referenced
this issue
Nov 8, 2022
**What this PR does / why we need it**: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. **Which issue(s) this PR fixes**: Fixes #1821 #7140
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this issue
Nov 8, 2022
…a#6725) **What this PR does / why we need it**: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. **Which issue(s) this PR fixes**: Fixes grafana#1821 grafana#7140
Abuelodelanada
pushed a commit
to canonical/loki
that referenced
this issue
Dec 1, 2022
…a#6725) **What this PR does / why we need it**: level=error ts=2022-07-20T04:07:11.881370946Z caller=flush.go:146 org_id=166256_8sxv2 msg="failed to flush user" err="store put chunk: gocql: no hosts available in the pool" The restart of cassandra （on k8s） caused the production accident that loki had to restart. It happened 4 times in our production environment in only half a year. I hope this problem can be fixed. The code of this PR is not particularly complete, but this PR seems to be available. If this PR does not merge, I also hope that other contributors in the loki community can propose another PR to fix this problem. The restart of cassandra that caused loki to be unavailable is already a conclusion recorded in our JIRA. **Which issue(s) this PR fixes**: Fixes grafana#1821 grafana#7140
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Using cassandra cluster as storage.When the cassandra cluster down, this error occurs.And it will never recover though the cassandra is UP.
And if we want to restart loki, loki will fall because of loop forever.
Describe the solution you'd like
As gocql wounld fix it (eg. gocql-915 created at 2017 and never closed.)
Would loki do something to reconect? Or do not retry to flush forever.
Describe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: