cluster stop working after 2 of 3 masters dead.

[liyimeng]

Environmental Info:
K3s Version:

1.25.6

Node(s) CPU architecture, OS, and Version:

ubuntu 22.04-lts

Cluster Configuration:

3 masters with embeded etcd + 1 node
Describe the bug:

Not sure if this is expected behaviour. If I shutdown the masters one by one, I expect ectd will elect the last survive one as leader and cluster continue to work. However, it is not the case, when two master go offline, the entire cluster fall apart. However, in early version (back to 1.23), this seems not a problem.

Steps To Reproduce:

• Installed K3s: 3 master with embedded etcd, and a worker node
• shutdown 1 master, cluster survive
• shutdown another master, cluster dead

Expected behavior:

cluster continue to work

Actual behavior:

cluster is dead
Additional context / logs:

[brandond]

If I shutdown the masters one by one, I expect ectd will elect the last survive one as leader and cluster continue to work.

No, that is not how etcd works, nor has it ever worked that way.
Ref: https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance

[liyimeng]

Thank @brandond