Sometimes dgraph cannot end a term

[makitka2007]

dgraph version: 1.0.13

we have this case periodically

in logs there are repeated messages about votes for term 39, and they never end. only when i restart nodes, the election completes fine

  | Mar 13 11:18:47.286 | i-0b6227338f21151a6 | dgraph-zero | [0xa] Read index context timed out
  | Mar 13 11:18:47.218 | i-0fbf4d4c6c9ee4907 | dgraph-zero | b [logterm: 39, index: 1103097, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:47.218 | i-019ff1d6e2e87bfed | dgraph-zero | c [logterm: 39, index: 1103097, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:47.217 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to c at term 39
  | Mar 13 11:18:47.217 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to b at term 39
  | Mar 13 11:18:47.216 | i-0b6227338f21151a6 | dgraph-zero | a received MsgPreVoteResp from a at term 39
  | Mar 13 11:18:47.216 | i-0b6227338f21151a6 | dgraph-zero | a became pre-candidate at term 39
  | Mar 13 11:18:47.216 | i-0b6227338f21151a6 | dgraph-zero | a is starting a new election at term 39
  | Mar 13 11:18:44.838 | i-0fbf4d4c6c9ee4907 | dgraph-zero | b [logterm: 39, index: 1103013, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:44.838 | i-019ff1d6e2e87bfed | dgraph-zero | c [logterm: 39, index: 1103013, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:44.837 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to c at term 39
  | Mar 13 11:18:44.836 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to b at term 39
  | Mar 13 11:18:44.836 | i-0b6227338f21151a6 | dgraph-zero | a received MsgPreVoteResp from a at term 39
  | Mar 13 11:18:44.836 | i-0b6227338f21151a6 | dgraph-zero | a became pre-candidate at term 39
  | Mar 13 11:18:44.836 | i-0b6227338f21151a6 | dgraph-zero | a is starting a new election at term 39
  | Mar 13 11:18:44.286 | i-0b6227338f21151a6 | dgraph-zero | [0xa] Read index context timed out
  | Mar 13 11:18:42.458 | i-0fbf4d4c6c9ee4907 | dgraph-zero | b [logterm: 39, index: 1102943, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:42.458 | i-019ff1d6e2e87bfed | dgraph-zero | c [logterm: 39, index: 1102943, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:42.457 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to b at term 39
  | Mar 13 11:18:42.456 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to c at term 39
  | Mar 13 11:18:42.456 | i-0b6227338f21151a6 | dgraph-zero | a received MsgPreVoteResp from a at term 39
  | Mar 13 11:18:42.456 | i-0b6227338f21151a6 | dgraph-zero | a became pre-candidate at term 39
  | Mar 13 11:18:42.456 | i-0b6227338f21151a6 | dgraph-zero | a is starting a new election at term 39
  | Mar 13 11:18:41.502 | i-0b6227338f21151a6 | dgraph-zero | Got connection request: cluster_info_only:true
  | Mar 13 11:18:41.286 | i-0b6227338f21151a6 | dgraph-zero | [0xa] Read index context timed out
  | Mar 13 11:18:40.078 | i-0fbf4d4c6c9ee4907 | dgraph-zero | b [logterm: 39, index: 1102859, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:40.078 | i-019ff1d6e2e87bfed | dgraph-zero | c [logterm: 39, index: 1102859, vote: c] rejected MsgPreVote from a [logterm: 39, index: 1099432] at term 39
  | Mar 13 11:18:40.077 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to c at term 39
  | Mar 13 11:18:40.077 | i-0b6227338f21151a6 | dgraph-zero | a [logterm: 39, index: 1099432] sent MsgPreVote request to b at term 39
  | Mar 13 11:18:40.076 | i-0b6227338f21151a6 | dgraph-zero | a received MsgPreVoteResp from a at term 39
  | Mar 13 11:18:40.076 | i-0b6227338f21151a6 | dgraph-zero | a became pre-candidate at term 39
  | Mar 13 11:18:40.076 | i-0b6227338f21151a6 | dgraph-zero | a is starting a new election at term 39
  | Mar 13 11:18:39.901 | i-0b6227338f21151a6 | dgraph-zero | Connected: cluster_info_only:true
  | Mar 13 11:18:38.285 | i-0b6227338f21151a6 | dgraph-zero | [0xa] Read index context timed out

bigger log entry here: https://drive.google.com/file/d/17tAMSGgqj_naqKHOnkp8nQmh8OTgtKC5/view?usp=sharing

[manishrjain]

Seems like this particular Zero is stuck trying to achieve a quorum. I think this might be fixed by #3138 .

[manishrjain]

Is this still an issue or should we close this?