[close #372] add circuit breaker and smart rawkv client #358
Conversation
marsishandsome
force-pushed
the
feature/support-CircuitBreaker
branch
27 times, most recently
from
38145a4
to
3b62a5c
Compare
|
/run-all-tests |
| this.listeners = new ArrayList<>(); | ||
| this.currentMetrics = new AtomicReference<>(new SingleWindowMetrics()); | ||
|
|
||
| scheduler = |
There was a problem hiding this comment.
should this background task be managed by the circuit breaker instead of the circuit metrics?
There was a problem hiding this comment.
I think it's a subtask belonging to the circuit breaker. For the metrics, it only needs to record the traffic statistics.
There was a problem hiding this comment.
BTW, we should not start this background thread when the circuit breaker is configured to be disabled.
There was a problem hiding this comment.
fixed in 5fcf19b
src/main/java/org/tikv/service/failsafe/CircuitBreakerMetricsImpl.java
Outdated
Show resolved
Hide resolved
Signed-off-by: marsishandsome <marsishandsome@gmail.com>
marsishandsome
force-pushed
the
feature/support-CircuitBreaker
branch
from
f42b506
to
929f3db
Compare
Signed-off-by: marsishandsome <marsishandsome@gmail.com>
marsishandsome
force-pushed
the
feature/support-CircuitBreaker
branch
from
2eea8ad
to
9932dab
Compare
Signed-off-by: marsishandsome <marsishandsome@gmail.com>
Signed-off-by: marsishandsome <marsishandsome@gmail.com>
| this.listeners = new ArrayList<>(); | ||
| this.currentMetrics = new AtomicReference<>(new SingleWindowMetrics()); | ||
|
|
||
| scheduler = |
There was a problem hiding this comment.
I think it's a subtask belonging to the circuit breaker. For the metrics, it only needs to record the traffic statistics.
| private final AtomicLong circuitOpened = new AtomicLong(-1); | ||
| private final AtomicReference<Status> status = new AtomicReference<>(Status.CLOSED); | ||
| private final AtomicLong attemptCount = new AtomicLong(0); | ||
| private final AtomicLong attemptSuccessCount = new AtomicLong(0); |
There was a problem hiding this comment.
-
do we really need to set these variables as atomic variables? I can only see they are accessed within the current TiSession.
-
we add the 'final' keyword to these fields but their values are changed during the execution time, seems a little counterintuitive.
There was a problem hiding this comment.
There are two-time windows and their corresponding traffic stats:
- the first time window is to record normal traffic stats when the circuit breaker is closed or disabled in the background and triggers circuit breaker, which is limited to
TiKV_CIRCUIT_BREAK_AVAILABILITY_WINDOW_IN_SECONDSand is called windowOfClosed. - the second time window to record the attempt traffic stats when the circuit breaker is half-opened, which is called windowOfHalfOpened.
these two windows can be posited like this: windowOfClosed -> (circuit breaker opened) -> (after sleep, circuit breaker half-opened) -> windowOfHalfOpened
So can we:
- Define two traffic metrics for these two time windows like the following and remove the atomic variables for attempts:
private final CircuitBreakerMetrics metricsOfClosed;
private final CircuitBreakerMetrics metricsOfHalfOpened- Make
CircuitBreakerImplbe the owner of these two windows, don't start the background task to collectmetricsOfClosedwhen the circuit breaker is disabled.
There was a problem hiding this comment.
do we really need to set these variables as atomic variables?
yes, in order to make CircuitBreakerImpl thread-safe.
There was a problem hiding this comment.
we add the 'final' keyword to these fields but their values are changed during the execution time, seems a little counterintuitive.
yes, we should add final and the object reference does not change. The object content does change.
| this.listeners = new ArrayList<>(); | ||
| this.currentMetrics = new AtomicReference<>(new SingleWindowMetrics()); | ||
|
|
||
| scheduler = |
There was a problem hiding this comment.
BTW, we should not start this background thread when the circuit breaker is configured to be disabled.
Signed-off-by: marsishandsome <marsishandsome@gmail.com>
zz-jason
changed the title
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
|
cherry pick to release-3.1 in PR #373 |
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: marsishandsome <marsishandsome@gmail.com>
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: marsishandsome <marsishandsome@gmail.com>
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: marsishandsome <marsishandsome@gmail.com>
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: marsishandsome <marsishandsome@gmail.com>
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: marsishandsome <marsishandsome@gmail.com> Co-authored-by: Liangliang Gu <marsishandsome@gmail.com>
Signed-off-by: marsishandsome marsishandsome@gmail.com
add Circuit Breaker