Control connections are retried too frequently causing the server outage

The control connection does not check if a reconnection attempt is already in progress. We get multiple remove events that time out on refreshNodeListAndTokenMap, each scheduling a concurrent reconnection attempt.

To clarify my previous comment, the control connection does prevent simultaneous attempts, but each new attempt cancel the previous one, which causes much shorter delays than what a single attempt would produce. There was also a subtle race in that cancellation. I'm refactoring the code to fix that race and let the initial attempt run.

There is still a race condition in AbstractReconnectionHandler that will lead to a thread leak :

1. Multiple threads create a new ARH instance simultaneously and call .start().

   1. Those ARH all share the same AtomicReference<ListenableFuture<?>> for the current attempt.

2. Those threads are going to schedule a task in the executor, which will result in method run() being called

3. Those threads are context-switched out

4. The executor now runs the newly submitted tasks that perform the following

   1. Check whether they have been canceled, which is currently false for both tasks

   2. Enter a sleeping loop until the isActive variable is set to true

5. The two previous threads are now put back in the CPU by the scheduler

   1. One of them will eventually cancel its associated task because they detect multiple tasks have been submitted.

6. The task that should be cancelled is not stucked in the sleep loop, as it will never check whether it is canceled anymore.

7. Eventually, the JVM will run out of native thread

Possible fixes include :

• call .cancel(true) so that the task's thread is actually interrupted

• replace the isActive boolean by a more suitable concurrent data structure (such as Semaphore or CountDownLatch) and remove the sleep loop

Just pushed the fix I did yesterday for that.