It seems that when a remote Cassandra node goes down, after the java driver receives notification from cassandra, it attempt to reconnect to that node without taking into account load balancing policy.
The scenario that was observed is that driver is configured to use TokenAware load balancing policy wrapped around the DCAware load balancing policy with usedHostsPerRemoteDC = 0. When driver receives a notification from Cassandra indicating that a node goes down, it attempts to handshake with that node regardless of DC where that node resides, even though all cassandra nodes in local DC were up and were reachable.
I'd suggest this behaviour to be changed, if possible, so that it's possible to avoid handshaking or reconnection attempts to nodes on remote DC, when DC aware load balancing policy is used with usedHostsPerRemoteDc = 0.
This behaviour is initially found Java driver version 1.0.5. Quick code inspection on the latest version 1.0.7 of the Java driver shows that the behaviour is still the same in 1.0.7.
You are correct and that's definitively an oversight. Pushed a simple fixed to not start the reconnection attempts if the node is "IGNORED". Thanks for the report.
Sylvain, thanks for the fix. Is it possible to incorporate this change into 1.x version as well ? That way it'd be available to 1.x users.
It was incorporated in the 1.x branch (as the 'fix version' indicates I must add). But I just realized that the patch actually break reconnection attempts completely () so you may want to wait on 1.0.8 before upgrading.
Thanks Sylvain. Will wait for version 1.0.8 then.