Driver should detect mixed version clusters (during upgrade) and use appropriate protocol version

Pushed some work in progress, don't look at it yet it's not tested. I will open a pull request when it's ready.

We've revisited this issue recently. Considering that everyone with a mixed cluster falls into that trap (and quite frankly the blame is not on you, given how tricky that is), we've decided to address it in driver 4. The solution is what I described in a previous comment:

Another approach would be to query system.peers.release_version on the initial node, and then make our decision based on that. There are a couple of (minor) drawbacks:

• we have to maintain a (release_version => protocol version) mapping in the driver's code, which kind of defeats the purpose of the protocol version.

• you still need negotiation before that first query, so the worst case scenario now reopens the initial connection twice.

This does add a bit of complexity to the initial connection sequence, but that's well worth it if it prevents users from shooting themselves in the foot.

As for the slower startup costs cant that be mitigated by the following:

1. if protocol version is specified then automatically just use that and skip the discovery step, and just behave like we do today. Those that are trying to optimize just will do so via manual protocol specification and therefore will follow best practice (and working but slow is better than down anyway. One is an optimization problem the other causes pagers to go off).
2. Eventually we could look at having cassandra broadcast version. I have to think this through but we do similar with web services in the app dev world. It's typically a mandatory part of any rolling upgrade support in a services world.

Final point, ghost nodes are a huge problem period I think rejecting a connection to it is what everyone would want ( I think surveying some end users would back that up for everyone ). Hope this helps.

The protocol version is negotiated with the first contact point: try first with the driver's highest version, if that doesn't work the server replies with its highest version, we close the connection and reopen a new one. Translated to the whole cluster, this would look like this:

• try each node with the driver's highest version

• wait for all responses

• pick the lowest version from all the replies (hopefully the higher nodes also support it, otherwise there is something very wrong in your mixed cluster)

There are a number of issues with this approach:

• performance, as mentioned by Alex (N connections instead of 1 or 2 connections)

• what if some nodes are down?

• by contract the driver is not supposed to connect to non-contact points until Cluster.connect, negotiation happens before that (Cluster.init)

• by contract the driver is not supposed to connect to nodes that are ignored by the load balancing policy (this can be dynamic, so some nodes might be ignored when you initialize the cluster, but become valid later).

Another approach would be to query system.peers.release_version on the initial node, and then make our decision based on that. There are a couple of (minor) drawbacks:

• we have to maintain a (release_version => protocol version) mapping in the driver's code, which kind of defeats the purpose of the protocol version.

• you still need negotiation before that first query, so the worst case scenario now reopens the initial connection twice.

IMHO the real solution would be to disable protocol negotiation by default. The mixed cluster issue is already documented (manual, javadoc), but because it's not something users need to actively enable, they usually miss that.

The main argument for negotiation by default is ease-of-use for first-time users. Unfortunately it gives them an opportunity to shoot themselves in the foot in the long run.

Shouldn't the cluster be able to know if it is operating in a mixed version environment?

The big annoyance here is that, if you don't fix the version of the driver to the prior version, rolling upgrades do not go well. Any connections established with a DSE 5.x node cannot then communicate with any 4.x node.

Until a DC is fully upgraded, you see lopsided performance bubbles across the nodes in the DC.