We currently have a SocketOptions.getReadTimeoutMillis() that is java-docd as The per-host read timeout in milliseconds.
However the fundamental assumption of the driver is that this be more than the default cassandra time out. When this time out occurs, number of driver behavior triggers such as marking the nodes as suspect. This time out is also used for Driver's internal communication on control connection, quick connection validation after node is suspect etc.
Some applications need the following behavior.
Do not wait longer than say 100 millis on a given node. If that node does not respond, simply move on to the next node.
You can not do that with SocketOptions.getReadTimeoutMillis() because it triggers all the undesired driver behavior on time out.
You can not do that with executeAsync() alone as that is meant for the whole request. We can kind of achieve this by executeAsync(low-time-out) followed by client retries. Even then, there are no guarantees that slow responding node is not picked again by load balancing policy specially on small clusters. Also every client wanting this will have to implement a client level retry every where they need this feature. Driver is not really helping much there.
We need driver to provide a true per host time-out that the client can use with out the driver side effects. On getting this time out, the driver should simply move on to the next node. Also driver should not use this time out for its internal purposes. This time-out should not have any relationship with cassandra server timeouts.
Though I don't think a ticket has been created yet, we've talked about adding some equivalent to the "rapid read retry" feature that Cassandra now implements. Basically, this would act like what you're describing here though would be slightly more flexible. Long story short, I agree something like this is needed, but this may not take the exact form described here (but if anything, it will be more general).
That ticket has been created: JAVA-561.
will meet our needs.