Driver shouldn't retry on client timeout if statement is not idempotent
As stated in CASSANDRA-9086, the driver will automatically retry a CAS transaction whose connection has been disconnected while the transaction is in progress. This transaction will then silently retry on a new connection. This can break the client-observed linearizability of LWT, as shown below.
While the client can work around this for "IF NOT EXISTS" transactions, this behavior is not fixable for arbitrary state CAS.
For example, in the model of a CAS register with two concurrent clients, we can have the following flow:
REGISTER has state 1.
CLIENT 1 starts a LWT transaction that CAS 1 to 4.
CLIENT 1's connection drops but the transaction succeeds.
REGISTER has state 4.
CLIENT 2 concurrently has a LWT transaction to CAS 4 to 2.
CLIENT 2's transaction succeeds.
REGISTER has state 2.
CLIENT 1 retries the CAS from 1 to 4.
CLIENT 1 receives a failure to apply, since REGISTER has state 2.
From the perspective of clients, there has been one failed CAS from 1 to 4 and one successful CAS from 4 to 2. Given this history, there is no linearizable path from 1 to 2 for the register.
In general, the driver should never non-configurably retry LWT, instead throwing an exception that the user can handle.
After some internal discussion, it has been suggested that a more elegant solution would be to broaden the scope of RetryPolicy to let users choose what to do in the event of client timeouts, connection exceptions and maybe more.
Currently, RetryPolicy handles the following situations:
But the following situations are handled internally by the driver (see RequestHandler), and the user cannot change this behavior:
OperationTimedOutException (aka client timeouts): decision is retry on next host with same consistency level;
ConnectionException: decision is retry on next host with same consistency level;
OVERLOADED response: decision is retry on next host with same consistency level;
IS_BOOTSTRAPPING response: decision is retry on next host with same consistency level;
SERVER_ERROR response: decision is retry on next host with same consistency level;
UNPREPARED response: decision is retry on same host with same consistency level;
unexpected response: decision is retry on next host with same consistency level;
The tentative API would be :
The new method would be called for all other cases listed above except PREPARED responses, that would remain internally handled (I cannot think of any usefulness of exposing it through the API). The flag mighHaveBeenApplied indicates whether there are chances that the request was applied server-side or not.
It has also been suggested that inspecting the Statement's idempotent flag could actually be done in a wrapper policy instead of in the default one.
would you mind giving us your opinion about this idea? Is there any particular reason why these situations were left out of RetryPolicy and had their behavior hardcoded?
I am broadening the scope of this ticket to all non-idempotent statements (and not only to CAS statements as the driver cannot distinguish those from other kinds of statements).
The proposed fix will consist of inspecting the statement's idempotent flag. This is a simple fix, but comes with a huge behavioral change as the driver assumes by default that statements are not idempotent; in practice, this means that most statements will not be retried anymore in the event of a client timeout.
One thing to consider is that the driver doesn't know a priori that the query is an LWT, from its point of view all queries are text strings (except when using QueryBuilder of course). We could use the idempotent flag for that, but that would have important consequences as statements are considered idempotent by default.
Needs to be backported to 2.1.x too