Linq: StatementFactory shouldn't cache failed prepared statements
Description
When accessing a table via linq that has been created in Cassandra after an initial attempt to access the table via linq an "unconfigured table" exception is thrown.
Reproduction Steps:
Attempt to insert into a non-existent table using linq
Create the non-existent table
Attempt to insert into the table using linq
Expected:
The second insert attempt to succeed
Actual
An "InvalidQueryException" is thrown with an "unconfigured table" message
I have created a gist to reproduce the issue. Just switch out the host and port if necessary.
https://gist.github.com/StephenDiligent/717f70f4a060641291bd69aa238260cb
Environment
Single node from docker image "cassandra:3.11.0"
Activity
The issue is related to how schema changes are propagated in your cluster.
The driver exposes SetMaxSchemaWaitSeconds() method in the ProtocolOptions to allow users to define the amount of time it should wait for all nodes to agree on the schema versions, which is 10 seconds by default.
When there isn't an agreement after that period of time, the driver will log a warning and continue. You should see the warning driver logs.
I have been reproducing this behavior on a single node cluster.
I have also reproduced the behavior on a 3 node cluster with a service that was running for hours after the schema changes were applied.
Also, the behavior is not seen when using simple or bound statements, only with linq2cql.
I have created a new gist that highlights this issue by performing linq, simple statement and prepared statement inserts repetitively for a set period of time.
The simple statement and prepared statement inserts succeed but the linq insert continues to fail.
https://gist.github.com/StephenDiligent/7437080982e8c64d52d5ff5eefb061b6
Thanks for a great snippet with minimal code to reproduce it!
I was able to reproduce it and find the underlying cause:
The internal prepared statement cache used by Linq caches the task used to prepare, if the task is faulted its added anyway and it will never recover from its faulted state: https://github.com/datastax/csharp-driver/blob/3.4.0/src/Cassandra/Mapping/Statements/StatementFactory.cs#L42
This looks like the desired behaviour, for example, when a query couldn't be prepared for a syntax error it doesn't make sense to recover from it. But in the case you mentioned, it isn't the desired behaviour.
The solution would be to look at the cached task and if faulted, re-prepare every time.
I've changed the title to make it easier to identify the problem and solution without reading the thread.
