Linq: StatementFactory shouldn't cache failed prepared statements

Description

When accessing a table via linq that has been created in Cassandra after an initial attempt to access the table via linq an "unconfigured table" exception is thrown.

Reproduction Steps:

  1. Attempt to insert into a non-existent table using linq

  2. Create the non-existent table

  3. Attempt to insert into the table using linq

Expected:
The second insert attempt to succeed

Actual
An "InvalidQueryException" is thrown with an "unconfigured table" message

I have created a gist to reproduce the issue. Just switch out the host and port if necessary.
https://gist.github.com/StephenDiligent/717f70f4a060641291bd69aa238260cb

Environment

Single node from docker image "cassandra:3.11.0"

Activity

Show:
Jorge Bay Gondra
February 19, 2018, 7:44 PM

The issue is related to how schema changes are propagated in your cluster.

The driver exposes SetMaxSchemaWaitSeconds() method in the ProtocolOptions to allow users to define the amount of time it should wait for all nodes to agree on the schema versions, which is 10 seconds by default.

When there isn't an agreement after that period of time, the driver will log a warning and continue. You should see the warning driver logs.

Stephen Payne
February 19, 2018, 8:05 PM

I have been reproducing this behavior on a single node cluster.
I have also reproduced the behavior on a 3 node cluster with a service that was running for hours after the schema changes were applied.
Also, the behavior is not seen when using simple or bound statements, only with linq2cql.

Stephen Payne
February 22, 2018, 11:02 AM

I have created a new gist that highlights this issue by performing linq, simple statement and prepared statement inserts repetitively for a set period of time.
The simple statement and prepared statement inserts succeed but the linq insert continues to fail.
https://gist.github.com/StephenDiligent/7437080982e8c64d52d5ff5eefb061b6

Jorge Bay Gondra
February 22, 2018, 9:12 PM

Thanks for a great snippet with minimal code to reproduce it!

I was able to reproduce it and find the underlying cause:
The internal prepared statement cache used by Linq caches the task used to prepare, if the task is faulted its added anyway and it will never recover from its faulted state: https://github.com/datastax/csharp-driver/blob/3.4.0/src/Cassandra/Mapping/Statements/StatementFactory.cs#L42

This looks like the desired behaviour, for example, when a query couldn't be prepared for a syntax error it doesn't make sense to recover from it. But in the case you mentioned, it isn't the desired behaviour.

The solution would be to look at the cached task and if faulted, re-prepare every time.

Jorge Bay Gondra
May 8, 2018, 11:20 PM

I've changed the title to make it easier to identify the problem and solution without reading the thread.

Done

Assignee

Jorge Bay Gondra

Reporter

Stephen Payne

Labels

Reproduced in

None

PM Priority

None

Fix versions

External issue ID

None

Doc Impact

None

Reviewer

None

Pull Request

None

Epic Link

None

Sprint

None

Pull Requests

None

Size

None

Components

Affects versions

Priority

Minor