For most of the day (it looks somewhat related to load) our production servers throw this exception when querying or updating a specific table in C*:
Exception Stacktrace
Cassandra driver:
CassandraCSharpDriver 3.16.0
Cassandra version:
3.11.3
We don't actively test the driver against Mono but I can try to find some time to reproduce this with that Mono version. Can you post the schema of that table and some example queries that your application executes?
Ok thanks very much. Have added script for creating the table and some example statements.
How large is the message column usually? I see that your example statement has bigintAsBlob(3) but I'm assuming this is just for example purposes, I imagine that the real statement that your application is sending has a much larger blob.
The reason I ask this is because this issue might be related to the frame size so I should use a similar size when I attempt to reproduce this (or at least a big enough size that matches your largest message size).
Yes bigintAsBlob(3) is just an example. Most message data is around 1-2 kb but in some cases it can be around 100-120 kb (it’s basically a serialized list of messages so arbitrary binary data). I think we had this error occur more often with customers that had more data in the messages column.
This might be related to CSHARP-526, on that ticket the suggested workaround is to switch to StreamingMode: https://docs.datastax.com/en/drivers/csharp/3.16/api/Cassandra.SocketOptions.html#Cassandra_SocketOptions_SetStreamMode_System_Boolean_
Which OS are you using with Mono? Ubuntu? Windows Server?
Taking into account I recommend using the workaround to enable the StreamingMode when using Mono until it is clear whether this is an inconsistency with the Mono implementation of SocketAsyncEventArgs or if it is actually something that the driver must handle.
Since the implementation is correct for the oficially supported platforms (.NET Core and .NET Framework) and considering that Mono is going to be unified with .NET in the near future, I propose we close this as "Won't Fix". However I would like to know if the "Streaming Mode" fixes your issue before closing this.
Ok thanks, the workaround of retrying requests seems to work fine so far so it is not causing us any issues. Would the “Streaming Mode” mean blocking I/O and hence more threads being used ? Just want to be careful as it only happens in our prod environment and I’m a bit hesitant about making major changes.
"Streaming Mode" is basically replacing SocketAsyncEventArgs with NetworkStream but both approaches perform non blocking I/O.
In theory, Network Stream should be less performant because it's a higher level of abstraction but in our benchmarks we found the difference to be very small in most cases and even not noticeable in some cases. However, we haven't performed any benchmarks on Mono so the best approach would be to perform some benchmarks with your app on a test environment before switching the mode on production.
This issue usually causes that specific connection to stay unusable until it is closed and a new one is opened so maybe the driver (or the server) is closing the connection and that's why your retry approach works?