For most of the day (it looks somewhat related to load) our production servers throw this exception when querying or updating a specific table in C*:
This might be related to CSHARP-526, on that ticket the suggested workaround is to switch to StreamingMode: https://docs.datastax.com/en/drivers/csharp/3.16/api/Cassandra.SocketOptions.html#Cassandra_SocketOptions_SetStreamMode_System_Boolean_
Which OS are you using with Mono? Ubuntu? Windows Server?
Taking into account I recommend using the workaround to enable the StreamingMode when using Mono until it is clear whether this is an inconsistency with the Mono implementation of SocketAsyncEventArgs or if it is actually something that the driver must handle.
Since the implementation is correct for the oficially supported platforms (.NET Core and .NET Framework) and considering that Mono is going to be unified with .NET in the near future, I propose we close this as "Won't Fix". However I would like to know if the "Streaming Mode" fixes your issue before closing this.
Ok thanks, the workaround of retrying requests seems to work fine so far so it is not causing us any issues. Would the “Streaming Mode” mean blocking I/O and hence more threads being used ? Just want to be careful as it only happens in our prod environment and I’m a bit hesitant about making major changes.
"Streaming Mode" is basically replacing SocketAsyncEventArgs with NetworkStream but both approaches perform non blocking I/O.
In theory, Network Stream should be less performant because it's a higher level of abstraction but in our benchmarks we found the difference to be very small in most cases and even not noticeable in some cases. However, we haven't performed any benchmarks on Mono so the best approach would be to perform some benchmarks with your app on a test environment before switching the mode on production.
This issue usually causes that specific connection to stay unusable until it is closed and a new one is opened so maybe the driver (or the server) is closing the connection and that's why your retry approach works?