TimeoutException on all requests (async deadlock ?)
Description
When deploying my app (which runs on cassandra) in production, I started to notice TimeoutExceptions throwing from time to time, while none of my 3 cassandra nodes had any load.
It started to happen more and more often as my server load was increasing. My server now rejects almost every request because of these timeouts... Which is, as you can imagine, a bit annoying
I managed to reproduce it with a simple example, which (on my local instance of cassandra or on my production cluster) results systematically in requests throwing TimeoutExceptions:
(code also attached in zip)
After a bit of investigation, it seems that there is an async deadlock somewhere.
When deploying my app (which runs on cassandra) in production, I started to notice TimeoutExceptions throwing from time to time, while none of my 3 cassandra nodes had any load.
It started to happen more and more often as my server load was increasing.
My server now rejects almost every request because of these timeouts... Which is, as you can imagine, a bit annoying
I managed to reproduce it with a simple example, which (on my local instance of cassandra or on my production cluster) results systematically in requests throwing TimeoutExceptions:
(code also attached in zip)
After a bit of investigation, it seems that there is an async deadlock somewhere.
I'm not clear about why it happens, though.
Am I missing something ?
Thanks !