I've tried several different scenarios, and it seems that regardless of whether or not a channel is being read from that OPTIONS messages are still being sent to the server for heartbeat purposes. I have a test that reproduces this behavior using simulacron here.
You can see that the sever is still showing the session as sending heartbeats, even though queries are being made and data is being returned. I'm not sure what is causing this as the netty idle behavior should be out of our hands.
Summarizing what we've found so far:
0. Heartbeat.fail enters us into an endless exception loop as Greg summarizes above. I think if we checked only for DriverTimeoutException (much like the 3.x callback handled for onTimeout only) that would solve the problem.
1. Heartbeats are getting sent even if the channel is busy. I think this is because HeartbeatHandler is too late in the pipeline (it should be before inflight handler)
2. Heartbeats are getting sent after the channel goes down. I think this is because we don't check to see if the channel is closed. Much like userEventTriggered in previous impl checked to see if the connection was initialized (we need to check that too) and that the connection isn't closed.
I think i have a handle on all of that, i'll take a stab at this and open up a PR.