Uploaded image for project: 'DataStax Java Driver for Apache Cassandra'
  1. JAVA-250

Low throughput sessions can throw "com.datastax.driver.core.exceptions.NoHostAvailableException" after a rolling restart of a cluster.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.4, 1.0.5
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      JRE 1.6 or 1.7

      Description

      {bold}Summary:{bold}

      Low throughput sessions can throw "com.datastax.driver.core.exceptions.NoHostAvailableException" after a rolling restart of a cluster. This is an issue because the driver doesn't attempt to reestablish connections to any available nodes on the first request following the restart.

      {bold}Steps to reproduce:{bold}

      1) Build cluster object, create session, run queries
      2) Stop running queries
      3) Roll a restart through the cluster, waiting for each node to come up before restarting the next
      4) Run a query. Error thrown: com.datastax.driver.core.exceptions.NoHostAvailableException

      {bold}Analysis:{bold}

      As the rolling restart progresses throught the cluster, connections (c.d.d.c.Connection) are marked as defunct by c.d.d.c.Connection.Dispatcher.channelClosed(). At the end of the rolling restart all connections have been marked as defunct. On the next call to c.d.d.c.RequestHandler.query() all connection throw a c.d.d.c.ConnecitonException from c.d.d.c.Connection.write() because the c.d.d.c.Connection.isDefunct flag is set to true. This causes the connections to be returned to the conneciton pool and be scheduled for reconnection, but that doesn't help save the current request. The current request fails with a c.d.d.c.NoHostAvailableException because all current connections are marked as defunct. This happens in c.d.d.c.RequestHandler.sendRequest() where all connections planned in the query fail because c.d.d.c.RequestHandler.query() returns false.
      I also plan to add a background thread to the KS REST api that runs a lightweight query periodically. This will prevent all connections from becoming defunct.

        Attachments

          Issue links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mpenick Michael Penick
              • Votes:
                7 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: