Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      cluster of 2 nodes created by ccm.
      uwsgi of 1 processes.

      Description

      I set my reconnection policy to ConstantReconnectPolicy, which is supposed to be reconnecting every 10 seconds.

      How to reproduce:
      Alll connections are created. Then I shut down the whole cluster with "ccm stop". uwsgi logs show:

      Connection <GeventConnection(67043600) 127.0.0.1:9042> closed by server
      Closing connection (67043600) to 127.0.0.1
      Closed socket to 127.0.0.1
      in-flight callbacks: {}
      Connection <GeventConnection(67915280) 127.0.0.1:9042> closed by server
      Closing connection (67915280) to 127.0.0.1
      Closed socket to 127.0.0.1
      in-flight callbacks: {}
      Connection <GeventConnection(67912336) 127.0.0.2:9042> closed by server
      Closing connection (67912336) to 127.0.0.2
      Closed socket to 127.0.0.2
      in-flight callbacks: {}

      As you can see reconnection is not initiated. "in-flight callbacks" line was added by me for debugging. It seems unless I send a new query, the driver won't reconnect.

      I noticed that reconnector is triggered when defunct/closed connection is returned to pool, which happens when there are callbacks in connection.

      The following code is how connection is closed from driver side, error_all_callbacks will call all pending callbacks, and if ResponseFuture._set_result callback in it, return_connection will be called and trigger the reconnect, otherwise no reconnect will be initiated:

      
          def close(self):
              with self.lock:
                  if self.is_closed:
                      return
                  self.is_closed = True
      
              log.debug("Closing connection (%s) to %s" % (id(self), self.host))
              if self._read_watcher:
                  self._read_watcher.kill(block=False)
              if self._write_watcher:
                  self._write_watcher.kill(block=False)
              if self._socket:
                  self._socket.close()
              log.debug("Closed socket to %s" % (self.host,))
      
              if not self.is_defunct:
                  self.*error_all_callbacks*(
                      ConnectionShutdown("Connection to %s was closed" % self.host))
                  # don't leave in-progress operations hanging
                  self.connected_event.set()
      

      I'm also wondering if a closed connection should be returned to pool?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              thoslin thoslin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: