Make connection and pool creation fully asynchronous
Description
When we create a Connection, we block on multiple futures: the ChannelFuture from the initial connect call, and the Connection.Future's from initialization queries.
HostConnectionPool creates its connections sequentially, so the thread that creates a pool blocks for each connection. For this reason, we schedule connection pool creations on dedicated thread pools (see SessionManager#maybeAddPool and SessionManager#forceRenewPool.
We could:
chain the futures in Connection and expose the final one as a Connection.readyFuture that you would have to wait on after calling the constructor;
aggregate the readyFuture's of all connections in a HostConnectionPool.readyFuture.
This way, connections would be opened in parallel, and we wouldn't need an extra thread to wait for the pool to be ready.
Environment
Pull Requests
Activity
Ran some time trials with the pull request code vs. the 2.0 branch. I'm using the same test from .
Tested with both a 12 node cluster and 40 node cluster. The cassandra nodes were n1-standard-2 gce instances running 2.1.3. The client ran on the same network as the cassandra nodes. "Thread count" indicates the configured size of the cluster.manager.executor thread pool. Time measured in milliseconds.
Observations:
java692 implementation outperforms 2.0 in every case.
the size of the executor pool does not impact connect time in the java692 implementation where in 2.0 it obviously does. This is evidence of the benefit of making pool creation fully asynchronous.
Having larger clusters does not significantly impact performance with java692 like it does with 2.0, though I assume the slowest connection will be the largest impact on total time since all connections are established at once.
Where largest improvement is shown is when using authentication. java692 implementation shows a ~3x improvement with a 12 node cluster and 8 threads, and a ~8x improvement with a 40 node cluster. I anticipate as the nodes or core connections grows, this improvement factor will grow.
Without Authentication
branch | node count | thread count | min | mean | 75% | 95% | max |
|---|---|---|---|---|---|---|---|
java692 | 12 | 2 | 26.99 | 55.93 | 62.50 | 99.89 | 114.37 |
java692 | 12 | 8 | 27.56 | 52.77 | 59.91 | 79.21 | 109.17 |
2.0 | 12 | 2 | 75.83 | 91.84 | 96.19 | 107.57 | 155.72 |
2.0 | 12 | 8 | 42.07 | 63.75 | 71.47 | 105.73 | 137.36 |
java692 | 40 | 2 | 47.61 | 112.95 | 140.81 | 203.09 | 226.87 |
java692 | 40 | 8 | 42.19 | 103.51 | 128.46 | 159.72 | 173.67 |
2.0 | 40 | 2 | 225.22 | 262.89 | 257.58 | 311.62 | 986.84 |
2.0 | 40 | 8 | 104.76 | 129.49 | 137.83 | 159.96 | 235.89 |
With Authentication
branch | node count | thread count | min | mean | 75% | 95% | max |
|---|---|---|---|---|---|---|---|
java692 | 12 | 2 | 178.37 | 214.54 | 226.62 | 273.87 | 331.41 |
java692 | 12 | 8 | 179.72 | 212.99 | 226.60 | 271.47 | 290.37 |
2.0 | 12 | 2 | 1487.23 | 1518.65 | 1522.95 | 1559.44 | 1718.34 |
2.0 | 12 | 8 | 649.66 | 673.46 | 678.25 | 713.86 | 757.79 |
java692 | 40 | 2 | 184.67 | 249.57 | 280.73 | 337.68 | 373.83 |
java692 | 40 | 8 | 189.70 | 251.80 | 281.41 | 331.48 | 361.83 |
2.0 | 40 | 2 | 4684.88 | 4748.61 | 4770.99 | 4833.74 | 4948.69 |
2.0 | 40 | 8 | 1901.24 | 1929.94 | 1939.57 | 1967.56 | 2080.85 |
