Make connection and pool creation fully asynchronous

Description

When we create a Connection, we block on multiple futures: the ChannelFuture from the initial connect call, and the Connection.Future's from initialization queries.

HostConnectionPool creates its connections sequentially, so the thread that creates a pool blocks for each connection. For this reason, we schedule connection pool creations on dedicated thread pools (see SessionManager#maybeAddPool and SessionManager#forceRenewPool.

We could:

  • chain the futures in Connection and expose the final one as a Connection.readyFuture that you would have to wait on after calling the constructor;

  • aggregate the readyFuture's of all connections in a HostConnectionPool.readyFuture.

This way, connections would be opened in parallel, and we wouldn't need an extra thread to wait for the pool to be ready.

Environment

None

Pull Requests

None

Activity

Show:
Andy Tolbert
April 9, 2015, 3:46 AM
Edited

Ran some time trials with the pull request code vs. the 2.0 branch. I'm using the same test from .

Tested with both a 12 node cluster and 40 node cluster. The cassandra nodes were n1-standard-2 gce instances running 2.1.3. The client ran on the same network as the cassandra nodes. "Thread count" indicates the configured size of the cluster.manager.executor thread pool. Time measured in milliseconds.

Observations:

  1. java692 implementation outperforms 2.0 in every case.

  2. the size of the executor pool does not impact connect time in the java692 implementation where in 2.0 it obviously does. This is evidence of the benefit of making pool creation fully asynchronous.

  3. Having larger clusters does not significantly impact performance with java692 like it does with 2.0, though I assume the slowest connection will be the largest impact on total time since all connections are established at once.

  4. Where largest improvement is shown is when using authentication. java692 implementation shows a ~3x improvement with a 12 node cluster and 8 threads, and a ~8x improvement with a 40 node cluster. I anticipate as the nodes or core connections grows, this improvement factor will grow.

Without Authentication

branch

node count

thread count

min

mean

75%

95%

max

java692

12

2

26.99

55.93

62.50

99.89

114.37

java692

12

8

27.56

52.77

59.91

79.21

109.17

2.0

12

2

75.83

91.84

96.19

107.57

155.72

2.0

12

8

42.07

63.75

71.47

105.73

137.36

java692

40

2

47.61

112.95

140.81

203.09

226.87

java692

40

8

42.19

103.51

128.46

159.72

173.67

2.0

40

2

225.22

262.89

257.58

311.62

986.84

2.0

40

8

104.76

129.49

137.83

159.96

235.89

With Authentication

branch

node count

thread count

min

mean

75%

95%

max

java692

12

2

178.37

214.54

226.62

273.87

331.41

java692

12

8

179.72

212.99

226.60

271.47

290.37

2.0

12

2

1487.23

1518.65

1522.95

1559.44

1718.34

2.0

12

8

649.66

673.46

678.25

713.86

757.79

java692

40

2

184.67

249.57

280.73

337.68

373.83

java692

40

8

189.70

251.80

281.41

331.48

361.83

2.0

40

2

4684.88

4748.61

4770.99

4833.74

4948.69

2.0

40

8

1901.24

1929.94

1939.57

1967.56

2080.85

Fixed

Assignee

Olivier Michallat

Reporter

Olivier Michallat

Labels

None

PM Priority

None

Affects versions

Fix versions

Pull Request

None

Doc Impact

None

Size

None

External issue ID

None

External issue ID

None

Priority

Minor