Make connection and pool creation fully asynchronous

Description

When we create a Connection, we block on multiple futures: the ChannelFuture from the initial connect call, and the Connection.Future's from initialization queries.

HostConnectionPool creates its connections sequentially, so the thread that creates a pool blocks for each connection. For this reason, we schedule connection pool creations on dedicated thread pools (see SessionManager#maybeAddPool and SessionManager#forceRenewPool.

We could:

chain the futures in Connection and expose the final one as a Connection.readyFuture that you would have to wait on after calling the constructor;
aggregate the readyFuture's of all connections in a HostConnectionPool.readyFuture.

This way, connections would be opened in parallel, and we wouldn't need an extra thread to wait for the pool to be ready.

Environment

None

Pull Requests

None

Linked work items

is duplicated by

JAVA-701

Improve connection initialization times

Activity

Andy Tolbert April 9, 2015 at 3:46 AM
Edited

Ran some time trials with the pull request code vs. the 2.0 branch. I'm using the same test from .

Tested with both a 12 node cluster and 40 node cluster. The cassandra nodes were n1-standard-2 gce instances running 2.1.3. The client ran on the same network as the cassandra nodes. "Thread count" indicates the configured size of the cluster.manager.executor thread pool. Time measured in milliseconds.

Observations:

java692 implementation outperforms 2.0 in every case.
the size of the executor pool does not impact connect time in the java692 implementation where in 2.0 it obviously does. This is evidence of the benefit of making pool creation fully asynchronous.
Having larger clusters does not significantly impact performance with java692 like it does with 2.0, though I assume the slowest connection will be the largest impact on total time since all connections are established at once.
Where largest improvement is shown is when using authentication. java692 implementation shows a ~3x improvement with a 12 node cluster and 8 threads, and a ~8x improvement with a 40 node cluster. I anticipate as the nodes or core connections grows, this improvement factor will grow.

Without Authentication

branch	node count	thread count	min	mean	75%	95%	max
java692	12	2	26.99	55.93	62.50	99.89	114.37
java692	12	8	27.56	52.77	59.91	79.21	109.17
2.0	12	2	75.83	91.84	96.19	107.57	155.72
2.0	12	8	42.07	63.75	71.47	105.73	137.36
java692	40	2	47.61	112.95	140.81	203.09	226.87
java692	40	8	42.19	103.51	128.46	159.72	173.67
2.0	40	2	225.22	262.89	257.58	311.62	986.84
2.0	40	8	104.76	129.49	137.83	159.96	235.89

With Authentication

branch	node count	thread count	min	mean	75%	95%	max
java692	12	2	178.37	214.54	226.62	273.87	331.41
java692	12	8	179.72	212.99	226.60	271.47	290.37
2.0	12	2	1487.23	1518.65	1522.95	1559.44	1718.34
2.0	12	8	649.66	673.46	678.25	713.86	757.79
java692	40	2	184.67	249.57	280.73	337.68	373.83
java692	40	8	189.70	251.80	281.41	331.48	361.83
2.0	40	2	4684.88	4748.61	4770.99	4833.74	4948.69
2.0	40	8	1901.24	1929.94	1939.57	1967.56	2080.85

Resize issue view side panel

Fixed

Details

Assignee

Olivier Michallat

Reporter

Olivier Michallat

Affects versions

2.0.9.2

2.1.5

Fix versions

2.0.10

Priority

Minor

Created March 16, 2015 at 10:49 AM

Updated April 20, 2015 at 1:30 PM

Resolved April 20, 2015 at 1:30 PM