Cassandra Timeouts Leaking Connections and causing deadlocks

Description

This seems to affect both 2.4.1 and 2.4.2 which we recently upgraded to when we started experiencing these issues. We are now on 2.4.2.

We've been experiencing a series of Spark job freezes recently as we've started seeing some read timeouts from Cassandra.

An example: this single stage has been "running" for 19 hours straight with no progression:

Note that there's 47 failed tasks. Some of those tasks are dead executors, some are timeouts reading from Cassandra.

Upon inspecting the tasks themselves, I noticed that there are 47 failed tasks and 21 running tasks. When inspecting the executors, we noticed there were two different types of "deadlocks". Note that we have 3 cores per executor.

1 of the threads (cores) is trying to establish a connection to Cassandra, the other two are waiting for the lock to be released:

You might notice the CustomizedCassandraConnectionFactory. Here is that class:

So nothing too crazy there (I think)

OR

1 of the threads is trying to create a PreparedStatement, and the other 2 are waiting to do one as well. We also noticed in the thread dump here a ThreadPool trying to call CassandraConnector#destroySession:

If it helps, only 1 executor had the prepared statement lock, all the others had the open connection lock.

The reason we think there are connection leaks is we did a netstat on the Cassandra nodes and it gave us back this:

Earlier this morning:

Number of Open Connections to Remote Host

IP Address of Remote Host

213

172.29.157.145

710

172.29.157.13

561

172.29.151.183

44

172.29.144.248

316

172.29.143.136

710

172.29.139.164

32

172.29.136.234

Later in the afternoon

Number of Open Connections to Remote Host

IP Address of Remote Host

212

172.29.157.145

710

172.29.157.13

736

172.29.151.183

584

172.29.144.248

316

172.29.143.136

710

172.29.139.164

387

172.29.136.234

Those IP addresses are the IPs of the spark worker nodes, which each have 30 cores, 3 cores per executor, so 10 executors per worker.
These numbers alone don't seem too crazy, since I think the appropriate number of connections per executor is 71 based on our configuration, however, if you look at the number of connections per PID on the workers, things seem really out of control:

Connections open to Cassandra

PID of process

8

129730

8

129477

8

129473

2

129181

32

129180

2

125796

339

79272

16

50754

266

40046

Ignoring the low numbers, that seems really high for that JVM. I checked a few hours later again and here are the new numbers on the same host:

Connections open to Cassandra

PID of process

25

129730

25

129477

25

129473

355

79272

16

23093

So something is definitely going wrong here. In this job we are querying Cassandra using the cassandraTable implicit defined on SparkContext if that helps.

Environment

None

Pull Requests

None

Status

Assignee

Piotr Kołaczkowski

Reporter

Ed Mitchell

Labels

None

Reviewer

None

Reviewer 2

None

Tester

None

Pull Request

None

Components

Affects versions

Priority

Major
Configure