Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Jenkins and Travis fail because of timeout on closing Connector

Description

1. Jenkins and Travis fail because connector does not shut down cleanly during integration tests. Shutting down process hangs until timeout elapses (200 minutes for Jenkins, 10 minutes for Travis).
Some examples:

https://travis-ci.org/datastax/spark-cassandra-connector/jobs/128178951
https://travis-ci.org/datastax/spark-cassandra-connector/jobs/131453192
http://jenkins2.datastax.lan:8080/job/DSE_SparkCassandraConnector_2.10_PullRequest_Builder/958/console

see thread

Cluster.CloseAsync.get hangs

2. Cannot allocate memory:

http://jenkins2.datastax.lan:8080/job/DSE_SparkCassandraConnector_2.10_PullRequest_Builder/956/consoleFull
http://jenkins2.datastax.lan:8080/job/DSE_SparkCassandraConnector_2.11_PullRequest_Builder/914/consoleFull
http://jenkins2.datastax.lan:8080/job/DSE_SparkCassandraConnector_2.11_PullRequest_Builder/923/console

3. Jenkins post build script fails:

http://jenkins2.datastax.lan:8080/job/DSE_SparkCassandraConnector_2.10_PullRequest_Builder/965/consoleFull

Pull Requests

https://github.com/datastax/spark-cassandra-connector/pull/976

Attachments

1

Activity

Show:

Jaroslaw Grabowski May 17, 2016 at 11:50 AM
Edited

1. It seams that shutting down cluster and all client connections at once lead to occasional deadlock, see

and exception from logs:

Shutting down cluster and clients sequentially should workaround the problem.
2. Was caused by (1), deadlocked processes occupied memory.
3. Only one Jenkins node is affected by this problem, need to investigate with admin team. problem was solved for that node.

Fixed

Details

Assignee

Reporter

Reviewer

Components

Priority

Created May 6, 2016 at 1:49 PM
Updated May 23, 2016 at 8:35 AM
Resolved May 23, 2016 at 8:35 AM