Repeated TypeConverter Registration Causes Degraded Performance and Crashes

Description

In a long-lived Spark Streaming application using TypeConverters, over time the ConnectionFactory will be invoked repeatedly as long-running stages cause us to hit 'spark.cassandra.connection.keep_alive_ms'.

The TypeConverters will be registered repeatedly. The SPARKC code appears to allow this, and the more times the TypeConverters are registered the worse the Cassandra write performance gets. Eventually our application will crash (performance gets bad, scheduling delay piles up...).

In the Datastax Java Driver, it simply logs a warning and does nothing if you attempt to register a Codec repeatedly. Maybe we want the same behavior for TypeConverters here?

Environment

None

Pull Requests

None

Status

Assignee

Unassigned

Reporter

JeffJ

Labels

None

Reviewer

None

Reviewer 2

None

Tester

None

Pull Request

None

Affects versions

Priority

Major
Configure