Talking with some of the driver team @Olivier Michallat we discussed how for some use-cases it would be beneficial for the connector to use a larger connection pool.
For ease of use I suggest we automatically set the host pool to math.max(defaultParallelism/ numExecutors, defaultParallelism) at runtime when creating and RDD. This will be the default behavior unless the end user sets a specific param
Talking with some of the driver team @Olivier Michallat we discussed how for some use-cases it would be beneficial for the connector to use a larger connection pool.
For ease of use I suggest we automatically set the host pool to
math.max(defaultParallelism/ numExecutors, defaultParallelism) at runtime when creating
and RDD. This will be the default behavior unless the end user sets a specific param
spark.cassandra.connection.connections_per_executor_max
For users not running a Spark Context we'll fall back to just 1.