"Unable to get Token Metadata" returned by Spark Connector 3.1.0 when a stale entry in system.peers is chosen
Description
After an upgrade to Spark Connector 3.1.0 was necessary for Cassandra 4.x, if a stale/invalid entry was left in the system.peers table and was used, the following would be returned and the connection would fail:
The Driver stacktrace:
Walmart regularly has toplogy changes that leave behind stale system.peers entries, they need the connection to still succeed in this scenario. This worked as expected in C* 3.11 with the 2.3.2 Connector
In case it matters, they set the following: listen_address: <IP>
After an upgrade to Spark Connector 3.1.0 was necessary for Cassandra 4.x, if a stale/invalid entry was left in the system.peers table and was used, the following would be returned and the connection would fail:
The Driver stacktrace:
Walmart regularly has toplogy changes that leave behind stale system.peers entries, they need the connection to still succeed in this scenario.
This worked as expected in C* 3.11 with the 2.3.2 Connector
In case it matters, they set the following:
listen_address: <IP>
rpc_address: 0.0.0.0
broadcast_rpc_address: <IP>