I recently migrated from spark streaming to structured streaming. After some adjustments to the code, it worked normally. However, sometimes it happens that in the middle of the execution the spark throws an exception.
The problem is that this keyspace exists, and the job sometimes works correctly for a long time before throwing this exception.
This is code for test:
My conf:
My sbt file:
I've already tested this on version 3.1.x too. Another thing I notice is that the error only happens to the cassandra cluster that is outside the server that is running the spark engine. In the project I have two databases. One is on the same host as the spark engine, and the other is on another server.
Major
I recently migrated from spark streaming to structured streaming. After some adjustments to the code, it worked normally. However, sometimes it happens that in the middle of the execution the spark throws an exception.
The problem is that this keyspace exists, and the job sometimes works correctly for a long time before throwing this exception.
This is code for test:
My conf:
My sbt file:
I've already tested this on version 3.1.x too. Another thing I notice is that the error only happens to the cassandra cluster that is outside the server that is running the spark engine. In the project I have two databases. One is on the same host as the spark engine, and the other is on another server.