CassandraSparkExtensions crash with Spark 3.1.0 & Databricks Runtime 7.x
If I use DirectJoin on the Spark 3.1.0-SNAPSHOT, or on Databricks Runtime 7.x, like this:
When trying to explain it or perform, it generates the exception.
Unfortunately, after discussion with developer, it looks like that return them back will cause another big refactoring that is hard to do before Spark 3.1 freeze. The fix on the SCC side is just changing imports in 2 files (PR will follow shortly)
Databricks Runtime 7.x (7.0/7.2/7.3), OSS Spark 3.1.0-SNAPSHOT
You can build from the open pull request
We have the same issue to upgrade spark too, thanks in advance!
Dear Team. Any update on this. Looks like this is a blocker for 3.1.1 upgrade :).