If I use DirectJoin on the Spark 3.1.0-SNAPSHOT, or on Databricks Runtime 7.x, like this:
When trying to explain it or perform, it generates the exception.
Unfortunately, after discussion with developer, it looks like that return them back will cause another big refactoring that is hard to do before Spark 3.1 freeze. The fix on the SCC side is just changing imports in 2 files (PR will follow shortly)
Databricks Runtime 7.x (7.0/7.2/7.3), OSS Spark 3.1.0-SNAPSHOT