Column names with some of the non-alphanumeric characters are not fully supported

Description

Spark 3.2 support implementation has uncovered a change in Spark behavior regarding column names (attributes) with some of the non-alphanumeric characters, like ~. Spark 3.2 adds backticks to attirbute names which breaks matching in SolrPredicateRules. The matching was fixed by removing the backticks, but we need make sure that all the functionalities work well with non-alphanumeric characters. Direct join works, but IN clause doesn’t (with the new version of Spark and with the older one). The goal of this ticket is to provide a comprehensive set of tests for non-alphanumerical characters across the whole connector.

Environment

None

Pull Requests

None

Activity

Show:

Details

Assignee

Reporter

Priority

Created April 22, 2022 at 11:41 AM
Updated May 26, 2022 at 1:25 PM