Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

More data type mappings and Custom types in Catalyst for Sql SQL integration

Description

For UUIDType and InetType, ...., etc. Investigate creating custom Catalyst types

Pull Requests

https://github.com/datastax/spark-cassandra-connector/pull/734 (b1.3 branch) https://github.com/datastax/spark-cassandra-connector/pull/748 (master branch)

Activity

Show:

Alex Liu June 26, 2015 at 5:00 PM

Added a new pull against master branch. There are some compiling deprecated warning at current master branch for data frame related methods, they will be fixed in another ticket.

Piotr Kołaczkowski June 26, 2015 at 4:07 PM

please do a separate PR for master.

Piotr Kołaczkowski June 26, 2015 at 4:07 PM
Edited

Merged into b1.3, but failed to merge cleanly into master.
There is some spark 1.4 incompatibility, the code does not compile.

Alex Liu June 1, 2015 at 4:41 PM

Because InetAddressType is not a Spark internal supported data type, there is no
CAST method to convert a InetAddress to a String. Spark internally converts it
to Double, so it errors out the query such as

The correct query should be

The same should apply to UUIDType.

If we keep UUID/InetAddress mapping to StringType, then we can use query like this

Either way is not a perfect answer unless we can modify OSS SPARK to add UUIDType and InetAddressType to internal type.

Alex Liu May 6, 2015 at 8:46 PM
Edited

Create InetAddressType and UUIDtype, maps data accordingly.

Fixed

Details

Assignee

Reporter

Reviewer

Reviewer 2

Fix versions

Priority

Created April 20, 2015 at 3:47 PM
Updated June 29, 2015 at 8:26 AM
Resolved June 29, 2015 at 8:26 AM