Support for TTL per insertion while using DataFrame.save method

Description

We want to add support of using "ttl" filed . For example currently spark supports TTL field using

rdd.saveToCassandra("test", "tab", writeConf = WriteConf(ttl = TTLOption.perRow("ttl")))

where rdd is something like ( last field is TTL in seconds)
val rdd = sc.makeRDD(Seq(
KeyValueWithTTL(1, 1L, "value1", 100),
KeyValueWithTTL(2, 2L, "value2", 200),
KeyValueWithTTL(3, 3L, "value3", 300)))

We want do similar with dataframe apis like
data.write
.format("org.apache.spark.sql.cassandra")
.options(Map( "table" -> tableName, "keyspace" -> dbName,
"spark_cassandra_connection_host" -> createStringList(hosts),
"spark_cassandra_output_metrics" -> "false",
"spark_cassandra_output_concurrent_writes" -> "20",
))
.mode(SaveMode.Append)
.save()
We want to use "ttl" field in our dataframe and want cassandra to apply it per row.

Pull Requests

None

Activity

Brian Cantoni 
May 16, 2016 at 7:18 PM

Setting fixversion to 1.6.0 because the PR is against master.

Alex Liu 
March 21, 2016 at 7:34 PM

Cassandra TTL can be applied to per column, but it's much easier to implement per insertion

Fixed

Details

Assignee

Reporter

Reviewer

Reviewer 2

Pull Request

Components

Fix versions

Priority

Created March 15, 2016 at 10:31 PM
Updated August 21, 2018 at 3:10 PM
Resolved June 8, 2016 at 9:01 PM