Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects versions: 2.0.6
    • Fix versions: None
    • Components: Core
    • Labels:
      None
    • Environment:

      Spark Cassandra version 2.0.6
      Cassandra version 3.0.15

      Description

      Hi,

      CREATE TYPE orderbooks.address (
          name text
      );
      
      CREATE TABLE orderbooks.person (
          name text PRIMARY KEY,
          addresses list<frozen<address>>
      );
      
      object MyTest3 {
      
        case class Address(name: String)
        case class Person(name:String, addresses: List[Address])
      
        def main(args: Array[String]) = {
          val spark = SparkSession.builder
            .appName("My Test")
            .master("local[*]")
            .config("spark.driver.bindAddress", "127.0.0.1")
            .getOrCreate()
      
          val p = Person("Peter", List(Address("Wall Street")))
          spark.sparkContext.parallelize(List.fill(2)(p), 1).saveToCassandra("orderbooks", "person")
          spark.close()
        }
      
      }
      
      cqlsh:orderbooks> select * from person ;
      
      @ Row 1
      -----------+------------------------------------------------
       name      | Peter
       addresses | [{name: 'Wall Street'}, {name: 'Wall Street'}]
      
      (1 rows)
      

      The address is duplicated.

      Please mind, that I set spark parallelism to 1. Is this expected behavior? My expectation was that the row will be completely overwritten and not duplicated.

      If I set parallelism to 2, it's not reproduced. There is a single address as expected.

        Attachments

          Activity

            People

            • Assignee:
              pkolaczk Piotr Kołaczkowski
              Reporter:
              aka.fe2s Oleksiy Dyagilev
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: