Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Haxx tests fail on the copy_table task

Description

The spark connector tests for cassandra fail. Sometimes it's during the copy_table phase:

2022-07-01 15:08:58,425 [jepsen :copy_table] INFO controller - :copy_table :start nil nil 2022-07-01 15:08:58,435 [jepsen :copy_table] INFO controller - Emit invoke - null 2022-07-01 15:08:58,436 [jepsen :copy_table] INFO server-0 - Executing on node server-0: cd /home/automaton/fallout_library/spark-cassandra-stress export SPARK_HOME=/home/automaton/fallout_library/spark ./run.sh apache copytable --trials 1 --totalOps 5000000000 --numTotalKeys 20000000 --distributedDataType dataframe -f /home/automaton/fallout_scratch/artifacts/spark_stress_results_copy_table.txt --master spark://10.101.34.60:7077 --conf spark.executor.memory=12g --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions --conf spark.cassandra.connection.host=10.101.34.60 > /home/automaton/fallout_scratch/artifacts/spark_stress_pipe_out_copy_table.txt 2>&1 2022-07-01 15:08:58,447 [jepsen :copy_table] INFO server-0 - Waiting for spark-stress at most 10 hours 2022-07-01 15:08:58,447 [jepsen :copy_table] INFO server-0 - Waiting for command (hard-timeout: 10 hr): cd /home/automaton/fallout_library/spark-cassandra-stress export SPARK_HOME=/home/automaton/fallout_library/spark ./run.sh apache copytable --trials 1 --totalOps 5000000000 --numTotalKeys 20000000 --distributedDataType dataframe -f /home/automaton/fallout_scratch/artifacts/spark_stress_results_copy_table.txt --master spark://10.101.34.60:7077 --conf spark.executor.memory=12g --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions --conf spark.cassandra.connection.host=10.101.34.60 > /home/automaton/fallout_scratch/artifacts/spark_stress_pipe_out_copy_table.txt 2>&1 2022-07-01 18:46:11,438 [FJP:979] ERROR server-0 - [CMD-FAIL] Command on node 'server-0' completed with non-zero exit code 1: cd /home/automaton/fallout_library/spark-cassandra-stress export SPARK_HOME=/home/automaton/fallout_library/spark ./run.sh apache copytable --trials 1 --totalOps 5000000000 --numTotalKeys 20000000 --distributedDataType dataframe -f /home/automaton/fallout_scratch/artifacts/spark_stress_results_copy_table.txt --master spark://10.101.34.60:7077 --conf spark.executor.memory=12g --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions --conf spark.cassandra.connection.host=10.101.34.60 > /home/automaton/fallout_scratch/artifacts/spark_stress_pipe_out_copy_table.txt 2>&1 2022-07-01 18:46:11,439 [jepsen :copy_table] INFO controller - Emit fail - spark_stress error 2022-07-01 18:46:11,455 [jepsen :copy_table] INFO controller - Emit end - null 2022-07-01 18:46:11,455 [jepsen :copy_table] INFO controller - :copy_table :info nil "completed" 2022-07-01 18:46:11,455 [jepsen :copy_table] INFO controller - :copy_table done

 

Sometimes, additionally it also fails to shutdown Cassandra during 60 seconds

WARN [StorageServiceShutdownHook] 2022-06-15 14:56:02,042 StorageService.java:4998 - Unable to terminate non-periodic tasks within 1 minute.

Environment

None

Pull Requests

None

Activity

Show:

Details

Assignee

Reporter

Components

Priority

Created July 19, 2022 at 7:04 AM
Updated July 19, 2022 at 7:04 AM

Flag notifications