spark-sql session doesn't exit cleanly
Description
Environment
Spark 3.0.1
SCC 3.0.0-beta
MacOS 10.12
Cassandra 4.0-beta2
Pull Requests
None
Activity
Show:
Jaroslaw Grabowski September 17, 2020 at 8:47 PM
Apparently hive closes the class loader responsible for loading these classes.
at org.apache.spark.util.MutableURLClassLoader.close(MutableURLClassLoader.java:40)
at org.apache.hadoop.hive.common.JavaUtils.closeClassLoader(JavaUtils.java:91)
at org.apache.hadoop.hive.common.JavaUtils.closeClassLoadersTo(JavaUtils.java:68)
at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1558)
at org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:293)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Jaroslaw Grabowski September 17, 2020 at 10:25 AM
Workaround:
set spark.cassandra.connection.keepAliveMS to 0
Jaroslaw Grabowski September 15, 2020 at 11:21 AM
This happens also for Spark 3.0.0. Only spark-sql is affected, spark-shell works fine for both versions.
I was playing with SCC & Spark 3.0.1, and noticed that I can't finish my work clearly - the Spark session just hangs with following errors, and even Ctrl-C doesn't help:
20/09/14 12:46:45 WARN DefaultPromise: An exception was thrown by com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded$$Lambda$1768/2053026937.operationComplete() java.lang.NoClassDefFoundError: com/datastax/oss/driver/internal/core/util/Loggers at com.datastax.oss.driver.internal.core.util.concurrent.UncaughtExceptions.log(UncaughtExceptions.java:49) at com.datastax.oss.driver.shaded.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) .... 20/09/14 12:46:45 WARN NioEventLoop: Unexpected exception in the selector loop. java.lang.NoClassDefFoundError: com/datastax/oss/driver/shaded/netty/channel/socket/ChannelInputShutdownReadComplete at com.datastax.oss.driver.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:109) at com.datastax.oss.driver.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:127)spark-sql was started with following command
bin/spark-sql --packages com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.0.0-beta \ --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions \ --conf spark.sql.catalog.casscatalog=com.datastax.spark.connector.datasource.CassandraCatalog