Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy

Description

I'm running an application using Cassandra 4.0, Spark 3.1.1, JDK 8, Tomcat 9.

It worked normally for 5 moths, but the application stopped when I configurate my pom.xml to new java-driver-core version : cassandra java-driver-core version 4.11.1 to fixed another problem about "Unsupported request opcode: 0 in protocol 6". The application stopped immediately.

This is complete stacktrace error :


java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1:9042} :: Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy (specified by basic.load-balancing-policy.class): com.datastax.oss.driver.internal.core.context.InternalDriverContext.getNodeFilter(Ljava/lang/String;)Ljava/util/function/Predicate;
at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:173) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.CassandraConnector$.$anonfun$sessionCache$1(CassandraConnector.scala:161) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:103) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.datasource.CassandraCatalog$.com$datastax$spark$connector$datasource$CassandraCatalog$$getMetadata(CassandraCatalog.scala:455) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at com.datastax.spark.connector.datasource.CassandraCatalog$.getTableMetaData(CassandraCatalog.scala:421) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at org.apache.spark.sql.cassandra.DefaultSource.getTable(DefaultSource.scala:68) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at org.apache.spark.sql.cassandra.DefaultSource.inferSchema(DefaultSource.scala:72) ~[spark-cassandra-connector_2.12-3.0.1.jar:3.0.1]
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:81) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:296) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at scala.Option.map(Option.scala:230) ~[scala-library-2.12.11.jar:na]
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:266) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at br.com.genesis.configuration.DataSourceCassandraConfig.setParametros(DataSourceCassandraConfig.java:40) ~[classes/:2.4.2]
at br.com.genesis.service.FalhaService.gerarGraficoModoFalhaTipo1(FalhaService.java:397) ~[classes/:2.4.2]
at br.com.genesis.controller.estatistico.EstModoFalhaController.gerarGraficos(EstModoFalhaController.java:134) ~[classes/:2.4.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.3.jar:5.3.3]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.3.jar:5.3.3]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.3.jar:5.3.3]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[servlet-api.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.3.jar:5.3.3]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[servlet-api.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-websocket.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.3.jar:5.3.3]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.3.jar:5.3.3]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126) [spring-boot-2.4.2.jar:2.4.2]
at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64) [spring-boot-2.4.2.jar:2.4.2]
at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101) [spring-boot-2.4.2.jar:2.4.2]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119) [spring-boot-2.4.2.jar:2.4.2]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.3.jar:5.3.3]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.3.jar:5.3.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [catalina.jar:9.0.46]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [catalina.jar:9.0.46]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [catalina.jar:9.0.46]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [catalina.jar:9.0.46]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [catalina.jar:9.0.46]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [catalina.jar:9.0.46]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [catalina.jar:9.0.46]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) [catalina.jar:9.0.46]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [catalina.jar:9.0.46]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [catalina.jar:9.0.46]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-coyote.jar:9.0.46]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:9.0.46]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-coyote.jar:9.0.46]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) [tomcat-coyote.jar:9.0.46]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:9.0.46]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:9.0.46]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
Caused by: java.lang.IllegalArgumentException: Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy (specified by basic.load-balancing-policy.class): com.datastax.oss.driver.internal.core.context.InternalDriverContext.getNodeFilter(Ljava/lang/String;)Ljava/util/function/Predicate;
at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:253) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfigProfiles(Reflection.java:162) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.buildLoadBalancingPolicies(DefaultDriverContext.java:338) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.util.concurrent.LazyReference.get(LazyReference.java:55) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.getLoadBalancingPolicies(DefaultDriverContext.java:687) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.init(DefaultSession.java:338) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.access$1100(DefaultSession.java:300) ~[java-driver-core-4.11.1.jar:na]
at com.datastax.oss.driver.internal.core.session.DefaultSession.lambda$init$0(DefaultSession.java:146) ~[java-driver-core-4.11.1.jar:na]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
... 1 common frames omitted
Caused by: java.lang.NoSuchMethodError: com.datastax.oss.driver.internal.core.context.InternalDriverContext.getNodeFilter(Ljava/lang/String;)Ljava/util/function/Predicate;
at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.<init>(LocalNodeFirstLoadBalancingPolicy.scala:40) ~[spark-cassandra-connector-driver_2.12-3.0.1.jar:3.0.1]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_292]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_292]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_292]
at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:247) ~[java-driver-core-4.11.1.jar:na]
... 14 common frames omitted

Pull Requests

None

Attachments

Linked work items

resolves

JAVA-2945

Reinstate InternalDriverContext.getNodeFilter method

Activity

Show:

Alex Dutra
May 26, 2021 at 8:51 AM

FYI https://datastax-oss.atlassian.net/browse/JAVA-2945 will put the missing method back. It’s going to be released in 4.11.2 very soon. Hopefully this will give Spark users a clear path towards using Spark with C* 4.0.

Luciana Oliveria
May 25, 2021 at 12:38 PM
(edited)

Thanks for your replay! I chanced the Java Driver to 4.10.0 and spark 3.0.0 (and the others compatibles dependencies), but annother error appear. The application did not start the Cassandra DB.

This error was corrected in java drive version 4.11.0 or 4.11.1 . I’m in trouble! If I fix one, the other appers. What could I do in this case?

[s0] Error connecting to Node(endPoint=127.0.0.1:9042, hostId=null, hashCode=fd2fa8e), trying next node (ConnectionInitException: [s0|control|id: 0x02f078a6, L:/127.0.0.1:40468 - R:/127.0.0.1:9042] Protocol initialization request, step 1 (OPTIONS): unexpected failure (java.lang.IllegalArgumentException: Unsupported request opcode: 0 in protocol 6))

Jarek Grabowski
May 25, 2021 at 8:58 AM

Hi!

Spark Cassandra Connector doesn’t support Spark 3.1 yet. You can follow this ticket to track the progress: SPARKC-646.

Also, Spark Cassandra Connector works with Java Driver 4.10

To get rid of these errors you need to downgrade the driver back to 4.10.0 and spark to 3.0.x.

Luciana Oliveria
May 24, 2021 at 5:03 PM

https://datastax-oss.atlassian.net/plugins/servlet/mobile?originPath=%2Fbrowse%2FJAVA-2945#issue/JAVA-2945

Luciana Oliveria
May 24, 2021 at 4:42 PM
(edited)

I opened a issue in the Java drive 4.11.1 and that is the reply:

Hi @Luciana Oliveria the method InternalDriverContext.getNodeFilter was removed in 4.11.0, see JAVA-2914. It was replaced by the method InternalDriverContext.getNodeDistanceEvaluator. It is also explained in our upgrade guide.

I guess the Spark library is calling the old InternalDriverContext.getNodeFilter method. You should report this to the Spark library’s maintainers. Thanks!

Could you help me please? The application is stopped to 3 weeks, and now I found this problem. I downgrade all the components and the error persist.

Best,

Resize issue view side panel

Details

Assignee

Jarek Grabowski

Reporter

Luciana Oliveria

Labels

Components

Core

DSE

SparkSQL

Priority

Major

Created May 21, 2021 at 1:21 PM

Updated May 26, 2021 at 8:51 AM

Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy

Description

Pull Requests

Attachments

Linked work items

resolves

Activity

Alex Dutra
May 26, 2021 at 8:51 AM

Luciana Oliveria
May 25, 2021 at 12:38 PM
(edited)

Jarek Grabowski
May 25, 2021 at 8:58 AM

Luciana Oliveria
May 24, 2021 at 5:03 PM

Luciana Oliveria
May 24, 2021 at 4:42 PM
(edited)

Details

Assignee

Reporter

Labels

Components

Priority

Flag notifications

Something's gone wrong

Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy

Description

Pull Requests

Attachments

Linked work items

resolves

Activity

Alex Dutra May 26, 2021 at 8:51 AM

Luciana Oliveria May 25, 2021 at 12:38 PM(edited)

Jarek Grabowski May 25, 2021 at 8:58 AM

Luciana Oliveria May 24, 2021 at 5:03 PM

Luciana Oliveria May 24, 2021 at 4:42 PM(edited)

Details

Assignee

Reporter

Labels

Components

Priority

Flag notifications

Something's gone wrong

Alex Dutra
May 26, 2021 at 8:51 AM

Luciana Oliveria
May 25, 2021 at 12:38 PM
(edited)

Jarek Grabowski
May 25, 2021 at 8:58 AM

Luciana Oliveria
May 24, 2021 at 5:03 PM

Luciana Oliveria
May 24, 2021 at 4:42 PM
(edited)