java.lang.OutOfMemoryError: Direct buffer memory
Description
It is a VPS instance with 1024MB memory, it runs a Jenkins. The Jenkins starts integration tests, the issue appears exactly after the 16th connect-disconnect in same JVM (possible connection leak?), the 16th connect to the cluster always throw the exception above.
I've tested the all version of the Datastax driver (successful test cycles / test cycles):
2.1.3: ok (4/4)
2.1.4: ok (4/4)
2.1.5: ok (4/4)
2.1.6: failed (0/4)
2.1.7: failed (0/4)
2.1.8: failed (0/4)
2.2.0-rc1: failed (0/4)
2.2.0-rc2: failed (0/4)
2.2.0-rc3: failed (0/4)
3.0.0-alpha5: failed (0/4)
The stack trace in the getErrors:
I've tried to set log level, and found a possible root cause of the issue, it is thrown after every disconnect from Cassandra cluster:
P.S.: the "Netty upgrade to 4.x" (https://datastax-oss.atlassian.net/browse/JAVA-622) issue resolved in the 2.1.6 version and the OutOfMemoryError: Direct buffer memory" appeared in the 2.1.6 version...
Environment
CentOS Linux release 7.1.1503, OpenJDK Runtime Environment (build 1.8.0_65-b17), DSC Cassandra 3.0.0 (tar.gz)
Pull Requests
Activity
Looks like a classloader issue. Seeing this:
Do your tests deploy code in a container? Is it properly undeployed after each test?
Netty 4 introduces buffer pooling. By default, the driver uses Netty's default allocator, which is static (source). If Netty is loaded by multiple classloaders that will create this default instance multiple times, and it pre-allocates memory at startup. So the key to the problem is to find out what is leaking these classloaders.
Yes, I will deep dig into the class loader of the Wildfly and the Cassandra... it looks like an interference between them...
Any fixes or news? Reproduced with driver v 3.0.0 on WildFly 10
This ticket has been closed on 2020-06-22 due to prolonged inactivity, as part of an automatic housekeeping procedure.
If you think that this was inappropriate, feel free to re-open the ticket. If possible, please provide any context that could explain why the issue described in this ticket is still relevant.
Also, please note that enhancements and feature requests cannot be accepted anymore for legacy driver versions (OSS 1.x, 2.x, 3.x and DSE 1.x).
Thank you for your understanding.
