ChannelPool leak when using FixedHostNameAddressTranslator

Description

When Cassandra is running on Kubernetes (w/o hostNetwork enabled) and the client application not, the driver needs to be configured with FixedHostNameAddressTranslator (or similar) to translate Cassandra Pods' Kubernetes private network IP addresses to the external IP/URL (eg. K8s LoadBalancer service IP) accessible to the client. Normally, in this case, we also configure contact points to that same IP/URL.

Internally, on initialization, driver creates a ChannelPool for every not IGNORED or FORCED_DOWN Node. However, when we setup the driver as above, it generates leaking network connections for all except 1 Node, which becomes a visible problem for the Cassandra clusters with large number of nodes having large number of client application instances.

The trace is the following:

  • On initial connection to DB, driver fetches its nodes info

  • Then nodes' endpoints are translated by FixedHostnameAddressTranslator to the same endpoint (K8s LoadBalancer URL)

  • Then if Node’s (translated) endpoint is already in contact points (and it always is), we put the contact point Node object instance to the nodes map, so the nodes map contains unique hostId keys mapped to the same contact point Node reference

  • After, we init ChannelPool for all Nodes (which are same), and put to the map of pools every initialized pool, using same Node object reference as a key

  • So in the end we have a pools map with a single entry, while leaking all other unmanaged pools running in the background which are never closed

We should avoid creating unmanaged duplicate pools in such scenarios.

Environment

Java webapp with Datastax Driver configured with FixedHostNameAddressTranslator calling Cassandra in K8s behind load balancer.

Pull Requests

None

Activity

Show:
Fixed

Details

Assignee

Reporter

Reproduced in

Fix versions

Size

S

Components

Priority

Created September 20, 2024 at 2:02 PM
Updated January 31, 2025 at 10:19 PM
Resolved January 31, 2025 at 10:19 PM

Flag notifications