This could be a config or software issue, but we've spent several days on it without a solution.
PHP spawns multiple child processes with multiple connections to each node of the cassandra cluster. These connections do not time out, so eventually, the machines run short of RAM.
What I mean by that is the following: [root@ ~]# netstat -onalp | grep php | wc -l 6771 [root@ ~]# netstat -onalp | grep 9042 | wc -l 7095
There are 251 PHP processes running, but only 10 active processes at this time.
I ran lsof -p on one of the processes connecting to Cassandra (pid=6000) and the output is 6000.txt and checked to see how many connections this single process made to each node of the cassandra cluster:
So for 1 process, we have 30 connections to the Cassandra cluster. These connections are persistent, so they never die. As the load goes up, more and more of these processes get spawned until the machine runs out of RAM.
The only solution we have found is to set pm.max_requests in PHP so the processes die after a specific work-load, but I don't think that's ideal.
1) Why does 1 process open multiple connections to each node. If it's persistent, shouldn't there be only 1 connection? 2) Why does it connect to all nodes (even those in different data centers). 3) Why do new processes get launched when old ones are still around 4) How do we get the processes to die when they are unused or after a period of inactivity.
Answers/suggestions/help would be greatly appreciated. Nodes that did fine with 16 G of RAM each (for MySQL + Redis) now require 32G, and we still run into problems without setting pm.max_requests.
This could be a config or software issue, but we've spent several days on it without a solution.
PHP spawns multiple child processes with multiple connections to each node of the cassandra cluster. These connections do not time out, so eventually, the machines run short of RAM.
What I mean by that is the following:
[root@ ~]# netstat -onalp | grep php | wc -l
6771
[root@ ~]# netstat -onalp | grep 9042 | wc -l
7095
There are 251 PHP processes running, but only 10 active processes at this time.
I ran lsof -p on one of the processes connecting to Cassandra (pid=6000) and the output is 6000.txt and checked to see how many connections this single process made to each node of the cassandra cluster:
[root@ ~]# grep -c cass01 /tmp/6000.txt
6
[root@ ~]# grep -c cass02 /tmp/6000.txt
8
[root@ ~]# grep -c cass03 /tmp/6000.txt
10
[root@ ~]# grep -c cass04 /tmp/6000.txt
6
So for 1 process, we have 30 connections to the Cassandra cluster. These connections are persistent, so they never die. As the load goes up, more and more of these processes get spawned until the machine runs out of RAM.
The only solution we have found is to set pm.max_requests in PHP so the processes die after a specific work-load, but I don't think that's ideal.
1) Why does 1 process open multiple connections to each node. If it's persistent, shouldn't there be only 1 connection?
2) Why does it connect to all nodes (even those in different data centers).
3) Why do new processes get launched when old ones are still around
4) How do we get the processes to die when they are unused or after a period of inactivity.
Answers/suggestions/help would be greatly appreciated. Nodes that did fine with 16 G of RAM each (for MySQL + Redis) now require 32G, and we still run into problems without setting pm.max_requests.