Weird behaviour of OpenSSH server eating slowly all the available memory
0
votes
1
answer
410
views
I have a machine running a pgbouncer server and an Open SSH server so clients can enstabilish a tunnel to the server and connect to the database.
Now all the clients keep the connection alive for around 3 minutes, then the connection is closed. As you can see the following image shows you that running the command:
ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
multiple times underline the fact that the number or sshd processes decreases over a shor period of time.
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
158 158 790
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
150 150 750
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
146 146 730
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
140 140 700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
140 140 700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
140 140 700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
140 140 700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
136 136 680
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
132 132 660
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
132 132 660
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
132 132 660
sosepe@pgbouncer:~$ netstat -nt | grep :22
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
298 298 1490
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
324 324 1620
**But not over a long period of time.**
What I mean is that that after some hours running that command again shows how actually the number of processes raises. That's because some of them never get actually closed.
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
324 324 1620
And after some hours this is the result:
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
1926 1926 9630
And these are the consequences:
Private + Shared = RAM used Program
4.0 KiB + 22.0 KiB = 26.0 KiB agetty
132.0 KiB + 27.5 KiB = 159.5 KiB vnstatd
128.0 KiB + 45.5 KiB = 173.5 KiB systemd-udevd
136.0 KiB + 65.0 KiB = 201.0 KiB cron
216.0 KiB + 33.0 KiB = 249.0 KiB tail
112.0 KiB + 239.5 KiB = 351.5 KiB systemd-timesyncd
504.0 KiB + 50.0 KiB = 554.0 KiB lvmetad
192.0 KiB + 481.5 KiB = 673.5 KiB vsftpd
908.0 KiB + 92.0 KiB = 1.0 MiB sudo
916.0 KiB + 103.0 KiB = 1.0 MiB pgbouncer
644.0 KiB + 604.0 KiB = 1.2 MiB vmtoolsd
1.3 MiB + 47.0 KiB = 1.3 MiB rsyslogd
1.2 MiB + 247.0 KiB = 1.5 MiB CloudEndure_Age (4)
1.4 MiB + 81.0 KiB = 1.5 MiB dbus-daemon
1.6 MiB + 331.0 KiB = 1.9 MiB su (4)
1.2 MiB + 892.0 KiB = 2.1 MiB VGAuthService
1.7 MiB + 582.0 KiB = 2.2 MiB collectd
2.7 MiB + 268.0 KiB = 3.0 MiB systemd-logind
2.9 MiB + 63.5 KiB = 3.0 MiB bash
4.3 MiB + 231.5 KiB = 4.5 MiB systemd-journald
2.7 MiB + 3.6 MiB = 6.2 MiB php-fpm7.3 (3)
4.8 MiB + 1.8 MiB = 6.6 MiB (sd-pam) (3)
6.5 MiB + 3.3 MiB = 9.8 MiB systemd (4)
5.8 MiB + 4.1 MiB = 9.9 MiB php-fpm7.2 (3)
5.7 MiB + 6.5 MiB = 12.2 MiB php-fpm5.6 (3)
16.1 MiB + 153.0 KiB = 16.3 MiB run_linux_migration_scripts_periodically (2)
17.2 MiB + 166.5 KiB = 17.3 MiB update_onprem_volumes (2)
18.5 MiB + 145.5 KiB = 18.6 MiB tailer (2)
11.8 MiB + 13.1 MiB = 24.9 MiB apache2 (11)
159.3 MiB + 180.0 KiB = 159.5 MiB java
741.6 MiB + 2.1 GiB = 2.8 GiB sshd (4469)
---------------------------------
3.1 GiB
=================================
SSHD ate almost 3 GB of memory, and this goes on until the machine is rebooted.
Any clue about where the problem could be?
Thanks!
P.S.
This is the conf file:
Port 10110
Protocol 1,2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
UsePrivilegeSeparation yes
SyslogFacility AUTHPRIV
LogLevel INFO
PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
PermitEmptyPasswords no
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
GatewayPorts yes
ClientAliveInterval 600
PermitTunnel yes
MaxSessions 50
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
Asked by Federico Loro
(1 rep)
Nov 24, 2021, 03:11 PM
Last activity: Oct 25, 2022, 05:24 AM
Last activity: Oct 25, 2022, 05:24 AM