auto restart mariadb.service if oom-kill killed main process
1
vote
1
answer
1092
views
I need some help trying to figure out why my mariadb service isn't automatically restarting upon crashing, likely due to insufficient memory on the server.
Essentially, my server operates at around 75%-90% memory utilisation all of the time. There's 8GB of memory on the server total. Today, at around 8am I observed
mariadb.service
going into the failed state. Moments before I noticed that my server's memory consumption was maxed out at around 98.5% for a few minutes.
Here's what I observed when running systemctl status mariadb.service
× mariadb.service - MariaDB 10.11 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
Active: failed (Result: oom-kill) since Thu 2024-03-28 04:00:36 EDT; 5min ago
Duration: 1d 1h 19min 29.156s
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 766 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
Process: 825 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 874 ExecStart=/usr/libexec/mariadbd --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=killed, signal=KILL)
Process: 1443 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
Main PID: 874 (code=killed, signal=KILL)
Status: "Taking your SQL requests now..."
CPU: 2h 39min 52.288s
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade: 1. Back-up your data before with 'mariadb-upgrade'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade: 2. Start the database daemon using 'systemctl start mariadb.service'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade: 3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade: Read more about 'mariadb-upgrade' usage at:
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade: https://mariadb.com/kb/en/mysql_upgrade/
Mar 27 02:41:07 domain-monitor-2022 systemd: Started MariaDB 10.11 database server.
Mar 28 04:00:36 domain-monitor-2022 systemd: mariadb.service: A process of this unit has been killed by the OOM killer.
Mar 28 04:00:36 domain-monitor-2022 systemd: mariadb.service: Main process exited, code=killed, status=9/KILL
Mar 28 04:00:36 domain-monitor-2022 systemd: mariadb.service: Failed with result 'oom-kill'.
Mar 28 04:00:36 domain-monitor-2022 systemd: mariadb.service: Consumed 2h 39min 52.288s CPU time.
I then ran systemctl restart mariadb.service
and it restarted without a problem, likely because the memory of the server came down a little.
However, it was down for a few minutes, why didn't it automatically try to restart? And how can I auto attempt to restart the database if a failure is observed.
I've since upgraded my server at a cost. From 8gb to 16gb of memory. And it's been around an hour of the database running and here's the status output:
● mariadb.service - MariaDB 10.11 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
Active: active (running) since Thu 2024-03-28 04:25:45 EDT; 29min ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 834 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
Process: 887 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 1182 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
Main PID: 946 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 94 (limit: 100205)
Memory: 463.1M
CPU: 3min 6.222s
CGroup: /system.slice/mariadb.service
└─946 /usr/libexec/mariadbd --basedir=/usr
Mar 28 04:25:44 domain-monitor-2022 systemd: Starting MariaDB 10.11 database server...
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: The datadir located at /var/lib/mysql needs to be upgraded using 'mariadb-upgrade' tool. This can be done using the fo>
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: 1. Back-up your data before with 'mariadb-upgrade'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: 2. Start the database daemon using 'systemctl start mariadb.service'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: 3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: Read more about 'mariadb-upgrade' usage at:
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade: https://mariadb.com/kb/en/mysql_upgrade/
Mar 28 04:25:45 domain-monitor-2022 systemd: Started MariaDB 10.11 database server.
So I'll conclude with what I need help with:
1. Why isn't mariadb.service
automatically restarted?
2. It initially failed because of: Failed with result 'oom-kill'. Why?
Output from running: SELECT 'SLEEPING MDB Ram use', COUNT(*),SUM(time),SUM(memory_used),SUM(max_memory_used) FROM information_schema.processlist WHERE command="Sleep";
- COUNT(*): 63:
- SUM(time): 341
- SUM(memory_used): 4947024
- SUM(max_memory_used): 9846392
Asked by Ryan H
(111 rep)
Mar 28, 2024, 08:58 AM
Last activity: Dec 20, 2024, 01:00 AM
Last activity: Dec 20, 2024, 01:00 AM