MariaDB won't restart after oom-kill on Oracle Linux 9
0
votes
1
answer
104
views
I am in the process of migrating some of our servers to a new operating system after CentOS went out of support; and, as a replacement, we decided to move to Oracle Linux 9; specifically, MariaDB is version
10.6.18
and the OS is Red Hat Enterprise Linux release 9.4 (Plow)
.
However, I am now facing an issue where, if the databases service gets killed by the OOM-killer, it won't restart automatically, while it did on CentOS.
(By the way, I know that the OOM-killer *shouldn't* hit the database, that is another matter and not the focus of this question...).
Here is the logs of the last time this happened on one of these servers:
2024-08-08 2:40:05 0 [Note] /opt/agews64/bin/mariadbd: ready for connections.
Version: '10.6.18-MariaDB' socket: '/opt/agews64/data/mysql/mysql.sock' port: 3306 MariaDB Server
240808 03:18:13 mysqld_safe Number of processes running now: 0
240808 03:18:13 mysqld_safe mysqld restarted
2024-08-08 3:18:13 0 [Note] Starting MariaDB 10.6.18-MariaDB source revision 887bb3f73555ff8a50138a580ca8308b9b5c069c as process 157377
2024-08-08 3:18:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-08-08 3:18:13 0 [Note] InnoDB: Using transactional memory
2024-08-08 3:18:13 0 [Note] InnoDB: Number of pools: 1
2024-08-08 3:18:13 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-08-08 3:18:13 0 [Note] InnoDB: Using Linux native AIO
2024-08-08 3:18:13 0 [Note] InnoDB: Initializing buffer pool, total size = 3221225472, chunk size = 134217728
2024-08-08 3:18:13 0 [Note] InnoDB: Completed initialization of buffer pool
2024-08-08 3:18:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=636943881408,636943881408
2024-08-08 3:18:13 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 34 row operations to undo
2024-08-08 3:18:13 0 [Note] InnoDB: Trx id counter is 44113220
2024-08-08 3:18:13 0 [Note] InnoDB: 128 rollback segments are active.
2024-08-08 3:18:14 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2024-08-08 3:18:14 0 [Note] InnoDB: Rolled back recovered transaction 44112559
2024-08-08 3:18:14 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2024-08-08 3:18:14 0 [Note] InnoDB: Removed temporary tablespace data file: "/opt/agews64/data/mysql/ibtmp1"
2024-08-08 3:18:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-08-08 3:18:14 0 [Note] InnoDB: Setting file '/opt/agews64/data/mysql/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-08-08 3:18:14 0 [Note] InnoDB: File '/opt/agews64/data/mysql/ibtmp1' size is now 12 MB.
2024-08-08 3:18:14 0 [Note] InnoDB: 10.6.18 started; log sequence number 636943881876; transaction id 44113222
2024-08-08 3:18:14 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-08-08 3:18:14 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-08-08 3:18:14 0 [Warning] 'proxies_priv' entry '@% root@vm-626suite-template' ignored in --skip-name-resolve mode.
2024-08-08 3:18:14 0 [Note] /opt/agews64/bin/mariadbd: ready for connections.
Version: '10.6.18-MariaDB' socket: '/opt/agews64/data/mysql/mysql.sock' port: 3306 MariaDB Server
2024-08-08 3:18:22 0 [Note] /opt/agews64/bin/mariadbd (initiated by: unknown): Normal shutdown
2024-08-08 3:18:22 0 [Note] InnoDB: FTS optimize thread exiting.
2024-08-08 3:18:22 0 [Note] InnoDB: Starting shutdown...
2024-08-08 3:18:23 0 [Note] InnoDB: Removed temporary tablespace data file: "/opt/agews64/data/mysql/ibtmp1"
2024-08-08 3:18:23 0 [Note] InnoDB: Shutdown completed; log sequence number 636943890130; transaction id 44113962
2024-08-08 3:18:23 0 [Note] /opt/agews64/bin/mariadbd: Shutdown complete
As you can see on line 3, where it says 240808 03:18:13 mysqld_safe Number of processes running now: 0
, it seems that mysqld_safe
correctly detected that fact that mariadbd
had died, and dutifully restarted it.
However, starting from 2024-08-08 3:18:22 0 [Note] /opt/agews64/bin/mariadbd (initiated by: unknown): Normal shutdown
, that is, a few seconds after mysqld_safe
restarted the database, immediately a new "clean" shutdown occurs.
After that, the database does not automatically restart.
For reference, here is the service definition for systemd:
[Unit]
Description=AGEws - Database
PartOf=agews.service
After=agews.service
[Service]
Type=forking
ExecStart=/opt/agews64/agews __start mysql
ExecStop=/opt/agews64/agews __stop mysql
ExecReload=/opt/agews64/agews __restart mysql
Restart=on-abort
KillMode=control-group
TimeoutSec=3600
OOMScoreAdjust=-750
MemoryAccounting=yes
[Install]
WantedBy=agews.service
The unit file uses Restart=on-abort
as it seemed recommended in another question I found on the network.
After starting, the unit is dinamically modified based on available RAM; for this specific server, it sets MemoryHigh=5.5G
and MemoryMax=6G
.
**What configuration change should I make so that MariaDB service automatically restart after being killed by OOM-killer?**
Asked by Matteo Tassinari
(187 rep)
Aug 8, 2024, 08:25 AM
Last activity: Aug 13, 2024, 09:28 AM
Last activity: Aug 13, 2024, 09:28 AM