Sample Header Ad - 728x90

MariaDB won't restart after oom-kill on Oracle Linux 9

0 votes
1 answer
104 views
I am in the process of migrating some of our servers to a new operating system after CentOS went out of support; and, as a replacement, we decided to move to Oracle Linux 9; specifically, MariaDB is version 10.6.18 and the OS is Red Hat Enterprise Linux release 9.4 (Plow). However, I am now facing an issue where, if the databases service gets killed by the OOM-killer, it won't restart automatically, while it did on CentOS. (By the way, I know that the OOM-killer *shouldn't* hit the database, that is another matter and not the focus of this question...). Here is the logs of the last time this happened on one of these servers:
2024-08-08  2:40:05 0 [Note] /opt/agews64/bin/mariadbd: ready for connections.
Version: '10.6.18-MariaDB'  socket: '/opt/agews64/data/mysql/mysql.sock'  port: 3306  MariaDB Server
240808 03:18:13 mysqld_safe Number of processes running now: 0
240808 03:18:13 mysqld_safe mysqld restarted
2024-08-08  3:18:13 0 [Note] Starting MariaDB 10.6.18-MariaDB source revision 887bb3f73555ff8a50138a580ca8308b9b5c069c as process 157377
2024-08-08  3:18:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-08-08  3:18:13 0 [Note] InnoDB: Using transactional memory
2024-08-08  3:18:13 0 [Note] InnoDB: Number of pools: 1
2024-08-08  3:18:13 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-08-08  3:18:13 0 [Note] InnoDB: Using Linux native AIO
2024-08-08  3:18:13 0 [Note] InnoDB: Initializing buffer pool, total size = 3221225472, chunk size = 134217728
2024-08-08  3:18:13 0 [Note] InnoDB: Completed initialization of buffer pool
2024-08-08  3:18:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=636943881408,636943881408
2024-08-08  3:18:13 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 34 row operations to undo
2024-08-08  3:18:13 0 [Note] InnoDB: Trx id counter is 44113220
2024-08-08  3:18:13 0 [Note] InnoDB: 128 rollback segments are active.
2024-08-08  3:18:14 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2024-08-08  3:18:14 0 [Note] InnoDB: Rolled back recovered transaction 44112559
2024-08-08  3:18:14 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2024-08-08  3:18:14 0 [Note] InnoDB: Removed temporary tablespace data file: "/opt/agews64/data/mysql/ibtmp1"
2024-08-08  3:18:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-08-08  3:18:14 0 [Note] InnoDB: Setting file '/opt/agews64/data/mysql/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-08-08  3:18:14 0 [Note] InnoDB: File '/opt/agews64/data/mysql/ibtmp1' size is now 12 MB.
2024-08-08  3:18:14 0 [Note] InnoDB: 10.6.18 started; log sequence number 636943881876; transaction id 44113222
2024-08-08  3:18:14 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-08-08  3:18:14 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-08-08  3:18:14 0 [Warning] 'proxies_priv' entry '@% root@vm-626suite-template' ignored in --skip-name-resolve mode.
2024-08-08  3:18:14 0 [Note] /opt/agews64/bin/mariadbd: ready for connections.
Version: '10.6.18-MariaDB'  socket: '/opt/agews64/data/mysql/mysql.sock'  port: 3306  MariaDB Server
2024-08-08  3:18:22 0 [Note] /opt/agews64/bin/mariadbd (initiated by: unknown): Normal shutdown
2024-08-08  3:18:22 0 [Note] InnoDB: FTS optimize thread exiting.
2024-08-08  3:18:22 0 [Note] InnoDB: Starting shutdown...
2024-08-08  3:18:23 0 [Note] InnoDB: Removed temporary tablespace data file: "/opt/agews64/data/mysql/ibtmp1"
2024-08-08  3:18:23 0 [Note] InnoDB: Shutdown completed; log sequence number 636943890130; transaction id 44113962
2024-08-08  3:18:23 0 [Note] /opt/agews64/bin/mariadbd: Shutdown complete
As you can see on line 3, where it says 240808 03:18:13 mysqld_safe Number of processes running now: 0, it seems that mysqld_safe correctly detected that fact that mariadbd had died, and dutifully restarted it. However, starting from 2024-08-08 3:18:22 0 [Note] /opt/agews64/bin/mariadbd (initiated by: unknown): Normal shutdown, that is, a few seconds after mysqld_safe restarted the database, immediately a new "clean" shutdown occurs. After that, the database does not automatically restart. For reference, here is the service definition for systemd:
[Unit]
Description=AGEws - Database
PartOf=agews.service
After=agews.service

[Service]
Type=forking
ExecStart=/opt/agews64/agews __start mysql
ExecStop=/opt/agews64/agews __stop mysql
ExecReload=/opt/agews64/agews __restart mysql
Restart=on-abort
KillMode=control-group
TimeoutSec=3600
OOMScoreAdjust=-750
MemoryAccounting=yes

[Install]
WantedBy=agews.service
The unit file uses Restart=on-abort as it seemed recommended in another question I found on the network. After starting, the unit is dinamically modified based on available RAM; for this specific server, it sets MemoryHigh=5.5G and MemoryMax=6G. **What configuration change should I make so that MariaDB service automatically restart after being killed by OOM-killer?**
Asked by Matteo Tassinari (187 rep)
Aug 8, 2024, 08:25 AM
Last activity: Aug 13, 2024, 09:28 AM