I am experiencing an issue with MySQL memory usage increasing over time to a point where OOM Killer kills the MySQL process.
I have 3 servers configured with Group Replication (1 Primary and 2 secondary).
The servers are running on AWS EC2:
m5.2xlarge (32GB RAM, 8vCPU,500GB Amazon EBS Storage)
Debian 11
MySQL 8.0.36
Link to additional requested information: https://justpaste.it/f0f4n
Updated additional information after 44 hours of server running: https://justpaste.it/6v4pl
The memory usage issue seems to occur faster on a server that is the PRIMARY server in the cluster but does affect all servers.
I have been trying, without success, to figure out what the cause of this could be.
The system memory usage for the MySQL process and the listed memory allocation in MySQL do not correspond.
*The additional memory usage does seem to be related to HIGH_NUMBER_OF_BYTES_USED of "memory/innodb/ut0rbt". It seems that this memory accounts for the reported memory discrepancy. In all instances where the memory is growing, this reported information is growing as well.*
Below is the output of
top - As you can see, the current physical RAM in use is by MySQL is 27.0G
Tasks: 118 total, 1 running, 117 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.4 us, 0.5 sy, 0.0 ni, 85.3 id, 0.5 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 31650.4 total, 263.6 free, 27895.9 used, 3490.9 buff/cache
MiB Swap: 4096.0 total, 0.1 free, 4095.9 used. 3302.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
757962 mysql 20 0 33.3g 27.0g 0 S 55.4 87.3 1104:36 mysqld
Below is the memory usage reported by MySQL:
mysql> select event_name,current_alloc from sys.memory_global_by_current_bytes limit 10;
+---------------------------------------------------------------+---------------+
| event_name | current_alloc |
+---------------------------------------------------------------+---------------+
| memory/innodb/buf_buf_pool | 8.18 GiB |
| memory/group_rpl/GCS_XCom::xcom_cache | 1023.71 MiB |
| memory/innodb/log_buffer_memory | 256.00 MiB |
| memory/temptable/physical_ram | 94.00 MiB |
| memory/performance_schema/events_statements_summary_by_digest | 40.28 MiB |
| memory/mysys/KEY_CACHE | 32.00 MiB |
| memory/innodb/ut0link_buf | 24.00 MiB |
| memory/sql/TABLE | 23.30 MiB |
| memory/innodb/lock0lock | 22.22 MiB |
| memory/innodb/memory | 21.93 MiB |
+---------------------------------------------------------------+---------------+
10 rows in set (0.01 sec)+----------------------------------+
| format_bytes(sum(current_alloc)) |
+----------------------------------+
| 9.95 GiB |
+----------------------------------+
1 row in set (0.00 sec)+------------------------------------------+--------------------+
| key_buffer_size | 32.000 MB |
| query_cache_size | 0.000 MB |
| innodb_buffer_pool_size | 8192.000 MB |
| innodb_additional_mem_pool_size | 0.000 MB |
| innodb_log_buffer_size | 256.000 MB |
+------------------------------------------+--------------------+
| BASE MEMORY | 8480.000 MB |
+------------------------------------------+--------------------+
| sort_buffer_size | 0.250 MB |
| read_buffer_size | 0.125 MB |
| read_rnd_buffer_size | 0.250 MB |
| join_buffer_size | 0.250 MB |
| thread_stack | 1.000 MB |
| binlog_cache_size | 0.031 MB |
| tmp_table_size | 16.000 MB |
+------------------------------------------+--------------------+
| MEMORY PER CONNECTION | 17.906 MB |
+------------------------------------------+--------------------+
| Max_used_connections | 129 |
| max_connections | 300 |
+------------------------------------------+--------------------+
| TOTAL (MIN) | 10789.906 MB |
| TOTAL (MAX) | 13851.875 MB |
+------------------------------------------+--------------------+memory/innodb/ut0rbt based on looking at the HIGH_NUMBER_OF_BYTES_USED being 16GB in the below report.
I cannot find much information on memory/innodb/ut0rbt when searching and am not sure where to go to from here. Any guidance would be appreciated.
SELECT * FROM performance_schema.memory_summary_global_by_event_name where SUM_NUMBER_OF_BYTES_ALLOC !=0 ORDER BY HIGH_NUMBER_OF_BYTES_USED DESC;
+--------------------------------------------------------------------------------+-------------+-------------+---------------------------+--------------------------+----------------+--------------------+-----------------+--------------------------+------------------------------+---------------------------+
| EVENT_NAME | COUNT_ALLOC | COUNT_FREE | SUM_NUMBER_OF_BYTES_ALLOC | SUM_NUMBER_OF_BYTES_FREE | LOW_COUNT_USED | CURRENT_COUNT_USED | HIGH_COUNT_USED | LOW_NUMBER_OF_BYTES_USED | CURRENT_NUMBER_OF_BYTES_USED | HIGH_NUMBER_OF_BYTES_USED |
+--------------------------------------------------------------------------------+-------------+-------------+---------------------------+--------------------------+----------------+--------------------+-----------------+--------------------------+------------------------------+---------------------------+
| memory/innodb/ut0rbt | 12367972599 | 12367971756 | 1175789585128 | 1175789498761 | 0 | 843 | 168276872 | 0 | 86367 | 16869347952 |
| memory/innodb/buf_buf_pool | 64 | 0 | 8783134720 | 0 | 0 | 64 | 64 | 0 | 8783134720 | 8783134720 |
| memory/temptable/physical_ram | 18599 | 18501 | 26578851040 | 26476087456 | 0 | 98 | 506 | 0 | 102763584 | 1634745920 |
| memory/mysys/TREE | 13933 | 13930 | 4655917920 | 4655878864 | 0 | 3 | 1568 | 0 | 39056 | 1119816488 |
| memory/group_rpl/GCS_XCom::xcom_cache | 767889 | 300297 | 1872249111 | 798535661 | 0 | 467592 | 495763 | 0 | 1073713450 | 1075974241 |
| memory/innodb/memory | 532219874 | 532207809 | 13596646725344 | 13596623477632 | 0 | 12065 | 87170 | 0 | 23247712 | 889900184 |
| memory/sql/String::value | 179746709 | 179745073 | 186307408232 | 186301810472 | 0 | 1636 | 8326 | 0 | 5597760 | 444524496 |
| memory/innodb/log_buffer_memory | 1 | 0 | 268436464 | 0 | 0 | 1 | 1 | 0 | 268436464 | 268436464 |
| memory/sql/Unique::merge_buffer | 36 | 36 | 301991328 | 301991328 | 0 | 0 | 26 | 0 | 0 | 218104848 |
| memory/sql/THD::main_mem_root | 200742977 | 200742846 | 3390843376248 | 3390840251216 | 0 | 131 | 3117 | 0 | 3125032 | 88720256 |
| memory/sql/Log_event | 43535365 | 43535358 | 20931402783 | 20931400073 | 0 | 7 | 269641 | 0 | 2710 | 49982659 |
| memory/sql/JSON | 9507512 | 9507456 | 859590368 | 859587008 | 0 | 56 | 549870 | 0 | 3360 | 47863656 |
| memory/group_rpl/certification_info | 1743130 | 1742155 | 164631088 | 158843768 | 0 | 975 | 426208 | 0 | 5787320 | 43207824 |
| memory/group_rpl/transaction_data | 2931445 | 2931445 | 3611045127 | 3611045127 | 0 | 0 | 15 | 0 | 0 | 42283650 |
| memory/performance_schema/events_statements_summary_by_digest | 1 | 0 | 42240000 | 0 | 0 | 1 | 1 | 0 | 42240000 | 42240000 |
| memory/group_rpl/Gcs_message_data::m_buffer | 711966 | 711966 | 3274265418 | 3274265418 | 0 | 0 | 4 | 0 | 0 | 38611767 |
| memory/mysys/KEY_CACHE | 3 | 0 | 33556016 | 0 | 0 | 3 | 3 | 0 | 33556016 | 33556016 |
| memory/sql/TABLE | 2498162 | 2490152 | 2662845439 | 2638261263 | 0 | 8010 | 9131 | 0 | 24584176 | 28309707 |
| memory/sql/Filesort_buffer::sort_keys | 3493367 | 3493366 | 115200850575 | 115200817775 | 0 | 1 | 636 | 0 | 32800 | 25885528 |
+--------------------------------------------------------------------------------+-------------+-------------+---------------------------+--------------------------+----------------+--------------------+-----------------+--------------------------+------------------------------+---------------------------+
Asked by O Genthe
(21 rep)
Feb 13, 2024, 02:11 PM
Last activity: Jul 11, 2025, 03:04 PM
Last activity: Jul 11, 2025, 03:04 PM