Persona Xtradb cluster CONFLICT DETECTED
1
vote
0
answers
228
views
I am working on a high-traffic platform with MySQL clustering using Percona XtraDB Cluster. I have a cluster consisting of 4 nodes. The database is receiving more than 500 requests per second for inserting, selecting, and updating data on the same tables. Below is my MySQL configuration:
[mysqld]
######## START General settings ###############
# General settings
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
mysqlx_socket=/tmp/xplugin.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
bind-address = 192.168.100.20
log-error-verbosity=3
innodb_redo_log_capacity=2147483648
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
auto_increment_increment = 5
auto_increment_offset = 3
############################## END General settings ##############################
############################## START wsrep ##############################
#-
server-id=2
# Path to Galera library
#-
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
#-
wsrep_cluster_address=gcomm://192.168.100.2,192.168.100.1,192.168.100.20,192.168.100.21
# In order for Galera to work correctly binlog format should be ROW
#-
binlog_format=ROW
# Slave thread to use
#-
wsrep_slave_threads=16
#-
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
#-
innodb_autoinc_lock_mode=2
# Node IP address
#-
wsrep_node_address=192.168.100.20
# Cluster name
#-
wsrep_cluster_name=pxc-cluster
#If wsrep_node_name is not specified, then system hostname will be used
#-
wsrep_node_name=pxc-cluster-node-3
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
#-
pxc_strict_mode=ENFORCING
# SST method
#-
wsrep_sst_method=xtrabackup-v2
#-
pxc_encrypt_cluster_traffic = OFF
############################## END wsrep ##############################
# Number of CPU cores
innodb_read_io_threads = 16
innodb_write_io_threads = 16
innodb_thread_concurrency = 16
innodb_purge_threads = 4
innodb_page_cleaners = 4
innodb_log_write_ahead_size=4
thread_cache_size = 64
#thread_handling = pool-of-threads
thread_handling = one-thread-per-connection
# Buffer pool size (assuming 128GB RAM)
innodb_buffer_pool_size = 100G
# Storage device type
innodb_io_capacity = 1500 # Adjust for NVMe SSD
innodb_io_capacity_max = 3000 # Adjust for NVMe SSD
############################## START Performance settings ##############################
default_storage_engine = InnoDB
key_buffer_size = 256M
innodb_buffer_pool_instances = 16
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 16M
innodb_flush_method = O_DIRECT
innodb_max_dirty_pages_pct = 90
innodb_flush_neighbors = 0
innodb_lru_scan_depth = 4096
innodb_change_buffer_max_size = 50
innodb_adaptive_hash_index = 0
innodb_doublewrite = 0
innodb_flush_sync = 0
innodb_max_undo_log_size = 2G
innodb_use_native_aio = 0
innodb_undo_log_truncate = ON
############################## END Performance settings ##############################
############################## Start Connection setting ##############################
# Connection settings
max_connections = 3000
max_allowed_packet = 256M
wait_timeout = 600
interactive_timeout = 600
############################## END Connection setting ##############################
############################## START Logging settings ##############################
log_error = /var/log/mysql/error.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 5
log_queries_not_using_indexes = 1
log_slow_admin_statements = 1
log_slow_replica_statements = 1
############################## END Logging settings ##############################
However, I am facing an issue with "CONFLICT DETECTED" And LOCK issue on my nodes, and the following errors are being reported:
2024-01-12T22:52:49.382682Z 16 [Note] [MY-011825] [InnoDB] *** WAITING FOR THIS LOCK TO BE GRANTED:
2024-01-12T22:52:49.382697Z 16 [Note] [MY-011825] [InnoDB] SQL1:
2024-01-12T22:52:49.382711Z 16 [Note] [MY-011825] [InnoDB] SQL2: INSERT INTO
requests_log
.....
2024-01-12T22:52:49.382739Z 16 [Note] [MY-000000] [WSREP] --------- CONFLICT DETECTED --------
2024-01-12T22:52:49.382756Z 16 [Note] [MY-000000] [WSREP] cluster conflict due to high priority abort for threads:
2024-01-12T22:52:49.382768Z 16 [Note] [MY-000000] [WSREP] Winning thread:
THD: 16, mode: high priority, state: exec, conflict: executing, seqno: 177984003
SQL: (null)
2024-01-12T22:52:49.382781Z 16 [Note] [MY-000000] [WSREP] Victim thread:
THD: 2338759, mode: local, state: exec, conflict: certifying, seqno: -1
SQL: INSERT INTO requests_log
......
2024-01-12T22:52:49.407148Z 2338784 [Note] [MY-011825] [InnoDB] *** Lock requesting TRANSACTION:
TRANSACTION 255572069, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
, undo log entries 1
MySQL thread id 2338784, OS thread handle 139937898251840, query id 37507384 wsrep: writing row for write-set (177984004)
2024-01-12T22:52:49.407423Z 2338784 [Note] [MY-011825] [InnoDB] *** Lock holder TRANSACTION:
TRANSACTION 255572067, ACTIVE (PREPARED) 0 sec
, undo log entries 1
MySQL thread id 16, OS thread handle 140003591190080, query id 37507363 wsrep: preparing to commit write set(177984003)
2024-01-12T22:52:50.780305Z 21 [Note] [MY-011825] [InnoDB] *** Priority TRANSACTION:
TRANSACTION 255572175, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT , undo log entries 1
MySQL thread id 21, OS thread handle 139994867009088, query id 37507865 wsrep: writing row for write-set (177984088)
2024-01-12T22:52:50.780456Z 21 [Note] [MY-011825] [InnoDB] *** Victim TRANSACTION:
TRANSACTION 255572174, ACTIVE 0 sec, thread declared inside InnoDB 4999
mysql tables in use 1, locked 1
, undo log entries 1
MySQL thread id 2338797, OS thread handle 139937889797696, query id 37507862 192.168.100.20 mixsimverify wsrep: replicating and certifying write set(-1)
update users
set balance
= ....
2024-01-12T22:52:50.780515Z 21 [Note] [MY-011825] [InnoDB] *** WAITING FOR THIS LOCK TO BE GRANTED:
2024-01-12T22:52:50.780530Z 21 [Note] [MY-011825] [InnoDB] SQL1:
2024-01-12T22:52:50.780545Z 21 [Note] [MY-011825] [InnoDB] SQL2: update users
set balance
= .....
2024-01-12T22:52:50.780564Z 21 [Note] [MY-000000] [WSREP] --------- CONFLICT DETECTED --------
2024-01-12T22:52:50.780588Z 21 [Note] [MY-000000] [WSREP] cluster conflict due to high priority abort for threads:
2024-01-12T22:52:50.780606Z 21 [Note] [MY-000000] [WSREP] Winning thread:
THD: 21, mode: high priority, state: exec, conflict: executing, seqno: 177984088
SQL: (null)
2024-01-12T22:52:50.780624Z 21 [Note] [MY-000000] [WSREP] Victim thread:
THD: 2338797, mode: local, state: exec, conflict: certifying, seqno: -1
SQL: update users
set balance
= ......
2024-01-12T22:52:50.802889Z 2338810 [Note] [MY-011825] [InnoDB] *** Lock requesting TRANSACTION:
TRANSACTION 255572178, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
MySQL thread id 2338810, OS thread handle 139937889797696, query id 37507868 Applying batch of row changes (update)
2024-01-12T22:52:50.803030Z 2338810 [Note] [MY-011825] [InnoDB] *** Lock holder TRANSACTION:
TRANSACTION 255572175, ACTIVE (PREPARED) 0 sec committing
, undo log entries 1
MySQL thread id 21, OS thread handle 139994867009088, query id 37507865 innobase_commit_low (177984088)
2024-01-12T22:52:51.291489Z 12 [Note] [MY-011825] [InnoDB] *** Priority TRANSACTION:
TRANSACTION 255572217, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT
MySQL thread id 12, OS thread handle 140007278396992, query id 37508017 Applying batch of row changes (update)
2024-01-12T22:52:51.291632Z 12 [Note] [MY-011825] [InnoDB] *** Victim TRANSACTION:
TRANSACTION 255572216, ACTIVE 0 sec, thread declared inside InnoDB 4999
mysql tables in use 1, locked 1
, undo log entries 1
MySQL thread id 2338798, OS thread handle 139954061203008, query id 37508015 192.168.100.20 mixsimverify wsrep: replicating and certifying write set(-1)
update users
set balance
= ....
2024-01-12T22:52:51.291691Z 12 [Note] [MY-011825] [InnoDB] *** WAITING FOR THIS LOCK TO BE GRANTED:
2024-01-12T22:52:51.291707Z 12 [Note] [MY-011825] [InnoDB] SQL1:
2024-01-12T22:52:51.291751Z 12 [Note] [MY-011825] [InnoDB] SQL2: update users
set balance
= ....
2024-01-12T22:52:51.291779Z 12 [Note] [MY-000000] [WSREP] --------- CONFLICT DETECTED --------
2024-01-12T22:52:51.291798Z 12 [Note] [MY-000000] [WSREP] cluster conflict due to high priority abort for threads:
How can I address and resolve this conflict and the LOCK issue?
Asked by Sam Wiki
(21 rep)
Jan 12, 2024, 11:04 PM