MySQL 8.0.35 mixed GTID replica/primary node: how to make primary tables replicate not just replica tables to downstream replica
0
votes
2
answers
40
views
LATER ADDITION: I resolved with a
reset master
on surface
which I had been hesitant to execute because it might negatively impact replication to offsite
which it did. Hopefully that simpler replication is easily restored.
This is a 4 node replica system: monitor->web->surface->offsite
where monitor
is a recent addition and adds a few tables to web
which had previously been the sole primary node. A parallel replication chain backend->surface->offsite
exists. web->surface
replication takes place for inserts to tables replicating from monitor
but does not occur for inserts to primary tables web
(or for creation of a test table on web
). I note that (not my choice) (1) monitor
implemented a different database name which is accommodated in web
's my.conf and (2) uses MySQL 8.0.42 (probably not relevant given that replication to web
works and the MySQL change documentation does not suggest breaking changes but included for completeness):
replicate-rewrite-db = monitor_db_name->web_db_name
To restore replication with new tables on web, a full dump on web
was made as follows:
mysqldump -u root -p --databases web_db_name --single-transaction --master-data --extended-insert --triggers --routines --events --set-gtid-purged=OFF | gzip > web_db.sql.gz
and manually changed the CHANGE MASTER line the the dump file:
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=220385066 for channel 'slave_channel_web';
On surface
,
stop replica for channel 'slave_channel_web';
Confirm replication stopped:
show replica status for channel 'slave_channel_web'\G;
insert dump file
start replica for channel 'slave_channel_web';
Confirm replication is working:
show replica status for channel 'slave_channel_web'\G;
Here are the contents of /etc/my.cnf
on web
:
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove the leading "# " to disable binary logging
# Binary logging captures changes between backups and is enabled by
# default. It's default setting is log_bin=binlog
# disable_log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#
# Remove leading # to revert to previous value for default_authentication_plugin,
# this will increase compatibility with older clients. For background, see:
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_default_authentication_plugin
# default-authentication-plugin=mysql_native_password
# Specifying the timezone for the databases
default-time-zone = 'US/Eastern'
# Setting the default authentication plugin
default_authentication_plugin=caching_sha2_password
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
symbolic-links=0
log-bin=/data/mysql-bin/mysql-bin
server-id=2
gtid_mode=ON
enforce-gtid-consistency=true
max_allowed_packet=64M
relay-log-info-repository=TABLE
master-info-repository=TABLE
#skip-slave-start=1
replicate-rewrite-db = monitor_db_name->web_db_name
relay-log=web-relay-bin
Here are the contents of /etc/my.cnf
on surface
:
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
# datadir=/var/lib/mysql
# Setting the timezone for the databases
default-time-zone = 'US/Eastern'
# Setting the default authentication plugin
default_authentication_plugin=caching_sha2_password
datadir=/data/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin=mysql-bin
server-id=3
gtid_mode=ON
enforce-gtid-consistency=true
relay-log=surf-relay-bin
relay-log-info-repository=TABLE
master-info-repository=TABLE
#skip-slave-start=1
innodb_buffer_pool_size=4G
#innodb_flush_log_at_trx_commit=1 # Set to 2 for faster mysql insert from dump and back to 1 afterwards for data integrity
innodb_log_file_size=256M
innodb_flush_method=O_DIRECT
Mysql replica status suggests replication working (via an ssh tunnel to port 13306 on surface
):
mysql> show replica status for channel 'slave_channel_web'\G;
*************************** 1. row ***************************
Replica_IO_State: Waiting for source to send event
Source_Host: 127.0.0.1
Source_User: repl
Source_Port: 13306
Connect_Retry: 60
Source_Log_File: mysql-bin.000002
Read_Source_Log_Pos: 235405762
Relay_Log_File: cdms-dqsurf-relay-bin-slave_channel_web.000002
Relay_Log_Pos: 15021022
Relay_Source_Log_File: mysql-bin.000002
Replica_IO_Running: Yes
Replica_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Source_Log_Pos: 235405762
Relay_Log_Space: 15021256
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Source_SSL_Allowed: No
Source_SSL_CA_File:
Source_SSL_CA_Path:
Source_SSL_Cert:
Source_SSL_Cipher:
Source_SSL_Key:
Seconds_Behind_Source: 0
Source_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Source_Server_Id: 2
Source_UUID: 8b3fb531-b226-11e8-81e7-b496912d2d25
Source_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
Source_Retry_Count: 86400
Source_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Source_SSL_Crl:
Source_SSL_Crlpath:
Retrieved_Gtid_Set: 1be71be7-2cf6-11f0-b3fe-e8cf832af101:177712-180447,
8b3fb531-b226-11e8-81e7-b496912d2d25:37213-40242
Executed_Gtid_Set: 1be71be7-2cf6-11f0-b3fe-e8cf832af101:175644-175649:175672-175675:175702-180447,
29dbdef6-b155-11e8-a7f1-b49691213024:8-2421409,
8b3fb531-b226-11e8-81e7-b496912d2d25:1-40777:40864-2418597,
d4ee172d-bc52-11e8-b3eb-b496912d2f49:1-31882
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: slave_channel_web
Source_TLS_Version:
Source_public_key_path:
Get_Source_public_key: 0
Network_Namespace:
1 row in set (0.00 sec)
That is consistent with new data from a table (chilled_water) on monitor
arriving on surface
. The surface
MySQL log file agrees that the replication connection to web
was successful. On surface
I see the chilled_water table from monitor
updating:
mysql> select id,timestamp from chilled_water order by id desc limit 5;
+---------+---------------------+
| id | timestamp |
+---------+---------------------+
| 1696323 | 2025-07-16 09:21:22 |
| 1696322 | 2025-07-16 09:21:12 |
| 1696321 | 2025-07-16 09:20:52 |
| 1696320 | 2025-07-16 09:21:02 |
| 1696319 | 2025-07-16 09:20:51 |
+---------+---------------------+
5 rows in set (0.00 sec)
However, information inserted on web
into web-specific tables (that do not exist in monitor) do not replicate to surface
as is seen in new information on the ups table on web
not showing up on surface
and a test table created on web not replicating to surface
.
mysql> select id,time from ups order by id desc limit 5;
+--------+---------------------+
| id | time |
+--------+---------------------+
| 359423 | 2025-07-15 11:36:03 |
| 359422 | 2025-07-15 11:35:02 |
| 359421 | 2025-07-15 11:34:02 |
| 359420 | 2025-07-15 11:33:02 |
| 359419 | 2025-07-15 11:32:02 |
+--------+---------------------+
5 rows in set (0.00 sec)
I'd appreciate any advise on how to proceed in debugging and addressing this.
The form asked what I've tried to fix the issue. I tried stopping and restarting the replica, redoing the above process from a fresh database backup, and combing the web and MySQL documentation for input.
Asked by Joel Sander
(1 rep)
Jul 16, 2025, 06:21 PM
Last activity: Jul 18, 2025, 07:58 PM
Last activity: Jul 18, 2025, 07:58 PM