Sample Header Ad - 728x90

MySQL 8.0.35 mixed GTID replica/primary node: how to make primary tables replicate not just replica tables to downstream replica

0 votes
2 answers
40 views
LATER ADDITION: I resolved with a reset master on surface which I had been hesitant to execute because it might negatively impact replication to offsite which it did. Hopefully that simpler replication is easily restored. This is a 4 node replica system: monitor->web->surface->offsite where monitor is a recent addition and adds a few tables to web which had previously been the sole primary node. A parallel replication chain backend->surface->offsite exists. web->surface replication takes place for inserts to tables replicating from monitor but does not occur for inserts to primary tables web (or for creation of a test table on web). I note that (not my choice) (1) monitor implemented a different database name which is accommodated in web's my.conf and (2) uses MySQL 8.0.42 (probably not relevant given that replication to web works and the MySQL change documentation does not suggest breaking changes but included for completeness):
replicate-rewrite-db = monitor_db_name->web_db_name
To restore replication with new tables on web, a full dump on web was made as follows:
mysqldump -u root -p --databases web_db_name --single-transaction --master-data --extended-insert --triggers --routines --events --set-gtid-purged=OFF | gzip > web_db.sql.gz
and manually changed the CHANGE MASTER line the the dump file: CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=220385066 for channel 'slave_channel_web'; On surface,
stop replica for channel 'slave_channel_web';
Confirm replication stopped:
show replica status for channel 'slave_channel_web'\G;
insert dump file
start replica for channel 'slave_channel_web';
Confirm replication is working:
show replica status for channel 'slave_channel_web'\G;
Here are the contents of /etc/my.cnf on web:
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove the leading "# " to disable binary logging
# Binary logging captures changes between backups and is enabled by
# default. It's default setting is log_bin=binlog
# disable_log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#
# Remove leading # to revert to previous value for default_authentication_plugin,
# this will increase compatibility with older clients. For background, see:
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_default_authentication_plugin 
# default-authentication-plugin=mysql_native_password

# Specifying the timezone for the databases 
default-time-zone = 'US/Eastern'

# Setting the default authentication plugin
default_authentication_plugin=caching_sha2_password

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

symbolic-links=0
log-bin=/data/mysql-bin/mysql-bin
server-id=2
gtid_mode=ON
enforce-gtid-consistency=true
max_allowed_packet=64M

relay-log-info-repository=TABLE
master-info-repository=TABLE
#skip-slave-start=1

replicate-rewrite-db = monitor_db_name->web_db_name

relay-log=web-relay-bin
Here are the contents of /etc/my.cnf on surface:
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
# datadir=/var/lib/mysql

# Setting the timezone for the databases
default-time-zone = 'US/Eastern'

# Setting the default authentication plugin 
default_authentication_plugin=caching_sha2_password

datadir=/data/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

log-bin=mysql-bin
server-id=3
gtid_mode=ON
enforce-gtid-consistency=true

relay-log=surf-relay-bin
relay-log-info-repository=TABLE
master-info-repository=TABLE

#skip-slave-start=1

innodb_buffer_pool_size=4G
#innodb_flush_log_at_trx_commit=1 # Set to 2 for faster mysql insert from dump and back to 1 afterwards for data integrity
innodb_log_file_size=256M
innodb_flush_method=O_DIRECT
Mysql replica status suggests replication working (via an ssh tunnel to port 13306 on surface):
mysql> show replica status for channel 'slave_channel_web'\G;
*************************** 1. row ***************************
             Replica_IO_State: Waiting for source to send event
                  Source_Host: 127.0.0.1
                  Source_User: repl
                  Source_Port: 13306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000002
          Read_Source_Log_Pos: 235405762
               Relay_Log_File: cdms-dqsurf-relay-bin-slave_channel_web.000002
                Relay_Log_Pos: 15021022
        Relay_Source_Log_File: mysql-bin.000002
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Source_Log_Pos: 235405762
              Relay_Log_Space: 15021256
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Source_SSL_Allowed: No
           Source_SSL_CA_File:
           Source_SSL_CA_Path:
              Source_SSL_Cert:
            Source_SSL_Cipher:
               Source_SSL_Key:
        Seconds_Behind_Source: 0
Source_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Source_Server_Id: 2
                  Source_UUID: 8b3fb531-b226-11e8-81e7-b496912d2d25
             Source_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                  Source_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Source_SSL_Crl:
           Source_SSL_Crlpath:
           Retrieved_Gtid_Set: 1be71be7-2cf6-11f0-b3fe-e8cf832af101:177712-180447,
8b3fb531-b226-11e8-81e7-b496912d2d25:37213-40242
            Executed_Gtid_Set: 1be71be7-2cf6-11f0-b3fe-e8cf832af101:175644-175649:175672-175675:175702-180447,
29dbdef6-b155-11e8-a7f1-b49691213024:8-2421409,
8b3fb531-b226-11e8-81e7-b496912d2d25:1-40777:40864-2418597,
d4ee172d-bc52-11e8-b3eb-b496912d2f49:1-31882
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name: slave_channel_web
           Source_TLS_Version:
       Source_public_key_path:
        Get_Source_public_key: 0
            Network_Namespace:
1 row in set (0.00 sec)
That is consistent with new data from a table (chilled_water) on monitor arriving on surface. The surface MySQL log file agrees that the replication connection to web was successful. On surface I see the chilled_water table from monitor updating:
mysql> select id,timestamp from chilled_water order by id desc limit 5;
+---------+---------------------+
| id      | timestamp           |
+---------+---------------------+
| 1696323 | 2025-07-16 09:21:22 |
| 1696322 | 2025-07-16 09:21:12 |
| 1696321 | 2025-07-16 09:20:52 |
| 1696320 | 2025-07-16 09:21:02 |
| 1696319 | 2025-07-16 09:20:51 |
+---------+---------------------+
5 rows in set (0.00 sec)
However, information inserted on web into web-specific tables (that do not exist in monitor) do not replicate to surface as is seen in new information on the ups table on web not showing up on surface and a test table created on web not replicating to surface.
mysql> select id,time from ups order by id desc limit 5;
+--------+---------------------+
| id     | time                |
+--------+---------------------+
| 359423 | 2025-07-15 11:36:03 |
| 359422 | 2025-07-15 11:35:02 |
| 359421 | 2025-07-15 11:34:02 |
| 359420 | 2025-07-15 11:33:02 |
| 359419 | 2025-07-15 11:32:02 |
+--------+---------------------+
5 rows in set (0.00 sec)
I'd appreciate any advise on how to proceed in debugging and addressing this. The form asked what I've tried to fix the issue. I tried stopping and restarting the replica, redoing the above process from a fresh database backup, and combing the web and MySQL documentation for input.
Asked by Joel Sander (1 rep)
Jul 16, 2025, 06:21 PM
Last activity: Jul 18, 2025, 07:58 PM