Sample Header Ad - 728x90

Unexpected occasional DEADLOCK when re-recreating database from split MySQL dump

1 vote
2 answers
45 views
To replace the content of a test database from a production database mysqldump output the following command was used: cat mysqldump-db-thisdate.sql | mysql -p ... mydb There has never been any issue with this command. However the DB grew a lot and this command takes several minutes. In order to reduce this time, a Perl script was written that - takes the mysqldump output as input - creates a single file having all DROP TABLE ... CREATE TABLE for each table - run this drop-creation file on a single thread, before doing the tables feeding below - creates as many files (see below) as there are tables (about 100 tables) - makes a fork() for each table file that is injected into the DB (all tables are dropped and created + fed in "parallel". table1..100 The DROP-CREATION file is something like DROP TABLE IF EXISTS mytable1; CREATE TABLE mytable1 ( someid1 int NOT NULL, ... PRIMARY KEY (someid1) ) ENGINE=InnoDB AUTO_INCREMENT=111 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; DROP TABLE IF EXISTS mytable2; CREATE TABLE mytable2 ( someid2 int NOT NULL, ... PRIMARY KEY (someid2) ) ENGINE=InnoDB AUTO_INCREMENT=222 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; ... Each table file mytableI.sql is like this, for instance for mytable1.sql /*!40103 SET TIME_ZONE='+00:00' */; SET foreign_key_checks = 0; SET unique_checks = 0; SET autocommit = 0; START TRANSACTION; LOCK TABLES mytable1 WRITE; INSERT INTO mytable1 VALUES (...),(...),...; UNLOCK TABLES; COMMIT; It's like doing, in parallel (pseudo code) for each table 1 to 100 do cat mytableI.sql | mysql -p ... mydb /* I is 1 ... 100 */ end for This method works very well, and saves from 50% to 75% of the time compared to the simple cat whole-dump | mysql usual method. However, from time to time (maybe 1 / 10), doing this parallel method, mysql throws an error > Deadlock found when trying to get lock; try restarting transaction It happens rarely, so just restarting the command is not a big deal. But why? Each table is processed at once, foreign keys are not checked... Doesn't MySQL, thanks to "LOCK TABLES" (and other mechanisms) protect itself against deadlocks in this case? *Addendum*: The mydb test database is not being accesses otherwise. ***edit testing other methods*** Trying to perform the DROP / CREATE operations in parallel, (each DROP / CREATE in the same thread, for each table), not even filling the tables with data, plenty of Deadlocks occur... Could it be that MySQL does not handle very well DROP/CREATE operations performed simultaneously? (should be done by a single DB admin?) Note: "*simultaneously*" and "*in parallel*" meaning each thread has its own MySQL connection.
Asked by Déjà vu (555 rep)
May 29, 2025, 07:29 AM
Last activity: May 31, 2025, 12:15 PM