Local SQL to Managed Cloud SQL via ADF. Copy Data throughput fell off a cliff
0
votes
0
answers
29
views
I have an on-site mini PC acting as a data extraction and replication SQL server that copies data nightly from the ERP production database that's running some legacy linux architecture. The copying is done using SSIS SQL Agent jobs, where they Truncate the SQL tables and Select * the production tables.
This mini PC has MS Integration Runtime installed and connected to our ADF environment. Similarly, I have ADF running nightly Copy Data pipelines that truncate our managed cloud SQL tables and Select * the on-site replication tables. This whole project is only about a month old.
For the first two weeks these ADF pipelines were seeing a throughput between 6 and 8 MB/s. But suddenly, over the weekend, they started only reaching ~550KB/s at best. Pipelines that were only taking 23 seconds on average now take 2 to 3 minutes. The longest pipeline was averaging 11 minutes for ~13 millions rows now takes about three hours. The local network Prod -> Replication SSIS jobs are still as fast as ever.
I have my IT friends looking at the problem from a networking perspective. Is there anything I can troubleshoot from a DB optimization angle?
Asked by funkyman50
(1 rep)
Feb 14, 2024, 02:42 PM