How do I bulk export data from an RDBMS without killing performance?
1
vote
1
answer
638
views
I'm building an app that lets the user export their data. When they click the Export button, we need to do a
SELECT * WHERE user_id =
, save it to a CSV file, and download it. There may be hundreds of thousands of entries.
If the user edits their data while the export is in progress, the export can include the edits, or not. Either is fine, as long as it's a consistent snapshot taken at any point in time during the export.
The question is how do I do it without killing performance for all other queries? I don't mind if the export takes longer. Can I tell the database to somehow run this query at a low priority?
Assume a MySQL or Postgres database.
The only solution I know is to set up a read replica just for exports. Exports can slow down the replica, but won't slow down the primary, thus ensuring fast performance for other queries. Is there a better solution to this problem?
Asked by Kartick Vaddadi
(113 rep)
Aug 17, 2022, 06:04 AM
Last activity: Aug 19, 2022, 05:45 AM
Last activity: Aug 19, 2022, 05:45 AM