How to efficiently extract data from a csv file and save it in mysql
2
votes
1
answer
328
views
I have a csv file, maybe 1000 or more rows, and echo row’s data is related to two table in mysql. Example,
table book
(id, title, …)
table scene
(id, book_id, short_content, position…)
And the csv file look like:
title,short_content1,position1,short_content2,position2,…
...
I want to extract data from it and save it to mysql. Also in the mysql maybe already have the same record, so I don’t want save the same data to mysql.
My solution is:
- First, fetch all the data related to the file, a big data set labeled BS
.
- Second, for each row in file compare to the BS
and create the book, and store the book_id for his scene
.
- Finally, batch create the scene
.
This is my poor solution, and if the csv file has 1000 rows, then the solution will take a large memory space and 1000 + 1 times mysql calls.
I see Bulk database update/insert from CSV file and Update Oracle database from CSV post. But I still confuse how to deal with csv file which related to multiple tables.
Can anyone give me a better solution.
Thanks.
Asked by hstk
(21 rep)
Jun 9, 2019, 02:01 PM
Last activity: Jul 21, 2021, 07:43 PM
Last activity: Jul 21, 2021, 07:43 PM