check patterns that don't exist in sqlite
4
votes
3
answers
329
views
I explained a similar situation with plain text files on https://unix.stackexchange.com/questions/29624/grep-huge-number-of-patterns-from-huge-file/30144 . Many people there said I should, so now I'm migrating my data to a sqlite database:
I have a file from which I extract about 10,000 patterns. Then I check if the database doesn't contain such patterns. If it doesn't, I need to save them externally in
file
for further processing:
for id in $(grep ^[0-9] keys); do
if [[ -z $(sqlite3 db.sqlite "select id from main where id = $id") ]]; then
echo $id >>file
fi
done
Since I'm new to SQL, I couldn't find a simple way to do this. Also, this loop is useless as it is 20 times slower than what I achieved with awk
on the mentioned URL.
Since the database is huge, keeps growing, and I run this loop very frequently, is it possible to make this faster?
Asked by admirabilis
(4792 rep)
Jan 31, 2012, 12:54 AM
Last activity: Nov 11, 2023, 01:26 PM
Last activity: Nov 11, 2023, 01:26 PM