Replace string in a huge (70GB), one line, text file
134
votes
14
answers
34713
views
I have a huge (70GB), **one line**, text file and I want to replace a string (token) in it.
I want to replace the token ``, with another dummy token (glove issue ).
I tried
sed
:
sed 's///g' corpus.txt.new
but the output file corpus.txt.new
has zero-bytes!
I also tried using perl:
perl -pe 's///g' corpus.txt.new
but I got an out of memory error.
For smaller files, both of the above commands work.
How can I replace a string is such a file?
This is a related question, but none of the answers worked for me.
**Edit**:
What about splitting the file in chunks of 10GBs (or whatever) each and applying sed
on each one of them and then merging them with cat
? Does that make sense? Is there a more elegant solution?
Asked by Christos Baziotis
(1467 rep)
Dec 29, 2017, 02:58 PM
Last activity: Apr 7, 2024, 02:06 PM
Last activity: Apr 7, 2024, 02:06 PM