Sample Header Ad - 728x90

Replace string in a huge (70GB), one line, text file

134 votes
14 answers
34713 views
I have a huge (70GB), **one line**, text file and I want to replace a string (token) in it. I want to replace the token ``, with another dummy token (glove issue ). I tried sed: sed 's///g' corpus.txt.new but the output file corpus.txt.new has zero-bytes! I also tried using perl: perl -pe 's///g' corpus.txt.new but I got an out of memory error. For smaller files, both of the above commands work. How can I replace a string is such a file? This is a related question, but none of the answers worked for me. **Edit**: What about splitting the file in chunks of 10GBs (or whatever) each and applying sed on each one of them and then merging them with cat? Does that make sense? Is there a more elegant solution?
Asked by Christos Baziotis (1467 rep)
Dec 29, 2017, 02:58 PM
Last activity: Apr 7, 2024, 02:06 PM