Storing longer text output in memory in shell variables vs Writing to disk

I have the below call to db2 database command line tool inside a loop, which runs 100k iterations. (the output from db2 is 5 rows of 20 chars approx per each call, total of approx 100k calls. The input to SP is prefixed to the output line from the SP and dumped to a log file.) while read line do db2 -x "call stored_procedure_XYZ($line)" |sed "s/^/$line/" >>log_file.txt done<$infile I am trying to make the whole thing run faster by taking out the sed and disk write from inside the loop. Is it advisable to store all the output in a variable, and later modify and write it to the disk? What would be the length the shell variable can hold? (bash, aix 6.1)