I have a machine with 2 TB of RAM and I am running a sort command on a file of size 150G where I have specified the buffer-size as 1000G, after doing my bit of reasearch on google, I got this piece of information "the more is the buffer size, the better is the performance". This is the command that I ran
sort -rk2 --buffer-size=1000G master_matrix_unsorted.csv > master_matrix_sorted.csv
But this is taking a lot of time and I have no clue on the progress of the task.
Any idea on what should be the best buffer size for this operation? I am planning to re-run this task with a new buffer-size.
Asked by Sambit Tripathy
(171 rep)
Jul 22, 2014, 10:55 PM
Last activity: Feb 14, 2024, 06:51 PM
Last activity: Feb 14, 2024, 06:51 PM