Sample Header Ad - 728x90

Transferring very large dataset from cluster to a storage server

1 vote
1 answer
540 views
We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast transfer tool that can be parallelized for individual files (because the individual files are each in terabytes). In this regard, I am looking for a tool that does not require admin rights and is still considerably faster than scp or rsync. If there is any tool that can be installed locally without admin rights, that will also be useful. I come across this link (https://unix.stackexchange.com/questions/227951/what-is-the-fastest-way-to-send-massive-amounts-of-data-between-two-computers) , which mentions the netcat way but we couldn't make it work. For information, we are trying to copy relatively few files with very large size (and not many many small files). Appreciate your time and help
Asked by Ikram Ullah (113 rep)
Mar 8, 2023, 02:45 PM
Last activity: Mar 17, 2023, 01:50 PM