Sample Header Ad - 728x90

xargs cat a file with aws cli (amazon s3 move)

0 votes
0 answers
501 views
I have a file called file.csv, which is a list of the files (or more precisely, file paths path in an S3 Bucket) that need to be moved to another folder in s3. This file lists 53,00,000 files. I have tried the following, but each move is taking a long time.
cat file.csv | xargs -I {} aws s3 mv s3://Bucket1/{} s3://Bucket2/{}
I am trying to speed up the process with the following:
cat file.csv | xargs -P50 -I {} aws s3 mv --recursive s3://Bucket1/{} s3://Bucket2/{}
...but it doesn't seem to be working. I also tried:
while read line; do
  echo ${line} | \
    xargs -n1 -P100 -I {} \
      aws s3 mv s3://Bucket1/{} s3://Bucket2/{} --recursive
done < file.csv
But that doesn't seem to work either. How can I run multiple aws cli commands with xargs by reading the input file?
Asked by hanu (1 rep)
Oct 25, 2023, 03:08 AM
Last activity: Oct 29, 2023, 11:07 AM