xargs cat a file with aws cli (amazon s3 move)
0
votes
0
answers
501
views
I have a file called
file.csv
, which is a list of the files (or more precisely, file paths path in an S3 Bucket) that need to be moved to another folder in s3.
This file lists 53,00,000 files. I have tried the following, but each move is taking a long time.
cat file.csv | xargs -I {} aws s3 mv s3://Bucket1/{} s3://Bucket2/{}
I am trying to speed up the process with the following:
cat file.csv | xargs -P50 -I {} aws s3 mv --recursive s3://Bucket1/{} s3://Bucket2/{}
...but it doesn't seem to be working.
I also tried:
while read line; do
echo ${line} | \
xargs -n1 -P100 -I {} \
aws s3 mv s3://Bucket1/{} s3://Bucket2/{} --recursive
done < file.csv
But that doesn't seem to work either.
How can I run multiple aws cli commands with xargs
by reading the input file?
Asked by hanu
(1 rep)
Oct 25, 2023, 03:08 AM
Last activity: Oct 29, 2023, 11:07 AM
Last activity: Oct 29, 2023, 11:07 AM