Using cURL, jq, and declaration and for loop condition, I tried to download multiple files from a GitLab private repo, but it downloaded only one
1
vote
1
answer
1374
views
I learned from the following sources:
- **
curl -O
**: [Download a file with curl on Linux / Unix command line](https://www.cyberciti.biz/faq/download-a-file-with-curl-on-linux-unix-command-line/)
- **jq:** [How to urlencode data for curl command?](https://stackoverflow.com/questions/296536/how-to-urlencode-data-for-curl-command/34407620#34407620)
- **Multiple files and curl -J
:** https://unix.stackexchange.com/questions/91798/download-pdf-files-from-using-curl
- **Condition for
loop:** https://unix.stackexchange.com/questions/377446/shell-how-to-use-2-variables-with-for-condition and https://unix.stackexchange.com/questions/291049/unable-to-download-data-using-curl-for-loop
Description of the script:
1. Variables, required by GitLab's [Repository files API](https://docs.gitlab.com/ee/api/repository_files.html) :
branch="master"
repo="my-dotfiles"
private_token="XXY_wwwwwx-qQQQRRSSS"
username="gusbemacbe"
2. I used a declaration for multiple files:
declare -a context_dirs=(
"home/.config/Code - Insiders/Preferences"
"home/.config/Code - Insiders/languagepacks.json"
"home/.config/Code - Insiders/rapid_render.json"
"home/.config/Code - Insiders/storage.json"
)
3. I used the condition for
loop with jq
to convert all files from the declaration context_dirs
to encoded URLs:
for urlencode in "${context_dirs[@]}"; do
paths=$(jq -nr --arg v "$urlencode" '$v|@uri')
done
4. I used the condition for
loop to download with curl
multiple files taken from paths
converted by jq
. It is important that I used -0
and -J
to output the file name, and -H
for "PRIVATE-TOKEN: $private_token"
:
for file in "${paths[@]}"; do
curl -sLOJH "PRIVATE-TOKEN: $private_token" "https://gitlab.com/api/v4/projects/$username%2F$repo/repository/files/$file/raw?ref=$branch "
done
Complete source code:
branch="master"
id="1911000X"
repo="my-dotfiles"
private_token="XXY_wwwwwx-qQQQRRSSS"
username="gusbemacbe"
declare -a context_dirs=(
"home/.config/Code - Insiders/Preferences"
"home/.config/Code - Insiders/languagepacks.json"
"home/.config/Code - Insiders/rapid_render.json"
"home/.config/Code - Insiders/storage.json"
)
for urlencode in "${context_dirs[@]}"; do
paths=$(jq -nr --arg v "$urlencode" '$v|@uri')
done
for file in "${paths[@]}"; do
curl -sLOJH "PRIVATE-TOKEN: $private_token" "https://gitlab.com/api/v4/projects/$username%2F$repo/repository/files/$file/raw?ref=$branch "
done
But the two conditions for
loop output only an encoded path and downloaded only a file.
Asked by Oo'-
(255 rep)
Jun 1, 2020, 06:20 AM
Last activity: Jun 1, 2020, 06:50 AM
Last activity: Jun 1, 2020, 06:50 AM