Sample Header Ad - 728x90

Curl Parallel requests using links source file

1 vote
2 answers
2997 views
I have this script to go through a list of URLs and the check return codes using Curl. Links file goes like this: https://link1/ ... https://link2/ ... https://link200/ ... (...) The script: INDEX=0 DIR="$(grep [WorkingDir file] | cut -d \" -f 2)" WORKDIR="${DIR}/base" ARQLINK="navbase.txt" for URL in $(cat $WORKDIR/$ARQLINK); do INDEX=$((INDEX + 1)) HTTP_CODE=$(curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' $URL) if [ $HTTP_CODE -eq 200 ]; then printf "\n%.3d => OK! - $URL" $INDEX; else printf "\n\n%.3d => FAIL! - $URL\n" $INDEX; fi done It takes a little while to run through every URL, so I was wondering how to speed those up curl requests. Maybe I could use some parallel Curl requests, but using "xargs" inside a "for" loop while also printing a message doesn't seem the way to go. I was able to use "xargs" out of the script and it sort of works, although not showing the correct HTTP code. cat navbase.txt | xargs -I % -P 10 curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' % I couldn't find a way to insert that into the script. Any tips?
Asked by markfree (425 rep)
Jan 4, 2022, 02:05 PM
Last activity: Mar 19, 2025, 11:10 AM