Using `sem` to make a script run in parallel
5
votes
2
answers
4006
views
I have the following shell script (one liner), which I wanted to use to identify directories that have the exact same contents. I'm using it, to identify and remove duplicate (child) directories.
When I try to run the same script with
sem
, I encounter No such file or directory
errors.
**Example - no parallel threads**
find -type d -links 2 | while read i; do \
find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \
| sort | md5sum | xargs -I {} echo {} $i ; \
done
Gives me:
e94d32e2a683d46d49c7580d649f7888 - ./Daft Punk/Alive 2007 2
e94d32e2a683d46d49c7580d649f7888 - ./Daft Punk/Alive 2007
---
**Example - using sem**
find -type d -links 2 | while read i; do sem -j+0 \
find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \
| sort | md5sum | xargs -I {} echo {} $i ; \
done; sem --wait
Gives me:
find: `./Daft': No such file or directory
find: `Punk/Alive': No such file or directory
find: `2007': No such file or directory
find: `2': No such file or directory
d41d8cd98f00b204e9800998ecf8427e - ./Daft Punk/Alive 2007 2
find: `./Daft': No such file or directory
find: `Punk/Alive': No such file or directory
find: `2007': No such file or directory
d41d8cd98f00b204e9800998ecf8427e - ./Daft Punk/Alive 2007
### Questions:
1. Why the difference in behaviour?
1. How do I remove/fix the No such file or directory
from the sem
script?
1. Are there any other improvements I could make in the script? (there is lots of awk
and xargs
)
Asked by Nick Grealy
(231 rep)
Jun 19, 2016, 03:04 AM
Last activity: Jun 21, 2016, 12:21 PM
Last activity: Jun 21, 2016, 12:21 PM