linux - Redirecting stdout with find -exec and without creating new shell

Question

I have one script that only writes data to stdout. I need to run it for multiple files and generate a different output file for each input file and I was wondering how to use find -exec for that. So I basically tried several variants of this (I replaced the script by cat just for testability purposes):

find * -type f -exec cat "{}" > "{}.stdout" \;

but could not make it work since all the data was being written to a file literally named{}.stdout.

Eventually, I could make it work with :

find * -type f -exec sh -c "cat {} > {}.stdout" \;

But while this latest form works well with cat, my script requires environment variables loaded through several initialization scripts, thus I end up with:

find * -type f -exec sh -c "initscript1; initscript2; ...;  myscript {} > {}.stdout" \;

Which seems a waste because I have everything already initialized in my current shell.

Is there a better way of doing this with find? Other one-liners are welcome.

Answer 1

You can do it with eval. It may be ugly, but so is having to make a shell script for this. Plus, it's all on one line. For example

find -type f -exec bash -c "eval md5sum {}  > {}.sum " \;

Answer 2

A simple solution would be to put a wrapper around your script:

#!/bin/sh

myscript "$1" > "$1.stdout"

Call it myscript2 and invoke it with find:

find . -type f -exec myscript2 {} \;

Note that although most implementations of find allow you to do what you have done, technically the behavior of find is unspecified if you use {} more than once in the argument list of -exec.

Answer 3

If you export your environment variables, they'll already be present in the child shell (If you use bash -c instead of sh -c, and your parent shell is itself bash, then you can also export functions in the parent shell and have them usable in the child; see export -f).

Moreover, by using -exec ... {} +, you can limit the number of shells to the smallest possible number needed to pass all arguments on the command line:

set -a # turn on automatic export of all variables
source initscript1
source initscript2

# pass as many filenames as possible to each sh -c, iterating over them directly
find * -name '*.stdout' -prune -o -type f \
  -exec sh -c 'for arg; do myscript "$arg" > "${arg}.stdout"' _ {} +

Alternately, you can just perform the execution in your current shell directly:

while IFS= read -r -d '' filename; do
  myscript "$filename" >"${filename}.out"
done < <(find * -name '*.stdout' -prune -o -type f -print0)

See UsingFind discussing safely and correctly performing bulk actions through find; and BashFAQ #24 discussing the use of process substitution (the <(...) syntax) to ensure that operations are performed in the parent shell.