159

我正在编写一个脚本来自动为我自己的网络服务器创建 Apache 和 PHP 的配置文件。我不想使用任何图形用户界面，如 CPanel 或 ISPConfig。

我有一些 Apache 和 PHP 配置文件的模板。Bash 脚本需要读取模板，进行变量替换并将解析的模板输出到某个文件夹中。最好的方法是什么？我可以想到几种方法。哪一个是最好的，或者可能有更好的方法来做到这一点？我想在纯 Bash 中做到这一点（例如在 PHP 中很容易）

1)如何替换文本文件中的 ${} 占位符？

模板.txt：

the number is ${i}
the word is ${word}

脚本.sh：

#!/bin/sh

#set variables
i=1
word="dog"
#read in template one line at the time, and replace variables
#(more natural (and efficient) way, thanks to Jonathan Leffler)
while read line
do
    eval echo "$line"
done < "./template.txt"

顺便说一句，我如何在这里将输出重定向到外部文件？如果变量包含引号，我是否需要转义？

2) 使用 cat & sed 将每个变量替换为其值：

给定模板.txt：

The number is ${i}
The word is ${word}

命令：

cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/"

对我来说似乎很糟糕，因为需要转义许多不同的符号并且有很多变量，这条线太长了。

你能想到其他一些优雅和安全的解决方案吗？

4

23 回答 23

192

Try envsubst

FOO=foo
BAR=bar
export FOO BAR

envsubst <<EOF
FOO is $FOO
BAR is $BAR
EOF
于 2012-06-15T12:48:40.780 回答
67
于 2010-05-26T19:35:58.720 回答
67

envsubst was new to me. Fantastic.

For the record, using a heredoc is a great way to template a conf file.

STATUS_URI="/hows-it-goin";  MONITOR_IP="10.10.2.15";

cat >/etc/apache2/conf.d/mod_status.conf <<EOF
<Location ${STATUS_URI}>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from ${MONITOR_IP}
</Location>
EOF
于 2012-08-17T13:33:14.693 回答
40

我同意使用 sed：它是搜索/替换的最佳工具。这是我的方法：

$ cat template.txt
the number is ${i}
the dog's name is ${name}

$ cat replace.sed
s/${i}/5/
s/${name}/Fido/

$ sed -f replace.sed template.txt > out.txt

$ cat out.txt
the number is 5
the dog's name is Fido
于 2010-05-26T18:13:52.723 回答
31

I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes

eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
于 2013-06-10T18:59:36.497 回答
25

I think eval works really well. It handles templates with linebreaks, whitespace, and all sorts of bash stuff. If you have full control over the templates themselves of course:

$ cat template.txt
variable1 = ${variable1}
variable2 = $variable2
my-ip = \"$(curl -s ifconfig.me)\"

$ echo $variable1
AAA
$ echo $variable2
BBB
$ eval "echo \"$(<template.txt)\"" 2> /dev/null
variable1 = AAA
variable2 = BBB
my-ip = "11.22.33.44"

This method should be used with care, of course, since eval can execute arbitrary code. Running this as root is pretty much out of the question. Quotes in the template need to be escaped, otherwise they will be eaten by eval.

You can also use here documents if you prefer cat to echo

$ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null

@plockc provoded a solution that avoids the bash quote escaping issue:

$ eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null

Edit: Removed part about running this as root using sudo...

Edit: Added comment about how quotes need to be escaped, added plockc's solution to the mix!

于 2012-09-14T10:08:08.803 回答
18

Edit Jan 6, 2017

I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps:

render_template() {
  eval "echo \"$(sed 's/\"/\\\\"/g' $1)\""
}

I can't think of keeping trailing new lines, but empty lines in between are kept.

Although it is an old topic, IMO I found out more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/

#!/bin/sh

# render a template configuration file
# expand variables + preserve formatting
render_template() {
  eval "echo \"$(cat $1)\""
}

user="Gregory"
render_template /path/to/template.txt > path/to/configuration_file

All credits to Grégory Pakosz.

于 2014-02-02T18:45:55.880 回答
14

Instead of reinventing the wheel go with envsubst Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.

If on mac make sure you have homebrew then link it from gettext:

brew install gettext
brew link --force gettext

./template.cfg

# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}

./.env:

SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2

./configure.sh

#!/bin/bash
cat template.cfg | envsubst > whatever.cfg

Now just use it:

# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables 
# if your solution depends on tools that utilise .env file 
# automatically like pipenv etc. 
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh
于 2018-08-31T08:45:07.450 回答
11

I'd have done it this way, probably less efficient, but easier to read/maintain.

TEMPLATE='/path/to/template.file'
OUTPUT='/path/to/output.file'

while read LINE; do
  echo $LINE |
  sed 's/VARONE/NEWVALA/g' |
  sed 's/VARTWO/NEWVALB/g' |
  sed 's/VARTHR/NEWVALC/g' >> $OUTPUT
done < $TEMPLATE
于 2010-05-28T11:07:05.333 回答
10

If you want to use Jinja2 templates, see this project: j2cli.

It supports:

  • Templates from JSON, INI, YAML files and input streams
  • Templating from environment variables
于 2014-06-25T15:16:58.747 回答
9

A longer but more robust version of the accepted answer:

perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt

This expands all instances of $VAR or ${VAR} to their environment values (or, if they're undefined, the empty string).

It properly escapes backslashes, and accepts a backslash-escaped $ to inhibit substitution (unlike envsubst, which, it turns out, doesn't do this).

So, if your environment is:

FOO=bar
BAZ=kenny
TARGET=backslashes
NOPE=engi

and your template is:

Two ${TARGET} walk into a \\$FOO. \\\\
\\\$FOO says, "Delete C:\\Windows\\System32, it's a virus."
$BAZ replies, "\${NOPE}s."

the result would be:

Two backslashes walk into a \bar. \\
\$FOO says, "Delete C:\Windows\System32, it's a virus."
kenny replies, "${NOPE}s."

If you only want to escape backslashes before $ (you could write "C:\Windows\System32" in a template unchanged), use this slightly-modified version:

perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt
于 2014-07-29T15:28:49.140 回答
6

Here's another pure bash solution:

  • it's using heredoc, so:
    • complexity doesn't increase because of additionaly required syntax
    • template can include bash code
      • that also allows you to indent stuff properly. See below.
  • it doesn't use eval, so:
    • no problems with the rendering of trailing empty lines
    • no problems with quotes in the template

$ cat code

#!/bin/bash
LISTING=$( ls )

cat_template() {
  echo "cat << EOT"
  cat "$1"
  echo EOT
}

cat_template template | LISTING="$LISTING" bash

$ cat template (with trailing newlines and double quotes)

<html>
  <head>
  </head>
  <body> 
    <p>"directory listing"
      <pre>
$( echo "$LISTING" | sed 's/^/        /' )
      <pre>
    </p>
  </body>
</html>

output

<html>
  <head>
  </head>
  <body> 
    <p>"directory listing"
      <pre>
        code
        template
      <pre>
    </p>
  </body>
</html>
于 2017-02-18T11:52:16.650 回答
6

Here is another solution: generate a bash script with all the variables and the contents of the template file, that script would look like this:

word=dog           
i=1                
cat << EOF         
the number is ${i} 
the word is ${word}

EOF

If we feed this script into bash it would produce the desired output:

the number is 1
the word is dog

Here is how to generate that script and feed that script into bash:

(
    # Variables
    echo word=dog
    echo i=1

    # add the template
    echo "cat << EOF"
    cat template.txt
    echo EOF
) | bash

Discussion

  • The parentheses opens a sub shell, its purpose is to group together all the output generated
  • Within the sub shell, we generate all the variable declarations
  • Also in the sub shell, we generate the cat command with HEREDOC
  • Finally, we feed the sub shell output to bash and produce the desired output

  • If you want to redirect this output into a file, replace the last line with:

    ) | bash > output.txt
于 2018-04-04T15:00:02.453 回答
5

Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes:

#!/bin/bash
regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}'
while read line; do
    while [[ "$line" =~ $regex ]]; do
        param="${BASH_REMATCH[1]}"
        line=${line//${BASH_REMATCH[0]}/${!param}}
    done
    echo $line
done
于 2014-10-22T14:17:09.630 回答
5

If using Perl is an option and you're content with basing expansions on environment variables only (as opposed to all shell variables), consider Stuart P. Bentley's robust answer.

This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use.

The goals are:

  • Support expansion of both ${name} and $name variable references.
  • Prevent all other expansions:
    • command substitutions ($(...) and legacy syntax `...`)
    • arithmetic substitutions ($((...)) and legacy syntax $[...]).
  • Allow selective suppression of variable expansion by prefixing with \ (\${name}).
  • Preserve special chars. in the input, notably " and \ instances.
  • Allow input either via arguments or via stdin.

Function expandVars():

expandVars() {
  local txtToEval=$* txtToEvalEscaped
  # If no arguments were passed, process stdin input.
  (( $# == 0 )) && IFS= read -r -d '' txtToEval
  # Disable command substitutions and arithmetic expansions to prevent execution
  # of arbitrary commands.
  # Note that selectively allowing $((...)) or $[...] to enable arithmetic
  # expressions is NOT safe, because command substitutions could be embedded in them.
  # If you fully trust or control the input, you can remove the `tr` calls below
  IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3')
  # Pass the string to `eval`, escaping embedded double quotes first.
  # `printf %s` ensures that the string is printed without interpretation
  # (after processing by by bash).
  # The `tr` command reconverts the previously escaped chars. back to their
  # literal original.
  eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`(['
}

Examples:

$ expandVars '\$HOME="$HOME"; `date` and $(ls)'
$HOME="/home/jdoe"; `date` and $(ls)  # only $HOME was expanded

$ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars
$SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded
  • For performance reasons, the function reads stdin input all at once into memory, but it's easy to adapt the function to a line-by-line approach.
  • Also supports non-basic variable expansions such as ${HOME:0:10}, as long as they contain no embedded command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
    • Such embedded substitutions actually BREAK the function (because all $( and ` instances are blindly escaped).
    • Similarly, malformed variable references such as ${HOME (missing closing }) BREAK the function.
  • Due to bash's handling of double-quoted strings, backslashes are handled as follows:
    • \$name prevents expansion.
    • A single \ not followed by $ is preserved as is.
    • If you want to represent multiple adjacent \ instances, you must double them; e.g.:
      • \\ -> \ - the same as just \
      • \\\\ -> \\
    • The input mustn't contain the following (rarely used) characters, which are used for internal purposes: 0x1, 0x2, 0x3.
  • There's a largely hypothetical concern that if bash should introduce new expansion syntax, this function might not prevent such expansions - see below for a solution that doesn't use eval.

If you're looking for a more restrictive solution that only supports ${name} expansions - i.e., with mandatory curly braces, ignoring $name references - see this answer of mine.

Here is an improved version of the bash-only, eval-free solution from the accepted answer:

The improvements are:

  • Support for expansion of both ${name} and $name variable references.
  • Support for \-escaping variable references that shouldn't be expanded.
  • Unlike the eval-based solution above,
    • non-basic expansions are ignored
    • malformed variable references are ignored (they don't break the script)
 IFS= read -d '' -r lines # read all input from stdin at once
 end_offset=${#lines}
 while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do
      pre=${BASH_REMATCH[1]} # everything before the var. reference
      post=${BASH_REMATCH[5]}${lines:end_offset} # everything after
      # extract the var. name; it's in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise
      [[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]}
      # Is the var ref. escaped, i.e., prefixed with an odd number of backslashes?
      if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then
           : # no change to $lines, leave escaped var. ref. untouched
      else # replace the variable reference with the variable's value using indirect expansion
           lines=${pre}${!varName}${post}
      fi
      end_offset=${#pre}
 done
 printf %s "$lines"
于 2015-03-25T23:32:52.490 回答
4

Perfect case for shtpl. (project of mine, so it is not widely in use and lacks in documentation. But here is the solution it offers anyhow. May you want to test it.)

Just execute: 

$ i=1 word=dog sh -c "$( shtpl template.txt )"

Result is:

the number is 1
the word is dog

Have fun.

于 2013-03-03T04:42:19.303 回答
3

This page describes an answer with awk

awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt
于 2012-03-06T19:39:52.680 回答
3 
# Usage: template your_file.conf.template > your_file.conf
template() {
        local IFS line
        while IFS=$'\n\r' read -r line ; do
                line=${line//\\/\\\\}         # escape backslashes
                line=${line//\"/\\\"}         # escape "
                line=${line//\`/\\\`}         # escape `
                line=${line//\$/\\\$}         # escape $
                line=${line//\\\${/\${}       # de-escape ${         - allows variable substitution: ${var} ${var:-default_value} etc
                # to allow arithmetic expansion or command substitution uncomment one of following lines:
#               line=${line//\\\$\(/\$\(}     # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE
#               line=${line//\\\$\(\(/\$\(\(} # de-escape $((        - allows $(( 1 + 2 ))
                eval "echo \"${line}\"";
        done < "$1"
}

This is the pure bash function adjustable to your liking, used in production and should not break on any input. If it breaks - let me know.

于 2017-10-12T18:50:21.320 回答
3

To follow up on plockc's answer on this page, here is a dash-suitable version, for those of you looking to avoid bashisms.

eval "cat <<EOF >outputfile
$( cat template.in )
EOF
" 2> /dev/null
于 2020-03-23T20:03:14.853 回答
1

You can also use bashible (which internally uses the evaluating approach described above/below).

There is an example, how to generate a HTML from multiple parts:

https://github.com/mig1984/bashible/tree/master/examples/templates

于 2015-09-03T13:30:47.750 回答
1

Look at simple variables substitution python script here: https://github.com/jeckep/vsubst

It is very simple to use:

python subst.py --props secure.properties --src_path ./templates --dst_path ./dist
于 2019-06-19T10:01:40.823 回答
0

Here's a bash function that preserves whitespace:

# Render a file in bash, i.e. expand environment variables. Preserves whitespace.
function render_file () {
    while IFS='' read line; do
        eval echo \""${line}"\"
    done < "${1}"
}
于 2016-01-01T03:27:00.857 回答
0

Here's a modified perl script based on a few of the other answers:

perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template

Features (based on my needs, but should be easy to modify):

  • Skips escaped parameter expansions (e.g. \${VAR}).
  • Supports parameter expansions of the form ${VAR}, but not $VAR.
  • Replaces ${VAR} with a blank string if there is no VAR envar.
  • Only supports a-z, A-Z, 0-9 and underscore characters in the name (excluding digits in the first position).
于 2016-12-22T02:02:54.540 回答