[rkennedy]

The sh.Exec family of functions used to use os.Expand to expand environment variables in command arguments, but that function offers no escape syntax — no way to avoid expanding dollar signs that the caller really wants to include on the command line. This commit implements a new, mostly compatible Expand function to allow escaping the environment-variable syntax, plus an accompanying Escape function to producing escaped strings.

This technically breaks backward compatibility; any existing code that has used the sequences "\\" or "\$" in its command arguments will need to be updated. Although the compatibility argument prevents os.Expand from changing, documentation for sh.Exec hasn't actually stated that it follows all the same rules as os.Expand, so it's possible to view this change as merely a bugfix, formalizing a new syntax that had not previously been established.

Fixes #434

[natefinch]

Thanks for tackling literal dollar signs in command arguments. I applied this patch to current master (0953947) to review it. I found two compatibility regressions that need addressing before merging; the inline comments include reproductions. The Escape round-trip worked in the cases I exercised, but replacing os.Expand also changes behavior for existing arguments unrelated to dollar escaping.

The updated patch also needs a small lint cleanup: use strings.ReplaceAll, fix the spelling of preceding, and use t.Setenv in the precedence test.

[natefinch]

Ordinary backslashes corrupt existing command arguments. The regexp matches every backslash followed by a character, and this branch drops both characters unless they are an escaped backslash or dollar sign. For example, Expand(C:\tools\mage, mapping) returns C:oolsage, and Expand(\d+\s, mapping) returns +; both inputs are preserved by os.Expand. This affects Windows paths and regex arguments even when they contain no environment-variable syntax. Please preserve unrecognized backslash sequences rather than reserving and deleting them, and add regression coverage for these cases. The compatibility impact goes beyond the \ and $ sequences mentioned in the PR description.

[rkennedy]

My intention was that code that didn't want backslash-processing should use sh.Escape.

Furthermore, my position was that it's impossible to have a true compatibility impact on behavior that was never documented in the first place. Since no backslash behavior was specified — only a vague mention of "$FOO format" — we can't actually break anything.

I realize that the documentation I wrote here for sh.Expand says that it "works just like os.Expand," but I think I meant that only in the sense that it searches for values looking like $var or ${var}, without really getting into the specific syntax.

But anyway, updated commits now match the syntax of os.Expand, with the only change being the new handling of \\ and \$ escape sequences.

[natefinch]

The new variable grammar is not compatible with os.Expand. Braced names in os.Expand are not limited to word characters. For example, with mapping := func(key string) string { return "<" + key + ">" }, os.Expand("${user.bitbucketAppPassword}", mapping) returns "<user.bitbucketAppPassword>", while this Expand returns "user.bitbucketAppPassword}" without calling the mapping for that name. ${FOO-BAR} is similarly affected. Also, $10 previously expands $1 followed by the literal 0, but this regexp treats 10 as one variable name. Please retain os.Expand's existing variable-name and special-parameter semantics outside the deliberately introduced escapes, with tests for these inputs.

[rkennedy]

As I mentioned on the other comment, it was never my intention to exactly match the syntax of os.Expand, mainly because its syntax isn't documented, but also because its behavior is a little weird, if you approach it expecting it to work like shell parameter expansion. It accepts syntax for variable names that shells like Bash don't accept, such as ${user.bitbucketAppPassword} and ${FOO-BAR}.

Should be fixed now, though.