Incorrect quoting of command-line arguments for all programs in Perl\exec

Hey @gamin-mb, just wanted to give you a quick update on the progress around this bug.

We found that the issue you had found is at least in part due to how Go and Windows handle passing arguments between processes. This may point to a bug in Go itself, for which a bug has been filed; syscall/exec_windows.go: appendEscapeArg does not escape all necessary characters · Issue #68313 · golang/go · GitHub

We’ve also working on a workaround for this bug for version v0.45.0 of the State Tool, due by the end of this month.

Hi Nathan,

Thanks for the followup.

I looked at your bug report and I’d say you’re on the right track by noticing some characters aren’t escaped as they should be.

Personally, I’d try to avoid the problem in the first place, as it seems the problem is created by the following call:

exec.Command("C:\\Windows\\System32\\cmd.exe", .....

This explicitly invokes a shell to run test.bat. It’s probably this shell that complains it “cannot find the file specified” because one of the roles of a shell is to interpret some characters (“<” in this case) to invoke system calls (define standard input in this case). This is what I meant by my second suggestion in a previous post: to use a different command to run the batch script, one that wouldn’t parse the arguments before invoking the batch script. I’m not so familiar with this aspect of Windows, but the first paragraph of this Perl documentation https://perldoc.perl.org/functions/system explains what I mean by avoiding using the shell (I learned the hard way because I once had the same problem with calling commands from Perl on Windows).

The downside of escaping characters is different shells use different characters; on systems that don’t use cmd.exe (Linux, MacOS), you’d need to be mindful of this.

However, I don’t know Go at all, and I don’t know the State code, so I may be totally wrong, in which case I apologize. But we’re making progress!

Martin

@gamin-mb for executors we do not run through cmd.exe, we instead invoke the bat file directly from Go. Unfortunately the problem with passing arguments exists in both these scenario’s.

End of the day the issue is due to Windows not handling progress arguments itself, it defers this entirely to the process being ran. Which Go and I imagine any other cross platform language handles by converting from/to an encoded argument string to a slice of strings, and this translation logic is unfortunately error-prone.

Basically it works fine if the process we’re trying to run assumes a slice of arguments, but if it expected an encoded argument string then the translation layer could introduce inconsistencies between what you type and what the process receives. Ideally I’d love to just get the exact input the user gave and forward that, but I’ve so far not found any way of doing that.

1 Like

Nathan,

I think I’m starting to understand better what you mean; you are talking about argument splitting (and using quotes to control that), while I was putting wildcard expansion and argument splitting in the same bucket, which was wrong. Both are part of argument processing, but they can be done in different places. Sorry, my fault.

For what it’s worth, the following blog says where you could get the raw command line (check the /link noarg.obj and _acmdln discussion near the end). But I suppose you’re already be aware of this…
Argument processing in Unix and Windows

Things are not as rosy on Windows. They are… fascinating or, better said, horrifying

Haha… sounds about right :sweat_smile:

That’s a very low level solution, and one that depends on C. I’m sure there’s a way to do the same in Go but at a certain point I start asking the question of “is it worth it” and given the current use-cases we’ve found (and solved) I wouldn’t say that it is, I think all things considered translating it both ways is a cleaner solution if the alternative is to have such low level code for a solved use-case.

Anyway, your issue will soon be solved in v0.45, at this point it’s more about how the sausage is made… :stuck_out_tongue:

1 Like

@gamin-mb v0.45.0 has been released, hopefully this addresses all your issues.

1 Like

Hi Nathan,

I just installed the latest state (v0.45) , and updated all my packages at the same time (but still using Perl 5.36.3, not yet 5.38). It all works!

Sorry for taking so long to test and respond; I was in the middle of something I couldn’t risk delaying.

I also quickly tested libXML and pp -M PAR, because these are tricky as well, and they also seem to work.

Thanks a lot!

-Martin

1 Like

Awesome, we got there eventually. Thanks for sticking with me through all the back and forth @gamin-mb !

1 Like