Incorrect quoting of command-line arguments for all programs in Perl\exec

Hi,

It seems all executables under Perl\exec have their command-line arguments interpreted by the system even if they are quoted. For instance, any of the following commands:
podchecker "a<b"
shasum "a<b"
exiftool -ver "-testName<DateTimeOriginal"
result in the following output, and no further processing:
The system cannot find the file specified.

This is a serious problem because it prevents passing the very common "-destination_tag<source_tag" construct to exiftool, to copy a source_tag to a destination_tag. The exiftool program isn’t called at all.

Bypassing the “exec” version of the program, like this:
C:\Perl\site\bin\exiftool -ver "-testName<DateTimeOriginal"
results in this message from exiftool, as expected:
12.76
No file specified

To be clear, this isn’t an exiftool problem, because all programs in the Perl\exec directory have the same problem. The behavior (and error message) is the same as if the quotes were removed:
exiftool -ver -testName<DateTimeOriginal (note the missing quotes)
yields:
The system cannot find the file specified.

Hypothesis:
Given that all programs under Perl\exec are in fact the same binary (bit-for-bit), my guess is this “surrogate” executable sets up a number of environment variables, reads the command line arguments, looks at ARGV0 to find its name, and calls the program of the same name down the path. But, when reading the command line arguments, the surrogate tries to interpret them, or it simply drops the quotes before calling the final executable. In any case, it’s a show stopper.

Any suggestions? Should I remove all programs from Perl\exec, except for the core Perl executables? By core Perl executables, I mean perl.exe, perl5.36.3.exe, runperl.exe, wperl.exe, par.exe, etc.

Thanks,

Martin

Hi @gamin-mb ,

That’s a very strange issue. The quotes being lost shouldn’t really be an issue, since we don’t run the underlying executable from a shell environment. We’re just passing the args as they are received by our executable onto the actual executable you’re trying to invoke. For what it’s worth, and since you seem technically inclined, this is where the “magic” happens:

I did a quick test to verify, and indeed it seems to work as expected.

I wonder if there might be some other issue with your runtime or your local checkout of it. Does state exec which exiftool return you a valid executable path?

Note state exec essentially runs whatever you give it within the context of your runtime. So you could also use it to run eg. state exec -- exiftool -ver "-testName<DateTimeOriginal".

Could you also share your State Tool version? state --version.

Hi Nathan,

Here’s the requested information, and some more for context:

C:\Users\root>state exec which exiftool
Operating on project gamin-mb/Perl-5.36-Windows, located at C:\Users\root\Perl-5.36-Windows.

/cygdrive/c/Perl/site/bin/exiftool

Obviously, I have Cygwin installed, because the “which” command is not available on Windows. This explains the “Linux” path format. Here’s the origin of my “which” command, and below is the output of the equivalent Windows command, “where”:

C:\Users\root>where which
C:\cygwin64\bin\which.exe

C:\Users\root>where exiftool
C:\Perl\exec\exiftool.exe
C:\Perl\site\bin\exiftool
C:\Perl\site\bin\exiftool.bat
C:\Users\root\AppData\Local\activestate\cache\bin\exiftool.exe

C:\Users\root>state exec -- exiftool -ver "-testName<DateTimeOriginal"
Operating on project gamin-mb/Perl-5.36-Windows, located at C:\Users\root\Perl-5.36-Windows.

The system cannot find the file specified.

C:\Users\root>state --version
ActiveState CLI by ActiveState Software Inc.
License BSD 3
Version 0.42.0-SHAcbce75e
Revision cbce75e6c093fd3195cbe18e6ed0c26a853459b4
Branch release
Built Mon Nov 27 2023 17:56:19 +0000 UTC

C:\Users\root>exiftool -ver "-testName<DateTimeOriginal"
The system cannot find the file specified.

C:\Users\root>C:\Perl\exec\exiftool.exe -ver "-testName<DateTimeOriginal"
The system cannot find the file specified.

C:\Users\root>C:\Users\root\AppData\Local\activestate\cache\bin\exiftool.exe -ver "-testName<DateTimeOriginal"
The system cannot find the file specified.

C:\Users\root>C:\Perl\site\bin\exiftool -ver "-testName<DateTimeOriginal"
12.76
No file specified

C:\Users\root>C:\Perl\site\bin\exiftool -ver -testName<DateTimeOriginal
The system cannot find the file specified.

So, of all the available exiftool executables, only the ones in site\bin runs properly; none of the other versions seem to even be invoked. We also get the error with the site\bin executable if the quotes are missing. I think the error message doesn’t come from Perl, because it’s a known Windows error string, but not a common Perl error string.

About the state tool:

All your questions involving the state tool puzzle me. Only the admin user (root) has the state tool installed on my computer, and I switch to a non-admin user for development. Consequently, state shouldn’t be a factor in deciding which executable runs. Indeed, I get the same results with both the admin and the non-admin users (except that invoking state as you asked fails for the non-admin user).

By the way, state update answers with “Update not found for release”, and I updated in early 2024, so I’m sure I had the latest version before my last state pull.

My personal assumptions and analysis:

It seems this is programmed in Go, which I don’t know. But I’m pretty sure the magic actually happens one line above and a few lines below:

10:        userArgs := os.Args[1:]
17:        if err := cmd.Run(); err != nil {

I wouldn’t be surprised if os.Args[] had no quotes left, and if cmd.Run() passed its then-unquoted arguments to some shell or system call which then interpreted the < as a standard input redirection, trying to read the file DateTimeOriginal. It would explain why “The system cannot find the file specified”.

On Windows, the Perl system() function sends its arguments to a shell, unless you carefully manage the argument list, so perhaps the Go Run() function does something similar.

Indeed, if I create an empty file named ‘DateTimeOriginal’ in the current directory, the behavior is consistent with the system looking for and finding a file named DateTimeOriginal (as if there were no quotes), as in this example:

C:\Users\root>dir /B DateTimeOriginal
DateTimeOriginal

C:\Users\root>exiftool -ver "-testName<DateTimeOriginal"
12.76
No file specified

I am really curious to understand why you don’t have the problem.

Just to be sure it’s not some interaction with Cygwin, I renamed my top C:\cygwin64 directory, to make it inaccessible; this didn’t change the behavior of the programs under Perl\exec.

For completeness: if I rename Perl\site\bin\exiftool.bat to something else, then I cannot run exiftool at all:

C:\Users\root>exiftool -ver
state-exec: run failed: cannot run command: command “c:\perl\site\bin\exiftool.bat” failed: exec: “c:\perl\site\bin\exiftool.bat”: file does not exist
state-exec: Not user serviceable; Please contact support for assistance.

This indicates that C:\Perl\exec\exiftool.exe wants to call C:\Perl\site\bin\exiftool.bat, but will not call C:\Perl\site\bin\exiftool (that’s probably a Windows PathExt thing). If I restore Perl\site\bin\exiftool.bat and call it directly, I get this:

C:\Users\root>C:\Perl\site\bin\exiftool.bat -ver "-testName<DateTimeOriginal"
12.76
No file specified

So the problem isn’t specifically with the batch file; it works if called directly, but not if C:\Perl\exec\exiftool.exe invokes it.

That’s as far as I can go. I hope this helps.

Thanks,

Martin

Hey @gamin-mb, so I’m a little confused as to why you think ActiveState is involved at all here. From what you’ve shared you’re running an exiftool that doesn’t come from us.

C:\Users\root>where exiftool
C:\Perl\exec\exiftool.exe
C:\Perl\site\bin\exiftool
C:\Perl\site\bin\exiftool.bat
C:\Users\root\AppData\Local\activestate\cache\bin\exiftool.exe

Only the last path here comes from ActiveState, unless you have an older installer that installs to C:\Perl? New installs with the State Tool do not use this path. If indeed you used an older installer, can you share how you installed this version of Perl?

Note state exec will essentially set up your runtime environment and then run the executable you passed it, but that environment will also include your global system environment, which contains exiftool. So in this case when you run state exec exiftool .. it’s really just running the same one as without state exec ...

Hi Nathan,

If you find the paths misleading, perhaps you didn’t realize that I chose my own --runtime-path? Let me go into details…

  1. I did request and install exiftool from ActiveState (the 4th package in the list):
C:\Users\root>state packages
█ Listing Packages

Operating on project gamin-mb/Perl-5.36-Windows, located at C:\Users\root\Perl-5.36-Windows.

  Name                   Version
──────────────────────────────────
  Excel-Writer-XLSX      Auto
  File-chdir             Auto
  File-Copy-Recursive    Auto
  Image-ExifTool         Auto
  PAR-Packer             Auto
  PadWalker              Auto
  Pod-Simple-Wiki        Auto
  Spreadsheet-Read       Auto
  Term-ProgressBar       Auto
  Test-Cmd               Auto
  Text-CSV_XS            Auto
  Win32-API              Auto
  Win32-Console          Auto
  Win32-File             Auto
  XML-LibXML             Auto

In addition, there are references to “exiftool.exe” and “c:\perl\site\bin\exiftool” in C:\Perl\exec\meta.as and in files under C:\Users\root\AppData\Local\ActiveState\cache\artifacts, which are files or folders created by ActiveState.

  1. I installed Perl as instructed by your support team, as discussed here: How do I get rid of extra runtimes?. More specifically, I did the following, in this order, on January 23, 2024:
I deleted C:\Perl
state clean uninstall --all
I deleted C:\Users\root\Perl-5.36.0-Windows
I downloaded state-remote-installer.exe from ActiveState
state-remote-installer.exe
state checkout gamin-mb/Perl-5.36-Windows --runtime-path C:/Perl
state use Perl-5.36-Windows

I also performed a few state pull; on January 25, February 16, and February 26 or 27.

  1. As I pointed out at the beginning of my first message, the error occurs not only with exiftool but with every program installed by the state tool in C:\Perl\exec or in …\AppData\Local\activestate\cache\bin. None of these programs seem to be allowed to run when passed a command-line argument containing an input redirection character (<), even if the character is quoted. See for instance what happens with as basic a command as this:

C:\Users\root>C:\Users\root\AppData\Local\activestate\cache\bin\pod2text.exe "a<b"
The system cannot find the file specified.

To simplify:

Let’s eliminate all these variables, and use a simple script that we can share: put the following line in a file called testarg.txt:
printf "Argument: '%s'.\n", $ARGV[0];
This script should print the first command-line argument and exit. Note that I chose to give the script a .txt extension, and I call perl explicitly on it to prevent Windows from choosing some command from the registry that might quote (or not) its arguments.

This is the output I get. The first one is expected, the second one is an error, the third one is an error and it also created a file named “b”, whose contents is “Argument: ‘a’.”:

C:\Users\root>C:\Users\root\AppData\Local\activestate\cache\bin\perl.exe testarg.txt “a<b”
Argument: ‘a<b’.

C:\Users\root>state exec perl testarg.txt “a<b”
Operating on project gamin-mb/Perl-5.36-Windows, located at C:\Users\root\Perl-5.36-Windows.

The system cannot find the file specified.

C:\Users\root>state exec perl testarg.txt “a>b”
Operating on project gamin-mb/Perl-5.36-Windows, located at C:\Users\root\Perl-5.36-Windows.

The second and third cases are consistent with Windows interpreting the argument as if the quotes weren’t present, which gives me a strong suspicion that state-exec drops the quotes around its arguments. I guess most of the executables under Perl\exec are in fact renamed copies of state-exec, and therefore they inherit the problem.

When you run my testarg.txt under the same conditions, what do you get? If it works for you, then I’ll scratch my head to find another explanation.

Thanks,

Martin

Thanks so much for going through the trouble of providing a test case, that is incredibly helpful! I was able to reproduce your issue. I’ve filed a bug for us to address this. We’ll likely get this fixed in version 0.44.0, which is due by the end of April. We’ll do some investigating and if the severity of this is high enough we might even get it into the next version which is due end of this month. If you like I can share an early beta version with you here.

For now all I can offer as a workaround is to use the underlying executable directly. You can obtain this by running:

state exec where exiftool

Though I’m pretty sure you already figured this out going by your previous messages.

Really the only downside of doing this is that it does not set up your environment for you. But that’s kind of how it is normally anyway, the fact that we set up your environment for you as the executable is ran is a quality of life feature that’s sadly just missbehaving.

Apologies for the inconvenience. And thanks for sticking with me!

1 Like

Well, thanks for sticking with me too.

I find I’m getting very good and timely support from ActiveState, especially given that I’m not paying for it. The least I can do is to be precise and helpful (and sometimes insistent :wink:).

Sure, I’ll be happy to test a beta, provided I can revert to a stable release if the beta is a show stopper.

Martin

1 Like