MSYS bash: maximum length of command line arguments

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

MSYS bash: maximum length of command line arguments

Axel Holzinger
Hi,

I'm a newbie on this list (and in MSYS still I'm afraid) , so please
forgive me, if this question was already answered. I did search the
archive, but I didn't find the answer.

On compiling FFmpeg's libavcodec from FFmpeg's main make (calling all
the subdirectories' make) I get the error "command line too long" and
obviously the command line is longer than 8192 characters/bytes. (For
the interested: the error happens at the stage of calling the
archiver, as in the command line every object file is referenced by
its relative path WITH the libavcodec subdirectory).

Now my question is how and why is bash.exe (renamed to sh.exe in MSYS)
relying on Windows' cmd.exe which's limit is 8192 characters/bytes
(resp. 8191 if we trust MSDN)?

bash.exe is a process itself or does it somehow call cmd.exe?

Is there a way to increase bash's maximum command line length (i.e. by
increasing a #define in the sources/headers)? Or if bash is relying on
cmd.ex, is it possible to use a different Windows shell that offers a
longer command line?

In some configure scripts there is a check for command line length
included ("checking the maximum length of command line arguments"). If
somebody has the code/script that I could use (by simply typing
./chkmaxcmdline or something like that) I would be very thankful.

Any hint would be highly appreciated.

Best regards and thank you
Axel


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command line arguments

Ralf Wildenhues
Hello Axel,

* Axel Holzinger wrote on Tue, Sep 07, 2010 at 10:47:14AM CEST:
> On compiling FFmpeg's libavcodec from FFmpeg's main make (calling all
> the subdirectories' make) I get the error "command line too long" and
> obviously the command line is longer than 8192 characters/bytes. (For
> the interested: the error happens at the stage of calling the
> archiver, as in the command line every object file is referenced by
> its relative path WITH the libavcodec subdirectory).

Please supply a link to the tarball you are building.

> In some configure scripts there is a check for command line length
> included ("checking the maximum length of command line arguments").

That check comes from Libtool macros.  Is libtool used for linking here?
It has means to avoid the command-line length limit (either by calling
the archiver repeatedly, or, with newer libtool, also using @file
support).  If that's broken, we'd like to know.

If the package in question doesn't use libtool, well, it could of course
implement either of those workarounds as well.

Thanks,
Ralf

------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Axel Holzinger
Hello Ralf et al,

On Tue, Sep 07, 2010 at 10:41 PM Ralf Wildenhues wrote:

> Hello Axel,
>
> * Axel Holzinger wrote on Tue, Sep 07, 2010 at 10:47:14AM CEST:
> > On compiling FFmpeg's libavcodec from FFmpeg's main make
> (calling all
> > the subdirectories' make) I get the error "command line too
> long" and
> > obviously the command line is longer than 8192
> characters/bytes. (For
> > the interested: the error happens at the stage of calling the
> > archiver, as in the command line every object file is referenced
by
> > its relative path WITH the libavcodec subdirectory).
>
> Please supply a link to the tarball you are building.

I'm buildung from SVN trunk, but I don't think there is an error in
the build/configuration.
svn://svn.ffmpeg.org/ffmpeg/trunk

> > In some configure scripts there is a check for command line length

> > included ("checking the maximum length of command line
arguments").
>
> That check comes from Libtool macros.  Is libtool used for
> linking here?

I don't think so.

> It has means to avoid the command-line length limit (either
> by calling the archiver repeatedly, or, with newer libtool,
> also using @file support).  If that's broken, we'd like to know.
>
> If the package in question doesn't use libtool, well, it
> could of course implement either of those workarounds as well.

I think I found the reason for MinGW/MSys bash having the 8192
characters limit. I think MinGW/MSys bash is compiled as a Windows
console application which brings up Windows' cmd.exe code which does
the Window handling and keyboard/mouse input. And this part is known
to have the 8192 characters input.

To gain the 32768 characters commandline that NT/XP/Vista/7 offers via
its CreateProcess function MinGW/MSys bash would need to be a native
Win32 application instead of a console application.

But this is still a wild guess. Somebody listening who knows how it
is?

Thank you
Axel


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Martin Mocko


On Thu, Sep 9, 2010 at 3:53 PM, Axel Holzinger <[hidden email]> wrote:
I think I found the reason for MinGW/MSys bash having the 8192
characters limit. I think MinGW/MSys bash is compiled as a Windows
console application which brings up Windows' cmd.exe code which does
the Window handling and keyboard/mouse input. And this part is known
to have the 8192 characters input.

I doubt console application have something to do with cmd.exe shell. I'd say OS just creates console (as in http://msdn.microsoft.com/en-us/library/ms682073(VS.85).aspx) for std handles when process can't inherit one from parent.
 

To gain the 32768 characters commandline that NT/XP/Vista/7 offers via
its CreateProcess function MinGW/MSys bash would need to be a native
Win32 application instead of a console application.

Nitpick warning: "native win32 application" is not correct term here. It is obvious what you mean ("GUI win32 application", the subsystem is called "GUI"). "Native" subsystem exists too and it is something different.
 

------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Ralf Wildenhues
In reply to this post by Axel Holzinger
* Axel Holzinger wrote on Thu, Sep 09, 2010 at 03:53:41PM CEST:

> On Tue, Sep 07, 2010 at 10:41 PM Ralf Wildenhues wrote:
> > * Axel Holzinger wrote on Tue, Sep 07, 2010 at 10:47:14AM CEST:
> > > On compiling FFmpeg's libavcodec from FFmpeg's main make (calling
> > > all the subdirectories' make) I get the error "command line too
> > > long" and obviously the command line is longer than 8192
> > > characters/bytes. (For the interested: the error happens at the
> > > stage of calling the archiver, as in the command line every object
> > > file is referenced by its relative path WITH the libavcodec
> > > subdirectory).
> >
> > Please supply a link to the tarball you are building.
>
> I'm buildung from SVN trunk, but I don't think there is an error in
> the build/configuration.
> svn://svn.ffmpeg.org/ffmpeg/trunk

I suggest you file a bug with ffmpeg.  You can suggest to them the
workarounds of creating the archive step by step, or using a response
file @file with ar.

Cheers,
Ralf

------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Axel Holzinger
In reply to this post by Martin Mocko
Hi Martin et al,

On Thu, Sep 9, 2010 at 7:12 PM, Martin Mocko wrote:
> On Thu, Sep 9, 2010 at 3:53 PM, Axel Holzinger
> <[hidden email]> wrote:
>
>
> I think I found the reason for MinGW/MSys bash having the 8192
> characters limit. I think MinGW/MSys bash is compiled
>     as a Windows console application which brings up Windows'
>     cmd.exe code which does the Window handling and keyboard/mouse
>     input. And this part is known to have the 8192 characters
input.
>
> I doubt console application have something to do with cmd.exe
> shell. I'd say OS just creates console (as in
> http://msdn.microsoft.com/en-us/library/ms682073(VS.85).aspx)
> for std handles when process can't inherit one from parent.

Hm, your link just leads me to a list of "Console Functions"-

I don't want to insist on cmd.exe being involved. Fact is that there
is code involved that doesn't come from bash sourcecode and makes all
the window drawing and - and this is just my opinion, no proove yet -
is responsible for the 8K commandline limit.

> To gain the 32768 characters commandline that
>     NT/XP/Vista/7 offers via its CreateProcess function
>     MinGW/MSys bash would need to be a native
> Win32 application instead of a console application.
>
> Nitpick warning: "native win32 application" is not correct
> term here. It is obvious what you mean ("GUI win32
> application", the subsystem is called "GUI"). "Native"
> subsystem exists too and it is something different.

Also nitpicking: What you mean is native NT. Win32 is already a
subsystem, like POSIX. I just meant a Win32 process without the
/console parameter that does all the windowing and keyboard input on
ist own. If you like, which does the GUI part on ist own. A Win32
process running in the Win32 subsystem can be a process without any
window/GUI at all.

Cheers
Axel


------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:

Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Martin Mocko


On Thu, Sep 9, 2010 at 11:12 PM, Axel Holzinger <[hidden email]> wrote:
Hi Martin et al,

On Thu, Sep 9, 2010 at 7:12 PM, Martin Mocko wrote:
> On Thu, Sep 9, 2010 at 3:53 PM, Axel Holzinger
> <[hidden email]> wrote:
 > I doubt console application have something to do with cmd.exe
> shell. I'd say OS just creates console (as in
> http://msdn.microsoft.com/en-us/library/ms682073(VS.85).aspx)
> for std handles when process can't inherit one from parent.
I don't want to insist on cmd.exe being involved. Fact is that there
is code involved that doesn't come from bash sourcecode and makes all
the window drawing and - and this is just my opinion, no proove yet -
is responsible for the 8K commandline limit.

but does the console code have anything to do with passing parameters? I don't say it is impossible, but I can't really imagine how. How would fact that program's output goes to console (and not file, pipe, or whatever) affect passing command line argument (eg. in CreateProcess(), which is called outside console code)? I can't really imagine.

BTW, I haven't heard about windows console using bash code, can you point me to somewhere where I learn more on this?
 

>       To gain the 32768 characters commandline that
>     NT/XP/Vista/7 offers via its CreateProcess function
>     MinGW/MSys bash would need to be a native
>       Win32 application instead of a console application.
>
> Nitpick warning: "native win32 application" is not correct
> term here. It is obvious what you mean ("GUI win32
> application", the subsystem is called "GUI"). "Native"
> subsystem exists too and it is something different.

Also nitpicking: What you mean is native NT. Win32 is already a
subsystem, like POSIX.

Heh-heh, right, good one ;)
 
I just meant a Win32 process without the
/console parameter that does all the windowing and keyboard input on
ist own. If you like, which does the GUI part on ist own. A Win32
process running in the Win32 subsystem can be a process without any
window/GUI at all.

Well, it's not really extra flag in Win32-subsystem NT executables. There is field "Subsystem" in IMAGE_OPTIONAL_HEADER, which can be IMAGE_SUBSYSTEM_NATIVE, IMAGE_SUBSYSTEM_WINDOWS_GUI, IMAGE_SUBSYSTEM_WINDOWS_CUI (or some other). It was in this context I called console a "subsystem" on par with native. But of course I agree from programmer's point of view, it appears more like a extra flag than a separate subsystem.

------------------------------------------------------------------------------
Automate Storage Tiering Simply
Optimize IT performance and efficiency through flexible, powerful,
automated storage tiering capabilities. View this brief to learn how
you can reduce costs and improve performance.
http://p.sf.net/sfu/dell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Charles Wilson-8
In reply to this post by Axel Holzinger
On 9/9/2010 5:12 PM, Axel Holzinger wrote:
> Also nitpicking: What you mean is native NT. Win32 is already a
> subsystem, like POSIX. I just meant a Win32 process without the
> /console parameter that does all the windowing and keyboard input on
> ist own.

bash is never ever going to be a gui program.  It is a command processor
(a shell) and that is all.  It has no concept of drawing, or of putting
windows -- or even text -- on the screen.  ALL it does it accept data on
its stdin, and put data on its stdout/stderr.

The terminal handles everything else.  In this case, that "terminal" may
be the "console" (*), but it could also be "mintty" or "rxvt" -- both of
which are available for MSYS.

(*) The "console" is that black box we associate with cmd.exe -- but it
is NOT, in fact, cmd.exe.  Just like bash, cmd.exe is a command
processor.  The console is handled by the windows subsystem (csrss.dll,
IIRC, on pre-Vista; another component of windows handles the consoles in
Vista and later).

I suggest you download msys-mintty

https://sourceforge.net/projects/mingw/files_beta/MSYS/mintty/mintty-0.8.3/

unpack it in C:\msys\1.0\, and create a shortcut that will launch it and
tell it to start msys-bash (see the README after you install).

That way, cmd.exe AND the win32 console support are NOT involved at all.
 If you still have the 8K limit...then the problem lies elsewhere. If
you don't...well, then, problem solved.

--
Chuck


------------------------------------------------------------------------------
Automate Storage Tiering Simply
Optimize IT performance and efficiency through flexible, powerful,
automated storage tiering capabilities. View this brief to learn how
you can reduce costs and improve performance.
http://p.sf.net/sfu/dell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Tor Lillqvist
In reply to this post by Axel Holzinger
Instead of just talking, how hard is it to experiment? Try these two programs:

p1.c:

#include <windows.h>
#include <process.h>
#include <stdio.h>

#define N 30000

int
main (int argc, char **argv)
{
  char *cmdline = malloc (N);
  int k;
  int rc;
  STARTUPINFO si;
  PROCESS_INFORMATION pi;

  strcpy (cmdline, ".\\p2.exe ");
  k = strlen (cmdline);
  memset (cmdline + k, 'x', N - 100);
  cmdline[k + N - 100] = '\0';

  printf ("p1: Trying system()\n");
  rc = system (cmdline);
  printf ("p1: system returned %d\n", rc);

  printf ("p1: Trying spawnl()\n");
  rc = spawnl (_P_WAIT, ".\\p2.exe", "p2.exe", strchr (cmdline, ' '), NULL);
  printf ("p1: spawnl returned %d\n", rc);

  memset (&si, 0, sizeof (si));
  si.cb = sizeof (si);

  printf ("p1: Trying CreateProcess()\n");
  rc = CreateProcess (".\\p2.exe", cmdline, NULL, NULL, TRUE, 0, NULL,
NULL, &si, &pi);
  if (!rc)
    printf ("p1: CreateProcess failed: %lu\n", GetLastError ());
  else
    printf ("p1: CreateProcess success, process %lu\n", pi.dwProcessId);

  exit (0);
}


and p2.c:

#include <windows.h>
#include <stdio.h>

int
main (int argc, char **argv)
{
  int i;
  char *cmdline = GetCommandLine ();

  if (cmdline)
    printf ("p2: GetCommandLine returned string of %d chars\n", strlen
(cmdline));
  else
    printf ("p2: GetCommandLine returned NULL\n");

  printf ("p2: argc: %d\n", argc);
  for (i = 0; i < argc; i++)
    printf ("p2: argv[%d]: %5d chars: %.30s%s\n",
            i, strlen(argv[i]), argv[i], strlen (argv[i]) > 30 ? "..." : "");

  exit (0);
}

Compile to p1.exe and p2.exe. Run p1.exe. See what output you get from
the two processes. Experiment with different settings of the subsystem
flag for p1.exe and p2.exe. Experiment with starting as such from
cmd.exe, and from bash, both in a console window, rxvt, mintty, etc.
With output redirected by the shell (or cmd.exe) to a file or pipe or
not. Experiment with different settings of the dwInheritHandles
parameter to CreateProcess(), and/or the standard handle fields in the
STARTUPINFO. Etc.

(I think he conclusion will be be that system() is best avoided as it
involves cmd.exe, but the spawn* functions work fine, and
CreateProcess() obviously gives the ultimate control, especially if
you use the wide-char variant.)

--tml

------------------------------------------------------------------------------
Automate Storage Tiering Simply
Optimize IT performance and efficiency through flexible, powerful,
automated storage tiering capabilities. View this brief to learn how
you can reduce costs and improve performance.
http://p.sf.net/sfu/dell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Axel Holzinger
Hi Tor et al,

On Friday, September 10, 2010 12:12 AM Tor Lillqvist wrote:

> Instead of just talking, how hard is it to experiment? Try
> these two programs:
>
> p1.c:
>
> #include <windows.h>
> #include <process.h>
> #include <stdio.h>
>
> #define N 30000
>
> int
> main (int argc, char **argv)
> {
>   char *cmdline = malloc (N);
>   int k;
>   int rc;
>   STARTUPINFO si;
>   PROCESS_INFORMATION pi;
>
>   strcpy (cmdline, ".\\p2.exe ");
>   k = strlen (cmdline);
>   memset (cmdline + k, 'x', N - 100);
>   cmdline[k + N - 100] = '\0';
>
>   printf ("p1: Trying system()\n");
>   rc = system (cmdline);
>   printf ("p1: system returned %d\n", rc);
>
>   printf ("p1: Trying spawnl()\n");
>   rc = spawnl (_P_WAIT, ".\\p2.exe", "p2.exe", strchr
> (cmdline, ' '), NULL);
>   printf ("p1: spawnl returned %d\n", rc);
>
>   memset (&si, 0, sizeof (si));
>   si.cb = sizeof (si);
>
>   printf ("p1: Trying CreateProcess()\n");
>   rc = CreateProcess (".\\p2.exe", cmdline, NULL, NULL, TRUE,
> 0, NULL, NULL, &si, &pi);
>   if (!rc)
>     printf ("p1: CreateProcess failed: %lu\n", GetLastError ());
>   else
>     printf ("p1: CreateProcess success, process %lu\n",
> pi.dwProcessId);
>
>   exit (0);
> }
>
>
> and p2.c:
>
> #include <windows.h>
> #include <stdio.h>
>
> int
> main (int argc, char **argv)
> {
>   int i;
>   char *cmdline = GetCommandLine ();
>
>   if (cmdline)
>     printf ("p2: GetCommandLine returned string of %d
> chars\n", strlen (cmdline));
>   else
>     printf ("p2: GetCommandLine returned NULL\n");
>
>   printf ("p2: argc: %d\n", argc);
>   for (i = 0; i < argc; i++)
>     printf ("p2: argv[%d]: %5d chars: %.30s%s\n",
>             i, strlen(argv[i]), argv[i], strlen (argv[i]) >
> 30 ? "..." : "");
>
>   exit (0);
> }
>
> Compile to p1.exe and p2.exe. Run p1.exe. See what output you
> get from the two processes. Experiment with different
> settings of the subsystem flag for p1.exe and p2.exe.
> Experiment with starting as such from cmd.exe, and from bash,
> both in a console window, rxvt, mintty, etc.
> With output redirected by the shell (or cmd.exe) to a file or
> pipe or not. Experiment with different settings of the
> dwInheritHandles parameter to CreateProcess(), and/or the
> standard handle fields in the STARTUPINFO. Etc.
>
> (I think he conclusion will be be that system() is best
> avoided as it involves cmd.exe, but the spawn* functions work
> fine, and
> CreateProcess() obviously gives the ultimate control,
> especially if you use the wide-char variant.)
>
> --tml

Thank you all for your posts and support. I was off for the weekend.
That's the reason I could only now do further tests.

I did run p1 and p2 under all possible command processors and there is
only one doing it right: TCC LE
http://www.jpsoft.com/tccledes.htm
At the end of the message is the test results.

I've also used mintty to compile FFmpeg, but the result is the same,
the commandline is too long. Eventually I will look into the mintty
sources (I did already for a little bit, but there is some magic going
on with hidden console windows, etc.).

Cheers
Axe

Here's the complete findings:
compiled with gcc under MSys/MinGW
----------------------------------
Msys (non rxvt):
$ p1
p1: Trying system()
The commandline is too long.
p1: system returned 1
p1: Trying spawnl()
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 4432
p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

Msys rxvt:
$ p1
The commandline is too long.
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: Trying system()
p1: system returned 1
p1: Trying spawnl()
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 3124

user@pc ~/p
$ p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

MSys mintty:
$ p1
The commandline is too long.
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: Trying system()
p1: system returned 1
p1: Trying spawnl()
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 512

user@pc ~/p
$ p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

cmd.exe:
~\p>p1
p1: Trying system()
The commandline is too long.
p1: system returned 1
p1: Trying spawnl()
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 3784

~\p>p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

tcc.exe:
[C:\msys\home\user\p]p1
p1: Trying system()
p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: system returned 0
p1: Trying spawnl()
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 3488
p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...


compiled with msvc (VS 2008)
----------------------------
MSys (non rxvt):
$ p1
p1: Trying system()
The commandline is too long.
p1: system returned 1
p1: Trying spawnl()
p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 3080

user@pc /c/proj/p/Debug
$ p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

MSys rxvt:
$ p1
The commandline is too long.
p2: GetCommandLine returned string of 29908 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: Trying system()
p1: system returned 1
p1: Trying spawnl()
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 5872

user@pc ~/p
$ p2: GetCommandLine returned string of 29909 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

MSys mintty:
$ p1
The commandline is too long.
p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: Trying system()
p1: system returned 1
p1: Trying spawnl()
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 5664

user@pc /c/proj/p/Debug
$ p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

cmd.exe:
C:\proj\p\Debug>p1
p1: Trying system()
The commandline is too long.
p1: system returned 1
p1: Trying spawnl()
p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 2660

C:\proj\p\Debug>p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

tcc.exe:
p1: Trying system()
p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: system returned 0
p1: Trying spawnl()
p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     6 chars: p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
p1: spawnl returned 0
p1: Trying CreateProcess()
p1: CreateProcess success, process 6104

[C:\proj\p\Debug]p2: GetCommandLine returned string of 1 chars
p2: argc: 2
p2: argv[0]:     8 chars: .\p2.exe
p2: argv[1]: 29900 chars: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...


------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Andy Koppe
On 13 September 2010 14:09, Axel Holzinger wrote:

> Hi Tor et al,
>
> On Friday, September 10, 2010 12:12 AM Tor Lillqvist wrote:
>> Instead of just talking, how hard is it to experiment? Try
>> these two programs:
>>
>> p1.c:
>>
>> #include <windows.h>
>> #include <process.h>
>> #include <stdio.h>
>>
>> #define N 30000
>>
>> int
>> main (int argc, char **argv)
>> {
>>   char *cmdline = malloc (N);
>>   int k;
>>   int rc;
>>   STARTUPINFO si;
>>   PROCESS_INFORMATION pi;
>>
>>   strcpy (cmdline, ".\\p2.exe ");
>>   k = strlen (cmdline);
>>   memset (cmdline + k, 'x', N - 100);
>>   cmdline[k + N - 100] = '\0';
>>
>>   printf ("p1: Trying system()\n");
>>   rc = system (cmdline);
>>   printf ("p1: system returned %d\n", rc);
>>
>>   printf ("p1: Trying spawnl()\n");
>>   rc = spawnl (_P_WAIT, ".\\p2.exe", "p2.exe", strchr
>> (cmdline, ' '), NULL);
>>   printf ("p1: spawnl returned %d\n", rc);
>>
>>   memset (&si, 0, sizeof (si));
>>   si.cb = sizeof (si);
>>
>>   printf ("p1: Trying CreateProcess()\n");
>>   rc = CreateProcess (".\\p2.exe", cmdline, NULL, NULL, TRUE,
>> 0, NULL, NULL, &si, &pi);
>>   if (!rc)
>>     printf ("p1: CreateProcess failed: %lu\n", GetLastError ());
>>   else
>>     printf ("p1: CreateProcess success, process %lu\n",
>> pi.dwProcessId);
>>
>>   exit (0);
>> }
>>
>>
>> and p2.c:
>>
>> #include <windows.h>
>> #include <stdio.h>
>>
>> int
>> main (int argc, char **argv)
>> {
>>   int i;
>>   char *cmdline = GetCommandLine ();
>>
>>   if (cmdline)
>>     printf ("p2: GetCommandLine returned string of %d
>> chars\n", strlen (cmdline));
>>   else
>>     printf ("p2: GetCommandLine returned NULL\n");
>>
>>   printf ("p2: argc: %d\n", argc);
>>   for (i = 0; i < argc; i++)
>>     printf ("p2: argv[%d]: %5d chars: %.30s%s\n",
>>             i, strlen(argv[i]), argv[i], strlen (argv[i]) >
>> 30 ? "..." : "");
>>
>>   exit (0);
>> }
>>
>> Compile to p1.exe and p2.exe. Run p1.exe. See what output you
>> get from the two processes. Experiment with different
>> settings of the subsystem flag for p1.exe and p2.exe.
>> Experiment with starting as such from cmd.exe, and from bash,
>> both in a console window, rxvt, mintty, etc.
>> With output redirected by the shell (or cmd.exe) to a file or
>> pipe or not. Experiment with different settings of the
>> dwInheritHandles parameter to CreateProcess(), and/or the
>> standard handle fields in the STARTUPINFO. Etc.
>>
>> (I think he conclusion will be be that system() is best
>> avoided as it involves cmd.exe, but the spawn* functions work
>> fine, and
>> CreateProcess() obviously gives the ultimate control,
>> especially if you use the wide-char variant.)
>>
>> --tml
>
> Thank you all for your posts and support. I was off for the weekend.
> That's the reason I could only now do further tests.
>
> I did run p1 and p2 under all possible command processors and there is
> only one doing it right: TCC LE
> http://www.jpsoft.com/tccledes.htm
> At the end of the message is the test results.

Well, looks like your build is invoking the system() function then.
From MSDN's page on that:

"The system function passes command to the command interpreter, which
executes the string as an operating-system command. system refers to
the COMSPEC and PATH environment variables that locate the
command-interpreter file (the file named CMD.EXE in Windows NT)."

In cmd.exe as well as in MSYS bash, COMSPEC points to cmd.exe by
default. Presumably under TCC, which unlike bash is
backward-compatible with cmd.exe, that's changed to tcc.exe, and
apparently that doesn't have the 8K limit that cmd.exe has. (Bash
doesn't have that limit either, but that doesn't actually matter
here.)

> I've also used mintty to compile FFmpeg, but the result is the same,
> the commandline is too long.

Which should come as no surprise, because mintty is just a terminal
emulator, not a command interpreter. It's still bash running inside
it, and system() will still invoke cmd.exe by default.

Andy

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

waterlan
  Andy Koppe schreef, Op 13-9-2010 22:44:
> In cmd.exe as well as in MSYS bash, COMSPEC points to cmd.exe by
> default. Presumably under TCC, which unlike bash is
> backward-compatible with cmd.exe, that's changed to tcc.exe, and
> apparently that doesn't have the 8K limit that cmd.exe has. (Bash
> doesn't have that limit either, but that doesn't actually matter here.)

Yes, in TCC COMSPEC points to TCC.EXE. So Axel's problem could be solved
by changing COMSPEC in MSYS to TCC.EXE.
It was fun trying TCC. Loooong time ago I used 4DOS.

--
Erwin Waterlander


------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Earnie Boyd
Erwin Waterlander wrote:

>   Andy Koppe schreef, Op 13-9-2010 22:44:
>> In cmd.exe as well as in MSYS bash, COMSPEC points to cmd.exe by
>> default. Presumably under TCC, which unlike bash is
>> backward-compatible with cmd.exe, that's changed to tcc.exe, and
>> apparently that doesn't have the 8K limit that cmd.exe has. (Bash
>> doesn't have that limit either, but that doesn't actually matter here.)
>
> Yes, in TCC COMSPEC points to TCC.EXE. So Axel's problem could be solved
> by changing COMSPEC in MSYS to TCC.EXE.
> It was fun trying TCC. Loooong time ago I used 4DOS.
>

Just as easily the COMSPEC could specify the MSYS sh.exe.  I haven't
tried it so use caution.  You could use the following to test it:

COMSPEC=/bin/sh start /msys

--
Earnie
-- http://www.for-my-kids.com

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

Re: MSYS bash: maximum length of command linearguments

Axel Holzinger
Hi all,

Earnie wrote:
> Erwin Waterlander wrote:
> >   Andy Koppe schreef, Op 13-9-2010 22:44:
> >> In cmd.exe as well as in MSYS bash, COMSPEC points to cmd.exe by
> >> default. Presumably under TCC, which unlike bash is
> >> backward-compatible with cmd.exe, that's changed to tcc.exe, and
> >> apparently that doesn't have the 8K limit that cmd.exe has. (Bash

> >> doesn't have that limit either, but that doesn't actually matter
> >> here.)
> >
> > Yes, in TCC COMSPEC points to TCC.EXE. So Axel's problem could be
> > solved by changing COMSPEC in MSYS to TCC.EXE.
> > It was fun trying TCC. Loooong time ago I used 4DOS.
> >
>
> Just as easily the COMSPEC could specify the MSYS sh.exe.  I
> haven't tried it so use caution.  You could use the following
> to test it:
>
> COMSPEC=/bin/sh start /msys

this doesn't help, commandline still too long, but setting COMSPEC to
tcc.exe does the trick!!!

Thanks to everybody, problem solved!

Best regards
Axel

PS: If anybody wants to check this for mingw64, the bad thing is,
there is no free tcc.exe x64 :-(


------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users
Reply | Threaded
Open this post in threaded view
|

exec(ve) arg count (was: Re: MSYS bash: maximum length of command linearguments)

Tomi Ollila-2
In reply to this post by Charles Wilson-8
This email got me confused (for a moment) to how long command line can
be launched from perl... but fortunately with 'exec' there is no such limit.

But it is interesting to see the speed of execution when trying  to launch
exec with many arguments from MSYS perl. Try this:

time perl -le '@x = ("x") x 1e5; exec "echo", @x'

on command line.

On MSYS/MSYSGIT (up to 1.0.12 or something) in my test machine this lasts
1 min 40 sec.

On CYGWIN this takes only 2 seconds.

For comparison; on one Ubuntu 9.04 x86_64

env time -p perl -le '@x = ("x") x 1e5; exec "echo", @x'

executed in just 0.06 seconds.

(Hmm, redirecting output to /dev/null improved CYGWIN execution time to
0.4 secs).

(Hmm2: using '1e6' makes perl on MSYS fail (cannot allocate memory for
cygwin heap). On cygwin this executes OK, in 2 secs with >/dev/null
redirection. More interestingly Ubuntu 9.04 x86_64 fails with this,
but Fedora 13 x86_64 does not).

What I am interested to hear whether the speed of this execution is
improved when using newer msys systems (anyone care to test), and if
not then I/we just need to know it.

Tomi

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
MinGW-users mailing list
[hidden email]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.sourceforge.net/lists/listinfo/mingw-users