python 2.7.12 on Linux behaving differently than on Windows

[Cameron Simpson]

On 07Dec2016 01:55, BartC <bc at freeuk.com> wrote:
>On 06/12/2016 22:35, Nathan Ernst wrote:
>>One other consideration in regards to globbing in the argument list:
>>there's a static limit to the byte length of argv. On windows, it's 8191
>>bytes
>
>That's the length of the command line, where parameters might still be 
>in the form *.*, ie. pre-splitting and pre-expansion. After converting 
>to argv format (which I believe is done by a C runtime function), that 
>will be in different memory with presumably its own limits.
>
>>(I'm assuming a null-terminator brings that to 8192, which is a weird
>>2**13). For Linux, as of kernal 2.6.25, apparently, the limit is 131072
>>bytes, an equally odd choice of 2**17 bytes.
>
>Maybe they started off at 4KB and 64KB but ran into problems.

I thought Linux had moved on to something more dynamic. Can't remember the 
citation though.

>But even Linux's 128KB will fill if someone wanted a command line that 
>listed 20,000 files individually. But it would be spectacularly bad 
>use of a command line interface which was designed for humans.

One might argue that this is exactly what command lines are to ease (either in 
the shell or in the app): expressing large things easily:

  cat *.txt     # millions of files, anyone?

>>I'm not sure if globs directly to commands bypass these restrictions, but
>>I've seen real world issues with build systems with thousands of files that
>>attempted to pass all dependencies to a target that blew up spectacularly
>>(one of the reasons cl & link on windows accept argument files as input
>>instead of just command line arguments).
>
>So does gcc on Linux (if you mean the @file syntax). It's a tidier way of 
>doing things.

Several things offer ways to read things-expected-to-be-large from files.  
Random example: wget will accept URLs from stdin or a file via its "-i" option, 
instead of always requiring them on the command line.

The typical workaround on UNIX for large argument lists is xargs, though it 
does not fit all use cases.

At some point I reach for (==> meaning "write") a small programme to work 
around such problems. Often a shell script or a python programme.

Cheers,
Cameron Simpson <cs at zip.com.au>