python 2.7.12 on Linux behaving differently than on Windows

[MRAB]

On 2016-12-06 13:08, eryk sun wrote:
> On Tue, Dec 6, 2016 at 12:26 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Tue, Dec 6, 2016 at 10:56 PM, BartC <bc at freeuk.com> wrote:
>>> In that directory (which was on Windows but accessible via a virtual Linux),
>>> typing any Linux command followed by * would have required all 3.4 million
>>> directory entries to be accessed in order to build a 3.4 million-element
>>> argv list. I've no idea how long that would have taken.
>>
>> I just asked Python to build me a 4-million-element list, and it took
>> no visible time - a small fraction of a second. Don't be afraid of
>> large argument lists. We're not writing 8088 Assembly Language
>> programs in 64KB of working memory here.
>
> The problem isn't building an arbitrary list with millions of
> elements. The problem is the time it would take to read millions of
> filenames from a directory. It depends on the performance of the disk
> and filesystem. Be careful with globbing. Think about the consequences
> before running a command, especially if you're in the habit of
> creating directories with hundreds of thousands, or millions, of
> files. It's not a problem that I've ever had to deal with.
>
Many years ago I was working with a database application running on 
MSDOS that stored predefined queries in files, one query per file. There 
were many queries (though fewer than a thousand), resulting in many 
small files in a single directory. Fetching one of those predefined 
queries was surprisingly slow.

I found that it was a lot faster to put them into a single file and then 
call an external program to extract the one wanted. It also took up a 
lot less disk space!