Is this a bug? Python intermittently stops dead for seconds

[Tim Peters]

[charlie strauss]
> I want to clarify that, on my computer, the first instance of the gap occurs way
> before the memory if filled. (at about 20% of physical ram).  Additionally the
> process monitor shows no page faults.

Python has no idea of how much RAM you have, or even of how much RAM
it's using.  See the `gc` module docs, function set_threshold(), for a
description of when Python decides to run a cyclic gc pass.  That's
all about the current excess (if any) of the number of container
allocations over the number of container deallocations since the last
time cyclic gc ran.

> ...
> (note that the array if filled as [1]*10, so there is actually only one "integer",
> but 10 array elements refering to it, per foo class.)

Nevertheless cyclic gc has to examine all 10 array elements each time
the list is scanned.

> ...
> For example when I write
>
> me.memory = [1]*nfoo
>
> perhaps internally, python is allocating an array of size foo and then __copying__ it
> into me.memory???

No.  Assignments in Python never copy anything.

> Since there is no reference to the intermediate it would then marked for future
> garbage collection.

Nothing in Python is "marked for future garbage collection".  From
time to time /all/ container objects are scanned to determine whether
any have become cyclic trash.  This takes time proportional to the
total number of containees across all container objects.

...

> foo was an election ballot holding 10 outcomes, and bar was a set of 100 ballots
> from 100 voting machines, and the total array was holding the ballot sets from a few
> thousand voting machines.
>
> Almost any inventory program is likely to have such a simmilar set of nested array,
> so it hardly seems unusual.

For memory-consumption reasons alone, a serious <wink> such program is
likely to use array.array objects at leaf level instead.  array.array
objects hold raw bits, not Python objects.  Since they don't hold
Python objects, they can't possibly be part of a cycle, so cyclic gc
doesn't need to scan array.array contents either.