[Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

[Martin Panter]

On 14 April 2016 at 09:15, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 13.04.16 14:40, Victor Stinner wrote:
>> By the way, I don't understand why "AMD64 OpenIndiana 3.x" is
>> considered as stable since it's failing with multiple issues since
>> many months and nobody is working on these failures. I suggest to move
>> this buildbot back to the unstable category.
>
> I think the main cause is the lack of memory in this buildbot. I tried to
> minimize memory consumption and leaks, but some leaks are left, and they
> provoke other tests failures, and additional resource leaks. Would be nice
> to add a feature for running every test in separate subprocess. This will
> isolate the effect of failed tests.

Last time I looked into the Open Indiana buildbot, I concluded that
the biggest problem was Python using fork() to spawn subprocesses. I
understand that OS does not do “memory overcommitment” like Linux
does, so every time you fork, the OS has to double the amount of
memory that is reserved. It is ironic, but running each test using the
current subprocess module (which uses fork) would probably make the
problem worse.

I suspect using posix_spawn() if possible would help a lot. But this
was rejected in <https://bugs.python.org/issue20104> for not being
flexible enough and making maintainence too complicated.