[Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

[David Bolen]

If you look back on the worker there's probably some other build on the
same worker still technically running that's blocking the rest.  It might
be old enough to not show on the main summary page, and you won't see if it
just looking at a single builder history, so look at the worker in general.

For example, it looks like there's a 3.6 build from 5 days ago, which
appears stuck in test_multiprocessing_fork for the past 123 hours.  So the
later builds are just waiting for the worker to be available.

I've seen the rare multiprocessing test hang on some of my slower Windows
workers.  I think only maybe within the past 5-6 months and probably less
than a few times overall.  I don't know why the regular test timeouts fail
to interrupt things.  But if you manually kill the stuck test process on
the worker, everything should free up for the rest.  Not sure if a master
side cancellation works or not.

The other cases are appear stuck in builds but in different places. The ARM
worker has a 3.x build  from 10 days ago stuck in test_pydoc.  The other
AIX worker is stuck in a checkout step somehow.  But in all cases, it's not
the waiting builds at issue, but the hung build that blocks the rest.

-- David

On Sat, Nov 10, 2018 at 10:00 AM Michael <aixtools at felt.demon.nl> wrote:

> Just noticed my build-bot (#161) and the other AIX build-bot (#10) are
> stuck acquiring locks.
>
> #161 stuck 5 days (actually, 159, 160, 161, 162 and 163)
>
> #10 stuck 2 days (9, 10 and 131) - with 9 pinging builder for 9 hours
> (builders 4 and 104 are quiet)
>
> FYI.
>
> p.s. Not unique to AIX - this one is acquiring locks for 9 days!
> https://buildbot.python.org/all/#/builders/8
>
> Michael
>
>
> _______________________________________________
> Python-Buildbots mailing list
> Python-Buildbots at python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>