From aixtools at felt.demon.nl  Sun Jun 16 07:59:56 2019
From: aixtools at felt.demon.nl (Michael)
Date: Sun, 16 Jun 2019 13:59:56 +0200
Subject: [Python-buildbots] Finally ! Many thanks!
Message-ID: <84dae9fa-483a-a6f2-8adf-e4328e14c793@felt.demon.nl>

Finally, got to this point that the tests are all passing on the bot - 3.X!!



p.s. - now that 3.8 is beta - Will there be a separate bot for 3.8 in
the near future?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-buildbots/attachments/20190616/7aaa6399/attachment.sig>

From rosuav at gmail.com  Sun Jun 16 21:20:06 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Jun 2019 11:20:06 +1000
Subject: [Python-buildbots] Current build stuck for four days(!)
Message-ID: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>

Turns out that this isn't a massive queue of builds happening, but
just one single build.

https://buildbot.python.org/all/#/builders/211/builds/46

The builder is saturating one CPU core and has been doing so for
literally days. I don't think it's going to get anywhere at this
point. Is there a way to bomb the build and tell it to start over?

ChrisA

From db3l.net at gmail.com  Sun Jun 16 21:49:10 2019
From: db3l.net at gmail.com (David Bolen)
Date: Sun, 16 Jun 2019 21:49:10 -0400
Subject: [Python-buildbots] Current build stuck for four days(!)
In-Reply-To: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>
References: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>
Message-ID: <CAFirtYVYKA6q18zogmszD0CsOD9TKJ5JW8Rj0DO67UhjfdJiVA@mail.gmail.com>

If you're signed in with your github.com account, then on the buildbot web
interface you should be able to stop the build as builder owner (if your
github.com account matches the worker information and is in the python
group - or something like that).  Although I suppose in this case it may
depend on how its hung.

In most cases for me, I usually just kill the test process locally on the
builder - that'll fail that build abruptly but then everything just moves
on.

-- David

On Sun, Jun 16, 2019 at 9:20 PM Chris Angelico <rosuav at gmail.com> wrote:

> Turns out that this isn't a massive queue of builds happening, but
> just one single build.
>
> https://buildbot.python.org/all/#/builders/211/builds/46
>
> The builder is saturating one CPU core and has been doing so for
> literally days. I don't think it's going to get anywhere at this
> point. Is there a way to bomb the build and tell it to start over?
>
> ChrisA
> _______________________________________________
> Python-Buildbots mailing list
> Python-Buildbots at python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>

From rosuav at gmail.com  Mon Jun 17 00:11:05 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Jun 2019 14:11:05 +1000
Subject: [Python-buildbots] Current build stuck for four days(!)
In-Reply-To: <CAFirtYVYKA6q18zogmszD0CsOD9TKJ5JW8Rj0DO67UhjfdJiVA@mail.gmail.com>
References: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>
 <CAFirtYVYKA6q18zogmszD0CsOD9TKJ5JW8Rj0DO67UhjfdJiVA@mail.gmail.com>
Message-ID: <CAPTjJmqDN_mrzzhNqAeS6BTD7Tw4wt28rXbeHYvW3XXawu2pMw@mail.gmail.com>

On Mon, Jun 17, 2019 at 11:49 AM David Bolen <db3l.net at gmail.com> wrote:
>
> If you're signed in with your github.com account, then on the buildbot web
> interface you should be able to stop the build as builder owner (if your
> github.com account matches the worker information and is in the python
> group - or something like that).  Although I suppose in this case it may
> depend on how its hung.

Hmm. I'm logged in and looking at the page, but can't see anything
about stopping the job. So it's probably not set up to recognize me as
an owner - possibly a groups issue.

> In most cases for me, I usually just kill the test process locally on the
> builder - that'll fail that build abruptly but then everything just moves
> on.

I took the easy way out and restarted the whole VM :) Thanks for the advice.

ChrisA

From vstinner at redhat.com  Mon Jun 17 03:01:04 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 17 Jun 2019 09:01:04 +0200
Subject: [Python-buildbots] Current build stuck for four days(!)
In-Reply-To: <CAPTjJmqDN_mrzzhNqAeS6BTD7Tw4wt28rXbeHYvW3XXawu2pMw@mail.gmail.com>
References: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>
 <CAFirtYVYKA6q18zogmszD0CsOD9TKJ5JW8Rj0DO67UhjfdJiVA@mail.gmail.com>
 <CAPTjJmqDN_mrzzhNqAeS6BTD7Tw4wt28rXbeHYvW3XXawu2pMw@mail.gmail.com>
Message-ID: <CA+3bQGE=b-VC0Jq3QaVkq_h6OeOp6ASHLWgxKWcg1Tx58U+pmA@mail.gmail.com>

It is the first for a very long time that I see a buildbot stuck in a
loop and don't produce any output. The main test runner process should
write the progress every 30 seconds. But here the job ran for 3 days.
Maybe it is a regression un test_asyncio, but it doesn't explain why
the main process fails to report update.

0:00:06 load avg: 0.58 [  8/423] test_pyexpat passed
0:00:07 load avg: 0.58 [  9/423] test_aifc passed
running: test_asyncio (37 sec 198 ms), test_html (30 sec 59 ms)
remoteFailed: [Failure instance: Traceback (failure with no frames):
<class 'twisted.internet.error.ConnectionLost'>: Connection to the
other side was lost in a non-clean fashion.
]

CIs are great. Every day, a new surprise! Every day I open my PyBB
email folder, I am excited, what will we get today??

Victor

Le lun. 17 juin 2019 ? 06:11, Chris Angelico <rosuav at gmail.com> a ?crit :
>
> On Mon, Jun 17, 2019 at 11:49 AM David Bolen <db3l.net at gmail.com> wrote:
> >
> > If you're signed in with your github.com account, then on the buildbot web
> > interface you should be able to stop the build as builder owner (if your
> > github.com account matches the worker information and is in the python
> > group - or something like that).  Although I suppose in this case it may
> > depend on how its hung.
>
> Hmm. I'm logged in and looking at the page, but can't see anything
> about stopping the job. So it's probably not set up to recognize me as
> an owner - possibly a groups issue.
>
> > In most cases for me, I usually just kill the test process locally on the
> > builder - that'll fail that build abruptly but then everything just moves
> > on.
>
> I took the easy way out and restarted the whole VM :) Thanks for the advice.
>
> ChrisA
> _______________________________________________
> Python-Buildbots mailing list
> Python-Buildbots at python.org
> https://mail.python.org/mailman/listinfo/python-buildbots



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From rosuav at gmail.com  Mon Jun 17 03:27:25 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 17 Jun 2019 17:27:25 +1000
Subject: [Python-buildbots] Current build stuck for four days(!)
In-Reply-To: <CA+3bQGE=b-VC0Jq3QaVkq_h6OeOp6ASHLWgxKWcg1Tx58U+pmA@mail.gmail.com>
References: <CAPTjJmqt8ZU76GM+WqYyBcFx6zSKxQA7hQ8L8yuJCAqi_XQH6Q@mail.gmail.com>
 <CAFirtYVYKA6q18zogmszD0CsOD9TKJ5JW8Rj0DO67UhjfdJiVA@mail.gmail.com>
 <CAPTjJmqDN_mrzzhNqAeS6BTD7Tw4wt28rXbeHYvW3XXawu2pMw@mail.gmail.com>
 <CA+3bQGE=b-VC0Jq3QaVkq_h6OeOp6ASHLWgxKWcg1Tx58U+pmA@mail.gmail.com>
Message-ID: <CAPTjJmrAqPKJpv3ejupoKb+-s+6TkEgPj1+uDOz=+HeJCHYStQ@mail.gmail.com>

On Mon, Jun 17, 2019 at 5:01 PM Victor Stinner <vstinner at redhat.com> wrote:
>
> It is the first for a very long time that I see a buildbot stuck in a
> loop and don't produce any output. The main test runner process should
> write the progress every 30 seconds. But here the job ran for 3 days.
> Maybe it is a regression un test_asyncio, but it doesn't explain why
> the main process fails to report update.
>
> 0:00:06 load avg: 0.58 [  8/423] test_pyexpat passed
> 0:00:07 load avg: 0.58 [  9/423] test_aifc passed
> running: test_asyncio (37 sec 198 ms), test_html (30 sec 59 ms)
> remoteFailed: [Failure instance: Traceback (failure with no frames):
> <class 'twisted.internet.error.ConnectionLost'>: Connection to the
> other side was lost in a non-clean fashion.
> ]
>

Yeah, I wasn't able to see anything in there either. The final
disconnect was probably as I powered down the VM itself - the process
was, apparently, unable to cleanly terminate. I'm not sure whether a
TCP RESET was sent during shutdown, or if the connection stayed open
until the build master next tried to ping it, but I suspect the
former. In any case, there was no application-level shutdown.

> CIs are great. Every day, a new surprise! Every day I open my PyBB email folder, I am excited, what will we get today??

You don't say! That's like my reason for subscribing to the
tz-announce list - what new timezone or DST insanity will we see next?
Tz-announce happens maybe ten times a year, but it's such gloriously
ridiculous stuff that it's worth it. (Egypt, I shall never forget the
day you reminded me that there were still worse things that could be
done with DST.)

ChrisA

From zachary.ware+pydev at gmail.com  Mon Jun 17 13:24:59 2019
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Mon, 17 Jun 2019 12:24:59 -0500
Subject: [Python-buildbots] Finally ! Many thanks!
In-Reply-To: <84dae9fa-483a-a6f2-8adf-e4328e14c793@felt.demon.nl>
References: <84dae9fa-483a-a6f2-8adf-e4328e14c793@felt.demon.nl>
Message-ID: <CAKJDb-OvsLdhp7AHzaJu7kTP5qB-5AV2k0jL4PUGrv4XFqVpKQ@mail.gmail.com>

On Sun, Jun 16, 2019 at 7:22 AM Michael <aixtools at felt.demon.nl> wrote:
> Finally, got to this point that the tests are all passing on the bot - 3.X!!

Congrats!

> p.s. - now that 3.8 is beta - Will there be a separate bot for 3.8 in
> the near future?

We do now have 3.8 builders, but those workers that were restricted to
3.x did not automatically have 3.8 builders added.  If tests are
passing on the 3.8 branch for you, I don't see any reason not to
accept a pull request to https://github.com/python/buildmaster-config
adding a 3.8 builder to your worker :)


--
Zach

From ncoghlan at gmail.com  Wed Jun 19 08:26:25 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 Jun 2019 22:26:25 +1000
Subject: [Python-buildbots] Finally ! Many thanks!
In-Reply-To: <84dae9fa-483a-a6f2-8adf-e4328e14c793@felt.demon.nl>
References: <84dae9fa-483a-a6f2-8adf-e4328e14c793@felt.demon.nl>
Message-ID: <CADiSq7dD-n1enWHW9sGQ4SZg5fMY-Cu+61NAsqEmCgAiYrXRDw@mail.gmail.com>

On Sun, 16 Jun 2019 at 22:22, Michael <aixtools at felt.demon.nl> wrote:
>
> Finally, got to this point that the tests are all passing on the bot - 3.X!!

Huzzah!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From aixtools at felt.demon.nl  Sun Jun 23 06:50:28 2019
From: aixtools at felt.demon.nl (Michael)
Date: Sun, 23 Jun 2019 12:50:28 +0200
Subject: [Python-buildbots] FYI: 'POWER6' bot offline for a few weeks
Message-ID: <883b6bde-c1e2-c666-4723-b6de4e08ec30@felt.demon.nl>

Hi all,

Now that the PPC64-AIX bot is also "green" for master (and I hope real
soon for 3.8 and 3.7) I feel 'comfortable' with having me server in
PowerOff status while I am off on vacation.

Will be back soon!

Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-buildbots/attachments/20190623/bca8f221/attachment.sig>

From vstinner at redhat.com  Tue Jun 25 06:24:33 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 25 Jun 2019 12:24:33 +0200
Subject: [Python-buildbots] New cleantest step on buildbots: remove old
 test_python_xxx directories
Message-ID: <CA+3bQGEkMyg-9c3n38_aPzajW7GRrFKv5ES28E3ocCBggdcBrg@mail.gmail.com>

Hi,

Since the end of last week, test_regrtest started to fail on more and
more buildbots, especially on x86 Gentoo Installed with X 3.x, with an
error message which contains:

RuntimeWarning: tests may fail, unable to create temporary directory
'/buildbot/tmp/tmpdir/test_python_28144': [Errno 17] File exists:
'/buildbot/tmp/tmpdir/test_python_28144'"

I knew for years that our buildbots are not great to cleanup things
between two builds: it's common that a failure is causing issues in
following builds and may require a manual action.

I added a new "cleantest" step which runs a new "./python -m test
--cleanup" command: it removes test_python_* directories in the
directory used by Python for tests (ex: ./build/ when building from
sources on Unix).

On x86 Gentoo Installed with X 3.x, my new step removed not least than
4146 old "test_python_xxx" directories! That's pretty big knowing that
Linux pids are limited to 32768: 4146 is already 13% of the available
ids, which makes the issue quite likely.

By the way, I also enhanced regrtest when using worker subprocesses.
Workers now create their sub-directory in the temporary directory of
the main process. Moreover, the main process now kills immediately
workers (and wait until they complete) on failure or if interrupted.
The whole test directory is always removed by the main process. The
last case where regrtest is when the main process does crash, which
happens often when tests are re-run.

Sadly, when tests are re-run, they are run sequentially in the main
process, rather than spawning a subprocess for each test file. That
should be fixed.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From cstratak at redhat.com  Thu Jun 27 08:39:29 2019
From: cstratak at redhat.com (Charalampos Stratakis)
Date: Thu, 27 Jun 2019 08:39:29 -0400 (EDT)
Subject: [Python-buildbots] Some more Fedora and RHEL machines willing to
 join the fleet
In-Reply-To: <1108471850.34448969.1561638693534.JavaMail.zimbra@redhat.com>
Message-ID: <1512713325.34449791.1561639169893.JavaMail.zimbra@redhat.com>

Hello,

I've recently got some more x86_64 machines that I'd like to add to the buildbot fleet.

More specifically, a Fedora 30 (the stable branch of Fedora), a RHEL7 and a RHEL8 machine.

I'm gonna add some naming suggestions, just to get things rolling :)

cstratak-fedora-stable, cstratak-RHEL7-x86_64, cstratak-RHEL8-x86_64

-- 
Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat

From cstratak at redhat.com  Thu Jun 27 10:19:08 2019
From: cstratak at redhat.com (Charalampos Stratakis)
Date: Thu, 27 Jun 2019 10:19:08 -0400 (EDT)
Subject: [Python-buildbots] Some more Fedora and RHEL machines willing
 to join the fleet
In-Reply-To: <1512713325.34449791.1561639169893.JavaMail.zimbra@redhat.com>
References: <1512713325.34449791.1561639169893.JavaMail.zimbra@redhat.com>
Message-ID: <1563456797.34481103.1561645148709.JavaMail.zimbra@redhat.com>



----- Original Message -----
> From: "Charalampos Stratakis" <cstratak at redhat.com>
> To: "python-buildbo." <python-buildbots at python.org>
> Sent: Thursday, June 27, 2019 2:39:29 PM
> Subject: [Python-buildbots] Some more Fedora and RHEL machines willing to join the fleet
> 
> Hello,
> 
> I've recently got some more x86_64 machines that I'd like to add to the
> buildbot fleet.
> 
> More specifically, a Fedora 30 (the stable branch of Fedora), a RHEL7 and a
> RHEL8 machine.
> 
> I'm gonna add some naming suggestions, just to get things rolling :)
> 
> cstratak-fedora-stable, cstratak-RHEL7-x86_64, cstratak-RHEL8-x86_64
> 

Or maybe even better cstratak-fedora-stable-x86_64. However the first fedora buildbot I added is cstratak-fedora. Could its name be changed to cstratak-fedora-x86_64?

> --
> Regards,
> 
> Charalampos Stratakis
> Software Engineer
> Python Maintenance Team, Red Hat
> _______________________________________________
> Python-Buildbots mailing list
> Python-Buildbots at python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
> 

-- 
Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat

From Paul.Monson at microsoft.com  Wed Jun 26 16:31:22 2019
From: Paul.Monson at microsoft.com (Paul Monson)
Date: Wed, 26 Jun 2019 20:31:22 +0000
Subject: [Python-buildbots] Windows ARM32 buildbot
Message-ID: <CY4PR2101MB0804EBD4E45B60F79F20F934F9E20@CY4PR2101MB0804.namprd21.prod.outlook.com>

Can someone merge this PR please?
https://github.com/python/buildmaster-config/pull/97

I suspect that the socket tests are flaky no Windows ARM32 because of a lack of resource, and I would like to test running one test at time to see if it improves test reliability.
The tests seems less flaky when not running in a buildbot worker, but it might just be that the sample size is too small when running manually

Thanks,
Paul