From rosuav at gmail.com  Sun Dec 17 09:40:35 2017
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Dec 2017 01:40:35 +1100
Subject: [Python-buildbots] Outage notifications aren't happening?
Message-ID: <CAPTjJmq-65bRBH2r5A9EgXVqBmF=ZWRr_=XvtygRRBOGjeFB+w@mail.gmail.com>

I just noticed by chance that my build slave was down. Not sure how
long it had been down for, but it would have been several days, maybe
a week or more. There used to be automated emails whenever contact was
lost; did those disappear with the buildbot version changes?

ChrisA
admin of the Debian Root build slave

From zachary.ware+pydev at gmail.com  Mon Dec 18 11:50:23 2017
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Mon, 18 Dec 2017 10:50:23 -0600
Subject: [Python-buildbots] Outage notifications aren't happening?
In-Reply-To: <CAPTjJmq-65bRBH2r5A9EgXVqBmF=ZWRr_=XvtygRRBOGjeFB+w@mail.gmail.com>
References: <CAPTjJmq-65bRBH2r5A9EgXVqBmF=ZWRr_=XvtygRRBOGjeFB+w@mail.gmail.com>
Message-ID: <CAKJDb-MiLeyO8Dtyy=Q-Z93nuNBdF0Wyn3eSWNO9+DQJG8Fwvw@mail.gmail.com>

On Sun, Dec 17, 2017 at 8:40 AM, Chris Angelico <rosuav at gmail.com> wrote:
> I just noticed by chance that my build slave was down. Not sure how
> long it had been down for, but it would have been several days, maybe
> a week or more. There used to be automated emails whenever contact was
> lost; did those disappear with the buildbot version changes?

I'm not sure what's up with that, especially since the buildbot-status
emails are working correctly.  I've opened
https://github.com/python/buildmaster-config/issues/27 to keep track
of it.

-- 
Zach

From zachary.ware at gmail.com  Mon Dec 18 11:56:57 2017
From: zachary.ware at gmail.com (Zachary Ware)
Date: Mon, 18 Dec 2017 10:56:57 -0600
Subject: [Python-buildbots] Potential master downtime
Message-ID: <CAKJDb-P9qjVGmCfocqgWtGmygftj_nYCQkHUrNRXAoCGmiZ87g@mail.gmail.com>

Hi all,

As part of the scheduled maintenance at OSUOSL on the 21st [1] the
buildbot master may be down long enough that your workers start to get
worried, but I'll be keeping an eye on it to try to keep downtime to a
minimum.  There shouldn't be any action required on the part of worker
owners; I'll update you here if there is.

[1] https://status.python.org/incidents/6v34ns57x31s

Thanks!
-- 
Zach

From victor.stinner at gmail.com  Thu Dec 21 17:27:26 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Dec 2017 23:27:26 +0100
Subject: [Python-buildbots] Internal buildbot error?
Message-ID: <CAMpsgwbszvqDLCN1Xag8XXweTRzc81MjMwbhPXym=uj2WvUFRQ@mail.gmail.com>

Hi,

What is the following failure? Can it be an internal buildbot bot?

http://buildbot.python.org/all/#/builders/69/builds/75

error in RunProcess._startCommand
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/buildslave/runprocess.py",
line 426, in start
self._startCommand()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/buildslave/runprocess.py",
line 550, in _startCommand
usePTY=self.usePTY)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/buildslave/runprocess.py",
line 572, in _spawnProcess
processProtocol, uid, gid, childFDs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/process.py",
line 747, in __init__
parentFD, forceReadHack=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/process.py",
line 156, in __init__
if not stat.S_ISFIFO(os.fstat(self.fileno()).st_mode):
OSError: [Errno 9] Bad file descriptor

Traceback (most recent call last):
  File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/internet/defer.py",
line 500, in errback
    self._startRunCallbacks(fail)
  File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/internet/defer.py",
line 567, in _startRunCallbacks
    self._runCallbacks()
  File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/internet/defer.py",
line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/internet/defer.py",
line 1442, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/internet/defer.py",
line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/data/buildbot/venv/lib/python3.4/site-packages/buildbot/process/buildstep.py",
line 705, in run
    results = yield self._start_deferred
builtins.TypeError: Can't convert 'bytes' object to str implicitly

Victor

From db3l.net at gmail.com  Thu Dec 21 21:26:34 2017
From: db3l.net at gmail.com (David Bolen)
Date: Thu, 21 Dec 2017 21:26:34 -0500
Subject: [Python-buildbots] Internal buildbot error?
In-Reply-To: <CAMpsgwbszvqDLCN1Xag8XXweTRzc81MjMwbhPXym=uj2WvUFRQ@mail.gmail.com>
References: <CAMpsgwbszvqDLCN1Xag8XXweTRzc81MjMwbhPXym=uj2WvUFRQ@mail.gmail.com>
Message-ID: <CAFirtYVcaLLbc3gs1uXaB-BbP8KhnmEoi=mZNxt7=jRS2=XZTg@mail.gmail.com>

It does seem like it might be a race condition of some sort - or at least
an unhandled case - in Twisted's process handling.

It's in the writer path, so I wonder if the umask operation completes
quickly enough, perhaps the output handle is in fact invalid by the time
the buildbot parent gets to that section of code.  The umask step looks
like it's only used by the DMG builder, not for regular test builds.

It's fairly uncommon - just twice so far across all branches under the new
master.  To be honest, we could probably just eliminate that step at this
point from the DMG builds (not sure it serves much purpose) or maybe add a
small sleep along with the umask to ensure the child process is around long
enough for the parent to complete the setup.

 -- David

On Thu, Dec 21, 2017 at 5:27 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> What is the following failure? Can it be an internal buildbot bot?
>
> http://buildbot.python.org/all/#/builders/69/builds/75
>
> error in RunProcess._startCommand
> Traceback (most recent call last):
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 426, in start
> self._startCommand()
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 550, in _startCommand
> usePTY=self.usePTY)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 572, in _spawnProcess
> processProtocol, uid, gid, childFDs)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/twisted/internet/process.py",
> line 747, in __init__
> parentFD, forceReadHack=True)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/twisted/internet/process.py",
> line 156, in __init__
> if not stat.S_ISFIFO(os.fstat(self.fileno()).st_mode):
> OSError: [Errno 9] Bad file descriptor
>
> Traceback (most recent call last):
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 500, in errback
>     self._startRunCallbacks(fail)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 567, in _startRunCallbacks
>     self._runCallbacks()
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 653, in _runCallbacks
>     current.result = callback(current.result, *args, **kw)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 1442, in gotResult
>     _inlineCallbacks(r, g, deferred)
> --- <exception caught here> ---
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 1386, in _inlineCallbacks
>     result = g.send(result)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> buildbot/process/buildstep.py",
> line 705, in run
>     results = yield self._start_deferred
> builtins.TypeError: Can't convert 'bytes' object to str implicitly
>
> Victor
> _______________________________________________
> Python-Buildbots mailing list
> Python-Buildbots at python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>

From db3l.net at gmail.com  Thu Dec 21 21:39:58 2017
From: db3l.net at gmail.com (David Bolen)
Date: Thu, 21 Dec 2017 21:39:58 -0500
Subject: [Python-buildbots] Internal buildbot error?
In-Reply-To: <CAFirtYVcaLLbc3gs1uXaB-BbP8KhnmEoi=mZNxt7=jRS2=XZTg@mail.gmail.com>
References: <CAMpsgwbszvqDLCN1Xag8XXweTRzc81MjMwbhPXym=uj2WvUFRQ@mail.gmail.com>
 <CAFirtYVcaLLbc3gs1uXaB-BbP8KhnmEoi=mZNxt7=jRS2=XZTg@mail.gmail.com>
Message-ID: <CAFirtYXODZVRd1h-Ps1n2OUU-DV+3qmtET06whfef_fcOqEDTg@mail.gmail.com>

BTW, I've found some old emails that the umask step was added as part of
some debugging back in 2010 when the DMG builder was first getting set up,
but the original issue has long since been resolved, so it doesn't really
serve any purpose and might as well get removed.

-- David

On Thu, Dec 21, 2017 at 9:26 PM, David Bolen <db3l.net at gmail.com> wrote:

> It does seem like it might be a race condition of some sort - or at least
> an unhandled case - in Twisted's process handling.
>
> It's in the writer path, so I wonder if the umask operation completes
> quickly enough, perhaps the output handle is in fact invalid by the time
> the buildbot parent gets to that section of code.  The umask step looks
> like it's only used by the DMG builder, not for regular test builds.
>
> It's fairly uncommon - just twice so far across all branches under the new
> master.  To be honest, we could probably just eliminate that step at this
> point from the DMG builds (not sure it serves much purpose) or maybe add a
> small sleep along with the umask to ensure the child process is around long
> enough for the parent to complete the setup.
>
>  -- David
>
> On Thu, Dec 21, 2017 at 5:27 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Hi,
>>
>> What is the following failure? Can it be an internal buildbot bot?
>>
>> http://buildbot.python.org/all/#/builders/69/builds/75
>>
>> error in RunProcess._startCommand
>> Traceback (most recent call last):
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 426, in start
>> self._startCommand()
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 550, in _startCommand
>> usePTY=self.usePTY)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 572, in _spawnProcess
>> processProtocol, uid, gid, childFDs)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/twisted/internet/process.py",
>> line 747, in __init__
>> parentFD, forceReadHack=True)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/twisted/internet/process.py",
>> line 156, in __init__
>> if not stat.S_ISFIFO(os.fstat(self.fileno()).st_mode):
>> OSError: [Errno 9] Bad file descriptor
>>
>> Traceback (most recent call last):
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 500, in errback
>>     self._startRunCallbacks(fail)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 567, in _startRunCallbacks
>>     self._runCallbacks()
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 653, in _runCallbacks
>>     current.result = callback(current.result, *args, **kw)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 1442, in gotResult
>>     _inlineCallbacks(r, g, deferred)
>> --- <exception caught here> ---
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 1386, in _inlineCallbacks
>>     result = g.send(result)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/buildbot/
>> process/buildstep.py",
>> line 705, in run
>>     results = yield self._start_deferred
>> builtins.TypeError: Can't convert 'bytes' object to str implicitly
>>
>> Victor
>> _______________________________________________
>> Python-Buildbots mailing list
>> Python-Buildbots at python.org
>> https://mail.python.org/mailman/listinfo/python-buildbots
>>
>
>