[Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks
Michael Felt
aixtools at felt.demon.nl
Tue Nov 13 16:07:49 EST 2018
On 11/12/2018 3:16 AM, Ned Deily wrote:
> On Nov 11, 2018, at 15:27, Michael <aixtools at felt.demon.nl> wrote:
>> Anyway, my bot went from having 9 to 10 fails (and 10k lines of output)
>> to 20 fails and roughly 22k lines of test.
>>
>> The last "near normal FAIL" is:
>> https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio
>> and the first mega fail is
>> https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio
> Michael,
>
> Thanks for tracking down where the additional failures began. I took a quick look at the buildbot listings for builds 325 and 325 and it seems like all of the additional failures that appear in 326 vs 325 are a result of a new build failure for the pyexpat module. That's visible at the end of the "compile" stage output:
>
> Failed to build these modules:
> pyexpat
>
> and near the top of the test output when the Makefile tries to rebuild any failed modules.
>
> Looking at the output of the git stage of 326, the git branch HEAD of the build is 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d. Running git show to get the commit message:
>
> $ git show 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
> commit 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
> Author: Gregory P. Smith <greg at krypto.org>
> Date: Wed Oct 17 18:10:46 2018 -0700
>
> bpo-35011: Restore use of pyexpatns.h in libexpat (GH-9939)
>
> Restores the use of pyexpatns.h to isolate our embedded copy of the expat C
> library so that its symbols do not conflict at link or dynamic loading time
> with an embedding application or other extension modules with their own
> version of libexpat.
>
> https://github.com/python/cpython/commit/5dc3f23b5fb0b510926012cb3732dae63cddea60#diff-3afaf7274c90ce1b7405f75ad825f545 inadvertently removed it when upgrading expat.
>
> That looks rather suspicious!
>
> So I guess the next step would be to try to figure out why that change (or some other in the range between buildbot run 325 and 326) causes the pyexpact module to stop building. There are a bunch of ld: 0711-317 ERROR: Undefined symbol errors in the link step of the "building 'pyexpat' extension" step. In the "configure" step, it is determined that it is not using an operating system provided version of expat:
>
> checking for --with-system-expat... no
>
> Which is to be expected. So it should be trying to pick up a version of expat that is provided elsewhere outside of the build.
Basically, as long as this bot has been running there has been a
libexpat.a in /opt/lib - at expat version 2.2.5.
buildbot at x064:[/home/buildbot]lslpp -L | grep expat
aixtools.expat.rte 2.2.5.0 C F aixtools expat
12-Nov-2017
aixtools.expat.share 2.2.5.0 C F aixtools expat
universal files
buildbot at x064:[/home/buildbot]lslpp -h aixtools.expat.rte
Fileset Level Action Status Date Time
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
aixtools.expat.rte
2.2.5.0 COMMIT COMPLETE 08/02/18 10:16:19
That has not changed.
I manually did a clone of cpython master and built using the same
arguments, userid and environment as I expect the bot to be using - And
I get 11 failed tests:
11 tests failed:
test__xxsubinterpreters test_binascii test_ctypes test_distutils
test_httpservers test_importlib test_multiprocessing_forkserver
test_multiprocessing_spawn test_pyexpat test_socket test_time
±±±±
I checked on my regular (manual) build system, and I see that I do not
have any libexpat.a library.
I recall (now), that I have occaisionally had to "manually" add -lexpat
to one of the modules - because of many undefined symbols - e.g.,
building 'pyexpat' extension
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -I/opt/include
-I/opt/buildaix/include -I./Include/internal -DHAVE_EXPAT_CONFIG_H=1
-DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat
-I./Include -I. -I/opt/include -I/opt/buildaix/include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0 -c
/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/pyexpat.c
-o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/pyexpat.o
"/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/pyexpat.c",
line 1221.23: 1506-068 (W) Operation between types "void*" and
"void(*)(void*,const char*,int)" is not allowed.
1500-030: (I) INFORMATION: PyInit_pyexpat: Additional optimization
may be attained by recompiling and specifying MAXMEM option with a value
greaterthan 8192.
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -I/opt/include
-I/opt/buildaix/include -I./Include/internal -DHAVE_EXPAT_CONFIG_H=1
-DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat
-I./Include -I. -I/opt/include -I/opt/buildaix/include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0 -c
/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlparse.c
-o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlparse.o
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -I/opt/include
-I/opt/buildaix/include -I./Include/internal -DHAVE_EXPAT_CONFIG_H=1
-DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat
-I./Include -I. -I/opt/include -I/opt/buildaix/include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0 -c
/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlrole.c
-o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlrole.o
xlc_r -O2 -I/opt/include -I/opt/buildaix/include -I/opt/include
-I/opt/buildaix/include -I./Include/internal -DHAVE_EXPAT_CONFIG_H=1
-DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat
-I./Include -I. -I/opt/include -I/opt/buildaix/include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Include
-I/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0 -c
/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmltok.c
-o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmltok.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L/opt/lib
-bmaxdata:0x40000000 -L/opt/lib -bmaxdata:0x40000000 -L/opt/lib
-bmaxdata:0x40000000 -I/opt/include -I/opt/buildaix/include
-I/opt/include -I/opt/buildaix/include
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/pyexpat.o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlparse.o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmlrole.o
build/temp.aix-7.1-3.8-pydebug/home/buildbot/buildarea/3.x.aixtools-aix-power6/build.0/Modules/expat/xmltok.o
-L/opt/lib -o build/lib.aix-7.1-3.8-pydebug/pyexpat.so
ld: 0711-317 ERROR: Undefined symbol: XML_SetStartElementHandler
ld: 0711-317 ERROR: Undefined symbol: XML_SetEndElementHandler
ld: 0711-317 ERROR: Undefined symbol: XML_SetProcessingInstructionHandler
ld: 0711-317 ERROR: Undefined symbol: XML_SetCharacterDataHandler
ld: 0711-317 ERROR: Undefined symbol: XML_SetUnparsedEntityDeclHandler
...
When I add "-lexpat" the module builds normally. But is "contrary" to
having it find a system libexpat (it is there, but in /opt/lib).
As my system that does not any libexpat does not need this "help", I am
going to guess that since configure finds the include files it builds
pyexpat differently than when the include file is not available.
I'll research further. (I'll start with renaming the expat include files).
All for tonight. Thanks for hints to look at.
> Perhaps there is an issue with it. Or the above change doesn't take something into account. To proceed further, it would be most helpful to have someone with AIX experience and/or familiarity with that buildbot's setup.
>
>
> --
> Ned Deily
> nad at python.org -- []
>
>
More information about the Python-Buildbots
mailing list