From status at bugs.python.org Fri May 2 18:07:53 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 2 May 2014 18:07:53 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140502160753.687B85691B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-04-25 - 2014-05-02) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4605 ( +5) closed 28582 (+61) total 33187 (+66) Open issues with patches: 2109 Issues opened (49) ================== #21350: bug in file.writelines accepting buffers http://bugs.python.org/issue21350 opened by bdkearns #21352: improve documentation indexing http://bugs.python.org/issue21352 opened by bgailer #21354: PyCFunction_New no longer exposed by python DLL breaking bdist http://bugs.python.org/issue21354 opened by mhammond #21356: Support LibreSSL (instead of OpenSSL): make RAND_egd optional http://bugs.python.org/issue21356 opened by Edd.Barrett #21357: Increase filecmp test coverage from 63% to 76% http://bugs.python.org/issue21357 opened by diana #21358: Augmented assignment doc: clarify 'only evaluated once' http://bugs.python.org/issue21358 opened by terry.reedy #21359: IDLE Redo command accelerator acts as Undo with current OS X C http://bugs.python.org/issue21359 opened by ned.deily #21360: mailbox.Maildir should ignore files named with a leading dot http://bugs.python.org/issue21360 opened by liw #21362: concurrent.futures does not validate that max_workers is prope http://bugs.python.org/issue21362 opened by Claudiu.Popa #21363: io.TextIOWrapper always closes wrapped files http://bugs.python.org/issue21363 opened by aronacher #21364: Documentation Recommends Broken Pattern http://bugs.python.org/issue21364 opened by aronacher #21365: asyncio.Task reference misses the most important fact about it http://bugs.python.org/issue21365 opened by pfalcon #21366: Document that return in finally overwrites prev value http://bugs.python.org/issue21366 opened by brandjon #21367: multiprocessing.JoinableQueue requires new kwarg http://bugs.python.org/issue21367 opened by lee.clemens #21368: Check for systemd locale on startup if current locale is set t http://bugs.python.org/issue21368 opened by ncoghlan #21370: segfault from simple traceback.format_exc call http://bugs.python.org/issue21370 opened by John.Rusnak #21372: multiprocessing.util.register_after_fork inconsistency http://bugs.python.org/issue21372 opened by schlamar #21373: robotparser: Automatically call modified function in read() http://bugs.python.org/issue21373 opened by mlorant #21375: Fix and test overflow behavior in the C version of heapq http://bugs.python.org/issue21375 opened by rhettinger #21376: asyncio docs refer to wrong TimeoutError http://bugs.python.org/issue21376 opened by qmega #21381: Python 3.4+ interpreter built on/with OS X 10.7 deployment tar http://bugs.python.org/issue21381 opened by Christian.Tismer #21382: Signal module doesnt raises ValueError Exception http://bugs.python.org/issue21382 opened by rsevcan #21383: "make touch" fails when the build directory is not the source http://bugs.python.org/issue21383 opened by ned.deily #21384: Windows: Make handle non inheritable by default http://bugs.python.org/issue21384 opened by haypo #21385: Compiling modified AST crashes on debug build unless linenumbe http://bugs.python.org/issue21385 opened by Ztane #21386: ipaddress.IPv4Address.is_global not implemented http://bugs.python.org/issue21386 opened by rluethi #21387: Memory leaks when embedded interpreter is reinitialized http://bugs.python.org/issue21387 opened by Evgeniy.Stepanov #21389: The repr of BoundMethod objects sometimes incorrectly identifi http://bugs.python.org/issue21389 opened by Steven.Barker #21390: readline: setlocale() returns NULL on Android http://bugs.python.org/issue21390 opened by lizhenhua.dev #21391: PATCH: using the abspath shortcut in Lib/shutil http://bugs.python.org/issue21391 opened by Yinon #21392: Python on Cygwin: subprocess.call BlockingIOError: [Errno 11] http://bugs.python.org/issue21392 opened by dellair.jie #21393: Python/random.c: close hCryptProv at exit http://bugs.python.org/issue21393 opened by haypo #21396: Python io implementation doesn't flush with write_through=True http://bugs.python.org/issue21396 opened by akira #21397: tempfile.py: Fix grammar in docstring, comment typos http://bugs.python.org/issue21397 opened by modocache #21398: LC_CTYPE=C: pydoc leaves terminal in an unusable state http://bugs.python.org/issue21398 opened by skrah #21400: Code coverage documentation is out-of-date. http://bugs.python.org/issue21400 opened by matrixise #21401: python2 -3 does not warn about str/unicode to bytes conversion http://bugs.python.org/issue21401 opened by Joshua.J.Cogliati #21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists http://bugs.python.org/issue21402 opened by Zero #21403: cElementTree's Element creation handles attrib argument differ http://bugs.python.org/issue21403 opened by santa4nt #21404: Compression level for tarfile/zipfile http://bugs.python.org/issue21404 opened by Sworddragon #21405: Allow using symbols from Unicode block "Superscripts and Subsc http://bugs.python.org/issue21405 opened by rominf #21406: Some socket constants are not enums http://bugs.python.org/issue21406 opened by pitrou #21408: delegation of `!=` to the right-hand side argument is not alwa http://bugs.python.org/issue21408 opened by exarkun #21409: setup.py check - should fail and retrun a non 0 exit code http://bugs.python.org/issue21409 opened by dundeemt #21410: setup.py check --restructuredtext -- appears to pass if docut http://bugs.python.org/issue21410 opened by dundeemt #21411: Enable Treat Warning as Error on 32-bit Windows http://bugs.python.org/issue21411 opened by zach.ware #21412: core dump in PyThreadState_Get when built --with-pymalloc http://bugs.python.org/issue21412 opened by jbeck #21413: urllib.request.urlopen dies on non-basic/digest auth schemes http://bugs.python.org/issue21413 opened by samwyse #21414: Add an intersperse function to itertools http://bugs.python.org/issue21414 opened by javawizard Most recent 15 issues with no replies (15) ========================================== #21414: Add an intersperse function to itertools http://bugs.python.org/issue21414 #21413: urllib.request.urlopen dies on non-basic/digest auth schemes http://bugs.python.org/issue21413 #21411: Enable Treat Warning as Error on 32-bit Windows http://bugs.python.org/issue21411 #21410: setup.py check --restructuredtext -- appears to pass if docut http://bugs.python.org/issue21410 #21397: tempfile.py: Fix grammar in docstring, comment typos http://bugs.python.org/issue21397 #21389: The repr of BoundMethod objects sometimes incorrectly identifi http://bugs.python.org/issue21389 #21386: ipaddress.IPv4Address.is_global not implemented http://bugs.python.org/issue21386 #21385: Compiling modified AST crashes on debug build unless linenumbe http://bugs.python.org/issue21385 #21383: "make touch" fails when the build directory is not the source http://bugs.python.org/issue21383 #21373: robotparser: Automatically call modified function in read() http://bugs.python.org/issue21373 #21372: multiprocessing.util.register_after_fork inconsistency http://bugs.python.org/issue21372 #21366: Document that return in finally overwrites prev value http://bugs.python.org/issue21366 #21360: mailbox.Maildir should ignore files named with a leading dot http://bugs.python.org/issue21360 #21352: improve documentation indexing http://bugs.python.org/issue21352 #21350: bug in file.writelines accepting buffers http://bugs.python.org/issue21350 Most recent 15 issues waiting for review (15) ============================================= #21397: tempfile.py: Fix grammar in docstring, comment typos http://bugs.python.org/issue21397 #21396: Python io implementation doesn't flush with write_through=True http://bugs.python.org/issue21396 #21393: Python/random.c: close hCryptProv at exit http://bugs.python.org/issue21393 #21391: PATCH: using the abspath shortcut in Lib/shutil http://bugs.python.org/issue21391 #21390: readline: setlocale() returns NULL on Android http://bugs.python.org/issue21390 #21386: ipaddress.IPv4Address.is_global not implemented http://bugs.python.org/issue21386 #21383: "make touch" fails when the build directory is not the source http://bugs.python.org/issue21383 #21375: Fix and test overflow behavior in the C version of heapq http://bugs.python.org/issue21375 #21373: robotparser: Automatically call modified function in read() http://bugs.python.org/issue21373 #21362: concurrent.futures does not validate that max_workers is prope http://bugs.python.org/issue21362 #21357: Increase filecmp test coverage from 63% to 76% http://bugs.python.org/issue21357 #21354: PyCFunction_New no longer exposed by python DLL breaking bdist http://bugs.python.org/issue21354 #21350: bug in file.writelines accepting buffers http://bugs.python.org/issue21350 #21347: Don't use a list argument together with shell=True in subproce http://bugs.python.org/issue21347 #21344: save scores or ratios in difflib get_close_matches http://bugs.python.org/issue21344 Top 10 most discussed issues (10) ================================= #21233: Add *Calloc functions to CPython memory allocation API http://bugs.python.org/issue21233 53 msgs #6839: zipfile can't extract file http://bugs.python.org/issue6839 36 msgs #21305: PEP 466: update os.urandom http://bugs.python.org/issue21305 16 msgs #18314: Have os.unlink remove junction points http://bugs.python.org/issue18314 13 msgs #2159: dbmmodule inquiry function is performance prohibitive http://bugs.python.org/issue2159 9 msgs #20962: Rather modest chunk size in gzip.GzipFile http://bugs.python.org/issue20962 9 msgs #21332: subprocess bufsize=1 docs are misleading http://bugs.python.org/issue21332 9 msgs #21398: LC_CTYPE=C: pydoc leaves terminal in an unusable state http://bugs.python.org/issue21398 9 msgs #21037: add an AddressSanitizer build option http://bugs.python.org/issue21037 8 msgs #21403: cElementTree's Element creation handles attrib argument differ http://bugs.python.org/issue21403 8 msgs Issues closed (58) ================== #9291: mimetypes initialization fails on Windows because of non-Latin http://bugs.python.org/issue9291 closed by tim.golden #9307: Py_TPFLAGS_LONG_SUBCLASS is not documented http://bugs.python.org/issue9307 closed by pitrou #9815: assertRaises as a context manager keeps tracebacks and frames http://bugs.python.org/issue9815 closed by pitrou #10872: Add mode to TextIOWrapper repr http://bugs.python.org/issue10872 closed by pitrou #13204: sys.flags.__new__ crashes http://bugs.python.org/issue13204 closed by pitrou #17145: memoryview(array.array) http://bugs.python.org/issue17145 closed by skrah #17386: Bring Doc/make.bat as close to Doc/Makefile as possible http://bugs.python.org/issue17386 closed by python-dev #18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods http://bugs.python.org/issue18185 closed by berker.peksag #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 closed by r.david.murray #18604: Consolidate gui available checks in test.support http://bugs.python.org/issue18604 closed by python-dev #18727: test for writing dictionary rows to CSV http://bugs.python.org/issue18727 closed by pitrou #19001: test_gdb fails on Fedora buildbot http://bugs.python.org/issue19001 closed by skrah #19385: dbm.dumb should be consistent when the database is closed http://bugs.python.org/issue19385 closed by python-dev #19420: Leak in _hashopenssl.c http://bugs.python.org/issue19420 closed by pitrou #19630: marshal.dump cannot write to temporary file http://bugs.python.org/issue19630 closed by tim.golden #19940: ssl.cert_time_to_seconds() returns wrong results if local time http://bugs.python.org/issue19940 closed by pitrou #19962: Create a 'python.bat' script to invoke interpreter from source http://bugs.python.org/issue19962 closed by zach.ware #19977: Use "surrogateescape" error handler for sys.stdin and sys.stdo http://bugs.python.org/issue19977 closed by haypo #20094: intermitent failures with test_dbm http://bugs.python.org/issue20094 closed by ethan.furman #20230: structseq types should expose _fields http://bugs.python.org/issue20230 closed by skrah #20355: -W command line options should have higher priority than PYTHO http://bugs.python.org/issue20355 closed by pitrou #20951: SSLSocket.send() returns 0 for non-blocking socket http://bugs.python.org/issue20951 closed by pitrou #21026: Document sitecustomize.py problems with pythonw http://bugs.python.org/issue21026 closed by python-dev #21040: socketserver: use selectors module http://bugs.python.org/issue21040 closed by neologix #21048: Index 'as' in import and with statements http://bugs.python.org/issue21048 closed by terry.reedy #21055: Index (augmented) assignment symbols http://bugs.python.org/issue21055 closed by python-dev #21057: TextIOWrapper does not support reading bytearrays or memoryvie http://bugs.python.org/issue21057 closed by pitrou #21138: mimetypes.MimeType UnicodeDecodeError http://bugs.python.org/issue21138 closed by tim.golden #21207: urandom persistent fd - not re-openned after fd close http://bugs.python.org/issue21207 closed by pitrou #21225: io.py: Improve docstrings for classes http://bugs.python.org/issue21225 closed by pitrou #21273: don't defined socket constants which are not implemented for G http://bugs.python.org/issue21273 closed by pitrou #21282: setup.py: More informative error msg for modules which built b http://bugs.python.org/issue21282 closed by python-dev #21312: Update thread_foobar.h to include timed locking and TLS suppor http://bugs.python.org/issue21312 closed by pitrou #21316: mark test_devpoll to be meaningful only for Solaris http://bugs.python.org/issue21316 closed by jcea #21320: dict() allows keyword expansion with integer keys, e.g. dict(* http://bugs.python.org/issue21320 closed by eric.smith #21321: itertools.islice() doesn't release reference to the source ite http://bugs.python.org/issue21321 closed by pitrou #21325: Missing Generic EXIF library for images in the standard librar http://bugs.python.org/issue21325 closed by terry.reedy #21340: Possible concurrency bug in asyncio, AttributeError in tasks.p http://bugs.python.org/issue21340 closed by python-dev #21341: Configuring 'font' with ttk.Style for 'TEntry' does not change http://bugs.python.org/issue21341 closed by terry.reedy #21349: crash in winreg SetValueEx with memoryview http://bugs.python.org/issue21349 closed by tim.golden #21351: refcounts not respected at process exit http://bugs.python.org/issue21351 closed by minrk #21353: document Popen.args attribute http://bugs.python.org/issue21353 closed by gregory.p.smith #21355: Shallow defaults to True, not 1 [filecmp.cmp] http://bugs.python.org/issue21355 closed by python-dev #21361: Add how to run a single test case to the devguide http://bugs.python.org/issue21361 closed by python-dev #21369: Extended modes for tarfile.TarFile() http://bugs.python.org/issue21369 closed by lars.gustaebel #21371: struct lconv does not have decimal_point on android platform http://bugs.python.org/issue21371 closed by skrah #21374: DecimalTuple.__module__ is set to _frozen_importlib http://bugs.python.org/issue21374 closed by skrah #21377: PyBytes_Concat could try to concat in-place http://bugs.python.org/issue21377 closed by pitrou #21378: ImportError: No module named 'concurrent.futures' http://bugs.python.org/issue21378 closed by Bill.Bergmann #21379: StopIteration is silenced when raised by generator within cont http://bugs.python.org/issue21379 closed by ncoghlan #21380: timezone support in strftime methods broken http://bugs.python.org/issue21380 closed by ned.deily #21388: decimal.InvalidContext is unused http://bugs.python.org/issue21388 closed by skrah #21394: Lib/random.py: use more efficient code to convert bytes to int http://bugs.python.org/issue21394 closed by haypo #21395: runtests.rst: link "build Python" to setup.rst#compiling http://bugs.python.org/issue21395 closed by python-dev #21399: inspect and class methods http://bugs.python.org/issue21399 closed by yselivanov #21407: Add function signatures to _decimal http://bugs.python.org/issue21407 closed by skrah #21415: Python __new__ method doc typo (it's a class and not a static http://bugs.python.org/issue21415 closed by steven.daprano #1533105: NetBSD build with --with-pydebug causes SIGSEGV http://bugs.python.org/issue1533105 closed by skrah From zachary.ware+pydev at gmail.com Fri May 2 22:06:57 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 2 May 2014 15:06:57 -0500 Subject: [Python-Dev] [Python-checkins] devguide: Fix broken link to Skip's optimizer paper, update bug link In-Reply-To: <3gL49p33dHz7Ll6@mail.python.org> References: <3gL49p33dHz7Ll6@mail.python.org> Message-ID: On Fri, May 2, 2014 at 3:02 PM, zach.ware wrote: > http://hg.python.org/devguide/rev/375b0b0b186b > changeset: 696:375b0b0b186b > user: Zachary Ware > date: Fri May 02 14:44:20 2014 -0500 > summary: > Fix broken link to Skip's optimizer paper, update bug link > > files: > compiler.rst | 4 ++-- > 1 files changed, 2 insertions(+), 2 deletions(-) > > > diff --git a/compiler.rst b/compiler.rst > --- a/compiler.rst > +++ b/compiler.rst > @@ -553,10 +553,10 @@ > .. _SPARK: http://pages.cpsc.ucalgary.ca/~aycock/spark/ > > .. [#skip-peephole] Skip Montanaro's Peephole Optimizer Paper > - (http://www.python.org/workshops/1998-11/proceedings/papers/montanaro/montanaro.html) > + (http://www.smontanaro.net/python/spam7/optimizer.html) Is this a good link or is there a 'better' one available? This was the first result of a Google search for "skip montanaro peephole optimizer" and seems to be straight from the source, so I went with it :) Regards, -- Zach From skip at pobox.com Sat May 3 00:32:19 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 2 May 2014 17:32:19 -0500 Subject: [Python-Dev] [Python-checkins] devguide: Fix broken link to Skip's optimizer paper, update bug link In-Reply-To: References: <3gL49p33dHz7Ll6@mail.python.org> Message-ID: On Fri, May 2, 2014 at 3:06 PM, Zachary Ware wrote: >> - (http://www.python.org/workshops/1998-11/proceedings/papers/montanaro/montanaro.html) >> + (http://www.smontanaro.net/python/spam7/optimizer.html) > > Is this a good link or is there a 'better' one available? Zach, That's as a good a link as I know of. (Lot of water under the bridge since then!) Skip From jessica.mckellar at gmail.com Mon May 5 05:02:22 2014 From: jessica.mckellar at gmail.com (Jessica McKellar) Date: Sun, 4 May 2014 20:02:22 -0700 Subject: [Python-Dev] Python under the sea and in space Message-ID: Hi folks, I'm trying to determine the greatest depth (in the ocean or underground) and highest altitude at which Python code has been executed. Please note that I'm interested in where the code was executed, and not, say, where data that Python analyzed was acquired. I know for example that NASA has analyzed plenty of data from space in Python, but the analysis was done in labs here on Earth. :) Do you have some good candidates? Please let me know! Regards, -Jessica -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon May 5 13:08:41 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 05 May 2014 04:08:41 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.1rc1 Message-ID: <53677139.6010008@hastings.org> On behalf of the Python development community and the Python 3.4 release team, I'm pleased to announce the availability of Python 3.4.1rc1. Python 3.4.1rc1 has over three hundred bugfixes and other improvements over 3.4.0. One notable change: the version of OpenSSL bundled with the Windows installer no longer has the "HeartBleed" vulnerability. You can download it here: https://www.python.org/download/releases/3.4.1 One note: the "topics" data file for the pydoc is broken in this release. This means the pydoc command and the built-in "help" will fail on some topics. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony.tuininga at gmail.com Mon May 5 23:32:46 2014 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 5 May 2014 15:32:46 -0600 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path Message-ID: Hi, I am the author of cx_Freeze which creates "frozen" executables from Python scripts. To this point I have been using frozen modules (compiled C) but this has the side effect of bundling parts of Python with a cx_Freeze release -- and this has bitten me on a number of occasions when newer versions of Python use slightly incompatible modules. :-) By scanning the code I discovered that Python automatically searches on startup /lib/pythonNN.zip where NN is replaced by the major and minor version that is being executed. Using this would allow me to ensure that bootstrap modules are not bundled with cx_Freeze but acquired from the distribution that is actually using it -- thereby eliminating the hassle noted above. I have tested this approach and found it works flawlessly. There is, however, no documentation that I can find for the contents of sys.path -- the documentation simply says that it is an installation default and doesn't specify what that default is. So my question is: can I safely make use of this "feature"? It has remained in place since at least Python 2.6 but I don't want to assume anything. Please advise! Thanks. Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon May 5 23:50:29 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 May 2014 22:50:29 +0100 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On 5 May 2014 22:32, Anthony Tuininga wrote: > So my question is: can I safely make use of this "feature"? It has remained > in place since at least Python 2.6 but I don't want to assume anything. > Please advise! Thanks. I believe this is a 100% supported feature of Python (since 2.6, as you mention). Some of the zip import/run features have unfortunately been underdocumented, and I suspect that this is simply one of those. Paul From anthony.tuininga at gmail.com Mon May 5 23:52:07 2014 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 5 May 2014 15:52:07 -0600 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On Mon, May 5, 2014 at 3:50 PM, Paul Moore wrote: > On 5 May 2014 22:32, Anthony Tuininga wrote: > > So my question is: can I safely make use of this "feature"? It has > remained > > in place since at least Python 2.6 but I don't want to assume anything. > > Please advise! Thanks. > > I believe this is a 100% supported feature of Python (since 2.6, as > you mention). Some of the zip import/run features have unfortunately > been underdocumented, and I suspect that this is simply one of those. > > Paul > That's great news. Thanks! Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Tue May 6 00:10:22 2014 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 5 May 2014 15:10:22 -0700 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On Mon, May 5, 2014 at 2:52 PM, Anthony Tuininga wrote: > On Mon, May 5, 2014 at 3:50 PM, Paul Moore wrote: > >> On 5 May 2014 22:32, Anthony Tuininga wrote: >> > So my question is: can I safely make use of this "feature"? It has >> remained >> > in place since at least Python 2.6 but I don't want to assume anything. >> > Please advise! Thanks. >> >> I believe this is a 100% supported feature of Python (since 2.6, as >> you mention). Some of the zip import/run features have unfortunately >> been underdocumented, and I suspect that this is simply one of those. >> >> Paul >> > > That's great news. Thanks! > > Anthony > There is at least one caveat to using a .zip file for the standard library: http://bugs.python.org/issue19081 Never change the .zip file in use by a running process. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 6 00:16:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 May 2014 08:16:05 +1000 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On 6 May 2014 07:51, "Paul Moore" wrote: > > On 5 May 2014 22:32, Anthony Tuininga wrote: > > So my question is: can I safely make use of this "feature"? It has remained > > in place since at least Python 2.6 but I don't want to assume anything. > > Please advise! Thanks. > > I believe this is a 100% supported feature of Python (since 2.6, as > you mention). Some of the zip import/run features have unfortunately > been underdocumented, and I suspect that this is simply one of those. sys.path initialisation in general is woefully underspecified and thoroughly confusing :( Fixing that is actually one of the motivating forces behind the proposed startup improvements in PEP 432. Cheers, Nick. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony.tuininga at gmail.com Tue May 6 00:14:51 2014 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 5 May 2014 16:14:51 -0600 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: Thanks. I think I can live with that restriction. :-) I do not read/write to the same zip file in the same process. Anthony On Mon, May 5, 2014 at 4:10 PM, Gregory P. Smith wrote: > > On Mon, May 5, 2014 at 2:52 PM, Anthony Tuininga < > anthony.tuininga at gmail.com> wrote: > >> On Mon, May 5, 2014 at 3:50 PM, Paul Moore wrote: >> >>> On 5 May 2014 22:32, Anthony Tuininga >>> wrote: >>> > So my question is: can I safely make use of this "feature"? It has >>> remained >>> > in place since at least Python 2.6 but I don't want to assume anything. >>> > Please advise! Thanks. >>> >>> I believe this is a 100% supported feature of Python (since 2.6, as >>> you mention). Some of the zip import/run features have unfortunately >>> been underdocumented, and I suspect that this is simply one of those. >>> >>> Paul >>> >> >> That's great news. Thanks! >> >> Anthony >> > > There is at least one caveat to using a .zip file for the standard > library: http://bugs.python.org/issue19081 > > Never change the .zip file in use by a running process. > > -gps > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony.tuininga at gmail.com Tue May 6 00:37:31 2014 From: anthony.tuininga at gmail.com (Anthony Tuininga) Date: Mon, 5 May 2014 16:37:31 -0600 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On Mon, May 5, 2014 at 4:16 PM, Nick Coghlan wrote: > > On 6 May 2014 07:51, "Paul Moore" wrote: > > > > On 5 May 2014 22:32, Anthony Tuininga > wrote: > > > So my question is: can I safely make use of this "feature"? It has > remained > > > in place since at least Python 2.6 but I don't want to assume anything. > > > Please advise! Thanks. > > > > I believe this is a 100% supported feature of Python (since 2.6, as > > you mention). Some of the zip import/run features have unfortunately > > been underdocumented, and I suspect that this is simply one of those. > > sys.path initialisation in general is woefully underspecified and > thoroughly confusing :( > > Fixing that is actually one of the motivating forces behind the proposed > startup improvements in PEP 432. > > Cheers, > Nick. > Thanks for the link. I would tend to agree with your assessment. :-) I've had to delve into the startup code a number of times to make cx_Freeze behave itself so any help here would be much appreciated! Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at 3dengineer.com Tue May 6 16:33:41 2014 From: james at 3dengineer.com (James Swift) Date: Tue, 6 May 2014 16:33:41 +0200 Subject: [Python-Dev] Behaviour change of object().format() Message-ID: Hi, In 3.3 I could do the following >>> "{x:s}".format(**{'x': [1, 2, 3]}) '[1, 2, 3]' But in 3.4 >>> "{x:s}".format(**{'x': [1, 2, 3]}) Traceback (most recent call last): File "", line 1, in TypeError: non-empty format string passed to object.__format__ Is this intentional? regards, James Swift -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at 3dengineer.com Tue May 6 16:45:52 2014 From: james at 3dengineer.com (James Swift) Date: Tue, 6 May 2014 16:45:52 +0200 Subject: [Python-Dev] Behaviour change of object().format() in 3.4 Message-ID: Hi, In 3.3 I could do the following >>> "{x:s}".format(**{'x': [1, 2, 3]}) '[1, 2, 3]' But in 3.4 >>> "{x:s}".format(**{'x': [1, 2, 3]}) Traceback (most recent call last): File "", line 1, in TypeError: non-empty format string passed to object.__format__ Is this intentional? regards, James Swift -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue May 6 17:48:16 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 06 May 2014 11:48:16 -0400 Subject: [Python-Dev] Behaviour change of object().format() in 3.4 In-Reply-To: References: Message-ID: <20140506154816.C239C250D5C@webabinitio.net> On Tue, 06 May 2014 16:45:52 +0200, James Swift wrote: > Hi, > > In 3.3 I could do the following > > >>> "{x:s}".format(**{'x': [1, 2, 3]}) > '[1, 2, 3]' > > But in 3.4 > > >>> "{x:s}".format(**{'x': [1, 2, 3]}) > Traceback (most recent call last): > File "", line 1, in > TypeError: non-empty format string passed to object.__format__ > > > Is this intentional? Yes. There was a deprecation warning for this in 3.3, and it is now an error in 3.4. For more information, see http://bugs.python.org/issue7994. --David From eric at trueblade.com Tue May 6 20:40:40 2014 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 06 May 2014 14:40:40 -0400 Subject: [Python-Dev] Behaviour change of object().format() In-Reply-To: References: Message-ID: <53692CA8.6060300@trueblade.com> On 05/06/2014 10:33 AM, James Swift wrote: > Hi, > > In 3.3 I could do the following > >>>> "{x:s}".format(**{'x': [1, 2, 3]}) > '[1, 2, 3]' > > But in 3.4 > >>>> "{x:s}".format(**{'x': [1, 2, 3]}) > Traceback (most recent call last): > File "", line 1, in > TypeError: non-empty format string passed to object.__format__ > > > Is this intentional? Yes. See http://bugs.python.org/issue7994 for the rationale. You can use: >>> "{x!s:15s}".format(**{'x': [1, 2, 3]}) '[1, 2, 3] ' That is, convert it first to a string (with !s) then use whatever string formatting specifier you want (here, '15s'). This should work under all 3.x versions (I think, haven't checked). BTW, for this specific case, you might want to use format_map: >>> "{x!s:15s}".format_map({'x': [1, 2, 3]}) '[1, 2, 3] ' Eric. From tjreedy at udel.edu Tue May 6 19:14:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 06 May 2014 13:14:01 -0400 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: On 5/5/2014 5:32 PM, Anthony Tuininga wrote: > Hi, > > I am the author of cx_Freeze which creates "frozen" executables from > Python scripts. To this point I have been using frozen modules (compiled > C) but this has the side effect of bundling parts of Python with a > cx_Freeze release -- and this has bitten me on a number of occasions > when newer versions of Python use slightly incompatible modules. :-) > > By scanning the code I discovered that Python automatically searches on > startup > > /lib/pythonNN.zip > > where NN is replaced by the major and minor version that is being > executed. Using this would allow me to ensure that bootstrap modules are > not bundled with cx_Freeze but acquired from the distribution that is > actually using it -- thereby eliminating the hassle noted above. > > I have tested this approach and found it works flawlessly. There is, > however, no documentation that I can find for the contents of sys.path > -- the documentation simply says that it is an installation default and > doesn't specify what that default is. > > So my question is: can I safely make use of this "feature"? It has > remained in place since at least Python 2.6 but I don't want to assume > anything. Please advise! Thanks. I would say yes. AFAIK /lib has always by used for the stdlib on Windows and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of the site module doc. We do not lightly change directory structure. PEP 273 then says "The zip archive must be treated exactly as a subdirectory tree,". I take this to mean that lib/pythonxy.zip should work the same as lib/pythonxy/ does. You can depend on the feature working as long as it works. Your bigger worry is accidental regression. I suggest testing release candidates (there is one for 3.4.1 now, I believe) and also an early alpha and beta for new release. The candidate may have a zip patch affecting central directory reading. If not, there is an issue on the tracker. You could monitor patches by filtering the new-issue announcements for 'zip'. -- Terry Jan Reedy From tjreedy at udel.edu Tue May 6 21:21:49 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 06 May 2014 15:21:49 -0400 Subject: [Python-Dev] Python under the sea and in space In-Reply-To: References: Message-ID: On 5/4/2014 11:02 PM, Jessica McKellar wrote: > Hi folks, > > I'm trying to determine the greatest depth (in the ocean or underground) > and highest altitude at which Python code has been executed. > > Please note that I'm interested in where the code was executed, and not, > say, where data that Python analyzed was acquired. I know for example > that NASA has analyzed plenty of data from space in Python, but the > analysis was done in labs here on Earth. :) > > Do you have some good candidates? Please let me know! I suggest posting this on python-list; its readers seem to work with a wide range of applications. -- Terry Jan Reedy From stefan at bytereef.org Tue May 6 23:35:33 2014 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 6 May 2014 23:35:33 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] Message-ID: <20140506213533.GA32765@sleipnir.bytereef.org> Just a warning, in case any of the new packaging team forgot to contact http://cve.mitre.org/ . Stefan Krah From victor.stinner at gmail.com Wed May 7 00:18:24 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 May 2014 00:18:24 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140506213533.GA32765@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> Message-ID: Hi, I don't understand your email. Can you please elaborate? Victor 2014-05-06 23:35 GMT+02:00 Stefan Krah : > Just a warning, in case any of the new packaging team forgot to contact > http://cve.mitre.org/ . > > > Stefan Krah > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From eric at trueblade.com Wed May 7 13:45:11 2014 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 07 May 2014 07:45:11 -0400 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: Message-ID: <536A1CC7.10908@trueblade.com> On 5/6/2014 1:14 PM, Terry Reedy wrote: > On 5/5/2014 5:32 PM, Anthony Tuininga wrote: >> Hi, >> >> I am the author of cx_Freeze which creates "frozen" executables from >> Python scripts. To this point I have been using frozen modules (compiled >> C) but this has the side effect of bundling parts of Python with a >> cx_Freeze release -- and this has bitten me on a number of occasions >> when newer versions of Python use slightly incompatible modules. :-) >> >> By scanning the code I discovered that Python automatically searches on >> startup >> >> /lib/pythonNN.zip >> >> where NN is replaced by the major and minor version that is being >> executed. Using this would allow me to ensure that bootstrap modules are >> not bundled with cx_Freeze but acquired from the distribution that is >> actually using it -- thereby eliminating the hassle noted above. >> >> I have tested this approach and found it works flawlessly. There is, >> however, no documentation that I can find for the contents of sys.path >> -- the documentation simply says that it is an installation default and >> doesn't specify what that default is. >> >> So my question is: can I safely make use of this "feature"? It has >> remained in place since at least Python 2.6 but I don't want to assume >> anything. Please advise! Thanks. > > I would say yes. AFAIK /lib has always by used for the stdlib on Windows > and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of > the site module doc. We do not lightly change directory structure. PEP > 273 then says "The zip archive must be treated exactly as a subdirectory > tree,". I take this to mean that lib/pythonxy.zip should work the same > as lib/pythonxy/ does. > > You can depend on the feature working as long as it works. Your bigger > worry is accidental regression. I suggest testing release candidates > (there is one for 3.4.1 now, I believe) and also an early alpha and beta > for new release. The candidate may have a zip patch affecting central > directory reading. If not, there is an issue on the tracker. You could > monitor patches by filtering the new-issue announcements for 'zip'. The best course of action would be to integrate any such tests into the stdlib test suite, where a regression would immediately be detected. Eric. From tjreedy at udel.edu Wed May 7 19:13:43 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 May 2014 13:13:43 -0400 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: <536A1CC7.10908@trueblade.com> References: <536A1CC7.10908@trueblade.com> Message-ID: On 5/7/2014 7:45 AM, Eric V. Smith wrote: > On 5/6/2014 1:14 PM, Terry Reedy wrote: >> On 5/5/2014 5:32 PM, Anthony Tuininga wrote: >>> Hi, >>> >>> I am the author of cx_Freeze which creates "frozen" executables from >>> Python scripts. To this point I have been using frozen modules (compiled >>> C) but this has the side effect of bundling parts of Python with a >>> cx_Freeze release -- and this has bitten me on a number of occasions >>> when newer versions of Python use slightly incompatible modules. :-) >>> >>> By scanning the code I discovered that Python automatically searches on >>> startup >>> >>> /lib/pythonNN.zip >>> >>> where NN is replaced by the major and minor version that is being >>> executed. Using this would allow me to ensure that bootstrap modules are >>> not bundled with cx_Freeze but acquired from the distribution that is >>> actually using it -- thereby eliminating the hassle noted above. >>> >>> I have tested this approach and found it works flawlessly. There is, >>> however, no documentation that I can find for the contents of sys.path >>> -- the documentation simply says that it is an installation default and >>> doesn't specify what that default is. >>> >>> So my question is: can I safely make use of this "feature"? It has >>> remained in place since at least Python 2.6 but I don't want to assume >>> anything. Please advise! Thanks. >> >> I would say yes. AFAIK /lib has always by used for the stdlib on Windows >> and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of >> the site module doc. We do not lightly change directory structure. PEP >> 273 then says "The zip archive must be treated exactly as a subdirectory >> tree,". I take this to mean that lib/pythonxy.zip should work the same >> as lib/pythonxy/ does. >> >> You can depend on the feature working as long as it works. Your bigger >> worry is accidental regression. I suggest testing release candidates >> (there is one for 3.4.1 now, I believe) and also an early alpha and beta >> for new release. The candidate may have a zip patch affecting central >> directory reading. If not, there is an issue on the tracker. You could >> monitor patches by filtering the new-issue announcements for 'zip'. > > The best course of action would be to integrate any such tests into the > stdlib test suite, where a regression would immediately be detected. We do not and should not put tests of 3rd party products that we do not distribute ourselves into the stdlib test suite. Anthony should test *his* pythonxy.dll himself. To the extent feasible, we should test the promised behavior of python.exe regardless of what any 3rd party developer does. If pythonxy.zip supplements rather than replacing pythonxy/*.py, this should be easy enough: copy .zip into place, run python in a subprocess, check the result, delete .zip. -- Terry Jan Reedy From eric at trueblade.com Wed May 7 20:02:19 2014 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 07 May 2014 14:02:19 -0400 Subject: [Python-Dev] Existence of pythonNN.zip in sys.path In-Reply-To: References: <536A1CC7.10908@trueblade.com> Message-ID: <536A752B.8030506@trueblade.com> On 05/07/2014 01:13 PM, Terry Reedy wrote: > On 5/7/2014 7:45 AM, Eric V. Smith wrote: >> On 5/6/2014 1:14 PM, Terry Reedy wrote: >>> On 5/5/2014 5:32 PM, Anthony Tuininga wrote: >>>> Hi, >>>> >>>> I am the author of cx_Freeze which creates "frozen" executables from >>>> Python scripts. To this point I have been using frozen modules >>>> (compiled >>>> C) but this has the side effect of bundling parts of Python with a >>>> cx_Freeze release -- and this has bitten me on a number of occasions >>>> when newer versions of Python use slightly incompatible modules. :-) >>>> >>>> By scanning the code I discovered that Python automatically searches on >>>> startup >>>> >>>> /lib/pythonNN.zip >>>> >>>> where NN is replaced by the major and minor version that is being >>>> executed. Using this would allow me to ensure that bootstrap modules >>>> are >>>> not bundled with cx_Freeze but acquired from the distribution that is >>>> actually using it -- thereby eliminating the hassle noted above. >>>> >>>> I have tested this approach and found it works flawlessly. There is, >>>> however, no documentation that I can find for the contents of sys.path >>>> -- the documentation simply says that it is an installation default and >>>> doesn't specify what that default is. >>>> >>>> So my question is: can I safely make use of this "feature"? It has >>>> remained in place since at least Python 2.6 but I don't want to assume >>>> anything. Please advise! Thanks. >>> >>> I would say yes. AFAIK /lib has always by used for the stdlib on Windows >>> and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of >>> the site module doc. We do not lightly change directory structure. PEP >>> 273 then says "The zip archive must be treated exactly as a subdirectory >>> tree,". I take this to mean that lib/pythonxy.zip should work the same >>> as lib/pythonxy/ does. >>> >>> You can depend on the feature working as long as it works. Your bigger >>> worry is accidental regression. I suggest testing release candidates >>> (there is one for 3.4.1 now, I believe) and also an early alpha and beta >>> for new release. The candidate may have a zip patch affecting central >>> directory reading. If not, there is an issue on the tracker. You could >>> monitor patches by filtering the new-issue announcements for 'zip'. >> >> The best course of action would be to integrate any such tests into the >> stdlib test suite, where a regression would immediately be detected. > > We do not and should not put tests of 3rd party products that we do not > distribute ourselves into the stdlib test suite. Anthony should test > *his* pythonxy.dll himself. To the extent feasible, we should test the > promised behavior of python.exe regardless of what any 3rd party > developer does. > > If pythonxy.zip supplements rather than replacing pythonxy/*.py, this > should be easy enough: copy .zip into place, run python in a subprocess, > check the result, delete .zip. I'm sorry, I misread this. I thought the question was about putting the stdlib itself in pythonxy.zip. I agree we shouldn't test someone else using that filename. Eric. From stefan at bytereef.org Thu May 8 14:12:02 2014 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 8 May 2014 14:12:02 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> Message-ID: <20140508121201.GA28465@sleipnir.bytereef.org> Victor Stinner wrote: > I don't understand your email. Can you please elaborate? There is nothing wrong with the package. The remark is a joke provoked by a long history of a campaign [1] against external packages on distutils-sig. Many tools (like crate.io, when it was still up) have made derogatory remarks about external packages. Now the latest version of the officially sanctioned download tool (pip) spits out copious warnings, one of which is the subject of this thread. External packages are being singled out unfairly: 1) Anyone can upload any package to PyPI (i.e. the index is not curated at all). 2) Last time I looked, access credentials (via "lost login") were sent out in plaintext. 3) AFAIK people can upload a different (malicious) version of a package with *the exact same name*. 4) pip generally downloads the latest version, so a malicious person can provide a good package for several years until people trust him, then change to a trojaned version. 5) Looking at the list of certificates that is in my default cert store, I don't find SSL trustworthy at all. 6) D.J. Bernstein, who is somewhat security minded, has been shipping his software *for years* with just plain HTTP and published checksums. To sum it up: 1) Don't use pip to install packages directly from PyPI if security really matters. 2) The best security we currently get is either a) with package signatures (*if* you can get the author's key via a trustworthy channel, which is rarely the case). b) with decent checksums that are recorded on public mailing lists at the time the package is announced (it would be hard for an attacker to modify all mailing list archives.) Whether a package is internal or external is orthogonal to both points. With all these points, I find it questionable for an "official" install tool to make security related remarks about just one category of weaknesses. After all, people might be led to believe that pip is some sort of apt-get and all uploaded packages are safe. Stefan Krah [1] Note that the joke is quite innocent in comparison to what I've read on distutils-sig about the subject. From donald at stufft.io Thu May 8 14:52:36 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 08:52:36 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508121201.GA28465@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> Message-ID: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> On May 8, 2014, at 8:12 AM, Stefan Krah wrote: > Victor Stinner wrote: >> I don't understand your email. Can you please elaborate? > > There is nothing wrong with the package. The remark is a joke provoked by > a long history of a campaign [1] against external packages on distutils-sig. > > Many tools (like crate.io, when it was still up) have made derogatory > remarks about external packages. Now the latest version of the officially > sanctioned download tool (pip) spits out copious warnings, one of which > is the subject of this thread. pip doesn't install externally hosted packaged by default because people should be aware of what servers they depend on. Installs that depend on externally hosted packages are brittle and more prone to failure. Every single external server adds *another* SPOF into any particular install set. Even if every external server has a 99.9% uptime, when you combine multiple of them the total uptime of any particular set of requirements drops quickly. This has been a major complaint that people have had over time. However they are not a security issue so they are easy to turn back on globally if a person wishes to make their own deployment more brittle (--allow-all-external). However in addition to externally hosted packages, there are also unverifiable packages. These are packages where there is no path to verify the download. Generally an unverifiable package is one that either is linked to directly but has no hash information encoded in the URL in a way that pip understands or, the more common case, it is a package that is linked from a page that is linked too on PyPI. Since we cannot build a path of trust to verify the package it is trivial for an attacker to MITM it and execute arbitrary Python code on the end users machine. This case *is* a security issue and pip goes out of it's way to *greatly* discourage depending on packages that require this kind of install. > > > External packages are being singled out unfairly: > > 1) Anyone can upload any package to PyPI (i.e. the index is not curated > at all). Sure, the threat model of PyPI isn't that you can install anything willy nilly and be safe. It's that when you ask for package X version Y, you should get what the *author* published and not what attacker who happens to be in a position to MITM you sends you. > > 2) Last time I looked, access credentials (via "lost login") were sent > out in plaintext. The existence of other security issues is not an excuse to not fix a security issue. There are other problems and we're slowly working on trying to clear them out. > > 3) AFAIK people can upload a different (malicious) version of a > package with *the exact same name*. Yes, a malicious author is needfully outside of the threat model for PyPI/pip. > > 4) pip generally downloads the latest version, so a malicious person > can provide a good package for several years until people trust > him, then change to a trojaned version. Yes, a malicious author is needfully outside of the threat model for PyPI/pip. > > 5) Looking at the list of certificates that is in my default cert > store, I don't find SSL trustworthy at all. TLS may not be the best method of authenticating package downloads, however it is an easy one and it is significantly better than having no authentication at all. You move your attack surface from "anyone who has a privileged network position" to "anyone who has a privileged network position and also has the ability to generate erroneously signed CA certificates". That is a significantly smaller attack surface. > > 6) D.J. Bernstein, who is somewhat security minded, has been shipping > his software *for years* with just plain HTTP and published checksums. Argument from authority doesn't really hold up very well when DJB doesn't distribute is software in a way that is intended to be consumed mechanically. Also while he may be a crypto expert he is far from an expert on successfully distributing software, unless you also think that the signature checking in most OS provided package managers is pointless. > > > To sum it up: > > 1) Don't use pip to install packages directly from PyPI if security > really matters. ?security? isn?t a useful term on it?s own. You have to define a threat model. A better answer here is don?t use pip to install packages directly from PyPI if a malicious author is within your threat model (modulo other issue we?re still working on, but as I said above, the existence of a second security issue isn?t a reason not to fix the first issue). > > 2) The best security we currently get is either > > a) with package signatures (*if* you can get the author's key via > a trustworthy channel, which is rarely the case). > > b) with decent checksums that are recorded on public mailing > lists at the time the package is announced (it would be > hard for an attacker to modify all mailing list archives.) > > Whether a package is internal or external is orthogonal to both points. Again as I said above, the internal vs external isn?t a security related decision it is a ?uptime? related decision. Verifiable vs unverifiable *is* a security related decision. > > > With all these points, I find it questionable for an "official" install > tool to make security related remarks about just one category of weaknesses. > > After all, people might be led to believe that pip is some sort of apt-get > and all uploaded packages are safe. > > > Stefan Krah > > > > [1] Note that the joke is quite innocent in comparison to what I've read on > distutils-sig about the subject. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Thu May 8 15:39:53 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 May 2014 15:39:53 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> Message-ID: <536B8929.6040407@egenix.com> Well, to be fair and leaving aside uptime concerns and the general desire to always install packages from some server instead of a safe and trusted local directory (probably too obvious ;-), it would certainly be possible to add support for trusted externally hosted packages. However, for some reason there's a strong resistance against doing this, which I frankly don't understand. I agree with Stefan that the warning message wording is less than ideal. You'd normally call such blanket statements FUD, esp. since there are plenty external hosting services which are reliable and safe to use. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rosuav at gmail.com Thu May 8 15:55:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 May 2014 23:55:05 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536B8929.6040407@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> Message-ID: On Thu, May 8, 2014 at 11:39 PM, M.-A. Lemburg wrote: > I agree with Stefan that the warning message wording is less > than ideal. You'd normally call such blanket statements FUD, > esp. since there are plenty external hosting services which > are reliable and safe to use. No, it's not FUD. Every external dependency adds another thing that can fail. I was just arguing this with one of my younger brothers this evening; he had some stuff stored in Google Docs, and he couldn't access it because of some outage. With a local git/hg repository, he would have had all his data right there, and even losing access to a central hub server just means you can't pull/push. Using someone else's server makes you depend on that server and everything in between. Obviously that's often a cost worth paying (hey, I'm posting this from Gmail, so I completely lose access to all these emails if it's inaccessible), but it's a legitimate concern, not FUD. (That doesn't stop the wording from being "less than ideal", though.) ChrisA From ncoghlan at gmail.com Thu May 8 15:57:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 May 2014 23:57:26 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536B8929.6040407@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> Message-ID: On 8 May 2014 23:39, M.-A. Lemburg wrote: > However, for some reason there's a strong resistance against > doing this, which I frankly don't understand. Because we're taking responsibility for the end-to-end user experience of PyPI, and are expressly trying to eliminate the elements of that user experience that are beyond the control of the PyPI admin team (even the question of "does this software actually work?" is in our sights if you consider a long enough time span). That's hard enough with just a couple of service providers (Fastly and Rackspace) in the mix - it quickly becomes impossible if every new dependency from an installation request adds a new point of failure. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu May 8 15:58:08 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 09:58:08 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536B8929.6040407@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> Message-ID: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: > Well, to be fair and leaving aside uptime concerns and the general > desire to always install packages from some server instead of > a safe and trusted local directory (probably too obvious ;-), > it would certainly be possible to add support for > trusted externally hosted packages. There is support for trusted externally hosted packages, you put the URL in PyPI and include a hash in the fragment like so: http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 The hash can be md5 or any of the sha-2 family. Now this does not mean that ``pip install cdecimal`` will automatically install this, because whether or not you're willing to install from servers other than PyPI[1] is a policy decision for the end user of pip. The only real contention point there is whether installing from other servers should be on or off by default. PEP438 selected off by default, and I agree with that decision. Installing externally hosted files, which are able to be safely downloaded[2], was a surprising behavior to *everyone* I've talked to who hadn't already discovered that pip/easy_install did that. For the people it wasn't surprising too, they said it was surprising when they had originally discovered it[3]. [1] To be specific, other than the configured index(es), which happens to default to PyPI. [2] For the definition of safe that PyPI/pip operate under, which is that the author of a package is assumed to be trusted by the person electing to download their package. [3] I suspect people who were around when PyPI *couldn't* host files and were only an index would be the exception to this. > > However, for some reason there's a strong resistance against > doing this, which I frankly don't understand. > > I agree with Stefan that the warning message wording is less > than ideal. You'd normally call such blanket statements FUD, > esp. since there are plenty external hosting services which > are reliable and safe to use. > I don't think the warning is FUD, and it doesn't mention anything security related at all. The exact text of the warning is in the subject of the email here: cdecimal an externally hosted file and may be unreliable Which is true as far as I can tell, it is externally hosted, and it may be unreliable[1]. If there is a better wording for that I?m happy to have it and will gladly commit it myself to pip. [1] In my experience dealing with complaints of pip's users, one of their big ones was that some dependency they use was, typically unknown to them, hosted externally and they found out it was hosted externally because the server it was hosted on went down. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 8 16:10:35 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 10:10:35 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> Message-ID: On May 8, 2014, at 9:58 AM, Donald Stufft wrote: > Now this does not mean that ``pip install cdecimal`` will automatically install > this, because whether or not you're willing to install from servers other than > PyPI[1] is a policy decision for the end user of pip. I forgot to add, for externally hosted files that are able to be safely downloaded pip's users can either elect to use any/all of them via: $ pip install --allow-all-external cdecimal $ PIP_ALLOW_ALL_EXTERNAL=1 pip install cdecimal $ echo "--allow-all-external\ncdecimal" > requirements.txt $ pip install -r requirements.txt $ echo "[global]\nallow-allow-external=true" > ~/.pip/pip.conf $ pip install cdecimal They can also elect to allow externally hosted files for *only* cdecimal using: $ pip install --allow-external cdecimal decimal $ PIP_ALLOW_EXTERNAL=cdecimal pip install cdecimal $ echo "--allow-external decimal\ncdecimal" > requirements.txt $ pip install -r requirements.txt $ echo "[global]\nallow-external=decimal" > ~/.pip/pip.conf $ pip install cdecimal I may have the syntax on the non command flag options slightly wrong, those are auto generated for us in our option system and I don't personally use those options. Also the reason for the extra verbosity in ``--allow-external`` is so you can allow it on something you don't directly depend on, like: $ pip install --allow-external cdecimal foobar-which-depends-on-cdecimal ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rdmurray at bitdance.com Thu May 8 16:11:39 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 08 May 2014 10:11:39 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> Message-ID: <20140508141140.4AEB1250DBB@webabinitio.net> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I???m happy to have it and > will gladly commit it myself to pip. > > [1] In my experience dealing with complaints of pip's users, one of their big > ones was that some dependency they use was, typically unknown to them, > hosted externally and they found out it was hosted externally because the > server it was hosted on went down. "unreliable" reads as "not safe", ie: insecure. You probably want something like "and access to it may be unreliable". --David From donald at stufft.io Thu May 8 16:21:28 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 10:21:28 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508141140.4AEB1250DBB@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> Message-ID: <54BDABBF-D8D9-4498-86E6-56F7F86C0A57@stufft.io> On May 8, 2014, at 10:11 AM, R. David Murray wrote: > On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: >> I don't think the warning is FUD, and it doesn't mention anything security >> related at all. The exact text of the warning is in the subject of the email >> here: >> >> cdecimal an externally hosted file and may be unreliable >> >> Which is true as far as I can tell, it is externally hosted, and it may be >> unreliable[1]. If there is a better wording for that I?m happy to have it and >> will gladly commit it myself to pip. >> >> [1] In my experience dealing with complaints of pip's users, one of their big >> ones was that some dependency they use was, typically unknown to them, >> hosted externally and they found out it was hosted externally because the >> server it was hosted on went down. > > "unreliable" reads as "not safe", ie: insecure. > > You probably want something like "and access to it may be unreliable". > > --David Done: https://github.com/pypa/pip/commit/69bf7067 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rdmurray at bitdance.com Thu May 8 16:21:34 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 08 May 2014 10:21:34 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508141140.4AEB1250DBB@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> Message-ID: <20140508142135.09EE0250DBC@webabinitio.net> On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" wrote: > On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: > > I don't think the warning is FUD, and it doesn't mention anything security > > related at all. The exact text of the warning is in the subject of the email > > here: > > > > cdecimal an externally hosted file and may be unreliable > > > > Which is true as far as I can tell, it is externally hosted, and it may be > > unreliable[1]. If there is a better wording for that I??????m happy to have it and > > will gladly commit it myself to pip. > > > > [1] In my experience dealing with complaints of pip's users, one of their big > > ones was that some dependency they use was, typically unknown to them, > > hosted externally and they found out it was hosted externally because the > > server it was hosted on went down. > > "unreliable" reads as "not safe", ie: insecure. > > You probably want something like "and access to it may be unreliable". Actually, thinking about this some more, *most* end-users aren't going to care that there's another point of failure here, they only care if it works or not when they try to install it. So something like "cdecimal is not hosted on pypi; download may fail if external server is unavailable" might be clearer. And once you're at that point, as a user I'm going to grumble, "Well, why the heck didn't you just try?", as I figure out how to re-execute the command so that it does try. --David From stephane at wirtel.be Thu May 8 16:17:46 2014 From: stephane at wirtel.be (=?utf-8?q?St=C3=A9phane?= Wirtel) Date: Thu, 08 May 2014 16:17:46 +0200 Subject: [Python-Dev] EuroPython CPython Sprint? Message-ID: Hi all, What do you think about a CPython sprint at EuroPython 2014? Regards, Stephane -- St?phane Wirtel - http://wirtel.be - @matrixise From solipsis at pitrou.net Thu May 8 16:31:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 May 2014 16:31:52 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> Message-ID: <20140508163152.4b554316@fsol> On Thu, 08 May 2014 10:21:34 -0400 "R. David Murray" wrote: > > > > "unreliable" reads as "not safe", ie: insecure. > > > > You probably want something like "and access to it may be unreliable". > > Actually, thinking about this some more, *most* end-users aren't going > to care that there's another point of failure here, they only care if it > works or not when they try to install it. So something like > "cdecimal is not hosted on pypi; download may fail if external server > is unavailable" might be clearer. > > And once you're at that point, as a user I'm going to grumble, "Well, why > the heck didn't you just try?", as I figure out how to re-execute the > command so that it does try. Agreed. That warning looks rather pointless and only aimed at trying to enforce the pip developers' ideological preferences. Regards Antoine. From bcannon at gmail.com Thu May 8 16:33:57 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 08 May 2014 14:33:57 +0000 Subject: [Python-Dev] EuroPython CPython Sprint? References: Message-ID: On Thu May 08 2014 at 10:25:44 AM, St?phane Wirtel wrote: > Hi all, > > What do you think about a CPython sprint at EuroPython 2014? > Great, although I think that answer would be considered obvious since there is no real negative to holding sprints. =) Are you indirectly asking if anyone plans to lead such a sprint? -Brett > > Regards, > > Stephane > -- > St?phane Wirtel - http://wirtel.be - @matrixise > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Thu May 8 16:36:50 2014 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 8 May 2014 16:36:50 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> Message-ID: <20140508143650.GA29239@sleipnir.bytereef.org> Donald Stufft wrote: > There is support for trusted externally hosted packages, you put the URL in > PyPI and include a hash in the fragment like so: > > http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 That is exactly the mode I was using until today. This mode produced the subject's warning message. Today I've switched to manual install mode with manual sha256sum verification which is *far* safer than anything you get via pip right now. > [2] For the definition of safe that PyPI/pip operate under, which is that the > author of a package is assumed to be trusted by the person electing to > download their package. No, there are other holes, which you have conceded in your previous mail. > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I?m happy to have it and > will gladly commit it myself to pip. Do you honestly not see a difference between the cited warning and the *intended* warning "the server's availability may be unreliable"? Even the latter is FUD or a truism (it applies to any server). The real question is: Why is there a warning if the person running pip has explicitly allowed external packages? Stefan Krah From donald at stufft.io Thu May 8 16:37:15 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 10:37:15 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508142135.09EE0250DBC@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> Message-ID: <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> On May 8, 2014, at 10:21 AM, R. David Murray wrote: > On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" wrote: >> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft wrote: >>> I don't think the warning is FUD, and it doesn't mention anything security >>> related at all. The exact text of the warning is in the subject of the email >>> here: >>> >>> cdecimal an externally hosted file and may be unreliable >>> >>> Which is true as far as I can tell, it is externally hosted, and it may be >>> unreliable[1]. If there is a better wording for that I???m happy to have it and >>> will gladly commit it myself to pip. >>> >>> [1] In my experience dealing with complaints of pip's users, one of their big >>> ones was that some dependency they use was, typically unknown to them, >>> hosted externally and they found out it was hosted externally because the >>> server it was hosted on went down. >> >> "unreliable" reads as "not safe", ie: insecure. >> >> You probably want something like "and access to it may be unreliable". > > Actually, thinking about this some more, *most* end-users aren't going > to care that there's another point of failure here, they only care if it > works or not when they try to install it. So something like > "cdecimal is not hosted on pypi; download may fail if external server > is unavailable" might be clearer. > > And once you're at that point, as a user I'm going to grumble, "Well, why > the heck didn't you just try?", as I figure out how to re-execute the > command so that it does try. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Most users are not going to care up until the point where the external server is unavailable, and then they care a whole lot. On the tin it sounds reasonable to just download the external file if the server is up however we've done that for a long time and the end result has been end user pain. Now requiring someone to add a flag in order to download an externally hosted file is also end user pain. The difference between the two pains is when they happen. The requiring a flag pain happens at the point of decision, when you decide to make your deployment depend on something hosted externally. The default to allow pain happens sometime in the future, when you may or may not have any idea why suddenly your installs aren't working (and when you look, PyPI is up so you're really very confused why this particular file doesn't work). Even worse is the case when a project has some old files, but the newer versions aren't hosted and suddenly people are getting very old releases which is even more confusing to the end users. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 8 16:38:38 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 10:38:38 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508163152.4b554316@fsol> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> <20140508163152.4b554316@fsol> Message-ID: <0ACCF3E1-E5CE-438C-8279-2B1C1FABE1DF@stufft.io> On May 8, 2014, at 10:31 AM, Antoine Pitrou wrote: > On Thu, 08 May 2014 10:21:34 -0400 > "R. David Murray" wrote: >>> >>> "unreliable" reads as "not safe", ie: insecure. >>> >>> You probably want something like "and access to it may be unreliable". >> >> Actually, thinking about this some more, *most* end-users aren't going >> to care that there's another point of failure here, they only care if it >> works or not when they try to install it. So something like >> "cdecimal is not hosted on pypi; download may fail if external server >> is unavailable" might be clearer. >> >> And once you're at that point, as a user I'm going to grumble, "Well, why >> the heck didn't you just try?", as I figure out how to re-execute the >> command so that it does try. > > Agreed. That warning looks rather pointless and only aimed at trying to > enforce the pip developers' ideological preferences. > > Regards > > Antoine. > The pip developers didn?t make this decision. It was discussed on distutils-sig hammered out in a PEP, and then accepted. We took part in that discussion, but ultimately we implemented PEP438. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Thu May 8 16:42:42 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 May 2014 16:42:42 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> Message-ID: <536B97E2.8070805@egenix.com> On 08.05.2014 15:58, Donald Stufft wrote: > > On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: > >> Well, to be fair and leaving aside uptime concerns and the general >> desire to always install packages from some server instead of >> a safe and trusted local directory (probably too obvious ;-), >> it would certainly be possible to add support for >> trusted externally hosted packages. > > There is support for trusted externally hosted packages, you put the URL in > PyPI and include a hash in the fragment like so: > > http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 > > The hash can be md5 or any of the sha-2 family. > > Now this does not mean that ``pip install cdecimal`` will automatically install > this, because whether or not you're willing to install from servers other than > PyPI[1] is a policy decision for the end user of pip. Hmm, if you call that feature "trusted externally hosted packages", pip should really do trust them, right ? ;-) I can understand that pip defaults to not trusting URLs which don't meet the above feature requirements, but not that it still warns about unreliable externally hosted packages even if the above feature is used. At the moment, pip will refuse to use an externally hosted files even if the package author uses the above hashed URLs; even with HTTPS and proper SSL certificate chain. > The only real contention > point there is whether installing from other servers should be on or off by > default. PEP438 selected off by default, and I agree with that decision. > Installing externally hosted files, which are able to be safely downloaded[2], > was a surprising behavior to *everyone* I've talked to who hadn't already > discovered that pip/easy_install did that. For the people it wasn't surprising > too, they said it was surprising when they had originally discovered it[3]. > > [1] To be specific, other than the configured index(es), which happens to > default to PyPI. > [2] For the definition of safe that PyPI/pip operate under, which is that the > author of a package is assumed to be trusted by the person electing to > download their package. > [3] I suspect people who were around when PyPI *couldn't* host files and were > only an index would be the exception to this. > >> >> However, for some reason there's a strong resistance against >> doing this, which I frankly don't understand. >> >> I agree with Stefan that the warning message wording is less >> than ideal. You'd normally call such blanket statements FUD, >> esp. since there are plenty external hosting services which >> are reliable and safe to use. >> > > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I?m happy to have it and > will gladly commit it myself to pip. The current version of pip writes: Downloading/unpacking pkg Could not find any downloads that satisfy the requirement pkg Some externally hosted files were ignored (use --allow-external pkg to allow). Cleaning up... No distributions at all found for pkg This wording if fine, IMO. The wording Stefan quoted gets generated for dependencies. This should probably be changed to the same wording (including the reference to the right command line option to use). > [1] In my experience dealing with complaints of pip's users, one of their big > ones was that some dependency they use was, typically unknown to them, > hosted externally and they found out it was hosted externally because the > server it was hosted on went down. I think that's a general problem, not one of some server being down: users put too much trust into the dependencies of packages they use. Regardless of how safe/reliable we make things w/r to file hosting, this problem does not go away. It's just too easy for people to get tricked into trusting packages they don't even know about. Nothing we'll ever change, though. People are lazy and easily drop all such concerns for ease of use :-( -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Thu May 8 16:52:46 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 May 2014 16:52:46 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> Message-ID: <536B9A3E.9030905@egenix.com> On 08.05.2014 15:57, Nick Coghlan wrote: > On 8 May 2014 23:39, M.-A. Lemburg wrote: >> However, for some reason there's a strong resistance against >> doing this, which I frankly don't understand. > > Because we're taking responsibility for the end-to-end user experience > of PyPI, and are expressly trying to eliminate the elements of that > user experience that are beyond the control of the PyPI admin team Oh, I guess you'd have to rewrite most of those 40k packages then :-) Seriously, the word "eliminate" in there does not sit well with our goals for openness. External services like github, sourceforge, bitbucket, dropbox, cdns, etc. are not per-se evil and unreliable. pip should acknowledge this and not try to "eliminate" all hosting services in the world per default [sound of Empire Strikes Back theme] ;-) > (even the question of "does this software actually work?" is in our > sights if you consider a long enough time span). That's hard enough > with just a couple of service providers (Fastly and Rackspace) in the > mix - it quickly becomes impossible if every new dependency from an > installation request adds a new point of failure. Like I said: the best option is to use a local directory which only contains packages files that you have inspected and actually trust :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Thu May 8 16:57:18 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 10:57:18 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508143650.GA29239@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> Message-ID: <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> On May 8, 2014, at 10:36 AM, Stefan Krah wrote: > Donald Stufft wrote: >> There is support for trusted externally hosted packages, you put the URL in >> PyPI and include a hash in the fragment like so: >> >> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 > > That is exactly the mode I was using until today. This mode produced the > subject's warning message. > > Today I've switched to manual install mode with manual sha256sum verification > which is *far* safer than anything you get via pip right now. It is not safer in any meaingful way. If someone is in a position to compromise the integrity of PyPI's TLS, they can replace the hash on that page with something else. Now you've attempted to work around this by telling people to go look up the release announcement hash. However if someone can compromise the integrity of PyPI's TLS, they can also compromise the integrity of https://mail.python.org/, or GMane, or any other TLS based website[1]. All of that assumes that the end user is going to bother to verify the hash *at all* which almost none of them will and they'll just check the http url into their requirements.txt file and be downloading things over HTTP and be vulnerable to arbitrary code execution via MITM. > > >> [2] For the definition of safe that PyPI/pip operate under, which is that the >> author of a package is assumed to be trusted by the person electing to >> download their package. > > No, there are other holes, which you have conceded in your previous mail. The presence of other holes is not a useful argument to avoid closing a hole. We're working to close all of them, and that ends up meaning we close one at a time. > > >> I don't think the warning is FUD, and it doesn't mention anything security >> related at all. The exact text of the warning is in the subject of the email >> here: >> >> cdecimal an externally hosted file and may be unreliable >> >> Which is true as far as I can tell, it is externally hosted, and it may be >> unreliable[1]. If there is a better wording for that I?m happy to have it and >> will gladly commit it myself to pip. > > Do you honestly not see a difference between the cited warning and the > *intended* warning "the server's availability may be unreliable?? Do I? No I don?t. However I?ve since adjusted the message based on R David Murray?s feedback to make sure it specifically says that access may be unreliable instead of just that the package itself may be unreliable. > > Even the latter is FUD or a truism (it applies to any server). No, because the use of an external host *adds* additional unreliability. If PyPI is down, then all packages are down, including ones that host externally. If the cdecimal server is down, then that one specific package is unavailable. It is impossible to do anything but reduce the overall availability by adding additional SPOFs. > > The real question is: Why is there a warning if the person running pip > has explicitly allowed external packages? > Why is there a warning? Originally that warning was there because the first part of the transition to this "mode" of defaults was to give an option to disable externally hosted files, but leave it on by default. In this phase we gave this warning to tell the people who just leave things to their default about this file. Should the warning itself still exist? I don't know, but a better avenue for asking for a change in pip is via our issue tracker instead of whining on python-dev. > Stefan Krah > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stephane at wirtel.be Thu May 8 16:59:44 2014 From: stephane at wirtel.be (=?utf-8?q?St=C3=A9phane?= Wirtel) Date: Thu, 08 May 2014 16:59:44 +0200 Subject: [Python-Dev] EuroPython CPython Sprint? In-Reply-To: References: Message-ID: <0B3881FC-46E7-40BB-875F-79B1AD7225F4@wirtel.be> On 8 May 2014, at 16:33, Brett Cannon wrote: > On Thu May 08 2014 at 10:25:44 AM, St?phane Wirtel > wrote: > >> Hi all, >> >> What do you think about a CPython sprint at EuroPython 2014? >> > > Great, although I think that answer would be considered obvious since there > is no real negative to holding sprints. =) Are you indirectly asking if > anyone plans to lead such a sprint? Yes, I do, I can make the request to the EuroPython team for the sprint but: * I am not a Python Core Dev * I can't find the right issues for the sprinters * I can't help the sprinters with the issues. * I am a newbie in the CPython code Thus, who would be interested in a CPython sprint? > > -Brett St?phane, -- St?phane Wirtel - http://wirtel.be - @matrixise From ncoghlan at gmail.com Thu May 8 17:05:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 May 2014 01:05:19 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536B9A3E.9030905@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <536B9A3E.9030905@egenix.com> Message-ID: On 9 May 2014 00:52, "M.-A. Lemburg" wrote: > > On 08.05.2014 15:57, Nick Coghlan wrote: > > > (even the question of "does this software actually work?" is in our > > sights if you consider a long enough time span). That's hard enough > > with just a couple of service providers (Fastly and Rackspace) in the > > mix - it quickly becomes impossible if every new dependency from an > > installation request adds a new point of failure. > > Like I said: the best option is to use a local directory which > only contains packages files that you have inspected and > actually trust :-) Correct, but that raises the barrier to entry too high. The pip defaults are aimed at providing an experience with the fewest points of failure that is currently achievable, with a minimal learning curve. We still have a long way to go, but if people want to influence those design decisions, the relevant lists are pypa-dev (for pip specific discussions) and distutils-sig (for higher level cross-tool design decisions) Cheers, Nick. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 08 2014) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Thu May 8 17:19:07 2014 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 8 May 2014 17:19:07 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> Message-ID: <20140508151906.GA29543@sleipnir.bytereef.org> Donald Stufft wrote: > hosted packages are brittle and more prone to failure. Every single external > server adds *another* SPOF into any particular install set. Even if every > external server has a 99.9% uptime, when you combine multiple of them the total > uptime of any particular set of requirements drops quickly. This has been a > major complaint that people have had over time. We have been through that many times; to me this is an indication that people are using pip under circumstances when they should not. pip is not apt-get. [I am aware that *you* know that, just stating it again for the broader audience.] > > 2) Last time I looked, access credentials (via "lost login") were sent > > out in plaintext. > > The existence of other security issues is not an excuse to not fix a security > issue. There are other problems and we're slowly working on trying to clear > them out. It is, however, a reason to avoid error messages that could *imply* (rightly or wrongly) to users that the combination of pip and internal packages is safe. > > 3) AFAIK people can upload a different (malicious) version of a > > package with *the exact same name*. > > Yes, a malicious author is needfully outside of the threat model for PyPI/pip. How so? I'm avoiding this attack by publishing sha256sums at release time. The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging up a checksum mismatch from the mailing list archives. > > 6) D.J. Bernstein, who is somewhat security minded, has been shipping > > his software *for years* with just plain HTTP and published checksums. > > Argument from authority doesn't really hold up very well when DJB doesn't > distribute is software in a way that is intended to be consumed mechanically. > Also while he may be a crypto expert he is far from an expert on successfully > distributing software, unless you also think that the signature checking in > most OS provided package managers is pointless. That is sort of a strawman. The whole point *is* that certain distribution models don't lend themselves to mechanical consumption. I cannot speak for DJB, perhaps he is just thinking that GPG signing is pointless if users can't validate the signature and SSL is pointless because one cannot trust governments. OS package signing is useful since the packages are curated. If anyone could upload a package to Debian, whereupon it would be signed with the official key, apt-get would lose its usefulness. Stefan Krah From rdmurray at bitdance.com Thu May 8 17:21:27 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 08 May 2014 11:21:27 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> Message-ID: <20140508152127.E4219250DBB@webabinitio.net> On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft wrote: > Most users are not going to care up until the point where the external server > is unavailable, and then they care a whole lot. On the tin it sounds reasonable > to just download the external file if the server is up however we've done > that for a long time and the end result has been end user pain. > > Now requiring someone to add a flag in order to download an externally hosted > file is also end user pain. The difference between the two pains is when they > happen. The requiring a flag pain happens at the point of decision, when you > decide to make your deployment depend on something hosted externally. The > default to allow pain happens sometime in the future, when you may or may not > have any idea why suddenly your installs aren't working (and when you look, > PyPI is up so you're really very confused why this particular file doesn't > work). Even worse is the case when a project has some old files, but the newer > versions aren't hosted and suddenly people are getting very old releases which > is even more confusing to the end users. Ah, I understand now. Your perspective is as someone who is using pip for *deployment*. I'm speaking from a python+plus+pip end-user perspective, which is going to be even more common now that it is part of the Python distribution. I'm not sure how you reconcile these two worlds. I would venture to suggest that if you are using it for deployment you really ought to be using a local package repository[*], not the global one; but, as someone observed, the sensible thing to do and what people actually do are often very different; and, since I haven't done this particular kind of deployment scenario in Python myself, there may be reasons I'm missing. However, your last mention of "end users" confuses me. Why are "end users" getting old packages in a deployment situation? Isn't it the developer/operations people (and the latter would, I assume, control the 'external packages' flag) who would be seeing that? Maybe you mean something by deployment different from how I use the word? --David [*] I found it *such* a pain to do this for perl/cpan. I have a project for a customer where I have to do this, and the hoops I had to jump through to get a reliable deployment (where packages wouldn't be unexpectedly upgraded under my feet) were nasty. (This was several years ago and I haven't revisited it, so maybe things have gotten better, or I just missed something.) I haven't had to do it for python yet, oddly enough, so I don't know how hard it is for Python. From donald at stufft.io Thu May 8 17:24:47 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 11:24:47 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508151906.GA29543@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <20140508151906.GA29543@sleipnir.bytereef.org> Message-ID: <5151B70D-7224-47B1-BA95-257652E60469@stufft.io> On May 8, 2014, at 11:19 AM, Stefan Krah wrote: > Donald Stufft wrote: >> hosted packages are brittle and more prone to failure. Every single external >> server adds *another* SPOF into any particular install set. Even if every >> external server has a 99.9% uptime, when you combine multiple of them the total >> uptime of any particular set of requirements drops quickly. This has been a >> major complaint that people have had over time. > > We have been through that many times; to me this is an indication that > people are using pip under circumstances when they should not. pip is > not apt-get. > > [I am aware that *you* know that, just stating it again for the broader > audience.] > > >>> 2) Last time I looked, access credentials (via "lost login") were sent >>> out in plaintext. >> >> The existence of other security issues is not an excuse to not fix a security >> issue. There are other problems and we're slowly working on trying to clear >> them out. > > It is, however, a reason to avoid error messages that could *imply* (rightly > or wrongly) to users that the combination of pip and internal packages is > safe. > > >>> 3) AFAIK people can upload a different (malicious) version of a >>> package with *the exact same name*. >> >> Yes, a malicious author is needfully outside of the threat model for PyPI/pip. > > How so? I'm avoiding this attack by publishing sha256sums at release time. > The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging > up a checksum mismatch from the mailing list archives. Practically speaking exactly zero people will ever bother to dig up the checksum from a mailing list archive, or even verify the hash that you have right on PyPI itself. Worse security that people actually use is infinitely better than better security that nobody uses. > > >>> 6) D.J. Bernstein, who is somewhat security minded, has been shipping >>> his software *for years* with just plain HTTP and published checksums. >> >> Argument from authority doesn't really hold up very well when DJB doesn't >> distribute is software in a way that is intended to be consumed mechanically. >> Also while he may be a crypto expert he is far from an expert on successfully >> distributing software, unless you also think that the signature checking in >> most OS provided package managers is pointless. > > That is sort of a strawman. The whole point *is* that certain distribution > models don't lend themselves to mechanical consumption. I cannot speak > for DJB, perhaps he is just thinking that GPG signing is pointless if > users can't validate the signature and SSL is pointless because one cannot > trust governments. > > OS package signing is useful since the packages are curated. If anyone > could upload a package to Debian, whereupon it would be signed with the > official key, apt-get would lose its usefulness. People are going to mechanically consume them. You can tell them it?s wrong all you want but they are going to do it. > > > > Stefan Krah > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 8 17:32:28 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 11:32:28 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508152127.E4219250DBB@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> <20140508152127.E4219250DBB@webabinitio.net> Message-ID: <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io> On May 8, 2014, at 11:21 AM, R. David Murray wrote: > On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft wrote: >> Most users are not going to care up until the point where the external server >> is unavailable, and then they care a whole lot. On the tin it sounds reasonable >> to just download the external file if the server is up however we've done >> that for a long time and the end result has been end user pain. >> >> Now requiring someone to add a flag in order to download an externally hosted >> file is also end user pain. The difference between the two pains is when they >> happen. The requiring a flag pain happens at the point of decision, when you >> decide to make your deployment depend on something hosted externally. The >> default to allow pain happens sometime in the future, when you may or may not >> have any idea why suddenly your installs aren't working (and when you look, >> PyPI is up so you're really very confused why this particular file doesn't >> work). Even worse is the case when a project has some old files, but the newer >> versions aren't hosted and suddenly people are getting very old releases which >> is even more confusing to the end users. > > Ah, I understand now. > > Your perspective is as someone who is using pip for *deployment*. Deployment, or any kind of situation where you want to have a reproducible build. Generally via deployment yes. > > I'm speaking from a python+plus+pip end-user perspective, which is going > to be even more common now that it is part of the Python distribution. > > I'm not sure how you reconcile these two worlds. I would venture to > suggest that if you are using it for deployment you really ought to > be using a local package repository[*], not the global one; but, as > someone observed, the sensible thing to do and what people actually > do are often very different; and, since I haven't done this particular > kind of deployment scenario in Python myself, there may be reasons > I'm missing. People simply don?t do this[1]. Especially in a world with things like Heroku existing which makes it stupid simple to use pip to install from PyPI but installing from your own server requires standing up some infrastructure. > > However, your last mention of "end users" confuses me. Why are "end > users" getting old packages in a deployment situation? Isn't it the > developer/operations people (and the latter would, I assume, control > the 'external packages' flag) who would be seeing that? Maybe you mean > something by deployment different from how I use the word? Someone using pip, this may be a developer who is just trying to recreate their production environment locally, this may be someone using chef/puppet to automate installing via pip, this may be someone pushing to Heroku. The old versions thing is more that it?s really confusing when you type ``pip install foo`` on a monday and get 2.0, and ``pip install foo`` on weds and get 0.4. > > --David > > [*] I found it *such* a pain to do this for perl/cpan. I have a > project for a customer where I have to do this, and the hoops I had > to jump through to get a reliable deployment (where packages wouldn't > be unexpectedly upgraded under my feet) were nasty. (This was several > years ago and I haven't revisited it, so maybe things have gotten better, > or I just missed something.) > > I haven't had to do it for python yet, oddly enough, so I don't know > how hard it is for Python. For Python with pip you can use a requirements.txt file to create a set of dependencies that are pinned to exact versions like: foo==2.0 bar==2.3 And pip will (theoretically, our dep solving is real bad ATM) install exactly those versions from your index server. Generally this means PyPI which means the author can delete the version and upload a new file with the same version number. However it?s also trivial to stand up your own server. It can be as easy as pointing nginx/Apache at a static directory with autoindex = True. (See: https://wheels.caremad.io/). On top of that there is peep which adds a secure message digest on it to make sure that the author/index didn?t swap things out on you, and there is some discussion about how best to add that to pip itself. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at bytereef.org Thu May 8 17:34:54 2014 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 8 May 2014 17:34:54 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> Message-ID: <20140508153454.GB29543@sleipnir.bytereef.org> Donald Stufft wrote: > > Today I've switched to manual install mode with manual sha256sum verification > > which is *far* safer than anything you get via pip right now. > > It is not safer in any meaingful way. > > If someone is in a position to compromise the integrity of PyPI's TLS, they > can replace the hash on that page with something else. Now you've attempted to > work around this by telling people to go look up the release announcement > hash. However if someone can compromise the integrity of PyPI's TLS, they can > also compromise the integrity of https://mail.python.org/, or GMane, or any > other TLS based website[1]. Of course it is safer. Suppose a file is stored on PyPI: 1) Attacker guesses my username (or is it even visible, I'm not sure). 2) Clicks on "lost login". 3) Intercepts mail (difficult, but far from the TLS attack category). Maybe on a home or university network. Or a rogue person at a mail provider. 4) Changes the uploaded file together with the hash. pip would be perfectly happy, checking the hash via Google would turn up a mismatch. Stefan Krah From mal at egenix.com Thu May 8 17:37:57 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 May 2014 17:37:57 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536B97E2.8070805@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> Message-ID: <536BA4D5.7080208@egenix.com> On 08.05.2014 16:42, M.-A. Lemburg wrote: > On 08.05.2014 15:58, Donald Stufft wrote: >> >> On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: >> >>> Well, to be fair and leaving aside uptime concerns and the general >>> desire to always install packages from some server instead of >>> a safe and trusted local directory (probably too obvious ;-), >>> it would certainly be possible to add support for >>> trusted externally hosted packages. >> >> There is support for trusted externally hosted packages, you put the URL in >> PyPI and include a hash in the fragment like so: >> >> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 >> >> The hash can be md5 or any of the sha-2 family. >> >> Now this does not mean that ``pip install cdecimal`` will automatically install >> this, because whether or not you're willing to install from servers other than >> PyPI[1] is a policy decision for the end user of pip. > > Hmm, if you call that feature "trusted externally hosted packages", > pip should really do trust them, right ? ;-) > > I can understand that pip defaults to not trusting URLs which don't > meet the above feature requirements, but not that it still warns > about unreliable externally hosted packages even if the above > feature is used. > > At the moment, pip will refuse to use an externally hosted files even > if the package author uses the above hashed URLs; even with HTTPS > and proper SSL certificate chain. Could this perhaps be changed/reconsidered for pip ? Note that easy_install/setuptools does not have such problems. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Thu May 8 17:43:37 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 11:43:37 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508153454.GB29543@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> <20140508153454.GB29543@sleipnir.bytereef.org> Message-ID: <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io> On May 8, 2014, at 11:34 AM, Stefan Krah wrote: > Donald Stufft wrote: >>> Today I've switched to manual install mode with manual sha256sum verification >>> which is *far* safer than anything you get via pip right now. >> >> It is not safer in any meaingful way. >> >> If someone is in a position to compromise the integrity of PyPI's TLS, they >> can replace the hash on that page with something else. Now you've attempted to >> work around this by telling people to go look up the release announcement >> hash. However if someone can compromise the integrity of PyPI's TLS, they can >> also compromise the integrity of https://mail.python.org/, or GMane, or any >> other TLS based website[1]. > > Of course it is safer. Suppose a file is stored on PyPI: > > 1) Attacker guesses my username (or is it even visible, I'm not sure). > > 2) Clicks on "lost login". > > 3) Intercepts mail (difficult, but far from the TLS attack category). > Maybe on a home or university network. Or a rogue person at a > mail provider. > > 4) Changes the uploaded file together with the hash. > > > pip would be perfectly happy, checking the hash via Google would turn > up a mismatch. I said ?meaningful?. Almost nobody is going to ever bother googling it and the likelihood that someone is able to MITM *you* specifically is far lesser than the likelihood that someone is going to MITM one of the cdecimal users. Additionally your messages aren?t signed and email isn?t an authenticated profile so if someone was able to get your password they could simply spoof and email from you to the mailing list with new hashes, or edit out the description telling people to go google some stuff. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu May 8 17:46:14 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 11:46:14 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536BA4D5.7080208@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> Message-ID: <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> On May 8, 2014, at 11:37 AM, M.-A. Lemburg wrote: > On 08.05.2014 16:42, M.-A. Lemburg wrote: >> On 08.05.2014 15:58, Donald Stufft wrote: >>> >>> On May 8, 2014, at 9:39 AM, M.-A. Lemburg wrote: >>> >>>> Well, to be fair and leaving aside uptime concerns and the general >>>> desire to always install packages from some server instead of >>>> a safe and trusted local directory (probably too obvious ;-), >>>> it would certainly be possible to add support for >>>> trusted externally hosted packages. >>> >>> There is support for trusted externally hosted packages, you put the URL in >>> PyPI and include a hash in the fragment like so: >>> >>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 >>> >>> The hash can be md5 or any of the sha-2 family. >>> >>> Now this does not mean that ``pip install cdecimal`` will automatically install >>> this, because whether or not you're willing to install from servers other than >>> PyPI[1] is a policy decision for the end user of pip. >> >> Hmm, if you call that feature "trusted externally hosted packages", >> pip should really do trust them, right ? ;-) >> >> I can understand that pip defaults to not trusting URLs which don't >> meet the above feature requirements, but not that it still warns >> about unreliable externally hosted packages even if the above >> feature is used. >> >> At the moment, pip will refuse to use an externally hosted files even >> if the package author uses the above hashed URLs; even with HTTPS >> and proper SSL certificate chain. > > Could this perhaps be changed/reconsidered for pip ? > > Note that easy_install/setuptools does not have such problems. Anything can be changes or reconsidered of course. I feel pretty strongly that an installer should not install things from places other than the index without a specific opt in. That discussion would be best done on distutils-sig as it would require reversing the decision in PEP438. I really don't feel strongly one way or the other about the *warning* that happens when you allow an external file. It exists primarily because at the time it was implemented external files were default to allowed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at bytereef.org Thu May 8 18:03:25 2014 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 8 May 2014 18:03:25 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> <20140508153454.GB29543@sleipnir.bytereef.org> <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io> Message-ID: <20140508160325.GA29870@sleipnir.bytereef.org> Donald Stufft wrote: > I said ?meaningful?. Almost nobody is going to ever bother googling it and > the likelihood that someone is able to MITM *you* specifically is far lesser > than the likelihood that someone is going to MITM one of the cdecimal users. I'm doing this for important installs. -- That is how I installed qmail and djbdns. > Additionally your messages aren?t signed and email isn?t an authenticated > profile so if someone was able to get your password they could simply spoof > and email from you to the mailing list with new hashes, or edit out the description > telling people to go google some stuff. Signing messages is pointless if the key isn't well connected. Also, I'm reading the lists and would notice a "release". Most importantly, the checksum mismatch would still be found, since the old messages with the correct sum would still exist under the scenario we're talking about (i.e. not GHCQ hacking into Belgacom routers). Stefan Krah From donald at stufft.io Thu May 8 18:22:16 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 12:22:16 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508160325.GA29870@sleipnir.bytereef.org> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> <20140508153454.GB29543@sleipnir.bytereef.org> <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io> <20140508160325.GA29870@sleipnir.bytereef.org> Message-ID: <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io> On May 8, 2014, at 12:03 PM, Stefan Krah wrote: > Donald Stufft wrote: >> I said ?meaningful?. Almost nobody is going to ever bother googling it and >> the likelihood that someone is able to MITM *you* specifically is far lesser >> than the likelihood that someone is going to MITM one of the cdecimal users. > > I'm doing this for important installs. -- That is how I installed qmail > and djbdns. > > >> Additionally your messages aren?t signed and email isn?t an authenticated >> profile so if someone was able to get your password they could simply spoof >> and email from you to the mailing list with new hashes, or edit out the description >> telling people to go google some stuff. > > Signing messages is pointless if the key isn't well connected. Also, I'm > reading the lists and would notice a "release". Most importantly, the > checksum mismatch would still be found, since the old messages with the > correct sum would still exist under the scenario we're talking about > (i.e. not GHCQ hacking into Belgacom routers). > > I?m unsure if you?re being willfully dense or if you?re just not understanding what I mean when I say ?almost?. Of course there are going to be a few outliers where people do bother to do that, but it?s not going to be common place at all. But whatever, I?ve removed the warning that occurs when you install an externally hosted file [1] and it will be included in pip 1.6. I have not changed the defaults for --allow-all-external nor have I removed the warning that occurs when someone elects to install an unverifiable download. [1] https://github.com/pypa/pip/commit/9f56b79e8d ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rdmurray at bitdance.com Thu May 8 18:42:16 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 08 May 2014 12:42:16 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> <20140508152127.E4219250DBB@webabinitio.net> <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io> Message-ID: <20140508164217.28F92B14088@webabinitio.net> On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft wrote: > On May 8, 2014, at 11:21 AM, R. David Murray wrote: > > Ah, I understand now. > > > > Your perspective is as someone who is using pip for *deployment*. > > Deployment, or any kind of situation where you want to have a reproducible > build. Generally via deployment yes. [...] > For Python with pip you can use a requirements.txt file to create a set of > dependencies that are pinned to exact versions like: > > foo==2.0 > bar==2.3 > > And pip will (theoretically, our dep solving is real bad ATM) install exactly > those versions from your index server. Generally this means PyPI which OK, this makes sense, then. (I wish perl/cpan had something similar...maybe it does, but I couldn't find it at the time.) This still leaves the fact that there is a disconnect between the "needs" of two different audiences for PIP: people who deploy things, and everyone else who just uses pip to install stuff. The second group is going to overwhelm the first group, if it doesn't already. And I think that's all the comments I have on this issue :) --David From donald at stufft.io Thu May 8 18:57:00 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 12:57:00 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140508164217.28F92B14088@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508141140.4AEB1250DBB@webabinitio.net> <20140508142135.09EE0250DBC@webabinitio.net> <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io> <20140508152127.E4219250DBB@webabinitio.net> <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io> <20140508164217.28F92B14088@webabinitio.net> Message-ID: On May 8, 2014, at 12:42 PM, R. David Murray wrote: > On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft wrote: >> On May 8, 2014, at 11:21 AM, R. David Murray wrote: >>> Ah, I understand now. >>> >>> Your perspective is as someone who is using pip for *deployment*. >> >> Deployment, or any kind of situation where you want to have a reproducible >> build. Generally via deployment yes. > [...] >> For Python with pip you can use a requirements.txt file to create a set of >> dependencies that are pinned to exact versions like: >> >> foo==2.0 >> bar==2.3 >> >> And pip will (theoretically, our dep solving is real bad ATM) install exactly >> those versions from your index server. Generally this means PyPI which > > OK, this makes sense, then. (I wish perl/cpan had something > similar...maybe it does, but I couldn't find it at the time.) > > This still leaves the fact that there is a disconnect between the > "needs" of two different audiences for PIP: people who deploy things, > and everyone else who just uses pip to install stuff. Yup balancing between the two is something we have to do in every decision we make. When PEP438 was being discussed I did a pretty extensive amount of investigation into what affect this change would have [1]. What I found was that: - The sizable majority was projects would host things on PyPI - There was a significant chunk of projects where a single release or two would only be available externally and it was an accident that they weren?t uploaded. - Of the links that were available externally, very few of them were available in a way that was verifiable and were thus insecure to install. Because of this it was determined that simply allowing externally hosted files without also allowing externally hosted and unverified files would not actually have a significant impact for the vast bulk of the projects that were not hosted on PyPI. > > The second group is going to overwhelm the first group, if it doesn't > already. Generally yes, because not every who uses pip to deploy uses pip to install locally, but most people who use pip to deploy also use pip locally. > > And I think that's all the comments I have on this issue :) [1]: https://github.com/dstufft/pypi.linkcheck ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brian at python.org Thu May 8 18:59:19 2014 From: brian at python.org (Brian Curtin) Date: Thu, 8 May 2014 11:59:19 -0500 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? Message-ID: This is mostly a question for Martin, but perhaps someone else would also know. I'm trying to build the 2.7 installers so I can backport the path option from 3.3, but I can't seem to figure out which version of Tix is necessary to have a complete build. So far any of them on http://svn.python.org/projects/external don't appear to be configured to work with the combination of tcl and tk that are used on 2.7, per the buildbot external scripts. It's another issue that Tix isn't even downloaded by the scripts, but I'll do it manually right now just to get this going. Any tips? From martin at v.loewis.de Thu May 8 21:36:22 2014 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 08 May 2014 21:36:22 +0200 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? In-Reply-To: References: Message-ID: <536BDCB6.6030300@v.loewis.de> Am 08.05.14 18:59, schrieb Brian Curtin: > This is mostly a question for Martin, but perhaps someone else would also know. > > I'm trying to build the 2.7 installers so I can backport the path > option from 3.3, but I can't seem to figure out which version of Tix > is necessary to have a complete build. So far any of them on > http://svn.python.org/projects/external don't appear to be configured > to work with the combination of tcl and tk that are used on 2.7, per > the buildbot external scripts. It's another issue that Tix isn't even > downloaded by the scripts, but I'll do it manually right now just to > get this going. > > Any tips? Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one where python.mak was pointing to Tcl 8.5.2. It may not have been tagged. In 2.7, it wasn't checked out out automatically because Tix requires the original Tcl/Tk path names, so building it automatically would have failed. Regards, Martin From p.f.moore at gmail.com Thu May 8 23:02:16 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 May 2014 22:02:16 +0100 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: On 8 May 2014 16:46, Donald Stufft wrote: > Anything can be changes or reconsidered of course. I feel pretty strongly that > an installer should not install things from places other than the index without > a specific opt in. That discussion would be best done on distutils-sig as it > would require reversing the decision in PEP438. I think it's worth reconsidering. Since this behaviour was implemented, there have been many instances of confusion and unhappiness with the situation, both from package developers and pip users. I don't think that's good for pip. I would like to see PEP 438 reviewed with the intention of working out how to fix the user experience (ideally while retaining the reliability enhancements, but accepting that compromises may be needed). Some points: 1. Often a user won't know that a package is externally hosted until an install fails. Having to add a flag and retry is a lousy experience. Why not at a minimum have an "externally hosted - are you sure?" prompt? 2. The way the flags are implemented (notably the need to repeat the package name) is clearly confusing and irritating for users, even if the reasons are logical. We should look at fixing that. 3. The user complaints against pip are significant and ongoing. I don't think they should be ignored. If PEP 438 cannot be reconsidered in the light of user feedback, then I think the pip developers may need to have a discussion about whether we conform to the PEP (clearly Donald thinks we should, I'm not sure I do, and I don't know about the others). But I don't think it's healthy for pip to be looking at deliberately ignoring an accepted PEP, so I'd prefer it if the debate was around addressing the issues in the PEP itself. 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP 438. The issues getting raised are end-user problems, and the users most affected are unlikely to be on that list. Socially, this change does not seem to be having the effect of persuading more package developers to host on PyPI. The stick doesn't appear to have worked, maybe we should be trying to find a carrot? Or maybe we have to accept that some developers have sound reasons for not hosting on PyPI and work with them to find an acceptable compromise? Has anyone checked what Stefan's reasons are for not hosting cdecimal on PyPI? Do they represent a use case that the PEP hasn't considered? > I really don't feel strongly one way or the other about the *warning* that > happens when you allow an external file. It exists primarily because at the > time it was implemented external files were default to allowed. I think it's reasonable to remove the warning. If the user chooses to allow an external file, it makes sense to assume they understand the implications and not nag them about their decision. Particularly given the level of controversy the warning is generating. On a personal note, I'm uncomfortable with the way this change is perceived as a case of *pip* enforcing a behaviour that the pip developers feel should be required. I actually don't like this change particularly. So having pip implement the behaviour required by that PEP is to me simply a case of compliance with the agreed standard. But now, as a pip developer, being held responsible for the resulting user pain, and being expected to defend it, does not make me happy. Paul From donald at stufft.io Thu May 8 23:22:05 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 17:22:05 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: On May 8, 2014, at 5:02 PM, Paul Moore wrote: > On 8 May 2014 16:46, Donald Stufft wrote: >> Anything can be changes or reconsidered of course. I feel pretty strongly that >> an installer should not install things from places other than the index without >> a specific opt in. That discussion would be best done on distutils-sig as it >> would require reversing the decision in PEP438. > > I think it's worth reconsidering. Since this behaviour was > implemented, there have been many instances of confusion and > unhappiness with the situation, both from package developers and pip > users. I don't think that's good for pip. I would like to see PEP 438 > reviewed with the intention of working out how to fix the user > experience (ideally while retaining the reliability enhancements, but > accepting that compromises may be needed). I think most of the confusion has been over the fact that ?allow-external takes a package name, not that it exists at all. > > Some points: > > 1. Often a user won't know that a package is externally hosted until > an install fails. Having to add a flag and retry is a lousy > experience. Why not at a minimum have an "externally hosted - are you > sure?" prompt? A prompt is OK with me. > 2. The way the flags are implemented (notably the need to repeat the > package name) is clearly confusing and irritating for users, even if > the reasons are logical. We should look at fixing that. Yea, there?s a ticket for this. > 3. The user complaints against pip are significant and ongoing. I > don't think they should be ignored. If PEP 438 cannot be reconsidered > in the light of user feedback, then I think the pip developers may > need to have a discussion about whether we conform to the PEP (clearly > Donald thinks we should, I'm not sure I do, and I don't know about the > others). But I don't think it's healthy for pip to be looking at > deliberately ignoring an accepted PEP, so I'd prefer it if the debate > was around addressing the issues in the PEP itself. I don?t think the problem with with the PEP. > 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP > 438. The issues getting raised are end-user problems, and the users > most affected are unlikely to be on that list. > > Socially, this change does not seem to be having the effect of > persuading more package developers to host on PyPI. The stick doesn't > appear to have worked, maybe we should be trying to find a carrot? Do you have any data to point to that says it hasn?t worked? Just to see what impact it has had, I?m running my scripts again that I ran a year ago to see what has changed, already I can see they are processing MUCH faster than last year. > Or > maybe we have to accept that some developers have sound reasons for > not hosting on PyPI and work with them to find an acceptable > compromise? Has anyone checked what Stefan's reasons are for not > hosting cdecimal on PyPI? Do they represent a use case that the PEP > hasn't considered? If I recall correctly his reasoning is that he finds the legal requirements associated with uploading to PyPI to be unsatisfactory. > >> I really don't feel strongly one way or the other about the *warning* that >> happens when you allow an external file. It exists primarily because at the >> time it was implemented external files were default to allowed. > > I think it's reasonable to remove the warning. If the user chooses to > allow an external file, it makes sense to assume they understand the > implications and not nag them about their decision. Particularly given > the level of controversy the warning is generating. The warning is gone as of a few hours ago. > > On a personal note, I'm uncomfortable with the way this change is > perceived as a case of *pip* enforcing a behaviour that the pip > developers feel should be required. I actually don't like this change > particularly. So having pip implement the behaviour required by that > PEP is to me simply a case of compliance with the agreed standard. But > now, as a pip developer, being held responsible for the resulting user > pain, and being expected to defend it, does not make me happy. > > Paul I think the pain is being overrepresented and the positives are being ignored. The problem is the benefits of this PEP are much like the benefits of TLS too. For the vast majority of people they don?t notice anything different except installing things is faster and more reliable. They don?t attribute that to the PEP or this decision, they just internalize it as the new norm. However the people who this does affect will seek out why it broke and raise an issue citing that thing specifically. This creates a perception of lots of pain for no gain when the reality is not that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From zachary.ware+pydev at gmail.com Thu May 8 23:20:53 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 8 May 2014 16:20:53 -0500 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? In-Reply-To: <536BDCB6.6030300@v.loewis.de> References: <536BDCB6.6030300@v.loewis.de> Message-ID: On Thu, May 8, 2014 at 2:36 PM, "Martin v. L?wis" wrote: > Am 08.05.14 18:59, schrieb Brian Curtin: >> This is mostly a question for Martin, but perhaps someone else would also know. >> >> I'm trying to build the 2.7 installers so I can backport the path >> option from 3.3, but I can't seem to figure out which version of Tix >> is necessary to have a complete build. So far any of them on >> http://svn.python.org/projects/external don't appear to be configured >> to work with the combination of tcl and tk that are used on 2.7, per >> the buildbot external scripts. It's another issue that Tix isn't even >> downloaded by the scripts, but I'll do it manually right now just to >> get this going. >> >> Any tips? > > Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one > where python.mak was pointing to Tcl 8.5.2. It may not have been tagged. > > In 2.7, it wasn't checked out out automatically because Tix requires the > original Tcl/Tk path names, so building it automatically would have failed. I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple of weeks ago (see http://bugs.python.org/issue21303), but hadn't gotten anything done with Tix yet. It should just need python.mak updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as soon as I can. -- Zach From ncoghlan at gmail.com Fri May 9 00:20:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 May 2014 08:20:10 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: On 9 May 2014 07:23, "Donald Stufft" wrote: > On May 8, 2014, at 5:02 PM, Paul Moore wrote: > > > Or > > maybe we have to accept that some developers have sound reasons for > > not hosting on PyPI and work with them to find an acceptable > > compromise? Has anyone checked what Stefan's reasons are for not > > hosting cdecimal on PyPI? Do they represent a use case that the PEP > > hasn't considered? > > If I recall correctly his reasoning is that he finds the legal requirements > associated with uploading to PyPI to be unsatisfactory. I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). I'll deal with that at work today. Cheers, Nick. P.S. Still the wrong list, folks. Most of the people you need to convince to get changes made to pip, PyPI and the packaging ecosystem in general aren't here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 9 00:22:30 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 May 2014 18:22:30 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io> On May 8, 2014, at 6:20 PM, Nick Coghlan wrote: > > On 9 May 2014 07:23, "Donald Stufft" wrote: > > On May 8, 2014, at 5:02 PM, Paul Moore wrote: > > > > > Or > > > maybe we have to accept that some developers have sound reasons for > > > not hosting on PyPI and work with them to find an acceptable > > > compromise? Has anyone checked what Stefan's reasons are for not > > > hosting cdecimal on PyPI? Do they represent a use case that the PEP > > > hasn't considered? > > > > If I recall correctly his reasoning is that he finds the legal requirements > > associated with uploading to PyPI to be unsatisfactory. > > I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). > > I'll deal with that at work today. > > > I?m pretty sure VanL wrote the terms and has explicitly said they won?t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page. [1] https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri May 9 01:07:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 May 2014 09:07:18 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io> Message-ID: On 9 May 2014 08:22, "Donald Stufft" wrote: > > > On May 8, 2014, at 6:20 PM, Nick Coghlan wrote: >> >> I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up). >> >> I'll deal with that at work today. >> >> > I?m pretty sure VanL wrote the terms and has explicitly said they won?t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page. Thanks for the additional references. It should be possible to clarify the terms to address Red Hat's (and other users') concerns while still addressing the points Van is worried about (for example, adding a footnote explaining the use cases that need to be covered, and being explicit that users must respect *both* licenses). However, this subthread is now even further off-topic for this list, so I'll take it to python-legal-sig later today with some specific proposed wording changes. Cheers, Nick. > > [1] https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri May 9 06:34:19 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 00:34:19 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> On May 8, 2014, at 5:22 PM, Donald Stufft wrote: >> Socially, this change does not seem to be having the effect of >> persuading more package developers to host on PyPI. The stick doesn't >> appear to have worked, maybe we should be trying to find a carrot? > > Do you have any data to point to that says it hasn?t worked? Just to see > what impact it has had, I?m running my scripts again that I ran a year > ago to see what has changed, already I can see they are processing > MUCH faster than last year. The data has finished processing, it represents a time diff of approximately one year. The pip release that caused all of this was released about 4-5 months ago. Overall PyPI has seen a 50% growth in installable projects in that time. If the change would have had no effect we'd expect to see a ~50% increase across the board. However what we've seen is a a 60% (+10% of expected) increase in projects that can only be installed from PyPI and a 12% decrease in projects that have any unsafe files (-62% of expected). Further more we can see that if pip were to change the default of --allow-all-external it would take 23 projects from unable to be installed by default to able to be installed by default. This represents 0.2% of installable projects on PyPI. It would take an additional 40 projects and make one or more additional files able to be downloaded by default. Some other data points: * We've gone from 86% of projects being installable from PyPI to 92%. * We've gone from 5% of projects being only unsafely installable to 3% * We've gone from 14% of projects having any files unsafe to install to 8% * We've gone from 0.004% of projects being safely hosted externally to 0.2% Looking at these numbers I think it's safe to say that in this time period that the "hosting hygiene" of a PyPI project is more likely to be a better state than it was a year ago. We cannot state for a fact if this is because of this change or not, however given that the fallout is ~23 (or ~63) projects out of 38,835 I think it is incredibly reasonable to leave the defaults alone since there is a reasonably high chance that they played at least some part in that change. I'd love to get these numbers to the point where the number of projects installable strictly from PyPI is 100% (or at least 100% installable safely), however 92% (or 92.2%) is getting pretty close to that and hopefully that number will just continue to grow until it hits 100%. For reference, here's the raw numbers as well as some summary of the data here: https://gist.github.com/dstufft/b14008d11c0a5760dbed And the repository where the raw data as well as the scripts used to collect and process it is here: https://github.com/dstufft/pypi.linkcheck linkcollector.py collections while linkwriter.py writes out the json file, and stats2.py processes and gives the numbers from the gist above. links.json is the data from a year ago, and 2014-05-08.links.json is the data from today. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Fri May 9 06:45:42 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 00:45:42 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> Message-ID: <3097645C-A16F-4386-B8BF-46CCBA9021D5@stufft.io> On May 9, 2014, at 12:34 AM, Donald Stufft wrote: > The data has finished processing, it represents a time diff of approximately > one year. The pip release that caused all of this was released about 4-5 months > ago. Oh I forgot to mention: In order to make the comparison as accurate as possible I've used the same collection script as I did a year ago. This is prior to the real advent of Wheels so these numbers do not take into accounts Wheels at all (which pip can also install) but it *does* include Eggs which pip cannot install. Further more it also includes #egg=dev urls which point to a VCS (and it would consider those an unverifiable/unsafe download). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Fri May 9 10:12:03 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 May 2014 10:12:03 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: <536C8DD3.6090601@egenix.com> On 08.05.2014 23:22, Donald Stufft wrote: > >> On a personal note, I'm uncomfortable with the way this change is >> perceived as a case of *pip* enforcing a behaviour that the pip >> developers feel should be required. I actually don't like this change >> particularly. So having pip implement the behaviour required by that >> PEP is to me simply a case of compliance with the agreed standard. But >> now, as a pip developer, being held responsible for the resulting user >> pain, and being expected to defend it, does not make me happy. > > I think the pain is being overrepresented and the positives are being ignored. > The problem is the benefits of this PEP are much like the benefits of TLS > too. For the vast majority of people they don?t notice anything different except > installing things is faster and more reliable. They don?t attribute that to the > PEP or this decision, they just internalize it as the new norm. However the > people who this does affect will seek out why it broke and raise an issue > citing that thing specifically. This creates a perception of lots of pain for no > gain when the reality is not that. Donald: I don't think anyone is arguing that hosting packages on PyPI is a bad thing and PyPI as a service has gotten a lot better than it was a few years ago. However, I find it troubling that we as Python developers are *forcing* the whole Python world to put their code into PyPI. There are plenty good reasons not to do this, and sometimes it's even impossible if you want to stay legal (see the PEP for details). Accordingly, we should respect those reasons make it possible for Python packages to live elsewhere, without having our tools put those packages into a bad light or making it harder for Python users to install such packages than needed. With the checksum uploaded to PyPI, the only argument against fetching packages from other places on the Internet is reliability, not security. PyPI is not the only reliable file hosting system on the Internet, so this argument is rather weak. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Fri May 9 11:01:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 10:01:35 +0100 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> Message-ID: On 9 May 2014 05:34, Donald Stufft wrote: > On May 8, 2014, at 5:22 PM, Donald Stufft wrote: > >>> Socially, this change does not seem to be having the effect of >>> persuading more package developers to host on PyPI. The stick doesn't >>> appear to have worked, maybe we should be trying to find a carrot? >> >> Do you have any data to point to that says it hasn?t worked? Just to see >> what impact it has had, I?m running my scripts again that I ran a year >> ago to see what has changed, already I can see they are processing >> MUCH faster than last year. > > The data has finished processing, it represents a time diff of approximately > one year. The pip release that caused all of this was released about 4-5 months > ago. > > Overall PyPI has seen a 50% growth in installable projects in that time. If the > change would have had no effect we'd expect to see a ~50% increase across the > board. However what we've seen is a a 60% (+10% of expected) increase in > projects that can only be installed from PyPI and a 12% decrease in projects > that have any unsafe files (-62% of expected). Donald, Thanks for taking the time to get those figures. It does appear that there are less cases that would be affected than the number of complaints would imply. The only concern I have about this type of analysis is that it doesn't "weight" projects. It may be (and again, I have no data to back this up) that the projects that are affected detrimentally by this change are unusually popular or otherwise significant. There's obviously no way to assess this sensibly other than by making a judgement on the level of complaints. But arguing numbers was never my intention here, so let's just say that I concede that the change has had a positive effect, which is great. Paul From stefan at bytereef.org Fri May 9 11:43:59 2014 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 9 May 2014 11:43:59 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io> References: <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <20140508143650.GA29239@sleipnir.bytereef.org> <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io> <20140508153454.GB29543@sleipnir.bytereef.org> <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io> <20140508160325.GA29870@sleipnir.bytereef.org> <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io> Message-ID: <20140509094359.GA6254@sleipnir.bytereef.org> Donald Stufft wrote: > I?m unsure if you?re being willfully dense or if you?re just not understanding > what I mean when I say ?almost?. Of course there are going to be a few outliers > where people do bother to do that, but it?s not going to be common place at > all. I suggest that you confine that style of discussion to distutils-sig. Stefan Krah From donald at stufft.io Fri May 9 13:44:25 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 07:44:25 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536C8DD3.6090601@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> Message-ID: On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: > On 08.05.2014 23:22, Donald Stufft wrote: >> >>> On a personal note, I'm uncomfortable with the way this change is >>> perceived as a case of *pip* enforcing a behaviour that the pip >>> developers feel should be required. I actually don't like this change >>> particularly. So having pip implement the behaviour required by that >>> PEP is to me simply a case of compliance with the agreed standard. But >>> now, as a pip developer, being held responsible for the resulting user >>> pain, and being expected to defend it, does not make me happy. >> >> I think the pain is being overrepresented and the positives are being ignored. >> The problem is the benefits of this PEP are much like the benefits of TLS >> too. For the vast majority of people they don?t notice anything different except >> installing things is faster and more reliable. They don?t attribute that to the >> PEP or this decision, they just internalize it as the new norm. However the >> people who this does affect will seek out why it broke and raise an issue >> citing that thing specifically. This creates a perception of lots of pain for no >> gain when the reality is not that. > > Donald: I don't think anyone is arguing that hosting packages on > PyPI is a bad thing and PyPI as a service has gotten a lot better > than it was a few years ago. Didn?t mean to imply that I thought it was otherwise. > > However, I find it troubling that we as Python developers are *forcing* > the whole Python world to put their code into PyPI. Forcing is a strong word. As of right now we?re "forcing" you to put it onto PyPI if you want to install it without *any* flags to pip. You're more then capable of hosting it externally if you wish, and pip will even tell the people who try to install it what they need to enable in order to install it. It even allows people to say "I don't care about any of this crap, just make all the external stuff work". Even if pip removed the external link handling all together and required you to do something like: $ pip install --extra-index-url https://example.com/our-packages/ foo $ pip install --find-links https://example.com/our-packages/ foo We still wouldn't be forcing anyone to upload things to PyPI. We are, however, discouraging people from not hosting on PyPI and providing incentives to doing that. > > There are plenty good reasons not to do this, and sometimes it's > even impossible if you want to stay legal (see the PEP for details). I re-read the reasons, I'm not sure I really agree with most (or all?) of them however if people really want to do it, then there is nothing stopping them. It's their choice and people on the *other* side of that who are installing these packages also get to make a choice if they want to allow it or not. > > Accordingly, we should respect those reasons make it possible for > Python packages to live elsewhere, without having our tools put > those packages into a bad light or making it harder for Python > users to install such packages than needed. I'm not sure what "put it into a bad light" is supposed to mean here. Is it about the warning? I've already removed that completely. As far as making it harder than it "needs" to be, that's a hard line to draw. I personally know people for whom things that are hosted externally have caused them a lot of pain to the point where they have to automatically spider PyPI for their own dependencies every hour or so in order to isolate themselves from the failure of random dependencies suddenly no longer being installable. This breakage is going to happen for basically any project that installs their dependencies with any frequency. One of the big projects that I'm aware of that has had this problem is Openstack who, in their CI, installs things several hundred times a day. If a service they depend on goes down that causes significant disruption for them. Now they've solved it by the above solution of hosting their own mirror of PyPI that includes spidering for externally installable things. They have one or two packages left that don't host directly on PyPI at which point they'll no longer need that afaik. I don't think telling every downstream of us of any size that in order to get reliable installs they are going to have to work around PyPI, and in some cases even develop custom software in order to do so effectively, is a very user friendly position. > > With the checksum uploaded to PyPI, the only argument against > fetching packages from other places on the Internet is reliability, > not security. I've never said differently. There are some minor privacy things in there but primarily it's about reliability and that people should generally be aware where their packages are coming from. > > PyPI is not the only reliable file hosting system on the Internet, > so this argument is rather weak. It actually doesn't matter how reliable the other systems are. Reliability wise the best possible outcome is the external host has 100% (extremely unlikely) uptime and it has no affect on reliability. The realistic outcome is that the external hosts will not have 100% uptime and will infact decrease the total availability of the install sets on PyPI. This isn't an opinion it is fact. But in reality reliability also includes more things than just the server temporarily going down. It also includes things like the maintainer has quit maintaining that package, left Python all together, died even. There are a decent number of packages on PyPI that people use which have no active maintainer at all. If people rely on a package that isn't hosted on PyPI they take a larger risk that their dependencies are going to become uninstallable due to the above. When scanning the unverifiable links it is amazing how many of those links are now 404s or 500s or even "nodename not founds". All of these cause issues for people and are not hypothetical, I've dealt with people complaining about one or another of them. When things are hosted on PyPI then we (we being the infra team) are able to ensure that the likelihood of ``pip install `` actually works and continues to work. When things are hosted externally the best we can do is say sorry. At that point the only major thing outside of our control that can cause someones installations to suddenly start failing is if the package author willfully removes their things from PyPI. That is unfortunate but it is also their right and there's nothing to be done for that. Thankfully that case is far less common than the case of "an external site just isn't there". I think it's important to point out that one of the driving factors that caused me to finally push for changes and what lead to PEP438 being created was that Mercurial's external hosted was being extremely flaky. I can't remember the exact details but I want to say that over the span of a week or two I was getting massive numbers of users complaining that ``pip install Mercurial`` was suddenly failing. This isn't to knock on the Mercurial folks or anything but to simply point out that these problems aren't things that just happen to (under|un)maintained software nor are they hypothetical. This PEP was born of the frustration that was being relayed to me by end users of PyPI/pip. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 13:55:57 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 12:55:57 +0100 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> Message-ID: On 9 May 2014 12:44, Donald Stufft wrote: > We still wouldn't be forcing anyone to upload things to PyPI. We are, however, > discouraging people from not hosting on PyPI and providing incentives to doing > that. But you're doing so by inflicting pain on people using pip to install those packages. Those users complain about *pip*, not about the packages. Better to directly impact the package maintainers, rather than their users (who are innocent victims). Better still of course to encourage people to improve things, not to punish them for not doing so. > I think it's important to point out that one of the driving factors that caused > me to finally push for changes and what lead to PEP438 being created was that > Mercurial's external hosted was being extremely flaky. I can't remember the > exact details but I want to say that over the span of a week or two I was > getting massive numbers of users complaining that ``pip install Mercurial`` > was suddenly failing. This isn't to knock on the Mercurial folks or anything > but to simply point out that these problems aren't things that just happen to > (under|un)maintained software nor are they hypothetical. This PEP was born of > the frustration that was being relayed to me by end users of PyPI/pip. So now "pip install Mercurial" always fails? And adding a flag allows it to work as well as before, but no better? How did that fix the issue? Seriously - I'm missing something here. Paul From donald at stufft.io Fri May 9 14:02:12 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 08:02:12 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io> Message-ID: <35017217-6F7A-455C-BF14-AEB58FB24955@stufft.io> On May 9, 2014, at 5:01 AM, Paul Moore wrote: > On 9 May 2014 05:34, Donald Stufft wrote: >> On May 8, 2014, at 5:22 PM, Donald Stufft wrote: >> >>>> Socially, this change does not seem to be having the effect of >>>> persuading more package developers to host on PyPI. The stick doesn't >>>> appear to have worked, maybe we should be trying to find a carrot? >>> >>> Do you have any data to point to that says it hasn?t worked? Just to see >>> what impact it has had, I?m running my scripts again that I ran a year >>> ago to see what has changed, already I can see they are processing >>> MUCH faster than last year. >> >> The data has finished processing, it represents a time diff of approximately >> one year. The pip release that caused all of this was released about 4-5 months >> ago. >> >> Overall PyPI has seen a 50% growth in installable projects in that time. If the >> change would have had no effect we'd expect to see a ~50% increase across the >> board. However what we've seen is a a 60% (+10% of expected) increase in >> projects that can only be installed from PyPI and a 12% decrease in projects >> that have any unsafe files (-62% of expected). > > Donald, > Thanks for taking the time to get those figures. It does appear that > there are less cases that would be affected than the number of > complaints would imply. Of course, I don?t like making claims without backing them up if I can :) > > The only concern I have about this type of analysis is that it doesn't > "weight" projects. It may be (and again, I have no data to back this > up) that the projects that are affected detrimentally by this change > are unusually popular or otherwise significant. There's obviously no > way to assess this sensibly other than by making a judgement on the > level of complaints. Yea, I don?t have a good way to weight those projects in any way. Normally I could get some sort of estimate by looking at the download numbers from PyPI but well ;) For the record, here?s the list of projects that are hosted *only* safely externally or that have *any* safely externally hosted files: https://gist.github.com/dstufft/1b16c305f97fff6cef2f Most of these don?t stand out to me at all. The only ones that do are: * pyOpenSSL which has one older release that is hosted that way * argparse which has the latest release hosted this way but has older releases hosted on PyPI * new relic which only hosts older releases externally * beautifulsoup4 which hosts things safely externally *and* on PyPI * Paste which has one ?external? thing which is actually only external because it used a cheeseshop.python.org link instead of a pypi.python.org link. * ipython which has one older release hosted safely externally but the latest is on PyPI * netifaces which has one older release hosted safely externally but the rest are on PyPI > > But arguing numbers was never my intention here, so let's just say > that I concede that the change has had a positive effect, which is > great. > Paul I didn?t mean to try to imply that it was :) I just wanted to make sure that *my* claims were true, or if they weren?t I wanted to be able to say that I was wrong. Since I had the numbers computed already it didn?t make any sense not to share them here. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri May 9 14:02:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 May 2014 22:02:53 +1000 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> Message-ID: On 9 May 2014 21:55, Paul Moore wrote: > On 9 May 2014 12:44, Donald Stufft wrote: >> I think it's important to point out that one of the driving factors that caused >> me to finally push for changes and what lead to PEP438 being created was that >> Mercurial's external hosted was being extremely flaky. I can't remember the >> exact details but I want to say that over the span of a week or two I was >> getting massive numbers of users complaining that ``pip install Mercurial`` >> was suddenly failing. This isn't to knock on the Mercurial folks or anything >> but to simply point out that these problems aren't things that just happen to >> (under|un)maintained software nor are they hypothetical. This PEP was born of >> the frustration that was being relayed to me by end users of PyPI/pip. > > So now "pip install Mercurial" always fails? And adding a flag allows > it to work as well as before, but no better? How did that fix the > issue? Seriously - I'm missing something here. Mercurial downloads are now available directly from PyPI. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri May 9 14:06:19 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 08:06:19 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> Message-ID: <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> On May 9, 2014, at 7:55 AM, Paul Moore wrote: > On 9 May 2014 12:44, Donald Stufft wrote: >> We still wouldn't be forcing anyone to upload things to PyPI. We are, however, >> discouraging people from not hosting on PyPI and providing incentives to doing >> that. > > But you're doing so by inflicting pain on people using pip to install > those packages. Those users complain about *pip*, not about the > packages. Better to directly impact the package maintainers, rather > than their users (who are innocent victims). Better still of course to > encourage people to improve things, not to punish them for not doing > so. We can?t directly impact the package maintainers and the vast bulk of people who have had a problem who have complained about it to pip also need to add the ?allow-unverifiable flag and would not simply be able to be fixed by allowing safely externally hosted files. Looking at the numbers and what packages are hosted externally, allowing safely externally hosted files would have practically no benefit to pip?s end users. The only case that I can see with a quick scan would be it would allow the latest version of argparse. TBH I think the biggest source of confusion reduction would be to remove the ?safely externally hosted? category all together and just make it hosted on PyPI -> Install by default, hosted off PyPI -> requires a per package flag. However I?m sure the vocal minority would have a problem with that. > >> I think it's important to point out that one of the driving factors that caused >> me to finally push for changes and what lead to PEP438 being created was that >> Mercurial's external hosted was being extremely flaky. I can't remember the >> exact details but I want to say that over the span of a week or two I was >> getting massive numbers of users complaining that ``pip install Mercurial`` >> was suddenly failing. This isn't to knock on the Mercurial folks or anything >> but to simply point out that these problems aren't things that just happen to >> (under|un)maintained software nor are they hypothetical. This PEP was born of >> the frustration that was being relayed to me by end users of PyPI/pip. > > So now "pip install Mercurial" always fails? And adding a flag allows > it to work as well as before, but no better? How did that fix the > issue? Seriously - I'm missing something here. No, This caused Mercurial to upload their packages to PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri May 9 14:21:14 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 13:21:14 +0100 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> Message-ID: On 9 May 2014 13:06, Donald Stufft wrote: >>> I think it's important to point out that one of the driving factors that caused >>> me to finally push for changes and what lead to PEP438 being created was that >>> Mercurial's external hosted was being extremely flaky. I can't remember the >>> exact details but I want to say that over the span of a week or two I was >>> getting massive numbers of users complaining that ``pip install Mercurial`` >>> was suddenly failing. This isn't to knock on the Mercurial folks or anything >>> but to simply point out that these problems aren't things that just happen to >>> (under|un)maintained software nor are they hypothetical. This PEP was born of >>> the frustration that was being relayed to me by end users of PyPI/pip. >> >> So now "pip install Mercurial" always fails? And adding a flag allows >> it to work as well as before, but no better? How did that fix the >> issue? Seriously - I'm missing something here. > > No, This caused Mercurial to upload their packages to PyPI. You're claiming that Mercurial moved to hosting on PyPI solely because users suddenly needed to add a flag to install from pip? As opposed to because PyPI gave them a more reliable hosting platform, for example? OK. I certainly can't give any evidence to dispute that claim, although I'm surprised. Paul From donald at stufft.io Fri May 9 14:25:34 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 08:25:34 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> Message-ID: <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io> On May 9, 2014, at 8:21 AM, Paul Moore wrote: > On 9 May 2014 13:06, Donald Stufft wrote: >>>> I think it's important to point out that one of the driving factors that caused >>>> me to finally push for changes and what lead to PEP438 being created was that >>>> Mercurial's external hosted was being extremely flaky. I can't remember the >>>> exact details but I want to say that over the span of a week or two I was >>>> getting massive numbers of users complaining that ``pip install Mercurial`` >>>> was suddenly failing. This isn't to knock on the Mercurial folks or anything >>>> but to simply point out that these problems aren't things that just happen to >>>> (under|un)maintained software nor are they hypothetical. This PEP was born of >>>> the frustration that was being relayed to me by end users of PyPI/pip. >>> >>> So now "pip install Mercurial" always fails? And adding a flag allows >>> it to work as well as before, but no better? How did that fix the >>> issue? Seriously - I'm missing something here. >> >> No, This caused Mercurial to upload their packages to PyPI. > > You're claiming that Mercurial moved to hosting on PyPI solely because > users suddenly needed to add a flag to install from pip? As opposed to > because PyPI gave them a more reliable hosting platform, for example? > OK. I certainly can't give any evidence to dispute that claim, > although I'm surprised. > > Paul I don?t know that for a fact but If my memory is correct that?s a reasonable assumption based on the timeline. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at bytereef.org Fri May 9 14:39:03 2014 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 9 May 2014 14:39:03 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> Message-ID: <20140509123903.GA7377@sleipnir.bytereef.org> Paul Moore wrote: > You're claiming that Mercurial moved to hosting on PyPI solely because > users suddenly needed to add a flag to install from pip? As opposed to > because PyPI gave them a more reliable hosting platform, for example? > OK. I certainly can't give any evidence to dispute that claim, > although I'm surprised. That is my understanding of the posted statistics, too. PyPI was overloaded, the improved infrastructure fixed that, so more projects now host on PyPI. Stefan Krah From p.f.moore at gmail.com Fri May 9 14:47:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 May 2014 13:47:49 +0100 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io> Message-ID: On 9 May 2014 13:25, Donald Stufft wrote: >> You're claiming that Mercurial moved to hosting on PyPI solely because >> users suddenly needed to add a flag to install from pip? As opposed to >> because PyPI gave them a more reliable hosting platform, for example? >> OK. I certainly can't give any evidence to dispute that claim, >> although I'm surprised. > > I don?t know that for a fact but If my memory is correct that?s a reasonable > assumption based on the timeline. As Stefan pointed out, improved PyPI infrastructure such as the CDN seems like a much more likely reason for Mercurial to have switched. Paul From solipsis at pitrou.net Fri May 9 14:59:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 May 2014 14:59:21 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io> <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io> Message-ID: <20140509145921.5fa6cee5@fsol> On Fri, 9 May 2014 13:47:49 +0100 Paul Moore wrote: > On 9 May 2014 13:25, Donald Stufft wrote: > >> You're claiming that Mercurial moved to hosting on PyPI solely because > >> users suddenly needed to add a flag to install from pip? As opposed to > >> because PyPI gave them a more reliable hosting platform, for example? > >> OK. I certainly can't give any evidence to dispute that claim, > >> although I'm surprised. > > > > I don?t know that for a fact but If my memory is correct that?s a reasonable > > assumption based on the timeline. > > As Stefan pointed out, improved PyPI infrastructure such as the CDN > seems like a much more likely reason for Mercurial to have switched. Here is the only reference I could find: http://www.selenic.com/pipermail/mercurial-devel/2013-July/052129.html Regards Antoine. From ethan at stoneleaf.us Fri May 9 15:09:28 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 09 May 2014 06:09:28 -0700 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> Message-ID: <536CD388.2040708@stoneleaf.us> On 05/08/2014 02:02 PM, Paul Moore wrote: > > Socially, this change does not seem to be having the effect of > persuading more package developers to host on PyPI. The stick doesn't > appear to have worked, maybe we should be trying to find a carrot? Or > maybe we have to accept that some developers have sound reasons for > not hosting on PyPI and work with them to find an acceptable > compromise? Has anyone checked what Stefan's reasons are for not > hosting cdecimal on PyPI? Do they represent a use case that the PEP > hasn't considered? Well, I do host a small handful of modules on PyPI, but I can say that some of my pain points are: - getting a good name: the obvious ones are taken, so the search begins to find a name that is not taken and yet still feels at least somewhat appropriate: my OO path module ended up being called strpath (dumb name); eventually changed to antipathy my script param line parser is called scription (okay name) my enum backport is called enum34 (blah) my dbf package is called dbf (lucky lucky lucky!) - reputation of PyPI is not that great: anybody can upload packages, no curating is done, once there the name is taken for all time, no standard version numbering, . . . (granted, these are my impressions and may not be entirely valid) Anyway, I know it's a volunteer effort I don't want to criticize those actually running it, but these are some of the problems that I see with the service itself. -- ~Ethan~ From mal at egenix.com Fri May 9 15:58:39 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 May 2014 15:58:39 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> Message-ID: <536CDF0F.6050407@egenix.com> On 09.05.2014 13:44, Donald Stufft wrote: > > On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >> Donald: I don't think anyone is arguing that hosting packages on >> PyPI is a bad thing and PyPI as a service has gotten a lot better >> than it was a few years ago. > > Didn?t mean to imply that I thought it was otherwise. > >> >> However, I find it troubling that we as Python developers are *forcing* >> the whole Python world to put their code into PyPI. > > Forcing is a strong word. As of right now we?re "forcing" you to put it onto > PyPI if you want to install it without *any* flags to pip. Which is what most people expect to be able to do. > You're more then > capable of hosting it externally if you wish, and pip will even tell the people > who try to install it what they need to enable in order to install it. It even > allows people to say "I don't care about any of this crap, just make all the > external stuff work". > > Even if pip removed the external link handling all together and required you > to do something like: > > $ pip install --extra-index-url https://example.com/our-packages/ foo > $ pip install --find-links https://example.com/our-packages/ foo > > We still wouldn't be forcing anyone to upload things to PyPI. We are, however, > discouraging people from not hosting on PyPI and providing incentives to doing > that. This is all true, but in reality, you are making externally hosted packages second class citizens in Python land by requiring extra options even for package listings that are perfectly safe to download from other servers. As mentioned before: I can understand that you want to make downloads safe for users, but if a package is hosted on some other reliable servers and pip can check that it's a valid package, there's little to argue for not allowing such downloads. >> There are plenty good reasons not to do this, and sometimes it's >> even impossible if you want to stay legal (see the PEP for details). > > I re-read the reasons, I'm not sure I really agree with most (or all?) of them > however if people really want to do it, then there is nothing stopping them. > It's their choice and people on the *other* side of that who are installing > these packages also get to make a choice if they want to allow it or not. You don't have to agree with those reasons. Fact is, they exist, prevent package authors from uploading to PyPI and we as Python developers should respect those reasons. With the new infrastructure, it's far more attractive to host packages on PyPI than it was before, so people who do not host on PyPI will have carefully thought about this. >> Accordingly, we should respect those reasons make it possible for >> Python packages to live elsewhere, without having our tools put >> those packages into a bad light or making it harder for Python >> users to install such packages than needed. > > I'm not sure what "put it into a bad light" is supposed to mean here. Is it > about the warning? I've already removed that completely. It's using the same unfortunate strategy that setuptools used by requiring an option called --single-version-externally-managed to be able to install a package without all the .pth and egg logic (an option that pip now uses per default, BTW ;-)). I snipped the rest of the discussion and reliability, using unmaintained packages and projects using their own mirrors (which should really be the standard, not an exceptional case), because it's not really leading anywhere: At the time we discussed the PEP, security was the main concern, not reliability. Now that there is a secure way to reference external distribution files, I think we should revisit the defaults decision in the light of the new possibilities. BTW: The PEP mentions re-hosting tools to upload their packages to PyPI and also says "This re-hosting tool MUST be available before automated hosting-mode changes are announced to package maintainers." I am not aware of such tools. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From phd at phdru.name Fri May 9 16:08:55 2014 From: phd at phdru.name (Oleg Broytman) Date: Fri, 9 May 2014 16:08:55 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536CD388.2040708@stoneleaf.us> References: <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536CD388.2040708@stoneleaf.us> Message-ID: <20140509140855.GA1076@phdru.name> On Fri, May 09, 2014 at 06:09:28AM -0700, Ethan Furman wrote: > On 05/08/2014 02:02 PM, Paul Moore wrote: > Well, I do host a small handful of modules on PyPI, but I can say that some of my pain points are: > > - getting a good name: the obvious ones are taken, so the search > begins to find a name that is not taken and yet still feels at > least somewhat appropriate: > > my OO path module ended up being called strpath (dumb name); > eventually changed to antipathy > > my script param line parser is called scription (okay name) > > my enum backport is called enum34 (blah) > > my dbf package is called dbf (lucky lucky lucky!) For many reasons I avoid github/gitorious/bitbucket, but there is one thing they do right -- two-levels namespaces. "Namespaces are one honking great idea -- let's do more of those!" Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From donald at stufft.io Fri May 9 17:39:02 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 11:39:02 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <536CDF0F.6050407@egenix.com> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> Message-ID: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > On 09.05.2014 13:44, Donald Stufft wrote: >> >> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >>> Donald: I don't think anyone is arguing that hosting packages on >>> PyPI is a bad thing and PyPI as a service has gotten a lot better >>> than it was a few years ago. >> >> Didn?t mean to imply that I thought it was otherwise. >> >>> >>> However, I find it troubling that we as Python developers are *forcing* >>> the whole Python world to put their code into PyPI. >> >> Forcing is a strong word. As of right now we?re "forcing" you to put it onto >> PyPI if you want to install it without *any* flags to pip. > > Which is what most people expect to be able to do. Sure, but sometimes it?s better to make an informed choice about things being installed instead of having it be installed by default and sometimes it?s better to disallow doing something at all. Further most people don?t expect an install to touch any server host other than PyPI. As far as I?m aware none of the other package repositories feature this, and even with the fact we support it barely anyone even cares to utilize it. > >> You're more then >> capable of hosting it externally if you wish, and pip will even tell the people >> who try to install it what they need to enable in order to install it. It even >> allows people to say "I don't care about any of this crap, just make all the >> external stuff work". >> >> Even if pip removed the external link handling all together and required you >> to do something like: >> >> $ pip install --extra-index-url https://example.com/our-packages/ foo >> $ pip install --find-links https://example.com/our-packages/ foo >> >> We still wouldn't be forcing anyone to upload things to PyPI. We are, however, >> discouraging people from not hosting on PyPI and providing incentives to doing >> that. > > This is all true, but in reality, you are making externally hosted > packages second class citizens in Python land by requiring > extra options even for package listings that are perfectly safe > to download from other servers. > > As mentioned before: I can understand that you want to make downloads > safe for users, but if a package is hosted on some other reliable > servers and pip can check that it's a valid package, there's little > to argue for not allowing such downloads. Except the fact that the only people I?ve ever seen *happy* that people can host packages externally are a handful (<5) of people (tbh primarily you and Stefan) and the feedback I get from every other person around the web in unequivocally yes please get rid of externally hosted files, they?ve done nothing but cause me pain. It?s not reasonable to allow a minority of users to negatively impact the majority. > >>> There are plenty good reasons not to do this, and sometimes it's >>> even impossible if you want to stay legal (see the PEP for details). >> >> I re-read the reasons, I'm not sure I really agree with most (or all?) of them >> however if people really want to do it, then there is nothing stopping them. >> It's their choice and people on the *other* side of that who are installing >> these packages also get to make a choice if they want to allow it or not. > > You don't have to agree with those reasons. Fact is, they exist, > prevent package authors from uploading to PyPI and we as > Python developers should respect those reasons. No I disagree that they are reasons at all. Half of the ones you enumerated are nonsensical or irrelevant, the other half can be fixed by adding features to or fixing PyPI. One or two read like reasons why someone might actually decide not to upload something to PyPI but which that reason isn?t really all that reasonable and finally a grand total of one or two of them look like an actual legit reasons and it only applies to Crypto software. I mean your reasoning included things like: > PyPI doesn?t let you upload two files with the same name, you gave the > example of UC2 vs UC4 The thing being if PyPI did allow you to do that then we?d have no way to determine which one is the right one and half of the people would just get randomly failing installs because we guessed the wrong one. > They don?t know it?s possible to upload to PyPI I don?t even know how to response to this one. > PyPI used to be unreliable but isn?t now and they don?t know about it Nor this one. > Not wanting people to download a package at all and instead check out > from a VCS repo This?ll randomly fail for people depending on if they happen to have that VCS installed. Also I don?t see any evidence that people are actually doing this outside of supporting a foo==dev install which pip won?t install by default either. > Not wanting to add latency for yourself for downloading from PyPI Instead you?re going to inflict extra latency on everyone else when you could just run a mirror or upload the packages to your own server. > File types that PyPI doesn?t support and which utilize a third party package > format, PyPM being a given example. Seriously? Should an apt repo support external links because you can?t install RPMs with apt? > Wanting people to have to read something before installing the package > because it relies on some additional setup. This is completely nonsensical if they want people to have to read something before they attempt to pip install it then they don?t want pip to discover the external file at all then. > It currently takes too long uploading that many files to > PyPI. This causes a problem, since in order to start the upload, > we have to register the release on PyPI, which tools will then > immediately find. However, during the upload time, they won't > necessarily find the right files to download and then fail. So if this is an actual problem file a bug and we can make it so PyPI lets you upload things as ?drafts? and then you can publish them all at once or some such thing. > having a strong need to know the number of downloads per > package and associated statistics such as downloads per > country, per year/month/day/hour If additional statistics are desired for PyPI file a bug and we can add them. > not wanting to give up access to the download log files This appears to be repeating the last reason with different wording. > the PyPI uploads not being compatible to their release process I?m not sure how it?s possible to have a release process where it isn?t compatible to upload to an additional server. Maybe ones that don?t currently do it but ones that are fundamentally incompatible? Almost all of your ?reasons? read like someone who maybe had one corner case where it might have made sense but didn?t want to admit that it was a corner case so tried to think up any random reason they could think of in order to make the list look longer and more impressive than it actually was. > > With the new infrastructure, it's far more attractive to > host packages on PyPI than it was before, so people who do > not host on PyPI will have carefully thought about this. > >>> Accordingly, we should respect those reasons make it possible for >>> Python packages to live elsewhere, without having our tools put >>> those packages into a bad light or making it harder for Python >>> users to install such packages than needed. >> >> I'm not sure what "put it into a bad light" is supposed to mean here. Is it >> about the warning? I've already removed that completely. > > It's using the same unfortunate strategy that setuptools used by > requiring an option called --single-version-externally-managed > to be able to install a package without all the .pth and egg > logic (an option that pip now uses per default, BTW ;-)). > > I snipped the rest of the discussion and reliability, using > unmaintained packages and projects using their own mirrors (which > should really be the standard, not an exceptional case), > because it's not really leading anywhere: Using your own mirror shouldn?t be the standard if all you?re doing is automatically updating that mirror. It?s a hack to get around unreliability and it should be seen of as a sign of a failure to provide a service that people can rely on and that?s how I see it. People depend on this service and it?s irresponsible to not treat it as a critical piece of infrastructure. > > At the time we discussed the PEP, security was the main concern, > not reliability. The main concern, not the only concern. > > Now that there is a secure way to reference external distribution > files, I think we should revisit the defaults decision in the > light of the new possibilities. Re-evaluate all you want. If they change it will be over my strenuous objections. The feedback I get from users is overwhelmly positive. Even the ones who get struck by needing to add extra flags tell me how they are glad that pip and PyPI are moving towards a more reliable model. > > > BTW: The PEP mentions re-hosting tools to upload their packages > to PyPI and also says "This re-hosting tool MUST be available > before automated hosting-mode changes are announced to package > maintainers." I am not aware of such tools. It appears this got missed. I?ll add it to my stack to remedy this. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, May 09 2014) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From status at bugs.python.org Fri May 9 18:07:57 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 9 May 2014 18:07:57 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140509160757.F39D6568E0@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-05-02 - 2014-05-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4611 ( +6) closed 28624 (+42) total 33235 (+48) Open issues with patches: 2119 Issues opened (35) ================== #18604: Consolidate gui available checks in test.support http://bugs.python.org/issue18604 reopened by ned.deily #21416: argparse should accept bytes arguments as originally passed http://bugs.python.org/issue21416 opened by underrun #21417: Compression level for zipfile http://bugs.python.org/issue21417 opened by Sworddragon #21418: Segv during call to super_init in application embedding Python http://bugs.python.org/issue21418 opened by snoeberger #21419: Use calloc() instead of malloc() for int << int (lshift) http://bugs.python.org/issue21419 opened by haypo #21420: Optimize 2 ** n: implement it as 1 << n http://bugs.python.org/issue21420 opened by haypo #21422: int << 0: return the number unmodified http://bugs.python.org/issue21422 opened by haypo #21423: concurrent.futures.ThreadPoolExecutor should accept an initial http://bugs.python.org/issue21423 opened by andreasvc #21424: Simplify and speed-up heaqp.nlargest() http://bugs.python.org/issue21424 opened by rhettinger #21425: Python 3 pipe handling breaks python mode in emacs on Windows http://bugs.python.org/issue21425 opened by marczellm #21427: installer not working http://bugs.python.org/issue21427 opened by ellipso #21428: Python suddenly cares about EOLs formats on windows http://bugs.python.org/issue21428 opened by pygnol #21429: Input.output error with multiprocessing http://bugs.python.org/issue21429 opened by mikaela #21430: Document ssl.pending() http://bugs.python.org/issue21430 opened by shevek #21431: 3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are http://bugs.python.org/issue21431 opened by ned.deily #21434: python -3 documentation is outdated http://bugs.python.org/issue21434 opened by berker.peksag #21436: Consider leaving importlib.abc.Loader.load_module() http://bugs.python.org/issue21436 opened by srittau #21437: document that asyncio.ProactorEventLoop doesn't support SSL http://bugs.python.org/issue21437 opened by akira #21439: Numerous minor issues in Language Reference http://bugs.python.org/issue21439 opened by zach.ware #21441: Buffer Protocol Documentation Error http://bugs.python.org/issue21441 opened by Jake.Vanderplas #21443: asyncio logging documentation clarifications http://bugs.python.org/issue21443 opened by hardkrash #21445: Some asserts in test_filecmp have the wrong messages http://bugs.python.org/issue21445 opened by Steven.Barker #21446: Update reload fixer to use importlib instead of imp http://bugs.python.org/issue21446 opened by berker.peksag #21447: Intermittent asyncio.open_connection / futures.InvalidStateErr http://bugs.python.org/issue21447 opened by ryder.lewis #21448: Email Parser use 100% CPU http://bugs.python.org/issue21448 opened by jader.fabiano #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 opened by josh.rosenberg #21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat http://bugs.python.org/issue21452 opened by mloskot #21453: Support of RPM subpackages in distutils http://bugs.python.org/issue21453 opened by vitalyisaev2 #21454: asyncio's loop.connect_read_pipe makes pipes non-blocking cont http://bugs.python.org/issue21454 opened by John Isidore #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 opened by neologix #21456: skip 2 tests in test_urllib2net.py if _ssl module not present http://bugs.python.org/issue21456 opened by rpointel #21457: NetBSD curses support improvements http://bugs.python.org/issue21457 opened by wiz #21461: Recognize -pthread http://bugs.python.org/issue21461 opened by wiz #21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds http://bugs.python.org/issue21462 opened by ncoghlan #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 opened by erik.bray Most recent 15 issues with no replies (15) ========================================== #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 #21461: Recognize -pthread http://bugs.python.org/issue21461 #21453: Support of RPM subpackages in distutils http://bugs.python.org/issue21453 #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 #21445: Some asserts in test_filecmp have the wrong messages http://bugs.python.org/issue21445 #21443: asyncio logging documentation clarifications http://bugs.python.org/issue21443 #21441: Buffer Protocol Documentation Error http://bugs.python.org/issue21441 #21439: Numerous minor issues in Language Reference http://bugs.python.org/issue21439 #21437: document that asyncio.ProactorEventLoop doesn't support SSL http://bugs.python.org/issue21437 #21434: python -3 documentation is outdated http://bugs.python.org/issue21434 #21429: Input.output error with multiprocessing http://bugs.python.org/issue21429 #21425: Python 3 pipe handling breaks python mode in emacs on Windows http://bugs.python.org/issue21425 #21424: Simplify and speed-up heaqp.nlargest() http://bugs.python.org/issue21424 #21418: Segv during call to super_init in application embedding Python http://bugs.python.org/issue21418 #21417: Compression level for zipfile http://bugs.python.org/issue21417 Most recent 15 issues waiting for review (15) ============================================= #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 #21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds http://bugs.python.org/issue21462 #21461: Recognize -pthread http://bugs.python.org/issue21461 #21457: NetBSD curses support improvements http://bugs.python.org/issue21457 #21456: skip 2 tests in test_urllib2net.py if _ssl module not present http://bugs.python.org/issue21456 #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 #21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat http://bugs.python.org/issue21452 #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 #21446: Update reload fixer to use importlib instead of imp http://bugs.python.org/issue21446 #21445: Some asserts in test_filecmp have the wrong messages http://bugs.python.org/issue21445 #21441: Buffer Protocol Documentation Error http://bugs.python.org/issue21441 #21434: python -3 documentation is outdated http://bugs.python.org/issue21434 #21423: concurrent.futures.ThreadPoolExecutor should accept an initial http://bugs.python.org/issue21423 #21422: int << 0: return the number unmodified http://bugs.python.org/issue21422 #21420: Optimize 2 ** n: implement it as 1 << n http://bugs.python.org/issue21420 Top 10 most discussed issues (10) ================================= #21233: Add *Calloc functions to CPython memory allocation API http://bugs.python.org/issue21233 15 msgs #21121: -Werror=declaration-after-statement is added even for extensio http://bugs.python.org/issue21121 14 msgs #21419: Use calloc() instead of malloc() for int << int (lshift) http://bugs.python.org/issue21419 10 msgs #21448: Email Parser use 100% CPU http://bugs.python.org/issue21448 8 msgs #20866: segfailt with os.popen and SIGPIPE http://bugs.python.org/issue20866 6 msgs #21083: Add get_content_disposition() to email.message.Message http://bugs.python.org/issue21083 6 msgs #21420: Optimize 2 ** n: implement it as 1 << n http://bugs.python.org/issue21420 6 msgs #21422: int << 0: return the number unmodified http://bugs.python.org/issue21422 6 msgs #21423: concurrent.futures.ThreadPoolExecutor should accept an initial http://bugs.python.org/issue21423 6 msgs #10752: build_ssl.py is relying on unreliable behaviour of os.popen http://bugs.python.org/issue10752 5 msgs Issues closed (42) ================== #5717: os.defpath includes unix /bin on windows http://bugs.python.org/issue5717 closed by tim.golden #13030: Be more generic when identifying the Windows main dir in insta http://bugs.python.org/issue13030 closed by tim.golden #17338: Add length_hint parameter to list, dict, set constructors to a http://bugs.python.org/issue17338 closed by rhettinger #17752: many distutils tests fail when run from the installed location http://bugs.python.org/issue17752 closed by doko #18154: make install failed on solaris http://bugs.python.org/issue18154 closed by palm.kevin #18211: -Werror=statement-after-declaration problem http://bugs.python.org/issue18211 closed by eric.araujo #18314: Have os.unlink remove junction points http://bugs.python.org/issue18314 closed by tim.golden #19414: iter(ordered_dict) yields keys not in dict in some circumstanc http://bugs.python.org/issue19414 closed by rhettinger #19643: shutil rmtree fails on readonly files in Windows http://bugs.python.org/issue19643 closed by tim.golden #20384: os.open() exception doesn't contain file name on Windows http://bugs.python.org/issue20384 closed by tim.golden #20642: Enhance deepcopy-ing for tuples http://bugs.python.org/issue20642 closed by python-dev #20737: 3.3 _thread lock.acquire() timeout and threading.Event().wait( http://bugs.python.org/issue20737 closed by kristjan.jonsson #20758: mimetypes initialization order http://bugs.python.org/issue20758 closed by tim.golden #21101: Extend the PyDict C API to handle cases where the hash value i http://bugs.python.org/issue21101 closed by rhettinger #21132: Failure to import win32api http://bugs.python.org/issue21132 closed by woakesd #21141: Don't mention Perl in Windows build output http://bugs.python.org/issue21141 closed by zach.ware #21157: Update imp docs for a PEP 451 world http://bugs.python.org/issue21157 closed by brett.cannon #21300: Docs (incorrectly) suggest email.policy.default is the default http://bugs.python.org/issue21300 closed by r.david.murray #21335: Update importlib.__init__ to reset _frozen_importlib's loader http://bugs.python.org/issue21335 closed by brett.cannon #21350: bug in file.writelines accepting buffers http://bugs.python.org/issue21350 closed by pitrou #21357: Increase filecmp test coverage from 63% to 76% http://bugs.python.org/issue21357 closed by python-dev #21366: Document that return in finally overwrites prev value http://bugs.python.org/issue21366 closed by zach.ware #21375: Fix and test overflow behavior in the C version of heapq http://bugs.python.org/issue21375 closed by rhettinger #21392: Python on Cygwin: subprocess.call BlockingIOError: [Errno 11] http://bugs.python.org/issue21392 closed by dellair.jie #21393: Python/random.c: close hCryptProv at exit http://bugs.python.org/issue21393 closed by haypo #21396: Python io implementation doesn't flush with write_through=True http://bugs.python.org/issue21396 closed by pitrou #21405: Allow using symbols from Unicode block "Superscripts and Subsc http://bugs.python.org/issue21405 closed by rominf #21414: Add an intersperse function to itertools http://bugs.python.org/issue21414 closed by rhettinger #21421: ABCs for MappingViews should declare __slots__ so subclasses a http://bugs.python.org/issue21421 closed by rhettinger #21426: Invisible characters in email related souce files. http://bugs.python.org/issue21426 closed by benjamin.peterson #21432: samba.git from git://git.samba.org/samba.git samba.netcmd.mai http://bugs.python.org/issue21432 closed by ned.deily #21433: "False = True" produces segfault http://bugs.python.org/issue21433 closed by ned.deily #21435: Segfault in gc with cyclic trash http://bugs.python.org/issue21435 closed by tim.peters #21438: Document which importlib.machinery loaders don't require an ar http://bugs.python.org/issue21438 closed by brett.cannon #21440: Use support.rmtree in test_zipfile http://bugs.python.org/issue21440 closed by tim.golden #21442: Win32 compiler warning in PyBytes_Concat http://bugs.python.org/issue21442 closed by zach.ware #21444: __len__ can't return big numbers http://bugs.python.org/issue21444 closed by benjamin.peterson #21450: [Issue 13630] IDLE: Find(ed) text is not highlighted while dia http://bugs.python.org/issue21450 closed by berker.peksag #21451: Improve error messages for malformed JSON http://bugs.python.org/issue21451 closed by r.david.murray #21458: MirBSD support http://bugs.python.org/issue21458 closed by brett.cannon #21459: DragonFlyBSD support http://bugs.python.org/issue21459 closed by brett.cannon #21460: distutils: use LDFLAGS http://bugs.python.org/issue21460 closed by wiz From stefan at bytereef.org Fri May 9 18:11:42 2014 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 9 May 2014 18:11:42 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> Message-ID: <20140509161142.GA8440@sleipnir.bytereef.org> Donald, I'm out of his discussion. I have one last request: please don't gossip about core devs in public as long as you have commit privs: https://botbot.me/freenode/python-requests/ Stefan Krah From mal at egenix.com Fri May 9 18:21:42 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 May 2014 18:21:42 +0200 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> Message-ID: <536D0096.3090405@egenix.com> On 09.05.2014 17:39, Donald Stufft wrote: > > On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > >> On 09.05.2014 13:44, Donald Stufft wrote: >>> >>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >>>> Donald: I don't think anyone is arguing that hosting packages on >>>> PyPI is a bad thing and PyPI as a service has gotten a lot better >>>> than it was a few years ago. >>> >>> Didn?t mean to imply that I thought it was otherwise. >>> >>>> >>>> However, I find it troubling that we as Python developers are *forcing* >>>> the whole Python world to put their code into PyPI. >>> >>> Forcing is a strong word. As of right now we?re "forcing" you to put it onto >>> PyPI if you want to install it without *any* flags to pip. >> >> Which is what most people expect to be able to do. > > Sure, but sometimes it?s better to make an informed choice about things being > installed instead of having it be installed by default and sometimes it?s better > to disallow doing something at all. > > Further most people don?t expect an install to touch any server host other > than PyPI. For some idea of "most people" that's probably true. I'd argue that most people probably don't really care all that much where their packages are coming from as long as they are getting the packages registered with PyPI, the Python package index, and can be sure that no one has tampered with them. But both arguments don't really count much, since it's all speculation. > As far as I?m aware none of the other package repositories > feature this, and even with the fact we support it barely anyone even cares to > utilize it. Most Linux repos comes with a list of standard repos to include. In Python land, Plone uses zc.buildout with a similar default list of repos. >>> You're more then >>> capable of hosting it externally if you wish, and pip will even tell the people >>> who try to install it what they need to enable in order to install it. It even >>> allows people to say "I don't care about any of this crap, just make all the >>> external stuff work". >>> >>> Even if pip removed the external link handling all together and required you >>> to do something like: >>> >>> $ pip install --extra-index-url https://example.com/our-packages/ foo >>> $ pip install --find-links https://example.com/our-packages/ foo >>> >>> We still wouldn't be forcing anyone to upload things to PyPI. We are, however, >>> discouraging people from not hosting on PyPI and providing incentives to doing >>> that. >> >> This is all true, but in reality, you are making externally hosted >> packages second class citizens in Python land by requiring >> extra options even for package listings that are perfectly safe >> to download from other servers. >> >> As mentioned before: I can understand that you want to make downloads >> safe for users, but if a package is hosted on some other reliable >> servers and pip can check that it's a valid package, there's little >> to argue for not allowing such downloads. > > Except the fact that the only people I?ve ever seen *happy* that people can > host packages externally are a handful (<5) of people (tbh primarily you and > Stefan) and the feedback I get from every other person around the web > in unequivocally yes please get rid of externally hosted files, they?ve done > nothing but cause me pain. > > It?s not reasonable to allow a minority of users to negatively impact the majority. I think you're getting this wrong: There *are* package authors who would like to host PyPI indexed packages external to PyPI and they all have good reasons to do so. It may well be a minority, but that's fine. Changing the default to allow secure external downloads is *not* impacting any majority. At worst, it's impacting the users of those packages, but that's really within the realm and responsibility of the package authors, not PyPI or the infrastructure team, so I don't understand why you are strongly objecting this. >>>> There are plenty good reasons not to do this, and sometimes it's >>>> even impossible if you want to stay legal (see the PEP for details). >>> >>> I re-read the reasons, I'm not sure I really agree with most (or all?) of them >>> however if people really want to do it, then there is nothing stopping them. >>> It's their choice and people on the *other* side of that who are installing >>> these packages also get to make a choice if they want to allow it or not. >> >> You don't have to agree with those reasons. Fact is, they exist, >> prevent package authors from uploading to PyPI and we as >> Python developers should respect those reasons. > > No I disagree that they are reasons at all. Half of the ones you enumerated are > nonsensical or irrelevant, the other half can be fixed by adding features to or > fixing PyPI. One or two read like reasons why someone might actually decide > not to upload something to PyPI but which that reason isn?t really all that > reasonable and finally a grand total of one or two of them look like an actual legit > reasons and it only applies to Crypto software. I guess continuing this discussion doesn't really make any sense. I'm trying to respect your reasoning, but you are apparently not respecting mine. Without such respect there's no basis for a discussion. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From donald at stufft.io Fri May 9 18:23:53 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 12:23:53 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140509161142.GA8440@sleipnir.bytereef.org> References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> <20140509161142.GA8440@sleipnir.bytereef.org> Message-ID: <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io> On May 9, 2014, at 12:11 PM, Stefan Krah wrote: > Donald, I'm out of his discussion. I have one last request: please don't > gossip about core devs in public as long as you have commit privs: > > https://botbot.me/freenode/python-requests/ I don?t really know how to respond to this. I was talking to some people I know and I gave them a summary of what was happening. I stand by everything I said there and I don?t think any of it was wrong or even gossip at all. If people feel that was inappropriate then go ahead and take my commit privs. I have them to make it easier for me to write PEPs and to update ensurepip. If they?re going to be used as an excuse to attempt to censor me then I?d rather not have them as I generally always speak my mind and I won?t stop doing so. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From graffatcolmingov at gmail.com Fri May 9 18:32:34 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 9 May 2014 11:32:34 -0500 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io> References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> <20140509161142.GA8440@sleipnir.bytereef.org> <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io> Message-ID: Stefan, If the only way you can think of to invalidate Donald's (vastly superior) arguments is to accuse of him of "gossip", you should probably reconsider your arguments. Looking at the conversation you didn't actually link to (https://botbot.me/freenode/python-requests/msg/14389415/) there is no gossip. There are no insinuations about your character. All that is there is a succinct description of the topic of this conversation. Cheers, Ian On Fri, May 9, 2014 at 11:23 AM, Donald Stufft wrote: > > On May 9, 2014, at 12:11 PM, Stefan Krah wrote: > >> Donald, I'm out of his discussion. I have one last request: please don't >> gossip about core devs in public as long as you have commit privs: >> >> https://botbot.me/freenode/python-requests/ > > I don?t really know how to respond to this. I was talking to some people I know > and I gave them a summary of what was happening. I stand by everything I > said there and I don?t think any of it was wrong or even gossip at all. > > If people feel that was inappropriate then go ahead and take my commit privs. I > have them to make it easier for me to write PEPs and to update ensurepip. If > they?re going to be used as an excuse to attempt to censor me then I?d rather > not have them as I generally always speak my mind and I won?t stop doing so. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com > From zachary.ware+pydev at gmail.com Fri May 9 19:00:04 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 9 May 2014 12:00:04 -0500 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? In-Reply-To: References: <536BDCB6.6030300@v.loewis.de> Message-ID: On Thu, May 8, 2014 at 4:20 PM, Zachary Ware wrote: > I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple > of weeks ago (see http://bugs.python.org/issue21303), but hadn't > gotten anything done with Tix yet. It should just need python.mak > updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as > soon as I can. Tix for 2.7 is now http://svn.python.org/projects/external/tix-8.4.3.5. You can build it with this monster of a command, run from tix-8.4.3.5\win: nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all Use "install" instead of "all" after building to install it to ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to be set if you're building Release, but MACHINE always has to be set so that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for 64-bit). -- Zach From rdmurray at bitdance.com Fri May 9 19:28:28 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 09 May 2014 13:28:28 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> Message-ID: <20140509172828.ECBA2250D4E@webabinitio.net> On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft wrote: > > On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: > > On 09.05.2014 13:44, Donald Stufft wrote: > >> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: > > I snipped the rest of the discussion and reliability, using > > unmaintained packages and projects using their own mirrors (which > > should really be the standard, not an exceptional case), > > because it's not really leading anywhere: > > Using your own mirror shouldn???t be the standard if all you???re doing > is automatically updating that mirror. It???s a hack to get around > unreliability and it should be seen of as a sign of a failure to provide > a service that people can rely on and that???s how I see it. People > depend on this service and it???s irresponsible to not treat it as a > critical piece of infrastructure. I don't understand this. Why it is our responsibility to provide a free service for a large project to repeatedly download a set of files they need? Why does it not make more sense for them to download them once, and only update their local copies when they change? That's almost completely orthogonal to making the service we do provide reliable. For perspective, Gentoo requests that people only do an emerge sync at most once a day, and if they have multiple machines to update, that they only do one pull, and they update the rest of their infrastructure from their local copy. As another point of information for comparison, Gentoo downloads files from wherever they are hosted first, and only if that fails falls back to a Gentoo provided mirror (if I remember correctly...I think the Gentoo mirror copy doesn't always exist?). --David From donald at stufft.io Fri May 9 20:12:22 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 14:12:22 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <20140509172828.ECBA2250D4E@webabinitio.net> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> <20140509172828.ECBA2250D4E@webabinitio.net> Message-ID: <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io> On May 9, 2014, at 1:28 PM, R. David Murray wrote: > On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft wrote: >> >> On May 9, 2014, at 9:58 AM, M.-A. Lemburg wrote: >>> On 09.05.2014 13:44, Donald Stufft wrote: >>>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg wrote: >>> I snipped the rest of the discussion and reliability, using >>> unmaintained packages and projects using their own mirrors (which >>> should really be the standard, not an exceptional case), >>> because it's not really leading anywhere: >> >> Using your own mirror shouldn?t be the standard if all you?re doing >> is automatically updating that mirror. It?s a hack to get around >> unreliability and it should be seen of as a sign of a failure to provide >> a service that people can rely on and that?s how I see it. People >> depend on this service and it?s irresponsible to not treat it as a >> critical piece of infrastructure. > > I don't understand this. Why it is our responsibility to provide a > free service for a large project to repeatedly download a set of files > they need? Why does it not make more sense for them to download them > once, and only update their local copies when they change? That's almost > completely orthogonal to making the service we do provide reliable. Well here?s the thing right. The large projects repeatedly downloading the same set of files is a canary. If any particular project goes uninstallable on PyPI (or if PyPI itself goes down) then nobody can install it, the people installing things over and over every day or the people who just happened to be installing it during that downtime. However intermittent failures and general insatiability is going to be noticed by the projects who install things over and over again quicker and easier and thus it becomes a lot easier to use them as a general gauge for what the average ?uptime? is. IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get a handful of ?small? installers (e.g. not the big projects) in each downtime, but a different set who are likely to shrug it off and just call treat it as the norm even though it?s very disruptive to what they?re doing. However the big project is highly likely to hit every single one of those downtimes and be able to say ?wow PyPI is flaky as hell?. To expand further on that if we assume that we want ``pip install `` to be reliable and not work sometimes and work at other times then we?re aiming for as high as uptime as possible. PyPI gets enough traffic that any single large project isn?t a noticeable drop in our bucket and due to the way our caching works it actually helps us to be faster and more reliable to have people constantly hitting packages because they?ll be in cache and able to be served without hitting the Origin servers. Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the month of April we served 71.4TB of data with 877.4 million requests of which 80.5% never made it to the actual servers that run PyPI and were served directly out of the geo distributed CDN that sits in front of PyPI. We are vastly better positioned to maintain a reliable infrastructure than ask that every large project that uses Python to do the same. The reason that it?s our responsibility for providing it is because we chose to provide it. There isn?t a moral imperative to run PyPI, but running PyPI badly seems like a crummy thing to do. > > For perspective, Gentoo requests that people only do an emerge sync at > most once a day, and if they have multiple machines to update, that they > only do one pull, and they update the rest of their infrastructure from > their local copy. To be clear, there are other reasons to run a local mirror but I don?t think that it?s reasonable to expect anyone who wants a reliable install using pip to stand up their own infrastructure. Further to this point here I?m currently working on adding caching by default for pip so that we minimize how often different people hit PyPI and we do it automatically and in a way that doesn?t generally require people to think about it and that also doesn?t require them to stand up their own infra. > > As another point of information for comparison, Gentoo downloads files > from wherever they are hosted first, and only if that fails falls back to > a Gentoo provided mirror (if I remember correctly...I think the Gentoo > mirror copy doesn't always exist?). > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tjreedy at udel.edu Fri May 9 22:20:32 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 09 May 2014 16:20:32 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io> References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> <20140509172828.ECBA2250D4E@webabinitio.net> <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io> Message-ID: On 5/9/2014 2:12 PM, Donald Stufft wrote: > > On May 9, 2014, at 1:28 PM, R. David Murray wrote: >> I don't understand this. Why it is our responsibility to provide a >> free service for a large project to repeatedly download a set of files >> they need? Why does it not make more sense for them to download them >> once, and only update their local copies when they change? That's almost >> completely orthogonal to making the service we do provide reliable. > > Well here?s the thing right. The large projects repeatedly downloading the > same set of files is a canary. If any particular project goes uninstallable on > PyPI (or if PyPI itself goes down) then nobody can install it, the people > installing things over and over every day or the people who just happened > to be installing it during that downtime. However intermittent failures and > general insatiability is going to be noticed by the projects who install things > over and over again quicker and easier and thus it becomes a lot easier > to use them as a general gauge for what the average ?uptime? is. I have had the same question as David, so I also appreciate your answer. > IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get > a handful of ?small? installers (e.g. not the big projects) in each downtime, > but a different set who are likely to shrug it off and just call treat it as the > norm even though it?s very disruptive to what they?re doing. However the > big project is highly likely to hit every single one of those downtimes and > be able to say ?wow PyPI is flaky as hell?. > > To expand further on that if we assume that we want ``pip install `` > to be reliable and not work sometimes and work at other times then we?re > aiming for as high as uptime as possible. PyPI gets enough traffic that > any single large project isn?t a noticeable drop in our bucket and due to the > way our caching works it actually helps us to be faster and more reliable > to have people constantly hitting packages because they?ll be in cache > and able to be served without hitting the Origin servers. > > Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the > month of April we served 71.4TB of data with 877.4 million requests of > which 80.5% never made it to the actual servers that run PyPI and were > served directly out of the geo distributed CDN that sits in front of PyPI. We > are vastly better positioned to maintain a reliable infrastructure than ask > that every large project that uses Python to do the same. > The reason that it?s our responsibility for providing it is because we chose > to provide it. There isn?t a moral imperative to run PyPI, but running PyPI > badly seems like a crummy thing to do. Agreed. >> For perspective, Gentoo requests that people only do an emerge sync at >> most once a day, and if they have multiple machines to update, that they >> only do one pull, and they update the rest of their infrastructure from >> their local copy. > > To be clear, there are other reasons to run a local mirror but I don?t think that > it?s reasonable to expect anyone who wants a reliable install using pip to > stand up their own infrastructure. Ok, you are not saying that caching is bad, but that having everyone reinvent caching, and possibly doing it badly, or at least not in thebest way, is bad. > Further to this point here I?m currently working on adding caching by default > for pip so that we minimize how often different people hit PyPI and we do it > automatically and in a way that doesn?t generally require people to think about > it and that also doesn?t require them to stand up their own infra. This seems like the right solution. It would sort of make each machine a micro-CDN node. -- Terry Jan Reedy From donald at stufft.io Fri May 9 22:38:27 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 May 2014 16:38:27 -0400 Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic] In-Reply-To: References: <20140506213533.GA32765@sleipnir.bytereef.org> <20140508121201.GA28465@sleipnir.bytereef.org> <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io> <536B8929.6040407@egenix.com> <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io> <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com> <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io> <536C8DD3.6090601@egenix.com> <536CDF0F.6050407@egenix.com> <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io> <20140509172828.ECBA2250D4E@webabinitio.net> <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io> Message-ID: On May 9, 2014, at 4:20 PM, Terry Reedy wrote: > On 5/9/2014 2:12 PM, Donald Stufft wrote: >> >> On May 9, 2014, at 1:28 PM, R. David Murray wrote: > >>> I don't understand this. Why it is our responsibility to provide a >>> free service for a large project to repeatedly download a set of files >>> they need? Why does it not make more sense for them to download them >>> once, and only update their local copies when they change? That's almost >>> completely orthogonal to making the service we do provide reliable. >> >> Well here?s the thing right. The large projects repeatedly downloading the >> same set of files is a canary. If any particular project goes uninstallable on >> PyPI (or if PyPI itself goes down) then nobody can install it, the people >> installing things over and over every day or the people who just happened >> to be installing it during that downtime. However intermittent failures and >> general insatiability is going to be noticed by the projects who install things >> over and over again quicker and easier and thus it becomes a lot easier >> to use them as a general gauge for what the average ?uptime? is. > > I have had the same question as David, so I also appreciate your answer. > >> IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get >> a handful of ?small? installers (e.g. not the big projects) in each downtime, >> but a different set who are likely to shrug it off and just call treat it as the >> norm even though it?s very disruptive to what they?re doing. However the >> big project is highly likely to hit every single one of those downtimes and >> be able to say ?wow PyPI is flaky as hell?. >> >> To expand further on that if we assume that we want ``pip install `` >> to be reliable and not work sometimes and work at other times then we?re >> aiming for as high as uptime as possible. PyPI gets enough traffic that >> any single large project isn?t a noticeable drop in our bucket and due to the >> way our caching works it actually helps us to be faster and more reliable >> to have people constantly hitting packages because they?ll be in cache >> and able to be served without hitting the Origin servers. >> >> Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the >> month of April we served 71.4TB of data with 877.4 million requests of >> which 80.5% never made it to the actual servers that run PyPI and were >> served directly out of the geo distributed CDN that sits in front of PyPI. We >> are vastly better positioned to maintain a reliable infrastructure than ask >> that every large project that uses Python to do the same. > >> The reason that it?s our responsibility for providing it is because we chose >> to provide it. There isn?t a moral imperative to run PyPI, but running PyPI >> badly seems like a crummy thing to do. > > Agreed. > >>> For perspective, Gentoo requests that people only do an emerge sync at >>> most once a day, and if they have multiple machines to update, that they >>> only do one pull, and they update the rest of their infrastructure from >>> their local copy. >> >> To be clear, there are other reasons to run a local mirror but I don?t think that >> it?s reasonable to expect anyone who wants a reliable install using pip to >> stand up their own infrastructure. > > Ok, you are not saying that caching is bad, but that having everyone reinvent caching, and possibly doing it badly, or at least not in thebest way, is bad. Yea, caching isn?t in general a bad thing, and actually PyPI uses it heavily. All access to /simple/ and /packages/ is cached for 24 hours by our CDN unless someone uploads or deletes a file, in which case we selectively purge those URLs from the CDN cache so that they (nearly) instantly get the updated results. Warehouse (aka PyPI 2.0) is designed to utilize our CDN cache even further and I?m hoping to get our cache rate even higher using it. > >> Further to this point here I?m currently working on adding caching by default >> for pip so that we minimize how often different people hit PyPI and we do it >> automatically and in a way that doesn?t generally require people to think about >> it and that also doesn?t require them to stand up their own infra. > > This seems like the right solution. It would sort of make each machine a micro-CDN node. Yes, it?s bog standard HTTP stuff just like a browser does it. The major difference is we?re limiting the maximum lifetime of a cache item in the client (pip) for the index pages but we are not doing that for the package files themselves. This is done to prevent a misconfigured server from causing pip to not see new versions for hours/days/weeks/years/whatever. Additionally this change also includes making pip smarter about HTTP requests in that if we have a stale item in the cache which as a Last-Modified or an ETag header we?ll do a conditional GET which will hopefully be returned with an HTTP 304 Not Modified so that we can simply refresh the stale item in the cache and use it again instead of needing to download an entire response body again. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From 4kir4.1i at gmail.com Fri May 9 21:53:06 2014 From: 4kir4.1i at gmail.com (akira) Date: Fri, 09 May 2014 23:53:06 +0400 Subject: [Python-Dev] should tests be thread-safe? Message-ID: <536D3222.5090109@gmail.com> Hi, May tests expect that unless they themselves start a thread then there are no threads to worry about? I see that some old tests are not thread-safe and I have not found it to be explicitly mentioned in the devguide. I've written a test for http://bugs.python.org/issue21332 that is known to be non-thread-safe. Is it acceptable for new tests? -- akira From benjamin at python.org Sat May 10 02:57:43 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 09 May 2014 17:57:43 -0700 Subject: [Python-Dev] pushing 2.7.7 back a week Message-ID: <1399683463.17589.115723569.5D169976@webmail.messagingengine.com> I'm pushing the release schedule for 2.7.7 back a week. That means the release candidate will be next weekend (May 17). This will hopefully save some trouble for the people who build our binaries, since 3.4.1 final is happening next weekend, too. Regards, Benjamin From ncoghlan at gmail.com Sat May 10 13:41:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 May 2014 21:41:40 +1000 Subject: [Python-Dev] should tests be thread-safe? In-Reply-To: <536D3222.5090109@gmail.com> References: <536D3222.5090109@gmail.com> Message-ID: On 10 May 2014 06:53, "akira" <4kir4.1i at gmail.com> wrote: > > Hi, > > May tests expect that unless they themselves start a thread then there are no threads to worry about? > > I see that some old tests are not thread-safe and I have not found it to be explicitly mentioned in the devguide. > > I've written a test for http://bugs.python.org/issue21332 that is known to be non-thread-safe. Is it acceptable for new tests? Thread safety is desirable, but not mandatory, since there is some process global state (e.g. the import system and the sys module in general) that the tests sometimes need to manipulate. Cheers, Nick. > > > -- > akira > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sat May 10 20:10:19 2014 From: brian at python.org (Brian Curtin) Date: Sat, 10 May 2014 13:10:19 -0500 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? In-Reply-To: References: <536BDCB6.6030300@v.loewis.de> Message-ID: On Fri, May 9, 2014 at 12:00 PM, Zachary Ware wrote: > On Thu, May 8, 2014 at 4:20 PM, Zachary Ware > wrote: >> I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple >> of weeks ago (see http://bugs.python.org/issue21303), but hadn't >> gotten anything done with Tix yet. It should just need python.mak >> updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as >> soon as I can. > > Tix for 2.7 is now > http://svn.python.org/projects/external/tix-8.4.3.5. You can build it > with this monster of a command, run from tix-8.4.3.5\win: > > nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 > TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 > INSTALL_DIR=..\..\tcltk all > > Use "install" instead of "all" after building to install it to > ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to > be set if you're building Release, but MACHINE always has to be set so > that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for > 64-bit). Awesome, thanks! So I now have a fully working setup, at least for 32-bit, and have backported the Path option from 3.3 (http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue with win32com getting a 64-bit installer built but didn't have time to look into it yet. If anyone wants to try a 2.7 installer with that Path option, here's a copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not signed, it'll warn you about that). From Steve.Dower at microsoft.com Sat May 10 20:35:17 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 10 May 2014 18:35:17 +0000 Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer? In-Reply-To: References: <536BDCB6.6030300@v.loewis.de> , Message-ID: <1399746936051.48735@microsoft.com> Brian Curtin wrote: > On Fri, May 9, 2014 at 12:00 PM, Zachary Ware > wrote: >> On Thu, May 8, 2014 at 4:20 PM, Zachary Ware >> wrote: >>> I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple >>> of weeks ago (see http://bugs.python.org/issue21303), but hadn't >>> gotten anything done with Tix yet. It should just need python.mak >>> updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as >>> soon as I can. >> >> Tix for 2.7 is now >> http://svn.python.org/projects/external/tix-8.4.3.5. You can build it >> with this monster of a command, run from tix-8.4.3.5\win: >> >> nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 >> TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 >> INSTALL_DIR=..\..\tcltk all >> >> Use "install" instead of "all" after building to install it to >> ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to >> be set if you're building Release, but MACHINE always has to be set so >> that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for >> 64-bit). > > Awesome, thanks! > > So I now have a fully working setup, at least for 32-bit, and have > backported the Path option from 3.3 > (http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue > with win32com getting a 64-bit installer built but didn't have time to > look into it yet. If the error is displayed immediately after the msm filename, it probably means you don't have that file. I had to pull the right ones out of our VS build directory, since I couldn't find a public version for SP1 (and apparently the QFE that updates the 32-bit MSM doesn't update the 64-bit one... guess I'll have to go look for that build now). > If anyone wants to try a 2.7 installer with that Path option, here's a > copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not > signed, it'll warn you about that). Looks good to me, and both of my builds worked fine. Cheers, Steve From gregory.szorc at gmail.com Sat May 10 22:05:54 2014 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Sat, 10 May 2014 13:05:54 -0700 Subject: [Python-Dev] python process creation overhead Message-ID: <536E86A2.600@gmail.com> I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant overhead and thus a concern for any CPython user. Full details are at [1]. tl;dr 10-18% of CPU time in test suite execution was spent doing the equivalent of `python -c 1` and 30-38% of CPU time was spent doing the equivalent of `hg version` (I suspect this is mostly dominated by module importing - and Mercurial has lazy module importing). My measurements show CPython is significantly slower than Perl and Ruby when it comes to process/interpreter creation/destruction. Furthermore, Python 3 appears to be >50% slower than Python 2. Any system spawning many Python processes will likely feel this overhead. I'm also a contributor to Firefox's build and testing infrastructure. Those systems collectively spawn thousands of Python processes. While I haven't yet measured, I fear that CPython process overhead is also contributing to significant overhead there. While more science and knowledge should be added before definite conclusions are reached, I'd like to ring the figurative bell about this problem. This apparent inefficiency has me concerned about the use of CPython in scenarios where process spawning is a common operation and decent latency is important. This includes CLI tools, build systems, and testing. I'm curious if others feel this is a problem and what steps can be taken to mitigate or correct it. [1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html Gregory Szorc gregory.szorc at gmail.com P.S. I'm not subscribed, so please CC me on responses. From alex.gaynor at gmail.com Sat May 10 23:18:22 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 10 May 2014 14:18:22 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: <3gR1CR0CDcz7LjY@mail.python.org> References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: Hi python-dev and Raymond, I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip. I agree that there's no need to repeat the same advice twice, but I'd much rather we kept the ".. warning:: " version, so users are more likely to actually read it. Also, there's a few errors with your commit message. First, we can reasonably assert that urandom provides an acceptable CSPRNG, mostly because it does on every platform I'm aware of. Second, urandom is still a psuedo-random number generator, however they are cryptographically secure, it's not "more random". Wikipedia does a good job laying out the necessary properties for a CSPRNG: https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator#Requirements Cheers, Alex On Sat, May 10, 2014 at 2:05 PM, raymond.hettinger < python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/b466dc34b86e > changeset: 90618:b466dc34b86e > parent: 90616:ce070040e1a6 > user: Raymond Hettinger > date: Sat May 10 14:05:28 2014 -0700 > summary: > Remove the redundant and poorly worded warning message. > > The paragraph above already says, clearly and correctly, that > "However, being completely deterministic, it is not suitable for > all purposes, and is completely unsuitable for cryptographic purposes." > > Also we should make any promises about SystemRandom or os.urandom() > being cryptographically secure (they may be, but be can't validate > that promise). Further, those are actual random number generators > not psuedo-random number generators. > > files: > Doc/library/random.rst | 6 ------ > 1 files changed, 0 insertions(+), 6 deletions(-) > > > diff --git a/Doc/library/random.rst b/Doc/library/random.rst > --- a/Doc/library/random.rst > +++ b/Doc/library/random.rst > @@ -43,12 +43,6 @@ > uses the system function :func:`os.urandom` to generate random numbers > from sources provided by the operating system. > > -.. warning:: > - > - The pseudo-random generators of this module should not be used for > - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if > - you require a cryptographically secure pseudo-random number generator. > - > > Bookkeeping functions: > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sat May 10 23:35:38 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 10 May 2014 14:35:38 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: On May 10, 2014, at 2:18 PM, Alex Gaynor wrote: > I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. People writing applications that need to be sure should probably have a howto guide. I don't think they are well served by filling the module docs with every possible way a person could make a security mistake). If you're writing a secure on-line poker game, you really need to know a great deal more about security than the warning message can supply. My understanding is that actual gaming systems use seeded CSPRNGs rather than urandom() because those systems need to be auditable. Raymond P.S. The docs for the random module had a successful 20 year history without the redundant big red warning box, so it is not really correct to say there has been a "considerable usability regression". -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat May 10 23:39:57 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 May 2014 23:39:57 +0200 Subject: [Python-Dev] should tests be thread-safe? In-Reply-To: <536D3222.5090109@gmail.com> References: <536D3222.5090109@gmail.com> Message-ID: If you need a well defined environement, run your test in a subprocess. Depending on the random function, your test may be run with more threads. On BSD, it changes for example which thread receives a signal. Importing the tkinter module creates a "hidden" C thread for the Tk loop. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat May 10 23:43:06 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 May 2014 23:43:06 +0200 Subject: [Python-Dev] python process creation overhead In-Reply-To: <536E86A2.600@gmail.com> References: <536E86A2.600@gmail.com> Message-ID: <20140510234306.342b2a94@fsol> Hello, On Sat, 10 May 2014 13:05:54 -0700 Gregory Szorc wrote: > I was investigating speeding up Mercurial's test suite (it runs ~13,000 > Python processes) and I believe I've identified CPython > process/interpreter creation and destruction as sources of significant > overhead and thus a concern for any CPython user. This was recently discussed in https://mail.python.org/pipermail/python-dev/2014-April/134028.html To give numbers: $ time python -c pass real 0m0.012s user 0m0.008s sys 0m0.004s $ time python -c "from mercurial import demandimport; demandimport.enable(); from mercurial.dispatch import request, dispatch; dispatch(request(['version']))" Mercurial Distributed SCM (version 3.0-rc+2-d924e387604f) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2014 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. real 0m0.056s user 0m0.044s sys 0m0.012s So, even for "hg version", most of the time is spent initializing Mercurial and its modules, not initializing Python itself. In this precise measurement, an infinitely fast Python startup would make "hg version" 20% faster (and other hg commands probably even less, since they have to load more Mercurial code and data). I'm wondering why Mercurial couldn't have a transparent daemon mode, where each "hg" invocation on the CLI would only load whatever required to communicate with the persistent daemon process. It already has a commandserver. > P.S. I'm not subscribed, so please CC me on responses. I would suggest following the list using the gmane NNTP gateway. Regards Antoine. From victor.stinner at gmail.com Sat May 10 23:46:23 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 10 May 2014 23:46:23 +0200 Subject: [Python-Dev] python process creation overhead In-Reply-To: <536E86A2.600@gmail.com> References: <536E86A2.600@gmail.com> Message-ID: Le 10 mai 2014 22:51, "Gregory Szorc" a ?crit : > Furthermore, Python 3 appears to be >50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat May 10 23:54:01 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 10 May 2014 23:54:01 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: <20140510235401.1b4774f2@fsol> On Sat, 10 May 2014 14:35:38 -0700 Raymond Hettinger wrote: > > In the past couple of years, we've grown an unfortunate tendency > to fill the docs with big warning boxes (the subprocess docs are > an example of implicitly communicating that the module is dangerous > and unusable). > > The preferred form of documentation is to be affirmatively worded, > telling how to use a tool correctly and what its known limitations are. > We save the warnings for cases of actual danger where otherwise > well informed users get tripped-up. > > Tim Peters used to be around to articulate the principle that we don't write > the docs with the assumption that the users are less bright than we are > or that they can't read. I agree with Alex. It's not about being bright or not, it's about being *willing* to eat walls of text. However pleasant it may be for some people to *write* documentation, for most readers (and especially non-native English readers, who read more slowly and more painfully than native ones), documentation is a piece of reference that they skim from, rather than read from start to end like a novel. So it makes sense to single out useful information. Knowing that the functions from the random module are not appropriate for cryptography is (IMO) more important than knowing that """Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2**19937-1. The underlying implementation in C is both fast and threadsafe""" Regards Antoine. From ncoghlan at gmail.com Sun May 11 00:10:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 May 2014 08:10:03 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: On 11 May 2014 07:37, "Raymond Hettinger" wrote: > > > On May 10, 2014, at 2:18 PM, Alex Gaynor wrote: > >> I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip > > > In the past couple of years, we've grown an unfortunate tendency > to fill the docs with big warning boxes (the subprocess docs are > an example of implicitly communicating that the module is dangerous > and unusable). > > The preferred form of documentation is to be affirmatively worded, > telling how to use a tool correctly and what its known limitations are. > We save the warnings for cases of actual danger where otherwise > well informed users get tripped-up. > > Tim Peters used to be around to articulate the principle that we don't write > the docs with the assumption that the users are less bright than we are > or that they can't read. That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation. This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked. As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning. A "Security Considerations" section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a "Here be dragons" warning early on noting that random *is* a potentially security sensitive module in a cryptographic context. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun May 11 00:16:08 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 18:16:08 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: <1C20E6FF-BF2A-4076-BFEC-9645E3290319@stufft.io> On May 10, 2014, at 6:10 PM, Nick Coghlan wrote: > > On 11 May 2014 07:37, "Raymond Hettinger" wrote: > > > > > > On May 10, 2014, at 2:18 PM, Alex Gaynor wrote: > > > >> I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip > > > > > > In the past couple of years, we've grown an unfortunate tendency > > to fill the docs with big warning boxes (the subprocess docs are > > an example of implicitly communicating that the module is dangerous > > and unusable). > > > > The preferred form of documentation is to be affirmatively worded, > > telling how to use a tool correctly and what its known limitations are. > > We save the warnings for cases of actual danger where otherwise > > well informed users get tripped-up. > > > > Tim Peters used to be around to articulate the principle that we don't write > > the docs with the assumption that the users are less bright than we are > > or that they can't read. > > That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation. > > This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked. > > As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning. > > A "Security Considerations" section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a "Here be dragons" warning early on noting that random *is* a potentially security sensitive module in a cryptographic context. > I completely agree with Alex, Antoine, and Nick here. I?m both an experienced Python programmer and someone who is generally aware of the security implications of various parts of software. However I appreciate when I look at documentation that explicitly calls out the ways I might screw it up, especially in a security sensitive context, but I appreciate it any context really. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From raymond.hettinger at gmail.com Sun May 11 00:21:21 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 10 May 2014 15:21:21 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: <20140510235401.1b4774f2@fsol> References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> Message-ID: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> On May 10, 2014, at 2:54 PM, Antoine Pitrou wrote: > It's not about being bright or not, it's about being > *willing* to eat walls of text. However pleasant it may be for some > people to *write* documentation, for most readers (and especially > non-native English readers, who read more slowly and more painfully > than native ones), documentation is a piece of reference that they skim > from, rather than read from start to end like a novel. Most users of the random module documentation are just normal people trying to create random numbers. People writing secure apps with cryptographically secure random numbers are not the primary audience. But we have this big red security warning that essentially says, if you read only one thing, read this. Before proceeding further with stamping distracting security warnings all over the module documentation, we should look to other languages to see what others have found necessary. This warning does not appear anywhere else I've looked (MS Excel docs, Java docs, Go lang docs, etc.) http://docs.oracle.com/javase/6/docs/api/java/util/Random.html http://golang.org/pkg/math/rand/ Those docs are clear, concise, not preachy, and not littered with distractions. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun May 11 00:24:40 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 10 May 2014 15:24:40 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: Give it up, Raymond. On Saturday, May 10, 2014, Raymond Hettinger wrote: > > On May 10, 2014, at 2:54 PM, Antoine Pitrou > > wrote: > > It's not about being bright or not, it's about being > *willing* to eat walls of text. However pleasant it may be for some > people to *write* documentation, for most readers (and especially > non-native English readers, who read more slowly and more painfully > than native ones), documentation is a piece of reference that they skim > from, rather than read from start to end like a novel. > > > Most users of the random module documentation are just > normal people trying to create random numbers. People > writing secure apps with cryptographically secure random > numbers are not the primary audience. But we have this > big red security warning that essentially says, if you read > only one thing, read this. > > Before proceeding further with stamping distracting security > warnings all over the module documentation, we should look > to other languages to see what others have found necessary. > This warning does not appear anywhere else I've looked > (MS Excel docs, Java docs, Go lang docs, etc.) > > http://docs.oracle.com/javase/6/docs/api/java/util/Random.html > http://golang.org/pkg/math/rand/ > > Those docs are clear, concise, not preachy, and not littered > with distractions. > > > Raymond > > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun May 11 00:33:10 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 18:33:10 -0400 Subject: [Python-Dev] python process creation overhead In-Reply-To: References: <536E86A2.600@gmail.com> Message-ID: <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io> On May 10, 2014, at 5:46 PM, Victor Stinner wrote: > Le 10 mai 2014 22:51, "Gregory Szorc" a ?crit : > > Furthermore, Python 3 appears to be >50% slower than Python 2. > > Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. > > There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. > For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ezio.melotti at gmail.com Sun May 11 00:35:28 2014 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sun, 11 May 2014 01:35:28 +0300 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> Message-ID: Hi, On Sun, May 11, 2014 at 12:35 AM, Raymond Hettinger wrote: > > On May 10, 2014, at 2:18 PM, Alex Gaynor wrote: > > I think this change is a considerable usability regression for the > documentation. Right now the warnings about CSPRNGs are hidden in the > introductory paragraph, which users are likely to skip > > > In the past couple of years, we've grown an unfortunate tendency > to fill the docs with big warning boxes If the problem is the scary red boxes, http://bugs.python.org/issue13515 still has a patch to make them less intimidating (and some interesting discussion about it). Best Regards, Ezio Melotti > (the subprocess docs are > an example of implicitly communicating that the module is dangerous > and unusable). > > The preferred form of documentation is to be affirmatively worded, > telling how to use a tool correctly and what its known limitations are. > We save the warnings for cases of actual danger where otherwise > well informed users get tripped-up. > > Tim Peters used to be around to articulate the principle that we don't write > the docs with the assumption that the users are less bright than we are > or that they can't read. > > People writing applications that need to be sure should probably have > a howto guide. I don't think they are well served by filling the module > docs with every possible way a person could make a security mistake). > > If you're writing a secure on-line poker game, you really need to know > a great deal more about security than the warning message can supply. > My understanding is that actual gaming systems use seeded CSPRNGs > rather than urandom() because those systems need to be auditable. > > > Raymond > > > P.S. The docs for the random module had a successful 20 year history > without the redundant big red warning box, so it is not really correct > to say there has been a "considerable usability regression". From cf.natali at gmail.com Sun May 11 00:43:26 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sat, 10 May 2014 23:43:26 +0100 Subject: [Python-Dev] should tests be thread-safe? In-Reply-To: References: <536D3222.5090109@gmail.com> Message-ID: > You might have forgotten to include Python-dev in the reply. Indeed, adding it back! > Thank you for the reply. I might have expressed the question poorely. I > meant: I have a script that I know is not thread-safe but it doesn't matter > because the test itself doesn't run any threads and the current tests are > never(?) run in multiple threads (-j uses processes). Should this *new* test > be fixed if e.g., there is a desire to be able to run (at least some) tests > in multiple threads concurrently in the future? The short answer is: no, you don't have to make you thread thread safe, as long as it can reliably run even in the presence of background threads (like the tkinter threads Victor mentions). From guido at python.org Sun May 11 00:50:14 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 10 May 2014 15:50:14 -0700 Subject: [Python-Dev] python process creation overhead In-Reply-To: <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io> References: <536E86A2.600@gmail.com> <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io> Message-ID: Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). On Sat, May 10, 2014 at 3:33 PM, Donald Stufft wrote: > > On May 10, 2014, at 5:46 PM, Victor Stinner > wrote: > > Le 10 mai 2014 22:51, "Gregory Szorc" a ?crit : > > Furthermore, Python 3 appears to be >50% slower than Python 2. > > Please mention the minor version. It looks like you compared 2.7 and 3.3. > Please test 3.4, we made interesting progress on the startup time. > > There is still something to do, especially on OS X. Depending on the OS, > different modules are loaded and some functions are implemented differently. > > > For what it's worth pip is the same way, about half of our test suite > involves > invoking (multiple) python processes. This has historically be really slow > (~30 minutes to run ~200 tests of that type). We've been able to get the > wall > clock run time down by parallelizing these but the sequential time is still > really slow. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun May 11 01:01:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 May 2014 09:01:33 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: On 11 May 2014 08:24, "Raymond Hettinger" wrote: > > Before proceeding further with stamping distracting security > warnings all over the module documentation, we should look > to other languages to see what others have found necessary. > This warning does not appear anywhere else I've looked > (MS Excel docs, Java docs, Go lang docs, etc.) > > http://docs.oracle.com/javase/6/docs/api/java/util/Random.html > http://golang.org/pkg/math/rand/ > > Those docs are clear, concise, not preachy, and not littered > with distractions. The fact that many (most?) programmers treat security considerations as a distraction is a core part of the problem we're trying to address. As you point out, most language development teams do very little to try to educate their users about security issues. The consequences of that are clearly visible in the world around us: when security is treated as an optional afterthought, you get widespread deployment of insecure software. At this point, we have two options: * continue with the same model as everyone else, and treat security as an optional extra users should feel free to ignore (or treat as an advanced topic only specialists need to worry about) * change our documentation practices to try to encourage the growth of a security aware development community around Python, trusting that our users will recognise that the security issues we're discussing are inherent in the way computers work, rather than being specific to Python. I'm obviously a strong advocate for the second path. Users aren't stupid, they'll figure out that almost all the security concerns we're warning about are inherent in the problem being solved, rather than being a Python-specific issue. Cheers, Nick. > > > Raymond > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun May 11 01:15:31 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 May 2014 01:15:31 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: Nick Coghlan, 11.05.2014 01:01: > As you point out, most language development teams do very little to try to > educate their users about security issues. The consequences of that are > clearly visible in the world around us: when security is treated as an > optional afterthought, you get widespread deployment of insecure software. > > At this point, we have two options: > > * continue with the same model as everyone else, and treat security as an > optional extra users should feel free to ignore (or treat as an advanced > topic only specialists need to worry about) > > * change our documentation practices to try to encourage the growth of a > security aware development community around Python, trusting that our users > will recognise that the security issues we're discussing are inherent in > the way computers work, rather than being specific to Python. > > I'm obviously a strong advocate for the second path. Users aren't stupid, > they'll figure out that almost all the security concerns we're warning > about are inherent in the problem being solved, rather than being a > Python-specific issue. Even if I know the problematic parts of a certain corner of software development or just of a specific tool, I prefer reading in the documentation that the authors of that tool are also aware of the (potential) problems. Makes me feel more comfortable with trusting the software. Total +1 on keeping these little bits around. Stefan From donald at stufft.io Sun May 11 01:19:50 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 May 2014 19:19:50 -0400 Subject: [Python-Dev] python process creation overhead In-Reply-To: References: <536E86A2.600@gmail.com> <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io> Message-ID: Yes right, sorry I didn?t mean to imply that all that time was spent in the Python start up time. I?ve personally never actually spent time to figure out which part of that was slow because getting visibility inside of a subprocess.Popen is a pain and I?m slowly trying to rewrite our tests to not require subprocesses at all. I was only trying to chime in with a me too here about slow subprocess tests since our test suite ends up starting a couple thousand subprocesses as well and those tests are definitely our slowest. On May 10, 2014, at 6:50 PM, Guido van Rossum wrote: > Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). > > > On Sat, May 10, 2014 at 3:33 PM, Donald Stufft wrote: > > On May 10, 2014, at 5:46 PM, Victor Stinner wrote: > >> Le 10 mai 2014 22:51, "Gregory Szorc" a ?crit : >> > Furthermore, Python 3 appears to be >50% slower than Python 2. >> >> Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. >> >> There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. >> > > For what it's worth pip is the same way, about half of our test suite involves > invoking (multiple) python processes. This has historically be really slow > (~30 minutes to run ~200 tests of that type). We've been able to get the wall > clock run time down by parallelizing these but the sequential time is still > really slow. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From raymond.hettinger at gmail.com Sun May 11 01:47:30 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 10 May 2014 16:47:30 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: On May 10, 2014, at 4:15 PM, Stefan Behnel wrote: > Total +1 on keeping these little bits around. Since all of you want a warning, I'll add one back but with improved wording. I'm not all at comfortable with the wording of the second sentence. I was the author of the SystemRandom() class and I only want to guarantee that it provides access to the operating system's source of random numbers. It is a bold claim to guarantee that it is cryptographically secure (many such claims in the past have turned-out to be false). We don't really know what it is going to do on a VM for example. Also, I don't want to call SystemRandom() a pseudo-random number generator. It purports to be an actual random number generator (or at least it purports to have used some real source of entropy at some stage). To me (the module maintainer), that is an important distinction. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 11 02:47:39 2014 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 10 May 2014 19:47:39 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: [Raymond Hettinger] > ... > I'm not all at comfortable with the wording of the second sentence. > I was the author of the SystemRandom() class and I only want > to guarantee that it provides access to the operating system's > source of random numbers. It is a bold claim to guarantee that > it is cryptographically secure (many such claims in the past have > turned-out to be false). We don't really know what it is going to > do on a VM for example. > > Also, I don't want to call SystemRandom() a pseudo-random number > generator. It purports to be an actual random number generator > (or at least it purports to have used some real source of entropy at > some stage). To me (the module maintainer), that is an important > distinction. It should be sufficient to say that SystemRandom() inherits all the properties of the operating system's os.urandom() implementation, yes? Since Python has nothing to do with that, it's most accurate and helpful to tell the user to refer to their OS urandom docs. On all platforms I'm aware of (two ;-)), urandom() is in fact a CSPRNG, not an "actual random number generator". That's why urandom() can get away with never blocking, potentially producing bits far faster than the system random() can accumulate fresh entropy. But all the hideous details don't belong in the Python docs - they belong in the OS's urandom docs. A phrasing I've found helpful is to tell users that "urandom() is as secure as the people who wrote your operating system knew how to make it". Linux users smile then, and Windows users groan ;-) From ethan at stoneleaf.us Sun May 11 07:27:36 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 10 May 2014 22:27:36 -0700 Subject: [Python-Dev] Values and objects In-Reply-To: References: <235C4BFA-9770-481A-9FCF-21C3F036769C@gmail.com> <87ppjpwafk.fsf@elektro.pacujo.net> <536ad8f2$0$29965$c3e8da3$5496439d@news.astraweb.com> <87zjiqbmy5.fsf@elektro.pacujo.net> <536d7a7d$0$29980$c3e8da3$5496439d@news.astraweb.com> <9cc0ebf9-dbed-4d3d-91fc-2abb9b0103d0@googlegroups.com> <536dc3f7$0$29980$c3e8da3$5496439d@news.astraweb.com> <536decca$0$29980$c3e8da3$5496439d@news.astraweb.com> <536E799D.6080602@stoneleaf.us> <536E9C3A.7060706@stoneleaf.us> Message-ID: <536F0A48.5040604@stoneleaf.us> [bringing back on-list] On 05/10/2014 07:30 PM, Devin Jeanpierre wrote: > On Sat, May 10, 2014 at 2:38 PM, Ethan Furman wrote: >> On 05/10/2014 02:03 PM, Devin Jeanpierre wrote: >>> >>> >>> spam is referring to a local variable that has not been bound. This is >>> not an implementation detail. >> >> The implementation detail is that in cpython there is a spot already >> reserved for what will be the 'spam' variable, as opposed to the module >> level where no such spots are reserved. > > I don't know what a "spot reserved" would even mean in a language > spec, so sure. But Chris did not bring up the idea of a "reserved > spot" in the post you were responding to, and I didn't either. Chris > claimed that Python had unbound variables, you disagreed, and I > pointed at the docs, which agree with Chris systematically. Chris was claiming that the exception UnboundLocalError, with complete text of: UnboundLocalError: local variable 'not_here' referenced before assignment is saying that the local variable does exist before it is assigned to; it is my contention that it does not. UnboundLocalError came into existence to aid in debugging a specific subclass of errors [1], and the text was chosen for that specific purpose. >>> Because module level variables work differently from local variables. >> >> Not by language definition. There are pythons where modifying function >> locals works, but the working or not-working is not a language guarantee. > > I did make the mistake of saying "locals", when I meant > "function-level locals". That was wrong. Um, what other kind of locals are there? > See https://docs.python.org/3.4/library/exceptions.html#UnboundLocalError > > "Raised when a reference is made to a local variable in a function or > method, but no value has been bound to that variable." Hmm, looks like a doc-patch is needed. ;) Seriously though, error messages are chosen to provide a simple and clear description that will help the user track down what went wrong, not for enshrining in exact detail the language semantics. Would you really rather have: Traceback (most recent call last): File "", line 1, in File "", line 2, in func UnboundLocalError: the name 'not_here' does not yet exist as you have not yet assigned anything to it so there is currently no variable by that name although at some point (in the future of this function, or perhaps in a branch that has been passed and did not execute) you will or did assign something to it so it will exist in the future of this function or may exist at this point in a future run of this function. or: Traceback (most recent call last): File "", line 1, in File "", line 2, in func UnboundLocalError: local variable 'not_here' referenced before assignment -- ~Ethan~ [1] The conditions being that a variable was referenced before is was created, but the text of NameError offered no help in figuring that out ("what do you mean it doesn't exist?!? It's right *there*!") From solipsis at pitrou.net Sun May 11 13:17:48 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 May 2014 13:17:48 +0200 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. References: <3gRBWD3q7bz7Ljd@mail.python.org> Message-ID: <20140511131748.611ce431@fsol> On Sun, 11 May 2014 06:04:56 +0200 (CEST) steve.dower wrote: > http://hg.python.org/devguide/rev/8d5d1f2c7378 > changeset: 698:8d5d1f2c7378 > user: Steve Dower > date: Sat May 10 21:01:39 2014 -0700 > summary: > Add myself to developer log and as a Windows expert. Welcome onboard, Steve ! Nice to have an additional Windows expert :-) Regards Antoine. From stephen at xemacs.org Sun May 11 15:34:20 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 11 May 2014 22:34:20 +0900 Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message. In-Reply-To: References: <3gR1CR0CDcz7LjY@mail.python.org> <20140510235401.1b4774f2@fsol> <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com> Message-ID: <874n0wh21v.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > As you point out, most language development teams do very little to > try to educate their users about security issues. That's partly because it isn't going to be terribly effective. Security is a difficult subject, not one that's going to be usefully treated in a couple of lines here, a couple more there. And it is generally an application issue, not one that is specific to individual features. If we're serious about this, I suggest following the RFC pattern: *every* module's documentation should have a "Security Considerations" section. Probably the content will be basically the same as the existing warning boxes, but with a consistent approach throughout the docs it could convey the importance of always thinking about security. > The consequences of that are clearly visible in the world around > us: when security is treated as an optional afterthought, But (FWIW) that's what warning boxes looks like to me. An afterthought. Not a systematic attempt to encourage security by teaching about secure programming. By your own words, we are nowhere close to a world where "a word, to the wise, is sufficient." From Steve.Dower at microsoft.com Sun May 11 15:57:47 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 11 May 2014 13:57:47 +0000 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: <20140511131748.611ce431@fsol> References: <3gRBWD3q7bz7Ljd@mail.python.org>,<20140511131748.611ce431@fsol> Message-ID: Thanks. For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.) Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Antoine Pitrou Sent: ?5/?11/?2014 4:18 To: python-dev at python.org Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. On Sun, 11 May 2014 06:04:56 +0200 (CEST) steve.dower wrote: > http://hg.python.org/devguide/rev/8d5d1f2c7378 > changeset: 698:8d5d1f2c7378 > user: Steve Dower > date: Sat May 10 21:01:39 2014 -0700 > summary: > Add myself to developer log and as a Windows expert. Welcome onboard, Steve ! Nice to have an additional Windows expert :-) Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From 4kir4.1i at gmail.com Sun May 11 14:02:31 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Sun, 11 May 2014 16:02:31 +0400 Subject: [Python-Dev] should tests be thread-safe? References: <536D3222.5090109@gmail.com> Message-ID: <877g5sle08.fsf@gmail.com> Victor Stinner writes: > If you need a well defined environement, run your test in a subprocess. > Depending on the random function, your test may be run with more threads. > On BSD, it changes for example which thread receives a signal. Importing > the tkinter module creates a "hidden" C thread for the Tk loop. Does it mean that non-thread-safe tests can't be run using a GUI test runner that is implemented using tkinter? -- akira From eliben at gmail.com Sun May 11 17:33:15 2014 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 11 May 2014 08:33:15 -0700 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> Message-ID: On Sun, May 11, 2014 at 6:57 AM, Steve Dower wrote: > Thanks. > > For those who missed the earlier discussions, Martin v. L?wis has handed > over responsibility for the Windows installers. It sounds like Brett Cannon > and I are both in a position to build 2.7 right now, and I hope to simplify > the setup required for 3.5 so that anyone can produce and test the > installers. (Martin is going to look after the 3.4.x builds.) > > Obviously I'm also here to help out with Windows in general. I haven't > quite managed to get 50% time from my employer (maybe 10%), but my > management at least is very supportive of my participation and keen to keep > Python running well. > Welcome Steve. It's awesome to have this, at least to some extent, officially endorsed by Microsoft. Python's Windows support is markedly better than other similar languages (Perl, Ruby), which I think contributes a lot to its success. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Sun May 11 20:04:37 2014 From: nad at acm.org (Ned Deily) Date: Sun, 11 May 2014 11:04:37 -0700 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> Message-ID: In article , Steve Dower wrote: > For those who missed the earlier discussions, Martin v. Lowis has handed over > responsibility for the Windows installers. Welcome to the the release team! --Ned -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Mon May 12 00:07:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 May 2014 08:07:22 +1000 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> Message-ID: On 11 May 2014 23:59, "Steve Dower" wrote: > > Thanks. Indeed, welcome! > For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, The other BC: Brian Curtin :) > Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well. Great to hear! Cheers, Nick. > > Cheers, > Steve > > Top-posted from my Windows Phone > ________________________________ > From: Antoine Pitrou > Sent: ?5/?11/?2014 4:18 > To: python-dev at python.org > Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. > > On Sun, 11 May 2014 06:04:56 +0200 (CEST) > steve.dower wrote: > > http://hg.python.org/devguide/rev/8d5d1f2c7378 > > changeset: 698:8d5d1f2c7378 > > user: Steve Dower > > date: Sat May 10 21:01:39 2014 -0700 > > summary: > > Add myself to developer log and as a Windows expert. > > Welcome onboard, Steve ! Nice to have an additional Windows expert :-) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon May 12 04:15:46 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 11 May 2014 19:15:46 -0700 Subject: [Python-Dev] Patch for robotparser.py Message-ID: <078F3CBC-98EC-44DC-A2E1-6E622BB258B4@gmail.com> If there is anyone here with an interest in web-spiders, it would be nice if someone else could take a look at http://bugs.python.org/issue21469 which addresses the risk of false positives with the robots.txt parser. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon May 12 05:30:52 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 11 May 2014 20:30:52 -0700 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org>, <20140511131748.611ce431@fsol> Message-ID: <238BC768-3DE1-4967-81E6-08DAB945EF20@gmail.com> On May 11, 2014, at 6:57 AM, Steve Dower wrote: > Thanks. > > For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.) > > Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well. Welcome to the party. :-) Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon May 12 20:43:22 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 12 May 2014 14:43:22 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_cpython_=28merge_3?= =?utf-8?q?=2E4_-=3E_default=29=3A_Merge_3=2E4-=3Edefault=3A_asynci?= =?utf-8?q?o=3A_Fix_upstream_issue_168=3A_StreamReader=2Eread=28-1?= =?utf-8?q?=29_from?= In-Reply-To: <3gS7nC2qJXz7LjP@mail.python.org> References: <3gS7nC2qJXz7LjP@mail.python.org> Message-ID: <20140512184322.A9D01250D4E@webabinitio.net> These changes appear to have caused several builbot failures, and there doesn't appear to be a bugs.python.org issue to report it to. One failure example: http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119 test_asyncio fails similarly for me on tip. On Mon, 12 May 2014 19:05:19 +0200, guido.van.rossum wrote: > http://hg.python.org/cpython/rev/2af5a52b9b87 > changeset: 90663:2af5a52b9b87 > parent: 90661:9493fdad2a75 > parent: 90662:909ea8cc86bb > user: Guido van Rossum > date: Mon May 12 10:05:04 2014 -0700 > summary: > Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from pipe may hang if data exceeds buffer limit. > > files: > Lib/asyncio/streams.py | 17 ++++-- > Lib/test/test_asyncio/test_streams.py | 36 +++++++++++++++ > 2 files changed, 47 insertions(+), 6 deletions(-) > > > diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py > --- a/Lib/asyncio/streams.py > +++ b/Lib/asyncio/streams.py > @@ -419,12 +419,17 @@ > return b'' > > if n < 0: > - while not self._eof: > - self._waiter = self._create_waiter('read') > - try: > - yield from self._waiter > - finally: > - self._waiter = None > + # This used to just loop creating a new waiter hoping to > + # collect everything in self._buffer, but that would > + # deadlock if the subprocess sends more than self.limit > + # bytes. So just call self.read(self._limit) until EOF. > + blocks = [] > + while True: > + block = yield from self.read(self._limit) > + if not block: > + break > + blocks.append(block) > + return b''.join(blocks) > else: > if not self._buffer and not self._eof: > self._waiter = self._create_waiter('read') > diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py > --- a/Lib/test/test_asyncio/test_streams.py > +++ b/Lib/test/test_asyncio/test_streams.py > @@ -1,7 +1,9 @@ > """Tests for streams.py.""" > > import gc > +import os > import socket > +import sys > import unittest > from unittest import mock > try: > @@ -583,6 +585,40 @@ > server.stop() > self.assertEqual(msg, b"hello world!\n") > > + @unittest.skipIf(sys.platform == 'win32', "Don't have pipes") > + def test_read_all_from_pipe_reader(self): > + # See Tulip issue 168. This test is derived from the example > + # subprocess_attach_read_pipe.py, but we configure the > + # StreamReader's limit so that twice it is less than the size > + # of the data writter. Also we must explicitly attach a child > + # watcher to the event loop. > + > + watcher = asyncio.get_child_watcher() > + watcher.attach_loop(self.loop) > + > + code = """\ > +import os, sys > +fd = int(sys.argv[1]) > +os.write(fd, b'data') > +os.close(fd) > +""" > + rfd, wfd = os.pipe() > + args = [sys.executable, '-c', code, str(wfd)] > + > + pipe = open(rfd, 'rb', 0) > + reader = asyncio.StreamReader(loop=self.loop, limit=1) > + protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop) > + transport, _ = self.loop.run_until_complete( > + self.loop.connect_read_pipe(lambda: protocol, pipe)) > + > + proc = self.loop.run_until_complete( > + asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop)) > + self.loop.run_until_complete(proc.wait()) > + > + os.close(wfd) > + data = self.loop.run_until_complete(reader.read(-1)) > + self.assertEqual(data, b'data') > + > > if __name__ == '__main__': > unittest.main() > > -- > Repository URL: http://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins From ericsnowcurrently at gmail.com Tue May 13 00:59:29 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 12 May 2014 16:59:29 -0600 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> Message-ID: On Sun, May 11, 2014 at 7:57 AM, Steve Dower wrote: > For those who missed the earlier discussions, Martin v. L?wis has handed > over responsibility for the Windows installers. It sounds like Brett Cannon > and I are both in a position to build 2.7 right now, and I hope to simplify > the setup required for 3.5 so that anyone can produce and test the > installers. (Martin is going to look after the 3.4.x builds.) You're a welcome addition to the group! > > Obviously I'm also here to help out with Windows in general. I haven't quite > managed to get 50% time from my employer (maybe 10%), but my management at > least is very supportive of my participation and keen to keep Python running > well. That's great news. Hopefully it's a growing trend. :) -eric From pcmanticore at gmail.com Tue May 13 00:16:36 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Tue, 13 May 2014 01:16:36 +0300 Subject: [Python-Dev] Questions regarding Windows buildbots Message-ID: Hello! I'm working on a patch for issue bugs.python.org/issue8579 (Add missing tests for FlushKey, LoadKey, and SaveKey in winreg). This issue requires the SeBackupPrivilege in order to use LoadKey and SaveKey. While acquiring the privilege isn't very complicated using ctypes, it fails with ERROR_NOT_ALL_ASSIGNED (1300) when the user has Administrative privileges, but it's not an Administrator, problem which can be eluded by running the script with elevated privileges. This leads me to a couple of questions: - should a Windows test be skipped if it can't acquire a certain privilege? - If we can acquire the privilege by elevating our process, does the Windows buildbots have UAC enabled and if so, how's the notification setting configured? For instance, elevating a process will trigger a new UAC window with the message "Do you want to allow the following program from an unknown publisher to make changes to this computer?" on the recommended configuration, but this doesn't happen when the configuration is set to "Never notify". Thank you. From brian at python.org Tue May 13 01:21:06 2014 From: brian at python.org (Brian Curtin) Date: Mon, 12 May 2014 18:21:06 -0500 Subject: [Python-Dev] Questions regarding Windows buildbots In-Reply-To: References: Message-ID: On Mon, May 12, 2014 at 5:16 PM, Claudiu Popa wrote: > Hello! > > I'm working on a patch for issue bugs.python.org/issue8579 (Add > missing tests for FlushKey, LoadKey, and SaveKey in winreg). This > issue requires the SeBackupPrivilege in order to use LoadKey and > SaveKey. While acquiring the privilege > isn't very complicated using ctypes, it fails with > ERROR_NOT_ALL_ASSIGNED (1300) when the > user has Administrative privileges, but it's not an Administrator, > problem which can be eluded by > running the script with elevated privileges. This leads me to a couple > of questions: > > - should a Windows test be skipped if it can't acquire a certain privilege? Yes. Check out any of the os.symlink tests - they're currently skipped when the symlink privilege isn't held. > - If we can acquire the privilege by elevating our process, does the > Windows buildbots have UAC > enabled and if so, how's the notification setting configured? For > instance, elevating a process will > trigger a new UAC window with the message "Do you want to allow the > following program from an unknown publisher to make changes to this > computer?" on the recommended configuration, but this doesn't happen > when the configuration is set to "Never notify". That probably depends on how each machine is setup. If they happen to get blocked on any individual slave, we'll just have to ask the owner to change that setting. Currently there are no Windows build slaves running as administrator. I used to have one but the machine died and I never replaced it. I also said a few months ago that I would get one setup again, but that hasn't happened yet. I can get a new machine up and running but probably not until next week as I'm at a conference. From gregory.szorc at gmail.com Tue May 13 01:22:52 2014 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Mon, 12 May 2014 16:22:52 -0700 Subject: [Python-Dev] python process creation overhead In-Reply-To: References: <536E86A2.600@gmail.com> Message-ID: <537157CC.1070603@gmail.com> On 5/10/2014 2:46 PM, Victor Stinner wrote: > Le 10 mai 2014 22:51, "Gregory Szorc" > a ?crit : >> Furthermore, Python 3 appears to be >50% slower than Python 2. > > Please mention the minor version. It looks like you compared 2.7 > and 3.3. Please test 3.4, we made interesting progress on the > startup time. > > There is still something to do, especially on OS X. Depending on > the OS, different modules are loaded and some functions are > implemented differently. 3.4.0 does appear to be faster than 3.3.5 on Linux - `python -c ''` is taking ~50ms (as opposed to ~60ms) on my i7-2600K. Great to see! But 3.4.0 is still slower than 2.7.6. And all versions of CPython are over 3x slower than Perl 5.18.2. This difference amounts to minutes of CPU time when thousands of processes are involved. That seems excessive to me. Why can't Python start as quickly as Perl or Ruby? From ericsnowcurrently at gmail.com Tue May 13 02:13:09 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 12 May 2014 18:13:09 -0600 Subject: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from In-Reply-To: <20140512184322.A9D01250D4E@webabinitio.net> References: <3gS7nC2qJXz7LjP@mail.python.org> <20140512184322.A9D01250D4E@webabinitio.net> Message-ID: On Mon, May 12, 2014 at 12:43 PM, R. David Murray wrote: > These changes appear to have caused several builbot failures, and > there doesn't appear to be a bugs.python.org issue to report it to. > One failure example: > > http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119 > > test_asyncio fails similarly for me on tip. Same for me on 3.4. -eric From dw+python-dev at hmmz.org Tue May 13 02:19:18 2014 From: dw+python-dev at hmmz.org (dw+python-dev at hmmz.org) Date: Tue, 13 May 2014 00:19:18 +0000 Subject: [Python-Dev] python process creation overhead In-Reply-To: <537157CC.1070603@gmail.com> References: <536E86A2.600@gmail.com> <537157CC.1070603@gmail.com> Message-ID: <20140513001918.GA734@k2> On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: > Why can't Python start as quickly as Perl or Ruby? On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from 81ms startup to 20ms by specifying -S, which disables site.py. Oblitering the .pth files immediately knocks 10ms off regular startup. I guess the question isn't why Python is slower than perl, but what aspects of site.py could be cached, reimplemented, or stripped out entirely. I'd personally love to see .pth support removed. David From guido at python.org Tue May 13 02:26:35 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 12 May 2014 17:26:35 -0700 Subject: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from In-Reply-To: References: <3gS7nC2qJXz7LjP@mail.python.org> <20140512184322.A9D01250D4E@webabinitio.net> Message-ID: Sorry about that. I will look into it soon, but won't have time right away. (Also I missed David's earlier message.) On May 12, 2014 5:14 PM, "Eric Snow" wrote: > On Mon, May 12, 2014 at 12:43 PM, R. David Murray > wrote: > > These changes appear to have caused several builbot failures, and > > there doesn't appear to be a bugs.python.org issue to report it to. > > One failure example: > > > > > http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119 > > > > test_asyncio fails similarly for me on tip. > > Same for me on 3.4. > > -eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 13 05:43:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 May 2014 13:43:20 +1000 Subject: [Python-Dev] python process creation overhead In-Reply-To: <20140513001918.GA734@k2> References: <536E86A2.600@gmail.com> <537157CC.1070603@gmail.com> <20140513001918.GA734@k2> Message-ID: On 13 May 2014 10:19, wrote: > On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: > >> Why can't Python start as quickly as Perl or Ruby? > > On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from > 81ms startup to 20ms by specifying -S, which disables site.py. > > Oblitering the .pth files immediately knocks 10ms off regular startup. I > guess the question isn't why Python is slower than perl, but what > aspects of site.py could be cached, reimplemented, or stripped out > entirely. I'd personally love to see .pth support removed. The startup code is currently underspecified and underdocumented, and quite fragile as a result. It represents 20+ years of organic growth without any systematic refactoring to simplify and streamline things. That's what PEP 432 aims to address, and is something I now expect to have time to get back to for Python 3.5. And yes, one thing those changes should enable is the creation of system Python runtimes on Linux that initialise faster than the current implementation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From me at the-compiler.org Tue May 13 16:49:08 2014 From: me at the-compiler.org (Florian Bruhin) Date: Tue, 13 May 2014 16:49:08 +0200 Subject: [Python-Dev] ConfigParser mangles keys with special chars In-Reply-To: <20140425142206.GD6735@lupin> References: <20140425142206.GD6735@lupin> Message-ID: <20140513144907.GA6735@lupin> * Florian Bruhin [2014-04-25 16:22:06 +0200]: > I noticed configparser does behave in a surprising way when a key > has a special meaning in ini-format. I've now submitted an issue here: http://bugs.python.org/issue21498 Florian -- () ascii ribbon campaign - stop html mail www.asciiribbon.org /\ www.the-compiler.org | I love long mails http://email.is-not-s.ms/ Blessed are the forgetful: for they get the better even of their blunders. -- Nietzsche -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From gregory.szorc at gmail.com Tue May 13 22:58:05 2014 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Tue, 13 May 2014 13:58:05 -0700 Subject: [Python-Dev] python process creation overhead In-Reply-To: References: <536E86A2.600@gmail.com> <537157CC.1070603@gmail.com> <20140513001918.GA734@k2> Message-ID: <5372875D.8080601@gmail.com> On 5/12/2014 8:43 PM, Nick Coghlan wrote: > On 13 May 2014 10:19, wrote: >> On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote: >> >>> Why can't Python start as quickly as Perl or Ruby? >> >> On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops >> from 81ms startup to 20ms by specifying -S, which disables >> site.py. >> >> Oblitering the .pth files immediately knocks 10ms off regular >> startup. I guess the question isn't why Python is slower than >> perl, but what aspects of site.py could be cached, reimplemented, >> or stripped out entirely. I'd personally love to see .pth >> support removed. > > The startup code is currently underspecified and underdocumented, > and quite fragile as a result. It represents 20+ years of organic > growth without any systematic refactoring to simplify and > streamline things. > > That's what PEP 432 aims to address, and is something I now expect > to have time to get back to for Python 3.5. And yes, one thing > those changes should enable is the creation of system Python > runtimes on Linux that initialise faster than the current > implementation. This is terrific news and something I greatly anticipate taking advantage of! But the great many of us still on 2.7 likely won't see a benefit, correct? How insane would it be for people to do things like pass -S in the shebang and manually implement the parts of site.py that are actually needed? From stephen at xemacs.org Wed May 14 04:33:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 14 May 2014 11:33:42 +0900 Subject: [Python-Dev] python process creation overhead In-Reply-To: <5372875D.8080601@gmail.com> References: <536E86A2.600@gmail.com> <537157CC.1070603@gmail.com> <20140513001918.GA734@k2> <5372875D.8080601@gmail.com> Message-ID: <87sioddr7d.fsf@uwakimon.sk.tsukuba.ac.jp> Gregory Szorc writes: > But the great many of us still on 2.7 likely won't see a benefit, > correct? How insane would it be for people to do things like pass -S > in the shebang and manually implement the parts of site.py that are > actually needed? Well, since it probably won't work .... That is to say, site.py typically provides different facilities to different programs -- that's why some parts of it show up as unneeded. So you need to carefully analyze *each* *subprocess* that you propose to invoke with -S and determine which parts of site.py it needs. In most cases I suspect you would better look at alternatives that avoid invoking a subprocess per task, but instead maintains a pool of worker processes (or threads). You might even be able to save network traffic or IPC by caching replies to common requests in the worker processes, which may save more per task than process invocation. Even where -S makes sense, I would suggest invoking "python -S" explicitly from the parent process rather than munging the shebang in the children. (You do need to be careful to audit changes in the child programs to be sure they aren't changed in ways that change site.py requirements. Putting -S in the shebang may help catch such problems early.) From bcannon at gmail.com Wed May 14 16:20:26 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 14 May 2014 14:20:26 +0000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? Message-ID: Over the past week or so there have been 2 patches to add support for various UNIX OSs. Now I thought we had stopped trying to add new esoteric OSs (e.g. I had never heard of MirOS until the patch for it came in), but I can't find a PEP that spells out what it takes to get a platform supported ( http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, not keeping them or adding them unless you are re-adding one which apparently just takes a volunteer). Do we want an official policy written down in a PEP (yes, I can write it)? Should I keep closing these patches and saying that we are not adding support for new operating systems and be hand-wavy about it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed May 14 16:28:20 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 14 May 2014 10:28:20 -0400 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: Message-ID: <20140514102820.0e13a141@anarchist.wooz.org> On May 14, 2014, at 02:20 PM, Brett Cannon wrote: >Do we want an official policy written down in a PEP (yes, I can write it)? >Should I keep closing these patches and saying that we are not adding >support for new operating systems and be hand-wavy about it? Yes, I think a PEP describing both policy and implementation (i.e. which platforms we officially support) is worthwhile. While I think you could write a new PEP, I also think it might just make sense to co-opt PEP 11 and broaden its scope, since as you say, removing support is only half the story. Cheers, -Barry From jsbueno at python.org.br Wed May 14 16:31:15 2014 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 14 May 2014 11:31:15 -0300 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: Message-ID: +1 for an official policy that comes with a "permanent maintainer for this platform required" as part of the list of requisites. js -><- On 14 May 2014 11:20, Brett Cannon wrote: > Over the past week or so there have been 2 patches to add support for > various UNIX OSs. Now I thought we had stopped trying to add new esoteric > OSs (e.g. I had never heard of MirOS until the patch for it came in), but I > can't find a PEP that spells out what it takes to get a platform supported > (http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, > not keeping them or adding them unless you are re-adding one which > apparently just takes a volunteer). > > Do we want an official policy written down in a PEP (yes, I can write it)? > Should I keep closing these patches and saying that we are not adding > support for new operating systems and be hand-wavy about it? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br > From solipsis at pitrou.net Wed May 14 16:42:47 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 May 2014 16:42:47 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? References: Message-ID: <20140514164247.18b252e5@fsol> On Wed, 14 May 2014 14:20:26 +0000 Brett Cannon wrote: > Over the past week or so there have been 2 patches to add support for > various UNIX OSs. Now I thought we had stopped trying to add new esoteric > OSs (e.g. I had never heard of MirOS until the patch for it came in), but I > can't find a PEP that spells out what it takes to get a platform supported ( > http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, > not keeping them or adding them unless you are re-adding one which > apparently just takes a volunteer). OTOH you can fix a platform bug without officially supporting it. If someone files an OpenBSD-specific patch, it may make sense to commit it even without officially supporting OpenBSD. In practice it all depends on how intrusive / reasonable the patch is, and whether it is working around a platform-specific bug rather than a standards-compliant limitation. (we could call those "stochastically supported platforms" :-)) Regards Antoine. From bcannon at gmail.com Wed May 14 16:45:20 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 14 May 2014 14:45:20 +0000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? References: <20140514164247.18b252e5@fsol> Message-ID: On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou wrote: > On Wed, 14 May 2014 14:20:26 +0000 > Brett Cannon wrote: > > Over the past week or so there have been 2 patches to add support for > > various UNIX OSs. Now I thought we had stopped trying to add new esoteric > > OSs (e.g. I had never heard of MirOS until the patch for it came in), > but I > > can't find a PEP that spells out what it takes to get a platform > supported ( > > http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, > > not keeping them or adding them unless you are re-adding one which > > apparently just takes a volunteer). > > OTOH you can fix a platform bug without officially supporting it. If > someone files an OpenBSD-specific patch, it may make sense to commit it > even without officially supporting OpenBSD. In practice it all depends > on how intrusive / reasonable the patch is, and whether it is working > around a platform-specific bug rather than a standards-compliant > limitation. > > (we could call those "stochastically supported platforms" :-)) > Very true, but these patches are all for e.g. configure to recognize a specific platform by listing them in some constant. Changing code to be more general I have no issue with since that's just good practice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed May 14 16:56:36 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 14 May 2014 10:56:36 -0400 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: Message-ID: <20140514145650.C0E17250D4F@webabinitio.net> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" wrote: > +1 for an official policy that comes with a "permanent maintainer for > this platform required" as part of the list > of requisites. > > js > -><- > > On 14 May 2014 11:20, Brett Cannon wrote: > > Over the past week or so there have been 2 patches to add support for > > various UNIX OSs. Now I thought we had stopped trying to add new esoteric > > OSs (e.g. I had never heard of MirOS until the patch for it came in), but I > > can't find a PEP that spells out what it takes to get a platform supported > > (http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, > > not keeping them or adding them unless you are re-adding one which > > apparently just takes a volunteer). > > > > Do we want an official policy written down in a PEP (yes, I can write it)? > > Should I keep closing these patches and saying that we are not adding > > support for new operating systems and be hand-wavy about it? In addition to a maintainer (who I think doesn't have to be a committer, though that would be ideal), I think a maintained buildbot should be a requirement for formal support. --David From bcannon at gmail.com Wed May 14 17:08:25 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 14 May 2014 15:08:25 +0000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? References: <20140514145650.C0E17250D4F@webabinitio.net> Message-ID: On Wed May 14 2014 at 11:02:50 AM, R. David Murray wrote: > On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" < > jsbueno at python.org.br> wrote: > > +1 for an official policy that comes with a "permanent maintainer for > > this platform required" as part of the list > > of requisites. > > > > js > > -><- > > > > On 14 May 2014 11:20, Brett Cannon wrote: > > > Over the past week or so there have been 2 patches to add support for > > > various UNIX OSs. Now I thought we had stopped trying to add new > esoteric > > > OSs (e.g. I had never heard of MirOS until the patch for it came in), > but I > > > can't find a PEP that spells out what it takes to get a platform > supported > > > (http://legacy.python.org/dev/peps/pep-0011/ is about removing > platforms, > > > not keeping them or adding them unless you are re-adding one which > > > apparently just takes a volunteer). > > > > > > Do we want an official policy written down in a PEP (yes, I can write > it)? > > > Should I keep closing these patches and saying that we are not adding > > > support for new operating systems and be hand-wavy about it? > > In addition to a maintainer (who I think doesn't have to be a committer, > though that would be ideal), I think a maintained buildbot should be a > requirement for formal support. > I would think someone how is/would be a core dev and a *stable* buildbot are requirements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Wed May 14 17:30:49 2014 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 14 May 2014 17:30:49 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140514145650.C0E17250D4F@webabinitio.net> Message-ID: <53738C29.7050008@ubuntu.com> Am 14.05.2014 17:08, schrieb Brett Cannon: > On Wed May 14 2014 at 11:02:50 AM, R. David Murray > wrote: > >> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" < >> jsbueno at python.org.br> wrote: >>> +1 for an official policy that comes with a "permanent maintainer for >>> this platform required" as part of the list >>> of requisites. >>> >>> js >>> -><- >>> >>> On 14 May 2014 11:20, Brett Cannon wrote: >>>> Over the past week or so there have been 2 patches to add support for >>>> various UNIX OSs. Now I thought we had stopped trying to add new >> esoteric >>>> OSs (e.g. I had never heard of MirOS until the patch for it came in), >> but I >>>> can't find a PEP that spells out what it takes to get a platform >> supported >>>> (http://legacy.python.org/dev/peps/pep-0011/ is about removing >> platforms, >>>> not keeping them or adding them unless you are re-adding one which >>>> apparently just takes a volunteer). >>>> >>>> Do we want an official policy written down in a PEP (yes, I can write >> it)? >>>> Should I keep closing these patches and saying that we are not adding >>>> support for new operating systems and be hand-wavy about it? >> >> In addition to a maintainer (who I think doesn't have to be a committer, >> though that would be ideal), I think a maintained buildbot should be a >> requirement for formal support. >> > > I would think someone how is/would be a core dev and a *stable* buildbot > are requirements. so, are aarch64-linux-gnu and powerpc64le-linux-gnu supported? I assume there are no buildbots and there won't be any for a long time. Otoh various distros do ship python on these architectures. Or are these just new architectures for an existing platform? If yes, then we should ask about architecture support too. The most prominent linux example are some RTLD constants which differ across some architectures. Matthias From bcannon at gmail.com Wed May 14 17:52:36 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 14 May 2014 15:52:36 +0000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? References: <20140514145650.C0E17250D4F@webabinitio.net> <53738C29.7050008@ubuntu.com> Message-ID: On Wed May 14 2014 at 11:33:27 AM, Matthias Klose wrote: > Am 14.05.2014 17:08, schrieb Brett Cannon: > > On Wed May 14 2014 at 11:02:50 AM, R. David Murray < > rdmurray at bitdance.com> > > wrote: > > > >> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" < > >> jsbueno at python.org.br> wrote: > >>> +1 for an official policy that comes with a "permanent maintainer for > >>> this platform required" as part of the list > >>> of requisites. > >>> > >>> js > >>> -><- > >>> > >>> On 14 May 2014 11:20, Brett Cannon wrote: > >>>> Over the past week or so there have been 2 patches to add support for > >>>> various UNIX OSs. Now I thought we had stopped trying to add new > >> esoteric > >>>> OSs (e.g. I had never heard of MirOS until the patch for it came in), > >> but I > >>>> can't find a PEP that spells out what it takes to get a platform > >> supported > >>>> (http://legacy.python.org/dev/peps/pep-0011/ is about removing > >> platforms, > >>>> not keeping them or adding them unless you are re-adding one which > >>>> apparently just takes a volunteer). > >>>> > >>>> Do we want an official policy written down in a PEP (yes, I can write > >> it)? > >>>> Should I keep closing these patches and saying that we are not adding > >>>> support for new operating systems and be hand-wavy about it? > >> > >> In addition to a maintainer (who I think doesn't have to be a committer, > >> though that would be ideal), I think a maintained buildbot should be a > >> requirement for formal support. > >> > > > > I would think someone how is/would be a core dev and a *stable* buildbot > > are requirements. > > so, are aarch64-linux-gnu and powerpc64le-linux-gnu supported? I assume > there > are no buildbots and there won't be any for a long time. Otoh various > distros do > ship python on these architectures. Or are these just new architectures > for an > existing platform? If yes, then we should ask about architecture support > too. > The most prominent linux example are some RTLD constants which differ > across > some architectures. > I consider CPU and compiler separate things. As long as we have a buildbot covering the CPU or compiler somehow I say they are covered (and someone is willing to help make sure they continue to work). I'm not going to say that we need a BSD ARM buildbot and a Linux ARM machine; having *a* machine with ARM should be enough to shake out most arch-specific issues IMO. Same goes with compilers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Wed May 14 18:04:36 2014 From: skip at pobox.com (Skip Montanaro) Date: Wed, 14 May 2014 11:04:36 -0500 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: Message-ID: I wonder if one or more people who maintain unofficial forks on minority platforms (OS/2, VMS, etc) could create an informational PEP about the process (benefits and pitfalls) of that kind of effort? Skip From db3l.net at gmail.com Wed May 14 23:36:30 2014 From: db3l.net at gmail.com (David Bolen) Date: Wed, 14 May 2014 17:36:30 -0400 Subject: [Python-Dev] Questions regarding Windows buildbots References: Message-ID: Brian Curtin writes: > On Mon, May 12, 2014 at 5:16 PM, Claudiu Popa wrote: (...) >> - If we can acquire the privilege by elevating our process, does the >> Windows buildbots have UAC >> enabled and if so, how's the notification setting configured? For >> instance, elevating a process will >> trigger a new UAC window with the message "Do you want to allow the >> following program from an unknown publisher to make changes to this >> computer?" on the recommended configuration, but this doesn't happen >> when the configuration is set to "Never notify". > > That probably depends on how each machine is setup. If they happen to > get blocked on any individual slave, we'll just have to ask the owner > to change that setting. For reference, my Windows 7 slave (bolen-windows7) is currently running with stock UAC settings, so I believe would prompt in such a scenario. The slave does run all builds with Windows error dialogs disabled, but I doubt that covers UAC. It's certainly no problem to disable the UAC prompting if it would help or if that becomes a more useful setting for the buildbot environment. It probably fits better with my existing model of disabling other sorts of error popups anyway, but just isn't something we've run up against yet in the build process. > Currently there are no Windows build slaves running as administrator. > I used to have one but the machine died and I never replaced it. I > also said a few months ago that I would get one setup again, but that > hasn't happened yet. I can get a new machine up and running but > probably not until next week as I'm at a conference. Yes - my slaves (XP and Win7) are running under administrative accounts, but not the literal Administrator account. Though I'm guessing this topic may be moot for the XP slave as it doesn't have UAC. Claudiu, if you've got any specific test code you'd like to have executed on either of my slaves to see how it behaves, I'd be happy to help. Just contact me directly. -- David From ncoghlan at gmail.com Thu May 15 04:35:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 May 2014 12:35:16 +1000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140514145650.C0E17250D4F@webabinitio.net> <53738C29.7050008@ubuntu.com> Message-ID: On 15 May 2014 01:52, Brett Cannon wrote: > > I consider CPU and compiler separate things. As long as we have a buildbot > covering the CPU or compiler somehow I say they are covered (and someone is > willing to help make sure they continue to work). I'm not going to say that > we need a BSD ARM buildbot and a Linux ARM machine; having a machine with > ARM should be enough to shake out most arch-specific issues IMO. Same goes > with compilers. We also handle some of that compatibility testing over in distro land. The feedback loop is a bit longer, but arch specific bugs are pretty rare anyway. For example: yes, CPython runs just fine on IBM s390x main frames, no, I'm not going to try to arrange for an s390x BuildBot upstream because that would be painful, and abstracting away CPU architecture issues is a key part of what the OS layer is for in the first place :) Anyway, +1 from me for expanding PEP 11 to also cover: - some rules of thumb for what kind of OS specific patches are acceptable to improve handling of unknown/unsupported OSes (e.g. switch from explicit OS detection to equivalent feature detection, yes, explicitly listing an unsupported OS in a conditional check, no) - some guidelines for what's needed for a new OS to be added as officially supported (with age and popularity likely being worth taking into account) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cs at zip.com.au Thu May 15 06:01:05 2014 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 15 May 2014 14:01:05 +1000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: Message-ID: <20140515040105.GA82654@cskk.homeip.net> On 14May2014 14:45, Brett Cannon wrote: >On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou >wrote: >> On Wed, 14 May 2014 14:20:26 +0000 >> Brett Cannon wrote: >> > Over the past week or so there have been 2 patches to add support for >> > various UNIX OSs. Now I thought we had stopped trying to add new esoteric >> > OSs (e.g. I had never heard of MirOS until the patch for it came in), >> but I >> > can't find a PEP that spells out what it takes to get a platform >> supported ( >> > http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms, >> > not keeping them or adding them unless you are re-adding one which >> > apparently just takes a volunteer). >> >> OTOH you can fix a platform bug without officially supporting it. If >> someone files an OpenBSD-specific patch, it may make sense to commit it >> even without officially supporting OpenBSD. In practice it all depends >> on how intrusive / reasonable the patch is, and whether it is working >> around a platform-specific bug rather than a standards-compliant >> limitation. >> >> (we could call those "stochastically supported platforms" :-)) > >Very true, but these patches are all for e.g. configure to recognize a >specific platform by listing them in some constant. Changing code to be >more general I have no issue with since that's just good practice. Recognition of a special platform isn't "full support", just addition of recognition making possible partial support for special cases. Unless that makes for some horrendous recognition decision tree somewhere I would have thought that's a pretty low bar to accept, and worth doing. Leaving aside any bug actually fixed, it makes it much easier for someone else to fix a platform specific bug by making a test constant available for turning on whatever special mode/code is wanted. More context on the example patch that triggered this query? Just 2c, Cameron Simpson From guido at python.org Thu May 15 06:05:49 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 14 May 2014 21:05:49 -0700 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: <20140515040105.GA82654@cskk.homeip.net> References: <20140515040105.GA82654@cskk.homeip.net> Message-ID: Main problem with rare platform support is not breaking it accidentally, since without buildbots we won't know when it's broken. This is why we don't make any promises. On May 14, 2014 9:02 PM, "Cameron Simpson" wrote: > On 14May2014 14:45, Brett Cannon wrote: > >> On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou >> wrote: >> >>> On Wed, 14 May 2014 14:20:26 +0000 >>> Brett Cannon wrote: >>> > Over the past week or so there have been 2 patches to add support for >>> > various UNIX OSs. Now I thought we had stopped trying to add new >>> esoteric >>> > OSs (e.g. I had never heard of MirOS until the patch for it came in), >>> but I >>> > can't find a PEP that spells out what it takes to get a platform >>> supported ( >>> > http://legacy.python.org/dev/peps/pep-0011/ is about removing >>> platforms, >>> > not keeping them or adding them unless you are re-adding one which >>> > apparently just takes a volunteer). >>> >>> OTOH you can fix a platform bug without officially supporting it. If >>> someone files an OpenBSD-specific patch, it may make sense to commit it >>> even without officially supporting OpenBSD. In practice it all depends >>> on how intrusive / reasonable the patch is, and whether it is working >>> around a platform-specific bug rather than a standards-compliant >>> limitation. >>> >>> (we could call those "stochastically supported platforms" :-)) >>> >> >> Very true, but these patches are all for e.g. configure to recognize a >> specific platform by listing them in some constant. Changing code to be >> more general I have no issue with since that's just good practice. >> > > Recognition of a special platform isn't "full support", just addition of > recognition making possible partial support for special cases. Unless that > makes for some horrendous recognition decision tree somewhere I would have > thought that's a pretty low bar to accept, and worth doing. > > Leaving aside any bug actually fixed, it makes it much easier for someone > else to fix a platform specific bug by making a test constant available for > turning on whatever special mode/code is wanted. > > More context on the example patch that triggered this query? > > Just 2c, > Cameron Simpson > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Thu May 15 13:27:46 2014 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 15 May 2014 14:27:46 +0300 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <52F61713.8000800@email.de> References: <20140207170745.ECC15568FA@psf.upfronthosting.co.za> <52F61713.8000800@email.de> Message-ID: Hi, On Sat, Feb 8, 2014 at 1:37 PM, francis wrote: > On 02/07/2014 06:07 PM, Python tracker wrote: >> >> Open issues with patches: 2045 > > > Has somebody done a graphic of that data againsttime? > You can find some charts here (it's still a work in progress though): http://bugs.python.org/issue?@template=stats Best Regards, Ezio Melotti > Regards, > francis > From skip at pobox.com Thu May 15 15:20:03 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 15 May 2014 08:20:03 -0500 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140515040105.GA82654@cskk.homeip.net> Message-ID: On Wed, May 14, 2014 at 11:05 PM, Guido van Rossum wrote: > Main problem with rare platform support is not breaking it accidentally, > since without buildbots we won't know when it's broken. This is why we don't > make any promises. Should we (or do we) offer to run (unofficial) buildbots for maintainers of minority platforms where possible? For example, I have no idea if a buildbot for MirOS is even feasible, but if the guy who submitted the patch is amenable and it is possible to run a buildbot slave for that OS, it still might be useful to have a "one stop" place for this. If failing, such buildbots wouldn't block a release, but would still provide tools for people to track down the source of breakage. Skip From solipsis at pitrou.net Thu May 15 15:26:24 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 May 2014 15:26:24 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140515040105.GA82654@cskk.homeip.net> Message-ID: <20140515152624.039883c7@fsol> On Thu, 15 May 2014 08:20:03 -0500 Skip Montanaro wrote: > On Wed, May 14, 2014 at 11:05 PM, Guido van Rossum wrote: > > Main problem with rare platform support is not breaking it accidentally, > > since without buildbots we won't know when it's broken. This is why we don't > > make any promises. > > Should we (or do we) offer to run (unofficial) buildbots for > maintainers of minority platforms where possible? For example, I have > no idea if a buildbot for MirOS is even feasible, but if the guy who > submitted the patch is amenable and it is possible to run a buildbot > slave for that OS, it still might be useful to have a "one stop" place > for this. If failing, such buildbots wouldn't block a release, but > would still provide tools for people to track down the source of > breakage. We already have such buildbots, they are in the "unstable" category. You can browse through existing buildbots here: https://www.python.org/dev/buildbot/ In the case of MirOS, though, I'm unsure core developers would proactively fix failures on such a niche platform :-) Regards Antoine. From skip at pobox.com Thu May 15 16:20:20 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 15 May 2014 09:20:20 -0500 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: <20140515152624.039883c7@fsol> References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> Message-ID: On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou wrote: > We already have such buildbots, they are in the "unstable" category. > You can browse through existing buildbots here: > https://www.python.org/dev/buildbot/ I can't see how to distinguish "stable" from "unstable" (or to view just the "unstable" category. What do those two categories have to do with "supported" and "unsupported"? Skip From bcannon at gmail.com Thu May 15 16:35:07 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 15 May 2014 14:35:07 +0000 Subject: [Python-Dev] Where is our official policy of what platforms we do support? References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> Message-ID: On Thu May 15 2014 at 10:24:45 AM, Skip Montanaro wrote: > On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou > wrote: > > We already have such buildbots, they are in the "unstable" category. > > You can browse through existing buildbots here: > > https://www.python.org/dev/buildbot/ > > I can't see how to distinguish "stable" from "unstable" (or to view > just the "unstable" category. Take http://buildbot.python.org/all/waterfall?category=3.x.stable&category=3.x.unstableand remove the category GET argument that you don't want to see, e.g. to only see unstable buildbots use http://buildbot.python.org/all/waterfall?category=3.x.unstable > What do those two categories have to do > with "supported" and "unsupported"? > Antoine can give the definitive answer, but I view stable buildbots as staying up and testing critical platforms. I.e. when I submit a patch I make sure the stable buildbots are always green (unless it's a transient failure) and I don't worry about the unstable ones (I view them as more informative than necessary). Basically red stable buildbot should block a release. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Thu May 15 16:40:33 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 15 May 2014 09:40:33 -0500 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> Message-ID: On Thu, May 15, 2014 at 9:35 AM, Brett Cannon wrote: > I view stable buildbots as staying up and testing critical platforms. Would "supported" and "unsupported" (or "critical" and "optional"?) make more sense? "Unstable" suggests "broken" to me, not "we don't really care about these." S From solipsis at pitrou.net Thu May 15 19:14:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 May 2014 19:14:55 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> Message-ID: <20140515191455.4b96baab@fsol> On Thu, 15 May 2014 09:40:33 -0500 Skip Montanaro wrote: > On Thu, May 15, 2014 at 9:35 AM, Brett Cannon wrote: > > I view stable buildbots as staying up and testing critical platforms. > > Would "supported" and "unsupported" (or "critical" and "optional"?) > make more sense? "Unstable" suggests "broken" to me, not "we don't > really care about these." I don't know who came up with these names in the first place. However there's a slight nuance here: some platform may be supported, but still some buildbot end up in the "unstable" category if it has issues of its own (for example the machine has a flaky network connection, etc.). And indeed there are Linux and Windows machines in the "unstable" category. Regards Antoine. From rdmurray at bitdance.com Thu May 15 20:34:07 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 15 May 2014 14:34:07 -0400 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: <20140515191455.4b96baab@fsol> References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> <20140515191455.4b96baab@fsol> Message-ID: <20140515183407.EE230250D5E@webabinitio.net> On Thu, 15 May 2014 19:14:55 +0200, Antoine Pitrou wrote: > On Thu, 15 May 2014 09:40:33 -0500 > Skip Montanaro wrote: > > On Thu, May 15, 2014 at 9:35 AM, Brett Cannon wrote: > > > I view stable buildbots as staying up and testing critical platforms. > > > > Would "supported" and "unsupported" (or "critical" and "optional"?) > > make more sense? "Unstable" suggests "broken" to me, not "we don't > > really care about these." > > I don't know who came up with these names in the first place. > However there's a slight nuance here: some platform may be supported, > but still some buildbot end up in the "unstable" category if it has > issues of its own (for example the machine has a flaky network > connection, etc.). And indeed there are Linux and Windows machines in > the "unstable" category. There's also nothing stopping us from putting a "niche platform" buildbot into the stable group if it normally builds fine. I suppose it would be pretty much supported by default then, though, if it being red was a release blocker. But we could decide to ignore a red 'niche' buildbot at release time; so, I think 'stable' vs 'unstable' is indeed the most descriptive: unstable buildbots are the ones that turn red "randomly"[*], or are always red because no one has fixed whatever the problem is (which might be on the buildbot or in our code). --David [*] Yes, our stable platforms do that sometimes too, but those are test instabilities, whereas unstable buildbots fail tests other than the known unstable tests. From status at bugs.python.org Fri May 16 18:07:57 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 16 May 2014 18:07:57 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140516160757.9498F560CB@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-05-09 - 2014-05-16) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4614 ( +3) closed 28674 (+50) total 33288 (+53) Open issues with patches: 2121 Issues opened (40) ================== #13916: disallow the "surrogatepass" handler for non utf-* encodings http://bugs.python.org/issue13916 reopened by haypo #19385: dbm.dumb should be consistent when the database is closed http://bugs.python.org/issue19385 reopened by serhiy.storchaka #21425: Interactive interpreter doesn't flush stderr prompty http://bugs.python.org/issue21425 reopened by pitrou #21459: DragonFlyBSD support http://bugs.python.org/issue21459 reopened by brett.cannon #21465: sqlite3 Row can return duplicate keys when using adapters http://bugs.python.org/issue21465 opened by BreamoreBoy #21468: NNTPLib connections become corrupt after long periods of activ http://bugs.python.org/issue21468 opened by James.Meneghello #21471: subprocess line-buffering only works in universal newlines mod http://bugs.python.org/issue21471 opened by pitrou #21472: Fix wsgiref handling of absolute HTTP Request-URI http://bugs.python.org/issue21472 opened by mouad #21473: Idle: test startup scripts. http://bugs.python.org/issue21473 opened by terry.reedy #21474: Idle: updata fixwordbreaks() for unicode identifiers http://bugs.python.org/issue21474 opened by terry.reedy #21475: Support the Sitemap and Crawl-delay extensions in robotparser http://bugs.python.org/issue21475 opened by rhettinger #21476: Inconsitent behaviour between BytesParser.parse and Parser.par http://bugs.python.org/issue21476 opened by ??ukasz.Kucharski #21477: Idle: improve idle_test,htest http://bugs.python.org/issue21477 opened by terry.reedy #21478: mock calls don't propagate to parent (autospec) http://bugs.python.org/issue21478 opened by and #21479: Document TarFile.open() as a classmethod http://bugs.python.org/issue21479 opened by berker.peksag #21480: A build now requires... http://bugs.python.org/issue21480 opened by skip.montanaro #21481: Argpase Namespace object methods __eq__ and __ne__ raise Type http://bugs.python.org/issue21481 opened by Joe.Borg #21482: get_versions() in cygwinccomiler.py cannot return correct gcc http://bugs.python.org/issue21482 opened by 3togo #21483: Skip os.utime() test on NFS? http://bugs.python.org/issue21483 opened by skip.montanaro #21484: More clarity needed about difference between "x += e" and "x = http://bugs.python.org/issue21484 opened by Kluzniak #21491: race condition in SocketServer.py ForkingMixIn collect_childre http://bugs.python.org/issue21491 opened by idsvandermolen #21493: Add test for ntpath.expanduser http://bugs.python.org/issue21493 opened by Claudiu.Popa #21495: Sane default for logging config http://bugs.python.org/issue21495 opened by guettli #21498: configparser accepts keys beginning with comment_chars when wr http://bugs.python.org/issue21498 opened by The Compiler #21500: Make use of the "load_tests" protocol in test_importlib packag http://bugs.python.org/issue21500 opened by eric.snow #21501: submitting mmap example for use in documentation http://bugs.python.org/issue21501 opened by hudson #21502: freeze.py not working properly with OS X framework builds http://bugs.python.org/issue21502 opened by yjiangnan #21503: Use test_both() consistently throughout test_importlib http://bugs.python.org/issue21503 opened by eric.snow #21504: can the subprocess module war using os.wait4 and so return usa http://bugs.python.org/issue21504 opened by donald.petravick #21505: cx_freeze multiprocessing bug http://bugs.python.org/issue21505 opened by shivani #21506: Windows MSI installer should mklink (symlink) python.exe to py http://bugs.python.org/issue21506 opened by edmorley #21507: set and frozenset constructor should use operator.length_hint http://bugs.python.org/issue21507 opened by lebedov #21508: C API PyArg_ParseTuple doc is innacurate http://bugs.python.org/issue21508 opened by Banger #21509: json.load fails to read UTF-8 file with (BOM) Byte Order Marks http://bugs.python.org/issue21509 opened by Kristian.Benoit #21510: fma documentation should provide better example. http://bugs.python.org/issue21510 opened by jayanthkoushik #21511: Thinko in Lib/quopri.py http://bugs.python.org/issue21511 opened by pfalcon #21513: speed up some ipaddress properties http://bugs.python.org/issue21513 opened by pitrou #21514: update json module docs in light of RFC 7159 & ECMA-404 http://bugs.python.org/issue21514 opened by cvrebert #21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile? http://bugs.python.org/issue21515 opened by haypo #21516: pathlib.Path(...).is_dir() crashes on some directories (Window http://bugs.python.org/issue21516 opened by theller Most recent 15 issues with no replies (15) ========================================== #21514: update json module docs in light of RFC 7159 & ECMA-404 http://bugs.python.org/issue21514 #21513: speed up some ipaddress properties http://bugs.python.org/issue21513 #21511: Thinko in Lib/quopri.py http://bugs.python.org/issue21511 #21505: cx_freeze multiprocessing bug http://bugs.python.org/issue21505 #21502: freeze.py not working properly with OS X framework builds http://bugs.python.org/issue21502 #21500: Make use of the "load_tests" protocol in test_importlib packag http://bugs.python.org/issue21500 #21498: configparser accepts keys beginning with comment_chars when wr http://bugs.python.org/issue21498 #21493: Add test for ntpath.expanduser http://bugs.python.org/issue21493 #21491: race condition in SocketServer.py ForkingMixIn collect_childre http://bugs.python.org/issue21491 #21479: Document TarFile.open() as a classmethod http://bugs.python.org/issue21479 #21477: Idle: improve idle_test,htest http://bugs.python.org/issue21477 #21476: Inconsitent behaviour between BytesParser.parse and Parser.par http://bugs.python.org/issue21476 #21473: Idle: test startup scripts. http://bugs.python.org/issue21473 #21472: Fix wsgiref handling of absolute HTTP Request-URI http://bugs.python.org/issue21472 #21468: NNTPLib connections become corrupt after long periods of activ http://bugs.python.org/issue21468 Most recent 15 issues waiting for review (15) ============================================= #21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile? http://bugs.python.org/issue21515 #21513: speed up some ipaddress properties http://bugs.python.org/issue21513 #21507: set and frozenset constructor should use operator.length_hint http://bugs.python.org/issue21507 #21503: Use test_both() consistently throughout test_importlib http://bugs.python.org/issue21503 #21493: Add test for ntpath.expanduser http://bugs.python.org/issue21493 #21479: Document TarFile.open() as a classmethod http://bugs.python.org/issue21479 #21472: Fix wsgiref handling of absolute HTTP Request-URI http://bugs.python.org/issue21472 #21463: RuntimeError when URLopener.ftpcache is full http://bugs.python.org/issue21463 #21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds http://bugs.python.org/issue21462 #21461: Recognize -pthread http://bugs.python.org/issue21461 #21459: DragonFlyBSD support http://bugs.python.org/issue21459 #21457: NetBSD curses support improvements http://bugs.python.org/issue21457 #21456: skip 2 tests in test_urllib2net.py if _ssl module not present http://bugs.python.org/issue21456 #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId http://bugs.python.org/issue21449 Top 10 most discussed issues (10) ================================= #21425: Interactive interpreter doesn't flush stderr prompty http://bugs.python.org/issue21425 18 msgs #13916: disallow the "surrogatepass" handler for non utf-* encodings http://bugs.python.org/issue13916 14 msgs #7776: http.client.HTTPConnection tunneling is broken http://bugs.python.org/issue7776 9 msgs #21506: Windows MSI installer should mklink (symlink) python.exe to py http://bugs.python.org/issue21506 9 msgs #21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile? http://bugs.python.org/issue21515 8 msgs #21027: difflib new cli interface http://bugs.python.org/issue21027 6 msgs #21304: PEP 466: Backport hashlib.pbkdf2_hmac to Python 2.7 http://bugs.python.org/issue21304 6 msgs #21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists http://bugs.python.org/issue21402 6 msgs #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 6 msgs #21507: set and frozenset constructor should use operator.length_hint http://bugs.python.org/issue21507 6 msgs Issues closed (52) ================== #9266: ctypes "ValueError: NULL pointer access" on Win7 x64 http://bugs.python.org/issue9266 closed by berker.peksag #10752: build_ssl.py is relying on unreliable behaviour of os.popen http://bugs.python.org/issue10752 closed by tim.golden #12985: Check signed arithmetic overflow in ./configure http://bugs.python.org/issue12985 closed by skrah #13869: CFLAGS="-UNDEBUG" build failure http://bugs.python.org/issue13869 closed by skrah #14142: getlocale(LC_ALL) behavior http://bugs.python.org/issue14142 closed by skrah #14198: Backport parts of the new memoryview documentation http://bugs.python.org/issue14198 closed by skrah #16531: Allow IPNetwork to take a tuple http://bugs.python.org/issue16531 closed by pitrou #18062: m68k FPU precision issue http://bugs.python.org/issue18062 closed by skrah #18104: Idle: make human-mediated GUI tests usable http://bugs.python.org/issue18104 closed by terry.reedy #19655: Replace the ASDL parser carried with CPython http://bugs.python.org/issue19655 closed by eli.bendersky #19721: Move all test_importlib utility code into test_importlib.util http://bugs.python.org/issue19721 closed by brett.cannon #19775: Provide samefile() on Path objects http://bugs.python.org/issue19775 closed by pitrou #20776: Add tests for importlib.machinery.PathFinder http://bugs.python.org/issue20776 closed by brett.cannon #20826: Faster implementation to collapse consecutive ip-networks http://bugs.python.org/issue20826 closed by pitrou #20998: fullmatch isn't matching correctly under re.IGNORECASE http://bugs.python.org/issue20998 closed by serhiy.storchaka #21037: add an AddressSanitizer build option http://bugs.python.org/issue21037 closed by neologix #21075: fileinput should use stdin.buffer for "rb" mode http://bugs.python.org/issue21075 closed by serhiy.storchaka #21088: curses addch() argument position reverses in Python3.4.0 http://bugs.python.org/issue21088 closed by larry #21156: Consider moving importlib.abc.InspectLoader.source_to_code() t http://bugs.python.org/issue21156 closed by brett.cannon #21226: PyImport_ExecCodeModuleObject not setting module attributes http://bugs.python.org/issue21226 closed by eric.snow #21237: Update Python 2/3 porting HOWTO's suggestion for dealing with http://bugs.python.org/issue21237 closed by brett.cannon #21306: PEP 466: backport hmac.compare_digest http://bugs.python.org/issue21306 closed by python-dev #21347: Don't use a list argument together with shell=True in subproce http://bugs.python.org/issue21347 closed by r.david.murray #21363: io.TextIOWrapper always closes wrapped files http://bugs.python.org/issue21363 closed by r.david.murray #21364: Documentation Recommends Broken Pattern http://bugs.python.org/issue21364 closed by pitrou #21370: segfault from simple traceback.format_exc call http://bugs.python.org/issue21370 closed by skrah #21383: "make touch" fails when the build directory is not the source http://bugs.python.org/issue21383 closed by ned.deily #21384: Windows: Make handle non inheritable by default http://bugs.python.org/issue21384 closed by haypo #21398: LC_CTYPE=C: pydoc leaves terminal in an unusable state http://bugs.python.org/issue21398 closed by haypo #21418: Segv during call to super_init in application embedding Python http://bugs.python.org/issue21418 closed by haypo #21419: Use calloc() instead of malloc() for int << int (lshift) http://bugs.python.org/issue21419 closed by haypo #21420: Optimize 2 ** n: implement it as 1 << n http://bugs.python.org/issue21420 closed by haypo #21422: int << 0: return the number unmodified http://bugs.python.org/issue21422 closed by haypo #21424: Simplify and speed-up heaqp.nlargest() http://bugs.python.org/issue21424 closed by rhettinger #21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat http://bugs.python.org/issue21452 closed by tim.golden #21464: fnmatch module uses regular expression with undefined result t http://bugs.python.org/issue21464 closed by pfalcon #21466: General Index link to del statement is wrong http://bugs.python.org/issue21466 closed by python-dev #21467: IDLE icon not included in Windows installer http://bugs.python.org/issue21467 closed by steve.dower #21469: False positive hazards in robotparser http://bugs.python.org/issue21469 closed by rhettinger #21470: Better seeding for the random module http://bugs.python.org/issue21470 closed by rhettinger #21485: remove unnecesary .flush() calls in the asyncio subprocess cod http://bugs.python.org/issue21485 closed by haypo #21486: optimize v4 & v6 netmask parsing http://bugs.python.org/issue21486 closed by pitrou #21487: Assorted ipaddress performance improvements http://bugs.python.org/issue21487 closed by pitrou #21488: codecs.encode/decode documentation inconsistency http://bugs.python.org/issue21488 closed by haypo #21489: Switching from -OO to -O still uses cached bytecode http://bugs.python.org/issue21489 closed by brett.cannon #21490: Add Py_ABS and Py_STRINGIFY macros http://bugs.python.org/issue21490 closed by haypo #21492: email.header.decode_header sometimes returns bytes, sometimes http://bugs.python.org/issue21492 closed by r.david.murray #21494: getopt error doesnot display correct error http://bugs.python.org/issue21494 closed by eric.smith #21496: pyvenv activate_this.py http://bugs.python.org/issue21496 closed by vinay.sajip #21497: faulthandler should handle sys.stderr being None gracefully http://bugs.python.org/issue21497 closed by haypo #21499: test_importlib incorrectly relies on .__builtins__ http://bugs.python.org/issue21499 closed by eric.snow #21512: time module becomes None after raise SystemExit http://bugs.python.org/issue21512 closed by ezio.melotti From bcannon at gmail.com Fri May 16 19:51:00 2014 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 16 May 2014 17:51:00 +0000 Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support Message-ID: Here is some proposed wording. Since it is more of a clarification of what it takes to garner support -- which is just a new section -- rather than a complete rewrite I'm including just the diff to make it easier to read the changes. *diff -r 49d18bb47ebc pep-0011.txt* *--- a/pep-0011.txt Wed May 14 11:18:22 2014 -0400* *+++ b/pep-0011.txt Fri May 16 13:48:30 2014 -0400* @@ -2,22 +2,21 @@ Title: Removing support for little used platforms Version: $Revision$ Last-Modified: $Date$ -Author: martin at v.loewis.de (Martin von L?wis) +Author: Martin von L?wis , + Brett Cannon Status: Active Type: Process Content-Type: text/x-rst Created: 07-Jul-2002 Post-History: 18-Aug-2007 + 16-May-2014 Abstract -------- -This PEP documents operating systems (platforms) which are not -supported in Python anymore. For some of these systems, -supporting code might be still part of Python, but will be removed -in a future release - unless somebody steps forward as a volunteer -to maintain this code. +This PEP documents how an operating system (platform) garners +support in Python as well as documenting past support. Rationale @@ -37,16 +36,53 @@ change to the Python source code will work on all supported platforms. -To reduce this risk, this PEP proposes a procedure to remove code -for platforms with no Python users. +To reduce this risk, this PEP specifies what is required for a +platform to be considered supported by Python as well as providing a +procedure to remove code for platforms with little or no Python +users. +Supporting platforms +-------------------- + +Gaining official platform support requires two things. First, a core +developer needs to volunteer to maintain platform-specific code. This +core developer can either already be a member of the Python +development team or be given contributor rights on the basis of +maintaining platform support (it is at the discretion of the Python +development team to decide if a person is ready to have such rights +even if it is just for supporting a specific platform). + +Second, a stable buildbot must be provided [2]_. This guarantees that +platform support will not be accidentally broken by a Python core +developer who does not have personal access to the platform. For a +buildbot to be considered stable it requires that the machine be +reliably up and functioning (but it is up to the Python core +developers to decide whether to promote a buildbot to being +considered stable). + +This policy does not disqualify supporting other platforms +indirectly. Patches which are not platform-specific but still done to +add platform support will be considered for inclusion. For example, +if platform-independent changes were necessary in the configure +script which was motivated to support a specific platform that would +be accepted. Patches which add platform-specific code such as the +name of a specific platform to the configure script will generally +not be accepted without the platform having official support. + +CPU architecture and compiler support are viewed in a similar manner +as platforms. For example, to consider the ARM architecture supported +a buildbot running on ARM would be required along with support from +the Python development team. In general it is not required to have +a CPU architecture run under every possible platform in order to be +considered supported. Unsupporting platforms ---------------------- -If a certain platform that currently has special code in it is -deemed to be without Python users, a note must be posted in this -PEP that this platform is no longer actively supported. This +If a certain platform that currently has special code in Python is +deemed to be without Python users or lacks proper support from the +Python development team and/or a buildbot, a note must be posted in +this PEP that this platform is no longer actively supported. This note must include: - the name of the system @@ -69,8 +105,8 @@ forward and offer maintenance. -Resupporting platforms ----------------------- +Re-supporting platforms +----------------------- If a user of a platform wants to see this platform supported again, he may volunteer to maintain the platform support. Such an @@ -101,7 +137,7 @@ release is made. Developers of extension modules will generally need to use the same Visual Studio release; they are concerned both with the availability of the versions they need to use, and with keeping -the zoo of versions small. The Python source tree will keep +the zoo of versions small. The Python source tree will keep unmaintained build files for older Visual Studio releases, for which patches will be accepted. Such build files will be removed from the source tree 3 years after the extended support for the compiler has @@ -223,6 +259,7 @@ ---------- .. [1] http://support.microsoft.com/lifecycle/ +.. [2] http://buildbot.python.org/3.x.stable/ Copyright --------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 17 07:14:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 May 2014 15:14:00 +1000 Subject: [Python-Dev] Returning None from methods that mutate object state Message-ID: During a conversation today, I realised that the convention of returning None from methods that change an object's state isn't captured the Programming Recommendations section of PEP 8. Specifically, I'm referring to this behaviour: >>> [].sort() is None True >>> "ABC".lower() is None False That's a deliberate design choice, and one that has been explained a few times on the list when folks ask why "[].sort().reverse()" doesn't work when "'ABC'.lower().replace('-', '_')" does. Would it be worth adding such a note? Or is it out of scope? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sat May 17 10:26:12 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 May 2014 04:26:12 -0400 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: On 5/17/2014 1:14 AM, Nick Coghlan wrote: > During a conversation today, I realised that the convention of > returning None from methods that change an object's state isn't > captured the Programming Recommendations section of PEP 8. > Specifically, I'm referring to this behaviour: > >>>> [].sort() is None > True >>>> "ABC".lower() is None > False When list.pop was added, the convention was changed to "do not return the 'self' parameter" >>> [1].pop() is None False Not returning 'self' allows some mutation functions to return something other than 'self'. I phrase the rule the way I did because a recursive collections can incidentally return itself. >>> L = [] >>> L.append(L) >>> L.pop() is L True it.__next__ is another mutator that returns neither self or None. Actually, if one regards file read and write as mutation, then returning None never was the rule. It seems to me that the actual Python rule is "Don't return 'self'. If there is nothing useful to return (other than self), return None." I believe this is true whether or not self is mutated. (Of course, there might be an exception I have overlooked.) > That's a deliberate design choice, and one that has been explained a > few times on the list when folks ask why "[].sort().reverse()" doesn't > work when "'ABC'.lower().replace('-', '_')" does. Do people ask why sys.stdout.write(line).write('\n') does not work? > Would it be worth adding such a note? Or is it out of scope? -- Terry Jan Reedy From ncoghlan at gmail.com Sat May 17 10:44:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 May 2014 18:44:28 +1000 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: On 17 May 2014 18:26, Terry Reedy wrote: > On 5/17/2014 1:14 AM, Nick Coghlan wrote: >> >> During a conversation today, I realised that the convention of >> returning None from methods that change an object's state isn't >> captured the Programming Recommendations section of PEP 8. >> Specifically, I'm referring to this behaviour: >> >>>>> [].sort() is None >> >> True >>>>> >>>>> "ABC".lower() is None >> >> False > > > When list.pop was added, the convention was changed to > "do not return the 'self' parameter" > >>>> [1].pop() is None > False > > Not returning 'self' allows some mutation functions to return something > other than 'self'. That's a good point - I wasn't thinking methods like pop() when phrasing it the way I did. > I phrase the rule the way I did because a recursive collections can > incidentally return itself. > >>>> L = [] >>>> L.append(L) >>>> L.pop() is L > True > > it.__next__ is another mutator that returns neither self or None. > > Actually, if one regards file read and write as mutation, then returning > None never was the rule. dict.pop() is a similar example. It was bad phrasing on my part, because I was thinking of a particular subset of mutating methods (the ones which don't have a more meaningful result that they need to return) > It seems to me that the actual Python rule is "Don't return 'self'. If there > is nothing useful to return (other than self), return None." I believe this > is true whether or not self is mutated. (Of course, there might be an > exception I have overlooked.) It's actually even more subtle than that - it's "Don't return self, unless required to do so by a protocol" (specifically, __iter__ and the _i*__ methods sometimes involve returning "self" for iterators and types that do in-place operations respectively) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat May 17 11:55:12 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 17 May 2014 11:55:12 +0200 Subject: [Python-Dev] Returning None from methods that mutate object state References: Message-ID: <20140517115512.1e1497f1@fsol> On Sat, 17 May 2014 15:14:00 +1000 Nick Coghlan wrote: > During a conversation today, I realised that the convention of > returning None from methods that change an object's state isn't > captured the Programming Recommendations section of PEP 8. This is more an API design guideline than a programming recommandation, IMO. If we start putting those in PEP 8, it will end up bloating the PEP. (perhaps we need another PEP?) Regards Antoine. From steve at pearwood.info Sat May 17 12:44:35 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 May 2014 20:44:35 +1000 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: <20140517104435.GS4273@ando> On Sat, May 17, 2014 at 04:26:12AM -0400, Terry Reedy wrote: > On 5/17/2014 1:14 AM, Nick Coghlan wrote: > >During a conversation today, I realised that the convention of > >returning None from methods that change an object's state isn't > >captured the Programming Recommendations section of PEP 8. > >Specifically, I'm referring to this behaviour: > > > >>>>[].sort() is None > >True > >>>>"ABC".lower() is None > >False > > When list.pop was added, the convention was changed to > "do not return the 'self' parameter" I don't think "return self" was ever the convention. Here's Python 1.5: [steve at ando ~]$ python1.5 Python 1.5.2 (#1, Aug 27 2012, 09:09:18) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> [].sort() is None 1 >>> [].reverse() is None 1 >>> [1, 2, 3].remove(2) is None 1 >>> {}.update({}) is None 1 (Wow. Returning 1 instead of True -- it's ages since I've seen that.) I think the rule "return None, not self" should be best understood as applying only to mutator methods which *only* operate for their side-effects (like sorting, reversing), not those which also have an obvious return value: >>> [2, 4, 6].pop() # Still Python 1.5. 6 I don't recall any other examples from 1.5 that did this. > it.__next__ is another mutator that returns neither self or None. > > Actually, if one regards file read and write as mutation, then returning > None never was the rule. I don't think we should regard reading as a mutation :-) Writing to a file on disk may be a side-effect, but I don't think it counts as a mutation of the file object. But even if we consider it as one, prior to 3.x writing returned None. This is from 2.7: py> open("/dev/null", "w").write("goodbye cruel world") is None True > It seems to me that the actual Python rule is "Don't return 'self'. If > there is nothing useful to return (other than self), return None." I > believe this is true whether or not self is mutated. (Of course, there > might be an exception I have overlooked.) The obvious exception is the iterator protocol: __iter__ of an iterator must return self. But that's an exceptional case. I don't think it is fair to say that Python imposes a *rule* "don't return self" -- it's more of a convention for built-ins only, not a rule for all Python classes. Ruby prefers methods that return self and are therefore chainable, while Python built-ins do not. I think it would be worth having PEP 8 make it clear that: - built-ins, and types which share the same interface as built-ins, should have mutators which return None rather than self; - but third-party types (including classes in the standard library, if necessary) may return self; - whichever choice you make, be consistant within a single API. E.g. don't have x.mutator1() return None, while x.mutator2() returns self. -- Steven From ncoghlan at gmail.com Sat May 17 16:18:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 May 2014 00:18:20 +1000 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: <20140517115512.1e1497f1@fsol> References: <20140517115512.1e1497f1@fsol> Message-ID: On 17 May 2014 19:56, "Antoine Pitrou" wrote: > > On Sat, 17 May 2014 15:14:00 +1000 > Nick Coghlan wrote: > > During a conversation today, I realised that the convention of > > returning None from methods that change an object's state isn't > > captured the Programming Recommendations section of PEP 8. > > This is more an API design guideline than a programming recommandation, > IMO. If we start putting those in PEP 8, it will end up bloating the > PEP. > (perhaps we need another PEP?) A new entry in https://docs.python.org/3/faq/design.html may be a reasonable option. It's not like it comes up *that* often, and as Nathaniel pointed out, the main advantage of writing it down somewhere is to be able to link to it in the future. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat May 17 15:39:28 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 17 May 2014 14:39:28 +0100 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: On Sat, May 17, 2014 at 6:14 AM, Nick Coghlan wrote: > During a conversation today, I realised that the convention of > returning None from methods that change an object's state isn't > captured the Programming Recommendations section of PEP 8. > Specifically, I'm referring to this behaviour: > >>>> [].sort() is None > True >>>> "ABC".lower() is None > False > > That's a deliberate design choice, and one that has been explained a > few times on the list when folks ask why "[].sort().reverse()" doesn't > work when "'ABC'.lower().replace('-', '_')" does. > > Would it be worth adding such a note? Or is it out of scope? Numpy also has a menagerie of in-place and out-of-place methods that follow this convention, and we also have to go round on the mailing list from time to time explaining to well-meaning beginners why for the in-place methods returning None is the right ("Pythonic") design. Having a canonical explanation of this to link to would be useful. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From tismer at stackless.com Sat May 17 16:45:54 2014 From: tismer at stackless.com (Christian Tismer) Date: Sat, 17 May 2014 16:45:54 +0200 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org>, <20140511131748.611ce431@fsol> Message-ID: <53777622.3010209@stackless.com> On 11.05.14 15:57, Steve Dower wrote: > Thanks. > > For those who missed the earlier discussions, Martin v. L?wis has > handed over responsibility for the Windows installers. It sounds like > Brett Cannon and I are both in a position to build 2.7 right now, and > I hope to simplify the setup required for 3.5 so that anyone can > produce and test the installers. (Martin is going to look after the > 3.4.x builds.) > > Obviously I'm also here to help out with Windows in general. I haven't > quite managed to get 50% time from my employer (maybe 10%), but my > management at least is very supportive of my participation and keen to > keep Python running well. > Very nice, great to read this. Welcome from me as well! cheers - Chris -- Christian Tismer :^) tismer at stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 17 17:50:47 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 17 May 2014 08:50:47 -0700 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: Please preserve the tradition. Adding it to PEP 8 sounds good! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Sun May 18 03:26:19 2014 From: bcannon at gmail.com (Dr. Brett Cannon) Date: Sun, 18 May 2014 01:26:19 +0000 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> Message-ID: I don't think you meant me for helping to build Windows binaries. :) On Sunday, May 11, 2014 9:58:16 AM, Steve Dower wrote: Thanks. For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.) Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well. Cheers, Steve Top-posted from my Windows Phone From: Antoine Pitrou Sent: ?5/?11/?2014 4:18 To: python-dev@ python.org Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. On Sun, 11 May 2014 06:04:56 +0200 (CEST) steve.dower checkins @ python.org > wrote: > http:// hg.python.org / devguide /rev/8d5d1f2c7378 > changeset: 698:8d5d1f2c7378 > user: Steve Dower @ microsoft.com > > date: Sat May 10 21:01:39 2014 -0700 > summary: > Add myself to developer log and as a Windows expert. Welcome onboard, Steve ! Nice to have an additional Windows expert :-) Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@ python.org https :// mail.python.org /mailman/ listinfo /python-dev Unsubscribe: https :// mail.python.org /mailman/options/python-dev/ steve.dower %40microsoft.com _______________________________________________ Python-Dev mailing list Python-Dev@ python.org https :// mail.python.org /mailman/ listinfo /python-dev Unsubscribe: https :// mail.python.org /mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun May 18 04:04:02 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 18 May 2014 02:04:02 +0000 Subject: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. In-Reply-To: References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol> , Message-ID: No, apologies for that. As Nick pointed out, I meant the *other* B.C. :) Top-posted from my Windows Phone ________________________________ From: Dr. Brett Cannon Sent: ?5/?17/?2014 18:26 To: Steve Dower; solipsis at pitrou.net; python-dev at python.org Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. I don't think you meant me for helping to build Windows binaries. :) On Sunday, May 11, 2014 9:58:16 AM, Steve Dower > wrote: Thanks. For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.) Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well. Cheers, Steve Top-posted from my Windows Phone From: Antoine Pitrou Sent: ?5/?11/?2014 4:18 To: python-dev@python.org Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert. On Sun, 11 May 2014 06:04:56 +0200 (CEST) steve.dower checkins@python.org> wrote: > http://hg.python.org/devguide/rev/8d5d1f2c7378 > changeset: 698:8d5d1f2c7378 > user: Steve Dower @microsoft.com> > date: Sat May 10 21:01:39 2014 -0700 > summary: > Add myself to developer log and as a Windows expert. Welcome onboard, Steve ! Nice to have an additional Windows expert :-) Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun May 18 15:00:54 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 May 2014 15:00:54 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: References: <20140515040105.GA82654@cskk.homeip.net> <20140515152624.039883c7@fsol> Message-ID: <5378AF06.7030704@v.loewis.de> Am 15.05.14 16:20, schrieb Skip Montanaro: > On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou wrote: >> We already have such buildbots, they are in the "unstable" category. >> You can browse through existing buildbots here: >> https://www.python.org/dev/buildbot/ > > I can't see how to distinguish "stable" from "unstable" (or to view > just the "unstable" category. What do those two categories have to do > with "supported" and "unsupported"? There are two competing definitions of "stable" vs. "unstable" when it comes to buildbots. 1. "stable buildbot" (my preferred definition) * the buildbot is available most of the time, an operator will deal with it when it happens to go down, and the builds always complete. * consequentially, an unstable buildbot is one that tends to disconnect, and takes a long time for the operator to recover 2. "stable platform" * all tests are expected to pass all the time; i.e. there are no tests that randomly fail, and no platform-specific failures * thus, a failed test is an indication that a new bug has been introduced When Antoine talked about "unofficial" buildbots, he was referring to the fact that we accept nearly any buildbot offered, but put them into the unstable category, which they often belong to by either definition of unstable. Regards, Martin From martin at v.loewis.de Sun May 18 15:10:55 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 May 2014 15:10:55 +0200 Subject: [Python-Dev] Where is our official policy of what platforms we do support? In-Reply-To: <20140514102820.0e13a141@anarchist.wooz.org> References: <20140514102820.0e13a141@anarchist.wooz.org> Message-ID: <5378B15F.9040207@v.loewis.de> Am 14.05.14 16:28, schrieb Barry Warsaw: > On May 14, 2014, at 02:20 PM, Brett Cannon wrote: > >> Do we want an official policy written down in a PEP (yes, I can write it)? >> Should I keep closing these patches and saying that we are not adding >> support for new operating systems and be hand-wavy about it? > > Yes, I think a PEP describing both policy and implementation (i.e. which > platforms we officially support) is worthwhile. While I think you could write > a new PEP, I also think it might just make sense to co-opt PEP 11 and broaden > its scope, since as you say, removing support is only half the story. I'd be careful to use the word "support". AFAIK, we do not offer support for Python on any platform, beyond accepting bug reports in the bug tracker. Paid support for Python is certainly offered by some companies, but not by us. IIUC, Brett is asking the question "What platforms is CPython supposed to work on?". Thanks to autoconf, it is impossible to give a complete list of such platforms. It might be that Python builds just fine, with no changes required, on a system that we've never heard of. So if you want to give a complete list of "supported" platforms, it might be the question "What platforms do we care about?", in the sense that someone would be willing to invest time if it stops working. Again, as we do not offer paid support, this list might get misleadingly long. I'd be personally willing to invest time looking into a lot of problems - just not within the next 12 months (but surely some day :-) The only reasonable positive platform list I can think of is "What platforms do we consider release-critical?", in the sense of not letting a release proceed if it fails to work on one of these platforms. We've had such a list for a long time, it is: - Microsoft Windows - Linux - Apple Mac OS X Regards, Martin From benjamin at python.org Mon May 19 02:49:18 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 May 2014 17:49:18 -0700 Subject: [Python-Dev] [RELEASED] Python 2.7.7 release candidate 1 Message-ID: <1400460558.31367.118834865.1A1F5777@webmail.messagingengine.com> Greetings Python users, Python 2.7.7 release candidate 1 is now available for download. Python 2.7.7 is a regularly scheduled bugfix release for the Python 2.7 series. The 2.7.7 release contains fixes for two severe, if arcane, potential security vulnerabilities. The first was the possibility of reading arbitrary process memory using JSONDecoder.raw_decode. [1] (No other json APIs are affected.) The second security issue is an integer overflow in the strop module. [2] (If you don't know what the strop module is, go ahead and forget it now.) This release also includes months of accumulated normal bugfixes. All the changes in Python 2.7.7 are described in detail in the Misc/NEWS file of the source tarball. You can view it online at http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS Downloads are at https://python.org/download/releases/2.7.7/ This is a testing release. Assuming no horrible bugs are found, 2.7.7 final will be released in two weeks time. Please consider testing your applications and libraries with the release candidate and reporting bugs to http://bugs.python.org/ Enjoy, Benjamin Peterson 2.7 Release Manager [1] http://bugs.python.org/issue21529 [2] http://bugs.python.org/issue21530 From guido at python.org Mon May 19 03:53:42 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 18 May 2014 18:53:42 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 Message-ID: On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson wrote: > Greetings Python users, > Python 2.7.7 release candidate 1 is now available for download. [...] > > http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS > So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon May 19 04:04:37 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 May 2014 19:04:37 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: References: Message-ID: <1400465077.15559.118851701.268719A5@webmail.messagingengine.com> On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote: > On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson > wrote: > > > Greetings Python users, > > Python 2.7.7 release candidate 1 is now available for download. [...] > > > > http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS > > > > So what became of PEP 466? This Misc/NEWS only mentions > hmac.compare_digest. It didn't get completely done. Mostly of it will land in 2.7.8 presumably. From benjamin at python.org Mon May 19 04:06:33 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 May 2014 19:06:33 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 Message-ID: <1400465193.16230.118851701.51F23E6F@webmail.messagingengine.com> On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote: > On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson > wrote: > > > Greetings Python users, > > Python 2.7.7 release candidate 1 is now available for download. [...] > > > > http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS > > > > So what became of PEP 466? This Misc/NEWS only mentions > hmac.compare_digest. It didn't get completely done. Mostly of it will land in 2.7.8 presumably. -- Regards, Benjamin From benjamin at python.org Mon May 19 04:06:34 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 May 2014 19:06:34 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 Message-ID: <1400465194.16051.118851701.6C5E6417@webmail.messagingengine.com> On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote: > On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson > wrote: > > > Greetings Python users, > > Python 2.7.7 release candidate 1 is now available for download. [...] > > > > http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS > > > > So what became of PEP 466? This Misc/NEWS only mentions > hmac.compare_digest. It didn't get completely done. Mostly of it will land in 2.7.8 presumably. -- Regards, Benjamin From donald at stufft.io Mon May 19 04:02:10 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 18 May 2014 22:02:10 -0400 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: References: Message-ID: On May 18, 2014, at 9:53 PM, Guido van Rossum wrote: > On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson wrote: > Greetings Python users, > Python 2.7.7 release candidate 1 is now available for download. [...] > > http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS > > So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io The SSL changes were too large to get done before 2.7.7 The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don?t feel strong enough in C code to call it a proper review. The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307) The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305) And that was everything from PEP 466. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Mon May 19 04:28:32 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 18 May 2014 19:28:32 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: References: Message-ID: Thanks for the update, Donald. Did anyone get any help from Red Hat or other distros? On Sun, May 18, 2014 at 7:02 PM, Donald Stufft wrote: > > On May 18, 2014, at 9:53 PM, Guido van Rossum wrote: > > On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson wrote: > >> Greetings Python users, >> Python 2.7.7 release candidate 1 is now available for download. [...] >> >> http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS >> > > So what became of PEP 466? This Misc/NEWS only mentions > hmac.compare_digest. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > The SSL changes were too large to get done before 2.7.7 > > The pbkdf2 has a patch sitting on the tracker ( > http://bugs.python.org/issue21304) Alex wanted someone to review before > commit. I looked over it but I don?t feel strong enough in C code to call > it a proper review. > > The guaranteed_algorithms bug depends on the pbkdf2 bug ( > http://bugs.python.org/issue21307) > > The os.urandom change had some argument and some concern that the related > change in 3.4 was still new and had some bugs being ironed out so it was > punted until 2.7.8 (http://bugs.python.org/issue21305) > > And that was everything from PEP 466. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon May 19 04:33:30 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 18 May 2014 22:33:30 -0400 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: References: Message-ID: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io> Well I believe Alex did what he did during his work day @ Rackspace. Distros specifically I don?t believe so, although both Alex and myself agreed that it made sense for the SSL changes to wait until after the other changes since it was the largest and most complicated. I think that?s the one where it makes the most sense to try and garner help from Red Hat or the like. Perhaps Nick knows someone at Red Hat we can poke to see if they?d be willing to do the SSL patch :) On May 18, 2014, at 10:28 PM, Guido van Rossum wrote: > Thanks for the update, Donald. Did anyone get any help from Red Hat or other distros? > > > On Sun, May 18, 2014 at 7:02 PM, Donald Stufft wrote: > > On May 18, 2014, at 9:53 PM, Guido van Rossum wrote: > >> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson wrote: >> Greetings Python users, >> Python 2.7.7 release candidate 1 is now available for download. [...] >> >> http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS >> >> So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest. >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > The SSL changes were too large to get done before 2.7.7 > > The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don?t feel strong enough in C code to call it a proper review. > > The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307) > > The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305) > > And that was everything from PEP 466. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > > -- > --Guido van Rossum (python.org/~guido) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Mon May 19 04:35:51 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 18 May 2014 19:35:51 -0700 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io> References: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io> Message-ID: At the very least PEP 466 needs to be updated to admit the failure -- it would be a shame if people read the PEP and assumed the promised features actually landed in 2.7.7 (which the PEP explicitly lists). -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 19 06:07:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 May 2014 14:07:27 +1000 Subject: [Python-Dev] Python 2.7.7 and PEP 466 In-Reply-To: References: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io> Message-ID: On 19 May 2014 12:35, Guido van Rossum wrote: > At the very least PEP 466 needs to be updated to admit the failure -- it > would be a shame if people read the PEP and assumed the promised features > actually landed in 2.7.7 (which the PEP explicitly lists). Will do - I'll update that to reference the specific issues tracking the implementation of the individual elements. On a related note, I was also thinking about adding a new section to the What's New in Python 2.7 doc. Specifically, a new "Security Enhancements in Maintenance Releases" section after the existing "The Future of Python 2.x" section. That would reference PEP 466 for background, and then list the specific maintenance releases where these features have been added (so just the one 2.7.7 entry for hmac.compare_digest to start with). I'd also add a direct link to PEP 373 (the 2.7 release schedule PEP) from the first bullet point under "The Future of Python 2.x" section (as well as rewording that point to better reflect the current state of things) Regards, Nick. P.S. As far as additional development resources for long term upstream CPython maintenance go - I'm working on it (and my understanding is that folks at other orgs are as well). Personally, I'm still in the gap between "that's likely a good idea" and actually translating the concept into available developer time. While Heartbleed has helped raise awareness of the whole "What are we depending on without committing sufficient development resources to long term maintenance?" problem, large orgs still don't tend to move that fast :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Mon May 19 06:52:58 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 May 2014 00:52:58 -0400 Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits, what do I do? Message-ID: An hour ago, I pulled recent commits to my local repository. I edited a couple of files, wrote commit messages, and pulled again, nothing new. I then did the usual: commit 2.7, commit 3.4, merge to 3.5. Problem: merge conflicts from 6 files I have never touched. Include/patchlevel.h Lib/distutils/__init__.py Lib/idlelib/idlever.py Lib/pydoc_data/topics.py Misc/RPM/python-3.5.spec README I stopped at this point and ran diskcheck. I then looked at the DAG and noticed 5 previous 3.4 patches that were not merged into 3.5: rev 90750 and 90751, Larry Hastings, 2 weeks ago!, rev 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 hours ago. From earliest to lastest: c67a19e11a71 7c5f1b200a24 35ea333f43bd 31211947387b 854fd6eeee2f If there was any notice on the Committer's list about not making commits, I did not get it. In fact, I do not remember getting anything on that list for at about a month. Besides that, I had no problem 3 days ago, and noticed the commit by Raymond, though not the absence of a merge. In any case, what do I do now, short of deleting and re-cloning, so I do not make anything worse? -- Terry Jan Reedy From raymond.hettinger at gmail.com Mon May 19 07:31:10 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 19 May 2014 06:31:10 +0100 Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits, what do I do? In-Reply-To: References: Message-ID: On May 19, 2014, at 5:52 AM, Terry Reedy wrote: > I stopped at this point and ran diskcheck. I then looked at the DAG and noticed 5 previous 3.4 patches that were not merged into 3.5: rev 90750 and 90751, Larry Hastings, 2 weeks ago!, rev > 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 hours ago. From earliest to lastest: > > c67a19e11a71 > 7c5f1b200a24 > 35ea333f43bd > 31211947387b > 854fd6eeee2f > > If there was any notice on the Committer's list about not making commits, I did not get it. In fact, I do not remember getting anything on that list for at about a month. Besides that, I had no problem 3 days ago, and noticed the commit by Raymond, though not the absence of a merge. > > In any case, what do I do now, short of deleting and re-cloning, so I do not make anything worse? I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges, all with conflicts). To get my repo back into a usable state, I ran "hg update --clean" and am waiting for Larry to do his cleans-up. Once his are in, I'll merge my last 3.4 commit (which should be easy since it has no conflicts). Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon May 19 08:08:05 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 May 2014 02:08:05 -0400 Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits, what do I do? In-Reply-To: References: Message-ID: On 5/19/2014 1:31 AM, Raymond Hettinger wrote: > > On May 19, 2014, at 5:52 AM, Terry Reedy > wrote: > >> I stopped at this point and ran diskcheck. I then looked at the DAG >> and noticed 5 previous 3.4 patches that were not merged into 3.5: rev >> 90750 and 90751, Larry Hastings, 2 weeks ago!, rev >> 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 >> hours ago. From earliest to lastest: >> >> c67a19e11a71 >> 7c5f1b200a24 >> 35ea333f43bd >> 31211947387b >> 854fd6eeee2f >> >> If there was any notice on the Committer's list about not making >> commits, I did not get it. In fact, I do not remember getting anything >> on that list for at about a month. Besides that, I had no problem 3 >> days ago, and noticed the commit by Raymond, though not the absence of >> a merge. >> >> In any case, what do I do now, short of deleting and re-cloning, so I >> do not make anything worse? > > I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges, > all with conflicts). > > To get my repo back into a usable state, I ran "hg update --clean" On Win 7, in TortoiseHG Workbench, command line py35% hg update --clean % hg update --clean abort: index 00changelog.i unknown format 2! [command returned code 255 Mon May 19 02:04:15 2014] py35% Same at command prompt in py35 directory. > am waiting for Larry to do his cleans-up. Once his are in, I'll merge my > last 3.4 commit (which should be easy since it has no conflicts). -- Terry Jan Reedy From larry at hastings.org Mon May 19 08:27:31 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 18 May 2014 23:27:31 -0700 Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits, what do I do? In-Reply-To: References: Message-ID: <5379A453.50404@hastings.org> On 05/18/2014 10:31 PM, Raymond Hettinger wrote: > > On May 19, 2014, at 5:52 AM, Terry Reedy > wrote: > >> I stopped at this point and ran diskcheck. I then looked at the DAG >> and noticed 5 previous 3.4 patches that were not merged into 3.5: rev >> 90750 and 90751, Larry Hastings, 2 weeks ago!, rev >> 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 >> hours ago. From earliest to lastest: >> >> c67a19e11a71 >> 7c5f1b200a24 >> 35ea333f43bd >> 31211947387b >> 854fd6eeee2f >> >> If there was any notice on the Committer's list about not making >> commits, I did not get it. In fact, I do not remember getting >> anything on that list for at about a month. Besides that, I had no >> problem 3 days ago, and noticed the commit by Raymond, though not the >> absence of a merge. >> >> In any case, what do I do now, short of deleting and re-cloning, so I >> do not make anything worse? > > I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges, > all with conflicts). > > To get my repo back into a usable state, I ran "hg update --clean" and > am waiting for Larry to do his cleans-up. Once his are in, I'll merge > my last 3.4 commit (which should be easy since it has no conflicts). I think I've got everything merged properly. So cry havoc and let slip the merge of howto/sockets.rst! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon May 19 09:54:28 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 19 May 2014 09:54:28 +0200 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.1 In-Reply-To: <53799E17.7040805@hastings.org> References: <53799E17.7040805@hastings.org> Message-ID: It's not easy to find the changelog. I found this page: https://docs.python.org/3.4/whatsnew/changelog.html Victor 2014-05-19 8:00 GMT+02:00 Larry Hastings : > > > On behalf of the Python development community and the Python 3.4 release > team, I'm pleased to announce the availability of Python 3.4.1. Python > 3.4.1 has over three hundred bugfixes and other improvements over 3.4.0. One > notable change: the version of OpenSSL bundled with the Windows installer no > longer has the "HeartBleed" vulnerability. > > You can download it here: > > https://www.python.org/download/releases/3.4.1 > > > > /arry > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > From larry at hastings.org Mon May 19 08:00:55 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 18 May 2014 23:00:55 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.1 Message-ID: <53799E17.7040805@hastings.org> On behalf of the Python development community and the Python 3.4 release team, I'm pleased to announce the availability of Python 3.4.1. Python 3.4.1 has over three hundred bugfixes and other improvements over 3.4.0. One notable change: the version of OpenSSL bundled with the Windows installer no longer has the "HeartBleed" vulnerability. You can download it here: https://www.python.org/download/releases/3.4.1 //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrvoje.niksic at avl.com Mon May 19 16:20:01 2014 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Mon, 19 May 2014 16:20:01 +0200 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: <537A1311.9010008@avl.com> On 05/17/2014 10:26 AM, Terry Reedy wrote: > When list.pop was added, the convention was changed to > "do not return the 'self' parameter" Do you have a reference for this? It is my understanding that the convention is for mutators to return None, in order to make it clear that the change is destructive. For example, the tutorial at https://docs.python.org/3.4/tutorial/datastructures.html says: """ You might have noticed that methods like insert, remove or sort that modify the list have no return value printed ? they return None. [1] This is a design principle for all mutable data structures in Python. """ Methods like list.pop and dict.pop would seem like a case of "practicality beats purity" because it's more convenient (and faster) to delete and retrieve value at the same go. From tjreedy at udel.edu Tue May 20 21:50:31 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 May 2014 15:50:31 -0400 Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits, what do I do? In-Reply-To: References: Message-ID: On 5/19/2014 2:08 AM, Terry Reedy wrote: > On 5/19/2014 1:31 AM, Raymond Hettinger wrote: >> To get my repo back into a usable state, I ran "hg update --clean" > py35% hg update --clean > % hg update --clean > abort: index 00changelog.i unknown format 2! After exiting Workbench and rebooting, update --clean worked fine. -- Terry Jan Reedy From tjreedy at udel.edu Tue May 20 23:47:06 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 May 2014 17:47:06 -0400 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: <537A1311.9010008@avl.com> References: <537A1311.9010008@avl.com> Message-ID: On 5/19/2014 10:20 AM, Hrvoje Niksic wrote: > On 05/17/2014 10:26 AM, Terry Reedy wrote: > > When list.pop was added, the convention was changed to > > "do not return the 'self' parameter" > > Do you have a reference for this? I think the fact that Guido accepted, in 2000, my 1999 proposal, with generalizations, at Tim Peter's suggestion, and that other pop methods and iterator.__next__ methods have been added since, speaks for itself. However, here is the reference to my original PEP-like post and the subsequent discussion. I consider the subthread about the convention as the most important part of the discussion, as I considered it the most salient objection to the proposal. https://groups.google.com/forum/#!topic/comp.lang.python/SKJq3S2ZYmg Mike Meyer said "While I personally like the idea of a .pop method for lists, it seems to violate one of the design principles that Guido uses: that methods either modify objects, or return values from/about it, but not both." John (Max) Skaller, in his second response, noted that Alex Stepanov used the same principle when designing STL, but the C++ modified his design for convenience and efficiency. In my first response, I suggested that if a convention prevents writing a proper stack, queue, or deque class, all of which are computer science standards, or if a convention prevents pairing a inverse with a function, as is standard in mathematics, then the convention should be re-examined. In Guido's first response he settled the particular question by saying "To implement a stack, one would need to add a list.pop() primitive (and no, I'm not against this particular one on the basis of any principle)." At the time, he was just more inclined "to leave the list data type alone and implement a stack as a class that uses a list for its representation." That covers half the thread. > It is my understanding that the > convention is for mutators to return None, I would say that the convention was to return nothing, which in Python means accepting the default return of None. In a language that allowed procedure methods, there would literally be no return. Python's interactive read-eval-print loop does not print None because it usually represents a non-return. > in order to make it clear that the change is destructive. and in particular, to disallow chaining by not returning 'self'. > For example, the tutorial at > https://docs.python.org/3.4/tutorial/datastructures.html says: > > """ > You might have noticed that methods like insert, remove or sort that > modify the list have no return value printed ? they return None. [1] > This is a design principle for all mutable data structures in Python. > """ Good catch. This only needs 'only' inserted before 'modify' to become true again. I opened http://bugs.python.org/issue21545 Note that the point of the comment is to explain the apparent absence of a return in the r.e.p. loop, the same as in a language with no-return procedures. >>> a.sort() >>> -- Terry Jan Reedy From chris.barker at noaa.gov Tue May 20 18:30:47 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 May 2014 09:30:47 -0700 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: > > >>>> [].sort() is None > > True > >>>> "ABC".lower() is None > > False > > > > That's a deliberate design choice, and one that has been explained a > > few times on the list when folks ask why "[].sort().reverse()" doesn't > > work when "'ABC'.lower().replace('-', '_')" does. > > > > Would it be worth adding such a note? Or is it out of scope? > Is there a reference anywhere as to *why* the convention in Python is to do it that way? Personally, I often miss the ability to chain operations on mutable objects, but I can only imagine that that design decision was made for good reason. However, as I teach Python, I find I have nothing to say other than "that's the way it's done in Python". I note in a way that there is a bit of backwards logic: immutable objects **have** to return something, so they naturally chain. It's almost as if the philosophy of the language is that you don't want chaining behavior, and it's really just a side effect of immutables that you get it with them. I suppose the argument could be that for mutable objects, returning None is an indicator that you are a) working with an mutable object, and b) that the method changes the internal state. But the .pop() example demonstrates that a method can both return something meaningful, and change internal state, so I'm not sure it's really a distinction worth making. So again: is there a clear explanation of the logic behind the convention posted somewhere? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed May 21 00:29:50 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 May 2014 10:29:50 +1200 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: <537BD75E.7090101@canterbury.ac.nz> Chris Barker wrote: > Personally, I often miss the ability to chain operations on mutable > objects, but I can only imagine that that design decision was made for > good reason. However, as I teach Python, I find I have nothing to say > other than "that's the way it's done in Python". Python has better ways of doing many of the things that method chaining is used for in other languages. In Java, for example, I often see it used as a somewhat klunky workaround for the lack of keyword arguments when initialising objects. Other than that, it seems to be mainly for stuffing multiple operations into one line, which is not something the Python style generally goes in for. > I suppose the argument could be that for mutable objects, returning None > is an indicator that you are a) working with an mutable object, and b) > that the method changes the internal state. But the .pop() example > demonstrates that a method can both return something meaningful, and > change internal state, so I'm not sure it's really a distinction worth > making. It's not just about mutating the object, it's about a mutating method with a name that could also plausibly be the name of a non-mutating method. The canonical example is sort(), which, if you didn't already know, could equally well be mutating or non-mutating. In those cases, I think it's worth making it difficult to get wrong. This doesn't apply so much to pop(), which sounds much more like a mutating method than a non-mutating one, so it's less likely you'll make a mistake. -- Greg From rdmurray at bitdance.com Wed May 21 01:50:14 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 20 May 2014 19:50:14 -0400 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: <20140520235015.813F6250D4D@webabinitio.net> On Tue, 20 May 2014 09:30:47 -0700, Chris Barker wrote: > > > > >>>> [].sort() is None > > > True > > >>>> "ABC".lower() is None > > > False > > > > > > That's a deliberate design choice, and one that has been explained a > > > few times on the list when folks ask why "[].sort().reverse()" doesn't > > > work when "'ABC'.lower().replace('-', '_')" does. > > > > > > Would it be worth adding such a note? Or is it out of scope? > > > > Is there a reference anywhere as to *why* the convention in Python is to do > it that way? > > Personally, I often miss the ability to chain operations on mutable > objects, but I can only imagine that that design decision was made for good > reason. However, as I teach Python, I find I have nothing to say other than > "that's the way it's done in Python". > > I note in a way that there is a bit of backwards logic: > > immutable objects **have** to return something, so they naturally chain. > It's almost as if the philosophy of the language is that you don't want > chaining behavior, and it's really just a side effect of immutables that > you get it with them. > > I suppose the argument could be that for mutable objects, returning None is > an indicator that you are a) working with an mutable object, and b) that > the method changes the internal state. But the .pop() example demonstrates > that a method can both return something meaningful, and change internal > state, so I'm not sure it's really a distinction worth making. > > So again: is there a clear explanation of the logic behind the convention > posted somewhere? I think it is exactly about not encouraging chaining of mutations, but I could be wrong :) --David From tjreedy at udel.edu Wed May 21 01:56:37 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 May 2014 19:56:37 -0400 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: On 5/20/2014 12:30 PM, Chris Barker wrote: > >>>> [].sort() is None > > True > >>>> "ABC".lower() is None > > False > Is there a reference anywhere as to *why* the convention in Python is to > do it that way? In short, reducing bugs induced by mutation of aliased objects. Functional languages evade the problem by prohibiting mutation (sometimes at the cost of inefficiency). In an alternate universe, the example above might become >>> a = []; a.sort() is a True >>> a = "ABC"' a.lower() is a False As I suggested earlier, having pure mutation methods not return anything made is easy to suggest a mutation + non-self return method, list.pop. If all mutation methods had previously returned 'self', there might have been disagreement over whether the item return should augment or replace the self return. Before you say the latter, consider the inconsistency of only sometimes returning self and the potential consistency between >>> most, last = 'a b c'.rsplit(maxsplit=1) >>> most, last ('a b', 'c') >>> most, last = [0, 1, 2].pop() >>> most, last ([0, 1], 2) One could also consider first, rest pairings. -- Terry Jan Reedy From guido at python.org Wed May 21 03:40:10 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 20 May 2014 18:40:10 -0700 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: <537BD75E.7090101@canterbury.ac.nz> References: <537BD75E.7090101@canterbury.ac.nz> Message-ID: On Tuesday, May 20, 2014, Greg Ewing wrote: > Chris Barker wrote: > >> Personally, I often miss the ability to chain operations on mutable >> objects, but I can only imagine that that design decision was made for good >> reason. However, as I teach Python, I find I have nothing to say other than >> "that's the way it's done in Python". >> > > Python has better ways of doing many of the things that > method chaining is used for in other languages. In Java, > for example, I often see it used as a somewhat klunky > workaround for the lack of keyword arguments when > initialising objects. > > Other than that, it seems to be mainly for stuffing > multiple operations into one line, which is not something > the Python style generally goes in for. > > I suppose the argument could be that for mutable objects, returning None >> is an indicator that you are a) working with an mutable object, and b) that >> the method changes the internal state. But the .pop() example demonstrates >> that a method can both return something meaningful, and change internal >> state, so I'm not sure it's really a distinction worth making. >> > > It's not just about mutating the object, it's about a > mutating method with a name that could also plausibly > be the name of a non-mutating method. The canonical > example is sort(), which, if you didn't already know, > could equally well be mutating or non-mutating. In > those cases, I think it's worth making it difficult > to get wrong. This is the key reason, by far the most important. > This doesn't apply so much to pop(), which sounds > much more like a mutating method than a non-mutating > one, so it's less likely you'll make a mistake. > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 22 01:33:54 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 May 2014 16:33:54 -0700 Subject: [Python-Dev] Returning None from methods that mutate object state In-Reply-To: References: Message-ID: Thanks all, Now I need to try to sum this all up to present to my students. ;-) -Chris On Tue, May 20, 2014 at 4:56 PM, Terry Reedy wrote: > On 5/20/2014 12:30 PM, Chris Barker wrote: > >> >>>> [].sort() is None >> > True >> >>>> "ABC".lower() is None >> > False >> > > Is there a reference anywhere as to *why* the convention in Python is to >> do it that way? >> > > In short, reducing bugs induced by mutation of aliased objects. Functional > languages evade the problem by prohibiting mutation (sometimes at the cost > of inefficiency). > > In an alternate universe, the example above might become > > >>> a = []; a.sort() is a > True > >>> a = "ABC"' a.lower() is a > False > > As I suggested earlier, having pure mutation methods not return anything > made is easy to suggest a mutation + non-self return method, list.pop. If > all mutation methods had previously returned 'self', there might have been > disagreement over whether the item return should augment or replace the > self return. Before you say the latter, consider the inconsistency of only > sometimes returning self and the potential consistency between > > >>> most, last = 'a b c'.rsplit(maxsplit=1) > >>> most, last > ('a b', 'c') > > >>> most, last = [0, 1, 2].pop() > >>> most, last > ([0, 1], 2) > > One could also consider first, rest pairings. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > chris.barker%40noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chandra.Srinivasan at synopsys.com Fri May 23 17:49:31 2014 From: Chandra.Srinivasan at synopsys.com (Chandra Srinivasan) Date: Fri, 23 May 2014 15:49:31 +0000 Subject: [Python-Dev] Unexpected increase of globals() refcount Message-ID: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com> Hi, I ran the following code in the Python interpreter and am trying to determine if the behavior I see is expected: import sys print sys.getrefcount(globals()) class Foo(object): def __init__(self): pass print sys.getrefcount(globals()) The first print statement above prints '4' and the second one prints '5'. However, if I remove the __init__ method from the class, the refcount stays the same. If I change the above code like this, the ref count stays the same: import gc import sys print sys.getrefcount(globals()) class Foo(object): def __init__(self): pass del Foo while gc.collect(): pass print sys.getrefcount(globals()) Can you let me know if this is a bug in the Python interpreter? -Chandra -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri May 23 18:03:03 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 23 May 2014 09:03:03 -0700 Subject: [Python-Dev] Unexpected increase of globals() refcount In-Reply-To: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com> References: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com> Message-ID: <1400860983.12094.120843053.24A2D612@webmail.messagingengine.com> On Fri, May 23, 2014, at 8:49, Chandra Srinivasan wrote: > Hi, > I ran the following code in the Python interpreter and am trying to > determine if the behavior I see is expected: > > import sys > print sys.getrefcount(globals()) > class Foo(object): > def __init__(self): > pass > print sys.getrefcount(globals()) > > The first print statement above prints '4' and the second one prints '5'. > However, if I remove the __init__ method from the class, the refcount > stays the same. > > If I change the above code like this, the ref count stays the same: > > import gc > import sys > print sys.getrefcount(globals()) > class Foo(object): > def __init__(self): > pass > del Foo > while gc.collect(): > pass > print sys.getrefcount(globals()) > > Can you let me know if this is a bug in the Python interpreter? No, functions hold a reference to the globals of the module they are defined in. From status at bugs.python.org Fri May 23 18:07:56 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 23 May 2014 18:07:56 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140523160756.50AD656A31@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-05-16 - 2014-05-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4622 ( +8) closed 28710 (+36) total 33332 (+44) Open issues with patches: 2129 Issues opened (26) ================== #21518: Expose RegUnloadKey in winreg http://bugs.python.org/issue21518 opened by Claudiu.Popa #21519: IDLE : Bug in keybinding validity check http://bugs.python.org/issue21519 opened by sahutd #21520: Erroneous zipfile test failure if the string 'bad' appears in http://bugs.python.org/issue21520 opened by larry #21521: Tkinter + OSX + Spaces : Multiple file dialogues created http://bugs.python.org/issue21521 opened by rovf #21524: Allowing to pass pathlib.Path object in mimetypes.guess_type f http://bugs.python.org/issue21524 opened by jorispilot #21526: Support new booleans in Tkinter http://bugs.python.org/issue21526 opened by serhiy.storchaka #21527: concurrent.futures.ThreadPoolExecutor does not use a default v http://bugs.python.org/issue21527 opened by Claudiu.Popa #21531: Sending a zero-length UDP packet to asyncore invokes handle_cl http://bugs.python.org/issue21531 opened by Tony.Gedge #21532: 2.7.7rc1 msi is lacking libpython27.a http://bugs.python.org/issue21532 opened by loewis #21533: built-in types dict docs - construct dict from iterable, not i http://bugs.python.org/issue21533 opened by wolma #21534: 404 on documentation download links http://bugs.python.org/issue21534 opened by zach.ware #21536: extension built with a shared python cannot be loaded with a s http://bugs.python.org/issue21536 opened by pitrou #21539: pathlib's Path.mkdir() should allow for "mkdir -p" functionali http://bugs.python.org/issue21539 opened by garrison #21541: Provide configure option --with-ssl for compilation with custo http://bugs.python.org/issue21541 opened by machyniak #21546: int('\0') gives wrong error message http://bugs.python.org/issue21546 opened by Kurt.Rose #21547: '!s' formatting documentation bug http://bugs.python.org/issue21547 opened by Joshua.Landau #21548: pydoc -k IndexError on empty docstring http://bugs.python.org/issue21548 opened by Dima.Tisnek #21549: Add the members parameter for TarFile.list() http://bugs.python.org/issue21549 opened by serhiy.storchaka #21550: Add Python implementation of the tar utility http://bugs.python.org/issue21550 opened by serhiy.storchaka #21552: String length overflow in Tkinter http://bugs.python.org/issue21552 opened by serhiy.storchaka #21555: gcmodule.c could use pytime.h http://bugs.python.org/issue21555 opened by pitrou #21556: try to use hashtable in pickle http://bugs.python.org/issue21556 opened by neologix #21557: os.popen & os.system lack shell-related security warnings http://bugs.python.org/issue21557 opened by cvrebert #21558: Fix a typo in the contextlib docs http://bugs.python.org/issue21558 opened by berker.peksag #21559: OverflowError should not happen for integer operations http://bugs.python.org/issue21559 opened by theme #21560: gzip.write changes trailer ISIZE field before type checking - http://bugs.python.org/issue21560 opened by wolma Most recent 15 issues with no replies (15) ========================================== #21559: OverflowError should not happen for integer operations http://bugs.python.org/issue21559 #21558: Fix a typo in the contextlib docs http://bugs.python.org/issue21558 #21557: os.popen & os.system lack shell-related security warnings http://bugs.python.org/issue21557 #21552: String length overflow in Tkinter http://bugs.python.org/issue21552 #21548: pydoc -k IndexError on empty docstring http://bugs.python.org/issue21548 #21533: built-in types dict docs - construct dict from iterable, not i http://bugs.python.org/issue21533 #21526: Support new booleans in Tkinter http://bugs.python.org/issue21526 #21524: Allowing to pass pathlib.Path object in mimetypes.guess_type f http://bugs.python.org/issue21524 #21521: Tkinter + OSX + Spaces : Multiple file dialogues created http://bugs.python.org/issue21521 #21514: update json module docs in light of RFC 7159 & ECMA-404 http://bugs.python.org/issue21514 #21505: cx_freeze multiprocessing bug http://bugs.python.org/issue21505 #21502: freeze.py not working properly with OS X framework builds http://bugs.python.org/issue21502 #21500: Make use of the "load_tests" protocol in test_importlib packag http://bugs.python.org/issue21500 #21498: configparser accepts keys beginning with comment_chars when wr http://bugs.python.org/issue21498 #21493: Add test for ntpath.expanduser http://bugs.python.org/issue21493 Most recent 15 issues waiting for review (15) ============================================= #21560: gzip.write changes trailer ISIZE field before type checking - http://bugs.python.org/issue21560 #21558: Fix a typo in the contextlib docs http://bugs.python.org/issue21558 #21556: try to use hashtable in pickle http://bugs.python.org/issue21556 #21555: gcmodule.c could use pytime.h http://bugs.python.org/issue21555 #21552: String length overflow in Tkinter http://bugs.python.org/issue21552 #21549: Add the members parameter for TarFile.list() http://bugs.python.org/issue21549 #21541: Provide configure option --with-ssl for compilation with custo http://bugs.python.org/issue21541 #21539: pathlib's Path.mkdir() should allow for "mkdir -p" functionali http://bugs.python.org/issue21539 #21533: built-in types dict docs - construct dict from iterable, not i http://bugs.python.org/issue21533 #21527: concurrent.futures.ThreadPoolExecutor does not use a default v http://bugs.python.org/issue21527 #21526: Support new booleans in Tkinter http://bugs.python.org/issue21526 #21519: IDLE : Bug in keybinding validity check http://bugs.python.org/issue21519 #21518: Expose RegUnloadKey in winreg http://bugs.python.org/issue21518 #21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile? http://bugs.python.org/issue21515 #21513: speed up some ipaddress properties http://bugs.python.org/issue21513 Top 10 most discussed issues (10) ================================= #14776: Add SystemTap static markers http://bugs.python.org/issue14776 8 msgs #20584: On FreeBSD, signal.NSIG is smaller than biggest signal value http://bugs.python.org/issue20584 7 msgs #21516: pathlib.Path(...).is_dir() crashes on some directories (Window http://bugs.python.org/issue21516 6 msgs #21556: try to use hashtable in pickle http://bugs.python.org/issue21556 6 msgs #21304: PEP 466: Backport hashlib.pbkdf2_hmac to Python 2.7 http://bugs.python.org/issue21304 5 msgs #20611: socket.create_connection() doesn't handle EINTR properly http://bugs.python.org/issue20611 4 msgs #21145: Add the @cached_property decorator http://bugs.python.org/issue21145 4 msgs #21511: Thinko in Lib/quopri.py http://bugs.python.org/issue21511 4 msgs #21518: Expose RegUnloadKey in winreg http://bugs.python.org/issue21518 4 msgs #21560: gzip.write changes trailer ISIZE field before type checking - http://bugs.python.org/issue21560 4 msgs Issues closed (35) ================== #7776: http.client.HTTPConnection tunneling is broken http://bugs.python.org/issue7776 closed by larry #9673: Entry Widget Not Editable under Windows XP http://bugs.python.org/issue9673 closed by terry.reedy #10744: ctypes arrays have incorrect buffer information (PEP-3118) http://bugs.python.org/issue10744 closed by python-dev #19866: tests aifc, sunau and wave failures on a fresh Win64 installat http://bugs.python.org/issue19866 closed by python-dev #20620: Update the min()/max() docs for the new default argument http://bugs.python.org/issue20620 closed by rhettinger #20635: Fix the grid geometry manager and add tests for geometry manag http://bugs.python.org/issue20635 closed by serhiy.storchaka #21027: difflib new cli interface http://bugs.python.org/issue21027 closed by rhettinger #21198: Minor tarfile documentation bug http://bugs.python.org/issue21198 closed by rhettinger #21213: Memory bomb by incorrect custom serializer to json.dumps http://bugs.python.org/issue21213 closed by rhettinger #21362: concurrent.futures does not validate that max_workers is prope http://bugs.python.org/issue21362 closed by bquinlan #21430: Document ssl.pending() http://bugs.python.org/issue21430 closed by pitrou #21455: add default backlog to socket.listen() http://bugs.python.org/issue21455 closed by neologix #21479: Document TarFile.open() as a classmethod http://bugs.python.org/issue21479 closed by rhettinger #21484: More clarity needed about difference between "x += e" and "x = http://bugs.python.org/issue21484 closed by rhettinger #21495: Sane default for logging config http://bugs.python.org/issue21495 closed by terry.reedy #21503: Use test_both() consistently throughout test_importlib http://bugs.python.org/issue21503 closed by eric.snow #21507: memory used by frozenset created from set differs from that of http://bugs.python.org/issue21507 closed by rhettinger #21517: installer Python default setting fails with mac Python Launche http://bugs.python.org/issue21517 closed by ned.deily #21522: Add more tkinter tests http://bugs.python.org/issue21522 closed by serhiy.storchaka #21523: quadratic-time compilation in the number of 'and' or 'or' expr http://bugs.python.org/issue21523 closed by pitrou #21525: Accept lists in Tkinter http://bugs.python.org/issue21525 closed by serhiy.storchaka #21528: Fix a number of typos in the documentation http://bugs.python.org/issue21528 closed by dstufft #21529: JSON module: reading arbitrary process memory http://bugs.python.org/issue21529 closed by benjamin.peterson #21530: Integer overflow in strop http://bugs.python.org/issue21530 closed by benjamin.peterson #21535: test_license_exists_at_url fails with 3.4.1, wrong/unexpected http://bugs.python.org/issue21535 closed by ned.deily #21537: functools.lru_cache does not cache exceptions http://bugs.python.org/issue21537 closed by rhettinger #21538: plistlib unable to load iOS7 Safari History.plist http://bugs.python.org/issue21538 closed by slo.sleuth #21540: PEP 8 should recommend "is not" and "not in" http://bugs.python.org/issue21540 closed by barry #21542: pprint doesn't work well for counters, sometimes shows them li http://bugs.python.org/issue21542 closed by rhettinger #21543: json library fails to serialize objects such as datetime http://bugs.python.org/issue21543 closed by r.david.murray #21544: Spam http://bugs.python.org/issue21544 closed by berker.peksag #21545: Tutorial: examples and comment about mutation methods http://bugs.python.org/issue21545 closed by terry.reedy #21551: Behavior of word boundaries in regexes unexpected http://bugs.python.org/issue21551 closed by r.david.murray #21553: Behaviour of modules depends on how they where imported http://bugs.python.org/issue21553 closed by eric.snow #21554: Possible Error in "Brief Tour of the Standard Library" http://bugs.python.org/issue21554 closed by rhettinger From herbertg at gmx.at Sat May 24 11:55:23 2014 From: herbertg at gmx.at (Herbert Griebel) Date: Sat, 24 May 2014 11:55:23 +0200 Subject: [Python-Dev] [RELEASED] Python 3.4.1 Message-ID: An HTML attachment was scrubbed... URL: From herbertg at gmx.at Sat May 24 12:15:37 2014 From: herbertg at gmx.at (Herbert Griebel) Date: Sat, 24 May 2014 12:15:37 +0200 Subject: [Python-Dev] [RELEASED] Python 3.4.1 In-Reply-To: <53799E17.7040805@hastings.org> References: <53799E17.7040805@hastings.org> Message-ID: <53807149.5090801@gmx.at> Found the issue. To reproduce the problem install Python 3.4.0, then rename the install folder (e.g. C:\Python34 to C:\Python34x) and install Python 3.4.1. From martin at v.loewis.de Sat May 24 19:23:47 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 May 2014 19:23:47 +0200 Subject: [Python-Dev] [RELEASED] Python 3.4.1 In-Reply-To: <53807149.5090801@gmx.at> References: <53799E17.7040805@hastings.org> <53807149.5090801@gmx.at> Message-ID: <5380D5A3.6060806@v.loewis.de> Am 24.05.14 12:15, schrieb Herbert Griebel: > Found the issue. To reproduce the problem install Python 3.4.0, then > rename the install folder (e.g. C:\Python34 to C:\Python34x) and install > Python 3.4.1. Please understand that installation of 3.4.1 attempts uninstallation of 3.4 first. Without testing, my guess is that it is the pip uninstallation which fails, due to python.exe not being there anymore. If you want to diagnose this further, run the installer with msiexec /i python-3.4.1.msi /l*v python.log Either study the log file yourself, or post it to bugs.python.org. Regards, Martin From Nikolaus at rath.org Sat May 24 21:35:42 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sat, 24 May 2014 12:35:42 -0700 Subject: [Python-Dev] Commit-ready patches in need of review Message-ID: <8738fzgeb5.fsf@vostro.rath.org> Hello, While my last appeal resulted in quite some commits (thanks!), I still have some more commit-ready patches waiting for review. It'd be great if some people could find time to take a look: * http://bugs.python.org/issue1738 (filecmp.dircmp does exact match only) Reviewed patch, rebased on current hg tip * http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to Argument Clinic across 2 files) I only wrote the patch for one file because I'd like to have feedback before tackling the second. However, the patches are independent so unless there are other problems this is ready for commit. * http://bugs.python.org/issue15955 (gzip, bz2, lzma: add option to limit output size) * http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is missing) Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From storchaka at gmail.com Tue May 27 10:58:51 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 May 2014 11:58:51 +0300 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: <3gcW0S2PBfz7LjV@mail.python.org> References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: 26.05.14 10:59, raymond.hettinger ???????(??): > + result = [(elem, i) for i, elem in zip(range(n), it)] Perhaps it is worth to add simple comment explaining why this is not equivalent to just list(zip(it, range(n))). Otherwise it can be unintentionally "optimized" in future. From rosuav at gmail.com Tue May 27 11:05:56 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 May 2014 19:05:56 +1000 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka wrote: > 26.05.14 10:59, raymond.hettinger ???????(??): >> >> + result = [(elem, i) for i, elem in zip(range(n), it)] > > > Perhaps it is worth to add simple comment explaining why this is not > equivalent to just list(zip(it, range(n))). Otherwise it can be > unintentionally "optimized" in future. > Where is the difference? I'm very much puzzled now. My first thought was based on differing-length iterables in zip, but the docs say it stops at the shortest of its args. ChrisA From murman at gmail.com Tue May 27 13:45:01 2014 From: murman at gmail.com (Michael Urman) Date: Tue, 27 May 2014 06:45:01 -0500 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: On Tue, May 27, 2014 at 4:05 AM, Chris Angelico wrote: > On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka wrote: >> 26.05.14 10:59, raymond.hettinger ???????(??): >>> >>> + result = [(elem, i) for i, elem in zip(range(n), it)] >> >> >> Perhaps it is worth to add simple comment explaining why this is not >> equivalent to just list(zip(it, range(n))). Otherwise it can be >> unintentionally "optimized" in future. >> > > Where is the difference? I'm very much puzzled now. My first thought > was based on differing-length iterables in zip, but the docs say it > stops at the shortest of its args. Due to how zip stops, it leaves the longer iterable in different places: >>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it) [(0, 'a'), (1, 'b'), (2, 'c')] 'd' >>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it) [('a', 0), ('b', 1), ('c', 2)] 'e' This seems like a potentially nasty gotcha, but I'm unclear what real use cases would be impacted. Michael From taleinat at gmail.com Tue May 27 14:44:33 2014 From: taleinat at gmail.com (Tal Einat) Date: Tue, 27 May 2014 15:44:33 +0300 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: On Tue, May 27, 2014 at 2:45 PM, Michael Urman wrote: > On Tue, May 27, 2014 at 4:05 AM, Chris Angelico wrote: >> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka wrote: >>> 26.05.14 10:59, raymond.hettinger ???????(??): >>>> >>>> + result = [(elem, i) for i, elem in zip(range(n), it)] >>> >>> >>> Perhaps it is worth to add simple comment explaining why this is not >>> equivalent to just list(zip(it, range(n))). Otherwise it can be >>> unintentionally "optimized" in future. >>> >> >> Where is the difference? I'm very much puzzled now. My first thought >> was based on differing-length iterables in zip, but the docs say it >> stops at the shortest of its args. > > Due to how zip stops, it leaves the longer iterable in different places: > >>>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it) > [(0, 'a'), (1, 'b'), (2, 'c')] > 'd' >>>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it) > [('a', 0), ('b', 1), ('c', 2)] > 'e' > > This seems like a potentially nasty gotcha, but I'm unclear what real > use cases would be impacted. If that's the issue, a comment explaining it should definitely be added. We could also use islice(enumerate(it), n)) to avoid the confusion. - Tal From j.wielicki at sotecware.net Tue May 27 12:01:35 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Tue, 27 May 2014 12:01:35 +0200 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: <5384627F.7090508@sotecware.net> On 27.05.2014 11:05, Chris Angelico wrote: > On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka wrote: >> 26.05.14 10:59, raymond.hettinger ???????(??): >>> >>> + result = [(elem, i) for i, elem in zip(range(n), it)] >> >> >> Perhaps it is worth to add simple comment explaining why this is not >> equivalent to just list(zip(it, range(n))). Otherwise it can be >> unintentionally "optimized" in future. >> > > Where is the difference? I'm very much puzzled now. My first thought > was based on differing-length iterables in zip, but the docs say it > stops at the shortest of its args. Nice puzzle. (elem, i) is not (i, elem), though. Thats really hard to see when not looking at it closely. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/j.wielicki%40sotecware.net > From raymond.hettinger at gmail.com Tue May 27 23:54:35 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 27 May 2014 14:54:35 -0700 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: <25554AEF-3E8C-4D91-9945-BB0705E79CCC@gmail.com> On May 27, 2014, at 1:58 AM, Serhiy Storchaka wrote: > Perhaps it is worth to add simple comment explaining why this is not equivalent to just list(zip(it, range(n))). Otherwise it can be unintentionally "optimized" in future. FWIW, that is covered by the test cases. If you substitute list(zip(it, range(n))), the tests fail. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed May 28 01:23:55 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 28 May 2014 00:23:55 +0100 Subject: [Python-Dev] cpython: Minor clean-ups for heapq. In-Reply-To: References: <3gcW0S2PBfz7LjV@mail.python.org> Message-ID: <53851E8B.2060600@btinternet.com> On 27/05/2014 13:44, Tal Einat wrote: > On Tue, May 27, 2014 at 2:45 PM, Michael Urman wrote: >> On Tue, May 27, 2014 at 4:05 AM, Chris Angelico wrote: >>> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka wrote: >>>> 26.05.14 10:59, raymond.hettinger ???????(??): >>>>> + result = [(elem, i) for i, elem in zip(range(n), it)] >>>> >>>> Perhaps it is worth to add simple comment explaining why this is not >>>> equivalent to just list(zip(it, range(n))). Otherwise it can be >>>> unintentionally "optimized" in future. >>>> >>> Where is the difference? I'm very much puzzled now. My first thought >>> was based on differing-length iterables in zip, but the docs say it >>> stops at the shortest of its args. >> Due to how zip stops, it leaves the longer iterable in different places: >> >>>>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it) >> [(0, 'a'), (1, 'b'), (2, 'c')] >> 'd' >>>>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it) >> [('a', 0), ('b', 1), ('c', 2)] >> 'e' >> >> This seems like a potentially nasty gotcha, but I'm unclear what real >> use cases would be impacted. > If that's the issue, a comment explaining it should definitely be added. +1. FWIW I couldn't see the difference between the 2 codings. I was relieved to see that the zip() doc says "The left-to-right evaluation order of the iterables is guaranteed." Even so, I don't think the reliance on this is obvious. Rob Cliffe > > We could also use islice(enumerate(it), n)) to avoid the confusion. > > - Tal > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > > ----- > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2014.0.4592 / Virus Database: 3950/7568 - Release Date: 05/26/14 From michael.haubenwallner at ssi-schaefer.com Wed May 28 16:51:19 2014 From: michael.haubenwallner at ssi-schaefer.com (Michael Haubenwallner) Date: Wed, 28 May 2014 16:51:19 +0200 Subject: [Python-Dev] use cases for "python-config" versus "pkg-config python" Message-ID: <5385F7E7.9090408@ssi-schaefer.com> Hello! Stumbling over problems on AIX (Modules/python.exp not found) building libxml2 as python module let me wonder about the intended use-cases for 'python-config' and 'pkg-config python'. FWIW, I can see these distinct use cases here, and I'm kindly asking if I got them right: * Build an application containing a python interpreter (like python$EXE itself): + link against libpython.so + re-export symbols from libpython.so for python-modules (platform-specific) + This is similar to build against any other library, thus = 'python.pc' is installed (for 'pkg-config python'). * Build a python-module (like build/lib.-/*.so): + no need to link against libpython.so, instead + expect symbols from libpython.so to be available at runtime, platform-specific either as + undefined symbols at build-time (Linux, others), or + a list of symbols to import from "the main executable" (AIX) + This is specific to python-modules, thus = 'python-config' is installed. Thank you! /haubi/ From brian at python.org Wed May 28 18:49:52 2014 From: brian at python.org (Brian Curtin) Date: Wed, 28 May 2014 11:49:52 -0500 Subject: [Python-Dev] Download Counts (was: Language Summit notes) Message-ID: On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou wrote: > I don't think we have recent download numbers since the Website > overhaul (do we?), but Python 3 isn't an "experimental concept > language" anymore (it hasn't been since 3.3 or 3.2, I'd say). Using the old logs, which are still good through 2013, I've found the following: The first year of a release series (month of final release month + 12mos): 2.6.x - 10.3 Million 2.7.x - 10.26M 3.2.x - 5.84M 3.3.x - 13.1M 2013 downloads (out of 34.79M across all possible versions): 2.6.x - 1.9M 2.7.x - 14.3M 3.2.x - 1.03M 3.3.x - 13.85M 3.3 had a big first year of availability (Oct '12-'13), and throughout 2013 it represented 48% of those versions listed above. From brian at python.org Wed May 28 19:57:49 2014 From: brian at python.org (Brian Curtin) Date: Wed, 28 May 2014 12:57:49 -0500 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: On May 28, 2014 12:49 PM, "Brian Curtin" wrote: > > On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou wrote: > > I don't think we have recent download numbers since the Website > > overhaul (do we?), but Python 3 isn't an "experimental concept > > language" anymore (it hasn't been since 3.3 or 3.2, I'd say). > > Using the old logs, which are still good through 2013, I've found the following: > > The first year of a release series (month of final release month + 12mos): > 2.6.x - 10.3 Million > 2.7.x - 10.26M > 3.2.x - 5.84M > 3.3.x - 13.1M > > 2013 downloads (out of 34.79M across all possible versions): > 2.6.x - 1.9M > 2.7.x - 14.3M > 3.2.x - 1.03M > 3.3.x - 13.85M > > 3.3 had a big first year of availability (Oct '12-'13), and throughout > 2013 it represented 48% of those versions listed above. Sorry for not being explicit: these are download counts for Windows installers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 28 20:10:00 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 28 May 2014 11:10:00 -0700 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: Is the Windows/Mac ratio still 70/30, with Linux in the single digits? On Wed, May 28, 2014 at 10:57 AM, Brian Curtin wrote: > On May 28, 2014 12:49 PM, "Brian Curtin" wrote: > > > > On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou > wrote: > > > I don't think we have recent download numbers since the Website > > > overhaul (do we?), but Python 3 isn't an "experimental concept > > > language" anymore (it hasn't been since 3.3 or 3.2, I'd say). > > > > Using the old logs, which are still good through 2013, I've found the > following: > > > > The first year of a release series (month of final release month + > 12mos): > > 2.6.x - 10.3 Million > > 2.7.x - 10.26M > > 3.2.x - 5.84M > > 3.3.x - 13.1M > > > > 2013 downloads (out of 34.79M across all possible versions): > > 2.6.x - 1.9M > > 2.7.x - 14.3M > > 3.2.x - 1.03M > > 3.3.x - 13.85M > > > > 3.3 had a big first year of availability (Oct '12-'13), and throughout > > 2013 it represented 48% of those versions listed above. > > Sorry for not being explicit: these are download counts for Windows > installers. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Wed May 28 22:05:48 2014 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 28 May 2014 13:05:48 -0700 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: On Wed, May 28, 2014 at 11:10 AM, Guido van Rossum wrote: > Is the Windows/Mac ratio still 70/30, with Linux in the single digits? > > Most Linux installs go through package managers which don't count here, no? Eli > > On Wed, May 28, 2014 at 10:57 AM, Brian Curtin wrote: > >> On May 28, 2014 12:49 PM, "Brian Curtin" wrote: >> > >> > On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou >> wrote: >> > > I don't think we have recent download numbers since the Website >> > > overhaul (do we?), but Python 3 isn't an "experimental concept >> > > language" anymore (it hasn't been since 3.3 or 3.2, I'd say). >> > >> > Using the old logs, which are still good through 2013, I've found the >> following: >> > >> > The first year of a release series (month of final release month + >> 12mos): >> > 2.6.x - 10.3 Million >> > 2.7.x - 10.26M >> > 3.2.x - 5.84M >> > 3.3.x - 13.1M >> > >> > 2013 downloads (out of 34.79M across all possible versions): >> > 2.6.x - 1.9M >> > 2.7.x - 14.3M >> > 3.2.x - 1.03M >> > 3.3.x - 13.85M >> > >> > 3.3 had a big first year of availability (Oct '12-'13), and throughout >> > 2013 it represented 48% of those versions listed above. >> >> Sorry for not being explicit: these are download counts for Windows >> installers. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Wed May 28 23:02:58 2014 From: brian at python.org (Brian Curtin) Date: Wed, 28 May 2014 16:02:58 -0500 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: On May 28, 2014 4:06 PM, "Eli Bendersky" wrote: > > > > > On Wed, May 28, 2014 at 11:10 AM, Guido van Rossum wrote: >> >> Is the Windows/Mac ratio still 70/30, with Linux in the single digits? >> > > Most Linux installs go through package managers which don't count here, no? I'll have to run something for the other non-Windows files, but the single digit Linux downloads he meant are the tarballs. We'll (probably) never know the true counts in the Linux world because of how pervasive Python is within basically every distro, but that's also likely the case on Mac. With Windows, since you must download Python it to use it, the numbers we see are probably the most useful on their own. I'm giving a talk at PyCon Russia that covers some of these numbers, so I'll probably try to dig up more and turn it into a blog post. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed May 28 23:27:59 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 May 2014 23:27:59 +0200 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: 2014-05-28 22:05 GMT+02:00 Eli Bendersky : > Most Linux installs go through package managers which don't count here, no? For Debian, there is the "popcorn" project which provides some statistics: http://qa.debian.org/popcon.php?package=python2.6 http://qa.debian.org/popcon.php?package=python2.7 http://qa.debian.org/popcon.php?package=python3.2 http://qa.debian.org/popcon.php?package=python3.3 http://qa.debian.org/popcon.php?package=python3.4 It looks like python2.6 is installed more often than python2.7 ! (if you look at the "Inst" column, not in the "Recent" column.) Python 3.4 recently became the default python3 package on Debian Unstable ("sid"). Debian Stable (Wheezy) still uses Python 3.2. I guess that Debian Testing uses Python 3.3. Victor From rosuav at gmail.com Wed May 28 23:37:57 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 29 May 2014 07:37:57 +1000 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: On Thu, May 29, 2014 at 7:27 AM, Victor Stinner wrote: > 2014-05-28 22:05 GMT+02:00 Eli Bendersky : >> Most Linux installs go through package managers which don't count here, no? > > For Debian, there is the "popcorn" project which provides some statistics: > > http://qa.debian.org/popcon.php?package=python2.6 > http://qa.debian.org/popcon.php?package=python2.7 > http://qa.debian.org/popcon.php?package=python3.2 > http://qa.debian.org/popcon.php?package=python3.3 > http://qa.debian.org/popcon.php?package=python3.4 > > It looks like python2.6 is installed more often than python2.7 ! (if > you look at the "Inst" column, not in the "Recent" column.) > > Python 3.4 recently became the default python3 package on Debian > Unstable ("sid"). Debian Stable (Wheezy) still uses Python 3.2. I > guess that Debian Testing uses Python 3.3. 2.6 is the default in oldstable (Squeeze), so every Debian system that hasn't upgraded from there is likely to be counted as a 2.6 and not a 2.7. (And Python 2 is a dependency of all sorts of things, so it's likely to get installed.) Wheezy ships 2.6.8, and I seem to have both that and 2.7.3 installed, so it's possible that even though 2.7 is the default, a lot of 2.6 installations are happening. Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3' package pulling in 3.3. That may change before Jessie becomes stable, I don't know. ChrisA From barry at python.org Thu May 29 01:05:50 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 28 May 2014 19:05:50 -0400 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: References: Message-ID: <20140528190550.096f16ee@limelight.wooz.org> On May 29, 2014, at 07:37 AM, Chris Angelico wrote: >Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3' >package pulling in 3.3. That may change before Jessie becomes stable, >I don't know. It already has: https://lists.debian.org/debian-python/2014/05/msg00162.html The question remains whether Jessie will drop Python 3.3, but I expect it to. Cheers, -Barry From rosuav at gmail.com Thu May 29 01:37:44 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 29 May 2014 09:37:44 +1000 Subject: [Python-Dev] Download Counts (was: Language Summit notes) In-Reply-To: <20140528190550.096f16ee@limelight.wooz.org> References: <20140528190550.096f16ee@limelight.wooz.org> Message-ID: On Thu, May 29, 2014 at 9:05 AM, Barry Warsaw wrote: > On May 29, 2014, at 07:37 AM, Chris Angelico wrote: > >>Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3' >>package pulling in 3.3. That may change before Jessie becomes stable, >>I don't know. > > It already has: > > https://lists.debian.org/debian-python/2014/05/msg00162.html > > The question remains whether Jessie will drop Python 3.3, but I expect it to. Well, there you go! :) I was almost going to say that my information was up to 24 hours out of date, because I hadn't run daily updates yet, and it turns out it was exactly that wrong :) ChrisA From glyph at twistedmatrix.com Thu May 29 00:26:38 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 28 May 2014 15:26:38 -0700 Subject: [Python-Dev] Language Summit Follow-Up Message-ID: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> At the language summit, Alex and I volunteered to put together some recommendations on what changes could be made to Python (the language) in order to facilitate a smoother transition from Python 2 to Python 3. One of the things that motivated this was the (surprising, to us) consideration that features like ensurepip might be added to the future versions of the 2.7 installers from python.org. The specific motivations for writing this are: Library maintainers have a rapidly expanding matrix that requires an increasing number of branches to satisfy. People with large corporate codebases absolutely cannot port all at once. If you don't have perfect test coverage then you can't make any progress. So these changes are intended to make porting from python 2 to python 3 more guided and incremental. We believe that these attributes are necessary. We would like to stress that we don't believe anything on this list is as important as the continuing efforts that everyone in the broader ecosystem is making. If you just want to ease the transition by working on anything at all, the best use of your time right now is porting https://warehouse.python.org/project/MySQL-python/ to Python 3. :) Nevertheless there are some things that the language and CPython could do. Unfortunately we had to reject any proposal that involved new __future__ imports, since unknown __future__ imports are un-catchable SyntaxErrors. Here are some ideas for Python 2.7+. Add ensurepip to the installers. Having pip reliably available increases the availability of libraries that help with porting, and will generally strengthen the broader ecosystem in the (increasingly long) transition period. Add some warnings about python 3 compatibility. It should at least be possible to get a warning for every single implicit string coercion. Old-style classes. Old-style division. Print statements. Old-style exception syntax. buffer(). bytes(memoryview(b'abc')) Importing old locations from the stdlib (see point 4.) Long integer syntax. Use of variables beyond the lifetime of an 'except Exception as e' block or a list comprehension. Backport 'yield from' to allow people to use Tulip and Tulip-compatible code, and to facilitate the development of Tulip-friendly libraries and a Tulip ecosystem. A robust Tulip ecosystem requires the participation of people who are not yet using Python 3. Add aliases for the renamed modules in the stdlib. This will allow people to "just write python 3" in a lot more circumstances. (re-)Enable warnings by default, including enabling -3 warnings. Right now all warnings are silent by default, which greatly reduces discoverability of future compatibility issues. I hope it's not controversial to say that most new Python code is still being written against Python 2.7 today; if people are writing that code in such a way that it's not 3-friendly, it should be a more immediately noticeable issue. Get rid of 2to3. Particularly, of any discussion of using 2to3 in the documentation. More than one very experienced, well-known Python developer in this discussion has told me that they thought 2to3 was the blessed way to port their code, and it's no surprise that they think so, given that the first technique mentions is still 2to3. We should replace 2to3 with something like . 2to3 breaks your code on python 2, and doesn't necessarily get it running on python 3. A more conservative approach that reduced the amount of work to get your code 2/3 compatible but was careful to leave everything working would be a lot more effective. Add a new 'bytes' type that actually behaves like the Python 3 bytes type (bytes(5)). We have rejected any changes for Python 3.5, simply because of the extremely long time to get those features into users hands. Any changes for Python 3 that we're proposing would need to get into a 3.4.x release, so that, for example, they can make their way into Ubuntu 14.04 LTS. Here are some ideas for Python 3.4.x: Usage of Python2 style syntax (for example, a print statement) or stdlib module names (for example, 'import urllib2') should result in a specific, informative warning, not a generic SyntaxError/ImportError. This will really help new users. Add 'unicode' back as an alias for 'str'. Just today I was writing some documentation where I had to resort to some awkward encoding tricks just to get a bytes object out without explaining the whole 2/3 dichotomy in some unrelated prose. We'd like to thank all the individuals who gave input and feedback in creating this list. -glyph & Alex Gaynor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu May 29 05:51:41 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 28 May 2014 21:51:41 -0600 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: Thanks for for putting this together. On Wed, May 28, 2014 at 4:26 PM, Glyph Lefkowitz wrote: > At the language summit, Alex and I volunteered to put together some > recommendations on what changes could be made to Python (the language) in > order to facilitate a smoother transition from Python 2 to Python 3. One of > the things that motivated this was the (surprising, to us) consideration > that features like ensurepip might be added to the future versions of the > 2.7 installers from python.org. > > The specific motivations for writing this are: > > 1. Library maintainers have a rapidly expanding matrix that requires an > increasing number of branches to satisfy. > 2. People with large corporate codebases absolutely cannot port all at once. > > > If you don't have perfect test coverage then you can't make any progress. > So these changes are intended to make porting from python 2 to python 3 more > guided and incremental. We believe that these attributes are necessary. One of the parallel discussions that came out of the language summit centers on basically the same idea: https://mail.python.org/pipermail/python-dev/2014-April/133986.html The key points are that there would be a separate tool that wraps the 2.7 interpreter and enforces the 2.7/3.x subset language but only for source files that have are tagged (a la the coding cookie). So there would be no need to make changes to Python itself. Some have also suggested this could also be accomplished via a pylint plugin. I'm working on an implementation called (for now) "pymigrate" that should eventually be available via the cheeseshop. A lot of what you've written here is really useful. > Here are some ideas for Python 2.7+. > > 1. Add ensurepip to the installers. Having pip reliably available increases > the availability of libraries that help with porting, and will generally > strengthen the broader ecosystem in the (increasingly long) transition > period. My understanding is that this is already the plan. > 2. Add some warnings about python 3 compatibility. > > 1. It should at least be possible to get a warning for every single implicit > string coercion. > 2. Old-style classes. > 3. Old-style division. > 4. Print statements. > 5. Old-style exception syntax. > 6. buffer(). > 7. bytes(memoryview(b'abc')) > 8. Importing old locations from the stdlib (see point 4.) > 9. Long integer syntax. > 10. Use of variables beyond the lifetime of an 'except Exception as e' block or > a list comprehension. +1 to improving the coverage of the -3 option in 2.7. > > 3. Backport 'yield from' to allow people to use Tulip and Tulip-compatible > code, and to facilitate the development of Tulip-friendly libraries and a > Tulip ecosystem. A robust Tulip ecosystem requires the participation of > people who are not yet using Python 3. What about Trellis? Regardless, adding yield from to 2.7 is going to be a tough sell. > 4. Add aliases for the renamed modules in the stdlib. This will allow people > to "just write python 3" in a lot more circumstances. I like this, but it seems to me to be a better fit for something like six or even pymigrate. > 5. (re-)Enable warnings by default, including enabling -3 warnings. Right now > all warnings are silent by default, which greatly reduces discoverability of > future compatibility issues. This is also something that fits better with a separate tool. > We have rejected any changes for Python 3.5, simply because of the extremely > long time to get those features into users hands. Any changes for Python 3 > that we're proposing would need to get into a 3.4.x release Good luck with that one! -eric From benjamin at python.org Thu May 29 06:50:15 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 28 May 2014 21:50:15 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: <1401339015.20566.122748037.2A7D52C6@webmail.messagingengine.com> On Wed, May 28, 2014, at 15:26, Glyph Lefkowitz wrote: > Add some warnings about python 3 compatibility. > It should at least be possible to get a warning for every single implicit > string coercion. > Old-style classes. > Old-style division. Fun fact: This can be achieved with the -Qwarn command line option. From songofacandy at gmail.com Thu May 29 07:22:31 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 29 May 2014 14:22:31 +0900 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: > We would like to stress that we don't believe anything on this list is as > important as the continuing efforts that everyone in the broader ecosystem > is making. If you just want to ease the transition by working on anything > at all, the best use of your time right now is porting > https://warehouse.python.org/project/MySQL-python/ to Python 3. :) > I've did it. https://github.com/PyMySQL/mysqlclient-python https://pypi.python.org/pypi/mysqlclient -- INADA Naoki From tjreedy at udel.edu Thu May 29 08:53:19 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 May 2014 02:53:19 -0400 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 5/29/2014 1:22 AM, INADA Naoki wrote: >> We would like to stress that we don't believe anything on this list is as >> important as the continuing efforts that everyone in the broader ecosystem >> is making. If you just want to ease the transition by working on anything >> at all, the best use of your time right now is porting >> https://warehouse.python.org/project/MySQL-python/ to Python 3. :) >> > > I've did it. > > https://github.com/PyMySQL/mysqlclient-python > https://pypi.python.org/pypi/mysqlclient I don't think that most people interpret Programming Language :: Python as indicating support for Python 3. So I suggest adding Programming Language :: Python :: 3 or even the more specific 3.3 and 3.4 -- Terry Jan Reedy From ncoghlan at gmail.com Thu May 29 13:43:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 May 2014 21:43:07 +1000 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 29 May 2014 08:26, Glyph Lefkowitz wrote: Thanks for the write-up! > Here are some ideas for Python 2.7+. > > Add ensurepip to the installers. Having pip reliably available increases > the availability of libraries that help with porting, and will generally > strengthen the broader ecosystem in the (increasingly long) transition > period. Agreed on this one - I only dropped it from 453 because I didn't think I could continue to push for it and still make the beta deadline for 3.4, not because I changed my mind. This mainly needs a champion to write the PEP to describe the exact details of what would be backported (the tricky parts of PEP 453 relating to pyvenv don't apply) and to actually do the work. Bundling the Python Launcher with the Windows installer would also be desirable. > Add some warnings about python 3 compatibility. > > It should at least be possible to get a warning for every single implicit > string coercion. > Old-style classes. > Old-style division. > Print statements. > Old-style exception syntax. > buffer(). > bytes(memoryview(b'abc')) > Importing old locations from the stdlib (see point 4.) > Long integer syntax. > Use of variables beyond the lifetime of an 'except Exception as e' block or > a list comprehension. A few more for the "missing -3 warning" list: - calling str.encode - calling unicode.decode - returning str from str.decode - returning unicode from unicode.encode The relevant TypeErrors in 3.x show where the latter two warnings need to go in 2.7. More controversially: warn for any calls to Py2 str methods that aren't present on Py3 bytes. > Backport 'yield from' to allow people to use Tulip and Tulip-compatible > code, and to facilitate the development of Tulip-friendly libraries and a > Tulip ecosystem. A robust Tulip ecosystem requires the participation of > people who are not yet using Python 3. Given that the target audience here is our more conservative user base, Tulip/Trollius backends for Twisted and gevent seem more conducive to that than supporting Tulip's native coroutine syntax. That said, it's a pure additive change without any significant backwards compatibility implications, so I personally wouldn't object if Guido said yes. > Add aliases for the renamed modules in the stdlib. This will allow people > to "just write python 3" in a lot more circumstances. This is what the module aliasing options provided by python-future are designed to do - it's much closer to "just write Python 3" than six is (although that comes at the price of being a bit more magical). The python-future experience also shows there's a potential backwards compatibility issue here - depending on exactly how you do it, you can end up conflicting with other libraries trying to do the same thing. > (re-)Enable warnings by default, including enabling -3 warnings. Right now > all warnings are silent by default, which greatly reduces discoverability of > future compatibility issues. I hope it's not controversial to say that most > new Python code is still being written against Python 2.7 today; if people > are writing that code in such a way that it's not 3-friendly, it should be a > more immediately noticeable issue. The reasons these were turned off haven't changed, and passing -3 already enables DeprecationWarnings generally (along with -Qwarn). However, several of the warnings mentioned in the list above are currently missing entirely, so they won't show up regardless of the warning state. > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the > documentation. More than one very experienced, well-known Python developer > in this discussion has told me that they thought 2to3 was the blessed way to > port their code, and it's no surprise that they think so, given that the > first technique mentions is > still 2to3. We should replace 2to3 with something like > . 2to3 breaks your code on > python 2, and doesn't necessarily get it running on python 3. A more > conservative approach that reduced the amount of work to get your code 2/3 > compatible but was careful to leave everything working would be a lot more > effective. +1, although I think python-future provides a better "subset language" experience than modernize+six does, and provides a 3->common subset converter in addition to a 2->common subset (see http://python-future.org/faq.html#relationship-between-python-future-and-other-compatibility-tools for more on the difference between the two approaches). The futurize script also has the nice property of supporting two different stages, with "--stage1" being a pure syntactic update - no new runtime dependencies needed. "Big bang" migrations are highly unlikely to be the right choice for anyone, and while the code produced by python-future will still have some quirks (e.g. avoiding direct use of any mapping methods as PEP 469 ended up recommending, plus a bunch of import noise at the start of the file) > Add a new 'bytes' type that actually behaves like the Python 3 bytes type > (bytes(5)). This is available in python-future. > We have rejected any changes for Python 3.5, simply because of the extremely > long time to get those features into users hands. Any changes for Python 3 > that we're proposing would need to get into a 3.4.x release, so that, for > example, they can make their way into Ubuntu 14.04 LTS. > > Here are some ideas for Python 3.4.x: > > Usage of Python2 style syntax (for example, a print statement) or stdlib > module names (for example, 'import urllib2') should result in a specific, > informative warning, not a generic SyntaxError/ImportError. This will > really help new users. > Add 'unicode' back as an alias for 'str'. Just today I was writing some > documentation where I had to resort to some awkward encoding tricks just to > get a bytes object out without explaining the whole 2/3 dichotomy in some > unrelated prose. While I realise it's out of scope for your "near term changes" list, there are still a few changes I think are worth exploring for the Python 3.5+ time frame: - the return of binary interpolation is already approved (but isn't implemented yet) - various other improvements to the Py3 binary data model (such as the ideas in PEP 467) - PEP 432 (cleaning up the startup sequence), or something along those lines, has now been identified as a prerequisite for decoupling Python 3 from the locale encoding on Linux (and hence letting us more easily ignore the entirely-inappropriate-for-the-21st-century C locale) - I'd like to see someone take a serious tilt at proposing "NAME " call statement support at the language level. IPython already supports that style of invocation for arbitrary callables (so print statements still appear work even in Python 3 if you use IPython rather than the default REPL or direct script execution), and there's a proof of concept patch at http://bugs.python.org/issue18788 that shows CPython's parser is able to handle the idea (see my final comment for additional restrictions a full proposal would need) For that last point, my interest is as much educational as it is in easing the transition from Python 2. The parentheses in "print('Hello world!')" mean introducing the idea of function calls early to explain how it works, while being able to omit them makes it easier to gloss over the distinction between statements and function calls initially and then cover it later after the basics of flow control have been nailed down. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bcannon at gmail.com Thu May 29 16:49:55 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 29 May 2014 14:49:55 +0000 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181 References: Message-ID: I think the memory leak was caused by http://hg.python.org/cpython/rev/7d20e30bd540 because http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the 'res' variable and then overwrites it unconditionally w/o PY_DECREF beforehand. On Thu May 29 2014 at 4:02:17 AM, wrote: > results for 8391f99208f6 on branch "default" > -------------------------------------------- > > test_collections leaked [0, 2, 4] references, sum=6 > test_collections leaked [0, 1, 2] memory blocks, sum=3 > test_functools leaked [0, 0, 3] memory blocks, sum=3 > test_importlib leaked [5, 5, 5] references, sum=15 > test_pkgutil leaked [1, 1, 1] references, sum=3 > test_site leaked [0, 0, 2] references, sum=2 > test_site leaked [0, 0, 2] memory blocks, sum=2 > test_zipimport leaked [47, 47, 47] references, sum=141 > test_zipimport_support leaked [2, 2, 2] references, sum=6 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x'] > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu May 29 17:10:44 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 29 May 2014 15:10:44 +0000 Subject: [Python-Dev] Language Summit Follow-Up References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On Wed May 28 2014 at 10:14:39 PM, Glyph Lefkowitz wrote: > At the language summit, Alex and I volunteered to put together some > recommendations on what changes could be made to Python (the language) in > order to facilitate a smoother transition from Python 2 to Python 3. One > of the things that motivated this was the (surprising, to us) consideration > that features like ensurepip might be added to the future versions of the > 2.7 installers from python.org. > > The specific motivations for writing this are: > > > 1. Library maintainers have a rapidly expanding matrix that requires > an increasing number of branches to satisfy. > 2. People with large corporate codebases absolutely cannot port all at > once. > > > If you don't have perfect test coverage then you can't make any progress. > So these changes are intended to make porting from python 2 to python 3 > more guided and incremental. We believe that these attributes are > necessary. > > We would like to stress that we don't believe anything on this list is as > important as the continuing efforts that everyone in the broader ecosystem > is making. If you just want to ease the transition by working on anything > at all, the best use of your time right now is porting > https://warehouse.python.org/project/MySQL-python/ to Python 3. :) > > Nevertheless there are some things that the language and CPython could do. > > Unfortunately we had to reject any proposal that involved new __future__ > imports, since unknown __future__ imports are un-catchable SyntaxErrors. > > Here are some ideas for Python 2.7+. > [SNIP] > > 1. Get rid of 2to3. Particularly, of any discussion of using 2to3 in > the documentation. More than one very experienced, well-known Python > developer in this discussion has told me that they thought 2to3 was the > blessed way to port their code, and it's no surprise that they think so, > given that the first technique < > https://docs.python.org/3/howto/pyporting.html> mentions is still > 2to3. We should replace 2to3 with something like < > https://github.com/mitsuhiko/python-modernize>. 2to3 breaks your code > on python 2, and doesn't necessarily get it running on python 3. A more > conservative approach that *reduced* the amount of work to get your > code 2/3 compatible but was careful to leave everything working would be a > lot more effective. > > Two things. One is the HOWTO mentions 2to3 *only* when you are dropping Python 3 support and since that makes your life the easiest it's listed as the first option. You will notice point 2 in the TL;DR is be source compatible if keeping Python 2 compatibility. The main part of the doc is also all about source compatibility and using 2to3 while maintaining compatibility is relegated to the Alternatives section at the end (heck, 2to3 is mentioned only 3 times in that entire document, one of which is in a code comment). Two, is how do you see python-future fit into this (I know others have brought it up but we have two different approaches now to keeping source compatibility that vary in magic and thus smoothing out edges)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 29 18:30:14 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 May 2014 12:30:14 -0400 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 5/28/2014 6:26 PM, Glyph Lefkowitz wrote: > I hope it's > not controversial to say that most new Python code is still being > written against Python 2.7 today; Given that Python 3 downloads now outnumber Python 2 downloads, I think 'most' might be an overstatement. But I think it a moot point. > if people are writing that code in such a way that it's not > 3-friendly, it should be a more immediately noticeable issue. If the truth were, conservatively, 1/4 of new Python code in 2.7, or even less, I would still be in favor of making 3-friendly 2.7 code easier. This is also important for the separate codebase approach, as in the stdlib. Just last week, I got a rejected chunk when backporting because a 2.7 idlelib module uses 'file(...' instead of 'open(...'. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Thu May 29 18:34:04 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 29 May 2014 10:34:04 -0600 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181 In-Reply-To: References: Message-ID: Good catch. I'll look into it. -eric On Thu, May 29, 2014 at 8:49 AM, Brett Cannon wrote: > I think the memory leak was caused by > http://hg.python.org/cpython/rev/7d20e30bd540 because > http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the > 'res' variable and then overwrites it unconditionally w/o PY_DECREF > beforehand. > > > On Thu May 29 2014 at 4:02:17 AM, wrote: >> >> results for 8391f99208f6 on branch "default" >> -------------------------------------------- >> >> test_collections leaked [0, 2, 4] references, sum=6 >> test_collections leaked [0, 1, 2] memory blocks, sum=3 >> test_functools leaked [0, 0, 3] memory blocks, sum=3 >> test_importlib leaked [5, 5, 5] references, sum=15 >> test_pkgutil leaked [1, 1, 1] references, sum=3 >> test_site leaked [0, 0, 2] references, sum=2 >> test_site leaked [0, 0, 2] memory blocks, sum=2 >> test_zipimport leaked [47, 47, 47] references, sum=141 >> test_zipimport_support leaked [2, 2, 2] references, sum=6 >> >> >> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', >> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x'] >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> https://mail.python.org/mailman/listinfo/python-checkins From ericsnowcurrently at gmail.com Thu May 29 20:46:11 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 29 May 2014 12:46:11 -0600 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181 In-Reply-To: References: Message-ID: Fixed! (test_site and test_functools are still leaking sporadically, but it looks unrelated to the import.c leak). -eric On Thu, May 29, 2014 at 10:34 AM, Eric Snow wrote: > Good catch. I'll look into it. > > -eric > > On Thu, May 29, 2014 at 8:49 AM, Brett Cannon wrote: >> I think the memory leak was caused by >> http://hg.python.org/cpython/rev/7d20e30bd540 because >> http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the >> 'res' variable and then overwrites it unconditionally w/o PY_DECREF >> beforehand. >> >> >> On Thu May 29 2014 at 4:02:17 AM, wrote: >>> >>> results for 8391f99208f6 on branch "default" >>> -------------------------------------------- >>> >>> test_collections leaked [0, 2, 4] references, sum=6 >>> test_collections leaked [0, 1, 2] memory blocks, sum=3 >>> test_functools leaked [0, 0, 3] memory blocks, sum=3 >>> test_importlib leaked [5, 5, 5] references, sum=15 >>> test_pkgutil leaked [1, 1, 1] references, sum=3 >>> test_site leaked [0, 0, 2] references, sum=2 >>> test_site leaked [0, 0, 2] memory blocks, sum=2 >>> test_zipimport leaked [47, 47, 47] references, sum=141 >>> test_zipimport_support leaked [2, 2, 2] references, sum=6 >>> >>> >>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', >>> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x'] >>> _______________________________________________ >>> Python-checkins mailing list >>> Python-checkins at python.org >>> https://mail.python.org/mailman/listinfo/python-checkins From josiah.carlson at gmail.com Fri May 30 02:34:15 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 29 May 2014 17:34:15 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: Pinging this thread 2 months later with a progress/status update. To those that have reviewed, commented, helped, or otherwise pushed this along, which includes (but is not limited to) Richard Oudkerk, eryksun, Giampaolo Rodola, thank you. The short version: As far as I can tell, the patch is ready: http://bugs.python.org/issue1191964 What is available: There are docs, tests, and obviously the functionality. Some code was moved from asyncio/windows_utils.py (which has a separate issue here: https://code.google.com/p/tulip/issues/detail?id=170). The API was changed slightly from what was proposed by Guido: sent = Popen.write_nonblocking(input, timeout=0) data = Popen.read_nonblocking(bufsize=4096) data = Popen.read_stderr_nonblocking(bufsize=4096) As a bonus feature, Windows communicate() calls no longer spawn worker threads, and instead use overlapped IO. I'm bringing this back up to python-dev to offer a slightly wider audience for commentary/concerns, and hopefully to get a stamp of approval that it is ready. Thank you, - Josiah On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson wrote: > I've got a patch with partial tests and documentation that I'm holding off > on upload because I believe there should be a brief discussion. > > Long story short, Windows needs a thread to handle writing in a > non-blocking fashion, regardless of the use of asyncio or plain subprocess. > > If you'd like to continue following this issue and participate in the > discussion, I'll see you over on http://bugs.python.org/issue1191964 . > > - Josiah > > > > On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson > wrote: > >> >> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum >> wrote: >> >>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson < >>> josiah.carlson at gmail.com> wrote: >>> >>>> >>>> If it makes you feel any better, I spent an hour this morning building >>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and >>>> not even using any part of asyncio (the Windows bits are in msvcrt and >>>> _winapi). It works in Python 3.3+. You can see it here: >>>> http://pastebin.com/0LpyQtU5 >>>> >>> >>> Seeing this makes *me* feel better. I think it's reasonable to add (some >>> variant) of that to the subprocess module in Python 3.5. It also belongs in >>> the Activestate cookbook. And no, the asyncio module hasn't made it >>> obsolete. >>> >> >> Cool. >> >> Josiah, you sound upset about the whole thing -- to the point of >>> writing unintelligible sentences and passive-aggressive digs at everyone >>> reading this list. I'm sorry that something happened that led you feel that >>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote >>> that code snippet -- it is completely clear what you want and why you want >>> it, and also what should happen next (a few rounds of code review on the >>> tracker). >>> >> >> I'm not always a prat. Something about python-dev brings out parts of me >> that I thought I had discarded from my personality years ago. Toss in a bit >> of needing to re-explain ideas that I've been trying to explain for almost >> 9 years? Frustration + formerly discarded personality traits = uck. That's >> pretty much why I won't be rejoining the party here regularly, you are all >> better off without me commenting on 95% of threads like I used to. >> >> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I >> was when I spend time on this list. That's *my* issue, not yours. That I >> spent any time redirecting my frustration towards you is BS, and if I could >> take back the email I sent just before getting Guido's, I would. >> >> I would advise everyone to write it off as the ramblings of a >> surprisingly young, angry old man. Or call me an a-hole. Both are pretty >> accurate. :) >> >> But that PEP? It's just a terrible PEP. It doesn't contain a single line >>> of example code. It doesn't specify the proposed interface, it just >>> describes in way too many sentences how it should work, and contains a >>> whole lot of references to various rants from which the reader is >>> apparently meant to become enlightened. I don't know which of the three >>> authors *really* wrote it, and I don't want to know -- I think the PEP is >>> irrelevant to the proposed feature, which is of "put it in the bug tracker >>> and work from there" category -- presumably the PEP was written based on >>> the misunderstanding that having a PEP would make acceptance of the patch >>> easier, or because during an earlier bikeshedding round someone said >>> "please write a PEP" (someone always says that). I propose to scrap the PEP >>> (set the status to Withdrawn) and just work on adding the methods to the >>> subprocess module. >>> >> >> I'm not going to argue. The first I read it was 2-3 days ago. >> >> If it were me, I'd define three methods, with longer names to clarify >>> what they do, e.g. >>> >>> proc.write_nonblocking(data) >>> data = proc.read_nonblocking() >>> data = proc.read_stderr_nonblocking() >>> >> >> Easily doable. >> >> I.e. add _nonblocking to the method names to clarify that they may return >>> '' when there's nothing available, and have a separate method for reading >>> stderr instead of a flag. And I'd wonder if there should be an unambiguous >>> way to detect EOF or whether the caller should just check for >>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable >>> when the other end is closed, and then the write() will fail. But maybe I >>> forget.) >>> >>> But that's all bikeshedding and it can happen on the tracker or directly >>> on the list just as easily; I don't see the need for a PEP. >>> >> >> Sounds good. >> >> - Josiah >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiah.carlson at gmail.com Fri May 30 02:35:21 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 29 May 2014 17:35:21 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: And as I was writing the "thank you" to folks, I hit send too early. Also thank you to Victor Stinner, Guido, Terry Reedy, and everyone else on this thread :) - Josiah On Thu, May 29, 2014 at 5:34 PM, Josiah Carlson wrote: > Pinging this thread 2 months later with a progress/status update. > > To those that have reviewed, commented, helped, or otherwise pushed this > along, which includes (but is not limited to) Richard Oudkerk, eryksun, > Giampaolo Rodola, thank you. > > > The short version: > As far as I can tell, the patch is ready: > http://bugs.python.org/issue1191964 > > What is available: > There are docs, tests, and obviously the functionality. Some code was > moved from asyncio/windows_utils.py (which has a separate issue here: > https://code.google.com/p/tulip/issues/detail?id=170). The API was > changed slightly from what was proposed by Guido: > > sent = Popen.write_nonblocking(input, timeout=0) > data = Popen.read_nonblocking(bufsize=4096) > data = Popen.read_stderr_nonblocking(bufsize=4096) > > As a bonus feature, Windows communicate() calls no longer spawn worker > threads, and instead use overlapped IO. > > > I'm bringing this back up to python-dev to offer a slightly wider audience > for commentary/concerns, and hopefully to get a stamp of approval that it > is ready. > > Thank you, > - Josiah > > > > > On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson > wrote: > >> I've got a patch with partial tests and documentation that I'm holding >> off on upload because I believe there should be a brief discussion. >> >> Long story short, Windows needs a thread to handle writing in a >> non-blocking fashion, regardless of the use of asyncio or plain subprocess. >> >> If you'd like to continue following this issue and participate in the >> discussion, I'll see you over on http://bugs.python.org/issue1191964 . >> >> - Josiah >> >> >> >> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson < >> josiah.carlson at gmail.com> wrote: >> >>> >>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum >>> wrote: >>> >>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson < >>>> josiah.carlson at gmail.com> wrote: >>>> >>>>> >>>>> If it makes you feel any better, I spent an hour this morning building >>>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and >>>>> not even using any part of asyncio (the Windows bits are in msvcrt and >>>>> _winapi). It works in Python 3.3+. You can see it here: >>>>> http://pastebin.com/0LpyQtU5 >>>>> >>>> >>>> Seeing this makes *me* feel better. I think it's reasonable to add >>>> (some variant) of that to the subprocess module in Python 3.5. It also >>>> belongs in the Activestate cookbook. And no, the asyncio module hasn't made >>>> it obsolete. >>>> >>> >>> Cool. >>> >>> Josiah, you sound upset about the whole thing -- to the point of >>>> writing unintelligible sentences and passive-aggressive digs at everyone >>>> reading this list. I'm sorry that something happened that led you feel that >>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote >>>> that code snippet -- it is completely clear what you want and why you want >>>> it, and also what should happen next (a few rounds of code review on the >>>> tracker). >>>> >>> >>> I'm not always a prat. Something about python-dev brings out parts of me >>> that I thought I had discarded from my personality years ago. Toss in a bit >>> of needing to re-explain ideas that I've been trying to explain for almost >>> 9 years? Frustration + formerly discarded personality traits = uck. That's >>> pretty much why I won't be rejoining the party here regularly, you are all >>> better off without me commenting on 95% of threads like I used to. >>> >>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I >>> was when I spend time on this list. That's *my* issue, not yours. That I >>> spent any time redirecting my frustration towards you is BS, and if I could >>> take back the email I sent just before getting Guido's, I would. >>> >>> I would advise everyone to write it off as the ramblings of a >>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty >>> accurate. :) >>> >>> But that PEP? It's just a terrible PEP. It doesn't contain a single line >>>> of example code. It doesn't specify the proposed interface, it just >>>> describes in way too many sentences how it should work, and contains a >>>> whole lot of references to various rants from which the reader is >>>> apparently meant to become enlightened. I don't know which of the three >>>> authors *really* wrote it, and I don't want to know -- I think the PEP is >>>> irrelevant to the proposed feature, which is of "put it in the bug tracker >>>> and work from there" category -- presumably the PEP was written based on >>>> the misunderstanding that having a PEP would make acceptance of the patch >>>> easier, or because during an earlier bikeshedding round someone said >>>> "please write a PEP" (someone always says that). I propose to scrap the PEP >>>> (set the status to Withdrawn) and just work on adding the methods to the >>>> subprocess module. >>>> >>> >>> I'm not going to argue. The first I read it was 2-3 days ago. >>> >>> If it were me, I'd define three methods, with longer names to clarify >>>> what they do, e.g. >>>> >>>> proc.write_nonblocking(data) >>>> data = proc.read_nonblocking() >>>> data = proc.read_stderr_nonblocking() >>>> >>> >>> Easily doable. >>> >>> I.e. add _nonblocking to the method names to clarify that they may >>>> return '' when there's nothing available, and have a separate method for >>>> reading stderr instead of a flag. And I'd wonder if there should be an >>>> unambiguous way to detect EOF or whether the caller should just check for >>>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable >>>> when the other end is closed, and then the write() will fail. But maybe I >>>> forget.) >>>> >>>> But that's all bikeshedding and it can happen on the tracker or >>>> directly on the list just as easily; I don't see the need for a PEP. >>>> >>> >>> Sounds good. >>> >>> - Josiah >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 30 18:07:58 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 30 May 2014 18:07:58 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140530160758.17F8056A46@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-05-23 - 2014-05-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4635 (+13) closed 28750 (+40) total 33385 (+53) Open issues with patches: 2127 Issues opened (41) ================== #16638: support multi-line docstring signatures in IDLE calltips http://bugs.python.org/issue16638 reopened by rhettinger #21561: help() on enum34 enumeration class creates only a dummy docume http://bugs.python.org/issue21561 opened by andymaier #21563: Segv during call to builtin_execfile in application embedding http://bugs.python.org/issue21563 opened by snoeberger #21564: Declaration of EVP_MD_CTX causes crash when switching between http://bugs.python.org/issue21564 opened by Ryan.Calhoun #21566: make use of the new default socket.listen() backlog argument http://bugs.python.org/issue21566 opened by neologix #21567: cannot create multipart alternative message with us-ascii char http://bugs.python.org/issue21567 opened by Viktor.Sz??pe #21568: Win32 pip doesn't use relative path to found site-packages. http://bugs.python.org/issue21568 opened by ??????.??? #21569: PEP 466: Python 2.7 What's New preamble changes http://bugs.python.org/issue21569 opened by ncoghlan #21571: Python build should check CPATH, C_INCLUDE_PATH for module dep http://bugs.python.org/issue21571 opened by JanKanis #21572: Use generic license web page rather than requiring release-spe http://bugs.python.org/issue21572 opened by ned.deily #21573: Clean up turtle.py code formatting http://bugs.python.org/issue21573 opened by jesstess #21574: Port image types detections from PIL to the imghdr module http://bugs.python.org/issue21574 opened by serhiy.storchaka #21576: Overwritten (custom) uuid inside dictionary http://bugs.python.org/issue21576 opened by beta990 #21577: Help for ImportError should show a more useful signature. http://bugs.python.org/issue21577 opened by eric.snow #21578: Misleading error message when ImportError called with invalid http://bugs.python.org/issue21578 opened by eric.snow #21579: Python 3.4: tempfile.close attribute does not work http://bugs.python.org/issue21579 opened by mmarkk #21580: PhotoImage(data=...) apparently has to be UTF-8 or Base-64 enc http://bugs.python.org/issue21580 opened by vadmium #21581: Consider dropping importlib.abc.Loader.create_module() http://bugs.python.org/issue21581 opened by brett.cannon #21582: use support.catpured context managers - test_asyncore http://bugs.python.org/issue21582 opened by diana #21583: use support.catpured_stderr context manager - test_logging http://bugs.python.org/issue21583 opened by diana #21585: Run Tkinter tests with wantobjects=False http://bugs.python.org/issue21585 opened by serhiy.storchaka #21588: Idle: make editor title bar user configurable http://bugs.python.org/issue21588 opened by terry.reedy #21590: Systemtap and DTrace support http://bugs.python.org/issue21590 opened by bkabrda #21591: "exec(a, b, c)" not the same as "exec a in b, c" in nested fun http://bugs.python.org/issue21591 opened by Robert.Jordens #21592: Make statistics.median run in linear time http://bugs.python.org/issue21592 opened by Thomas.Dybdahl.Ahle #21593: Clarify re.search documentation first match http://bugs.python.org/issue21593 opened by Joshua.Landau #21594: asyncio.create_subprocess_exec raises OSError http://bugs.python.org/issue21594 opened by Sebastian.Kreft.Deezer #21595: Creating many subprocess generates lots of internal BlockingIO http://bugs.python.org/issue21595 opened by Sebastian.Kreft.Deezer #21596: asyncio.wait fails when futures list is empty http://bugs.python.org/issue21596 opened by Sebastian.Kreft.Deezer #21597: Allow turtledemo code pane to get wider. http://bugs.python.org/issue21597 opened by terry.reedy #21599: Argument transport in attach and detach method in Server class http://bugs.python.org/issue21599 opened by vajrasky #21600: mock.patch.stopall doesn't work with patch.dict to sys.modules http://bugs.python.org/issue21600 opened by kakuma #21601: Cancel method for Asyncio Task is not documented http://bugs.python.org/issue21601 opened by vajrasky #21603: IDLE SaveAs drops the extension in the prompted filename http://bugs.python.org/issue21603 opened by rhettinger #21604: Misleading 2to3 fixer name in documentation: standard_error http://bugs.python.org/issue21604 opened by wluebbe #21605: Add tests for Tkinter images http://bugs.python.org/issue21605 opened by serhiy.storchaka #21608: logging.handlers.HTTPHandler.setFormatter() has no effect http://bugs.python.org/issue21608 opened by bk2zsto #21610: load_module not closing opened files http://bugs.python.org/issue21610 opened by mattip #21611: int() docstring - unclear what number is http://bugs.python.org/issue21611 opened by and #21612: IDLE should not open multiple instances of one file http://bugs.python.org/issue21612 opened by irdb #21613: Installer for mac doesn't store the installation location http://bugs.python.org/issue21613 opened by ustinov Most recent 15 issues with no replies (15) ========================================== #21612: IDLE should not open multiple instances of one file http://bugs.python.org/issue21612 #21611: int() docstring - unclear what number is http://bugs.python.org/issue21611 #21605: Add tests for Tkinter images http://bugs.python.org/issue21605 #21604: Misleading 2to3 fixer name in documentation: standard_error http://bugs.python.org/issue21604 #21601: Cancel method for Asyncio Task is not documented http://bugs.python.org/issue21601 #21600: mock.patch.stopall doesn't work with patch.dict to sys.modules http://bugs.python.org/issue21600 #21599: Argument transport in attach and detach method in Server class http://bugs.python.org/issue21599 #21597: Allow turtledemo code pane to get wider. http://bugs.python.org/issue21597 #21596: asyncio.wait fails when futures list is empty http://bugs.python.org/issue21596 #21593: Clarify re.search documentation first match http://bugs.python.org/issue21593 #21591: "exec(a, b, c)" not the same as "exec a in b, c" in nested fun http://bugs.python.org/issue21591 #21582: use support.catpured context managers - test_asyncore http://bugs.python.org/issue21582 #21577: Help for ImportError should show a more useful signature. http://bugs.python.org/issue21577 #21568: Win32 pip doesn't use relative path to found site-packages. http://bugs.python.org/issue21568 #21564: Declaration of EVP_MD_CTX causes crash when switching between http://bugs.python.org/issue21564 Most recent 15 issues waiting for review (15) ============================================= #21610: load_module not closing opened files http://bugs.python.org/issue21610 #21608: logging.handlers.HTTPHandler.setFormatter() has no effect http://bugs.python.org/issue21608 #21605: Add tests for Tkinter images http://bugs.python.org/issue21605 #21601: Cancel method for Asyncio Task is not documented http://bugs.python.org/issue21601 #21599: Argument transport in attach and detach method in Server class http://bugs.python.org/issue21599 #21595: Creating many subprocess generates lots of internal BlockingIO http://bugs.python.org/issue21595 #21585: Run Tkinter tests with wantobjects=False http://bugs.python.org/issue21585 #21583: use support.catpured_stderr context manager - test_logging http://bugs.python.org/issue21583 #21582: use support.catpured context managers - test_asyncore http://bugs.python.org/issue21582 #21580: PhotoImage(data=...) apparently has to be UTF-8 or Base-64 enc http://bugs.python.org/issue21580 #21578: Misleading error message when ImportError called with invalid http://bugs.python.org/issue21578 #21572: Use generic license web page rather than requiring release-spe http://bugs.python.org/issue21572 #21566: make use of the new default socket.listen() backlog argument http://bugs.python.org/issue21566 #21564: Declaration of EVP_MD_CTX causes crash when switching between http://bugs.python.org/issue21564 #21563: Segv during call to builtin_execfile in application embedding http://bugs.python.org/issue21563 Top 10 most discussed issues (10) ================================= #21477: Idle: improve idle_test.htest http://bugs.python.org/issue21477 18 msgs #20383: Add a keyword-only spec argument to types.ModuleType http://bugs.python.org/issue20383 13 msgs #17172: Add turtledemo to IDLE menu http://bugs.python.org/issue17172 10 msgs #20611: socket.create_connection() doesn't handle EINTR properly http://bugs.python.org/issue20611 9 msgs #20689: socket.AddressFamily is absent in pydoc output http://bugs.python.org/issue20689 7 msgs #21579: Python 3.4: tempfile.close attribute does not work http://bugs.python.org/issue21579 7 msgs #3015: tkinter with wantobjects=False has been broken for some time http://bugs.python.org/issue3015 6 msgs #21235: importlib's spec module create algorithm is not exposed http://bugs.python.org/issue21235 6 msgs #8243: curses writing to window's bottom right position raises: `_cur http://bugs.python.org/issue8243 5 msgs #21561: help() on enum34 enumeration class creates only a dummy docume http://bugs.python.org/issue21561 5 msgs Issues closed (39) ================== #1641: asyncore delayed calls feature http://bugs.python.org/issue1641 closed by haypo #7094: Add alternate float formatting styles to new-style formatting. http://bugs.python.org/issue7094 closed by eric.smith #8743: set() operators don't work with collections.Set instances http://bugs.python.org/issue8743 closed by rhettinger #10203: sqlite3.Row doesn't support sequence protocol http://bugs.python.org/issue10203 closed by serhiy.storchaka #13742: Add a key parameter (like sorted) to heapq.merge http://bugs.python.org/issue13742 closed by rhettinger #14315: zipfile.ZipFile() unable to open zip File with a short extra h http://bugs.python.org/issue14315 closed by gregory.p.smith #14710: pkgutil.get_loader and find_loader fail when module is missing http://bugs.python.org/issue14710 closed by brett.cannon #15246: Line coverage for collectionts.abc.Set http://bugs.python.org/issue15246 closed by rhettinger #16774: Additional recipes for itertools docs http://bugs.python.org/issue16774 closed by rhettinger #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 closed by rhettinger #19925: Add unit test for spwd module http://bugs.python.org/issue19925 closed by serhiy.storchaka #20197: Support WebP image format detection in imghdr module http://bugs.python.org/issue20197 closed by serhiy.storchaka #21137: Better repr for threading.Lock() http://bugs.python.org/issue21137 closed by rhettinger #21343: os.path.relpath returns inconsistent types http://bugs.python.org/issue21343 closed by serhiy.storchaka #21376: asyncio docs refer to wrong TimeoutError http://bugs.python.org/issue21376 closed by haypo #21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists http://bugs.python.org/issue21402 closed by serhiy.storchaka #21434: python -3 documentation is outdated http://bugs.python.org/issue21434 closed by python-dev #21439: Numerous minor issues in Language Reference http://bugs.python.org/issue21439 closed by rhettinger #21454: asyncio's loop.connect_read_pipe makes pipes non-blocking cont http://bugs.python.org/issue21454 closed by haypo #21481: Argpase Namespace object methods __eq__ and __ne__ raise Type http://bugs.python.org/issue21481 closed by rhettinger #21493: Add test for ntpath.expanduser http://bugs.python.org/issue21493 closed by serhiy.storchaka #21513: speed up some ipaddress properties http://bugs.python.org/issue21513 closed by pitrou #21534: 404 on documentation download links http://bugs.python.org/issue21534 closed by benjamin.peterson #21546: int('\0') gives wrong error message http://bugs.python.org/issue21546 closed by terry.reedy #21555: gcmodule.c could use pytime.h http://bugs.python.org/issue21555 closed by pitrou #21558: Fix a typo in the contextlib docs http://bugs.python.org/issue21558 closed by rhettinger #21562: curses getsxy() should be curses getxy() in https://docs.pytho http://bugs.python.org/issue21562 closed by jcea #21565: multiprocessing: use contex-manager protocol for synchronizati http://bugs.python.org/issue21565 closed by neologix #21570: String being confused with datetime.datetime object. http://bugs.python.org/issue21570 closed by brandon #21575: list.sort() should show arguments in tutorial http://bugs.python.org/issue21575 closed by rhettinger #21584: Unraised overflow error in sqlite3.Row indexing http://bugs.python.org/issue21584 closed by serhiy.storchaka #21586: Minor error "How To Sockets" http://bugs.python.org/issue21586 closed by python-dev #21587: Tabs in source http://bugs.python.org/issue21587 closed by python-dev #21589: Use better idiom in unittest example http://bugs.python.org/issue21589 closed by ezio.melotti #21598: Is __getitem__ and __len__ implementations enough to make a us http://bugs.python.org/issue21598 closed by santa4nt #21602: smtplib.py socket.create_connection() also doesn't handle EINT http://bugs.python.org/issue21602 closed by neologix #21606: No visual feedback when entering japanese Characters in Entry http://bugs.python.org/issue21606 closed by ned.deily #21607: results of `zip` are displayed as ' http://bugs.python.org/issue21607 closed by SilentGhost #21609: Documentation for datetime.datetime uses microseconds instead http://bugs.python.org/issue21609 closed by Miquel.Garcia From chris.barker at noaa.gov Fri May 30 18:46:52 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 30 May 2014 09:46:52 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On Thu, May 29, 2014 at 4:43 AM, Nick Coghlan wrote: > For that last point, my interest is as much educational as it is in > easing the transition from Python 2. The parentheses in "print('Hello > world!')" mean introducing the idea of function calls early to explain > how it works, while being able to omit them makes it easier to gloss > over the distinction between statements and function calls initially > and then cover it later after the basics of flow control have been > nailed down. > I've been doing a lot of intro-to-python teaching lately (py2 so far...), and I understand this desire. IN fact, a lot of notes point to the fact that python's "hello world" is simply : print "hello world", rather than what is required in some languages to do something that simple. However, I also believe that when teaching it's better to introduce the "right way" to do something up front, rather than a "beginners' way", then later say, well, you really SHOULD do it this other way... So if we want our students to use print as a function, we might a well start them off that way. Saying that their very first easy program is: print("hello world") is fine -- they don't have to know or understand what a function call is -- they simply copy the syntax. And frankly, we get to simple function calls, VERY early in the program -- you can't really do anything without them... In fact, in my latest class, we've made an effort to introduce forward-thinking up front, even before we explain quite what it all means: use u"a string" to make a string you write a class like: class C(object): ... before we talk about subclassing, or what "object" is... just my $0.02 -Chris > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From wizzat at gmail.com Fri May 30 19:40:18 2014 From: wizzat at gmail.com (Mark Roberts) Date: Fri, 30 May 2014 10:40:18 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On Thu, May 29, 2014 at 9:30 AM, Terry Reedy wrote: > On 5/28/2014 6:26 PM, Glyph Lefkowitz wrote: > > I hope it's >> not controversial to say that most new Python code is still being >> written against Python 2.7 today; >> > > Given that Python 3 downloads now outnumber Python 2 downloads, I think > 'most' might be an overstatement. But I think it a moot point. How can you determine that Python3 downloads actually outnumber Python2 downloads? It seems that looking at Windows downloads (as I saw in a thread earlier this month) is fallacious at absolute best. Linux and OSX both ship with Python2, and most downloads happen via individual package management tools. Even including Python3.4 as a default Python3 doesn't mean that it's the default Python on the system. >From my perspective as an engineer and library maintainer: Pypi seems to indicate an overwhelming number of Python2 usage, and so do the job requisitions that I've seen. Furthermore, even Python based libraries for cutting edge technologies are still written in Python2 and later converted to Python3. I just don't understand how we (as a community) can make the assertion that most "new Python" is written in Python 3. What I'd really like to see is a Python 2.8 that makes sufficient changes to Python 2 that writing libraries which cross the boundary between 2 and 3 is relatively easy instead of a painful nightmarish chore. Because when push comes to shove, Python 2 support is still infinitely more important than Python 3. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri May 30 19:51:00 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 31 May 2014 03:51:00 +1000 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On Sat, May 31, 2014 at 3:40 AM, Mark Roberts wrote: > What I'd really like to see is a Python 2.8 that makes sufficient changes to > Python 2 that writing libraries which cross the boundary between 2 and 3 is > relatively easy instead of a painful nightmarish chore. Because when push > comes to shove, Python 2 support is still infinitely more important than > Python 3. That's of absolutely ZERO value to anyone who has to maintain support for Python 2.3 or even 2.7. Will this hypothetical 2.8 run, unchanged, all code written for 2.7? If not, all you've done is add a third branch to maintain, and solved nothing; and if it will, how much can you really change? ChrisA From breamoreboy at yahoo.co.uk Fri May 30 20:06:51 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 30 May 2014 19:06:51 +0100 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 30/05/2014 18:40, Mark Roberts wrote: > > What I'd really like to see is a Python 2.8 that makes sufficient > changes to Python 2 that writing libraries which cross the boundary > between 2 and 3 is relatively easy instead of a painful nightmarish > chore. Because when push comes to shove, Python 2 support is still > infinitely more important than Python 3. > > -Mark > What is the point of flogging a horse that's been dead for so long that it's already down to a skeleton? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From ethan at stoneleaf.us Fri May 30 19:01:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 30 May 2014 10:01:42 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: <5388B976.8050305@stoneleaf.us> On 05/30/2014 09:46 AM, Chris Barker wrote: > On Thu, May 29, 2014 at 4:43 AM, Nick Coghlan wrote: >> >> For that last point, my interest is as much educational as it is in >> easing the transition from Python 2. The parentheses in "print('Hello >> world!')" mean introducing the idea of function calls early to explain >> how it works, while being able to omit them makes it easier to gloss >> over the distinction between statements and function calls initially >> and then cover it later after the basics of flow control have been >> nailed down. > However, I also believe that when teaching it's better to introduce the "right way" to do something up front, rather > than a "beginners' way", then later say, well, you really SHOULD do it this other way... +1 Function calls are not that hard to understand. Anybody who has ever asked someone to do something for them should have a basic grasp of the nature of a function call. -- ~Ethan~ From ericsnowcurrently at gmail.com Fri May 30 21:39:12 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 30 May 2014 13:39:12 -0600 Subject: [Python-Dev] [Python-checkins] cpython: Issue #20383: Introduce importlib.util.module_from_spec(). In-Reply-To: <3ggFND0tdbz7SSy@mail.python.org> References: <3ggFND0tdbz7SSy@mail.python.org> Message-ID: On Fri, May 30, 2014 at 12:55 PM, brett.cannon wrote: > http://hg.python.org/cpython/rev/b26d021081d2 > changeset: 90915:b26d021081d2 > parent: 90913:69011f6ce573 > user: Brett Cannon > date: Fri May 30 14:55:29 2014 -0400 > summary: > Issue #20383: Introduce importlib.util.module_from_spec(). > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -581,20 +581,19 @@ > return loader > > > -def _load_module_shim(self, fullname): > +def _load_module_shim(spec, fullname): > """Load the specified module into sys.modules and return it. This should have stayed "self", no? -eric From bcannon at gmail.com Fri May 30 22:28:46 2014 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 30 May 2014 20:28:46 +0000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #20383: Introduce importlib.util.module_from_spec(). References: <3ggFND0tdbz7SSy@mail.python.org> Message-ID: On Fri May 30 2014 at 3:39:47 PM, Eric Snow wrote: > On Fri, May 30, 2014 at 12:55 PM, brett.cannon > wrote: > > http://hg.python.org/cpython/rev/b26d021081d2 > > changeset: 90915:b26d021081d2 > > parent: 90913:69011f6ce573 > > user: Brett Cannon > > date: Fri May 30 14:55:29 2014 -0400 > > summary: > > Issue #20383: Introduce importlib.util.module_from_spec(). > > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > > --- a/Lib/importlib/_bootstrap.py > > +++ b/Lib/importlib/_bootstrap.py > > @@ -581,20 +581,19 @@ > > return loader > > > > > > -def _load_module_shim(self, fullname): > > +def _load_module_shim(spec, fullname): > > """Load the specified module into sys.modules and return it. > > This should have stayed "self", no? > > Yes. Fixed with an added comment to explain why since it isn't obvious in isolation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri May 30 22:47:31 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 May 2014 22:47:31 +0200 Subject: [Python-Dev] Language Summit Follow-Up References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: <20140530224731.1abe5ede@fsol> On Wed, 28 May 2014 15:26:38 -0700 Glyph Lefkowitz wrote: > Backport 'yield from' to allow people to use Tulip and Tulip-compatible code, and to facilitate the development of Tulip-friendly libraries and a Tulip ecosystem. A robust Tulip ecosystem requires the participation of people who are not yet using Python 3. I was wondering whether you were trolling or not on this one. From a quality assurance point of view, adding major features to a bugfix branch is extremely destructive, so I'm strongly -1 on it. > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the documentation. More than one very experienced, well-known Python developer in this discussion has told me that they thought 2to3 was the blessed way to port their code, and it's no surprise that they think so, given that the first technique mentions is still 2to3. 2to3 is certainly fine if you are porting to 3.x without looking to keep your code 2.x-compatible. Until there's a better alternative, of course. So what we should do is better explain the choice (if you want to port your code to 3.x, use 2to3; if you want to maintain dual-compatible code, use six or something similar). Regards Antoine. From guido at python.org Sat May 31 00:39:22 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 30 May 2014 15:39:22 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: <20140530224731.1abe5ede@fsol> References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> <20140530224731.1abe5ede@fsol> Message-ID: 2to3 is poorly named. With different fixers it is a fine tool for converting 2-only code to 2-and-3 straddling code. Even when using six, you need to do things like convert print statements to print() calls with future import, use 'as' in except clauses, and so on. On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou wrote: > On Wed, 28 May 2014 15:26:38 -0700 > Glyph Lefkowitz wrote: > > Backport 'yield from' to allow people to use Tulip and Tulip-compatible > code, and to facilitate the development of Tulip-friendly libraries and a > Tulip ecosystem. A robust Tulip ecosystem requires the participation of > people who are not yet using Python 3. > > I was wondering whether you were trolling or not on this one. > From a quality assurance point of view, adding major features to a > bugfix branch is extremely destructive, so I'm strongly -1 on it. > > > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the > documentation. More than one very experienced, well-known Python developer > in this discussion has told me that they thought 2to3 was the blessed way > to port their code, and it's no surprise that they think so, given that the > first technique mentions > is still 2to3. > > 2to3 is certainly fine if you are porting to 3.x without looking to > keep your code 2.x-compatible. Until there's a better alternative, of > course. > So what we should do is better explain the choice (if you want to port > your code to 3.x, use 2to3; if you want to maintain dual-compatible > code, use six or something similar). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 31 03:46:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 31 May 2014 11:46:33 +1000 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> <20140530224731.1abe5ede@fsol> Message-ID: On 31 May 2014 08:42, "Guido van Rossum" wrote: > > 2to3 is poorly named. With different fixers it is a fine tool for converting 2-only code to 2-and-3 straddling code. Even when using six, you need to do things like convert print statements to print() calls with future import, use 'as' in except clauses, and so on. Both modernize & futurize build on lib2to3 to do the heavy lifting - they don't reinvent that wheel, they just change which fixes are applied by default and add a few more of their own. However, even if going straight to Python 3, I suspect folks would still be better off doing something like: * futurize --stage1 (migrates to 2.6+, but adds no new runtime dependencies - the output should be fairly idiomatic Python 2.6/7 code) * 2to3 (wholesale migration to Python 3) The trick, though, is that the granularity of that approach is at the process level - the entire application must be converted at once. You also can't start your own migration until all your dependencies are available in Python 3. By contrast, migration via the common subset can be incremental and opportunistic: * the granularity of migration is individual modules, rather than entire processes, so you can make a low risk gradual transition, even without a comprehensive regression test suite * you initially stay on Python 2, so you can start whenever is convenient for you, rather than having to wait for all your dependencies Cheers, Nick. > > > On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou wrote: >> >> On Wed, 28 May 2014 15:26:38 -0700 >> Glyph Lefkowitz wrote: >> > Backport 'yield from' to allow people to use Tulip and Tulip-compatible code, and to facilitate the development of Tulip-friendly libraries and a Tulip ecosystem. A robust Tulip ecosystem requires the participation of people who are not yet using Python 3. >> >> I was wondering whether you were trolling or not on this one. >> From a quality assurance point of view, adding major features to a >> bugfix branch is extremely destructive, so I'm strongly -1 on it. >> >> > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the documentation. More than one very experienced, well-known Python developer in this discussion has told me that they thought 2to3 was the blessed way to port their code, and it's no surprise that they think so, given that the first technique mentions is still 2to3. >> >> 2to3 is certainly fine if you are porting to 3.x without looking to >> keep your code 2.x-compatible. Until there's a better alternative, of >> course. >> So what we should do is better explain the choice (if you want to port >> your code to 3.x, use 2to3; if you want to maintain dual-compatible >> code, use six or something similar). >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 31 04:05:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 31 May 2014 12:05:06 +1000 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 31 May 2014 02:47, "Chris Barker" wrote: > However, I also believe that when teaching it's better to introduce the "right way" to do something up front, rather than a "beginners' way", then later say, well, you really SHOULD do it this other way... So if we want our students to use print as a function, we might a well start them off that way. Saying that their very first easy program is: > > print("hello world") > > is fine -- they don't have to know or understand what a function call is -- they simply copy the syntax. And frankly, we get to simple function calls, VERY early in the program -- you can't really do anything without them... OK, that's a fair criticism of that particular argument, so I'll stop using it. There are others, though: * improve consistency between IPython's existing implied call support and vanilla Python 3.5+ * retroactively make a lot of Python 2 examples also valid for Python 3.5+ * ease the migration to Python 3 for ad hoc utility scripts (which often use print heavily) * reduce the syntactic noise in migration patches (although large applications tend to use print less than scripts do) * largely eliminate one of the *human* barriers to migration from Python 2 (forgetting the parens for print at the interactive console and in ad hoc scripts) I already have too many other things on my todo list to work this up into a full PEP, but the proof of concept (along with IPython's existing support) shows there's no *technical* barrier to adding the feature. It's a question of whether or not we *want* to blur the boundary between simple statements and simple imperative function calls like that. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 31 04:32:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 31 May 2014 12:32:51 +1000 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On 31 May 2014 03:42, "Mark Roberts" wrote: > > What I'd really like to see is a Python 2.8 that makes sufficient changes to Python 2 that writing libraries which cross the boundary between 2 and 3 is relatively easy instead of a painful nightmarish chore. That's what projects like python-future are for. Python 2.8 wouldn't help, since most folks still target 2.6 for compatibility with stable Linux platforms like RHEL, CentOS, Debian Stable & Ubuntu LTS. This is a key point folks often miss in these discussions: even putting Python 3 aside entirely, the migration of the overall ecosystem from Python 2.6 to Python *2.7* is not yet finished. RHEL 7 (which uses 2.7 as the system Python) is currently only available as a release candidate, and the same necessarily holds true for its downstream rebuilds like CentOS. We saw this happen with Python 2.4 as well: for a lot of library developers, the day CentOS 6 finally landed (with Python 2.6 as the system Python) was the day they finally decided to drop support for Python 2.4. It's that slow adoption cycle for new feature releases *within* the Python 2 series that means the effort that would be needed to create a Python 2.8 release is better put into tools and utilities that people can use *now* (like PyPI backports of standard library modules, python-future and the "pymigrate" utility Steve Dower suggested and Eric Snow has started working on), tweaks to 2.7 itself (like PEP 466 and a possible future backport of the ensurepip changes) and Python 3 changes that improve both Python 3 *and* the subset it shares with Python 2 (like the restoration of binary interpolation support approved for 3.5). Cheers, Nick. P.S. I've written more about adoption cycles for new Python versions at http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#wouldn-t-a-python-2-8-release-help-ease-the-transition -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat May 31 05:32:04 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 May 2014 23:32:04 -0400 Subject: [Python-Dev] Updating turtle.py Message-ID: I have two areas of questions about updating turtle.py. First the module itself, then a turtle tracker issue versus code cleanup policies. A. Unlike most stdlib modules, turtle is copyrighted and licensed by an individual. ''' # turtle.py: a Tkinter based turtle graphics module for Python # Version 1.1b - 4. 5. 2009 # Copyright (C) 2006 - 2010 Gregor Lingl # email: glingl at aon.at ''' I am not sure what the copyright covers other than the exact text contributed, with updates, by Gregor. It certainly does not cover the API and whatever code he copied from the previous version (unless that was also by him, and I have no idea how much he copied when reimplementing). I don't think it should cover additions made by others either. Should there be another line to cover these? ''' # Permission is granted to anyone to use this software for any purpose, # including commercial applications, and to alter it and redistribute it # freely, subject to the following restrictions: # # 1. The origin of this software must not be misrepresented; you must not # claim that you wrote the original software. If you use this software # in a product, an acknowledgment in the product documentation would be # appreciated but is not required. # 2. Altered source versions must be plainly marked as such, and must not be # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. ''' As to point 2, the source has been altered a bit (by others) but it is not marked as such. How should it be? ''' _ver = "turtle 1.1b- - for Python 3.1 - 4. 5. 2009" ''' Obsolete; delete or alter (how)? When this replaced the previous turtle.py, it was considered 'owned' by Gregor in that changes had to go through him. However, he became inactive soon after and maintenance ceased. There has been only one turtle-specific code change that I know of (by Ned Daily, a month ago, for OSX. So is turtle.py unpatchable by anyone or fair game for anyone? A particular example: Gregor added intermediate layers to isolate turtle from tkinter. (He had a plan to add other backends, but I believe he abandoned that.) If someone wanted to reduce the layering and make the code easier to understand and maintain, while speeding execution, would that be allowed now? B. Lets assuming that turtle.py is, at least to some degree, fair game for fixes and enhancements. PSF Python PyLadies (Jessica Keller, Lynn Root) are participating in the 2014 GNOME Outreach Program for Women (OPW) https://wiki.python.org/moin/OPW/2014 . One of the projects (bottem of that page) is Graphical Python, in particular Turtle. A few days ago, Jessica posted http://bugs.python.org/issue21573 Clean up turtle.py code formatting "Lib/turtle.py has some code formatting issues. Let's clean them up to make the module easier to read as interns start working on it this summer." She want to follow cleanup with code examination, fixes, and enhancements. Responding today, I cautioned that clean-up only patches, such as she apparently would like to start with, are not in favor. But I cannot say how the policy applies to her proposal. Does the 'promise' of later work on the code make preliminary clean-up ok? Since she only marked the issue for 3.5, I also cautioned that 3.5-only cleanups would make fixing bugs in other issues harder. Is the code clean-up policy the same for all branches? Test? you ask?. There are apparently no unittests for turtle. On the other hand, turtledemo tests the overall function of the module and of many (most) of the top-level turtle functions. -- Terry Jan Reedy From guido at python.org Sat May 31 05:38:14 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 30 May 2014 20:38:14 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: On Fri, May 30, 2014 at 7:05 PM, Nick Coghlan wrote: > I already have too many other things on my todo list to work this up into > a full PEP, but the proof of concept (along with IPython's existing > support) shows there's no *technical* barrier to adding the feature. > That doesn't follow. A PoC and even IPython can ignore all kinds of *technical* issues (e.g. giving code that's valid today a different meaning). Please drop this idea or at least take it out of this thread and focus on what *can* and *should* be done, in particular adding warnings with good hints about what went wrong when Python 3 encounters what appears to be a print statement. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 31 05:39:15 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 30 May 2014 20:39:15 -0700 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> <20140530224731.1abe5ede@fsol> Message-ID: Right. The point is that the current HOWTO gives information that is not useful for most people who are tasked with a port. On Fri, May 30, 2014 at 6:46 PM, Nick Coghlan wrote: > > On 31 May 2014 08:42, "Guido van Rossum" wrote: > > > > 2to3 is poorly named. With different fixers it is a fine tool for > converting 2-only code to 2-and-3 straddling code. Even when using six, you > need to do things like convert print statements to print() calls with > future import, use 'as' in except clauses, and so on. > > Both modernize & futurize build on lib2to3 to do the heavy lifting - they > don't reinvent that wheel, they just change which fixes are applied by > default and add a few more of their own. > > However, even if going straight to Python 3, I suspect folks would still > be better off doing something like: > > * futurize --stage1 (migrates to 2.6+, but adds no new runtime > dependencies - the output should be fairly idiomatic Python 2.6/7 code) > * 2to3 (wholesale migration to Python 3) > > The trick, though, is that the granularity of that approach is at the > process level - the entire application must be converted at once. You also > can't start your own migration until all your dependencies are available in > Python 3. > > By contrast, migration via the common subset can be incremental and > opportunistic: > > * the granularity of migration is individual modules, rather than entire > processes, so you can make a low risk gradual transition, even without a > comprehensive regression test suite > * you initially stay on Python 2, so you can start whenever is convenient > for you, rather than having to wait for all your dependencies > > Cheers, > Nick. > > > > > > > On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou > wrote: > >> > >> On Wed, 28 May 2014 15:26:38 -0700 > >> Glyph Lefkowitz wrote: > >> > Backport 'yield from' to allow people to use Tulip and > Tulip-compatible code, and to facilitate the development of Tulip-friendly > libraries and a Tulip ecosystem. A robust Tulip ecosystem requires the > participation of people who are not yet using Python 3. > >> > >> I was wondering whether you were trolling or not on this one. > >> From a quality assurance point of view, adding major features to a > >> bugfix branch is extremely destructive, so I'm strongly -1 on it. > >> > >> > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the > documentation. More than one very experienced, well-known Python developer > in this discussion has told me that they thought 2to3 was the blessed way > to port their code, and it's no surprise that they think so, given that the > first technique mentions > is still 2to3. > >> > >> 2to3 is certainly fine if you are porting to 3.x without looking to > >> keep your code 2.x-compatible. Until there's a better alternative, of > >> course. > >> So what we should do is better explain the choice (if you want to port > >> your code to 3.x, use 2to3; if you want to maintain dual-compatible > >> code, use six or something similar). > >> > >> Regards > >> > >> Antoine. > >> > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat May 31 10:09:19 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 31 May 2014 17:09:19 +0900 Subject: [Python-Dev] Updating turtle.py In-Reply-To: References: Message-ID: <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > As to point 2, the source has been altered a bit (by others) but it is > not marked as such. How should it be? I would suggest adding """ Based on turtle 1.1b for Python 3.1 (4.5.2009) by Gregor Lingl. This is a revised version including changes from the Python community. Change history is available from the Python repository: . """ Including the URL is questionable as updates are likely to be overlooked if the repo should ever move. Including the first sentence is a matter of taste. > ''' > _ver = "turtle 1.1b- - for Python 3.1 - 4. 5. 2009" > ''' > Obsolete; delete or alter (how)? Delete definition and references, or replace with more generic wording, I think. E.g., '_ver = "turtle for Python 3"'. It's a pain to maintain as is. It's not information for determining copyright, so is covered by Gregor's permissive license. > When this replaced the previous turtle.py, it was considered 'owned' by > Gregor in that changes had to go through him. However, he became > inactive soon after and maintenance ceased. There has been only one > turtle-specific code change that I know of (by Ned Daily, a month ago, > for OSX. So is turtle.py unpatchable by anyone or fair game for anyone? Legally, it's fair game. Socially, it's a matter of project policy. AFAICT Python policy is that someone should ask Gregor (a precedent is the Fredrik Lundh/ElementTree case). AIUI, there's been a five-year span since Gregor's been active, so I would think it's basically a matter of courtesy. Most likely he's not interested in returning as maintainer, or he can't be contacted with reasonable effort. Then it's open to anyone. If he's interested in maintaining control but obstructive toward third party contributions, that's messy but in the end the PSF, or its delegates, as owners of the repo have legal control, and social legitimacy to exercise it as seems best for the project. If the PSF does "go over Gregor's head" to open up the file in trunk for modifications, the wording above might want to change to "This fork includes changes from ...". An alternative would be to fork into a new module with a different name. That can be done at any time, with the only downsides being redundancy and potential bad will between Python and Gregor Lingl. > B. Lets assuming that turtle.py is, at least to some degree, fair game > for fixes and enhancements. PSF Python PyLadies (Jessica Keller, Lynn > Root) are participating in the 2014 GNOME Outreach Program for Women > (OPW) https://wiki.python.org/moin/OPW/2014 . One of the projects > (bottem of that page) is Graphical Python, in particular Turtle. I don't think there is any issue at all here. Legally there's no barrier at all to working on Turtle or pushing changes anywhere. Socially, there's no barrier to anything but pushing to release branches (including trunk). To the extent that OPW (GSoC, etc) requires integration with the parent project, they can push to a hg.python.org branch pending clarification of the maintainership issue. Best of all, you've identified volunteers for searching for Gregor! IMHO YMMV Steve From stephen at xemacs.org Sat May 31 10:18:59 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 31 May 2014 17:18:59 +0900 Subject: [Python-Dev] Language Summit Follow-Up In-Reply-To: References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com> Message-ID: <87wqd25pjg.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Barker writes: > that way. Saying that their very first easy program is: > > print("hello world") > > is fine I have had similar experience on a small scale. Also I've been teaching R recently. The students who know Python (Python 3, we don't have backward compatibility issues in our work) got the concept that all "real work" in R is done by functions, immediately. Those who don't, have trouble with the concept that "help" and "q" (for "quit") need parentheses to get them to work. Unfortunately R doesn't have Python's Easter Eggs, so they get a code dump rather than help when they type the bare names. From martin at v.loewis.de Sat May 31 20:05:55 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 31 May 2014 20:05:55 +0200 Subject: [Python-Dev] Updating turtle.py In-Reply-To: References: Message-ID: <538A1A03.4020405@v.loewis.de> Am 31.05.14 05:32, schrieb Terry Reedy: > I have two areas of questions about updating turtle.py. First the module > itself, then a turtle tracker issue versus code cleanup policies. > > A. Unlike most stdlib modules, turtle is copyrighted and licensed by an > individual. > ''' > # turtle.py: a Tkinter based turtle graphics module for Python > # Version 1.1b - 4. 5. 2009 > # Copyright (C) 2006 - 2010 Gregor Lingl > # email: glingl at aon.at > ''' > I am not sure what the copyright covers other than the exact text > contributed, with updates, by Gregor. It certainly does not cover the > API and whatever code he copied from the previous version (unless that > was also by him, and I have no idea how much he copied when > reimplementing). I don't think it should cover additions made by others > either. Should there be another line to cover these? He should provide a contributor form, covering his past contributions. Would you like to contact him about this? Adding a license up-front (as you propose) is counter-productive; the author may not agree to your specific licensing terms. If he was unwilling to agree to the contributor form (which I doubt, knowing him personally), the only option would be to remove the code from the distribution. > _ver = "turtle 1.1b- - for Python 3.1 - 4. 5. 2009" > ''' > Obsolete; delete or alter (how)? Delete. It's a private variable, and the true version number is maintained by Mercurial, and not in the code itself (or else is the version of Python it ships with). > A particular example: Gregor added intermediate layers to isolate turtle > from tkinter. (He had a plan to add other backends, but I believe he > abandoned that.) If someone wanted to reduce the layering and make the > code easier to understand and maintain, while speeding execution, would > that be allowed now? It would be good if there would be a new maintainer of the module. Maybe Gregor would be willing to reactivate his contributions when asked, maybe he would be willing to hand it over to somebody else. A maintainer would have the ultimate say in architectural changes. Without a maintainer, I'd rather not make architectural changes. > Responding today, I cautioned that clean-up only patches, such as she > apparently would like to start with, are not in favor. I would not say that. I recall that I asked Gregor to make a number of style changes before he submitted the code, and eventually agreed to the code when I thought it was "good enough". However, continuing on that path sounds reasonable to me. It is the mixing of clean-up patches with functional changes that is not in favor. > Since she only marked the issue for 3.5, I also cautioned that 3.5-only > cleanups would make fixing bugs in other issues harder. Is the code > clean-up policy the same for all branches? I don't think that we should be taken hostage by merging restrictions of the DVCS - we switched to the DVCS precisely with the promise that merging would be easier. Given the number of bug fixes that the turtle module has seen, I'd suggest that it is less work to restrict cleanup to 3.5, and then deal with any forward-porting of bug fixing when it actually happens. Regards, Martin From martin at v.loewis.de Sat May 31 20:09:30 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 31 May 2014 20:09:30 +0200 Subject: [Python-Dev] Updating turtle.py In-Reply-To: <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <538A1ADA.2060805@v.loewis.de> Am 31.05.14 10:09, schrieb Stephen J. Turnbull: > AFAICT Python policy is that someone should ask Gregor (a precedent is > the Fredrik Lundh/ElementTree case). AIUI, there's been a five-year > span since Gregor's been active, so I would think it's basically a > matter of courtesy. Most likely he's not interested in returning as > maintainer, or he can't be contacted with reasonable effort. I would not be so sure about this, see http://python4kids.net/turtle.html Regards, Martin