From report at bugs.python.org  Sun Jan  1 00:50:05 2012
From: report at bugs.python.org (Berker Peksag)
Date: Sat, 31 Dec 2011 23:50:05 +0000
Subject: [issue13641] decoding functions in the base64 module could accept
	unicode strings
In-Reply-To: <1324386392.48.0.968435833624.issue13641@psf.upfronthosting.co.za>
Message-ID: <1325375405.88.0.30215814654.issue13641@psf.upfronthosting.co.za>


Changes by Berker Peksag <berker.peksag at gmail.com>:


----------
keywords: +patch
nosy: +berkerpeksag
Added file: http://bugs.python.org/file24119/issue13641_v1.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13641>
_______________________________________

From report at bugs.python.org  Sun Jan  1 05:09:38 2012
From: report at bugs.python.org (Jonathan Rogers)
Date: Sun, 01 Jan 2012 04:09:38 +0000
Subject: [issue12930] reindent.py inserts spaces in multiline literals
In-Reply-To: <1315405777.71.0.767109553281.issue12930@psf.upfronthosting.co.za>
Message-ID: <1325390978.63.0.544571687415.issue12930@psf.upfronthosting.co.za>


Jonathan Rogers <jonathanrrogers at gmail.com> added the comment:

I don't think reindent.py should change any bytes inside string literals since it can't know anything about what those strings mean or how they'll be used by the program at run time. Unfortunately, it starts out by unconditionally calling the .expandtabs() method on each input line, so tab characters are lost. The only change to a string literal I can imagine that would be safe is to replace tab characters with '\t'.

I am trying to use reindent.py on Python source files which include triple-quoted, multi-line string literals containing makefile and Python snippets. In both cases, running reindent.py changes the meaning of of that contained in the literal.

----------
nosy: +Jonathan.Rogers

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12930>
_______________________________________

From report at bugs.python.org  Sun Jan  1 07:07:51 2012
From: report at bugs.python.org (Jonathan Rogers)
Date: Sun, 01 Jan 2012 06:07:51 +0000
Subject: [issue12930] reindent.py inserts spaces in multiline literals
In-Reply-To: <1315405777.71.0.767109553281.issue12930@psf.upfronthosting.co.za>
Message-ID: <1325398071.25.0.517038069341.issue12930@psf.upfronthosting.co.za>


Jonathan Rogers <jonathanrrogers at gmail.com> added the comment:

Rather than expanding tab characters inside string literals, it's safer to replace them with '\t'.

----------
Added file: http://bugs.python.org/file24120/save_strings.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12930>
_______________________________________

From report at bugs.python.org  Sun Jan  1 09:22:32 2012
From: report at bugs.python.org (Devin Jeanpierre)
Date: Sun, 01 Jan 2012 08:22:32 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
Message-ID: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>


New submission from Devin Jeanpierre <jeanpierreda at gmail.com>:

What follows is a copy-paste of a shell session. Notice that at the end, rather than being inside the online help utility, I'm still in the interactive interpreter. I was able to duplicate this on python3.2, python2.7, and python2.6 (verifying it on other versions would have required installing them). Reading the source in trunk, there is nothing that looks like it actually should run this interactive help session. It's just missing.

I guess nobody used this, eh? I've attached a patch that should fix it. I'm not sure how you want to handle adding a test for this, so please advise me on that.

-----

>>> help('help')

Welcome to Python 3.2!  This is the online help utility.

If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.

Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules.  To quit this help utility and
return to the interpreter, just type "quit".

To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics".  Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".

>>>

----------
components: Library (Lib)
files: r74214.diff
keywords: patch
messages: 150427
nosy: Devin Jeanpierre
priority: normal
severity: normal
status: open
title: pydoc help (or help('help')) claims to run a help utility; does nothing
versions: Python 2.6, Python 2.7, Python 3.2
Added file: http://bugs.python.org/file24121/r74214.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Sun Jan  1 12:18:45 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 11:18:45 +0000
Subject: [issue13683] Docs in Python 3:raise statement mistake
In-Reply-To: <1325259254.7.0.957455919083.issue13683@psf.upfronthosting.co.za>
Message-ID: <E1RhJR5-0005Ot-LG@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 420e01156272 by Sandro Tosi in branch '3.2':
Issue #13683: raise with no exception in scope throws a RuntimeError; fix by Ramchandra Apte
http://hg.python.org/cpython/rev/420e01156272

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13683>
_______________________________________

From report at bugs.python.org  Sun Jan  1 12:19:22 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 11:19:22 +0000
Subject: [issue13683] Docs in Python 3:raise statement mistake
In-Reply-To: <1325259254.7.0.957455919083.issue13683@psf.upfronthosting.co.za>
Message-ID: <1325416762.55.0.948458227648.issue13683@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13683>
_______________________________________

From report at bugs.python.org  Sun Jan  1 12:57:28 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 11:57:28 +0000
Subject: [issue13690] Add DEBUG flag to documentation of re.compile
In-Reply-To: <1325355977.96.0.0770563449241.issue13690@psf.upfronthosting.co.za>
Message-ID: <E1RhK2Z-0008WQ-8B@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9aebb4d07ddf by Sandro Tosi in branch '2.7':
Issue #13690: add re.DEBUG; patch by Filip Gruszczy?ski
http://hg.python.org/cpython/rev/9aebb4d07ddf

New changeset f4a9c7cf98dd by Sandro Tosi in branch '3.2':
Issue #13690: add re.DEBUG; patch by Filip Gruszczy?ski
http://hg.python.org/cpython/rev/f4a9c7cf98dd

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13690>
_______________________________________

From report at bugs.python.org  Sun Jan  1 12:58:49 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 11:58:49 +0000
Subject: [issue13690] Add DEBUG flag to documentation of re.compile
In-Reply-To: <1325355977.96.0.0770563449241.issue13690@psf.upfronthosting.co.za>
Message-ID: <1325419129.75.0.0185807998933.issue13690@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python, sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13690>
_______________________________________

From report at bugs.python.org  Sun Jan  1 13:03:31 2012
From: report at bugs.python.org (Henrik Holmboe)
Date: Sun, 01 Jan 2012 12:03:31 +0000
Subject: [issue13692] 2to3 mangles from . import frobnitz
Message-ID: <1325419411.86.0.637139477325.issue13692@psf.upfronthosting.co.za>


New submission from Henrik Holmboe <henrik at holmboe.se>:

It seems that 2to3 mangles::

 from . import frobnitz

into::

 from ... import frobnitz

This was noticed in the port of ipython to py3k. See https://github.com/ipython/ipython/issues/1197

----------
components: 2to3 (2.x to 3.x conversion tool)
messages: 150430
nosy: holmbie
priority: normal
severity: normal
status: open
title: 2to3 mangles from . import frobnitz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13692>
_______________________________________

From report at bugs.python.org  Sun Jan  1 14:01:53 2012
From: report at bugs.python.org (Thomas Kluyver)
Date: Sun, 01 Jan 2012 13:01:53 +0000
Subject: [issue13692] 2to3 mangles from . import frobnitz
In-Reply-To: <1325419411.86.0.637139477325.issue13692@psf.upfronthosting.co.za>
Message-ID: <1325422913.24.0.359192722349.issue13692@psf.upfronthosting.co.za>


Thomas Kluyver <takowl at gmail.com> added the comment:

A couple of things to note:

- This was with the Python 3.1 implementation of 2to3 - the problem doesn't appear with the Python 3.2 version.
- The import statement in question was inside a method definition. I wonder if the extra two dots correspond to the class and method scopes.

----------
nosy: +takluyver

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13692>
_______________________________________

From report at bugs.python.org  Sun Jan  1 16:09:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 01 Jan 2012 15:09:32 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <1325430572.21.0.986328186401.issue13645@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> You could add the requisite path_size() method to get the value, and
> assume 0 means unsupported

I thought:
- calling two methods means two stat calls per file, this could be slightly inefficient
- if future extensions of the import mechanism require yet more stat information (for example owner or chmod), it will be yet another bunch of stat'ing methods to create

(besides, calling int() on the timestamp is a loss of information, I don't understand why this must be done in path_mtime() rather than let the consumer do whatever it wants with the higher-precision timestamp)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:06:06 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 17:06:06 +0000
Subject: [issue13680] Aifc comptype write fix
In-Reply-To: <1325237015.55.0.920988533262.issue13680@psf.upfronthosting.co.za>
Message-ID: <E1RhOrD-0007ZK-Fg@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a9cdc3ff2b8e by Sandro Tosi in branch '3.2':
Issue #13680: add lowecase compression type to write header; patch by Oleg Plakhotnyuk
http://hg.python.org/cpython/rev/a9cdc3ff2b8e

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13680>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:07:42 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 17:07:42 +0000
Subject: [issue13680] Aifc comptype write fix
In-Reply-To: <1325237015.55.0.920988533262.issue13680@psf.upfronthosting.co.za>
Message-ID: <1325437662.22.0.477687385325.issue13680@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13680>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:25:00 2012
From: report at bugs.python.org (kxroberto)
Date: Sun, 01 Jan 2012 17:25:00 +0000
Subject: [issue13693] email.Header.Header incorrect/non-smart on international
	charset address fields
Message-ID: <1325438700.75.0.319524788564.issue13693@psf.upfronthosting.co.za>


New submission from kxroberto <kxroberto at users.sourceforge.net>:

the email.* package seems to over-encode international charset address fields - resulting even in display errors in the receivers reader - , 
when message header composition is done as recommended in http://docs.python.org/library/email.header.html 

Python 2.7.2
>>> e=email.Parser.Parser().parsestr(getcliptext())
>>> e['From']
'=?utf-8?q?Martin_v=2E_L=C3=B6wis?= <report at bugs.python.org>'
# note the par
>>> email.Header.decode_header(_)
[('Martin v. L\xc3\xb6wis', 'utf-8'), ('<report at bugs.python.org>', None)]
# unfortunately there is no comfortable function for this:
>>> u='Martin v. L\xc3\xb6wis'.decode('utf8') + ' <report at bugs.python.org>'
>>> u
u'Martin v. L\xf6wis <report at bugs.python.org>'
>>> msg=email.Message.Message()
>>> msg['From']=u
>>> msg.as_string()
'From: =?utf-8?b?TWFydGluIHYuIEzDtndpcyA8cmVwb3J0QGJ1Z3MucHl0aG9uLm9yZz4=?=\n\n'
>>> msg['From']=str(u)
>>> msg.as_string()
'From: =?utf-8?b?TWFydGluIHYuIEzDtndpcyA8cmVwb3J0QGJ1Z3MucHl0aG9uLm9yZz4=?=\nFrom: Martin v. L\xf6wis <report at bugs.python.org>\n\n'
>>> msg['From']=email.Header.Header(u)
>>> msg.as_string()
'From: =?utf-8?b?TWFydGluIHYuIEzDtndpcyA8cmVwb3J0QGJ1Z3MucHl0aG9uLm9yZz4=?=\nFrom: Martin v. L\xf6wis <report at bugs.python.org>\nFrom: =?utf-8?b?TWFydGluIHYuIEzDtndpcyA8cmVwb3J0QGJ1Z3MucHl0aG9uLm9yZz4=?=\n\n'
>>> 

(BTW: strange is that multiple msg['From']=... _assignments_ end up as multiple additions !???   also msg renders 8bit header lines without warning/error or auto-encoding, while it does auto on unicode!??)

Whats finally arriving at the receiver is typically like:

From: "=?utf-8?b?TWFydGluIHYuIEzDtndpcyA8cmVwb3J0QGJ1Z3MucHl0aG9uLm9yZz4=?=" <report at bugs.python.org>

because the servers seem to want the address open, they extract the address and _add_ it (duplicating) as ASCII. => error

I have not found any emails in my archives where address header fields are so over-encoded like python does. Even in non-address fields mostly only those words/groups are encoded which need it.

I assume the sophisticated/high-level looking email.* package doesn't expect that the user fiddles things together low-level? with parseaddr, re.search, make_header Header.encode , '.join ... Or is it indeed (undocumented) so? IMHO it should be auto-smart enough.

Note: there is a old deprecated function mimify.mime_encode_header which seemed to try to cautiously auto-encode correct/sparsely (but actually fails too on all examples tried).

----------
components: Library (Lib)
messages: 150434
nosy: kxroberto
priority: normal
severity: normal
status: open
title: email.Header.Header incorrect/non-smart on international charset address fields
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13693>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:35:42 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 17:35:42 +0000
Subject: [issue13640] add mimetype for application/vnd.apple.mpegurl
In-Reply-To: <1324345079.76.0.172332572268.issue13640@psf.upfronthosting.co.za>
Message-ID: <E1RhPJs-0000ns-LJ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 7926f594e333 by Sandro Tosi in branch 'default':
Issue #13640: add application/vnd.apple.mpegurl MIME type; (partial) patch by Hiroaki Kawai
http://hg.python.org/cpython/rev/7926f594e333

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13640>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:36:27 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 17:36:27 +0000
Subject: [issue13640] add mimetype for application/vnd.apple.mpegurl
In-Reply-To: <1324345079.76.0.172332572268.issue13640@psf.upfronthosting.co.za>
Message-ID: <1325439387.96.0.302530586852.issue13640@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

I've also added 'm3u' which is the companion of 'm3u8'.

----------
nosy: +sandro.tosi
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13640>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:38:49 2012
From: report at bugs.python.org (kxroberto)
Date: Sun, 01 Jan 2012 17:38:49 +0000
Subject: [issue10839] email module should not allow some header field
	repetitions
In-Reply-To: <1294263603.86.0.308955870327.issue10839@psf.upfronthosting.co.za>
Message-ID: <1325439529.8.0.861444843746.issue10839@psf.upfronthosting.co.za>


kxroberto <kxroberto at users.sourceforge.net> added the comment:

I think really ill/strange is that kind of item _assignments_  do _add_ multiple.

If   msg[field] = xy    would just add-first/replace-frist , and only msg.add_xxxx/.append(field, xy)  would  add multiples that would be clear and understandable/readable. 
(The sophisticated check dictionary is unnecessary IMHO, I don't expect the class to be ever smart enough for a full RFC checklist.)

e.g. I remember a bug like

msg[field] = xy
if special_condition:
     msg[field] = abc   # just wanted a alternative


Never ever expected a double header here!

"="  with adding behavior is absurd IMHO. Certainly doesn't allow readable code.

----------
nosy: +kxroberto

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10839>
_______________________________________

From report at bugs.python.org  Sun Jan  1 18:57:28 2012
From: report at bugs.python.org (Jay Freeman (saurik))
Date: Sun, 01 Jan 2012 17:57:28 +0000
Subject: [issue2481] locale.strxfrm does not work with Unicode strings
In-Reply-To: <1206455635.63.0.674389192114.issue2481@psf.upfronthosting.co.za>
Message-ID: <1325440648.47.0.681157073786.issue2481@psf.upfronthosting.co.za>


Jay Freeman (saurik) <saurik at saurik.com> added the comment:

Given that Python 3.x is still not ready for general use (and when this is discussed people make it quite clear that this is to be expected, and that a many year timeline was originally proposed for the Python 3.0 transition), it seems like this bug fix should have been backported to 2.x at some point in the last four years it has been open. :(

----------
nosy: +saurik

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2481>
_______________________________________

From report at bugs.python.org  Sun Jan  1 19:27:40 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 01 Jan 2012 18:27:40 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1325430572.21.0.986328186401.issue13645@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM1GHAkhqZC=t8UdttN-8R5Hr2GZPRZbEtW=_vN0sWO0Ww@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

The patch looks good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Sun Jan  1 19:57:08 2012
From: report at bugs.python.org (kxroberto)
Date: Sun, 01 Jan 2012 18:57:08 +0000
Subject: [issue13693] email.Header.Header incorrect/non-smart on international
	charset address fields
In-Reply-To: <1325438700.75.0.319524788564.issue13693@psf.upfronthosting.co.za>
Message-ID: <1325444228.14.0.542077111485.issue13693@psf.upfronthosting.co.za>


kxroberto <kxroberto at users.sourceforge.net> added the comment:

now I tried to render this address field header 

u'Name <abc\u03a3 at xy>, abc at ewf, "Name?" <weofij at fjeio>'

with 
h = email.Header.Header(continuation_ws='')
h.append ... / email.Header.make_header via these chunks:

[('Name <', us-ascii), ('abc\xce\xa3', utf-8), ('@xy>, abc at ewf, "', us-ascii), ('Name\xc3\x9f', utf-8), ('" <weofij at fjeio>', us-ascii)]

the outcome is:

'Name < =?utf-8?b?YWJjzqM=?= @xy>, abc at ewf, " =?utf-8?b?TmFtZcOf?=\n " <weofij at fjeio>'


(note: local part of email address can be utf too)

It seems to be impossible to avoid the erronous extra spaces from outside within that email.Header framework.
Thus I guess it was not possible up to now to decently format a beyond-ascii MIME message using the official email.Header mechanism? - even when pre-digesting things

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13693>
_______________________________________

From report at bugs.python.org  Sun Jan  1 21:18:23 2012
From: report at bugs.python.org (Petri Lehtinen)
Date: Sun, 01 Jan 2012 20:18:23 +0000
Subject: [issue13676] sqlite3: Zero byte truncates string contents
In-Reply-To: <1325161654.99.0.774888973707.issue13676@psf.upfronthosting.co.za>
Message-ID: <1325449103.14.0.937560597192.issue13676@psf.upfronthosting.co.za>


Petri Lehtinen <petri at digip.org> added the comment:

Attached an updated patch. The custom text_factory case is now fixed, and bytes, bytearray and custom factory are all tested.

I also added back the pysqlite_unicode_from_string() function, as this makes the patch a bit smaller. It also seems to me (only by looking at the code) that the sqlite3.OptimizedUnicode factory isn't currently working as documented.

Antoine: Do you happen to know what's the status of the OptimizeUnicode thingie? Has it been changed for a reason or is it just an error that happened during the py3k transition?

----------
Added file: http://bugs.python.org/file24122/sqlite3_zero_byte_v3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13676>
_______________________________________

From report at bugs.python.org  Sun Jan  1 21:21:17 2012
From: report at bugs.python.org (Petri Lehtinen)
Date: Sun, 01 Jan 2012 20:21:17 +0000
Subject: [issue13676] sqlite3: Zero byte truncates string contents
In-Reply-To: <1325161654.99.0.774888973707.issue13676@psf.upfronthosting.co.za>
Message-ID: <1325449277.5.0.962906558823.issue13676@psf.upfronthosting.co.za>


Changes by Petri Lehtinen <petri at digip.org>:


Removed file: http://bugs.python.org/file24122/sqlite3_zero_byte_v3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13676>
_______________________________________

From report at bugs.python.org  Sun Jan  1 21:21:38 2012
From: report at bugs.python.org (Petri Lehtinen)
Date: Sun, 01 Jan 2012 20:21:38 +0000
Subject: [issue13676] sqlite3: Zero byte truncates string contents
In-Reply-To: <1325161654.99.0.774888973707.issue13676@psf.upfronthosting.co.za>
Message-ID: <1325449298.74.0.151265361849.issue13676@psf.upfronthosting.co.za>


Petri Lehtinen <petri at digip.org> added the comment:

(Whoops, I didn't mean to change the magic source coding comment. Updating the patch once again.)

----------
Added file: http://bugs.python.org/file24123/sqlite3_zero_byte_v3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13676>
_______________________________________

From report at bugs.python.org  Sun Jan  1 22:54:26 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 21:54:26 +0000
Subject: [issue13594] Aifc markers write fix
In-Reply-To: <1323803286.63.0.487230833708.issue13594@psf.upfronthosting.co.za>
Message-ID: <E1RhTMG-0007A8-Rs@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset c7a4405835e8 by Sandro Tosi in branch '3.2':
Issue #13594: various fixes to aifc module; patch by Oleg Plakhotnyuk
http://hg.python.org/cpython/rev/c7a4405835e8

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13594>
_______________________________________

From report at bugs.python.org  Sun Jan  1 22:55:17 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 21:55:17 +0000
Subject: [issue13594] Aifc markers write fix
In-Reply-To: <1323803286.63.0.487230833708.issue13594@psf.upfronthosting.co.za>
Message-ID: <1325454917.17.0.140925771449.issue13594@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13594>
_______________________________________

From report at bugs.python.org  Sun Jan  1 23:13:37 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 01 Jan 2012 22:13:37 +0000
Subject: [issue13676] sqlite3: Zero byte truncates string contents
In-Reply-To: <1325449103.14.0.937560597192.issue13676@psf.upfronthosting.co.za>
Message-ID: <1325455954.4016.2.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Attached an updated patch. The custom text_factory case is now fixed,
> and bytes, bytearray and custom factory are all tested.

Thanks, looks good to me.

> Antoine: Do you happen to know what's the status of the
> OptimizeUnicode thingie? Has it been changed for a reason or is it
> just an error that happened during the py3k transition?

It looks obsolete in 3.x to me. If you look at the 2.7 source code, it
had a real meaning there. Probably we could simplify the 3.x source code
by removing that option (but better to do it in a separate patch).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13676>
_______________________________________

From report at bugs.python.org  Sun Jan  1 23:15:41 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 01 Jan 2012 22:15:41 +0000
Subject: [issue13641] decoding functions in the base64 module could accept
	unicode strings
In-Reply-To: <1324386392.48.0.968435833624.issue13641@psf.upfronthosting.co.za>
Message-ID: <1325456141.2.0.0985394015414.issue13641@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks for the patch, Berker. It seems a bit too simple, though. You should add some tests in Lib/test/test_base64.py and run them (using "./python -m test -v test_base64"), this will allow you to see if your changes are correct.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13641>
_______________________________________

From report at bugs.python.org  Sun Jan  1 23:43:02 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 01 Jan 2012 22:43:02 +0000
Subject: [issue13302] Clarification needed in C API arg parsing
In-Reply-To: <1320057763.71.0.58387430149.issue13302@psf.upfronthosting.co.za>
Message-ID: <E1RhU7I-0001Ze-AJ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b2b7104691c9 by Sandro Tosi in branch '2.7':
Issue #13302: backport part of 3ed28f28466f
http://hg.python.org/cpython/rev/b2b7104691c9

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13302>
_______________________________________

From report at bugs.python.org  Sun Jan  1 23:44:21 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 01 Jan 2012 22:44:21 +0000
Subject: [issue13302] Clarification needed in C API arg parsing
In-Reply-To: <1320057763.71.0.58387430149.issue13302@psf.upfronthosting.co.za>
Message-ID: <1325457861.61.0.808058917465.issue13302@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Thanks Antoine for the pointer.

----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
versions:  -Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13302>
_______________________________________

From report at bugs.python.org  Mon Jan  2 01:00:49 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 02 Jan 2012 00:00:49 +0000
Subject: [issue2481] locale.strxfrm does not work with Unicode strings
In-Reply-To: <1206455635.63.0.674389192114.issue2481@psf.upfronthosting.co.za>
Message-ID: <1325462449.77.0.919217753064.issue2481@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

saurik: can you propose a patch?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2481>
_______________________________________

From report at bugs.python.org  Mon Jan  2 03:15:16 2012
From: report at bugs.python.org (Matt Joiner)
Date: Mon, 02 Jan 2012 02:15:16 +0000
Subject: [issue13694] asynchronous connect in asyncore.dispatcher does not set
	addr
Message-ID: <1325470516.27.0.519900720027.issue13694@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

Patch attached

----------
components: Library (Lib)
files: dispatcher_connect_addr.patch
keywords: patch
messages: 150449
nosy: anacrolix
priority: normal
severity: normal
status: open
title: asynchronous connect in asyncore.dispatcher does not set addr
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24124/dispatcher_connect_addr.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13694>
_______________________________________

From report at bugs.python.org  Mon Jan  2 03:25:58 2012
From: report at bugs.python.org (Jay Freeman (saurik))
Date: Mon, 02 Jan 2012 02:25:58 +0000
Subject: [issue2481] locale.strxfrm does not work with Unicode strings
In-Reply-To: <1206455635.63.0.674389192114.issue2481@psf.upfronthosting.co.za>
Message-ID: <1325471158.62.0.959875868891.issue2481@psf.upfronthosting.co.za>


Jay Freeman (saurik) <saurik at saurik.com> added the comment:

I have attached a tested patch against Python-2.7.2.tgz (as I do not know how to use hg currently). It should be noted that I also am not 100% certain how the Python build environment works, but the way I added the wcsxfrm test was to add it to configure.in, then run autoheader and autoconf.

It also should be noted that the original code called strxfrm and did not check for an error result: neither does my new code (which is mostly based on formulaic modifications of the existing code in addition to educated guesses with regards to coding and formatting standards: feel free to change, obviously).

Finally, I noticed while working on this that --enable-unicode=no does not work (there is a check that enforces that it must be either ucs2 or ucs4): seems like an easy fix. That said, I ran into numerous other issues trying to make a non-Unicode build, and in the end gave up. My code looks like it should work, however, were someone to figure out how to build a non-Unicode Python 2.7.

----------
keywords: +patch
Added file: http://bugs.python.org/file24125/wcsxfrm.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2481>
_______________________________________

From report at bugs.python.org  Mon Jan  2 06:48:10 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 02 Jan 2012 05:48:10 +0000
Subject: [issue13695] "type specific" to "type-specific"
Message-ID: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>


New submission from Bo?tjan Mejak <bostjan.mejak at gmail.com>:

Please visit the following link and fix the below text:

http://docs.python.org/library/unittest.html#unittest.TestCase.assertEqual

Changed in version 2.7: Added the automatic calling of >>>type-specific<<< equality function.

Just add the hyphen. Thanks.

----------
assignee: docs at python
components: Documentation
messages: 150451
nosy: Retro, docs at python
priority: normal
severity: normal
status: open
title: "type specific" to "type-specific"
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan  2 13:19:53 2012
From: report at bugs.python.org (Piotr Dobrogost)
Date: Mon, 02 Jan 2012 12:19:53 +0000
Subject: [issue2698] Extension module build fails for MinGW: missing
	vcvarsall.bat
In-Reply-To: <1209230424.66.0.154562223128.issue2698@psf.upfronthosting.co.za>
Message-ID: <1325506793.5.0.592977725239.issue2698@psf.upfronthosting.co.za>


Piotr Dobrogost <pd at python.dobrogost.net> added the comment:

@Lehmann

You have to have either Visual Studio 2008 or Visual C++ Express 2008 installed. The folder where vcvarsall.bat file is being looked for is read from the registry. It's either HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0\Setup\VC (for Visual Studio) or HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VCExpress\9.0\Setup\VC (for Visual C++ Express).

----------
nosy: +piotr.dobrogost

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2698>
_______________________________________

From report at bugs.python.org  Mon Jan  2 13:34:53 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 02 Jan 2012 12:34:53 +0000
Subject: [issue13565] test_multiprocessing.test_notify_all() hangs on "AMD64
	Snow Leopard 02 03.x"
In-Reply-To: <1323435539.17.0.162534043929.issue13565@psf.upfronthosting.co.za>
Message-ID: <1325507693.84.0.68942460349.issue13565@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Alright, it seems to be fixed.
We can still reopen if this happens again.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13565>
_______________________________________

From report at bugs.python.org  Mon Jan  2 13:59:18 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 02 Jan 2012 12:59:18 +0000
Subject: [issue13694] asynchronous connect in asyncore.dispatcher does not set
	addr
In-Reply-To: <1325470516.27.0.519900720027.issue13694@psf.upfronthosting.co.za>
Message-ID: <1325509158.7.0.45596040823.issue13694@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +giampaolo.rodola, josiahcarlson, stutzbach
stage:  -> patch review
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13694>
_______________________________________

From report at bugs.python.org  Mon Jan  2 14:00:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 02 Jan 2012 13:00:28 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325509228.42.0.719279892398.issue13609@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks for the updated patch.
I think I would like other people's opinions about the dual-function approach.

----------
nosy: +loewis, neologix, rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 14:14:04 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Mon, 02 Jan 2012 13:14:04 +0000
Subject: [issue13694] asynchronous connect in asyncore.dispatcher does not set
	addr
In-Reply-To: <1325470516.27.0.519900720027.issue13694@psf.upfronthosting.co.za>
Message-ID: <1325510044.27.0.234126461796.issue13694@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

I'm not sure how useful this is as addr will be set later, when connection is established.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13694>
_______________________________________

From report at bugs.python.org  Mon Jan  2 14:55:34 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Mon, 02 Jan 2012 13:55:34 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325512534.29.0.621067522119.issue13609@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

What I would do:

- build the namedtuple in Python rather than in C
- I don't particularly like CamelCased names for nametuples (I would use "size" instead of "TerminalSize")
- on UNIX, the ioctl() stuff can be done in python rather than in C
- use C only on Windows to deal with GetStdHandle/GetConsoleScreenBufferInfo; function name should be accessed as nt._get_terminal_size similarly to what we did for shutil.disk_usage in issue12442.
- do not raise NotImplementedError; if the required underlying functions are not available os.get_terminal_size should not exists in the first place (same as we did for shutil.disk_usage / issue12442)


Also, I'm not sure I like fallback=(80, 25) as default, followed by:

        try:
            size = query_terminal_size(sys.__stdout__.fileno())
        except OSError:
            size = TerminalSize(fallback)

That means we're never going to get an exception.
Instead I would:
- provide fallback as None
- let OSError propate unless fallback is not None

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 15:58:03 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 02 Jan 2012 14:58:03 +0000
Subject: [issue9975] Incorrect use of flowinfo and scope_id in IPv6 sockaddr
	tuple
In-Reply-To: <1285698946.58.0.377750793336.issue9975@psf.upfronthosting.co.za>
Message-ID: <E1RhjKr-00042V-Gi@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 0c10061df711 by Charles-Fran?ois Natali in branch '2.7':
Issue #9975: socket: Fix incorrect use of flowinfo and scope_id. Patch by
http://hg.python.org/cpython/rev/0c10061df711

New changeset cc346a672091 by Charles-Fran?ois Natali in branch '3.2':
Issue #9975: socket: Fix incorrect use of flowinfo and scope_id. Patch by
http://hg.python.org/cpython/rev/cc346a672091

New changeset 9222b8e7a7bc by Charles-Fran?ois Natali in branch 'default':
Issue #9975: socket: Fix incorrect use of flowinfo and scope_id. Patch by
http://hg.python.org/cpython/rev/9222b8e7a7bc

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9975>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:01:44 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Mon, 02 Jan 2012 15:01:44 +0000
Subject: [issue13684] httplib tunnel infinite loop
In-Reply-To: <1325260587.05.0.642107056326.issue13684@psf.upfronthosting.co.za>
Message-ID: <1325516504.36.0.261574793831.issue13684@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13684>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:01:48 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Mon, 02 Jan 2012 15:01:48 +0000
Subject: [issue13694] asynchronous connect in asyncore.dispatcher does not set
	addr
In-Reply-To: <1325470516.27.0.519900720027.issue13694@psf.upfronthosting.co.za>
Message-ID: <1325516508.39.0.485862981529.issue13694@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13694>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:39:01 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 15:39:01 +0000
Subject: [issue13690] Add DEBUG flag to documentation of re.compile
In-Reply-To: <1325355977.96.0.0770563449241.issue13690@psf.upfronthosting.co.za>
Message-ID: <1325518741.33.0.249266573548.issue13690@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> Display debug information about compiled expression.

IMO it would be helpful to define ?display? more precisely (e.g. ?Print debugging information on stdout during pattern compilation?).

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13690>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:50:44 2012
From: report at bugs.python.org (Ralf Schlatterbeck)
Date: Mon, 02 Jan 2012 15:50:44 +0000
Subject: [issue1467619] Header.decode_header eats up spaces
Message-ID: <1325519444.69.0.779454686273.issue1467619@psf.upfronthosting.co.za>


Ralf Schlatterbeck <rsc at runtux.com> added the comment:

I've been bitten by this too (in python up to 2.7 in roundup the bug-tracker). We're currently using a workaround that re-inserts spaces, see git on roundup.sourceforge.net file mailgw.py method _decode_header_to_utf8

RFC2047 even has a test-case at the end, it specifies:

encoded form                                displayed as
(=?ISO-8859-1?Q?a?= b)                      (a b)

note the space between 'a' and 'b' above. Spaces between non-encoded and encoded parts should be preserved. And it's probably a good idea to put the examples from the RFC into the regression test.

----------
nosy: +runtux

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1467619>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:58:35 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 15:58:35 +0000
Subject: [issue13402] Document absoluteness of sys.executable
In-Reply-To: <1321289437.84.0.982753479785.issue13402@psf.upfronthosting.co.za>
Message-ID: <1325519915.46.0.381333680236.issue13402@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

This discussion seems relevant: http://mail.python.org/pipermail/python-dev/2006-March/062453.html (it talks about the meaning of sys.executable, special cases such as embedded Python and functions that implement it (so we can have a look at that code)).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13402>
_______________________________________

From report at bugs.python.org  Mon Jan  2 16:58:40 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 02 Jan 2012 15:58:40 +0000
Subject: [issue1467619] Header.decode_header eats up spaces
Message-ID: <1325519920.67.0.559868887922.issue1467619@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1467619>
_______________________________________

From report at bugs.python.org  Mon Jan  2 17:09:53 2012
From: report at bugs.python.org (Ralf Schlatterbeck)
Date: Mon, 02 Jan 2012 16:09:53 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1188637019.25.0.0476259625696.issue1079@psf.upfronthosting.co.za>
Message-ID: <1325520593.45.0.455559184503.issue1079@psf.upfronthosting.co.za>


Ralf Schlatterbeck <rsc at runtux.com> added the comment:

maybe it would be a good start to include the examples at the end of RFC2047 into the regression tests? These examples at least support the case that a '?' may immediately follow an encoded string:

encoded form                                displayed as
(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)     (ab)

when trying this in python 2.7:

>>> decode_header ('(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)')
[('(', None), ('a', 'iso-8859-1'), ('=?ISO-8859-1?Q?b?=)', None)]

this fails. So I consider this a bug.

Note that although RFC2047 is vague concerning the interpretation if two encoded strings could follow each other without a whitespace, these *are* seen in the wild and *are* interpreted correctly by the mailers I've tested: mutt, thunderbird, exchange in various versions, even lotus notes seems to get this right. So I guess python should be "liberal in what you accept" and parse something like 
'(=?ISO-8859-1?Q?a?==?ISO-8859-1?Q?b?=)'
into
[ ('(', None)
, ('a', 'iso-8859-1')
, ('b', 'iso-8859-1')
, (')', None)
]

----------
nosy: +runtux

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________

From report at bugs.python.org  Mon Jan  2 17:46:54 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Mon, 02 Jan 2012 16:46:54 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325522814.07.0.278832095525.issue13609@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

Nice patch :-)

I think the two function approach works well.

Since you have already checked that termsize is not NULL, Py_DECREF can be used instead of Py_CLEAR.

Would it not be better to use sys.__stdout__ instead of 1 in the documentation and to use STDOUT_FILENO instead of 1 in the code?

A brief google suggested that Solaris requires sys/termios.h for TIOCGWINSZ. Perhaps also only define TERMSIZE_USE_IOCTL if TIOCGWINSZ is defined.
Like so:
#if defined(HAVE_SYS_IOCTL_H)
#include <sys/ioctl.h>
#if defined(TIOCGWINSZ)
#define TERMSIZE_USE_IOCTL
#else
#define TERMSIZE_USE_NOTIMPLEMENTED
#endif
#elif defined(HAVE_CONIO_H)
#include <windows.h>
#include <conio.h>
#define TERMSIZE_USE_CONIO
#else
#define TERMSIZE_USE_NOTIMPLEMENTED
#endif

(didn't check the windows parts)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 17:52:21 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 16:52:21 +0000
Subject: [issue12394] packaging: generate scripts from callable (dotted paths)
In-Reply-To: <1308906864.25.0.392964418022.issue12394@psf.upfronthosting.co.za>
Message-ID: <1325523141.56.0.030289668623.issue12394@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I?ll get back to this issue later, but now I just wanted to add a link about building the binary exe files: http://mail.python.org/pipermail/python-dev/2006-April/063846.html (it comes from the time where setuptools was supposed to be added to the stdlib)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12394>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:11:26 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 17:11:26 +0000
Subject: [issue13511] Specifying multiple lib and include directories on linux
In-Reply-To: <1322683661.17.0.138081414237.issue13511@psf.upfronthosting.co.za>
Message-ID: <1325524286.19.0.233366758097.issue13511@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> Whether or not includedir and libdir are supposed to allow multiple packages is beyond me at this
> point so I'll change the topic to more reflect the problem I am having.  More importantly (and
> possibly related to includedir and libdir) is the fact that python 2.7 does not allow specifying
> multiple lib and include directories in linux.  Is there one way to do this that I overlooked? I
> included the bit about includedir, libdir, CFLAGS, and LDFLAGS in my original post to show what I
> tried.

At present I know less than you about these.  Maybe Martin can chime in?

If it?s not possible to use multiple directories to configure Python, then it?s a limitation, and changing it would qualify as a new feature given our policy, and as such be inappropriate for 2.7.

----------
nosy: +loewis
versions:  -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13511>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:14:51 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 17:14:51 +0000
Subject: [issue6983] Add specific get_platform() for freebsd
In-Reply-To: <1253751823.43.0.939101436857.issue6983@psf.upfronthosting.co.za>
Message-ID: <1325524491.87.0.0524627385962.issue6983@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the reply.  distutils2 won?t support eggs, and I don?t think setuptools or distribute will be reworked to work on distutils2, so if your problem is only with eggs there is nothing that we can do to solve it in distutils2.  (As I said, distutils is frozen and can?t be changed to address this either.)  Currently distutils2 build sdists and bdists of the dumb, msi and wininst varieties.  If these formats have the same issue that you describe, then we can change them.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6983>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:16:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 17:16:55 +0000
Subject: [issue13561] os.listdir documentation should mention surrogateescape
In-Reply-To: <1323395400.31.0.796436036954.issue13561@psf.upfronthosting.co.za>
Message-ID: <1325524615.2.0.0566884090746.issue13561@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Patch looks good, apart from a missing ?the?, but I think it should be expanded: Right now it tells that surrogateescape is used, but IMO it would be more useful if it also mentioned practical implications, i.e. what Michael ?The Beard? Foord says in the first message.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13561>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:21:57 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 02 Jan 2012 17:21:57 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325524917.97.0.84103640418.issue9993@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Antoine, would you mind taking this one?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:34:38 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 02 Jan 2012 17:34:38 +0000
Subject: [issue13663] pootle.python.org is outdated.
In-Reply-To: <1324903809.07.0.39088932351.issue13663@psf.upfronthosting.co.za>
Message-ID: <1325525678.72.0.793112267676.issue13663@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13663>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:36:15 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 02 Jan 2012 17:36:15 +0000
Subject: [issue13665] TypeError: string or integer address expected instead of
	str instance
In-Reply-To: <1324921834.92.0.644281007494.issue13665@psf.upfronthosting.co.za>
Message-ID: <1325525775.23.0.898350978814.issue13665@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
stage:  -> needs patch
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13665>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:43:27 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 02 Jan 2012 17:43:27 +0000
Subject: [issue13693] email.Header.Header incorrect/non-smart on international
	charset address fields
In-Reply-To: <1325438700.75.0.319524788564.issue13693@psf.upfronthosting.co.za>
Message-ID: <1325526207.3.0.418282512916.issue13693@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Actually, no, the local part cannot be in anything other than ascii (see RFC 5335, which desires to address this problem among others).  Also, an encoded word cannot occur inside quotation marks.  If you correct those two bugs, you can generate an RFC-valid address using Header.append.

There is a project underway to make all of this header parsing and formatting stuff work better: see the http://pypi.python.org/pypi/email.

By the way, this is easier already in python 3.2.  There you can do:

   >>> formataddr(('Name?', 'weofij at fjeio'))
   '=?utf-8?b?TmFtZcOf?= <weofij at fjeio>'

----------
nosy: +r.david.murray
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13693>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:51:10 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 02 Jan 2012 17:51:10 +0000
Subject: [issue10839] email module should not allow some header field
	repetitions
In-Reply-To: <1294263603.86.0.308955870327.issue10839@psf.upfronthosting.co.za>
Message-ID: <1325526670.48.0.462180574101.issue10839@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Regardless of what anybody thinks about the design, it is what it is and can't be changed for backward compatibility reasons.  The best we can do is reject creating duplicate headers for headers that may only appear once.  That feature has already been coded in the new version of the email package (see http://pypi.python.org/pypi/email), but has not yet been committed to the trunk, which is why this issue is still open.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10839>
_______________________________________

From report at bugs.python.org  Mon Jan  2 18:55:24 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 02 Jan 2012 17:55:24 +0000
Subject: [issue1467619] Header.decode_header eats up spaces
Message-ID: <1325526924.83.0.333424343711.issue1467619@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Antoine, I marked this for Python 3.3 only because there is no good way to fix it in 2.7/3.2.  (If someone comes up with a way I'll be happy to review it, though!)

----------
versions:  -Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1467619>
_______________________________________

From report at bugs.python.org  Mon Jan  2 19:14:27 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 02 Jan 2012 18:14:27 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1188637019.25.0.0476259625696.issue1079@psf.upfronthosting.co.za>
Message-ID: <1325528067.61.0.988640391873.issue1079@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

The RFC isn't at all vague about encoded words not separated by white space.  That isn't allowed by the BNF.  As you say, though, they occur in the wild and should be parsed correctly.

In your other point I think you mean "immediately followed by a )", right?  Yes, that is allowed and no, we don't currently parse that correctly.

Adding the RFC tests would be great (patches gladly accepted).  Fixes for ones we fail would be great, too, but at the very least we can mark them as expected failures.  I don't usually like adding tests that we expect to fail, but in the case of externally defined tests such as the RFC examples I think it is worthwhile, so that we can check in a complete test set.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________

From report at bugs.python.org  Mon Jan  2 19:22:27 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 02 Jan 2012 18:22:27 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325528547.28.0.437210543983.issue9993@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Ow, I'm sorry for overlooking this. I thought you would take it.
Unfortunately the patch is now broken because of 5b61334bb776.
(technically it applies, but the tests don't run anymore because the "other filesystem" now uses a mocking approach)

By the way, I think this shouldn't be applied to bugfix branches, since it's a behaviour change.

----------
versions:  -Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Mon Jan  2 19:40:30 2012
From: report at bugs.python.org (CRicky)
Date: Mon, 02 Jan 2012 18:40:30 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
Message-ID: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>


New submission from CRicky <cricky.kmp at gmx.fr>:

I had an HTTP redirection that worked perfectly on version 3.1.

On version 3.2, I get a HTTP error 302. In this redirection, I actually have 2 redirections. The last one does not work because it is a relative redirection, so urlparts.scheme is empty.

Some lines have been added in version 3.2 for security reason, but it also blocks relative links in 302 return.

To correct, I have added empty scheme in check:
        if not urlparts.scheme in ('http', 'https', 'ftp', ''):

With that, it works correctly.
I don't make you any for 3 new chars. ;)

Best regards,
CRicky

----------
components: Library (Lib)
messages: 150473
nosy: CRicky
priority: normal
severity: normal
status: open
title: [urllib.request.HTTPRedirectHandler.http_error_302] Relative Redirect issue
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Mon Jan  2 19:57:25 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 02 Jan 2012 18:57:25 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325530645.89.0.518264793167.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Thanks for the reviews!

> - build the namedtuple in Python rather than in C
I started this way, but if two functions are used, it is nicer to have them both
return the same type. If it was defined in the Python part, there would be
loop in the sense that os would import the function from _posixmodule, and
_posixmodule would import the namedtuple from os. The code for the tuple is fairly simple,
mostly docs, and saving a few lines just doesn't seem to be worth the complication of doing
it other way around.

> - I don't particularly like CamelCased names for nametuples (I would use "size" instead of "TerminalSize")
I was following the style in _posixmodule: there is WaitidResultType and SchedParamType.
"size" seems to generic, maybe something like "terminal_size" would be better (SchedParamType
is exported as sched_param).

Will change to "terminal_size".

- on UNIX, the ioctl() stuff can be done in python rather than in C
It can, but it requires extracting the bytes by hand, doing it in C is cleaner.
The C implementation is only 5 lines (in addition to the code necessary for windows)
and again it seems simpler this way. Also the configuration is kept in one
place in the #ifdefs at the top, not in two different files.

> - use C only on Windows to deal with GetStdHandle/GetConsoleScreenBufferInfo; function name should be
> accessed as nt._get_terminal_size similarly to what we did for shutil.disk_usage in issue12442.
> - do not raise NotImplementedError; if the required underlying functions are not available 
> os.get_terminal_size should not exists in the first place (same as we did for shutil.disk_usage / issue12442)

I think that there is a difference in "importance" -- shutil.disk_usage cannot be
faked, or ignored, and if it is unavailable, then some work-around must be implemented.
OTOH, the terminal size is an embellishment, and if a wrong terminal size is used, the output
might wrap around or be a bit narrow (the common case), but like in argparse.FancyFormatter,
just continuing is probably reasonable. So to save the users of the function the trouble
to wrap it in some try..except, let's cater for the common case.

This same argument goes for defining the function only if an implementation is available:
argparse would need something like:
try:
  os.get_terminal_size
except NameError:
  def get_terminal_size():
    return (80, 25)
which just doesn't seem to _gain_ anything.

> Also, I'm not sure I like fallback=(80, 25) as default, followed by:
> [...]
> That means we're never going to get an exception.
> Instead I would:
> - provide fallback as None
> - let OSError propate unless fallback is not None
Again, if the user forgets to add the fallback, and only tests in a terminal, she 
would get an unnecessary surprise when running the thing in a pipe. So the fallback
would have to be almost always provided... and it will almost always be (80, 25).
So let's just provide it upfront, and let the user call the low-level function if
they want full control.

-----------------------------------------------------------------------------------

> I think the two function approach works well.
:)

> Since you have already checked that termsize is not NULL, Py_DECREF can be used instead of Py_CLEAR.
OK.

> Would it not be better to use sys.__stdout__ instead of 1 in the documentation and to use STDOUT_FILENO 
> instead of 1 in the code?
OK.

> A brief google suggested that Solaris requires sys/termios.h for TIOCGWINSZ. Perhaps also only define
> TERMSIZE_USE_IOCTL if TIOCGWINSZ is defined.
Hm, I don't have solaris available, so I won't be able to check easily, but
maybe sys/termios.h should be imported if TIOCGWINSZ is not available. Would 
be nice to provide the implementation if possible.

> Like so:
Something like:

> #if defined(HAVE_SYS_IOCTL_H)
> #include <sys/ioctl.h>
> #if defined(TIOCGWINSZ)
> #define TERMSIZE_USE_IOCTL
> #else

#if defined(HAVE_SYS_TERMIOS_H)
#include <sys/termios.h>
#if defined(TIOCGWINSZ)
#define TERMSIZE_USE_IOCTL
#else
#define TERMSIZE_USE_NOTIMPLEMENTED

> #endif
> #elif defined(HAVE_CONIO_H)
> #include <windows.h>
> #include <conio.h>
> #define TERMSIZE_USE_CONIO
> #else
> #define TERMSIZE_USE_NOTIMPLEMENTED
> #endif

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 20:49:27 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Mon, 02 Jan 2012 19:49:27 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325533767.4.0.551659567826.issue13609@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

I'll try & investigate Solaris a bit...

Also, what should be the default terminal size?

Gnome-terminal and xterm seem to default to 80x24, not 80x25.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 21:25:38 2012
From: report at bugs.python.org (Bill Janssen)
Date: Mon, 02 Jan 2012 20:25:38 +0000
Subject: [issue2698] Extension module build fails for MinGW: missing
	vcvarsall.bat
In-Reply-To: <1325506793.5.0.592977725239.issue2698@psf.upfronthosting.co.za>
Message-ID: <CAHkzgdidn0h-TAuozkqVFhX1Ptrviex5NiE8+KRbSj8gwqjeHw@mail.gmail.com>


Bill Janssen <bill.janssen at gmail.com> added the comment:

Actually, when using setup.py with MinGW, you just need to say the right thing:

  % python setup.py build --compiler=mingw32 install

That removes the check for vcvarsall.bat.

Bill

On Mon, Jan 2, 2012 at 6:19 AM, Piotr Dobrogost <report at bugs.python.org> wrote:
>
> Piotr Dobrogost <pd at python.dobrogost.net> added the comment:
>
> @Lehmann
>
> You have to have either Visual Studio 2008 or Visual C++ Express 2008 installed. The folder where vcvarsall.bat file is being looked for is read from the registry. It's either HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0\Setup\VC (for Visual Studio) or HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VCExpress\9.0\Setup\VC (for Visual C++ Express).
>
> ----------
> nosy: +piotr.dobrogost
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue2698>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2698>
_______________________________________

From report at bugs.python.org  Mon Jan  2 22:13:13 2012
From: report at bugs.python.org (Robert Collins)
Date: Mon, 02 Jan 2012 21:13:13 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
Message-ID: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>


New submission from Robert Collins <robertc at robertcollins.net>:

This affects the python implementation of RLock only.

If a signal occurs during RLock.acquire() or release() and then operates on the same lock to acquire() or release() it, process hangs or assertions can be triggered.

The attached test script demonstrates the issue on Python 2.6 and 3.2, and code inspection suggests this is still valid for 2.7 and 3.4.

To use it, run the script and then 'kill -SIGUSR2' the process id it prints out. Keep sending SIGUSR2 until you get bored or:
 - a traceback occurs
 - the process stops printing out '.'s

We found this debugging application server hangs during log rotation (https://bugs.launchpad.net/launchpad/+bug/861742) where if the thread that the signal is received in was about to log a message the deadlock case would crop up from time to time. Of course, this wasn't ever safe code, and we're changing it (to have the signal handler merely set a integer flag that the logging handler can consult without locking).

However, I wanted to raise this upstream, and I note per issue #3618 that the 3.x IO module apparently includes a minimal version of the Python RLock implementation (or just uses RLock itself). Either way this bug probably exists for applications simply doing IO, either when there is no C RLock implementation available, or all the time (depending on exactly what the IO module is doing nowadays).

----------
components: Library (Lib)
files: hang.py
messages: 150477
nosy: rbcollins
priority: normal
severity: normal
status: open
title: python RLock implementation unsafe with signals
versions: Python 2.7, Python 3.4
Added file: http://bugs.python.org/file24126/hang.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Mon Jan  2 22:46:28 2012
From: report at bugs.python.org (endolith)
Date: Mon, 02 Jan 2012 21:46:28 +0000
Subject: [issue13698] Should not use mboxo format
Message-ID: <1325540787.88.0.149207146949.issue13698@psf.upfronthosting.co.za>


New submission from endolith <endolith at gmail.com>:

The documentation states:

"Several variations of the mbox format exist to address perceived shortcomings in the original. In the interest of compatibility, mbox implements the original format, which is sometimes referred to as mboxo."

http://docs.python.org/dev/library/mailbox.html#mailbox.mbox

But this format is fundamentally broken, corrupting lines that start with ">From ", and I can't find any justification for using it in Python.  In fact, all four links included in that section argue against this format.

If only one mbox format is used, it should be mboxrd.  Otherwise, include support for all the variants, with mboxrd as the default.

----------
components: Library (Lib)
messages: 150478
nosy: endolith
priority: normal
severity: normal
status: open
title: Should not use mboxo format
type: behavior
versions: Python 2.7, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13698>
_______________________________________

From report at bugs.python.org  Mon Jan  2 22:47:36 2012
From: report at bugs.python.org (endolith)
Date: Mon, 02 Jan 2012 21:47:36 +0000
Subject: [issue13698] Mailbox module should not use mboxo format
In-Reply-To: <1325540787.88.0.149207146949.issue13698@psf.upfronthosting.co.za>
Message-ID: <1325540856.7.0.173329549773.issue13698@psf.upfronthosting.co.za>


Changes by endolith <endolith at gmail.com>:


----------
title: Should not use mboxo format -> Mailbox module should not use mboxo format

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13698>
_______________________________________

From report at bugs.python.org  Mon Jan  2 22:55:04 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 02 Jan 2012 21:55:04 +0000
Subject: [issue13698] Mailbox module should support other mbox formats in
	addition to mboxo
In-Reply-To: <1325540787.88.0.149207146949.issue13698@psf.upfronthosting.co.za>
Message-ID: <1325541304.09.0.759799373949.issue13698@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Well, supporting the other variants would be good (I'll review any proposed patches), but I think the default will have to stay mboxo for backward compatibility reasons (unless the consensus is to go through the warning/deprecation cycle to change it).

As a new feature, this could only go into 3.3 or later.

----------
nosy: +r.david.murray
stage:  -> needs patch
title: Mailbox module should not use mboxo format -> Mailbox module should support other mbox formats in addition to mboxo
type: behavior -> enhancement
versions:  -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13698>
_______________________________________

From report at bugs.python.org  Mon Jan  2 23:04:23 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Mon, 02 Jan 2012 22:04:23 +0000
Subject: [issue13685] argparse does not sanitize help strings for % signs
In-Reply-To: <1325284167.16.0.584942168313.issue13685@psf.upfronthosting.co.za>
Message-ID: <1325541863.16.0.936835857227.issue13685@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

In case I wasn't clear, I mean that the help string supports %-formatting variable expansion, such as "%(default)s". I think it would be good if a sentence were added to the end of the "help" section, saying something like:

Because the help string supports %-formatting, if you want a literal "%" to appear in the help string, you must escape it as "%%".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13685>
_______________________________________

From report at bugs.python.org  Mon Jan  2 23:30:40 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Mon, 02 Jan 2012 22:30:40 +0000
Subject: [issue13653] reorder set.intersection parameters for better
	performance
In-Reply-To: <1324601653.94.0.248406097539.issue13653@psf.upfronthosting.co.za>
Message-ID: <1325543440.86.0.875963437038.issue13653@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13653>
_______________________________________

From report at bugs.python.org  Mon Jan  2 23:43:19 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 02 Jan 2012 22:43:19 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325544199.27.0.232024457063.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Updated patch termsize.diff.3

- s/TerminalSize/terminal_size/ in Python code
- change the fallback to (80, 24) (seems to be the commonest default)
- s/Py_CLEAR/Py_DECREF/
- use STDOUT_FILENO
- add more hrefs in docs
- include <termios.h> is available (untested fix for solaris compatibility)
- rename TerminalSize as terminal_size in Python code

I tested this iteration only on linux and windows, but it is not substantially changed, so should still work on *bsd. (I previously
tested on debian/kfreebsd and dragonflybsd and it seemed functional).

----------
Added file: http://bugs.python.org/file24127/termsize.diff.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan  2 23:54:08 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 02 Jan 2012 22:54:08 +0000
Subject: [issue8245] email examples don't actually work (SMTP.connect is not
	called)
In-Reply-To: <1269714048.68.0.587279851175.issue8245@psf.upfronthosting.co.za>
Message-ID: <1325544848.28.0.316049256994.issue8245@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

This has already been fixed in issue11883 changing the SMTP constructor call.

----------
nosy: +sandro.tosi
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> Call connect() before sending an email with smtplib
versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8245>
_______________________________________

From report at bugs.python.org  Tue Jan  3 00:10:17 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Mon, 02 Jan 2012 23:10:17 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325545817.39.0.818130273512.issue13697@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Adding myself to the nosy list because I am interested in the issue, but I can't take care of this myself for a few weeks, at least.

Robert, would you consider to try to write a patch?

----------
nosy: +jcea
versions: +Python 3.3 -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 00:19:01 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Mon, 02 Jan 2012 23:19:01 +0000
Subject: [issue13699] test_gdb has recently started failing
Message-ID: <1325546341.52.0.130827872784.issue13699@psf.upfronthosting.co.za>


New submission from Vinay Sajip <vinay_sajip at yahoo.co.uk>:

test_gdb has started failing recently on my Ubuntu Natty system:

vinay at eta-natty:~/tools/cpython$ ./python Lib/test/regrtest.py -v test_gdb
== CPython 3.3.0a0 (default:3ed5a6030c9b, Jan 2 2012, 23:04:11) [GCC 4.5.2]
==   Linux-2.6.38-13-generic-i686-with-debian-squeeze-sid little-endian
==   /home/vinay/tools/cpython/build/test_python_14449
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0)
[1/1] test_gdb

[successful tests snipped]

======================================================================
FAIL: test_print_after_up (test.test_gdb.PyPrintTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vinay/tools/cpython/Lib/test/test_gdb.py", line 670, in test_print_after_up
    r".*\nlocal 'c' = 3\nlocal 'b' = 2\nlocal 'a' = 1\n.*")
  File "/home/vinay/tools/cpython/Lib/test/test_gdb.py", line 171, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x80ad5c6: file Python/bltinmodule.c, line 951.\n[Thread debugging using libthread_db enabled]\n\nBreakpoint 1, builtin_id (self=<module at remote 0xb7dd012c>, v=42) at Python/bltinmodule.c:951\n951\t    return PyLong_FromVoidPtr(v);\n#6 (unable to read python frame information)\nUnable to read information on python frame\nUnable to read information on python frame\nUnable to read information on python frame\n' did not match ".*\\nlocal 'c' = 3\\nlocal 'b' = 2\\nlocal 'a' = 1\\n.*"

======================================================================
FAIL: test_locals_after_up (test.test_gdb.PyLocalsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vinay/tools/cpython/Lib/test/test_gdb.py", line 702, in test_locals_after_up
    r".*\na = 1\nb = 2\nc = 3\n.*")
  File "/home/vinay/tools/cpython/Lib/test/test_gdb.py", line 171, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x80ad5c6: file Python/bltinmodule.c, line 951.\n[Thread debugging using libthread_db enabled]\n\nBreakpoint 1, builtin_id (self=<module at remote 0xb7dd012c>, v=42) at Python/bltinmodule.c:951\n951\t    return PyLong_FromVoidPtr(v);\n#6 (unable to read python frame information)\nUnable to read information on python frame\n' did not match '.*\\na = 1\\nb = 2\\nc = 3\\n.*'

----------------------------------------------------------------------
Ran 42 tests in 13.809s

FAILED (failures=2, skipped=11)
test test_gdb failed
1 test failed:
    test_gdb

vinay at eta-natty:~/tools/cpython$ ./python
Python 3.3.0a0 (default:3ed5a6030c9b, Jan  2 2012, 23:04:11) 
[GCC 4.5.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_vars()['PY_CFLAGS']
'-Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes'
>>> 

vinay at eta-natty:~/tools/cpython$ gdb --version
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

vinay at eta-natty:~/tools/cpython$ gcc --version
gcc (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

vinay at eta-natty:~/tools/cpython$ uname -a
Linux eta-natty 2.6.38-13-generic #53-Ubuntu SMP Mon Nov 28 19:23:39 UTC 2011 i686 i686 i386 GNU/Linux

Marking haypo as nosy since it might be related to recent changes for #13628.

In earlier tests (conducted on 12 Dec, before the #13628 fix was checked in) test_gdb did not fail on the same system.

----------
components: Library (Lib)
messages: 150484
nosy: haypo, vinay.sajip
priority: normal
severity: normal
status: open
title: test_gdb has recently started failing
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13699>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:30:04 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 00:30:04 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325550604.94.0.64640986645.issue13697@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
stage:  -> needs patch
type:  -> behavior
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:30:08 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 00:30:08 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325550608.69.0.256447156065.issue13697@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:40:20 2012
From: report at bugs.python.org (Phill)
Date: Tue, 03 Jan 2012 00:40:20 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325551220.5.0.84927236565.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

I have run into the same problem with python 3.2 & 2.7 on windows 7 with the Listener object. It opens the same port twice for some wierd reason

I have tried the example in this bug report and i am getting the same behavoir as the original poster

----------
nosy: +gearb0x

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:41:58 2012
From: report at bugs.python.org (Phill)
Date: Tue, 03 Jan 2012 00:41:58 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325551318.74.0.996291799178.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

Im not sure whether to open a new bug report for my issue or just leave it here.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:42:50 2012
From: report at bugs.python.org (Matt Joiner)
Date: Tue, 03 Jan 2012 00:42:50 +0000
Subject: [issue13694] asynchronous connect in asyncore.dispatcher does not set
	addr
In-Reply-To: <1325470516.27.0.519900720027.issue13694@psf.upfronthosting.co.za>
Message-ID: <1325551370.93.0.887366175636.issue13694@psf.upfronthosting.co.za>


Matt Joiner <anacrolix at gmail.com> added the comment:

I don't believe it is. dispatcher.addr is only set if the connection is immediately established. It's set explicitly in dispatcher.__init__ if a socket is provided that is already connected. It's *not* set after a connection completes.

There are 2 solutions as I see it:

1) Set addr after a successful call to socket.connect indicates that a connection is being established. Currently this only occurs if the connection is not delayed.
2) Set the addr when a connect event completes. This would require making getpeername calls like in dispatcher.__init__ and would be significantly more expensive.

My patch implements method 1. This conforms to existing addr-setting behaviour in dispatcher.bind.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13694>
_______________________________________

From report at bugs.python.org  Tue Jan  3 01:52:43 2012
From: report at bugs.python.org (Robert Collins)
Date: Tue, 03 Jan 2012 00:52:43 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325551963.58.0.256309919671.issue13697@psf.upfronthosting.co.za>


Robert Collins <robertc at robertcollins.net> added the comment:

I'm not sure it is sensibly implementable in pure python: the semantics of signal handling (AIUI) are that the vm is interrupted, sets a flag to say 'when the GIL is released or the next bytecode interpretation happens, please process signal X' [the former for C extensions, the latter for regular code], and then the OS level signal handler returns. The current C or bytecode execution completes and then we dispatch to the python signal handler.

Now, what we would want here is that attempts to acquire/release the RLock in a signal handler would behave as though the RLock.acquire/release methods were atomic. The general way to enforce such atomicity is via a lock or mutex. Now, because the call stack looks like this:
<signal>thread.acquire()
<non-signal-code>thread.acquire()

The other acquire, if it already holds this supposed mutex, would block the python signal handler acquire call from obtaining it; The inner acquire would have to error *whether or not a timeout was passed*, because the non-signal-code won't progress and complete until the python signal handler code returns. One could, in principle, use stack inspection to determine how far through a nested acquire/release the outer code is, but that seems, uhm, pathological to me :).

The crux of the problem is detecting whether the non-reentrant thread.lock is held because (a) another thread holds it or (b) this thread holds it. If we can inspect its internals, we could determine that this thread holds it - and thus we must be reentered into the acquire/release methods. With that tricky thing established, we could look at the remaining logic to make sure that each line is either atomic or conditional on reentrancy.

Simpler I think to require CRLocks always, and provide a portable C implementation which will always complete before returning to execution in the calling thread, avoiding this issue entirely.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 02:11:29 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 01:11:29 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
Message-ID: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>


New submission from Erno Tukia <erno.tukia at iki.fi>:

>>> import imaplib
>>> imap = imaplib.IMAP4_SSL("imap.example.com")

>>> authcb = lambda resp: "{0}\x00{0}\x00{1}".format("username","password")
>>> imap.authenticate("PLAIN", authcb)
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    imap.authenticate("PLAIN", authcb)
  File "/usr/lib/python3.1/imaplib.py", line 361, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mech)
  File "/usr/lib/python3.1/imaplib.py", line 1075, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.1/imaplib.py", line 889, in _command
    literal = literator(self.continuation_response)
  File "/usr/lib/python3.1/imaplib.py", line 1238, in process
    return self.encode(ret)
  File "/usr/lib/python3.1/imaplib.py", line 1257, in encode
    e = binascii.b2a_base64(t)
TypeError: must be bytes or buffer, not str

... and ...

>>> authcb = lambda resp: "{0}\x00{0}\x00{1}".format("username","password").encode()
>>> imap.authenticate("PLAIN", authcb)
Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    imap.authenticate("PLAIN", authcb)
  File "/usr/lib/python3.1/imaplib.py", line 361, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mech)
  File "/usr/lib/python3.1/imaplib.py", line 1075, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.1/imaplib.py", line 889, in _command
    literal = literator(self.continuation_response)
  File "/usr/lib/python3.1/imaplib.py", line 1238, in process
    return self.encode(ret)
  File "/usr/lib/python3.1/imaplib.py", line 1259, in encode
    oup = oup + e[:-1]
TypeError: Can't convert 'bytes' object to str implicitly

----------
components: Library (Lib)
messages: 150489
nosy: etukia
priority: normal
severity: normal
status: open
title: imaplib.IMAP4.authenticate authobject fails with PLAIN mechanism
type: behavior
versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 02:19:23 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 03 Jan 2012 01:19:23 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1325553563.26.0.535848483761.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Also

http://guide.python-distribute.org/quickstart.html#lay-out-your-project

needs to have the first sentence fixed: "python" to "Python"


and

http://docs.python.org/library/functools.html#functools.cmp_to_key

needs to have the word "Py3.x" fixed to "Python 3" (note: not to "Python 3.x" because functools.reduce() function also has the word "Python 3", so be consistent)

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Tue Jan  3 02:29:30 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 03 Jan 2012 01:29:30 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1325554170.15.0.347291495802.issue13695@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
stage:  -> needs patch
type:  -> enhancement
versions:  -Python 3.1, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Tue Jan  3 02:59:58 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 01:59:58 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325555998.99.0.979221409228.issue13700@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

The first argument to authenticate must be bytes.  This is not well documented.  It might also be a bug, since I'm not sure anyone has done a thorough audit of what should be bytes and what should be string in imaplib.

3.1 no longer gets bug fixes, so I'm removing it from versions.  Likewise I remove 3.4 since that applies only to changes that will *not* be put in 3.3 for some reason.

----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python, r.david.murray
versions:  -Python 3.1, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 03:11:10 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 02:11:10 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325556670.67.0.027361597351.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

The same problems exists even if I use b"PLAIN" as the first argument in authenticate().

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 03:24:24 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 02:24:24 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325557464.39.0.305235489566.issue13700@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Gah, I was looking at the wrong source code when I wrote that.  A string first argument is indeed valid.  I'm not sure where the problem is coming from since the internal CRAM_MD5 returns a string and that seems to work...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 03:59:53 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 02:59:53 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325559593.36.0.524552869569.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

File "/usr/lib/python3.1/imaplib.py", line 1257, in encode
    e = binascii.b2a_base64(t)

imaplib._Authenticator.encode() calls binascii.b2a_base64() function. In Python 2.6 that function returns a string, and in Python 3.1 it returns bytes. What is returned from b2a_base64() function is later in the encode() function concatenated with a string, with bytes that is not possible.

Should binascii.b2a_base64() return a string (2.6) or bytes (3.1)?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 04:01:46 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Tue, 03 Jan 2012 03:01:46 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
Message-ID: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

Decimal has compatibility for Python 2.3 (such as not using decorators) as mentioned in comments in the source code for it.
This compatibility should be removed in Python 3.3 and if possible, in Python 3.2.

----------
components: Library (Lib)
messages: 150495
nosy: ramchandra.apte
priority: normal
severity: normal
status: open
title: Remove Decimal Python 2.3 Compatibility
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Tue Jan  3 04:18:02 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 03:18:02 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325560682.04.0.164895039767.issue13700@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Bytes definitely.  We hashed that out a while ago.

My point is that CRAM_MD5 login calls authenticate, and its authenticator returns a string, just like your example does.  So it ought to be going through the same code path.  I haven't followed the logic in detail, though, so there must be some difference...I'm pretty sure the MD5 login has a test now (but not 100% sure...)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 04:43:02 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 03 Jan 2012 03:43:02 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325562182.06.0.945846510022.issue13697@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 04:50:09 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 03 Jan 2012 03:50:09 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <1325562609.88.0.962035744907.issue13696@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +orsenthil
stage:  -> test needed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Tue Jan  3 05:10:09 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 03 Jan 2012 04:10:09 +0000
Subject: [issue13585] Add contextlib.ContextStack
In-Reply-To: <1323654590.86.0.53041507768.issue13585@psf.upfronthosting.co.za>
Message-ID: <1325563809.59.0.681199656575.issue13585@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13585>
_______________________________________

From report at bugs.python.org  Tue Jan  3 07:12:20 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Tue, 03 Jan 2012 06:12:20 +0000
Subject: [issue12364] Deadlock in test_concurrent_futures
In-Reply-To: <1308504315.81.0.454658658034.issue12364@psf.upfronthosting.co.za>
Message-ID: <1325571140.25.0.93506173628.issue12364@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

Retrieving the result of a future after the executor has been shut down can cause a hang.

It seems like this regression was introduced in a76257a99636. This regression exists only for ProcessPoolExecutor.

The problem is that even if there are pending work items, the processes are still signaled to shut down leaving the pending work items permanently unfinished. The patch simply removes the call to shut down the processes when there are pending work items.

Attached is a patch.

----------
keywords: +patch
nosy: +rosslagerwall
Added file: http://bugs.python.org/file24128/issue.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12364>
_______________________________________

From report at bugs.python.org  Tue Jan  3 07:12:54 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Tue, 03 Jan 2012 06:12:54 +0000
Subject: [issue12364] Deadlock in test_concurrent_futures
In-Reply-To: <1308504315.81.0.454658658034.issue12364@psf.upfronthosting.co.za>
Message-ID: <1325571174.44.0.838862161903.issue12364@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


Added file: http://bugs.python.org/file24129/itest.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12364>
_______________________________________

From report at bugs.python.org  Tue Jan  3 08:32:06 2012
From: report at bugs.python.org (Srikanth S)
Date: Tue, 03 Jan 2012 07:32:06 +0000
Subject: [issue1372770] email.Header should preserve original FWS
Message-ID: <1325575926.55.0.500963367776.issue1372770@psf.upfronthosting.co.za>


Changes by Srikanth S <sriksubr at gmail.com>:


----------
nosy: +srikanths

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1372770>
_______________________________________

From report at bugs.python.org  Tue Jan  3 08:37:54 2012
From: report at bugs.python.org (Srikanth S)
Date: Tue, 03 Jan 2012 07:37:54 +0000
Subject: [issue11492] email.header.Header doesn't fold headers correctly
In-Reply-To: <1300087978.39.0.703069545984.issue11492@psf.upfronthosting.co.za>
Message-ID: <1325576274.66.0.581708088533.issue11492@psf.upfronthosting.co.za>


Changes by Srikanth S <sriksubr at gmail.com>:


----------
nosy: +srikanths

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11492>
_______________________________________

From report at bugs.python.org  Tue Jan  3 14:57:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 13:57:25 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1325599045.74.0.0648987519292.issue13701@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +facundobatista, mark.dickinson, rhettinger, skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Tue Jan  3 15:33:52 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 14:33:52 +0000
Subject: [issue12364] Deadlock in test_concurrent_futures
In-Reply-To: <1308504315.81.0.454658658034.issue12364@psf.upfronthosting.co.za>
Message-ID: <1325601232.28.0.228427227764.issue12364@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well I was sure I had added this code for a reason, but the tests seem to run without...
Just a comment: the test isn't ProcessPoolExecutor-specific, so it should really be in the generic tests.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12364>
_______________________________________

From report at bugs.python.org  Tue Jan  3 15:40:36 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 14:40:36 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325601636.45.0.388769650609.issue13697@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Yes, using synchronization primitives or doing I/O in Python signal handlers isn't a good idea. Perhaps the signal docs should be clearer about that.

> Of course, this wasn't ever safe code, and we're changing it (to have the signal handler merely set a integer flag that the logging handler can consult without locking)

Indeed, setting a variable (or using set_wakeup_fd()) is the right approach.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 16:06:16 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 15:06:16 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325603176.26.0.392799478167.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

In Python 2.6 PLAIN authentication works, in Python 3.1 not.

Lib/test/test_imaplib.py does not test IMAP4.authenticate() or IMAP4.login_cram_md5() functions, only IMAP4.login().

I would still like to go back to imaplib._Authenticator.encode() function. The function is below.

# inp = authobject(response)
def encode(self, inp):
    oup = ''
    while inp:
        if len(inp) > 48:
            t = inp[:48]
            inp = inp[48:]
        else:
            t = inp
            inp = ''
        e = binascii.b2a_base64(t)
        if e:
            oup = oup + e[:-1]
    return oup

binascii.b2a_base64() takes bytes, so inp must therefore be bytes, and returns bytes (Python 3). Then str + bytes (out + e[:-1]) fails.

The fix would then be changing 
            oup = oup + e[:-1]
to
            oup = oup + e[:-1].decode()

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 16:25:30 2012
From: report at bugs.python.org (Ralf Schlatterbeck)
Date: Tue, 03 Jan 2012 15:25:30 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1325528067.61.0.988640391873.issue1079@psf.upfronthosting.co.za>
Message-ID: <20120103152528.GA27700@runtux.com>


Ralf Schlatterbeck <rsc at runtux.com> added the comment:

Fine, I see what you mean, this involves very careful reading of the RFC
and could have been a little more verbose ...

Right. Should have been a ')'

> Adding the RFC tests would be great (patches gladly accepted).  Fixes
> for ones we fail would be great, too, but at the very least we can
> mark them as expected failures.  I don't usually like adding tests
> that we expect to fail, but in the case of externally defined tests
> such as the RFC examples I think it is worthwhile, so that we can
> check in a complete test set.

Patch attached (against current tip, 74241:120a79b8bb11). We currently
fail *all* of the tests in the RFC due to the same problem, the closing
')', I've marked them accordingly.

I've made the 5th test (with newline in the string) two cases, one with
\r\n for the newline, one with only \n. They fail differently.

I plan to look into this a little more, my current plan is to make the
outer regex non-greedy (if possible) and remove the trailing whitespace.
That would involve parsing (and ignoring) additional whitespace
*between* encoded words but not at the boundary to a non-encoded word.

Any objections/further infos?

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office at runtux.com
osAlliance member                       email: rsc at osalliance.com

----------
Added file: http://bugs.python.org/file24130/python.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________
-------------- next part --------------
diff -r 120a79b8bb11 Lib/test/test_email/test_email.py
--- a/Lib/test/test_email/test_email.py	Tue Jan 03 06:26:13 2012 +0200
+++ b/Lib/test/test_email/test_email.py	Tue Jan 03 16:16:09 2012 +0100
@@ -2056,6 +2056,67 @@
         self.assertEqual(decode_header(s),
                         [(b'andr\xe9=zz', 'iso-8659-1')])
 
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_1(self):
+        # 1st testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_2(self):
+        # 2nd testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= b)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b' b)', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_3(self):
+        # 3rd testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b'b', 'iso-8859-1'),
+             (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_4(self):
+        # 4th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?=  =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b'b', 'iso-8859-1'),
+             (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_5a(self):
+        # 5th testcase at end of rfc2047 newline is \r\n
+        s = '(=?ISO-8859-1?Q?a?=\r\n    =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b'b', 'iso-8859-1'),
+             (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_5b(self):
+        # 5th testcase at end of rfc2047 newline is \n
+        s = '(=?ISO-8859-1?Q?a?=\n    =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b'b', 'iso-8859-1'),
+             (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_6(self):
+        # 6th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a_b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a b', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_7(self):
+        # 7th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-2?Q?_b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b' b', 'iso-8859-2'),
+             (b')', None)])
+
 
 # Test the MIMEMessage class
 class TestMIMEMessage(TestEmailBase):

From report at bugs.python.org  Tue Jan  3 16:26:37 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Tue, 03 Jan 2012 15:26:37 +0000
Subject: [issue12394] packaging: generate scripts from callable (dotted paths)
In-Reply-To: <1308906864.25.0.392964418022.issue12394@psf.upfronthosting.co.za>
Message-ID: <1325604397.21.0.108914382616.issue12394@psf.upfronthosting.co.za>


Vinay Sajip <vinay_sajip at yahoo.co.uk> added the comment:

@?ric: you may also be interested in a standalone launcher which I wrote for the pythonv branch:

https://bitbucket.org/vinay.sajip/simple_launcher/

This is built using Visual Studio and is not based on setuptools code, but uses the same Windows API for child process creation and synchronisation as the PEP 397 launcher (rather than execv/spawnv as the setuptools launcher does). It also links with the runtime statically rather than linking with msvcrt.dll.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12394>
_______________________________________

From report at bugs.python.org  Tue Jan  3 16:26:53 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Tue, 03 Jan 2012 15:26:53 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1325604413.55.0.744028724927.issue13701@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Did you have specific changes in mind?

While we're still maintaining 2.x and 3.x code in parallel, there's a benefit to not having the versions of decimal.py diverge too much.  Given that the 2.3-compatible code isn't actually broken, I'm not sure that there's really that much to be gained by changing it.

The ugliest part of the current code is probably the post-application of classmethod instead of using a decorator;  I wouldn't object to fixing that.

I do agree with the general principle that the 3.x version of decimal.py doesn't need to stay backwards compatible with Python 2.3.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Tue Jan  3 16:36:57 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 15:36:57 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <1325605017.27.0.588451913955.issue13696@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

CRicky's proposed changed looks reasonable to me - although it would be better with a unit test too :)

----------
nosy: +gvanrossum, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Tue Jan  3 16:41:32 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 15:41:32 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325605292.67.0.148395473196.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

I tried to fix the problem and the correct fix is to change
    oup = ''
to
    oup = b''

in imaplib._Authenticator.encode() function, and not what I suggested in my previous post.

After changing that PLAIN authentication works.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:11:59 2012
From: report at bugs.python.org (Ralf Schlatterbeck)
Date: Tue, 03 Jan 2012 16:11:59 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <20120103152528.GA27700@runtux.com>
Message-ID: <20120103161157.GA32383@runtux.com>


Ralf Schlatterbeck <rsc at runtux.com> added the comment:

enclosed please find a fixed patch -- decode_header consolidates
multiple encoded strings with the same encoding into a single entry in
the returned parts.
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office at runtux.com
osAlliance member                       email: rsc at osalliance.com

----------
Added file: http://bugs.python.org/file24131/python.patch.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________
-------------- next part --------------
diff -r 120a79b8bb11 Lib/test/test_email/test_email.py
--- a/Lib/test/test_email/test_email.py	Tue Jan 03 06:26:13 2012 +0200
+++ b/Lib/test/test_email/test_email.py	Tue Jan 03 17:09:34 2012 +0100
@@ -2056,6 +2056,63 @@
         self.assertEqual(decode_header(s),
                         [(b'andr\xe9=zz', 'iso-8659-1')])
 
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_1(self):
+        # 1st testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_2(self):
+        # 2nd testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= b)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b' b)', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_3(self):
+        # 3rd testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_4(self):
+        # 4th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?=  =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_5a(self):
+        # 5th testcase at end of rfc2047 newline is \r\n
+        s = '(=?ISO-8859-1?Q?a?=\r\n    =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_5b(self):
+        # 5th testcase at end of rfc2047 newline is \n
+        s = '(=?ISO-8859-1?Q?a?=\n    =?ISO-8859-1?Q?b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_6(self):
+        # 6th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a_b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a b', 'iso-8859-1'), (b')', None)])
+
+    @unittest.expectedFailure
+    def test_rfc2047_rfc2047_7(self):
+        # 7th testcase at end of rfc2047
+        s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-2?Q?_b?=)'
+        self.assertEqual(decode_header(s),
+            [(b'(', None), (b'a', 'iso-8859-1'), (b' b', 'iso-8859-2'),
+             (b')', None)])
+
 
 # Test the MIMEMessage class
 class TestMIMEMessage(TestEmailBase):

From report at bugs.python.org  Tue Jan  3 17:16:06 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 03 Jan 2012 16:16:06 +0000
Subject: [issue13598] string.Formatter doesn't support empty curly braces "{}"
In-Reply-To: <1323835162.4.0.792542335688.issue13598@psf.upfronthosting.co.za>
Message-ID: <1325607366.24.0.350565206807.issue13598@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

test_string.diff looks good, except that it should probably only test the exception type, not the message (they are not a guaranteed part of the Python language and may change arbitrarily between versions or implementations (e.g. PyPy), so better not to add tests that depend on exact words).

I don?t have anything specific to say about issue13598.diff; if it makes the test pass, then it?s good.  ?if manual == True? should just be replaced by ?if manual?.

If you?d like to, you can make one patch with fix + tests that addresses my comments and remove the older diffs.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13598>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:17:11 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 16:17:11 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1188637019.25.0.0476259625696.issue1079@psf.upfronthosting.co.za>
Message-ID: <1325607431.79.0.487950824251.issue1079@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Well, a caution that tweaking the regex can have unexpected consequences as past issues have proven (but by all means go for it), and a note that the parsing strategy is going to change completely in email6 (see http://pypi.python.org/email and http://hg.python.org/features/email6).  I think your tests should pass on that branch; I'll be interested to try it when I get some time.

(Note: I'm removing 3.1 from versions since it doesn't get bug fixes any more.)

Also, I'm not sure the (non-essential) change to consolidate like-charset encoded words is appropriate for a bug fix.  It's hard to see how it would break anything, but why take the risk if it isn't needed to fix the bug.

----------
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:17:58 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 03 Jan 2012 16:17:58 +0000
Subject: [issue4755] Add function to get common path prefix
In-Reply-To: <1230350435.61.0.221193783724.issue4755@psf.upfronthosting.co.za>
Message-ID: <1325607478.76.0.524558288474.issue4755@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo
title: Common path prefix -> Add function to get common path prefix
type: behavior -> enhancement
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4755>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:18:27 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 16:18:27 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1188637019.25.0.0476259625696.issue1079@psf.upfronthosting.co.za>
Message-ID: <1325607507.28.0.286544517263.issue1079@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Gah, that's what I get for not reading carefully (or looking at the patch first).  Your test change is fine, of course.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:20:58 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 03 Jan 2012 16:20:58 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325607658.51.0.851761875258.issue13700@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Would you be interested in providing a patch that includes tests?  I think Antoine set up a test framework for testing the login as part of issue 4471.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:40:49 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 03 Jan 2012 16:40:49 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325608849.25.0.783307105368.issue8184@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

That's because SocketListener uses SO_REUSEADDR.
It seems that, with SO_REUSEADDR, Windows allows binding to a port even though there's a socket already bound to the same port in the LISTEN state: this is wrong, the semantics of SO_REUSEADDR was intended for sockets in TIME-WAIT state, and Linux and BSD systems implement this properly (i.e. fail with EADDRINUSE when there's a socket in LISTEN state).
The problem, if we remove this flag, is that managers binding to a specific port will get EADDRINUSE in case of rapid restart.
Since I'm not convinced that this is really an issue, I'd suggest to close this as "won't fix". Another option would be to complain to Microsoft :-)

@Phill: I'm not sure I understand your problem: could you be more specific (or open a new issue)?

----------
nosy: +neologix, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:42:43 2012
From: report at bugs.python.org (Patrick von Reth)
Date: Tue, 03 Jan 2012 16:42:43 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken
Message-ID: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>


New submission from Patrick von Reth <patrick.vonreth at gmail.com>:

when extracting http://www.openssl.org/source/openssl-1.0.0d.tar.gz with python3.2 on windows 7 extraction fails with 

  File "C:\python32\lib\tarfile.py", line 2175, in extract
    set_attrs=set_attrs)
  File "C:\python32\lib\tarfile.py", line 2259, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "C:\python32\lib\tarfile.py", line 2359, in makelink
    targetpath)
  File "C:\python32\lib\tarfile.py", line 2251, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\python32\lib\tarfile.py", line 2292, in makefile
    target = bltn_open(targetpath, "wb")
IOError: [Errno 22] Invalid argument: 'R:\\tmp\\os\\openssl-1.0.0d\\apps\\md4.c'

the reason is that the symlink is broken

R:\>dir R:\tmp\os\openssl-1.0.0d\apps\md4.c
 Volume in drive R has no label.
 Volume Serial Number is E8F0-7223
 Directory of R:\tmp\os\openssl-1.0.0d\apps
02.01.2012  20:13    <SYMLINK>      md4.c [../crypto/md4/md4.c]

it must be backslashes instead of front slashes and that's why python cant access the file the symlink is pointing to.

----------
components: Windows
messages: 150512
nosy: Patrick.von.Reth
priority: normal
severity: normal
status: open
title: relative symlinks in tarfile.extract broken
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:43:28 2012
From: report at bugs.python.org (Patrick von Reth)
Date: Tue, 03 Jan 2012 16:43:28 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325609008.44.0.367484568428.issue13702@psf.upfronthosting.co.za>


Changes by Patrick von Reth <patrick.vonreth at gmail.com>:


----------
title: relative symlinks in tarfile.extract broken -> relative symlinks in tarfile.extract broken (windows)

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:43:43 2012
From: report at bugs.python.org (Jeff Yurkiw)
Date: Tue, 03 Jan 2012 16:43:43 +0000
Subject: [issue13685] argparse does not sanitize help strings for % signs
In-Reply-To: <1325284167.16.0.584942168313.issue13685@psf.upfronthosting.co.za>
Message-ID: <1325609023.14.0.716583718235.issue13685@psf.upfronthosting.co.za>


Jeff Yurkiw <jeff at cyan.com> added the comment:

That would probably work too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13685>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:48:09 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 03 Jan 2012 16:48:09 +0000
Subject: [issue818201] distutils: clean does not use build_base option from
	build
Message-ID: <1325609289.77.0.301054523328.issue818201@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> Where was this fixed?  It is still a problem in Python 2.6.6.

I assumed it was fixed after looking at the code: clean does take build-* options from the build command.

> For example, if I do: 
> python setup.py build_ext --compiler=mingw32  build --build-platlib=build\win64
> Then follow it up with:
> python setup.py clean --build-base=build\win64 -a
> This is what it does:
> running clean
> 'build\lib.win-amd64-2.6' does not exist -- can't clean it
> removing 'build\bdist.win-amd64' (and everything under it)
> 'build\scripts-2.6' does not exist -- can't clean it
> As you can see, the base directory argument is ignored.

I?m not sure if this is a distutils bug or if you have to use the same options (i.e. build-lib both times, not build-platlib then build-base).  The original report used -b (build-base) for both commands, so I?ll turn that into a test (unless you?d like to do it?) to see if it works as intended or not.

----------
resolution: fixed -> 
stage: committed/rejected -> test needed
status: closed -> open
versions:  -Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue818201>
_______________________________________

From report at bugs.python.org  Tue Jan  3 17:51:57 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 03 Jan 2012 16:51:57 +0000
Subject: [issue9975] Incorrect use of flowinfo and scope_id in IPv6 sockaddr
	tuple
In-Reply-To: <1285698946.58.0.377750793336.issue9975@psf.upfronthosting.co.za>
Message-ID: <1325609517.32.0.45016954114.issue9975@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Should be fixed now.
Vilmos, thanks for the patch!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9975>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:04:29 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 17:04:29 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325610269.85.0.498899743552.issue8184@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

There's a length MSDN article about this:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621%28v=vs.85%29.aspx

Executive summary: it's a can of worms.

However, let me point out the following sentence:
?Ports without SO_EXCLUSIVEADDRUSE set may be reused as soon as the socket on which bind was previously called is closed.?

...which seems to suggest we shouldn't use SO_REUSEADDR under Windows, since Windows sockets appear to have the Unix SO_REUSEADDR semantics by default.

----------
resolution: fixed -> 
versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:20:57 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 03 Jan 2012 17:20:57 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325611257.91.0.274440731606.issue13697@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

The core of the problem is that we don't just want those methods to be atomic or thread-safe, but reentrant (or rather async-safe).
As such, protecting by a lock isn't enough (and it's not really feasible in Python).

Note that the RLock implementation already checks whether the lock is already acquire by the current thread, but there's a "race":
if self._owner == me:
    [...]
    self._count = self._count + 1
    return 1
rc = self._block.acquire(blocking, timeout)
if rc:
    self._owner = me

If the signal handler is called after _block has been acquired, but before self._owner is set, the next call to acquire from the signal handler won't "realize" that the block has already been acquired by the current thread, and will deadlock.

Now, the fun part: this affects not only RLock, but every Python code performing "atomic" actions: condition variables, barriers, etc. There are some constraints on what can be done from a signal handler, and it should probably be documented.

Note that another solution would be to use a dedicated thread for signal management (like Java does), but that's another story.

Also, this shouldn't be a problem for the buffered I/O code, since the code already accounts for this possibility (if the lock is already acquired by the current thread, an exception is raised).

Now, there's something I don't understand: I've just had a quick look, but AFAICT, there's no reason why the C version of RLock could not be available: am I missing something? Why do we even have a Python implementation?

----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:29:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 17:29:28 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325611257.91.0.274440731606.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325611700.3356.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Note that another solution would be to use a dedicated thread for
> signal management (like Java does), but that's another story.

That sounds like a good solution in the middle-term. Are there any
drawbacks? (apart from launching a thread)

> Also, this shouldn't be a problem for the buffered I/O code, since the
> code already accounts for this possibility (if the lock is already
> acquired by the current thread, an exception is raised).

Yes, but raising an exception is less helpful than doing what the user
asked for :)

> Now, there's something I don't understand: I've just had a quick look,
> but AFAICT, there's no reason why the C version of RLock could not be
> available: am I missing something? Why do we even have a Python
> implementation?

The C version is quite recent, and there's a school of thought that we
should always provide fallback Python implementations.
(also, arguably a Python implementation makes things easier to
prototype, although I don't think it's the case for an RLock)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:36:43 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 03 Jan 2012 17:36:43 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325612203.48.0.699765981988.issue8184@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

OK, so just removing SO_REUSEADDR on Windows should do the trick...
Seriously, why can't they simply conform to existing standards :-(
If someone wants to provide a patch + test, go ahead!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:39:11 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 03 Jan 2012 17:39:11 +0000
Subject: [issue9349] document argparse's help=SUPPRESS
In-Reply-To: <1279892796.46.0.177553560445.issue9349@psf.upfronthosting.co.za>
Message-ID: <E1Ri8KL-0004YJ-P0@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 572ddf2770bc by Sandro Tosi in branch '3.2':
Issue #9349: add argparse.SUPPRESS to help doc
http://hg.python.org/cpython/rev/572ddf2770bc

New changeset 17b7b856cbe8 by Sandro Tosi in branch '2.7':
Issue #9349: add argparse.SUPPRESS to help doc
http://hg.python.org/cpython/rev/17b7b856cbe8

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9349>
_______________________________________

From report at bugs.python.org  Tue Jan  3 18:41:39 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Tue, 03 Jan 2012 17:41:39 +0000
Subject: [issue9349] document argparse's help=SUPPRESS
In-Reply-To: <1279892796.46.0.177553560445.issue9349@psf.upfronthosting.co.za>
Message-ID: <1325612499.26.0.235654041421.issue9349@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9349>
_______________________________________

From report at bugs.python.org  Tue Jan  3 19:10:45 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Tue, 03 Jan 2012 18:10:45 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <1325614245.24.0.646372662308.issue13696@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

(This is in reference to issue 11662.)

I can't think of a way that this proposed change would bring back the original vulnerability, so go ahead.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Tue Jan  3 20:08:11 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Tue, 03 Jan 2012 19:08:11 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1325617691.39.0.307248962131.issue13701@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee:  -> rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Tue Jan  3 20:36:50 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Tue, 03 Jan 2012 19:36:50 +0000
Subject: [issue13703] Hash collision security issue
Message-ID: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>


New submission from Barry A. Warsaw <barry at python.org>:

This is already publicly known and in deep discussion on python-dev.  The proper fix is still TBD.  Essentially, hash collisions can be exploited to DoS a web framework that automatically parses input forms into dictionaries.

Start here:

http://mail.python.org/pipermail/python-dev/2011-December/115116.html

----------
components: Interpreter Core
messages: 150522
nosy: barry, benjamin.peterson, georg.brandl
priority: release blocker
severity: normal
status: open
title: Hash collision security issue
type: security
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 20:43:25 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Tue, 03 Jan 2012 19:43:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325619805.02.0.0980351026353.issue13703@psf.upfronthosting.co.za>


Changes by Guido van Rossum <guido at python.org>:


----------
nosy: +gvanrossum

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 20:48:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 19:48:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325620133.88.0.283299261126.issue13703@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +christian.heimes, pitrou
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:01:06 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Tue, 03 Jan 2012 20:01:06 +0000
Subject: [issue13114] check -r fails with non-ASCII unicode long_description
In-Reply-To: <1317907362.37.0.133459111826.issue13114@psf.upfronthosting.co.za>
Message-ID: <1325620866.36.0.757489784078.issue13114@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I recently encountered this error, and in the process of troubleshooting, developed this one-line to reproduce the problem. I'm including it here for posterity (and in case anybody wants to test a given version of Python for the error):

python -c "from distutils.core import setup; setup(name='foo.project', version='1.0', url='http://www.example.com', author='Foo', author_email='foo at example.com', long_description = u'\xc0',)" check -r

----------
nosy: +jason.coombs

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13114>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:17:01 2012
From: report at bugs.python.org (Phill)
Date: Tue, 03 Jan 2012 20:17:01 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325621821.94.0.514452167191.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

@neologix: nah its fine, if you guys are gonna re open this one I wont worry about opening a new bug.

If the above gets solved on windows my problem will just go away, thanks

----------
versions: +Python 2.6 -Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:19:26 2012
From: report at bugs.python.org (Christian Heimes)
Date: Tue, 03 Jan 2012 20:19:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325621966.44.0.819775459627.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

I had a short chat with Guido yesterday. I'll try to sum up the conversation. Guido, please correct me if I got something wrong or missed a point.

Guido wants the fix as simple and less intrusive as possible as he wants to provide/apply a patch for Python 2.4 to 3.3. This means any new stuff is off the table unless it's really, really necessary. Say goodbye to my experimental MurmurHash3 patch.

We haven't agreed whether the randomization should be enabled by default or disabled by default. IMHO it should be disabled for all releases except for the upcoming 3.3 release. The env var PYTHONRANDOMHASH=1 would enable the randomization. It's simple to set the env var in e.g. Apache for mod_python and mod_wsgi.

----------
stage: needs patch -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:24:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 20:24:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325621966.44.0.819775459627.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325622203.3356.4.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> We haven't agreed whether the randomization should be enabled by
> default or disabled by default. IMHO it should be disabled for all
> releases except for the upcoming 3.3 release.

I think on the contrary it must be enabled by default. Leaving security
holes open is wrong.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:24:33 2012
From: report at bugs.python.org (Christian Heimes)
Date: Tue, 03 Jan 2012 20:24:33 +0000
Subject: [issue13704] Random number generator in Python core
Message-ID: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>


New submission from Christian Heimes <lists at cheimes.de>:

All proposed fixes for a randomized hashing function raise and fall with
a good random number generator to feed the random seed. The seed must be
created very early in the startup phase of the interpreter, preferable
before the basic types are initialized. CPython already have multiple
sources for random data (win32_urandom in Modules/posixmodule.c, urandom
in Lib/os.py, Mersenne twister in Modules/_randommodule.c). However we
can't use them because they are wrapped inside Python modules which
require infrastructure like initialized base types.

Discussion at http://mail.python.org/pipermail/python-dev/2012-January/115263.html

My proposed changes are implemented in my feature fork but not yet well tested. Windows build files needs modification, too.

----------
assignee: christian.heimes
hgrepos: 98
messages: 150527
nosy: barry, benjamin.peterson, christian.heimes, georg.brandl, gvanrossum, pitrou
priority: release blocker
severity: normal
stage: patch review
status: open
title: Random number generator in Python core
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:28:14 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 03 Jan 2012 20:28:14 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325622494.07.0.162951513063.issue13704@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

(for the record, you can use the "create patch" button which creates a reviewable diff)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:31:16 2012
From: report at bugs.python.org (Christian Heimes)
Date: Tue, 03 Jan 2012 20:31:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325622676.69.0.910088403863.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

> I think on the contrary it must be enabled by default. Leaving security
> holes open is wrong.

We can't foresee the implications of the randomization and only a small number of deployments is affected by the problem. But I won't start a fight on the matter. ;)

----------
dependencies: +Random number generator in Python core
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:42:40 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 20:42:40 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325623360.44.0.245579357892.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

Here's a patch with test.

I am not an IMAP guru, so please verify my patch.

----------
keywords: +patch
Added file: http://bugs.python.org/file24132/issue13700.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:47:46 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Tue, 03 Jan 2012 20:47:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325623666.01.0.717139057796.issue13703@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

I'm with Antoine -- turn it on by default.  Maybe there should be a release candidate to test the waters.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:49:40 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Tue, 03 Jan 2012 20:49:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325622203.3356.4.camel@localhost.localdomain>
Message-ID: <20120103154934.2712c6c3@resist.wooz.org>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 03, 2012, at 08:24 PM, Antoine Pitrou wrote:

>I think on the contrary it must be enabled by default. Leaving security
>holes open is wrong.

Unless there's evidence of performance regressions or backward
incompatibilities, I agree.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 21:56:19 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 03 Jan 2012 20:56:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325624179.37.0.265842045767.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:20:59 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 03 Jan 2012 21:20:59 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325625659.84.0.0165439875969.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Unless there's evidence of performance regressions
> or backward incompatibilities, I agree.

If hash() is modified, str(dict) and str(set) will change for example. It may break doctests. Can we consider that the application should not rely (indirectly) on hash and so fix (for example) their doctests? Or is it a backward incompatibility?

hash() was already modified in major Python versions.

For this specific issue, I consider that security is more important than str(dict).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:43:35 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Tue, 03 Jan 2012 21:43:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325627015.26.0.118754080338.issue13703@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Barry, when this gets fixed, shall we coordinate release times?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:44:13 2012
From: report at bugs.python.org (Ralf Schlatterbeck)
Date: Tue, 03 Jan 2012 21:44:13 +0000
Subject: [issue1079] decode_header does not follow RFC 2047
In-Reply-To: <1325607431.79.0.487950824251.issue1079@psf.upfronthosting.co.za>
Message-ID: <20120103214410.GB32383@runtux.com>


Ralf Schlatterbeck <rsc at runtux.com> added the comment:

Attached please find a patch that
- keeps all spaces between non-encoded and encoded parts
- doesn't create spaces between non-encoded and encoded parts in case
  these are already there or not needed (because they are non-ctext
  characters of RFC822 like ')') in the methods "encode" and "__str__"
  of class Header.
in all other cases spaces are still inserted, this keeps many tests
happy and probably won't break too much existing code.

I've re-read RFC2047 (and parts of 822) and now share your opinion that
it requires that encoded parts *must* be followed by a
'linear-white-space' if the following (or preceding) token is text or ctext.
(p.7 5. Use of encoded-words in message headers)

With the special-casing of ctext characters mentioned above,
roundtripping is now possible, so if you parse a normalized string
consisting of encoded and non-encoded parts, (even multiple) whitespace
is preserved.

I still think we should do it like everyone else and *not* automatically
insert whitespace at boundaries between encoded and non-encoded words,
even if the RFC requires it. Someone wanting to create headers
consisting of mixed encoded/non-encoded parts without whitespace must
know what to do anyway -- the previous implementation also didn't check
for all border cases.

I've *not yet* tested this against the email6 branch you mentioned.

Note that I didn't have to make the regex non-greedy, it already
was. I've just removed the whitespace at the end of the regex.

I've changed all the tests that test for removal of whitespace between
non-encoded and encoded parts. Obviously I've also changed a test that
relied on failing to parse adjacent encoded strings. But please look at
my changes of the tests.

The rfc2047 tests now all pass.

The patch also fixes issue1467619 "Header.decode_header eats up spaces"

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: office at runtux.com
osAlliance member                       email: rsc at osalliance.com

----------
Added file: http://bugs.python.org/file24133/python.patch3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1079>
_______________________________________
-------------- next part --------------
diff -r 3609d32cec46 Lib/email/header.py
--- a/Lib/email/header.py	Tue Jan 03 17:48:19 2012 +0100
+++ b/Lib/email/header.py	Tue Jan 03 22:19:30 2012 +0100
@@ -40,7 +40,6 @@
   \?                    # literal ?
   (?P<encoded>.*?)      # non-greedy up to the next ?= is the encoded string
   \?=                   # literal ?=
-  (?=[ \t]|$)           # whitespace or the end of the string
   ''', re.VERBOSE | re.IGNORECASE | re.MULTILINE)
 
 # Field name regexp, including trailing colon, but not separating whitespace,
@@ -86,8 +85,12 @@
     words = []
     for line in header.splitlines():
         parts = ecre.split(line)
+        first = True
         while parts:
-            unencoded = parts.pop(0).strip()
+            unencoded = parts.pop(0)
+            if first:
+                unencoded = unencoded.lstrip()
+                first = False
             if unencoded:
                 words.append((unencoded, None, None))
             if parts:
@@ -95,6 +98,16 @@
                 encoding = parts.pop(0).lower()
                 encoded = parts.pop(0)
                 words.append((encoded, encoding, charset))
+    # Now loop over words and remove words that consist of whitespace
+    # between two encoded strings.
+    import sys
+    droplist = []
+    for n, w in enumerate(words):
+        if n>1 and w[1] and words[n-2][1] and words[n-1][0].isspace():
+            droplist.append(n-1)
+    for d in reversed(droplist):
+        del words[d]
+
     # The next step is to decode each encoded word by applying the reverse
     # base64 or quopri transformation.  decoded_words is now a list of the
     # form (decoded_word, charset).
@@ -217,22 +230,27 @@
         self._normalize()
         uchunks = []
         lastcs = None
+        lastspace = None
         for string, charset in self._chunks:
             # We must preserve spaces between encoded and non-encoded word
             # boundaries, which means for us we need to add a space when we go
             # from a charset to None/us-ascii, or from None/us-ascii to a
             # charset.  Only do this for the second and subsequent chunks.
+            # Don't add a space if the None/us-ascii string already has
+            # a space (trailing or leading depending on transition)
             nextcs = charset
             if nextcs == _charset.UNKNOWN8BIT:
                 original_bytes = string.encode('ascii', 'surrogateescape')
                 string = original_bytes.decode('ascii', 'replace')
             if uchunks:
+                hasspace = string and self._nonctext(string[0])
                 if lastcs not in (None, 'us-ascii'):
-                    if nextcs in (None, 'us-ascii'):
+                    if nextcs in (None, 'us-ascii') and not hasspace:
                         uchunks.append(SPACE)
                         nextcs = None
-                elif nextcs not in (None, 'us-ascii'):
+                elif nextcs not in (None, 'us-ascii') and not lastspace:
                     uchunks.append(SPACE)
+            lastspace = string and self._nonctext(string[-1])
             lastcs = nextcs
             uchunks.append(string)
         return EMPTYSTRING.join(uchunks)
@@ -286,6 +304,11 @@
             s.encode(output_charset, errors)
         self._chunks.append((s, charset))
 
+    def _nonctext(self, s):
+        """True if string s is not a ctext character of RFC822.
+        """
+        return s.isspace() or s in ('(', ')', '\\')
+
     def encode(self, splitchars=';, \t', maxlinelen=None, linesep='\n'):
         r"""Encode a message header into an RFC-compliant format.
 
@@ -329,7 +352,20 @@
             maxlinelen = 1000000
         formatter = _ValueFormatter(self._headerlen, maxlinelen,
                                     self._continuation_ws, splitchars)
+        lastcs = None
+        hasspace = lastspace = None
         for string, charset in self._chunks:
+            if hasspace is not None:
+                hasspace = string and self._nonctext(string[0])
+                import sys
+                if lastcs not in (None, 'us-ascii'):
+                    if not hasspace or charset not in (None, 'us-ascii'):
+                        formatter.add_transition()
+                elif charset not in (None, 'us-ascii') and not lastspace:
+                    formatter.add_transition()
+            lastspace = string and self._nonctext(string[-1])
+            lastcs = charset
+            hasspace = False
             lines = string.splitlines()
             if lines:
                 formatter.feed('', lines[0], charset)
@@ -346,6 +382,7 @@
                     formatter.feed(fws, sline, charset)
             if len(lines) > 1:
                 formatter.newline()
+        if self._chunks:
             formatter.add_transition()
         value = formatter._str(linesep)
         if _embeded_header.search(value):
diff -r 3609d32cec46 Lib/test/test_email/test_asian_codecs.py
--- a/Lib/test/test_email/test_asian_codecs.py	Tue Jan 03 17:48:19 2012 +0100
+++ b/Lib/test/test_email/test_asian_codecs.py	Tue Jan 03 22:19:30 2012 +0100
@@ -41,7 +41,7 @@
 Hello World! =?iso-2022-jp?b?GyRCJU8lbSE8JW8hPCVrJUkhKhsoQg==?=
  =?iso-8859-1?q?Gr=FC=DF_Gott!?=""")
         eq(decode_header(h.encode()),
-           [(b'Hello World!', None),
+           [(b'Hello World! ', None),
             (b'\x1b$B%O%m!<%o!<%k%I!*\x1b(B', 'iso-2022-jp'),
             (b'Gr\xfc\xdf Gott!', gcode)])
         subject_bytes = (b'test-ja \xa4\xd8\xc5\xea\xb9\xc6\xa4\xb5'
diff -r 3609d32cec46 Lib/test/test_email/test_email.py
--- a/Lib/test/test_email/test_email.py	Tue Jan 03 17:48:19 2012 +0100
+++ b/Lib/test/test_email/test_email.py	Tue Jan 03 22:19:30 2012 +0100
@@ -1999,9 +1999,9 @@
  foo bar =?mac-iceland?q?r=8Aksm=9Arg=8Cs?="""
         dh = decode_header(s)
         eq(dh, [
-            (b'Re:', None),
+            (b'Re: ', None),
             (b'r\x8aksm\x9arg\x8cs', 'mac-iceland'),
-            (b'baz foo bar', None),
+            (b' baz foo bar ', None),
             (b'r\x8aksm\x9arg\x8cs', 'mac-iceland')])
         header = make_header(dh)
         eq(str(header),
@@ -2010,36 +2010,38 @@
 Re: =?mac-iceland?q?r=8Aksm=9Arg=8Cs?= baz foo bar =?mac-iceland?q?r=8Aksm?=
  =?mac-iceland?q?=9Arg=8Cs?=""")
 
-    def test_whitespace_eater_unicode(self):
+    def test_whitespace_keeper_unicode(self):
         eq = self.assertEqual
         s = '=?ISO-8859-1?Q?Andr=E9?= Pirard <pirard at dom.ain>'
         dh = decode_header(s)
         eq(dh, [(b'Andr\xe9', 'iso-8859-1'),
-                (b'Pirard <pirard at dom.ain>', None)])
+                (b' Pirard <pirard at dom.ain>', None)])
         header = str(make_header(dh))
         eq(header, 'Andr\xe9 Pirard <pirard at dom.ain>')
 
-    def test_whitespace_eater_unicode_2(self):
+    def test_whitespace_keeper_unicode_2(self):
         eq = self.assertEqual
         s = 'The =?iso-8859-1?b?cXVpY2sgYnJvd24gZm94?= jumped over the =?iso-8859-1?b?bGF6eSBkb2c=?='
         dh = decode_header(s)
-        eq(dh, [(b'The', None), (b'quick brown fox', 'iso-8859-1'),
-                (b'jumped over the', None), (b'lazy dog', 'iso-8859-1')])
+        eq(dh, [(b'The ', None), (b'quick brown fox', 'iso-8859-1'),
+                (b' jumped over the ', None), (b'lazy dog', 'iso-8859-1')])
         hu = str(make_header(dh))
         eq(hu, 'The quick brown fox jumped over the lazy dog')
 
     def test_rfc2047_missing_whitespace(self):
         s = 'Sm=?ISO-8859-1?B?9g==?=rg=?ISO-8859-1?B?5Q==?=sbord'
         dh = decode_header(s)
-        self.assertEqual(dh, [(s, None)])
-
-    def test_rfc2047_with_whitespace(self):
-        s = 'Sm =?ISO-8859-1?B?9g==?= rg =?ISO-8859-1?B?5Q==?= sbord'
-        dh = decode_header(s)
         self.assertEqual(dh, [(b'Sm', None), (b'\xf6', 'iso-8859-1'),
                               (b'rg', None), (b'\xe5', 'iso-8859-1'),
                               (b'sbord', None)])
 
+    def test_rfc2047_with_whitespace(self):
+        s = 'Sm =?ISO-8859-1?B?9g==?= rg =?ISO-8859-1?B?5Q==?= sbord'
+        dh = decode_header(s)
+        self.assertEqual(dh, [(b'Sm ', None), (b'\xf6', 'iso-8859-1'),
+                              (b' rg ', None), (b'\xe5', 'iso-8859-1'),
+                              (b' sbord', None)])
+
     def test_rfc2047_B_bad_padding(self):
         s = '=?iso-8859-1?B?%s?='
         data = [                                # only test complete bytes
@@ -2056,62 +2058,56 @@
         self.assertEqual(decode_header(s),
                         [(b'andr\xe9=zz', 'iso-8659-1')])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_1(self):
         # 1st testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'a', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_2(self):
         # 2nd testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a?= b)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'a', 'iso-8859-1'), (b' b)', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_3(self):
         # 3rd testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-1?Q?b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_4(self):
         # 4th testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a?=  =?ISO-8859-1?Q?b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_5a(self):
         # 5th testcase at end of rfc2047 newline is \r\n
         s = '(=?ISO-8859-1?Q?a?=\r\n    =?ISO-8859-1?Q?b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_5b(self):
         # 5th testcase at end of rfc2047 newline is \n
         s = '(=?ISO-8859-1?Q?a?=\n    =?ISO-8859-1?Q?b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'ab', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_6(self):
         # 6th testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a_b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'a b', 'iso-8859-1'), (b')', None)])
 
-    @unittest.expectedFailure
     def test_rfc2047_rfc2047_7(self):
         # 7th testcase at end of rfc2047
         s = '(=?ISO-8859-1?Q?a?= =?ISO-8859-2?Q?_b?=)'
         self.assertEqual(decode_header(s),
             [(b'(', None), (b'a', 'iso-8859-1'), (b' b', 'iso-8859-2'),
              (b')', None)])
+        self.assertEqual(make_header(decode_header(s)).encode(), s.lower())
+        self.assertEqual(str(make_header(decode_header(s))), '(a b)')
 
 
 # Test the MIMEMessage class
@@ -4463,11 +4459,11 @@
         h = make_header(decode_header(s))
         eq(h.encode(), s)
 
-    def test_whitespace_eater(self):
+    def test_whitespace_keeper(self):
         eq = self.assertEqual
         s = 'Subject: =?koi8-r?b?8NLP18XSy8EgzsEgxsnOwczYztk=?= =?koi8-r?q?=CA?= zz.'
         parts = decode_header(s)
-        eq(parts, [(b'Subject:', None), (b'\xf0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 \xce\xc1 \xc6\xc9\xce\xc1\xcc\xd8\xce\xd9\xca', 'koi8-r'), (b'zz.', None)])
+        eq(parts, [(b'Subject: ', None), (b'\xf0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 \xce\xc1 \xc6\xc9\xce\xc1\xcc\xd8\xce\xd9\xca', 'koi8-r'), (b' zz.', None)])
         hdr = make_header(parts)
         eq(hdr.encode(),
            'Subject: =?koi8-r?b?8NLP18XSy8EgzsEgxsnOwczYztnK?= zz.')

From report at bugs.python.org  Tue Jan  3 22:47:15 2012
From: report at bugs.python.org (Erno Tukia)
Date: Tue, 03 Jan 2012 21:47:15 +0000
Subject: [issue13700] imaplib.IMAP4.authenticate authobject fails with PLAIN
	mechanism
In-Reply-To: <1325553089.14.0.504978991098.issue13700@psf.upfronthosting.co.za>
Message-ID: <1325627235.83.0.275841447291.issue13700@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

Here's another patch that should fix the CRAM-MD5 authentication. My previous patch is required with this one. The patch includes a test.

----------
Added file: http://bugs.python.org/file24134/cram_md5.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13700>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:48:21 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Tue, 03 Jan 2012 21:48:21 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325627301.7.0.0276796982644.issue13703@psf.upfronthosting.co.za>


Changes by Dave Malcolm <dmalcolm at redhat.com>:


----------
nosy: +dmalcolm

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:48:51 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 03 Jan 2012 21:48:51 +0000
Subject: [issue6031] BaseServer.shutdown documentation is incomplete
In-Reply-To: <1242431876.91.0.139343855315.issue6031@psf.upfronthosting.co.za>
Message-ID: <E1RiCDx-0006Qa-Mb@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4fad6b811c8b by Sandro Tosi in branch '2.7':
Issue #6031: improve serve_forever() description
http://hg.python.org/cpython/rev/4fad6b811c8b

New changeset 4a30d36a9c69 by Sandro Tosi in branch '3.2':
Issue #6031: improve serve_forever() description
http://hg.python.org/cpython/rev/4a30d36a9c69

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6031>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:50:03 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Tue, 03 Jan 2012 21:50:03 +0000
Subject: [issue6031] BaseServer.shutdown documentation is incomplete
In-Reply-To: <1242431876.91.0.139343855315.issue6031@psf.upfronthosting.co.za>
Message-ID: <1325627403.6.0.292950497723.issue6031@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

yep indeed, I've removed the deadlock part and committed.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6031>
_______________________________________

From report at bugs.python.org  Tue Jan  3 22:51:43 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 03 Jan 2012 21:51:43 +0000
Subject: [issue13636] Python SSL Stack doesn't have a Secure Default set of
	ciphers
In-Reply-To: <1324292408.0.0.0336711576939.issue13636@psf.upfronthosting.co.za>
Message-ID: <E1RiCGj-0006j1-Ue@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 25c2d24e1b11 by Antoine Pitrou in branch '3.2':
Issue #13636: Weak ciphers are now disabled by default in the ssl module
http://hg.python.org/cpython/rev/25c2d24e1b11

New changeset ace54f5e75d7 by Antoine Pitrou in branch 'default':
Issue #13636: Weak ciphers are now disabled by default in the ssl module
http://hg.python.org/cpython/rev/ace54f5e75d7

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13636>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:00:51 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 03 Jan 2012 22:00:51 +0000
Subject: [issue13636] Python SSL Stack doesn't have a Secure Default set of
	ciphers
In-Reply-To: <1324292408.0.0.0336711576939.issue13636@psf.upfronthosting.co.za>
Message-ID: <E1RiCPa-0007Z9-EC@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f9122975fd80 by Antoine Pitrou in branch '2.7':
Issue #13636: Weak ciphers are now disabled by default in the ssl module
http://hg.python.org/cpython/rev/f9122975fd80

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13636>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:02:46 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Tue, 03 Jan 2012 22:02:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325627015.26.0.118754080338.issue13703@psf.upfronthosting.co.za>
Message-ID: <20120103170241.015affb4@resist.wooz.org>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 03, 2012, at 09:43 PM, Benjamin Peterson wrote:

>Barry, when this gets fixed, shall we coordinate release times?

Yes!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:06:41 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Tue, 03 Jan 2012 22:06:41 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1271397834.22.0.547082855086.issue8416@psf.upfronthosting.co.za>
Message-ID: <1325628401.45.0.895529150089.issue8416@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi Georg, I've verified doc search works with 2.6.[467], so if the fix is so easy to just add that js, could you please give it a look? TIA

----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:08:19 2012
From: report at bugs.python.org (Christian Heimes)
Date: Tue, 03 Jan 2012 22:08:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325628499.85.0.576811724807.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Randomized hashing destabilizes the unit tests of Python, too. Here are the outputs of four test runs:

11 tests failed:
    test_collections test_dbm test_dis test_gdb test_inspect
    test_packaging test_set test_symtable test_ttk_textonly
    test_urllib test_urlparse

9 tests failed:
    test_dbm test_dis test_gdb test_json test_packaging test_set
    test_symtable test_urllib test_urlparse

10 tests failed:
    test_dbm test_dis test_gdb test_inspect test_packaging test_set
    test_symtable test_ttk_textonly test_urllib test_urlparse

9 tests failed:
    test_collections test_dbm test_dict test_dis test_gdb
    test_packaging test_symtable test_urllib test_urlparse

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:13:08 2012
From: report at bugs.python.org (Sinisa Segvic)
Date: Tue, 03 Jan 2012 22:13:08 +0000
Subject: [issue13705] Raising exceptions from finally works better than
	advertised in the documentation
Message-ID: <1325628788.79.0.655122317093.issue13705@psf.upfronthosting.co.za>


New submission from Sinisa Segvic <sinisa.segvic at fer.hr>:

Hi,

The documentation says:
"""
If the finally clause raises another exception (...) the saved exception is lost. 
"""

This does not appear to be true. 
In the example below the backtrace shows both exceptions.

>>> import math
>>> try:
...   1/0
... finally:
...   math.sqrt(-1)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
ValueError: math domain error

Cheers,

Sini?a

----------
assignee: docs at python
components: Documentation
messages: 150544
nosy: docs at python, ssegvic
priority: normal
severity: normal
status: open
title: Raising exceptions from finally works better than advertised in the documentation
type: enhancement
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13705>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:17:05 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 03 Jan 2012 22:17:05 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325629025.47.0.991947240587.issue13704@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I disagree with that approach, basically because I disagree that we need a C implementation of MT. Platforms that don't provide /dev/urandom will just have to be less secure. Using the current time (in milliseconds if available) plus the current pid ought to be good enough as a random seed.

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:18:27 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 03 Jan 2012 22:18:27 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325629107.16.0.651684072756.issue13704@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24135/3106cc0a2024.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:19:44 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Tue, 03 Jan 2012 22:19:44 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325629184.76.0.224505543539.issue13704@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:19:51 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Tue, 03 Jan 2012 22:19:51 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325629191.09.0.854201769183.issue13703@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:24:10 2012
From: report at bugs.python.org (Sinisa Segvic)
Date: Tue, 03 Jan 2012 22:24:10 +0000
Subject: [issue13705] Raising exceptions from finally works better than
	advertised in the documentation
In-Reply-To: <1325628788.79.0.655122317093.issue13705@psf.upfronthosting.co.za>
Message-ID: <1325629450.42.0.764871693607.issue13705@psf.upfronthosting.co.za>


Sinisa Segvic <sinisa.segvic at fer.hr> added the comment:

Link to the documentation:
http://docs.python.org/py3k/reference/compound_stmts.html#id2

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13705>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:31:02 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 03 Jan 2012 22:31:02 +0000
Subject: [issue13705] Raising exceptions from finally works better than
	advertised in the documentation
In-Reply-To: <1325628788.79.0.655122317093.issue13705@psf.upfronthosting.co.za>
Message-ID: <E1RiCsl-0002AV-Gc@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset c39fbb24b3f4 by Benjamin Peterson in branch '3.2':
exception support is correct now (closes #13705)
http://hg.python.org/cpython/rev/c39fbb24b3f4

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13705>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:36:25 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 03 Jan 2012 22:36:25 +0000
Subject: [issue13706] Unicode fill characters no longer work in numeric
	formatting
Message-ID: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

It used to be possible to specify Unicode fill characters in numeric
formatting:

Python 3.3.0a0 (default:1dd6908df8f5, Jul 16 2011, 11:16:00) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> format(1234, "\u2007<7")
'1234\u2007\u2007\u2007'
[64682 refs]
>>> 


Now it doesn't work:

Python 3.3.0a0 (default:65ac469d30e6, Jan  3 2012, 23:23:07) 
[GCC 4.4.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> format(1234, "\u2007<7")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: fill character too large

----------
components: Interpreter Core
messages: 150548
nosy: eric.smith, haypo, loewis, mark.dickinson, skrah
priority: normal
severity: normal
status: open
title: Unicode fill characters no longer work in numeric formatting
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:37:31 2012
From: report at bugs.python.org (Christian Heimes)
Date: Tue, 03 Jan 2012 22:37:31 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325630251.75.0.0575980289331.issue13704@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

We already have a C implementation of MT in Modules/_randommodule.c. I just suggest that we move the implementation to a place, where we can use it as seed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:44:11 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Tue, 03 Jan 2012 22:44:11 +0000
Subject: [issue13706] Unicode fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325630651.43.0.947076224853.issue13706@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

It's still possible; it's just apparently limited to ASCII characters.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:47:50 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 03 Jan 2012 22:47:50 +0000
Subject: [issue13706] Unicode fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325630870.13.0.496082945126.issue13706@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:48:55 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 03 Jan 2012 22:48:55 +0000
Subject: [issue13706] Unicode fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325630935.21.0.512176719766.issue13706@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Hum, somehow I always refuse to acknowledge that ASCII is a subset
of Unicode. :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:49:16 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 03 Jan 2012 22:49:16 +0000
Subject: [issue13706] non-ascii fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325630956.82.0.976796432738.issue13706@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
title: Unicode fill characters no longer work in numeric formatting -> non-ascii fill characters no longer work in numeric formatting

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:51:27 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Tue, 03 Jan 2012 22:51:27 +0000
Subject: [issue13706] non-ascii fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325631087.53.0.921233956643.issue13706@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

I assume this is left over from the PEP 393 changes. I think the right thing to do is delete this code from line 277 of formatter_unicode.c:

    if (format->fill_char > 127 || format->align > 127 ||
        format->sign > 127) {
        PyErr_SetString(PyExc_ValueError, "fill character too large");
        return 0;
    }

I'm not sure such a restriction needs to exist any more. But I'll admit to not having thought it through.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Tue Jan  3 23:59:53 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 03 Jan 2012 22:59:53 +0000
Subject: [issue13706] non-ascii fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325631593.36.0.0505643403688.issue13706@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I assume this is left over from the PEP 393 changes.

Correct.

> I'm not sure such a restriction needs to exist any more.

The restriction was introduced to simplify the implementation. maxchar has to be computed exactly in format_string_internal(), format_int_or_long_internal(), format_float_internal() and format_complex_internal().

For format_string_internal(), the following change is enough (untested):

    if (lpad != 0 || rpad != 0)
        maxchar = Py_MAX(maxchar, format->fill_char);

For number formatting functions, spec->n_lpadding, spec->n_spadding and spec->n_rpadding have to be checked. Something like:

    if (spec->n_lpadding || spec->n_spadding || spec->n_rpadding)
        maxchar = Py_MAX(maxchar, format->fill_char);

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:07:35 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 03 Jan 2012 23:07:35 +0000
Subject: [issue13706] non-ascii fill characters no longer work in numeric
	formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325632055.33.0.0319034176152.issue13706@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Removing the if condition would be incorrect. The maximum char is computed at the beginning of the formatting. If, during formatting, need for a padding character is determined, the padding character must not be larger than the maximum char of the target string - which is 127 unless 'c' formatting is used.

One solution would be to determine whether the padding character is used in advance. Another solution would be to widen the string when the need for non-ASCII padding is detected.

I have no intention of fixing this issue myself; I don't mind if non-ASCII padding characters are not supported in 3.3.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:41:12 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 03 Jan 2012 23:41:12 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1271397834.22.0.547082855086.issue8416@psf.upfronthosting.co.za>
Message-ID: <1325634072.56.0.0170060777438.issue8416@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Santiago, if you are still running 2.6.5 code, use the most recent 2.6 docs at
http://docs.python.org/release/2.6.6/
This will have all the corrections made after the 2.6.5 release. Contrary to what you might think the header line says, there is no particular connection between the 2.6.5 code release and the obsolete 2.6.5 doc release.

George, since the continuously updated x.y docs released with x.y.z really document x.y and not each x.y.z bugfix release, I am a bit surprised that they are labelled x.y.z docs with the claim "This is the documentation for Python x.y.z", especially since they are updated after the x.y.z code release.

The current '2.7.2' docs, last updated today, would more truthfully be called either '2.7' docs or '2.7.3a0' docs, as they are are a preview of what will be released with 2.7.3 and are not what was released with 2.7.2. If there *were* (unusally) any new features in 2.7.3a0, they would already be listed in the so-called '2.7.2' docs. (There *was* such a bugfix addition for difflib.SequenceMatcher in 2.7.1, and I presume it did appear in the updated '2.7.0' online docs.)

I am not sure we should have obsolete snapshot versions online. They only serve to be an attractive nuisance as illustrated by this issue. With the initial 3.3 release being called 3.3.0, there would be no ambiguity in calling the 3.3 docs just that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:42:09 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 03 Jan 2012 23:42:09 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1271397834.22.0.547082855086.issue8416@psf.upfronthosting.co.za>
Message-ID: <1325634129.11.0.362876890412.issue8416@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Sorry, /George/Georg/

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:44:16 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 03 Jan 2012 23:44:16 +0000
Subject: [issue13706] non-ascii fill characters no longer work in formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1325634256.66.0.419765190817.issue13706@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Actually the issue is not restricted to numeric formatting. It's not
possible to pad a Unicode string with a non-ascii whitespace:

>>> format("abcd", "\u2007<7")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: fill character too large



I'd be more than happy to restrict all numerical I/O operations to
ASCII. This includes input strings for int(), float(), Decimal().


It does break backwards compatibility though and the situation
for string formatting above seems odd to me. It is worse when
the rejected fill character is already present in the string:


>>> format("\u2007abcd", "\u2007<7")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: fill character too large

----------
title: non-ascii fill characters no longer work in numeric formatting -> non-ascii fill characters no longer work in formatting

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:52:48 2012
From: report at bugs.python.org (Paul McMillan)
Date: Tue, 03 Jan 2012 23:52:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325634768.26.0.450418980694.issue13703@psf.upfronthosting.co.za>


Paul McMillan <paul at mcmillan.ws> added the comment:

I agree that we should enable randomness by default, and provide an easy way for users to disable it if necessary (unit test suites that explicitly depend on order being an obvious candidate).

I'll link my proposed algorithm change here, for the record:
https://gist.github.com/0a91e52efa74f61858b5

I've gotten confirmation from several other sources that the fix recommended by the presenters (just a random initialization seed) only prevents the most basic form of the attack.

----------
nosy: +PaulMcMillan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 00:55:12 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 03 Jan 2012 23:55:12 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1271397834.22.0.547082855086.issue8416@psf.upfronthosting.co.za>
Message-ID: <1325634912.39.0.0837993457432.issue8416@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:22:37 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 00:22:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325636557.52.0.9529812418.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Christian Heimes proposes the following change in its randomhash branch (see issue #13704):

-    x = (Py_uhash_t) *p << 7;
+    x = Py_RndHashSeed + ((Py_uhash_t) *p << 7);
     for (i = 0; i < len; i++)
         x = (1000003U * x) ^ (Py_uhash_t) *p++;
     x ^= (Py_uhash_t) len;

This change doesn't add any security if the attacker can inject any string and retreive the hash value. You can retreive directly Py_RndHashSeed using:

Py_RndHashSeed = intmask((hash("a") ^ len("a") ^ ord("a")) * DIVIDE) - (ord("a") << 7)

where intmask() truncates to a long (x mod 2^(long bits)) and DIVIDE = 1/1000003 mod 2^(long bits). For example, DIVIDE=2021759595 for 32 bits long.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:33:10 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Wed, 04 Jan 2012 00:33:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325637190.12.0.420267270643.issue13703@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:35:15 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Wed, 04 Jan 2012 00:35:15 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325637315.68.0.254355757018.issue13704@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy: +Arfrever, ezio.melotti, michael.foord

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:38:29 2012
From: report at bugs.python.org (Christian Heimes)
Date: Wed, 04 Jan 2012 00:38:29 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325637509.62.0.233651420659.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Victor, please ignore my code related to hash randomization for now. I've deliberately not linked my branch to this bug report. I'm well aware that it's not secure and that it's pretty easy to reverse engineer the seed from a hash of a short string. The code is a proof of concept to detect failing tests and other issues.

I'm in private contact with Paul and we are working together. He has done extended research and I'll gladly follow his expertise. I've already discussed the issue with small strings, but I can't recall if it was a private mail to Paul or a public one to the dev list.

Paul:
I still think that you should special case short strings (five or few chars sound good). An attacker can't do much harm with one to five char strings but such short strings may make it too easy to calculate the seed.

16kb of seed is still a lot. Most CPUs have about 16 to 32, maybe 64kb L1 cache for data. 1024 to 4096 bytes should increase cache locality and reduce speed impacts.

PS: I'm going to reply to your last mail tomorrow.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:40:53 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 04 Jan 2012 00:40:53 +0000
Subject: [issue13707] Clarify hash() lifetime
Message-ID: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>


New submission from Terry J. Reedy <tjreedy at udel.edu>:

Current 3.2.2 docs:

id(object) Return the ?identity? of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime. [model]

hash(object) Return the hash value of the object (if it has one). Hash values are integers. They are used to quickly compare dictionary keys 

Suggestion: change "Hash values are integers. They ..." to
"This should be an integer which is constant for this object during its lifetime. Hash values ..."

Rationale: For builtin class instances, hash values are guaranteed to be constant that long, and only that long, as the default hash(ob) for object() instances is currently, for my win7, 64 bit, 3.2.2 CPython, id(ob) // 16 (the minimum object size). User class instance hashes (with custom __hash__) *should* have the same lifetime. But since Python cannot enforce that, I did not say 'guaranteed'.

User code should *not* depend on a longer lifetime, just as for id() output. It seems worth implying that, as for id(), because (based on recent pydev discussion) people seems to be prone to over-generalize the current longer-term stability of number and string hashes, which itself may disappear in future releases. (see #13703)

----------
assignee: docs at python
components: Documentation
messages: 150561
nosy: docs at python, terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: Clarify hash() lifetime
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:49:38 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Wed, 04 Jan 2012 00:49:38 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325638178.7.0.154817611421.issue13704@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy:  -ezio.melotti, michael.foord

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:55:05 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 04 Jan 2012 00:55:05 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325638505.95.0.983259782726.issue13703@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

In #13707 I suggest a change to the current hash() entry which is needed independently of this issue, because the default hash (for object()), being tied to id() is already limited to an object's lifetime. But this change will become more imperative if hash() is made run-dependent for numbers and strings.

There does not seems to presently *be* a security hole for 64 bit builds, so if there is any noticeable slowdown on 64 bit builds and it is sensibly easy to tie the default to the bitness, I would think it should be off for such builds.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 01:56:28 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Wed, 04 Jan 2012 00:56:28 +0000
Subject: [issue13707] Clarify hash() lifetime
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325638588.27.0.581367994431.issue13707@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:00:55 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 01:00:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325638855.8.0.411042960043.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Paul first proposition (on python-dev) was to replace:

    ...
    x = (ord(s[0]) << 7)
    while i < length:
        x = intmask((1000003*x) ^ ord(s[i]))
        ...

by:

    ...
    x = (ord(s[0]) << 7)
    while i < length:
        x = intmask((1000003*x) ^ ord(s[i])) ^ r[x % len_r]
        ...

This change has a vulnerability similar than the one of Christian's suggested changed. The "r" array can be retreived directly with:

r2 = []
for i in xrange(len(r)):
    s = chr(intmask(i * UNSHIFT7) % len(r))
    h = intmask(hash(s) ^ len(s) ^ ord(s) ^ ((ord(s) << 7) * MOD))
    r2.append(chr(h))
r2 = ''.join(r2)

where UNSHIFT7 = 1/2**7 mod 2^(long bits).

By the way, this change always use r[0] to hash all string of one ASCII character (U+0000-U+007F).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:17:17 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Wed, 04 Jan 2012 01:17:17 +0000
Subject: [issue13707] Clarify hash() lifetime
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325639837.32.0.774190850224.issue13707@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

-1. The hash has nothing to do with the lifetime, but with the value of an object.

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:30:01 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 01:30:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325637509.62.0.233651420659.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325640529.3356.6.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I'm in private contact with Paul and we are working together. He has
> done extended research and I'll gladly follow his expertise. I've
> already discussed the issue with small strings, but I can't recall if
> it was a private mail to Paul or a public one to the dev list.

Can all this be discussed on this issue now that it's the official point
of reference? It will avoid the repetition of arguments we see here and
there.

(I don't think special-casing small strings makes sense, because then
you have two algorithms to audit rather than one)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:31:56 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 01:31:56 +0000
Subject: [issue13636] Python SSL Stack doesn't have a Secure Default set of
	ciphers
In-Reply-To: <1324292408.0.0.0336711576939.issue13636@psf.upfronthosting.co.za>
Message-ID: <1325640716.75.0.318413884937.issue13636@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I've committed a conservative version of the patch, plus a test.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13636>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:45:33 2012
From: report at bugs.python.org (Santiago Gala)
Date: Wed, 04 Jan 2012 01:45:33 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1325634912.46.0.837525244404.issue8416@psf.upfronthosting.co.za>
Message-ID: <CAPXTYCVEW_GLSzdZ_rT2vg=A9YHa0dSS_x4CUVmpr4oXZE3n1A@mail.gmail.com>


Santiago Gala <sgala at apache.org> added the comment:

Still http://docs.python.org/release/2.6.6/search.html?q=regular+expression
works, while http://docs.python.org/release/2.6.5/search.html?q=regular+expression
fails, and http://docs.python.org/release/2.6.5/searchindex.js gives a
404, while http://docs.python.org/release/2.6.6/searchindex.js works
so the bug report stands as it was reported...

Regards
Santiago

On Wed, Jan 4, 2012 at 12:55 AM, Ezio Melotti <report at bugs.python.org> wrote:
>
> Changes by Ezio Melotti <ezio.melotti at gmail.com>:
>
>
> ----------
> nosy: +ezio.melotti
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue8416>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:54:53 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 01:54:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325642093.98.0.114808804311.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> https://gist.github.com/0a91e52efa74f61858b5

Please, attach directly a file to the issue, or copy/paste the code in your comment. Interesting part the code:
---

#Proposed replacement
#--------------------------------------
import os, array
size_exponent = 14 #adjust as a memory/security tradeoff
r = array.array('l', os.urandom(2**size_exponent))
len_r = len(r)

def _hash_string2(s):
    """The algorithm behind compute_hash() for a string or a unicode."""
    length = len(s)
    #print s
    if length == 0:
        return -1
    x = (ord(s[0]) << 7) ^ r[length % len_r]
    i = 0
    while i < length:
        x = intmask((1000003*x) ^ ord(s[i]))
        x ^= r[x % len_r]
        i += 1
    x ^= length
    return intmask(x)
---

> r = array.array('l', os.urandom(2**size_exponent))
> len_r = len(r)

r size should not depend on the size of a long. You should write something like:

sizeof_long = ctypes.sizeof(ctypes.c_long)
r_bits = 8
r = array.array('l', os.urandom((2**r_bits) * sizeof_long))
r_mask = 2**r_bits-1

and then replace "% len_r" by "& r_mask".

What is the minimum value of r_bits? For example, would it be safe to use a single long integer? (r_bits=1)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 02:58:04 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 01:58:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325642093.98.0.114808804311.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325642213.3356.8.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > r = array.array('l', os.urandom(2**size_exponent))
> > len_r = len(r)
> 
> r size should not depend on the size of a long. You should write something like:
> 
> sizeof_long = ctypes.sizeof(ctypes.c_long)
> r_bits = 8
> r = array.array('l', os.urandom((2**r_bits) * sizeof_long))
> r_mask = 2**r_bits-1

The final code will be in C and will use neither ctypes nor array.array.
Arguing about this looks quite pointless IMO.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:14:55 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 02:14:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325643295.39.0.0632700309253.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

For the record, here is what "man urandom" says about random seed size:

?[...] no cryptographic primitive available today can hope to promise 
more than 256  bits of  security,  so  if  any  program  reads more than 
256 bits (32 bytes) from the kernel random pool per invocation, or per 
reasonable  reseed  interval (not less than one minute), that should be
taken as a sign that its cryptography  is  not  skilfully  implemented.?

In that light, reading a 64 bytes seed from /dev/urandom is already a lot, and 4096 bytes is simply insane.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:16:27 2012
From: report at bugs.python.org (Zhiping Deng)
Date: Wed, 04 Jan 2012 02:16:27 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325643387.27.0.47876997278.issue13703@psf.upfronthosting.co.za>


Changes by Zhiping Deng <kofreestyler at gmail.com>:


----------
nosy: +Zhiping.Deng

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:27:07 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 04 Jan 2012 02:27:07 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325644027.07.0.488667331632.issue13704@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee: christian.heimes -> rhettinger
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:33:30 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 02:33:30 +0000
Subject: [issue13708] Document ctypes.wintypes
Message-ID: <1325644409.99.0.366875630851.issue13708@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
nosy: ramchandra.apte
priority: normal
severity: normal
status: open
title: Document ctypes.wintypes

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13708>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:33:58 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 02:33:58 +0000
Subject: [issue13708] Document ctypes.wintypes
Message-ID: <1325644438.51.0.00118311970114.issue13708@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

Document ctypes.wintypes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13708>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:37:50 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 02:37:50 +0000
Subject: [issue13708] Document ctypes.wintypes
In-Reply-To: <1325644438.51.0.00118311970114.issue13708@psf.upfronthosting.co.za>
Message-ID: <1325644670.18.0.218756607694.issue13708@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13708>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:38:50 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 04 Jan 2012 02:38:50 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325644730.01.0.0342595612945.issue13707@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Martin, I do not understand. The default hash is based on id (as is default equality comparison), not value. Are you OK with hash values changing if the 'value' changes? My understanding is that changing hash values for objects in sets and dicts is bad, which is why mutable builtins with value-based equality do not have hash values.

----------
title: Clarify hash() lifetime -> Clarify hash() constancy period

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:40:38 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 02:40:38 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325644838.03.0.91190374723.issue13707@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

You can define a __hash__ that changes if the object changes. It is not recommended, but it's possible. So I agree with Martin that your proposed clarification is wrong.
(I also think that it wouldn't bring anything, either)

Suggest closing as invalid/rajected.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:46:44 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 02:46:44 +0000
Subject: [issue13709] Capitalization mistakes in the documentation for ctypes
Message-ID: <1325645203.91.0.562347938508.issue13709@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

In section 15.17.1.17 in the ctypes documentation, the documentation says
"It is funny to see that on linux the sort function seems to work much more efficiently, it is doing less comparisons"

"It is quite interesting to see that the Windows qsort() function needs
more comparisons than the linux version!"

"linux" should be capitalized in both these sentences.

----------
assignee: docs at python
components: Documentation
messages: 150574
nosy: docs at python, ramchandra.apte
priority: normal
severity: normal
status: open
title: Capitalization mistakes in the documentation for ctypes

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13709>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:51:01 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 04 Jan 2012 02:51:01 +0000
Subject: [issue9201] IDLE: raises Exception TclError in a special case
In-Reply-To: <1278600015.52.0.587656960584.issue9201@psf.upfronthosting.co.za>
Message-ID: <1325645461.74.0.789890193008.issue9201@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

The problem was originally reported in Issue4676 and fixed partly there and partly in Issue3851.

----------
nosy: +ned.deily
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> python3 closes + home keys

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9201>
_______________________________________

From report at bugs.python.org  Wed Jan  4 03:51:11 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Wed, 04 Jan 2012 02:51:11 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325611700.3356.3.camel@localhost.localdomain>
Message-ID: <CAH_1eM0u5aMhGu3WnLa_CckJ_p48zsSHMpUqpaVj2w39XCLt4w@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> That sounds like a good solution in the middle-term. Are there any
> drawbacks? (apart from launching a thread)

Just to be clear: the approach I was suggesting is to have a resident
thread dedicated to signal management, not to spawn a new one when
needed. Another advantage is that we could mask signals in all threads
except this one, and have a consistent cross-platform behavior with
respect to signals+threads.

However I see two drawbacks:
- it seems that we want to allow building Python without threads
support. In that case, this wouldn't work, or we would need the
current implementation as a fallback, but this would complicate the
code somewhat.
- the thread would have to be respawned after a fork()

> The C version is quite recent, and there's a school of thought that we
> should always provide fallback Python implementations.
> (also, arguably a Python implementation makes things easier to
> prototype, although I don't think it's the case for an RLock)

Hmmm, I'm not convinced by this argument in this specific case: since
the C version is now the default, is faster and more reliable, can't
we drop the Python version?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:08:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 03:08:15 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325646495.38.0.255430053195.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I read that the attack cannot be computed with actual computers (it's too expensive) against Python 64 bits. I tried to change str.__hash__ in Python 32 bits to compute the hash in 64 bits and than truncate the hash to 32 bits: it doesn't change anything, the hash values are the same, so it doesn't improve the security.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:15:53 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 03:15:53 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325646953.48.0.683443362251.issue13697@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> This affects the python implementation of RLock only.

In the issue #13550, it was discussed to remove completly the logging machinery from the threading module. If we remove it, we don't need the Python implementation of RLock.

We already removed the Python implementation of subprocess.Popen._execute_child() in Python 3.3 because of issues with fork, signals and threads.

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:25:33 2012
From: report at bugs.python.org (Robert Collins)
Date: Wed, 04 Jan 2012 03:25:33 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325647533.22.0.105180601826.issue13697@psf.upfronthosting.co.za>


Robert Collins <robertc at robertcollins.net> added the comment:

Normally I advocate very strongly for Python implementation of C accelerated modules, but when the two implementations are not equivalent, having a simpler Python one around does not help anyone (not users, other language implementors etc). True reentrancy is possible but quite hard to achieve. 

So, FWIW, +1 on just having a C implementation.

The dedicated signal thread sounds useful too - it would ease the issues for folk using other parts of the stdlib, and would in principle permit a pure-python RLock to hang around.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:34:28 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 04 Jan 2012 03:34:28 +0000
Subject: [issue13558] multiprocessing package incompatible with PyObjC
In-Reply-To: <1323375974.18.0.406983892582.issue13558@psf.upfronthosting.co.za>
Message-ID: <1325648068.12.0.934206883195.issue13558@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

Issue8713 proposes adding the option of using processes for multiprocessing rather than bare forks.  Let's consolidate this issue with that one.

----------
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> multiprocessing needs option to eschew fork() under Linux

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13558>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:37:27 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 03:37:27 +0000
Subject: [issue13699] test_gdb has recently started failing
In-Reply-To: <1325546341.52.0.130827872784.issue13699@psf.upfronthosting.co.za>
Message-ID: <1325648247.71.0.208261087262.issue13699@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

"test_gdb has started failing recently on my Ubuntu Natty system:
...
>>> sysconfig.get_config_vars()['PY_CFLAGS']
'-Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes'
...
Marking haypo as nosy since it might be related to recent changes for #13628."

Before my last change for #13628, test_gdb was skipped completly when Python was compiled in release mode (with gcc optimization, which is the default).

Except if you have an idea to improve python-gdb.py to make it find the frame pointer, the simple fix is to skip the failing tests if Python is optimized. I already add some @unittest.skipIf(python_is_optimized(), "Python was compiled with optimizations") in my commit 0b03cb97dac0.

Would you like to write such patch?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13699>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:37:48 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 04 Jan 2012 03:37:48 +0000
Subject: [issue8713] multiprocessing needs option to eschew fork() under Linux
In-Reply-To: <1273853591.12.0.245920632829.issue8713@psf.upfronthosting.co.za>
Message-ID: <1325648268.1.0.932385507475.issue8713@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

See also consolidated Issue13558 for additional justification for processes option on OS X.

----------
nosy: +mrmekon, ned.deily

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:39:19 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 04 Jan 2012 03:39:19 +0000
Subject: [issue13556] When tzinfo.utcoffset is out-of-bounds,
	the exception message is misleading
In-Reply-To: <1323364991.44.0.541305100354.issue13556@psf.upfronthosting.co.za>
Message-ID: <1325648359.9.0.0805136164629.issue13556@psf.upfronthosting.co.za>


Changes by Ned Deily <nad at acm.org>:


----------
nosy: +belopolsky, lemburg

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13556>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:40:26 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Wed, 04 Jan 2012 03:40:26 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325538793.4.0.376965160112.issue13697@psf.upfronthosting.co.za>
Message-ID: <1325648426.75.0.762256622997.issue13697@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:43:45 2012
From: report at bugs.python.org (Eli Bendersky)
Date: Wed, 04 Jan 2012 03:43:45 +0000
Subject: [issue13709] Capitalization mistakes in the documentation for ctypes
In-Reply-To: <1325645203.91.0.562347938508.issue13709@psf.upfronthosting.co.za>
Message-ID: <1325648625.55.0.473507705698.issue13709@psf.upfronthosting.co.za>


Eli Bendersky <eliben at gmail.com> added the comment:

And while we're at it, I would consider revising both sentences in a more fundamental way, since they're too chatty for the official docs.

----------
nosy: +eli.bendersky

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13709>
_______________________________________

From report at bugs.python.org  Wed Jan  4 04:51:31 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Wed, 04 Jan 2012 03:51:31 +0000
Subject: [issue13556] When tzinfo.utcoffset is out-of-bounds,
	the exception message is misleading
In-Reply-To: <1323364991.44.0.541305100354.issue13556@psf.upfronthosting.co.za>
Message-ID: <1325649091.3.0.632765285004.issue13556@psf.upfronthosting.co.za>


Alexander Belopolsky <alexander.belopolsky at gmail.com> added the comment:

Is 3.3 message better?

>>> datetime.now(tz=X())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: offset must be a timedelta strictly between -timedelta(hours=24) and timedelta(hours=24).

In 2.7, the message is indeed misleading:

>>> datetime.now(tz=X())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: tzinfo.utcoffset() returned 1440; must be in -1439 .. 1439

I am not sure fixing this in 2.x is worth the trouble, but I would consider improving the message in 3.x by adding information about the actual offset.  I vaguely remember that there was a reason for leaving that info out in 3.x.

----------
assignee:  -> belopolsky
stage:  -> needs patch
versions: +Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13556>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:07:22 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 04 Jan 2012 04:07:22 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325650042.15.0.203440691379.issue13707@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Given that the doc says that use of hash() is to compare dict keys, it does not seem wrong to me to suggest that hash() should be usable to do so.

I believe id() and consequently hash() are unique among builtins in being run-dependent. That is currently documented for id() but not for hash(). Given that people seriously asked whether we can randomize hash() with each run, because 'people' 'expect' it to remain rather constant, it does not seem useless to clarify that it can change with each run. I am sure my wording could be improved. An alternative would be 'Hash values for built-in objects are constant for each run but not necessarily thereafter."

If you take into account what people can do with special methods, some of the other entries seem more wrong that my suggestion. For instance:
"len(s) Return the length (the number of items) of an object." and
"str(obj ... When only object is given, this returns its nicely printable representation." These are true only for built-in objects, but the policy is to leave out the qualification.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:32:57 2012
From: report at bugs.python.org (Brian Curtin)
Date: Wed, 04 Jan 2012 04:32:57 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325651577.8.0.332517127251.issue13702@psf.upfronthosting.co.za>


Changes by Brian Curtin <brian at python.org>:


----------
nosy: +brian.curtin
stage:  -> test needed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:36:13 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 04:36:13 +0000
Subject: [issue13550] Rewrite logging hack of the threading module
In-Reply-To: <1323297626.55.0.00727033353184.issue13550@psf.upfronthosting.co.za>
Message-ID: <1325651773.41.0.433895753286.issue13550@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13550>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:37:31 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 04:37:31 +0000
Subject: [issue4188] test_threading hang when running as verbose
In-Reply-To: <1224793703.28.0.169476277835.issue4188@psf.upfronthosting.co.za>
Message-ID: <1325651851.15.0.266249740525.issue4188@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4188>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:38:49 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 04 Jan 2012 04:38:49 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325651929.71.0.602422868047.issue13707@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

-1 I concur with Martin.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:39:52 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 04 Jan 2012 04:39:52 +0000
Subject: [issue13128] httplib debuglevel on CONNECT doesn't print response
	headers
In-Reply-To: <1318050704.97.0.551622649018.issue13128@psf.upfronthosting.co.za>
Message-ID: <1325651992.34.0.831207896251.issue13128@psf.upfronthosting.co.za>


Changes by Ned Deily <nad at acm.org>:


----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13128>
_______________________________________

From report at bugs.python.org  Wed Jan  4 05:58:50 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 04:58:50 +0000
Subject: [issue13710] hash() on string containing only null characters returns
	the length of the string
Message-ID: <1325653130.87.0.569297022599.issue13710@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

If you run hash on strings containing only null characters it returns the length of the string
>>> hash("\0")
1
>>> hash("\0\0")
2
>>> hash("\0"*1000)
1000
This behaviour is not like proper hash functions.
The hashes of these strings should exhibit the avalanche effect like in a proper hash function.

----------
components: Interpreter Core
messages: 150587
nosy: ramchandra.apte
priority: normal
severity: normal
status: open
title: hash() on string containing only null characters returns the length of the string
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13710>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:00:38 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 05:00:38 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325653238.68.0.839319796103.issue13703@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:01:49 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 05:01:49 +0000
Subject: [issue13710] hash() on string containing only null characters returns
	the length of the strings
In-Reply-To: <1325653130.87.0.569297022599.issue13710@psf.upfronthosting.co.za>
Message-ID: <1325653309.41.0.915887425164.issue13710@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
title: hash() on string containing only null characters returns the length of the string -> hash() on string containing only null characters returns the length of the strings

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13710>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:02:10 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Wed, 04 Jan 2012 05:02:10 +0000
Subject: [issue13710] hash() on strings containing only null characters
	returns the length of the strings
In-Reply-To: <1325653130.87.0.569297022599.issue13710@psf.upfronthosting.co.za>
Message-ID: <1325653330.17.0.864140936958.issue13710@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
title: hash() on string containing only null characters returns the length of the strings -> hash() on strings containing only null characters returns the length of the strings

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13710>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:05:25 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Wed, 04 Jan 2012 05:05:25 +0000
Subject: [issue13710] hash() on strings containing only null characters
	returns the length of the strings
In-Reply-To: <1325653130.87.0.569297022599.issue13710@psf.upfronthosting.co.za>
Message-ID: <1325653525.23.0.301390379409.issue13710@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

No. Python's dictionary implementation is designed to handle hash functions with the properties of Python's.

----------
nosy: +benjamin.peterson
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13710>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:07:20 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 05:07:20 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325653640.91.0.0427207033699.issue13707@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:09:02 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 05:09:02 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325653742.74.0.129109001196.issue13704@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:09:59 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 05:09:59 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325653799.93.0.461049922261.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Yet another random hash function, simplified version of Paul's function. It always use exactly 256 bits of entropy and so 32 bytes of memory, and doesn't keep the loop. I don't expect my function to be secure, but just give more work to the attacker to compute the data for an attack against our dict implementation.

---
import os, array, struct
sizeof_long = struct.calcsize("l")
r_bits = 256
r_len = r_bits // (sizeof_long * 8)
r_mask = r_len - 1
r = array.array('l', os.urandom(r_len * sizeof_long))

def randomhash(s):
    length = len(s)
    if length == 0:
        return -2
    x = ord(s[0])
    x ^= r[x & r_mask]
    x <<= 7
    for ch in s:
        x = intmask(1000003 * x)
        x ^= ord(ch)
    x ^= length
    x ^= r[x & r_mask]
    return intmask(x)
---

The first "x ^= r[x & r_mask]" may be replaced by "x ^= r[(x ^ length) & r_mask]".

The binary shift is done after the first xor with r, because 2**7 and r_len are not coprime (r_len is a multipler of 2**7), and so (ord(s[0] << 7) & r_mask is always zero.

randomhash(s)==hash(s) if we used twice the same index in the r array. I don't know if this case gives useful information.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:14:51 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 04 Jan 2012 05:14:51 +0000
Subject: [issue13710] hash() on strings containing only null characters
	returns the length of the strings
In-Reply-To: <1325653130.87.0.569297022599.issue13710@psf.upfronthosting.co.za>
Message-ID: <1325654091.8.0.350286593882.issue13710@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Object hashes are not crypto hashes. Use hashlib module for crypto.

----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13710>
_______________________________________

From report at bugs.python.org  Wed Jan  4 06:59:30 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Wed, 04 Jan 2012 05:59:30 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <1325656770.42.0.805590268198.issue13696@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Here is the patch against 3.2. Something is wrong with the mercurial at the moment where 3.2 is shown as inactive. Once that is corrected, I shall commit and push this to 3.2 and cpython codelines.

This bug is not present in 2.7 as different logic is followed in there.

----------
keywords: +patch
Added file: http://bugs.python.org/file24136/13696.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Wed Jan  4 07:00:38 2012
From: report at bugs.python.org (Paul McMillan)
Date: Wed, 04 Jan 2012 06:00:38 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325656838.58.0.858793240155.issue13703@psf.upfronthosting.co.za>


Paul McMillan <paul at mcmillan.ws> added the comment:

A couple of things here:

First, my proposed change is not cryptographically secure. There simply aren't any cryptographic hashing algorithms available that are in the performance class we need. My proposal does make the collision attack quite difficult to carry out, even if the raw output values from the hash are available to the attacker (they should not usually be).

I favor using the same algorithm between 32 and 64 bit builds for consistency of behavior. Developers would be startled to find that ordering stays consistent on a 64 bit build but varies on 32 bit builds. Additionally, the impracticality of attacking of 64 bit builds rests on the fact that these particular researchers didn't devise a way to do it. I'd hate to make this change and then have a clever mathematician publish some elegant point requiring us to go fix the problem all over again. 

I could be convinced either way on small strings. I like that they can't be used to attack the secret. At the same time, I worry that combining 2 different hashing routines into the same output space may introduce unexpected collisions and other difficult to debug edge-case conditions. It also means that the order of the hashes of long strings will vary while the order of short strings will not - another inconsistency which will encourage bugs.

Thank you Victor for the improvements to the python demonstration code. As Antoine said, it's only demo code, but better demo code is always good.

Antoine: That section of the manpage is referring to the overall security of a key generated using urandom. 256 bits is overkill for this application. We could take 256 bits and use them to generate a key using a cryptographically appropriate algorithm, but it's simpler to read more bits and use them directly as the key.

Additionally, that verbiage has been in the man page for urandom for quite some time (probably since the earliest version in the mid 90's). The PRNG has been improved since then.

Minimum length of r is a hard question. The shorter it is, the higher the correlation of the output. In my tests, 16kb was the amount necessary to generally do reasonably well on my test suite for randomness even with problematic input. Obviously our existing output isn't random, so it doesn't pass those tests at all. Using a fairly small value (4k) should not make the results much worse from a security perspective, but might be problematic from a collision/distribution standpoint. It's clear that we don't need cryptographically good randomness here, but passing the test suite is not a bad thing when considering the distribution.

When we settle on a C implementation, I'd like to run it through the smhasher set of tests to make sure we aren't making distribution worse, especially for very small values of r.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 07:14:34 2012
From: report at bugs.python.org (=?utf-8?q?Lars_Gust=C3=A4bel?=)
Date: Wed, 04 Jan 2012 06:14:34 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325657674.76.0.672152072339.issue13702@psf.upfronthosting.co.za>


Changes by Lars Gust?bel <lars at gustaebel.de>:


----------
assignee:  -> lars.gustaebel
nosy: +lars.gustaebel
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Wed Jan  4 07:47:45 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 04 Jan 2012 06:47:45 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <E1RiKdU-0002az-7L@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 86141d28b20d by Senthil Kumaran in branch '3.2':
Issue13696 - Fix 302 Redirection for Relative urls.
http://hg.python.org/cpython/rev/86141d28b20d

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Wed Jan  4 08:26:30 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Wed, 04 Jan 2012 07:26:30 +0000
Subject: [issue13696] [urllib.request.HTTPRedirectHandler.http_error_302]
	Relative Redirect issue
In-Reply-To: <1325529630.8.0.473539706261.issue13696@psf.upfronthosting.co.za>
Message-ID: <1325661990.91.0.178317209359.issue13696@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
resolution:  -> fixed
stage: test needed -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13696>
_______________________________________

From report at bugs.python.org  Wed Jan  4 08:30:56 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 04 Jan 2012 07:30:56 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325662256.01.0.0467049291251.issue13704@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Why is this listed as a release blocker?  It is questionable whether it should be done at all?  It is a very aggressive change.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 09:17:42 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Wed, 04 Jan 2012 08:17:42 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325644730.01.0.0342595612945.issue13707@psf.upfronthosting.co.za>
Message-ID: <4F040B24.7030403@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> Martin, I do not understand. The default hash is based on id (as is
> default equality comparison), not value.

In the default implementation, the id *is* the object's value (i.e.
objects, by default, only compare equal if they are identical). So
the default implementation is just a special case of the more general
rule that hashes need to be consistent with equality.

> Are you OK with hash values changing if the 'value' changes?

An object that can change its value (i.e. a mutable object) should
fail to hash.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:13:56 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 04 Jan 2012 09:13:56 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325644730.01.0.0342595612945.issue13707@psf.upfronthosting.co.za>
Message-ID: <4F04184E.6020704@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Terry J. Reedy wrote:
> 
> Terry J. Reedy <tjreedy at udel.edu> added the comment:
> 
> Martin, I do not understand. The default hash is based on id (as is default equality comparison), not value. Are you OK with hash values changing if the 'value' changes? My understanding is that changing hash values for objects in sets and dicts is bad, which is why mutable builtins with value-based equality do not have hash values.

Hash values are based on the object values, not their id(). See the
various type implementations as reference. The id() is only used
as hash for objects which don't have a "value" (and thus cannot be
compared).

Given that we have the invariant "a==b => hash(a)==hash(b)" in Python,
it immediately follows that hash values for objects with comparison
method cannot have a lifetime - at least not within the same process
and, depending how you look at it, also not in multi-process
applications.

----------
nosy: +lemburg

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:18:32 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 04 Jan 2012 09:18:32 +0000
Subject: [issue8416] python 2.6.5 documentation can't search
In-Reply-To: <1271397834.22.0.547082855086.issue8416@psf.upfronthosting.co.za>
Message-ID: <1325668712.03.0.321521614795.issue8416@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

The continually updated docs are built from the stable branches, whose version remains at (e.g.) 2.7.2 until 2.7.3a1 is released, at which point the continuous updating stops until 2.7.3 is final.

I don't think presenting docs with an alpha version on the http://docs.python.org/ frontpage is useful. On the other hand, I do think it is important to have doc fixed reflected (more or less) instantly somewhere, so that e.g. people reporting typos can see the fixes. The status quo is a compromise between these two needs.

When we do make backwards incompatible changes or additions during a stable cycle, they need to be marked with "new/changed in version 2.7.X+1" anyway.  So the SequenceMatcher change would alert users itself. If not, that's a bug.

About the "obsolete" snapshots: I don't know what you're referring to there: if it's the released docs for specific versions, then I think that's standard practice to have a doc version released for a specific Python version; and I wouldn't change that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8416>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:29:25 2012
From: report at bugs.python.org (Christian Heimes)
Date: Wed, 04 Jan 2012 09:29:25 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325669365.75.0.214069563566.issue13704@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Release blocker: I was following the example in #13703. A RNG (PRNG or CSPRNG) is required for randomized hashing function.

The patch contains more than just the RNG changes. Only Include/pyrandom.h, Modules/_randommodule.c, Modules/posixmodule.c, Python/hash.c and parts of Makefile.pre.in are relevant for this tracker item. Sorry for the inconvenience!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:39:47 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 04 Jan 2012 09:39:47 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1325669987.93.0.33730640633.issue13704@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee: rhettinger -> christian.heimes

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:46:44 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 04 Jan 2012 09:46:44 +0000
Subject: [issue13707] Clarify hash() constancy period
In-Reply-To: <1325637653.51.0.0128051850964.issue13707@psf.upfronthosting.co.za>
Message-ID: <1325670404.57.0.870054111372.issue13707@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

[Antoine]
> Suggest closing as invalid/rajected.

[Martin]
> -1. The hash has nothing to do with the lifetime, 
> but with the value of an object.

----------
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13707>
_______________________________________

From report at bugs.python.org  Wed Jan  4 10:59:35 2012
From: report at bugs.python.org (Mark Shannon)
Date: Wed, 04 Jan 2012 09:59:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325671175.23.0.440344351776.issue13703@psf.upfronthosting.co.za>


Changes by Mark Shannon <mark at hotpy.org>:


----------
nosy: +Mark.Shannon

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 11:52:14 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 10:52:14 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <CAH_1eM0u5aMhGu3WnLa_CckJ_p48zsSHMpUqpaVj2w39XCLt4w@mail.gmail.com>
Message-ID: <1325674262.3458.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > That sounds like a good solution in the middle-term. Are there any
> > drawbacks? (apart from launching a thread)
> 
> Just to be clear: the approach I was suggesting is to have a resident
> thread dedicated to signal management, not to spawn a new one when
> needed. Another advantage is that we could mask signals in all threads
> except this one, and have a consistent cross-platform behavior with
> respect to signals+threads.

Hmm, but that would break single-threaded programs which expect their
select() (or other) to return EINTR when a signal is received (which is
a perfectly valid expectation in that case).

> However I see two drawbacks:
> - it seems that we want to allow building Python without threads
> support. In that case, this wouldn't work, or we would need the
> current implementation as a fallback, but this would complicate the
> code somewhat.

I don't know if that's still useful to build Python without threads. I
would expect most platforms to have a compatible threads implementation
(and Python probably can't run on very small embedded platforms).
Perhaps you can ask on python-dev.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Wed Jan  4 12:03:00 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 11:03:00 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325674980.48.0.821134686697.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Using a fairly small value (4k) should not make the results much worse 
> from a security perspective, but might be problematic from a
> collision/distribution standpoint.

Keep in mind the average L1 data cache size is between 16KB and 64KB. 4KB is already a significant chunk of that.

Given a hash function's typical loop is to feed back the current result into the next computation, I don't see why a small value (e.g. 256 bytes) would be detrimental.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 13:07:37 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 04 Jan 2012 12:07:37 +0000
Subject: [issue13699] test_gdb has recently started failing
In-Reply-To: <1325546341.52.0.130827872784.issue13699@psf.upfronthosting.co.za>
Message-ID: <E1RiPd1-0004j4-Tu@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset dfffb293f4b3 by Vinay Sajip in branch 'default':
Closes #13699. Skipped two tests if Python is optimised.
http://hg.python.org/cpython/rev/dfffb293f4b3

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13699>
_______________________________________

From report at bugs.python.org  Wed Jan  4 14:26:47 2012
From: report at bugs.python.org (=?utf-8?q?Manuel_B=C3=A4renz?=)
Date: Wed, 04 Jan 2012 13:26:47 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
Message-ID: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>


New submission from Manuel B?renz <manuel at enigmage.de>:

I've attached a script which demonstrates the bug.

When feeding a <script> that contains a comment tag with the actual script and the script containing tags itself (e.g. a 'document.write(<td></td>)'), the parser doesn't call handle_comment and handle_starttag.

----------
components: Library (Lib)
files: htmlparserbug.py
messages: 150603
nosy: turion
priority: normal
severity: normal
status: open
title: html.parser.HTMLParser doesn't parse tags in comments in scripts correctly
type: behavior
versions: Python 3.2
Added file: http://bugs.python.org/file24137/htmlparserbug.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 14:38:28 2012
From: report at bugs.python.org (=?utf-8?q?Manuel_B=C3=A4renz?=)
Date: Wed, 04 Jan 2012 13:38:28 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325684308.37.0.206467719002.issue13711@psf.upfronthosting.co.za>


Manuel B?renz <manuel at enigmage.de> added the comment:

I forgot to say, I'm using python version 3.2.2.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 14:55:45 2012
From: report at bugs.python.org (R. David Murray)
Date: Wed, 04 Jan 2012 13:55:45 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325685345.02.0.645510031048.issue13711@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

The content of a script tag is CDATA.  Why would you expect it to be parsed?

----------
nosy: +ezio.melotti, r.david.murray

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 15:25:36 2012
From: report at bugs.python.org (=?utf-8?q?Manuel_B=C3=A4renz?=)
Date: Wed, 04 Jan 2012 14:25:36 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325687136.45.0.124276819372.issue13711@psf.upfronthosting.co.za>


Manuel B?renz <manuel at enigmage.de> added the comment:

Oh, I wasn't aware of that.
Then, the bug is actually calling handle_endtag.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 15:28:47 2012
From: report at bugs.python.org (=?utf-8?q?Manuel_B=C3=A4renz?=)
Date: Wed, 04 Jan 2012 14:28:47 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325687327.65.0.888576083389.issue13711@psf.upfronthosting.co.za>


Manuel B?renz <manuel at enigmage.de> added the comment:

To clarify this even further: Consider
parser_instance.feed("<script><td></td></script>")

It should call:
parser_instance.handle_starttag("script", [])
parser_instance.handle_data("<td></td>")
parser_instance.handle_endtag("script", [])

Instead, it calls:
parser_instance.handle_starttag("script", [])
parser_instance.handle_data("<td>")
parser_instance.handle_endtag("td", [])
parser_instance.handle_endtag("script", [])

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 15:42:30 2012
From: report at bugs.python.org (R. David Murray)
Date: Wed, 04 Jan 2012 14:42:30 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325688150.92.0.000381258245158.issue13711@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

I believe this was fixed recently as part of issue 670664.  Ezio will know for sure.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 15:52:27 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 04 Jan 2012 14:52:27 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325688747.71.0.727033282609.issue13703@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

If test_packaging fails because it relies on dict order / hash details, that?s a bug.  Can you copy the full tb (possibly in another report, I can fix it independently of this issue)?

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 15:58:04 2012
From: report at bugs.python.org (Christian Heimes)
Date: Wed, 04 Jan 2012 14:58:04 +0000
Subject: [issue13712] test_packaging depends on hash order
Message-ID: <1325689084.75.0.043764224634.issue13712@psf.upfronthosting.co.za>


New submission from Christian Heimes <lists at cheimes.de>:

As requested in http://bugs.python.org/issue13703#msg150609

./python Lib/test/regrtest.py test_packaging
[1/1] test_packaging
Warning -- threading._dangling was modified by test_packaging
Warning -- sysconfig._SCHEMES was modified by test_packaging
test test_packaging failed -- Traceback (most recent call last):
  File "/home/heimes/dev/python/randomhash/Lib/packaging/tests/test_create.py", line 168, in test_convert_setup_py_to_cfg
    """))
AssertionError: '[metadata]\nname = pyxfoil\nversion = 0.2\nsummary = Python bindings for the Xf [truncated]... != '[metadata]\nname = pyxfoil\nversion = 0.2\nsummary = Python bindings for the Xf [truncated]...
  [metadata]
  name = pyxfoil
  version = 0.2
  summary = Python bindings for the Xfoil engine
  download_url = UNKNOWN
  home_page = http://www.python-science.org/project/pyxfoil
  maintainer = Andr? Espaze
  maintainer_email = andre.espaze at logilab.fr
  description = My super Death-scription
         |barbar is now on the public domain,
         |ho, baby !
  
  [files]
  packages = pyxfoil
      babar
      me
  modules = my_lib
      mymodule
  scripts = my_script
      bin/run
- extra_files = setup.py
+ extra_files = Martinique/Lamentin/dady
+     Martinique/Lamentin/mumy
+     Martinique/Lamentin/sys
+     Martinique/Lamentin/bro
+     setup.py
      README
-     pyxfoil/fengine.so
      Pom
      Flora
      Alexander
+     pyxfoil/fengine.so
-     Martinique/Lamentin/dady
-     Martinique/Lamentin/mumy
-     Martinique/Lamentin/sys
-     Martinique/Lamentin/bro
  
  resources =
      README.rst = {doc}
      pyxfoil.1 = {man}
  


1 test failed:
    test_packaging

----------
assignee: eric.araujo
components: Distutils2
messages: 150610
nosy: alexis, christian.heimes, eric.araujo
priority: normal
severity: normal
status: open
title: test_packaging depends on hash order
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13712>
_______________________________________

From report at bugs.python.org  Wed Jan  4 16:02:18 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Wed, 04 Jan 2012 15:02:18 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325689338.22.0.909986291768.issue13711@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Yep, this was fixed in #670664.
With the development version of Python (AFAIK the fix has not be released yet) and the example parser found in the doc[0] I get this:

>>> parser = MyHTMLParser()
>>> parser.feed('<script><td></td></script>')
Encountered a start tag: script
Encountered   some data: <td></td>
Encountered  an end tag: script


[0]: http://docs.python.org/dev/library/html.parser.html#example-html-parser-application

----------
assignee:  -> ezio.melotti
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> HTMLParser.py - more robust SCRIPT tag parsing

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 16:02:57 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Wed, 04 Jan 2012 15:02:57 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325662256.01.0.0467049291251.issue13704@psf.upfronthosting.co.za>
Message-ID: <20120104100251.78decfc2@limelight.wooz.org>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 04, 2012, at 07:30 AM, Raymond Hettinger wrote:

>Why is this listed as a release blocker?  It is questionable whether it
>should be done at all?  It is a very aggressive change.

It's a release blocker so that the issue won't get ignored before the next
release.  That doesn't necessarily mean it must be fixed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan  4 16:08:37 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Wed, 04 Jan 2012 15:08:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325656838.58.0.858793240155.issue13703@psf.upfronthosting.co.za>
Message-ID: <20120104100832.00e0e5a3@limelight.wooz.org>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 04, 2012, at 06:00 AM, Paul McMillan wrote:

>Developers would be startled to find that ordering stays consistent on a 64
>bit build but varies on 32 bit builds.

Well, one positive outcome of this issue is that users will finally viscerally
understand that dictionary (and set) order should never be relied upon, even
between successive runs of the same Python executable.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 17:19:16 2012
From: report at bugs.python.org (=?utf-8?q?Manuel_B=C3=A4renz?=)
Date: Wed, 04 Jan 2012 16:19:16 +0000
Subject: [issue13711] html.parser.HTMLParser doesn't parse tags in comments in
	scripts correctly
In-Reply-To: <1325683607.39.0.829837031313.issue13711@psf.upfronthosting.co.za>
Message-ID: <1325693956.68.0.359581759953.issue13711@psf.upfronthosting.co.za>


Manuel B?renz <manuel at enigmage.de> added the comment:

Great! Thank you!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13711>
_______________________________________

From report at bugs.python.org  Wed Jan  4 17:38:40 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Wed, 04 Jan 2012 16:38:40 +0000
Subject: [issue13713] Regression for http.client read()
Message-ID: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>


New submission from Ross Lagerwall <rosslagerwall at gmail.com>:

806cfe39f729 introduced a regression for http.client read(len).

To see this:
$ ./python test.py
$ wget http://archives.fedoraproject.org/pub/archive/fedora/linux/core/1/SRPMS/
$ diff index.html index2.html

This is a difference in the files (which there shouldn't be).

The change which introduced the problem was:
changeset:   73875:806cfe39f729
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Dec 06 22:33:57 2011 +0100
summary:     Issue #13464: Add a readinto() method to http.client.HTTPResponse.

----------
components: Library (Lib)
files: test.py
messages: 150615
nosy: orsenthil, pitrou, rosslagerwall
priority: normal
severity: normal
stage: needs patch
status: open
title: Regression for http.client read()
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24138/test.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 17:42:06 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 04 Jan 2012 16:42:06 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325656838.58.0.858793240155.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F048156.30006@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Some comments:

1. The security implications in all this is being somewhat overemphasized.

There are many ways you can do a DoS attack on web servers. It's the
responsibility of the used web frameworks and servers to deal with
the possible cases.

It's a good idea to provide some way to protect against hash
collision attacks, but that will only solve one possible way of
causing a resource attack on a server.

There are other ways you can generate lots of CPU overhead with
little data input (e.g. think of targeting the search feature on
many Zope/Plone sites).

In order to protect against such attacks in general, we'd have to
provide a way to control CPU time and e.g. raise an exception if too
much time is being spent on a simple operation such as a key insertion.
This can be done using timers, signals or even under OS control.

The easiest way to protect against the hash collision attack is by
limiting the POST/GET/HEAD request size.

The second best way would be to limit the number of parameters that a
web framework accepts for POST/GET/HEAD request.

2. Changing the semantics of hashing in a dot release is not allowed.

If randomization of the hash start vector or some other method is
enabled by default in a dot release, this will change the semantics
of any application switching to that dot release.

The hash values of Python objects are not only used by the Python
dictionary implementation, but also by other storage mechanisms
such as on-disk dictionaries, inter-process object exchange via
share memory, memcache, etc.

Hence, if changed, the hash change should be disabled per default
for dot releases and enabled for 3.3.

3. Changing the way strings are hashed doesn't solve the problem.

Hash values of other types can easily be guessed as well, e.g.
take integers which use a trivial hash function.

We'd have to adapt all hash functions of the basic types in Python
or come up with a generic solution using e.g. double-hashing
in the dictionary/set implementations.

4. By just using a random start vector you change the absolute
hash values for specific objects, but not the overall hash sequence
or its period.

An attacker only needs to create many hash collisions, not
specific ones. It's the period of the hash function that's
important in such attacks and that doesn't change when moving to
a different start vector.

5. Hashing needs to be fast.

It's one of the most used operations in Python. Please get experts into
the boat like Tim Peters and Christian Tismer, who both have worked
on the dict implementation and the hash functions, before experimenting
with ad-hoc fixes.

6. Counting collisions could solve the issue without having to
change hashing.

Another idea would be counting the collisions and raising an
exception if the number of collisions exceed a certain
threshold.

Such a change would work for all hashable Python objects and
protect against the attack without changing any hash function.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

----------
nosy: +lemburg

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 17:53:45 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Wed, 04 Jan 2012 16:53:45 +0000
Subject: [issue12660] test_gdb fails when installed
In-Reply-To: <1312038409.05.0.274432685937.issue12660@psf.upfronthosting.co.za>
Message-ID: <1325696025.67.0.154170342183.issue12660@psf.upfronthosting.co.za>


Vinay Sajip <vinay_sajip at yahoo.co.uk> added the comment:

Pending the real fix, I've attached a patch to skip the test if it's not a source build.

----------
keywords: +patch
nosy: +vinay.sajip
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24139/test-gdb-patch.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12660>
_______________________________________

From report at bugs.python.org  Wed Jan  4 17:58:15 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 04 Jan 2012 16:58:15 +0000
Subject: [issue13712] test_packaging depends on hash order
In-Reply-To: <1325689084.75.0.043764224634.issue13712@psf.upfronthosting.co.za>
Message-ID: <1325696295.18.0.573513878598.issue13712@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks, I will check this.

----------
versions: +3rd party

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13712>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:14:54 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 17:14:54 +0000
Subject: [issue13713] Regression for http.client read()
In-Reply-To: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>
Message-ID: <1325697294.08.0.0510890589989.issue13713@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +Jon.Kuhn
priority: normal -> critical

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:18:30 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 04 Jan 2012 17:18:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F048156.30006@egenix.com>
Message-ID: <4F0489DF.4020101@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Marc-Andre Lemburg wrote:
> 
> 3. Changing the way strings are hashed doesn't solve the problem.
> 
> Hash values of other types can easily be guessed as well, e.g.
> take integers which use a trivial hash function.

Here's an example for integers on a 64-bit machine:

>>> g = ((x*(2**64 - 1), hash(x*(2**64 - 1))) for x in xrange(1, 1000000))
>>> d = dict(g)

This takes ages to complete and only uses very little memory.
The input data has some 32MB if written down in decimal numbers
- not all that much data either.

32397634

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:22:43 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 04 Jan 2012 17:22:43 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325697763.57.0.620603987811.issue13703@psf.upfronthosting.co.za>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

The email interface ate part of my reply:

>>> g = ((x*(2**64 - 1), hash(x*(2**64 - 1))) for x in xrange(1, 1000000))
>>> s = ''.join(str(x) for x in g)
>>> len(s)
32397634
>>> g = ((x*(2**64 - 1), hash(x*(2**64 - 1))) for x in xrange(1, 1000000))
>>> d = dict(g)
... lots of time for coffee, pizza, taking a walk, etc. :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:41:21 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 04 Jan 2012 17:41:21 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325698881.85.0.77647586716.issue13703@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

To expand on Marc-Andre's point 1: the DOS attack on web servers is possible because servers are generally dumb at the first stage. Upon receiving a post request, all key=value pairs are mindlessly packaged into a hash table that is then passed on to a page handler that typically ignores the invalid keys.

However, most pages do not need any key,value pairs and forms that do have a pre-defined set of expected and recognized keys. If there were a possibly empty set of keys associated with each page, and the set were checked against posted keys, then a DOS post with thousands of effectively random keys could quickly (in O(1) time) be rejected as erroneous.

In Python, the same effect could be accomplished by associating a class with slots with each page and having the server create an instance of the class. Attempts to create an undefined attribute would then raise an exception. Either way, checking input data for face validity before processing it in a time-consuming way is one possible solution for nearly all web pages and at least some other applications.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:44:50 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Wed, 04 Jan 2012 17:44:50 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325699090.97.0.797030531574.issue13703@psf.upfronthosting.co.za>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

Except, it's a totally non-scalable approach.  People have vulnerabilities all over their sites which they don't realize.  Some examples:

django-taggit (an application I wrote for handling tags) parses tags out an input, it stores these in a set to check for duplicates.  It's vulnerable.

Another site I'm writing accepts JSON POSTs, you can put arbitrary keys in the JSON.  It's vulnerable.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:48:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 17:48:28 +0000
Subject: [issue13713] Regression for http.client read()
In-Reply-To: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>
Message-ID: <1325699308.27.0.778966338754.issue13713@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The fix is quite trivial. Here is a patch + tests.

----------
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24140/readinto_chunked.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:56:23 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Wed, 04 Jan 2012 17:56:23 +0000
Subject: [issue13713] Regression for http.client read()
In-Reply-To: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>
Message-ID: <1325699783.91.0.327805470433.issue13713@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

The patch looks right and seems to fix the issue. Thanks :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:58:11 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 04 Jan 2012 17:58:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F048156.30006@egenix.com>
Message-ID: <4F04932B.8080907@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Marc-Andre Lemburg wrote:
> 
> 1. The security implications in all this is being somewhat overemphasized.
> 
> There are many ways you can do a DoS attack on web servers. It's the
> responsibility of the used web frameworks and servers to deal with
> the possible cases.
> 
> It's a good idea to provide some way to protect against hash
> collision attacks, but that will only solve one possible way of
> causing a resource attack on a server.
> 
> There are other ways you can generate lots of CPU overhead with
> little data input (e.g. think of targeting the search feature on
> many Zope/Plone sites).
> 
> In order to protect against such attacks in general, we'd have to
> provide a way to control CPU time and e.g. raise an exception if too
> much time is being spent on a simple operation such as a key insertion.
> This can be done using timers, signals or even under OS control.
> 
> The easiest way to protect against the hash collision attack is by
> limiting the POST/GET/HEAD request size.

For GET and HEAD, web servers normally already apply such limitations
at rather low levels:

http://stackoverflow.com/questions/686217/maximum-on-http-header-values

So only HTTP methods which carry data in the body part of the HTTP
request are effected, e.g. POST and various WebDAV methods.

> The second best way would be to limit the number of parameters that a
> web framework accepts for POST/GET/HEAD request.

Depending on how parsers are implemented, applications taking
XML/JSON/XML-RPC/etc. as data input may also be vulnerable, e.g.
non validating XML parsers which place element attributes into
a dictionary or a JSON parser that has to read the JSON version of
the dict I generated earlier on.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:58:37 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 04 Jan 2012 17:58:37 +0000
Subject: [issue13464] HTTPResponse is missing an implementation of readinto
In-Reply-To: <1322072062.06.0.738377139897.issue13464@psf.upfronthosting.co.za>
Message-ID: <E1RiV6i-0006zx-E7@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4b21f651eeee by Antoine Pitrou in branch 'default':
Issue #13713: fix a regression in HTTP chunked reading after 806cfe39f729
http://hg.python.org/cpython/rev/4b21f651eeee

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13464>
_______________________________________

From report at bugs.python.org  Wed Jan  4 18:58:37 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 04 Jan 2012 17:58:37 +0000
Subject: [issue13713] Regression for http.client read()
In-Reply-To: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>
Message-ID: <E1RiV6i-0006zx-DG@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4b21f651eeee by Antoine Pitrou in branch 'default':
Issue #13713: fix a regression in HTTP chunked reading after 806cfe39f729
http://hg.python.org/cpython/rev/4b21f651eeee

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 19:00:56 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 04 Jan 2012 18:00:56 +0000
Subject: [issue13713] Regression for http.client read()
In-Reply-To: <1325695120.58.0.937190089641.issue13713@psf.upfronthosting.co.za>
Message-ID: <1325700056.48.0.123766312938.issue13713@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Ok, committed!
(Jon, don't worry, such things happen :-))

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13713>
_______________________________________

From report at bugs.python.org  Wed Jan  4 19:35:18 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 04 Jan 2012 18:35:18 +0000
Subject: [issue7098] g formatting for decimal types should always strip
	trailing zeros.
In-Reply-To: <1255198932.03.0.979291900289.issue7098@psf.upfronthosting.co.za>
Message-ID: <1325702118.62.0.598422254119.issue7098@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

[Mark]
> So I think the current code is correct.

I agree with this. Currently the 'g' format is like to_sci_string()
with the added possibility of adjusting the number of significant
digits. It's probably hard to come up with a better way to handle
this for Decimal.

----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7098>
_______________________________________

From report at bugs.python.org  Wed Jan  4 23:14:01 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 04 Jan 2012 22:14:01 +0000
Subject: [issue11648] openlog()s 'logopt' keyword broken in syslog module
In-Reply-To: <1300848156.85.0.89651150439.issue11648@psf.upfronthosting.co.za>
Message-ID: <1325715241.3.0.477112999696.issue11648@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

This has already been fixed with 71f7175e2b34 & friends.

----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions:  -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11648>
_______________________________________

From report at bugs.python.org  Wed Jan  4 23:26:42 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 04 Jan 2012 22:26:42 +0000
Subject: [issue10772] Several actions for argparse arguments missing from docs
In-Reply-To: <1293327184.11.0.454888760887.issue10772@psf.upfronthosting.co.za>
Message-ID: <E1RiZI8-0003Dr-GU@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 278fbd7b9608 by Sandro Tosi in branch '2.7':
Issue #10772: add count and help argparse action; patch by Marc Sibson
http://hg.python.org/cpython/rev/278fbd7b9608

New changeset 326f755962e3 by Sandro Tosi in branch '3.2':
Issue #10772: add count and help argparse action; patch by Marc Sibson
http://hg.python.org/cpython/rev/326f755962e3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10772>
_______________________________________

From report at bugs.python.org  Wed Jan  4 23:27:48 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 04 Jan 2012 22:27:48 +0000
Subject: [issue10772] Several actions for argparse arguments missing from docs
In-Reply-To: <1293327184.11.0.454888760887.issue10772@psf.upfronthosting.co.za>
Message-ID: <1325716068.1.0.141298160136.issue10772@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Thanks Marc for the patch, I've just committed it.

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10772>
_______________________________________

From report at bugs.python.org  Thu Jan  5 00:21:56 2012
From: report at bugs.python.org (Berker Peksag)
Date: Wed, 04 Jan 2012 23:21:56 +0000
Subject: [issue13641] decoding functions in the base64 module could accept
	unicode strings
In-Reply-To: <1324386392.48.0.968435833624.issue13641@psf.upfronthosting.co.za>
Message-ID: <1325719316.31.0.122459781773.issue13641@psf.upfronthosting.co.za>


Berker Peksag <berker.peksag at gmail.com> added the comment:

Hi Antoine,

I added some tests for b64decode function.

Also, I wrote some tests for b32decode and b16decode functions and failed. I think my patch is not working for b32decode and b16decode functions. I'll dig into code and try to find a way.

Thanks!

----------
Added file: http://bugs.python.org/file24141/issue13641_v2_with_tests.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13641>
_______________________________________

From report at bugs.python.org  Thu Jan  5 00:42:52 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 23:42:52 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325720572.43.0.859996838691.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Work-in-progress patch implementing my randomized hash function (random.patch):
 - add PyOS_URandom() using CryptoGen, SSL (only on VMS!!) or /dev/urandom, will a fallback on a dummy LCG if the OS urandom failed
 - posix.urandom() is always defined and reuses PyOS_URandom()
 - hash(str) is now randomized using two random Py_hash_t values: don't touch the critical loop, only add a prefix and a suffix

Notes:
 - PyOS_URandom() reuses mostly code from Modules/posixmodule.c, except dev_urandom() and fallback_urandom() which are new
 - I removed memset(PyBytes_AS_STRING(result), 0, howMany); from win32_urandom() because it doesn't really change anything because the LCG is used if win32_urandom() fails
 - Python refuses to start if the OS urandom is missing.
 - Python/random.c code may be moved into Python/pythonrun.c if it is an issue to add a new file in old Python versions.
 - If the OS urandom fails to generate the unicode hash secret, no warning is emitted (because the LCG is used). I don't know if a warning is needed in this case.
 - os.urandom() argument is now a Py_ssize_t instead of an int

TODO:
 - add an environment option to ignore the OS urandom and only uses the LCG
 - fix all tests broken because of the randomized hash(str)
 - PyOS_URandom() raises exceptions whereas it is called before creating the interpreter state. I suppose that it cannot work like this.
 - review and test PyOS_URandom()
 - review and test the new randomized hash(str)

----------
keywords: +patch
Added file: http://bugs.python.org/file24142/random.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 00:54:25 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 04 Jan 2012 23:54:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325721265.95.0.809609196313.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> add PyOS_URandom() using CryptoGen, SSL (only on VMS!!)
> or /dev/urandom

Oh, OpenSSL (RAND_pseudo_bytes) should be used on Windows, Linux, Mac OS X, etc. if OpenSSL is available. I was just too lazy to add a define or pyconfig.h option to indicate if OpenSSL is available or not. FYI RAND_pseudo_bytes() is now exposed in the ssl module of Python 3.3.

> will a fallback on a dummy LCG

It's the Linear congruent generator (LCG) used by Microsoft Visual C++ and PHP:

x(n+1) = (x(n) * 214013 + 2531011) % 2^32

I only use bits 23..16 (bits 15..0 are not really random).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:01:02 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 00:01:02 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325721265.95.0.809609196313.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325721587.3458.13.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > add PyOS_URandom() using CryptoGen, SSL (only on VMS!!)
> > or /dev/urandom
> 
> Oh, OpenSSL (RAND_pseudo_bytes) should be used on Windows, Linux, Mac
> OS X, etc. if OpenSSL is available.

Apart from the large dependency, the OpenSSL license is not
GPL-compatible which may be a problem for some Python-embedding
applications:
http://en.wikipedia.org/wiki/OpenSSL#Licensing

> > will a fallback on a dummy LCG
> 
> It's the Linear congruent generator (LCG) used by Microsoft Visual C++
> and PHP:
> 
> x(n+1) = (x(n) * 214013 + 2531011) % 2^32
> 
> I only use bits 23..16 (bits 15..0 are not really random).

If PHP uses it, I'm confident it is secure.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:02:48 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:02:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325721768.46.0.595826990627.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

+            printf("read %i bytes\n", size);

Oops, I forgot a debug message.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:11:03 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:11:03 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325722263.31.0.319189986882.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> If PHP uses it, I'm confident it is secure.

If I remember correctly, it is only used for the Windows version of PHP, but PHP doesn't implement it correctly because it uses all bits.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:31:43 2012
From: report at bugs.python.org (Paul McMillan)
Date: Thu, 05 Jan 2012 00:31:43 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325723503.71.0.150293983738.issue13703@psf.upfronthosting.co.za>


Paul McMillan <paul at mcmillan.ws> added the comment:

This is not something that can be fixed by limiting the size of POST/GET. 

Parsing documents (even offline) can generate these problems. I can create books that calibre (a Python-based ebook format shifting tool) can't convert, but are otherwise perfectly valid for non-python devices. If I'm allowed to insert usernames into a database and you ever retrieve those in a dict, you're vulnerable. If I can post things one at a time that eventually get parsed into a dict (like the tag example), you're vulnerable. I can generate web traffic that creates log files that are unparsable (even offline) in Python if dicts are used anywhere. Any application that accepts data from users needs to be considered.

Even if the web framework has a dictionary implementation that randomizes the hashes so it's not vulnerable, the entire python standard library uses dicts all over the place. If this is a problem which must be fixed by the framework, they must reinvent every standard library function they hope to use.

Any non-trivial python application which parses data needs the fix. The entire standard library needs the fix if is to be relied upon by applications which accept data. It makes sense to fix Python.

Of course we must fix all the basic hashing functions in python, not just the string hash. There aren't that many. 

Marc-Andre:
If you look at my proposed code, you'll notice that we do more than simply shift the period of the hash. It's not trivial for an attacker to create colliding hash functions without knowing the key.

Since speed is a concern, I think that the proposal to avoid using the random hash for short strings is a good idea. Additionally, randomizing only some of the characters in longer strings will allow us to improve security without compromising speed significantly.

I suggest that we don't randomize strings shorter than 6 characters. For longer strings, we randomize the first and last 5 characters. This means we're only adding additional work to a max of 10 rounds of the hash, and only for longer strings. Collisions with the hash from short strings should be minimal.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:32:56 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:32:56 +0000
Subject: [issue13699] test_gdb has recently started failing
In-Reply-To: <1325546341.52.0.130827872784.issue13699@psf.upfronthosting.co.za>
Message-ID: <1325723576.83.0.633637900239.issue13699@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> New changeset dfffb293f4b3 by Vinay Sajip in branch 'default'

The fix should also be applied to 3.2.

----------
resolution: fixed -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13699>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:36:11 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:36:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325723771.39.0.672443575759.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

"Since speed is a concern, I think that the proposal to avoid using the random hash for short strings is a good idea."

My proposition only adds two XOR to hash(str) (outside the loop on Unicode characters), so I expect a ridiculous overhead. I don't know yet how hard it is to guess the secret from hash(str) output.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:36:52 2012
From: report at bugs.python.org (Christian Heimes)
Date: Thu, 05 Jan 2012 00:36:52 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325723812.53.0.978598000253.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Thanks Victor!

> - hash(str) is now randomized using two random Py_hash_t values: 
> don't touch the critical loop, only add a prefix and a suffix

At least for Python 2.x hash(str) and hash(unicode) have to yield the same result for ASCII only strings. 

>  - PyOS_URandom() raises exceptions whereas it is called before
> creating the interpreter state. I suppose that it cannot work like this.

My patch compensates for the issue and calls Py_FatalError() when the random seed hasn't been initialized yet.

You aren't special casing small strings. I fear that an attacker may guess the seed from several small strings. How about using another initial seed for strings shorter than 4 code points?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:39:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 00:39:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325723812.53.0.978598000253.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325723899.3458.14.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> You aren't special casing small strings. I fear that an attacker may
> guess the seed from several small strings.

How would (s)he do?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:44:25 2012
From: report at bugs.python.org (Paul McMillan)
Date: Thu, 05 Jan 2012 00:44:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325723771.39.0.672443575759.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRUV+P54dLaJ9xNS07nh1bZd8jmiBpFMT8Qh7W59Yq38aw@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> My proposition only adds two XOR to hash(str) (outside the loop on Unicode characters), so I expect a ridiculous overhead. I don't know yet how hard it is to guess the secret from hash(str) output.

It doesn't work much better than a single random seed. Calculating the
hash of a null byte gives you the xor of your two seeds. An attacker
can still cause collisions inside the vulnerable hash function, your
change doesn't negate those internal collisions. Also, strings of all
null bytes collide trivially.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:49:04 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:49:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325724544.3.0.711149644235.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I fear that an attacker may guess the seed from several small strings

hash(a) ^ hash(b) "removes" the suffix, but I don't see how to guess the prefix from this new value. It doesn't mean that it is not possible, just that I don't have a strong background in crytography :-)

I don't expect that adding 2 XOR would change our dummy (fast but unsafe) hash function into a cryptographic hash function. We cannot have security for free. If we want a strong cryptographic hash function, it would be much slower (Paul wrote that it would be 4x slower). But we prefer speed over security, so we have to do compromise.

I don't know if you can retreive hash values in practice. I suppose that you can only get hash(str) & (size - 1) with size=size of the dict internal array, so only the lower bits. Using a large dict, you may be able to retreive more bits of the hash value.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:53:57 2012
From: report at bugs.python.org (Christian Heimes)
Date: Thu, 05 Jan 2012 00:53:57 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325724837.8.0.410157062214.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Given that a user has an application with an oracle function that returns the hash of a unicode string, an attacker can probe tenth of thousand one and two character unicode strings. That should give him/her enough data to calculate both seeds. hash("") already gives away lots of infomration about the seeds, too.

- hash("") should always return 0

- for small strings we could use a different seed than for larger strings

- for larger strings we could use Paul's algorithm but limit the XOR op to the first and last 16 elements instead of all elements.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:57:04 2012
From: report at bugs.python.org (Paul McMillan)
Date: Thu, 05 Jan 2012 00:57:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325724837.8.0.410157062214.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRXwYVLGsx8A=EpVoKETD0qmD8EL=xfqBYbc_Pos0ibjyw@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> - for small strings we could use a different seed than for larger strings

Or just leave them unseeded with our existing algorithm. Shifting them
into a different part of the hash space doesn't really gain us much.

> - for larger strings we could use Paul's algorithm but limit the XOR op to the first and last 16 elements instead of all elements.

Agreed. It does have to be both the first and the last though. We
can't just do one or the other.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:58:39 2012
From: report at bugs.python.org (Christian Heimes)
Date: Thu, 05 Jan 2012 00:58:39 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325725119.42.0.196857111498.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Paul wrote:
> I suggest that we don't randomize strings shorter than 6 characters. For longer strings, we randomize the first and last 5 characters. This means we're only adding additional work to a max of 10 rounds of the hash, and only for longer strings. Collisions with the hash from short strings should be minimal.

It's too surprising for developers when just the strings with 6 or more chars are randomized. Barry made a good point http://bugs.python.org/issue13703#msg150613

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 01:58:44 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 00:58:44 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325725124.08.0.369442188913.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

"Calculating the hash of a null byte gives you the xor of your two seeds."

Not directly because prefix is first multiplied by 1000003. So hash("\0") gives you (prefix*1000003) % 2^32 ^ suffix.

Example:

$ ./python 
secret={b7abfbbf, db6cbb4d}
Python 3.3.0a0 (default:547e918d7bf5+, Jan  5 2012, 01:36:39) 
>>> hash("")
1824997618
>>> hash("\0")
-227042383
>>> hash("\0"*2)
1946249080
>>> 0xb7abfbbf ^ 0xdb6cbb4d
1824997618
>>> (0xb7abfbbf * 1000003) & 0xffffffff ^ 0xdb6cbb4d
4067924912
>>> hash("\0") & 0xffffffff
4067924913

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 02:05:59 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 01:05:59 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325725559.56.0.557718997796.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> At least for Python 2.x hash(str) and hash(unicode) have to yield
> the same result for ASCII only strings. 

Ah yes, I forgot Python 2: I wrote my patch for Python 3.3. The two hash functions should be modified to be randomized.

> hash("") should always return 0

Ok, I can add a special case. Antoine told me that hash("") gives prefix ^ suffix, which is too much information for the attacker :-)

> for small strings we could use a different seed
> than for larger strings

Why? The attack doesn't work with short strings? What do you call a "short string"?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 02:09:06 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 01:09:06 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325725746.35.0.274718290076.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 2:
 - hash("") is always 0
 - Remove a debug message

----------
Added file: http://bugs.python.org/file24143/random-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 02:17:04 2012
From: report at bugs.python.org (Christian Heimes)
Date: Thu, 05 Jan 2012 01:17:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325726224.1.0.0660844510127.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

In reply to MAL's message http://bugs.python.org/issue13703#msg150616

> 2. Changing the semantics of hashing in a dot release is not allowed.

I concur with Marc. The change is too intrusive and may cause too much trouble for the issue. Also it seems to be unnecessary for platforms with 64bit hash.

Marc: Fred told me that ZODB isn't affected. One thing less to worry. ;)


> 5. Hashing needs to be fast.

Good point, we should include Tim and Christian Tiesmer once we have a solution we can agree upon

PS: I'm missing "Reply to message" and a threaded view for lengthy topics

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 04:45:59 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 05 Jan 2012 03:45:59 +0000
Subject: [issue13712] pysetup create should not convert package_data to
	extra_files
In-Reply-To: <1325689084.75.0.043764224634.issue13712@psf.upfronthosting.co.za>
Message-ID: <1325735159.7.0.77150134567.issue13712@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

The bug is caused by code in packaging.create that iterates over a dict (package_data) to extend a list (extra_files).  Instead of just calling sorted to make output deterministic, I?d prefer to fix that more serious behavior bug (see also #13463, #11805 and #5302 for more !fun package_data bugs).  Problem is that the setup.cfg syntax does not define how to give more than one value.  If it?s judged acceptable to disallow paths with embedded spaces, we could do something like this:

[files]
package_data =
    spam = first second third

Otherwise we?d need to use multiple lines (requested in #5302):

[files]
package_data =
    spam = first
    spam = second
    spam = third

We probably don?t want that.  An intermediate idea:

[files]
package_data =
    spam = first
           second
           third

Not sure this would be the nicest thing for people to write, and for us (me) to extend the setup.cfg parser for.

Anyway, attached patch fixes the code so that package_data in setup.py becomes package_data in setup.cfg and adapts the tests to check that, disabling multi-value package_data for now.  I tested it with distutils2 and pypy, so it should fix the hash change in your clone.

----------
keywords: +patch
nosy: +erik.bray
title: test_packaging depends on hash order -> pysetup create should not convert package_data to extra_files
Added file: http://bugs.python.org/file24144/fix-pysetup-create-package_data.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13712>
_______________________________________

From report at bugs.python.org  Thu Jan  5 05:36:10 2012
From: report at bugs.python.org (Sworddragon)
Date: Thu, 05 Jan 2012 04:36:10 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
Message-ID: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>


New submission from Sworddragon <sworddragon2 at aol.com>:

If a client gets a reconnect and a new ip from the provider the methods of ftplib can't handle this and are hanging in an infinite loop. For example if a file is transfered with storbinary() and the client gets a new ip address the script will never end. I'm using the Linux Kernel 3.2 on a 64 bit system and Python 2.7 is affected too.

----------
components: Library (Lib)
messages: 150654
nosy: Sworddragon
priority: normal
severity: normal
status: open
title: Methods of ftplib never ends if the ip address changes
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 07:25:13 2012
From: report at bugs.python.org (Huzaifa Sidhpurwala)
Date: Thu, 05 Jan 2012 06:25:13 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325744713.17.0.31507819307.issue13703@psf.upfronthosting.co.za>


Huzaifa Sidhpurwala <sidhpurwala.huzaifa at gmail.com> added the comment:

I am wondering if a CVE id has been assigned to this security issue yet?

----------
nosy: +Huzaifa.Sidhpurwala

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 10:01:15 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 05 Jan 2012 09:01:15 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325723503.71.0.150293983738.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0566D1.1010100@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Paul McMillan wrote:
> 
> This is not something that can be fixed by limiting the size of POST/GET. 
> 
> Parsing documents (even offline) can generate these problems. I can create books that calibre (a Python-based ebook format shifting tool) can't convert, but are otherwise perfectly valid for non-python devices. If I'm allowed to insert usernames into a database and you ever retrieve those in a dict, you're vulnerable. If I can post things one at a time that eventually get parsed into a dict (like the tag example), you're vulnerable. I can generate web traffic that creates log files that are unparsable (even offline) in Python if dicts are used anywhere. Any application that accepts data from users needs to be considered.
> 
> Even if the web framework has a dictionary implementation that randomizes the hashes so it's not vulnerable, the entire python standard library uses dicts all over the place. If this is a problem which must be fixed by the framework, they must reinvent every standard library function they hope to use.
> 
> Any non-trivial python application which parses data needs the fix. The entire standard library needs the fix if is to be relied upon by applications which accept data. It makes sense to fix Python.

Agreed: Limiting the size of POST requests only applies to *web* applications.
Other applications will need other fixes.

Trying to fix the problem in general by tweaking the hash function to
(apparently) make it hard for an attacker to guess a good set of
colliding strings/integers/etc. is not really a good solution. You'd
only be making it harder for script kiddies, but as soon as someone
crypt-analysis the used hash algorithm, you're lost again.

You'd need to use crypto hash functions or universal hash functions
if you want to achieve good security, but that's not an option for
Python objects, since the hash functions need to be as fast as possible
(which rules out crypto hash functions) and cannot easily drop the invariant
"a=b => hash(a)=hash(b)" (which rules out universal hash functions, AFAICT).

IMO, the strategy to simply cap the number of allowed collisions is
a better way to achieve protection against this particular resource
attack. The probability of having valid data reach such a limit is
low and, if configurable, can be made 0.

> Of course we must fix all the basic hashing functions in python, not just the string hash. There aren't that many. 

... not in Python itself, but if you consider all the types in Python
extensions and classes implementing __hash__ in user code, the number
of hash functions to fix quickly becomes unmanageable.

> Marc-Andre:
> If you look at my proposed code, you'll notice that we do more than simply shift the period of the hash. It's not trivial for an attacker to create colliding hash functions without knowing the key.

Could you post it on the ticket ?

BTW: I wonder how long it's going to take before someone figures out
that our merge sort based list.sort() is vulnerable as well... its
worst-case performance is O(n log n), making attacks somewhat harder.
The popular quicksort which Python used for a long time has O(n?),
making it much easier to attack, but fortunately, we replaced it
with merge sort in Python 2.3, before anyone noticed ;-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 10:18:05 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Thu, 05 Jan 2012 09:18:05 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325755085.96.0.520351917105.issue13714@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

It seems expected behavior to me, and the same issue should apply to all other network libs as well. What would you expect ftplib to do in such case?

----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 10:39:31 2012
From: report at bugs.python.org (Sworddragon)
Date: Thu, 05 Jan 2012 09:39:31 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325756371.55.0.743156070033.issue13714@psf.upfronthosting.co.za>


Sworddragon <sworddragon2 at aol.com> added the comment:

If the connection gets lost and reconnected again but the ip address doesn't change storbinary() continues the data transfer. But if the ip address was changed due to the reconnect storbinary() hangs in a loop.

I expect either that storbinary() detects the change of the ip address and continues the data transfer like it does if the ip address has never changed or it should throw an exception.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 10:43:33 2012
From: report at bugs.python.org (Mark Shannon)
Date: Thu, 05 Jan 2012 09:43:33 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325724837.8.0.410157062214.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0570BF.2050701@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

What is the mechanism by which the attacker can determine the seeds?
The actual hash value is not directly observable externally.
The attacker can only determine the timing effects of multiple 
insertions into a dict, or have I missed something?

> - hash("") should always return 0

Why should hash("") always return 0?
I can't find it in the docs anywhere.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:00:31 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Thu, 05 Jan 2012 10:00:31 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325757631.48.0.60273994216.issue13714@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

What storbinary does is just using a socket to send data.
There's no way for storbinary to ask the socket whether an unpredicted event such as an IP change occurred and neither it should.

As a user, you just shouldn't change the IP address while a network app is running on that network interface and expect it to keep working or raise an exception.  
The consequences are unpredictable and are probably subject to change depending on what platform you're on.

In summary, this is not a problem which should be dealt with by base ftplib or any other network lib in the stdlib.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:09:40 2012
From: report at bugs.python.org (Sworddragon)
Date: Thu, 05 Jan 2012 10:09:40 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325758180.96.0.743778527051.issue13714@psf.upfronthosting.co.za>


Sworddragon <sworddragon2 at aol.com> added the comment:

The problem is that it is for example here in germany very common that the provider disconnects the client every 24 hours and gives him a new ip address if his router reconnects. This makes it very difficult to send big files with ftplib.

For example for daily backups I have written an automatic backup script which uses ftplib. The transfer needs some hours and very often it fails (it silently never ends) because I got a new ip address.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:20:27 2012
From: report at bugs.python.org (Christian Heimes)
Date: Thu, 05 Jan 2012 10:20:27 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325758827.39.0.163970385347.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

It's quite possible that a user has created a function (by mistake or deliberately) that gives away the hash of an arbitrary string. We haven't taught developers that (s)he shouldn't disclose the hash of a string.

> Why should hash("") always return 0?
> I can't find it in the docs anywhere.

hash("") should return something constant that doesn't reveal information about the random seeds. 0 is an arbitrary choice that is as good as anything else. hash("") already returns 0, hence my suggestion for 0.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:28:14 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Thu, 05 Jan 2012 10:28:14 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325759294.48.0.829856889395.issue13714@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

Since you say the connection hangs I think you can set a timeout:

>>> ftp = ftplib.FTP(..., timeout=30)

That is applied to both control and data connection (and hence storbinary). This way you should get a socket.timeout exception after 30 seconds.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:41:00 2012
From: report at bugs.python.org (Sworddragon)
Date: Thu, 05 Jan 2012 10:41:00 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325760060.65.0.963053102683.issue13714@psf.upfronthosting.co.za>


Sworddragon <sworddragon2 at aol.com> added the comment:

If i set the timeout argument an exception s thrown if the ip address is changed. At least it's a workaround but we should think about if Python shouldn't try to detect changes of the ip address.

It would be nicer to continue the file transfer like it does if the connection gets lost without a change of the ip address instead of sending the complete data again.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:41:41 2012
From: report at bugs.python.org (Mark Shannon)
Date: Thu, 05 Jan 2012 10:41:41 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325758827.39.0.163970385347.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F057E63.4020102@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

But that's not the issue we are supposed to be dealing with.
A single (genuinely random) seed will deal with the attack described in 
the talk and it is (almost) as fast as using 0 as a seed.
Why make things complicated dealing with a hypothetical problem?

>> Why should hash("") always return 0?
>> I can't find it in the docs anywhere.
> 
> hash("") should return something constant that doesn't reveal information about the random seeds. 0 is an arbitrary choice that is as good as anything else. hash("") already returns 0, hence my suggestion for 0.

Is special casing arbitrary values really any more secure?
If we special case "", the attacker will just start using "\0" and so on...

> 
> ----------
> 
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 11:50:37 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Thu, 05 Jan 2012 10:50:37 +0000
Subject: [issue13714] Methods of ftplib never ends if the ip address changes
In-Reply-To: <1325738170.02.0.180646245412.issue13714@psf.upfronthosting.co.za>
Message-ID: <1325760637.5.0.106744884629.issue13714@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

Python can't do that. It's a socket implementation detail. Python just exposes the underlying socket implementation as-is.
I'm closing this out as rejected.

----------
assignee:  -> giampaolo.rodola
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13714>
_______________________________________

From report at bugs.python.org  Thu Jan  5 12:46:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 05 Jan 2012 11:46:47 +0000
Subject: [issue13699] test_gdb has recently started failing
In-Reply-To: <1325546341.52.0.130827872784.issue13699@psf.upfronthosting.co.za>
Message-ID: <E1RilmP-0004Jp-O6@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a3d4cde1c357 by Vinay Sajip in branch '3.2':
Closes #13699. Skipped two tests if Python is optimised.
http://hg.python.org/cpython/rev/a3d4cde1c357

New changeset 7d87ebbbd718 by Vinay Sajip in branch 'default':
Closes #13699: merged fix from 3.2.
http://hg.python.org/cpython/rev/7d87ebbbd718

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13699>
_______________________________________

From report at bugs.python.org  Thu Jan  5 13:41:26 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 12:41:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325726224.1.0.0660844510127.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325767213.8907.10.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I concur with Marc. The change is too intrusive and may cause too much
> trouble for the issue.

Do you know if mod_wsgi et al. are tackling the issue on their side?

> Also it seems to be unnecessary for platforms with 64bit hash.

We still support Python on 32-bit platforms, so this can't be a serious
argument.
If you think that no-one runs a server on a 32-bit kernel nowadays, I
would point out that "no-one" apparently doesn't include ourselves ;-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 16:44:38 2012
From: report at bugs.python.org (Erik Bray)
Date: Thu, 05 Jan 2012 15:44:38 +0000
Subject: [issue13712] pysetup create should not convert package_data to
	extra_files
In-Reply-To: <1325689084.75.0.043764224634.issue13712@psf.upfronthosting.co.za>
Message-ID: <1325778278.99.0.0169650152109.issue13712@psf.upfronthosting.co.za>


Erik Bray <erik.m.bray at gmail.com> added the comment:

FWIW, I'm for the first option for specifying package_data:

[files]
package_data =
    spam = first second third

I'm pretty sure this is how I ended up implementing it in d2to1, since I needed this functionality.

Theoretically spaces could be supported with an escape sequence, but I don't think that's worth complicating things for if package_data is deprecated anyways.  I'm all for making it difficult for anyone trying to include filenames with spaces in their source code.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13712>
_______________________________________

From report at bugs.python.org  Thu Jan  5 16:47:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 15:47:39 +0000
Subject: [issue13128] httplib debuglevel on CONNECT doesn't print response
	headers
In-Reply-To: <1318050704.97.0.551622649018.issue13128@psf.upfronthosting.co.za>
Message-ID: <1325778459.35.0.849616372815.issue13128@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +orsenthil
versions: +Python 3.3 -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13128>
_______________________________________

From report at bugs.python.org  Thu Jan  5 17:47:21 2012
From: report at bugs.python.org (Eli Collins)
Date: Thu, 05 Jan 2012 16:47:21 +0000
Subject: [issue13715] typo in unicodedata documentation
Message-ID: <1325782041.56.0.783502480563.issue13715@psf.upfronthosting.co.za>


New submission from Eli Collins <elic at assurancetechnologies.com>:

I noticed a minor typo in the unicodedata.normalize() documentation...

The line reading 'U+0327 (COMBINING CEDILLA) U+0043 (LATIN CAPITAL LETTER C)' is not proper unicode, it should be in the reverse order: 'U+0043 (LATIN CAPITAL LETTER C) U+0327 (COMBINING CEDILLA)'.

attached is a small patch to fix this.

----------
assignee: docs at python
components: Documentation
files: docfix.patch
keywords: patch
messages: 150670
nosy: docs at python, eli.collins
priority: normal
severity: normal
status: open
title: typo in unicodedata documentation
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file24145/docfix.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13715>
_______________________________________

From report at bugs.python.org  Thu Jan  5 17:47:31 2012
From: report at bugs.python.org (=?utf-8?q?Lars_Gust=C3=A4bel?=)
Date: Thu, 05 Jan 2012 16:47:31 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325782051.86.0.0510246606263.issue13702@psf.upfronthosting.co.za>


Lars Gust?bel <lars at gustaebel.de> added the comment:

You actually hit two bugs at the same time here: The target of the created symlink was not translated from unix to windows path delimiters and is therefore broken. The second bug is issue12926 which leads to the error in TarFile.makefile(). 

Brian, AFAIK all file-specific functions on windows accept forward slashes in pathnames, right? Has this been discussed in the course of the windows implementation of os.symlink()? I could certainly fix the slash translation in tarfile.py, but may be it's os.symlink() that should been fixed.

----------
dependencies: +tarfile tarinfo.extract*() broken with symlinks

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:21:01 2012
From: report at bugs.python.org (Patrick von Reth)
Date: Thu, 05 Jan 2012 17:21:01 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325784061.1.0.521553235482.issue13702@psf.upfronthosting.co.za>


Patrick von Reth <patrick.vonreth at gmail.com> added the comment:

to ignore the bug I also tried dereference=True, but it looks like python3 is ignoring it for extraction.
Is this the normal behavior or just another bug?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:24:13 2012
From: report at bugs.python.org (=?utf-8?q?Lars_Gust=C3=A4bel?=)
Date: Thu, 05 Jan 2012 17:24:13 +0000
Subject: [issue13702] relative symlinks in tarfile.extract broken (windows)
In-Reply-To: <1325608963.69.0.695216553795.issue13702@psf.upfronthosting.co.za>
Message-ID: <1325784253.51.0.704965780213.issue13702@psf.upfronthosting.co.za>


Lars Gust?bel <lars at gustaebel.de> added the comment:

The dereference option is only used for archive creation, so the contents of the file a symbolic link is pointing to is added instead of the symbolic link itself.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13702>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:24:18 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 05 Jan 2012 17:24:18 +0000
Subject: [issue13715] typo in unicodedata documentation
In-Reply-To: <1325782041.56.0.783502480563.issue13715@psf.upfronthosting.co.za>
Message-ID: <1325784258.68.0.113390699909.issue13715@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
assignee: docs at python -> ezio.melotti
nosy: +ezio.melotti
stage:  -> patch review
type:  -> enhancement
versions:  -Python 2.6, Python 3.1, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13715>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:47:47 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 05 Jan 2012 17:47:47 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1325291864.93.0.942110099636.issue12760@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM0t_wsGTRPYr6fDA77ieBgwZKk_DrFmzcxmqKnO-4sM6Q@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I've done a small review.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:49:54 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 05 Jan 2012 17:49:54 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1325621821.94.0.514452167191.issue8184@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3fMzG_sEVLW4eQ+ebR9H-yywwLstWx6yttO4_G90d=0g@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> If the above gets solved on windows my problem will just go away, thanks

Would you like to propose a patch with test?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:57:00 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 05 Jan 2012 17:57:00 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <1325674262.3458.3.camel@localhost.localdomain>
Message-ID: <CAH_1eM3sqsstA8qk+VuzEnbeP=2QJoFe_w4LcwiZc72U4HYQ_Q@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Hmm, but that would break single-threaded programs which expect their
> select() (or other) to return EINTR when a signal is received (which is
> a perfectly valid expectation in that case).

Yes, that's why I said "that"s another story" ;-)
EINTR is really a pain, and relying on it to return from a syscall
upon signal reception is a bad idea (certain OS restart syscalls by
default - not select() though - and if the signal is received before
you call the syscall, you'll deadlock). This would IMHO be the best
way to go, but I know we can't reasonably change this now.

> I don't know if that's still useful to build Python without threads. I
> would expect most platforms to have a compatible threads implementation
> (and Python probably can't run on very small embedded platforms).
> Perhaps you can ask on python-dev.

There are another problems, for example it's very well known that
signals and threads don't mix well (so for example you'd have to block
all signals before spawning the new thread, and reenable them
afterwards).
I'm not sure it's worth the extra complication. I can still try to
write a quick patch to see if it gets somewhere (and doesn't break
test_threading and test_signals).

The first and most simple thing we could do would be to nuke the
Python version (and also the loggin hack at the same time): does that
sound reasonable ?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Thu Jan  5 18:59:00 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 05 Jan 2012 17:59:00 +0000
Subject: [issue12926] tarfile tarinfo.extract*() broken with symlinks
In-Reply-To: <1315389851.08.0.471880114138.issue12926@psf.upfronthosting.co.za>
Message-ID: <E1Rirac-0005uu-KO@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 573fc99873bd by Lars Gust?bel in branch '3.2':
Issue #12926: Fix a bug in tarfile's link extraction.
http://hg.python.org/cpython/rev/573fc99873bd

New changeset 5936c2005ab7 by Lars Gust?bel in branch 'default':
Merge from 3.2: Issue #12926: Fix a bug in tarfile's link extraction.
http://hg.python.org/cpython/rev/5936c2005ab7

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12926>
_______________________________________

From report at bugs.python.org  Thu Jan  5 19:07:31 2012
From: report at bugs.python.org (=?utf-8?q?Lars_Gust=C3=A4bel?=)
Date: Thu, 05 Jan 2012 18:07:31 +0000
Subject: [issue12926] tarfile tarinfo.extract*() broken with symlinks
In-Reply-To: <1315389851.08.0.471880114138.issue12926@psf.upfronthosting.co.za>
Message-ID: <1325786851.55.0.500824384575.issue12926@psf.upfronthosting.co.za>


Lars Gust?bel <lars at gustaebel.de> added the comment:

This should be fixed now, thanks.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12926>
_______________________________________

From report at bugs.python.org  Thu Jan  5 19:08:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 05 Jan 2012 18:08:39 +0000
Subject: [issue11984] Wrong "See also" in symbol and token module docs
In-Reply-To: <1304405152.56.0.259676009595.issue11984@psf.upfronthosting.co.za>
Message-ID: <E1Rirjx-00072m-AZ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset d8102ccc5bf7 by Sandro Tosi in branch '2.7':
Issue #11984: remove reference to parser, it's not showing symbol usage anymore
http://hg.python.org/cpython/rev/d8102ccc5bf7

New changeset b326d90ce9c9 by Sandro Tosi in branch '3.2':
Issue #11984: remove reference to parser, it's not showing symbol usage anymore
http://hg.python.org/cpython/rev/b326d90ce9c9

New changeset f375a1be031c by Sandro Tosi in branch 'default':
Issue #11984: merge with 3.2
http://hg.python.org/cpython/rev/f375a1be031c

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11984>
_______________________________________

From report at bugs.python.org  Thu Jan  5 19:09:41 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 05 Jan 2012 18:09:41 +0000
Subject: [issue11984] Wrong "See also" in symbol and token module docs
In-Reply-To: <1304405152.56.0.259676009595.issue11984@psf.upfronthosting.co.za>
Message-ID: <1325786981.8.0.791494254587.issue11984@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi Davi, thanks for your report! I've removed the reference from symbol to parser, given this last module doesn't show anymore how to user symbol, and it's encouraging to use ast module instead.

----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11984>
_______________________________________

From report at bugs.python.org  Thu Jan  5 19:51:05 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 05 Jan 2012 18:51:05 +0000
Subject: [issue12042] What's New multiprocessing example error
In-Reply-To: <1304967317.91.0.912225759197.issue12042@psf.upfronthosting.co.za>
Message-ID: <E1RisP1-00031B-6q@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3353f9747a39 by Sandro Tosi in branch '2.7':
Issue #12042: a queue is only used to retrive results; preliminary patch by Jordan Stadler
http://hg.python.org/cpython/rev/3353f9747a39

New changeset 0d4bb1356f39 by Sandro Tosi in branch '3.2':
Issue #12042: a queue is only used to retrive results; preliminary patch by Jordan Stadler
http://hg.python.org/cpython/rev/0d4bb1356f39

New changeset e379617b4c4c by Sandro Tosi in branch 'default':
Issue #12042: merge with 3.2
http://hg.python.org/cpython/rev/e379617b4c4c

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12042>
_______________________________________

From report at bugs.python.org  Thu Jan  5 19:55:12 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 05 Jan 2012 18:55:12 +0000
Subject: [issue12042] What's New multiprocessing example error
In-Reply-To: <1304967317.91.0.912225759197.issue12042@psf.upfronthosting.co.za>
Message-ID: <1325789712.46.0.442673508827.issue12042@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Thanks Davi for the report and Jordan for the patch! Jordan, a tip for your (hopefully) future contributions: for doc patches, please don't re-indent the whole paragraph, since it makes harder to identify the actual changes, just change what is needed, and the committer will take care (if needed) to re indent the paragraph.

I've re-edited a bit and committed the fix in the active branches.

----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12042>
_______________________________________

From report at bugs.python.org  Thu Jan  5 20:02:16 2012
From: report at bugs.python.org (Florent Xicluna)
Date: Thu, 05 Jan 2012 19:02:16 +0000
Subject: [issue13716] distutils doc contains lots of XXX
Message-ID: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>


New submission from Florent Xicluna <florent.xicluna at gmail.com>:

http://docs.python.org/distutils/apiref.html?highlight=XXX#module-distutils.ccompiler

We find lots of "XXX" and "XXX see also." which give no information.

----------
assignee: docs at python
components: Documentation
messages: 150683
nosy: docs at python, flox
priority: normal
severity: normal
stage: needs patch
status: open
title: distutils doc contains lots of XXX
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Thu Jan  5 20:04:00 2012
From: report at bugs.python.org (Florent Xicluna)
Date: Thu, 05 Jan 2012 19:04:00 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325790240.43.0.42478832186.issue13716@psf.upfronthosting.co.za>


Changes by Florent Xicluna <florent.xicluna at gmail.com>:


----------
nosy: +eric.araujo, tarek
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:12:55 2012
From: report at bugs.python.org (Atle Pedersen)
Date: Thu, 05 Jan 2012 20:12:55 +0000
Subject: [issue13717] print fails on unicode '\udce5' surrogates not allowed
Message-ID: <1325794375.32.0.250961574863.issue13717@psf.upfronthosting.co.za>


New submission from Atle Pedersen <atle.pedersen at gmail.com>:

I've made a short program to traverse file tree and print file names.

for root, dirs, files in os.walk(path):
        for f in files:
                hex = ' '.join(["%02X"%ord(x) for x in f])
                print('file is',hex,f)

This fails with the following file:

file is 67 72 DCE5 6B 61 6C 6C 65 6E 2E 6A 70 67 2E 68 74 6D 6C Traceback (most recent call last):
  File "/home/atle/bin/findpictures.py", line 16, in <module>
    print('file is',hexa,f)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce5' in position 2: surrogates not allowed

I don't really understand the issue, but this works with Python 2, and fails using 3.1.4 (gentoo: dev-lang/python-3.1.4-r3)

Same code using Python 2.7.2 gives:
('file is', '67 72 E5 6B 61 6C 6C 65 6E 2E 6A 70 67 2E 68 74 6D 6C', 'gr\xe5kallen.jpg.html')

----------
components: Unicode
messages: 150684
nosy: Atle.Pedersen, ezio.melotti
priority: normal
severity: normal
status: open
title: print fails on unicode '\udce5' surrogates not allowed
type: behavior
versions: Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13717>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:21:21 2012
From: report at bugs.python.org (Glenn Linderman)
Date: Thu, 05 Jan 2012 20:21:21 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325794881.53.0.369102435709.issue13703@psf.upfronthosting.co.za>


Changes by Glenn Linderman <v+python at g.nevcal.com>:


----------
nosy: +v+python

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:23:12 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 05 Jan 2012 20:23:12 +0000
Subject: [issue13717] print fails on unicode '\udce5' surrogates not allowed
In-Reply-To: <1325794375.32.0.250961574863.issue13717@psf.upfronthosting.co.za>
Message-ID: <1325794992.66.0.774129359074.issue13717@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

On Python 3, os.walk() uses the surrogateescape error handler.  If the filename is in e.g. iso-8859-* and the filesystem encoding is UTF-8, decoding '\xe5' will then result in '\udce5', and '\udce5' can't then be printed because it's a lone surrogate.

See also http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables

----------
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13717>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:23:43 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 20:23:43 +0000
Subject: [issue13717] print fails on unicode '\udce5' surrogates not allowed
In-Reply-To: <1325794375.32.0.250961574863.issue13717@psf.upfronthosting.co.za>
Message-ID: <1325795023.34.0.478284090844.issue13717@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The file tree contains a file which has an undecodable character in it. It ends up mangled as specified in PEP 383.
Priting such filenames is not directly supported (since they have invalid characters in them), but you can workaround it in several ways, for example escaping all non-ASCII chars: `print(ascii(f))`.

(note that opening the file will still work fine; only outputting the filename without special care will fail)

Python 2 is different since it doesn't attempt to decode filenames at all, it just treats them as opaque bytes.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13717>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:48:21 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 05 Jan 2012 20:48:21 +0000
Subject: [issue13511] Specifying multiple lib and include directories on linux
In-Reply-To: <1322683661.17.0.138081414237.issue13511@psf.upfronthosting.co.za>
Message-ID: <1325796501.72.0.884538307167.issue13511@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

See http://www.gnu.org/software/autoconf/manual/autoconf.html
for a description of the includedir and libdir options:

? Variable: includedir
The directory for installing C header files.
? Variable: libdir
The directory for installing object code libraries.

So it just doesn't make sense to have multiple directories in these options, and you shouldn't be passing directories that that you want to be searched.

Closing this report as invalid.

----------
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13511>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:54:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 20:54:39 +0000
Subject: [issue13697] python RLock implementation unsafe with signals
In-Reply-To: <CAH_1eM3sqsstA8qk+VuzEnbeP=2QJoFe_w4LcwiZc72U4HYQ_Q@mail.gmail.com>
Message-ID: <1325796804.8907.13.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> The first and most simple thing we could do would be to nuke the
> Python version (and also the loggin hack at the same time): does that
> sound reasonable ?

To me, yes, but I think it's better to ask on python-dev as for nuking
the Python version.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13697>
_______________________________________

From report at bugs.python.org  Thu Jan  5 21:55:00 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 05 Jan 2012 20:55:00 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325796900.41.0.764307621112.issue13609@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I haven't read much of this issue, but I strongly dislike the use of named tuples. Either we want people to use named fields, then we should use a regular class (possibly with slots), or we want to define the result as two values, then there should be a plain tuple result.

named tuples should only be used as a compatibility mechanism, when the first design used tuples, and it was later found that additional values need to be returned which would change the number of values (or the original design was considered bad because it returned too many positional values to properly keep track).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:03:11 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 05 Jan 2012 21:03:11 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1325797391.73.0.507235259876.issue13701@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I suggest to remove all mentionings of 2.3 compatibility from the file (I actually could find only a single one), for 3.3. Changing 3.2 is out of scope, as it isn't a bug fix (except that the one place referring to 2.3 claims that there is a comment at the top of the file which actually doesn't exist anymore).

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:12:49 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 05 Jan 2012 21:12:49 +0000
Subject: [issue10521] str methods don't accept non-BMP fillchar on a narrow
	Unicode build
In-Reply-To: <1290612325.35.0.372651063965.issue10521@psf.upfronthosting.co.za>
Message-ID: <1325797969.07.0.822503530627.issue10521@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I'm just going to close this and say "use 3.3".

----------
nosy: +benjamin.peterson
resolution:  -> out of date
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10521>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:12:49 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 05 Jan 2012 21:12:49 +0000
Subject: [issue10542] Py_UNICODE_NEXT and other macros for surrogates
In-Reply-To: <1290788167.05.0.282092739881.issue10542@psf.upfronthosting.co.za>
Message-ID: <1325797969.81.0.309541890365.issue10542@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Closing now.

----------
nosy: +benjamin.peterson
resolution:  -> out of date
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10542>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:24:53 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Thu, 05 Jan 2012 21:24:53 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325798693.8.0.0681137479856.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> I haven't read much of this issue, but I strongly dislike the use of 
> named tuples.
I don't really have a very strong opinion, but (cols, rows) does seem a lot like a tuple -- it really is just a pair of values without other function or state. Still I would much prefer to say
  get_terminal_size().columns
than
  get_terminal_size()[0]
So a bare tuple seems like the worst choice.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:40:04 2012
From: report at bugs.python.org (Paul McMillan)
Date: Thu, 05 Jan 2012 21:40:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325799604.17.0.952753487848.issue13703@psf.upfronthosting.co.za>


Paul McMillan <paul at mcmillan.ws> added the comment:

Marc-Andre: Victor already pasted the relevant part of my code:
http://bugs.python.org/issue13703#msg150568
The link to the fuller version, with revision history and a copy of the code before I modified it is here:
https://gist.github.com/0a91e52efa74f61858b5

>Why? The attack doesn't work with short strings? What do you call a "short string"?

Well, the demonstrated collision is for 16 character ascii strings. Worst case UTF-8, we're looking at 3 manipulable bytes per character, but they may be harder to collide since some of those bytes are fixed.

> only be making it harder for script kiddies, but as soon as someone
> crypt-analysis the used hash algorithm, you're lost again.

Not true. What I propose is to make the amount of information necessary to analyze and generate collisions impractically large. My proposed hash function is certainly broken if you brute force the lookup table. There are undoubtedly other problems with it too. The point is that it's hard enough. We aren't going for perfect security - we're going for enough to make this attack impractical.

What are the downsides to counting collisions? For one thing, it's something that needs to be kept track of on a per-dict basis, and can't be cached the way the hash results are. How do you choose a good value for the limit? If you set it to something conservative, you still pay the collision price every time a dict  is created to discover that the keys collide. This means that it's possible to feed to bad data up to exactly the limit, and suddenly the python app is inexplicably slow. If you set the limit too aggressively, then sometimes valid data gets caught, and python randomly dies in hard to debug ways with an error the programmer has never seen in testing and cannot reproduce.

It adds a new way to kill most python applications, and so programs are going to have to be re-written to cope with it. It also introduces a new place to cause errors - if the WSGI server dies, it's hard for my application to catch that and recover gracefully.

>... not in Python itself, but if you consider all the types in Python
> extensions and classes implementing __hash__ in user code, the number
> of hash functions to fix quickly becomes unmanageable.

When we looked at the Django project, we wouldn't have anything to fix since ours end up relying on the python internal values eventually. I suspect a lot of other code is similar.

Mark said:
>What is the mechanism by which the attacker can determine the seeds?

The biggest information leak is probably the ordering in which dict entries are returned. This can be used to deduce the underlying hash values. This is much easier than trying to do it via timing.

> But that's not the issue we are supposed to be dealing with.
> A single (genuinely random) seed will deal with the attack described in 
> the talk and it is (almost) as fast as using 0 as a seed.

This is not true. A single random seed shifts the hash table, but does not actually prevent an attacker from generating collisions. Please see my other posts on the topic here and on the mailing list.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:42:15 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 21:42:15 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1325798693.8.0.0681137479856.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325799659.8907.15.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

The point of named tuples here is that you can use both
    get_terminal_size().columns
or
    columns, rows = get_terminal_size()
depending on the situation.
Also, the better repr() makes debugging easier.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:53:38 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Thu, 05 Jan 2012 21:53:38 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325800418.55.0.438945483442.issue9993@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

I took the liberty to fix the tests.

Basically I've adapted them to the new mock based cross file system approach (that doesn't depend on luck anymore :)).

I also had to add one more `os.path.realpath` because on some OS (like OS X) the tmp directory path already consists of symlinks.

I didn't touch the actual code.

Tested on OS X and Ubuntu Linux (Oneiric).

----------
Added file: http://bugs.python.org/file24146/shutil_move_symlinks.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Thu Jan  5 22:56:01 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 05 Jan 2012 21:56:01 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325800561.14.0.00685125657639.issue12415@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi ?ric, did you reconsider the text of the patch or this issue just passed under your radar?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Thu Jan  5 23:04:36 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 05 Jan 2012 22:04:36 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325801076.92.0.858313006774.issue9993@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks! I think we also need a doc update for the change in behaviour (with a "versionchanged" tag).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Thu Jan  5 23:49:33 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 05 Jan 2012 22:49:33 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325803773.15.0.326274033406.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> What I propose is to make the amount of information necessary
> to analyze and generate collisions impractically large.

Not only: the attacker has to compute the collisions for the new seed. I don't know how long it is, the code to generate collisions is not public yet. I suppose than generating collisions is longer if we change the hash function to add more instructions (I don't know how much).

If generating the collisions requires a farm of computers / GPUs / something else and 7 days, it doesn't matter if it's easy to retreive the secret.

If the attack wants to precompute collisions for all possible seeds, (s)he will also have to store them. With 64 bits of entropy, if an attack is 1 byte long, you have to store 2^64 bytes (16,777,216 TB).

It is a problem if it takes less than a day with a desktop PC to generate data for an attack. In this case, it should be difficult to compute the secret.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 01:19:02 2012
From: report at bugs.python.org (Phill)
Date: Fri, 06 Jan 2012 00:19:02 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325809142.89.0.201067219737.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

Normally I would be happy to but my combined python experience is about 30 minutes so I am probably not the man

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 01:19:30 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 06 Jan 2012 00:19:30 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325809170.41.0.294935319431.issue13609@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Some comments about termsize.diff.3.

I don't see why there are two functions, one should be enough: get_terminal_size() should be dropped, and query_terminal_size() renamed to get_terminal_size(). As said before, I don't think that reading ROWS and COLUMNS environment variables is useful. If a program chose to rely on these variables, it can reimplement its own "try env var or fallback on get_terminal_size()" function.

get_terminal_size() should not have a fallback value: it should raise an error, and the caller is responsible to decide what to do in this case (e.g. catch the exception and use its own default value). Most functions in the posix module work like this, and it avoids the difficult choice of the right default value. (fallback=None is an hack to avoid an exception, it's not the pythonic.)

I don't think that it's possible that stdin, stdout and/or stderr have its own terminal. I suppose that the 3 streams are always attached to the same terminal. So I don't see why the function would take an argument. Tell me if I am wrong.

Instead of using sys.__stdout__.fileno(), you can directly use 1 because Python always create sys.__stdout__ from the file descriptor 1.

I think that a tuple (columns, rows) would be just fine. A namedtuple helps when you have a variable number of fields, or more than 3 fields, but here you just have 2 fields, it's not too much difficult to remember which one contains the columns.

I would prefer an optional function, instead of implementing a function raising a NotImplementedError. All other posix functions are defined like this.

ioctl() is already exposed in the fcntl module, I don't see a specific test for <sys/ioctl.h> header. It looks like the module is always compiled on Unix, I don't see how fcntl and ioctl are tested in setup.py.

I don't think that you need <conio.h> to get GetConsoleScreenBufferInfo(), <windows.h> should be enough. So just check for "#ifdef MS_WINDOWS".

Your function is helpful, and it is surprising that nobody proposed to implement it in Python. Some libraries did already implement their own function (like the "py" library).

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 01:23:11 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 06 Jan 2012 00:23:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325809391.0.0.140533755387.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Note for myself, random-2.patch: _PyRandom_Init() must generate a prefix and a suffix different than zero (call PyOS_URandom in a loop, and fail after 100 tries).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 01:26:48 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 00:26:48 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1325809170.41.0.294935319431.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325809532.8907.30.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I don't think that it's possible that stdin, stdout and/or stderr have
> its own terminal. I suppose that the 3 streams are always attached to
> the same terminal. So I don't see why the function would take an
> argument. Tell me if I am wrong.

I think it can be useful in case the program creates its own
session/terminal using openpty?

> Instead of using sys.__stdout__.fileno(), you can directly use 1
> because Python always create sys.__stdout__ from the file descriptor
> 1.

>From pythonrun.c:

    /* Set sys.stdin */
    fd = fileno(stdin);
    [...]
    /* Set sys.stdout */
    fd = fileno(stdout);
    [...]
    /* Set sys.stderr, replaces the preliminary stderr */
    fd = fileno(stderr);

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 01:48:26 2012
From: report at bugs.python.org (Phill)
Date: Fri, 06 Jan 2012 00:48:26 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325810906.99.0.144104621195.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

I have commented out the line:
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

In lib/multiprocessing/connection.py

as a test and it works fine, the problem still persists for named pipes (im not sure if thats how named pipes are supposed to behave though)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 02:06:02 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 01:06:02 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325811962.96.0.0313477738573.issue8184@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> im not sure if thats how named pipes are supposed to behave though

I'm not sure what you mean. Are you creating named pipes yourself?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 02:09:47 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 06 Jan 2012 01:09:47 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325812187.85.0.130998974385.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

"Given that a user has an application with an oracle function that returns the hash of a unicode string, an attacker can probe tenth of thousand one and two character unicode strings. That should give him/her enough data to calculate both seeds. hash("") already gives away lots of infomration about the seeds, too."

Sorry, but I don't see how you compute the secret using these data.

You are right, hash("\0") gives some information about the secret. With my patch, hash("\0")^1 gives: ((prefix * 1000003) & HASH_MASK) ^ suffix.

(hash("\0")^1) ^ (hash("\0\0")^2) gives ((prefix * 1000003) & HASH_MASK) ^ ((prefix * 1000003**2)  & HASH_MASK).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 02:44:18 2012
From: report at bugs.python.org (Christian Heimes)
Date: Fri, 06 Jan 2012 01:44:18 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325814258.07.0.160546102396.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Either we are really paranoid (I know that I am *g*) or Perl's and Ruby's randomized hashing function suffer from the issues we are worried about. They don't compensate for hash(''), hash(n * '\0') or hash(shortstring).

Perl 5.12.4 hv.h:

#define PERL_HASH(hash,str,len) \
     STMT_START { \
        register const char * const s_PeRlHaSh_tmp = str; \
        register const unsigned char *s_PeRlHaSh = (const unsigned char *)s_PeRlHaSh_tmp; \
        register I32 i_PeRlHaSh = len; \
        register U32 hash_PeRlHaSh = PERL_HASH_SEED; \
        while (i_PeRlHaSh--) { \
            hash_PeRlHaSh += *s_PeRlHaSh++; \
            hash_PeRlHaSh += (hash_PeRlHaSh << 10); \
            hash_PeRlHaSh ^= (hash_PeRlHaSh >> 6); \
        } \
        hash_PeRlHaSh += (hash_PeRlHaSh << 3); \
        hash_PeRlHaSh ^= (hash_PeRlHaSh >> 11); \
        (hash) = (hash_PeRlHaSh + (hash_PeRlHaSh << 15)); \
    } STMT_END

Ruby 1.8.7-p357 st.c:strhash()

#define CHAR_BIT 8
hash_seed = rb_genrand_int32() # Mersenne Twister

    register unsigned long val = hash_seed;

    while ((c = *string++) != '\0') {
        val = val*997 + c;
        val = (val << 13) | (val >> (sizeof(st_data_t) * CHAR_BIT - 13));
    }

    return val + (val>>5);

I wasn't able to find Java's fix quickly. Anybody else?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 02:50:08 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Fri, 06 Jan 2012 01:50:08 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325814608.37.0.931640244089.issue13703@psf.upfronthosting.co.za>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

Perl is so paranoid they obscure their variable names!  In all seriousness, both Perl and Ruby are vulnerable to the timing attacks, and as far as I know the JVM is not patching this themselves, but telling applications to fix it themselves (I know JRuby switched to Murmurhash).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 03:18:53 2012
From: report at bugs.python.org (Phill)
Date: Fri, 06 Jan 2012 02:18:53 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325816333.91.0.969165023049.issue8184@psf.upfronthosting.co.za>


Phill <beerb0x at gmail.com> added the comment:

Rather than listening on a socket, listening on a named pipe

eg:
address = (r'\\.\pipe\Test', 'AF_PIPE')
listener = Listener(*address)
conn = listener.accept()

It doesnt raise an exception when i run the script again a second time.

Like I said, I dont know much about named pipes and im not even sure thats how they are intended to work in this context. IE: if one process is listening, can another listen on that named pipe as well?

Phill

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 03:27:30 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Fri, 06 Jan 2012 02:27:30 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1325816850.55.0.653037923483.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <oh8gdv at gmail.com> added the comment:

> Regarding unittests instead, there is already a method called
> test_userpass_inurl which could be extended with some tests on a
> password containing spaces ( Lib/test/test_urllib.py:263). But what
> I haven't yet understood is: does it really exists a user:pass in
> python.org?

Note Lib/test/test_urllib.py:261 ; there is a fake HTTP wrapper in the test. So the request is not really sent.

I modified FakeHTTPConnection to store the sent HTTP request. I also copied the test you mentioned from python3 to 2.7. The second test I add in the patch fails. The test should pass with python2.5 from OS X (did not run the test but checked headers against netcat).

Please take a look at the tests I added. I'm not sure if geturl() should return the quoted version or not. But certainly the quoted version must not be used in the base64. If you think geturl() should return the quoted version, I'm fine with that - in principle characters like \n in the password could be bad in an URL unless quoted.

Maybe the tests could ALSO be added to some other places, but I think this full path makes sense to check like this.

----------
keywords: +patch
Added file: http://bugs.python.org/file24147/tests-and-fakehttp-request-storing.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Fri Jan  6 03:35:43 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Fri, 06 Jan 2012 02:35:43 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1325817343.23.0.982647386171.issue13701@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Mark, do you think the C version of decimal is going to happen for 3.3?

If so, it make little sense to make any changes at all the current version of the pure python code.

Another advantage to leaving the pure python code alone is that it will make maintenance easier if we have to make updates (remember, if the spec gets updated, that will be considered a bug fix and backported).

----------
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Fri Jan  6 03:50:28 2012
From: report at bugs.python.org (Paul McMillan)
Date: Fri, 06 Jan 2012 02:50:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325814608.37.0.931640244089.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRVw9YW2VCE-voXwkgE=4BNXkc+wZ1ViwvsSDJxS0bpFNw@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

As Alex said, Java has refused to fix the issue.

I believe that Ruby 1.9 (at least the master branch code that I looked
at) is using murmurhash2 with a random seed.

In either case, yes, these functions are vulnerable to a number of
attacks. We're solving the problem more completely than they did.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 03:57:40 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 06 Jan 2012 02:57:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325818660.63.0.112419628547.issue13703@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Those who use or advocate a simple randomized starting hash (Perl, Ruby, perhaps MS, and the CCC presenters) are presuming that the randomized hash values are kept private. Indeed, they should be (and the docs could note this) unless an attacker has direct access to the interpreter. An attacker who does, as in a Python programming class, can much more easily freeze the interpreter by 'accidentally' writing code equivalent to "while True: pass".

I do not think we, as Python developers, should be concerned about esoteric timing attacks. They strike me as a site issue rather than a language issue. As I understand them, they require *large* numbers of probes coupled with responses based on the same hash function. So a site being so probed already has bit of a problem. And if hashing were randomized per process, and probes were randomly distributed among processes, and processes were periodically killed and restarted with new seeds, could such an attack get anywhere (besides the DOS effect of the probing)? The point of the CCC talk was that with one constant known hash, one could lock up a server for a long while with just one upload.

So I think we should copy Perl and Ruby, do the easy thing, and add a random seed to 3.3 hashing, subject to keeping equality for equal numbers. Let whatever thereby fails, fail, and be updated. For prior versions, add an option for strings and perhaps numbers, and document that some tests will fail if enabled.

We could also consider, for 3.3, making the output of hash() be different from the internal values used for dicts, perhaps by switching random seeds in hash(). So even if someone does return hash(x) values to potential attackers, they are not the values used in dicts. (This would require a slight change in the doc.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 04:06:05 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Fri, 06 Jan 2012 03:06:05 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1325819165.62.0.417108248999.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

Updated patch for 2.7 hg tip attached. Please review, test and if ok, port to 3.x.

I guess the URL needs to be quoted so commented out the assertion for the URL being equal. I added unquote in the base64 encoding of the password, which makes the test pass. Seems to work for me and no urllib tests were broken. Did not run others.

http://test.webdav.org/ has some basic auth test accounts configured if you want to try it out. You can use wireshark to grab the base64 from the unencrypted http. Fancy opener works for me with this now, too.

----------
Added file: http://bugs.python.org/file24148/tests-and-fakehttp-request-storing-2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Fri Jan  6 07:10:02 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 06 Jan 2012 06:10:02 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1325799659.8907.15.camel@localhost.localdomain>
Message-ID: <20120106071000.Horde.lmWaTNjz9kRPBpA4mrbjqdA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Zitat von Antoine Pitrou <report at bugs.python.org>:

> Antoine Pitrou <pitrou at free.fr> added the comment:
>
> The point of named tuples here is that you can use both
>     get_terminal_size().columns
> or
>     columns, rows = get_terminal_size()
> depending on the situation.

And my point is that we should make the choice which
of these is more obvious, and drop the other.

> Also, the better repr() makes debugging easier.

A class could still have a nice repr.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 09:19:22 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 06 Jan 2012 08:19:22 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1325816333.91.0.969165023049.issue8184@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2XBsGC16uB9VG4aRx=ZpT49Hcq-vc-+PyCNveir7C+Qw@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Like I said, I dont know much about named pipes and im not even sure thats how they are intended to work in this context. IE: if one process is listening, can another listen on that named pipe as well?

Under Unix, you'd get a EADDRINUSE with a Unix domain socket.

I don't know much about Windows, but here's what CreateNamedPipe doc says:
"""
FILE_FLAG_FIRST_PIPE_INSTANCE
0x00080000
If you attempt to create multiple instances of a pipe with this flag,
creation of the first instance succeeds, but creation of the next
instance fails with ERROR_ACCESS_DENIED.
Windows 2000:  This flag is not supported until Windows 2000 SP2 and Windows XP.
"""

So it seems that we should probably pass FILE_FLAG_FIRST_PIPE_INSTANCE
under Windows.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 09:25:04 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Fri, 06 Jan 2012 08:25:04 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325838304.44.0.241372840347.issue9993@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

Oh sorry, I didn't look into the doc patch.

I unified both into one patch and added the versionchanged tag and also updated the docstring of shutil.move.

----------
Added file: http://bugs.python.org/file24149/shutil_move_symlinks.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Fri Jan  6 10:08:11 2012
From: report at bugs.python.org (Mark Shannon)
Date: Fri, 06 Jan 2012 09:08:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325818660.63.0.112419628547.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F06B9F8.9050609@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

I agree.

+1 for strings. -0 for numbers.

This might cause problems with dict subclasses and the like,
so I'm -1 on this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 10:31:13 2012
From: report at bugs.python.org (Mark Shannon)
Date: Fri, 06 Jan 2012 09:31:13 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325842273.5.0.235565271939.issue13703@psf.upfronthosting.co.za>


Mark Shannon <mark at hotpy.org> added the comment:

Without the context, that last message didn't make much sense.

I agree with Terry that we should copy Perl and Ruby (for strings).
I'm -1 on hash() returning a different value than dict uses internally.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 11:53:51 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Fri, 06 Jan 2012 10:53:51 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1325847231.6.0.631468216986.issue13642@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Some review comments. Instead of doing the inline unquote like this -

-            auth = base64.b64encode(user_passwd).strip()
+            auth = base64.b64encode(unquote(user_passwd)).strip()

It is better to do the explicitly above the b64 encoding step.
Just as host has been unquoted.

                    user_passwd, host = splituser(host)
                    host = unquote(host)

Also, you have done this only for https_open, the same would need be replicated for http_open and also for proxy_passwd.  Also on tests, Modifying sendall with

             def sendall(self, str):
-                pass
+                FakeHTTPConnection.request += str

seems a bit odd to me, you are using a class level object and adding a str. I think, there should be better way to do. (I shall provide an example). Also str term can replaced, even if it was coming from old code.

----------
assignee:  -> orsenthil
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Fri Jan  6 12:24:01 2012
From: report at bugs.python.org (Michael Kesper)
Date: Fri, 06 Jan 2012 11:24:01 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
Message-ID: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>


New submission from Michael Kesper <mkesper at schokokeks.org>:

http://docs.python.org/library/string.html#format-specification-mini-language mentions:
Changed in version 2.7: Added the ',' option (see also PEP 378).

PEP 378 tells me:
The ',' option is defined as shown above for types 'd', 'e', 'f', 'g', 'E', 'G', '%', 'F' and ''.

However:

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> '{:.2,%}'.format(.537568)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid conversion specification
>>> '{:2,%}'.format(.537568)
'53.756800%'

----------
messages: 150721
nosy: mkesper
priority: normal
severity: normal
status: open
title: Format Specification Mini-Language does not accept comma for percent value
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:30:17 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 12:30:17 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325853017.71.0.936491533882.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Following comments by Martin and Victor, here is next version: termsize.diff.4

Changes:
- just check for defined(MS_WINDOWS) and rely on <windows.h>.
- rename query_terminal_size to get_terminal_size_raw
     This way it should be clearer that the second one is low-level,
     and should be less exposed. I don't want to call it
     _get_terminal_size() because it is not just an implementation
     detail and would sometimes be called directly.
- NotImplementedError is gone. get_terminal_size_raw() is not
  defined if not possible.

Non-changes:
- sys.__stdout__.fileno() is not changed to 1, because as Antoine
  pointed out, it is set at runtime.
- STDOUT_FILENO: not defined on windows, so just use 1 and add a comment
- fd argument is retained, because we might want to test terminals
  opened with openpty.
- two functions: still there. I think that get_terminal_size() should
  provide an easy-to-use, even trivial-to-use, way to get a sensible
  value without writing a wrapper. In the _majority_ of cases the
  wrapper would be something like get_terminal_size() currently.
- named tuple: like Antoine said, it gives nice syntax.
- testing for <sys/ioctl.h>: in termios and other modules, setup.py
  first tests if we are on unix. But there might be unices without
  TIOCGWINSZ, and non-unix systems where TIOCGWINSZ _is_ defined,
  so it seems a functional test is simpler and more robust.

Tested on linux and windows XP.

----------
Added file: http://bugs.python.org/file24150/termsize.diff.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:30:38 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 12:30:38 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1325816333.91.0.969165023049.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325852961.3341.2.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

Le vendredi 06 janvier 2012 ? 02:18 +0000, Phill a ?crit :
> Phill <beerb0x at gmail.com> added the comment:
> 
> Rather than listening on a socket, listening on a named pipe
> 
> eg:
> address = (r'\\.\pipe\Test', 'AF_PIPE')
> listener = Listener(*address)
> conn = listener.accept()
> 
> It doesnt raise an exception when i run the script again a second time.

According to MSDN, this is normal:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365150%28v=vs.85%29.aspx

However, there's also a flag named FILE_FLAG_FIRST_PIPE_INSTANCE that we
could use in PipeListener, which would then raise an error if a listener
pipe was created a second time. It would probably make more sense,
although I don't know whether some programs may rely on creating a pipe
multiple times.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:49:17 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 12:49:17 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325842273.5.0.235565271939.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F06EDC3.7040600@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Before continuing down the road of adding randomness to hash
functions, please have a good read of the existing dictionary
implementation:

"""
Major subtleties ahead:  Most hash schemes depend on having a "good" hash
function, in the sense of simulating randomness.  Python doesn't:  its most
important hash functions (for strings and ints) are very regular in common
cases:

[0, 1, 2, 3]
>>> map(hash, ("namea", "nameb", "namec", "named"))
[-1658398457, -1658398460, -1658398459, -1658398462]
>>>

This isn't necessarily bad!  To the contrary, in a table of size 2**i, taking
the low-order i bits as the initial table index is extremely fast, and there
are no collisions at all for dicts indexed by a contiguous range of ints.
The same is approximately true when keys are "consecutive" strings.  So this
gives better-than-random behavior in common cases, and that's very desirable.
...
"""

There's also a file called dictnotes.txt which has more interesting
details about how the implementation is designed.

Please note that the term "collision" is used in a slightly different
way: it refers to trying to find an empty slot in the dictionary
table. Having a collision implies that the hash values of two distinct
objects are the same, but you also get collisions in case two distinct
objects with different hash values get mapped to the same table entry.

An attack can be based on trying to find many objects with the same
hash value, or trying to find many objects that, as they get inserted
into a dictionary, very often cause collisions due to the collision
resolution algorithm not finding a free slot.

In both cases, the (slow) object comparisons needed to find an
empty slot is what makes the attack practical, if the application
puts too much trust into large blobs of input data - which is
the actual security issues we're trying to work around here...

Given the dictionary implementation notes, I'm even less certain
that the randomization change is a good idea. It will likely
introduce a performance hit due to both the added complexity in
calculating the hash as well as the reduced cache locality of
the data in the dict table.

I'll upload a patch that demonstrates the collisions counting
strategy to show that detecting the problem is easy. Whether
just raising an exception is a good idea, is another issue.

It may be better to change the tp_hash slot in Python 3.3
to take an argument, so that the dict implementation can
use the hash function as universal hash family function
(see http://en.wikipedia.org/wiki/Universal_hash).

The dict implementation could then alter the hash parameter
and recreate the dict table in case the number of collisions
exceeds a certain limit, thereby actively taking action
instead of just relying on randomness solving the issue in
most cases.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:52:21 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 12:52:21 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325854341.31.0.572411522028.issue13703@psf.upfronthosting.co.za>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Demo patch implementing the collision limit idea for Python 2.7.

----------
Added file: http://bugs.python.org/file24151/hash-attack.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:56:09 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 12:56:09 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325854341.31.0.572411522028.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F06EF5F.2060702@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

The hash-attack.patch solves the problem for the integer case
I posted earlier on and doesn't cause any problems with the
test suite.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'too many hash collisions'

It also doesn't change the hashing or dict repr in existing
applications.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 13:56:49 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 12:56:49 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325854609.56.0.387311609534.issue13703@psf.upfronthosting.co.za>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Stupid email interface again... here's the full text:

The hash-attack.patch solves the problem for the integer case
I posted earlier on and doesn't cause any problems with the
test suite.

>>> d = dict((x*(2**64 - 1), hash(x*(2**64 - 1))) for x in xrange(1, 100))
>>> d = dict((x*(2**64 - 1), hash(x*(2**64 - 1))) for x in xrange(1, 1000))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'too many hash collisions'

It also doesn't change the hashing or dict repr in existing
applications.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:04:38 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 13:04:38 +0000
Subject: [issue13041] argparse: terminal width is not detected properly
In-Reply-To: <1316902039.33.0.744560309521.issue13041@psf.upfronthosting.co.za>
Message-ID: <1325855078.79.0.589356059948.issue13041@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

New version to use after #13609 is implemented: patch2.diff

----------
Added file: http://bugs.python.org/file24152/patch2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13041>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:18:38 2012
From: report at bugs.python.org (Ralf Schmitt)
Date: Fri, 06 Jan 2012 13:18:38 +0000
Subject: [issue13719] bdist_msi upload fails
Message-ID: <1325855918.9.0.207720990162.issue13719@psf.upfronthosting.co.za>


New submission from Ralf Schmitt <python-bugs at systemexit.de>:

Running setup.py bdist_msi upload fails with something like:

c:\Python27\python.exe setup.py bdist_msi upload
running bdist_msi
running build
running build_ext
installing to build\bdist.win-amd64\msi
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\msi
creating build\bdist.win-amd64\msi\Lib
creating build\bdist.win-amd64\msi\Lib\site-packages
copying build\lib.win-amd64-2.7\greenlet.pyd -> build\bdist.win-amd64\msi\Lib\site-packages
running install_headers
creating build\bdist.win-amd64\msi\Include
creating build\bdist.win-amd64\msi\Include\greenlet
copying greenlet.h -> build\bdist.win-amd64\msi\Include\greenlet
running install_egg_info
Writing build\bdist.win-amd64\msi\Lib\site-packages\greenlet-0.3.3-py2.7.egg-info
creating dist
removing 'build\bdist.win-amd64\msi' (and everything under it)
running upload
Linking c:\Users\ralf\home\greenlet\build\lib.win-amd64-2.7\greenlet.pyd to c:\Users\ralf\home\greenlet\greenlet.pyd
error: greenlet-0.3.3: No such file or directory

The attached patch fixes it.

----------
assignee: tarek
components: Distutils
files: fix-bdist_msi-upload.patch
keywords: patch
messages: 150729
nosy: eric.araujo, loewis, schmir, tarek
priority: normal
severity: normal
status: open
title: bdist_msi upload fails
versions: Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file24153/fix-bdist_msi-upload.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13719>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:19:58 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 13:19:58 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1325855998.02.0.814986313066.issue8184@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

(oops, hadn't seen Charles-Fran?ois's answer before replying)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:27:11 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 13:27:11 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325856431.52.0.541848499517.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

One more comment on $COLUMNS overriding the actual terminal size:

> Zbyszek, I just looked at [1] and I disagree that the environment
> variable should have higher precedence. In fact, I believe it should
> have lower precedence, and should be used as a fallback.

To fix issue #9553 "test_argparse.py: 80 failures if COLUMNS env var set to a value other than 80", sys.env['COLUMNS']=80 is set during tests.
If issue #13041 is fixed and a real terminal size is used, then
unless $COLUMNS have higher preference, the tests would break again.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:44:57 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Fri, 06 Jan 2012 13:44:57 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325857497.66.0.387909902654.issue13609@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Some remarks on the Windows implementation in termsize.diff.4:

- On Windows, the C runtime always sets fileno(stdout) to 1, so hardcoded values are OK.
But on Unix, I'm quite sure that embedded interpreters (mod_python?) sometimes close the standard descriptor, so fd=1 can refer to something entirely different.
Does it makes sense to initialize fd=fileno(stdout) (this is C code) instead?

- When GetStdHandle() returns INVALID_HANDLE_VALUE, PyErr_SetFromWindowsErr(0) should be used.
And it's not necessary to use GetLastError(), 0 is enough.

- GetStdHandle will return NULL in a non-console application (try with pythonw.exe or IDLE), I think a specific error message should be raised in this case.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 14:55:48 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 13:55:48 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
Message-ID: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>


New submission from Zbyszek Szmek <zbyszek at in.waw.pl>:

% cat test_argparse_narrow.py
import argparse
argparse.ArgumentParser().print_help()

% COLUMNS=15 ./python test_argparse_narrow.py
Traceback (most recent call last):
  File "test_argparse_narrow.py", line 2, in <module>
    argparse.ArgumentParser().print_help()
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 2347, in print_help
    self._print_message(self.format_help(), file)
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 2321, in format_help
    return formatter.format_help()
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 276, in format_help
    help = self._root_section.format_help()
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 206, in format_help
    func(*args)
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 206, in format_help
    func(*args)
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 514, in _format_action
    help_lines = self._split_lines(help_text, help_width)
  File "/home/zbyszek/python/cpython/Lib/argparse.py", line 615, in _split_lines
    return _textwrap.wrap(text, width)
  File "/home/zbyszek/python/cpython/Lib/textwrap.py", line 316, in wrap
    return w.wrap(text)
  File "/home/zbyszek/python/cpython/Lib/textwrap.py", line 291, in wrap
    return self._wrap_chunks(chunks)
  File "/home/zbyszek/python/cpython/Lib/textwrap.py", line 220, in _wrap_chunks
    raise ValueError("invalid width %r (must be > 0)" % self.width)
ValueError: invalid width -1 (must be > 0)

argparse should not fail if the user resizes the window to something very thin...

----------
components: Library (Lib)
files: test_argparse_narrow.py
messages: 150733
nosy: zbysz
priority: normal
severity: normal
status: open
title: argparse print_help() fails if COLUMNS is set to a low value
versions: Python 3.3
Added file: http://bugs.python.org/file24154/test_argparse_narrow.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Fri Jan  6 15:53:48 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Fri, 06 Jan 2012 14:53:48 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1325861628.99.0.523505302995.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

> It is better to do the explicitly above the b64 encoding step.
> Just as host has been unquoted.
> 
>                     user_passwd, host = splituser(host)
>                     host = unquote(host)

Ok. So it needs to be done on the line after import base64.

> Also, you have done this only for https_open, the same would need be
> replicated for http_open and also for proxy_passwd.

So four cases where this may need to be fixed and my test only covers one of them:

* http without proxy
* http with proxy
* https without proxy
* https with proxy

Copypasted code :(

Can the https and the proxy auth be tested with the same fake http connection, when the request is stored?

>              def sendall(self, str):
> -                pass
> +                FakeHTTPConnection.request += str
> 
> seems a bit odd to me

Agreed, not clean and needs to be fixed. Works here, but could cause the test code to become unreadable later on. Wrote it at 5 am, not getting sleep. Please provide a cleaner alternative. :)

One question: how come the fake http is given HTTP headers in some tests and payload only in others? Is it emulating the TCP stream or the payload stream handle? It can't do both, can it? Are the headers in some tests actually doing anything?

I would not mind if someone else finished the patch :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Fri Jan  6 16:11:39 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Fri, 06 Jan 2012 15:11:39 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
In-Reply-To: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>
Message-ID: <1325862699.07.0.783354870644.issue13718@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +eric.smith, ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Fri Jan  6 16:20:06 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Fri, 06 Jan 2012 15:20:06 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
In-Reply-To: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>
Message-ID: <1325863206.65.0.0216872958685.issue13718@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
assignee:  -> eric.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:27:52 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:27:52 +0000
Subject: [issue13615] setup.py register fails with -r argument
In-Reply-To: <1324070497.95.0.248126770194.issue13615@psf.upfronthosting.co.za>
Message-ID: <1325867272.14.0.782777876994.issue13615@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Maybe related to (or same as) #11473.

----------
assignee: tarek -> eric.araujo
components: +Distutils2
nosy: +alexis
stage:  -> test needed
title: `setup.py register` fails with -r argument -> setup.py register fails with -r argument
versions: +3rd party, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13615>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:33:19 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:33:19 +0000
Subject: [issue11473] upload command no longer accepts repository by section
	name
In-Reply-To: <1299882717.19.0.490703720004.issue11473@psf.upfronthosting.co.za>
Message-ID: <1325867599.23.0.41298167243.issue11473@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

This is a strange bug.  I added a test using -r server2, using the already-existing PYPIRC_LONG_PASSWORD string as .pypirc contents.  The test passes.  To make sure changing one test would not affect another one, I created a new .pypirc file, PYPIRC_CUSTOM_SERVER, and then I got to see your bug!  I tried various combinations of keys (realm or not, user or not), changed the markup (: or = as delimiter, spaces or not) but could not easily find the root of the bug.

Can you publish a .pypirc file that works with 2.7 and not with 3.2?  That would be a good starting point.

msg144569 - (view) 	Author: Jason R. Coombs (jason.coombs) * (Python committer) 	Date: 2011-09-28 18:09

I now seem to be unable to reproduce the issue. I do keep my .pypirc in a revision control system, so I have reasonable confidence that my .pypirc contained the content that I'm attaching now (passwords scrubbed of course).

> One possible factor is that my .pypirc is symlinked from ~/.pypirc to config/python/.pypirc and
> it's conceivable that Python 3.2.0 has some issues with symlink resolution that caused it to not
> process the .pypirc properly.

Are these Windows symlinks?  Are they supposed to be dereferenced transparently by the OS (i.e. during open) like unix symlinks?

> Sure enough. I just confirmed that with Python 3.2.0, if .pypirc is symlinked, distutils behaves
> as if the .pypirc isn't present at all, but if that same .pypirc is copied, it behaves as
> expected. If one deletes the .pypirc altogether, it produces the same error.

OK.  For distutils2, I think we?ll want better logging/error reporting.  The user should see ?no /what/ever/.pypirc found?, not ?section not found?.

> I suggest we mark this bug as resolved (as the primary cause was due to the file not being read at
> all), but also apply your patch (which has some other good fixes) without the CUSTOM_SERVER, or
> with a version of the CUSTOM_SERVER that works.

Agreed.  I?ll resume work on that patch to understand why it fails.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11473>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:33:43 2012
From: report at bugs.python.org (Ray)
Date: Fri, 06 Jan 2012 16:33:43 +0000
Subject: [issue13511] Specifying multiple lib and include directories on linux
In-Reply-To: <1322683661.17.0.138081414237.issue13511@psf.upfronthosting.co.za>
Message-ID: <1325867623.9.0.571021737523.issue13511@psf.upfronthosting.co.za>


Ray <rpqpro at hotmail.com> added the comment:

Martin, did you read my last post? Did you read the updated topic? libdir and includedir are *not* the issue.

So it makes sense to allow configuring multiple directories for python installation for darwin but not linux? Why?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13511>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:35:05 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 06 Jan 2012 16:35:05 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325854609.56.0.387311609534.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwbMEe4zmsbkDL-iZGrErut7g-bsYamc5RHWkTuBWy8_CQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

hash-attack.patch does never decrement the collision counter.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:42:42 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:42:42 +0000
Subject: [issue13033] Add shutil.chowntree
In-Reply-To: <1316769849.87.0.0298236424523.issue13033@psf.upfronthosting.co.za>
Message-ID: <1325868162.49.0.798309417833.issue13033@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the patch.  Before you do any more work, do other core developers agree that this function is a good addition or is it obsoleted by the generic improved-walk-with-callback that Nick is working on?

Doc/library/shutil.rst:

I don?t think the note directives are needed.  The doc is not big, I assume people will read all of it and see the caveats.  (I?ll also want to group some small paragraphs.)

Lib/shutil.py:

+def chowntree(path, user=None, group=None, followlinks=False):
+ [...]
+    The dictionary _modified_items, will keep track of the old ownership details,
What _modified_items dictionary?

Apart from a few stylistic violations which can be fixed by the committer, the function looks good.

Lib/test/test_shutil.py:

Looks good and needs more tests.  Currently it only calls chowntree on a directory without children, so it does not test that the chown is indeed recursive.

----------
nosy: +giampaolo.rodola, haypo, ncoghlan, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13033>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:43:12 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:43:12 +0000
Subject: [issue1040439] Missing documentation on how to link with libpython
Message-ID: <1325868192.22.0.587437984751.issue1040439@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Attached patch adds the useful bits from the review I did; I hope you won?t think it?s pointless nitpicking.

----------
Added file: http://bugs.python.org/file24155/docembed-edits.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1040439>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:44:50 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:44:50 +0000
Subject: [issue13562] Notes about module load path
In-Reply-To: <1323397324.61.0.220189805252.issue13562@psf.upfronthosting.co.za>
Message-ID: <1325868290.99.0.256887957781.issue13562@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I did the markup edits but then found out that the doc for Py_Initialize in c-api/init already mentions the module load path and the module table, so I?m going to make another patch with just a link to that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13562>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:49:21 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:49:21 +0000
Subject: [issue13719] bdist_msi upload fails
In-Reply-To: <1325855918.9.0.207720990162.issue13719@psf.upfronthosting.co.za>
Message-ID: <1325868561.73.0.127993472282.issue13719@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks!  Given the fragility of distutils, we try to add a test for every fix we commit, but I don?t remember if we already have a basic HTTP server to test the upload command.  If not, I think I?ll commit this after manual testing (my Windows VM crashes on boot, so it may take some time for me to get to this).

----------
assignee: tarek -> eric.araujo
components: +Distutils2
nosy: +alexis
stage:  -> test needed
versions: +3rd party, Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13719>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:49:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:49:55 +0000
Subject: [issue11473] upload command no longer accepts repository by section
	name
In-Reply-To: <1299882717.19.0.490703720004.issue11473@psf.upfronthosting.co.za>
Message-ID: <1325868595.82.0.280255294078.issue11473@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150736

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11473>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:50:31 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:50:31 +0000
Subject: [issue11473] upload command no longer accepts repository by section
	name
In-Reply-To: <1299882717.19.0.490703720004.issue11473@psf.upfronthosting.co.za>
Message-ID: <1325868631.73.0.746869465106.issue11473@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

[copy/paste failure in my previous message, please disregard]

> One possible factor is that my .pypirc is symlinked from ~/.pypirc to config/python/.pypirc and
> it's conceivable that Python 3.2.0 has some issues with symlink resolution that caused it to not
> process the .pypirc properly.

Are these Windows symlinks?  Are they supposed to be dereferenced transparently by the OS (i.e. during open) like unix symlinks?

> Sure enough. I just confirmed that with Python 3.2.0, if .pypirc is symlinked, distutils behaves
> as if the .pypirc isn't present at all, but if that same .pypirc is copied, it behaves as
> expected. If one deletes the .pypirc altogether, it produces the same error.

OK.  For distutils2, I think we?ll want better logging/error reporting.  The user should see ?no /what/ever/.pypirc found?, not ?section not found?.

> I suggest we mark this bug as resolved (as the primary cause was due to the file not being read at
> all), but also apply your patch (which has some other good fixes) without the CUSTOM_SERVER, or
> with a version of the CUSTOM_SERVER that works.

Agreed.  I?ll resume work on that patch to understand why it fails.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11473>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:51:46 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:51:46 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325868706.26.0.24608303108.issue12415@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I merely forgot, thanks for the prod.  I?m taking your message as a +1 to the wording and will commit when the server I use for tunnels is back online.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:54:13 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 16:54:13 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325868853.17.0.612749312845.issue13716@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Tarek ruled that time spent improving the distutils docs was time lost and better spent on distutils2.  Accordingly, I only fix clear bugs in the doc but don?t improve them in any way.  I?m inclined to close this as wontfix, or I can remove the XXX if you?re strongly against them.  (For the packaging docs, I will fix them properly.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Fri Jan  6 17:55:02 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 16:55:02 +0000
Subject: [issue1040439] Missing documentation on how to link with libpython
In-Reply-To: <1325868192.22.0.587437984751.issue1040439@psf.upfronthosting.co.za>
Message-ID: <1325868825.3341.7.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Attached patch adds the useful bits from the review I did; I hope you
> won?t think it?s pointless nitpicking.

Looks good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1040439>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:02:03 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Fri, 06 Jan 2012 17:02:03 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325869323.36.0.156979773145.issue12415@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

http://docs.python.org/dev/documenting/building.html also needs to be updated (it still mentions Subversion and probably obsolete versions of Sphinx and other packages).  If you are going to add the text proposed in your second message, you might want to drop a couple of "p"s from there.

----------
nosy: +ezio.melotti
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:03:08 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 17:03:08 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwbMEe4zmsbkDL-iZGrErut7g-bsYamc5RHWkTuBWy8_CQ@mail.gmail.com>
Message-ID: <4F072943.8090603@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner at haypocalc.com> added the comment:
> 
> hash-attack.patch does never decrement the collision counter.

Why should it ? It's only used as local variable in the lookup function.

Note that the limit only triggers on a per-key basis. It's not
a limit on the total number of collisions in the table, so you don't
need to keep the number of collisions stored on the object.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:05:18 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 17:05:18 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325869518.48.0.0483429952209.issue12415@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> http://docs.python.org/dev/documenting/building.html also needs to be updated (it
> still mentions Subversion and probably obsolete versions of Sphinx and other
> packages).
Well, the doc makefile still uses Subversion.  As for the versions, if the ones listed in the doc don?t match the ones used in the makefile, I?ll sync them.

> If you are going to add the text proposed in your second message, you might want
> to drop a couple of "p"s from there.
I don?t understand: "p"s?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:11:11 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 17:11:11 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1325869871.33.0.877368574599.issue13691@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

IMO, help('help') should document the help function, not start an interactive help session (that?d be help()).

----------
nosy: +eric.araujo
versions: +Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:12:49 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Fri, 06 Jan 2012 17:12:49 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325869969.77.0.557897583364.issue12415@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

> Well, the doc makefile still uses Subversion. 
Indeed, I was thinking about getting a CPython checkout with the doc, not about the tools used to built it (maybe it could be clarified).

> I don?t understand: "p"s?
>From "_Developpers's Guide: http://docs.python.org/devguide"

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:16:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 06 Jan 2012 17:16:55 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1325870215.95.0.0353341882579.issue12415@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> Indeed, I was thinking about getting a CPython checkout with the doc
Right.  I?ll edit my wording to mention Mercurial.

>> I don?t understand: "p"s?
>From "_Developpers's Guide: http://docs.python.org/devguide"
The curse of being French :)  I always want to write ?developpers? guide? when it?s ?developer?s guide?.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:31:44 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 06 Jan 2012 17:31:44 +0000
Subject: [issue11473] upload command no longer accepts repository by
	section	name
In-Reply-To: <1325868631.73.0.746869465106.issue11473@psf.upfronthosting.co.za>
Message-ID: <95F84BBD-520A-4569-90D8-5631F636B8E7@jaraco.com>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

Windows symlinks, much like Unix symlinks, should dereference naturally when open is called, although there is a bug in later versions of the c runtime calling stat on a symlink. See issue6727 for details.

----------
title: upload command no longer accepts repository by section name -> upload command no longer accepts repository by section	name
Added file: http://bugs.python.org/file24156/smime.p7m

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11473>
_______________________________________
-------------- next part --------------
Content-Type: multipart/signed; micalg=sha1;
	boundary="Apple-Mail-65462CE5-42F6-47A8-8F19-8E32853CCBD1";
	protocol="application/pkcs7-signature"

--Apple-Mail-65462CE5-42F6-47A8-8F19-8E32853CCBD1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Windows symlinks, much like Unix symlinks, should dereference naturally when=
 open is called, although there is a bug in later versions of the c runtime c=
alling stat on a symlink. See issue6727 for details.=

--Apple-Mail-65462CE5-42F6-47A8-8F19-8E32853CCBD1
Content-Disposition: attachment; filename="smime.p7s"
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIVrjCCBjQw
ggQcoAMCAQICASIwDQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0
Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAn
BgNVBAMTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDMzM1oX
DTE3MTAyNDIxMDMzM1owgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSsw
KQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFy
dENvbSBDbGFzcyAzIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBALmjSW4SPiDKlAinvVeLZOVfItiuP1aRHL530E7QUc9icCwL33+P
H+Js1HAh8CgWFl34sOxx1FJyS/C4VLPRsqDfP72jtzCVUAL0DAxZ7wgzQvFz7x61jGxfhYhqYb1+
PPOLkYBbkRIrPMg3dLEdKmXIYJYXDH+mB/V/jLo73/Kb7h/rNoNg/oHHSv5Jolyvp5IY2btfcTBf
W/telEFj5rDTX2juTvZ3Qhf3XQX5ca3Q7A10zrUV/cWJOJ7F5RltbEIaboZmX5JBUb3FhUiAdBot
ehAX6DbDOuYoJtVxmGof6GuVGcPo98K4TJf8FHo+UA9EOVDp/W7fCqKT4sXk/XkCAwEAAaOCAa0w
ggGpMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBR7iZySlyShhEcC
y3T8LvSs3DLl8zAfBgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRa
MFgwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYh
aHR0cDovL3d3dy5zdGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6
Ly93d3cuc3RhcnRzc2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5j
b20vc2ZzY2EuY3JsMIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0
dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cu
c3RhcnRzc2wuY29tL2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAABznBL1seNF
AhfG5hdukfQLzqsDJ452dB9CVqgUeTobh4x55PngizYKZ6+ieVlDtPd2MB7U2ck6L7tyMxMpvRoa
74Sokhl5ZgwTQZb7xB9rKoUPxygFX2biNm+r+l04+COmJGb02iLSPR9OFgqhk5qNK83IpT8QJD5V
XP7heTH3TdHGFWXimRYGXBmuVOXEpXYaZjn7syQuXr7k4RZuFp32FOzfzTBrZWDgixjJB+OJkMfp
yu6GvEXeYWrks+ILjHt78VUj8DHxB2i6JTkmA51vR181KEiAYtZbiDK21UgncNxEIeU3b0vhpE4c
KT5kcZaXFK58fLRthoHae6br4lQsZIMEbHhScnzH9uAg3qL7AGBILDwu6iasejOhkoSp6opZDeUS
UfEPE0tHwKFur3Ts1K52EyXBAr1aA67ECgL9ka2yPt1YA/mj7frpI9li/F311dn7+pHT1C3lFPRR
zOc6i9gAXjBOWJq+Yl0dSubTn2xVdS+swwfjR0uLWqoSH9Hra4sXDHxslb/KaDXJw8g1EvBzgP0Z
UHolzyrDgvwL/WOuLWek6koXP89CRPuWtC0iC/vZ4IcCgGuhniwYV4XhrdAW9FKlKdRi8SEBGYpV
QiCPVhC5wF2oQvsEs6szAThCuamfAmZnFCUBdCcVTXQuX/OXoQe4Fl70ePaZhjS6MIIHpTCCBo2g
AwIBAgICALswDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENv
bSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYD
VQQDEy9TdGFydENvbSBDbGFzcyAzIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTAeFw0x
MTExMjQwODIzMjZaFw0xMzExMjUwMTQ2NDBaMIGfMSAwHgYDVQQNExc1NzQwMDMtZENER0JlOGhC
NVQ1UTVWcjELMAkGA1UEBhMCVVMxHTAbBgNVBAgTFERpc3RyaWN0IG9mIENvbHVtYmlhMRMwEQYD
VQQHEwpXYXNoaW5ndG9uMRgwFgYDVQQDEw9KYXNvbiBSLiBDb29tYnMxIDAeBgkqhkiG9w0BCQEW
EWphcmFjb0BqYXJhY28uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtIz2vssW
Ao03XQ0a/2nhvrtEXgzBiIorvVjpqug/KtFOCmfQUTf6Syvx3YVcALdVWoSPR9ZVJr9MlcatsqYi
5uEUX+k3x7fTYzcnK43GFIEnr16ahIfXO/Gm6WzxzRFvku+6fxzj9Eogutvk1nPd7GviModadzjO
jYxDcq9ZeFFc8QvUxp1gUwv8Bp8KQsWDTM25G6P/I8k+9p9mSnRVWcBpY9QkAOPK/wKFJcxle+sj
xcdbV6hwHZMRzMfzeouD6+OWyYfkDplII0RLFCrWgrOnAX5gufkQLcm+M0a/hedZLhkHP9/P/IVj
txcotoyqOTOJnJihfxOoASWpq0rb6QIDAQABo4ID+jCCA/YwCQYDVR0TBAIwADALBgNVHQ8EBAMC
BLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBQHApdr6WTS4rXR9eil
zbBGH8yIpDAfBgNVHSMEGDAWgBR7iZySlyShhEcCy3T8LvSs3DLl8zBqBgNVHREEYzBhgRFqYXJh
Y29AamFyYWNvLmNvbYERamFyYWNvQGphcmFjby5jb22BEWphcmFjb0Bjcy5ubXQuZWR1gRBqYXJh
Y29AZ21haWwuY29tgRRqYXJhY29Ac3BlYWtlYXN5Lm5ldDCCAiEGA1UdIASCAhgwggIUMIICEAYL
KwYBBAGBtTcBAgIwggH/MC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xp
Y3kucGRmMDQGCCsGAQUFBwIBFihodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9pbnRlcm1lZGlhdGUu
cGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAD
AgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNzdWVkIGFjY29yZGluZyB0byB0aGUgQ2xhc3Mg
MyBWYWxpZGF0aW9uIHJlcXVpcmVtZW50cyBvZiB0aGUgU3RhcnRDb20gQ0EgcG9saWN5LCByZWxp
YW5jZSBvbmx5IGZvciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBjb21wbGlhbmNlIG9mIHRoZSBy
ZWx5aW5nIHBhcnR5IG9ibGlnYXRpb25zLjCBnAYIKwYBBQUHAgIwgY8wJxYgU3RhcnRDb20gQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBAhpkTGlhYmlsaXR5IGFuZCB3YXJyYW50aWVzIGFyZSBs
aW1pdGVkISBTZWUgc2VjdGlvbiAiTGVnYWwgYW5kIExpbWl0YXRpb25zIiBvZiB0aGUgU3RhcnRD
b20gQ0EgcG9saWN5LjA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9j
cnR1My1jcmwuY3JsMIGOBggrBgEFBQcBAQSBgTB/MDkGCCsGAQUFBzABhi1odHRwOi8vb2NzcC5z
dGFydHNzbC5jb20vc3ViL2NsYXNzMy9jbGllbnQvY2EwQgYIKwYBBQUHMAKGNmh0dHA6Ly9haWEu
c3RhcnRzc2wuY29tL2NlcnRzL3N1Yi5jbGFzczMuY2xpZW50LmNhLmNydDAjBgNVHRIEHDAahhho
dHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wDQYJKoZIhvcNAQEFBQADggEBALj5oD4GgcCRCQcnaaRB
8+UyCkPMmJCaXvsRPseEv+zozL0Iq/s3Uj1sMir9W+vK74q22kPxvuaPnTUTePa6m/y4RpQpIh/Y
d3WkaeHtM2zvIXpeK2FsoOObIWo71yR18uyttTIQOEbg94ZYriIReBkob0jhUq35wzpqoxURSB+2
IRQ1Wwpgft7PjO/0e1yuHPuXtg5i7FlupZqCyo9NuIbWi1ZMaRl22V9b4hjQaxCAU4W/unqoQgY3
Lhz2Zz5YmXsbKGdIYIpKmRGqXE55RawY0uZc/7FqKfP3NQHp50srze53tkA5Rc27WtWOCD2FrEdn
PTflmqnf43IikwO43CgwggfJMIIFsaADAgECAgEBMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYT
AklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0
aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQDEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0wNjA5MTcxOTQ2MzZaFw0zNjA5MTcxOTQ2MzZaMH0xCzAJBgNVBAYTAklMMRYwFAYDVQQK
Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu
aW5nMSkwJwYDVQQDEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMGI2wm8bEZ8eJ+Ve7UzkPJyYtbBNiAiJF7O6XfyQwqiBmSk
zI42+DjmI/BubbE83XKjhRyh0z20MyvTL6/+6rBBWWe2xAZ9Cp50hdZ5TIA3et85BVJZ9/QbRkOk
0oWF0sNx83ViNLosin8ej+7tNNARx5bNUj26M9bdTd4LO0pLn8ImL/q1FhxyNXfKPF3myuEmixo2
dlwB23QUJf7ttaCID914yi0fB5cwAS1yefpG1hMqqLmmq4NJHeXy793kAY4YCo9jUxaFYqkOGTrM
tWamwmt0B+Qr4XY+tG3Y9kThc2IfO8S+oFNWJWxRCfeqq8q/dv1tm/Od2789ZrwMVqqvmEiVOkvf
p1hQ2Th1qVvqQwwC/5nr6GxNcFspZZzdql3MrwEx7Azr0o3o6px75m73J2YMGkjXbkLjP94hPnvh
DXD7Y6qobBpUtFwlesmiyYsWprssfhdeBU1YbhIdAe4SEA3GMn8Y//z0+s1ukeg2Sb4aSGmLwpZN
GhKyaRfBCpDW+nkiSL+6e2n4cMf6ejfY2A3Sdk9X/5C345HS3e/CYLdnOt3+qpzw1It/ciLOxp+X
tviviqAQqNn7GMa2tVxSPIm2GSpzAQoPA7MSYPJ6L4Hbo27/JjCX9YvdiVe2rT2zryvFt3YC8KXW
K5qGFCpy9uMzjF0JSxPfu4x0E1JLAgMBAAGjggJSMIICTjAMBgNVHRMEBTADAQH/MAsGA1UdDwQE
AwIBrjAdBgNVHQ4EFgQUTgvvGqRAW6UXaYcwyjRoQ9BBrvIwZAYDVR0fBF0wWzAsoCqgKIYmaHR0
cDovL2NlcnQuc3RhcnRjb20ub3JnL3Nmc2NhLWNybC5jcmwwK6ApoCeGJWh0dHA6Ly9jcmwuc3Rh
cnRjb20ub3JnL3Nmc2NhLWNybC5jcmwwggFdBgNVHSAEggFUMIIBUDCCAUwGCysGAQQBgbU3AQEB
MIIBOzAvBggrBgEFBQcCARYjaHR0cDovL2NlcnQuc3RhcnRjb20ub3JnL3BvbGljeS5wZGYwNQYI
KwYBBQUHAgEWKWh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9pbnRlcm1lZGlhdGUucGRmMIHQBggr
BgEFBQcCAjCBwzAnFiBTdGFydCBDb21tZXJjaWFsIChTdGFydENvbSkgTHRkLjADAgEBGoGXTGlt
aXRlZCBMaWFiaWxpdHksIHJlYWQgdGhlIHNlY3Rpb24gKkxlZ2FsIExpbWl0YXRpb25zKiBvZiB0
aGUgU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgUG9saWN5IGF2YWlsYWJsZSBhdCBo
dHRwOi8vY2VydC5zdGFydGNvbS5vcmcvcG9saWN5LnBkZjARBglghkgBhvhCAQEEBAMCAAcwOAYJ
YIZIAYb4QgENBCsWKVN0YXJ0Q29tIEZyZWUgU1NMIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MA0G
CSqGSIb3DQEBBQUAA4ICAQAWbJn0Zgw09dCFXn0K7NoQTjgcXt+mJQVLkTLB6DvxPd1ECVsHSYop
y2YCt7Ga9yWYCTyOG+HdNocrS7to0zlmPaAmx/I5kR1Rq4J7ftXOWuTiA1dwaZcI+V5YpgrfjAaa
RRYWOApeV/Zix3oCBea8HrXynvSpKYP4shTjbiiHRMOQGt44qTysQ01kRc7dKKlc8nN7BPgX6Kux
8y5cZG5zMToSuLyzEeR9j4FRmjuNifRNk2Z7PAPt05odmvNlUPWg0HWfL6/w6oJDmPhpnIl5xEOO
RnLjZDYSr/clHjiJkHd+w2tqucPLREuseJCL58csHksRRMg0UifNCl2fhcGJ1Rp48pUQUzLdgIRm
ddm1aCj7YS6+hKg4wJkShqUeZ2StBi4vqXCFx5YPfIll9Y5DVA6r3aWAOZRgwDTJlnAsoxL1H0h7
vRx+a7edkPQiO674/CrK+oJSoO+vS1WT68G18CKLrDROJiIEoYcsdUq35X0T17gMZMA20skvhhKM
IwnBG4I7c0mjaleHlOXWeMWZQ2PjTeB3LeFlmXJpBBpHCeYPAVYk+x+/DnmpWC65xAkBfpW6bQAG
PrLqShA52NAr9b/sdb+XAsUJGwjcVTfigfs3hENiIMrnVktl6v5swSSTJKE06wX/miKum30/8WVR
CqYwarP0iByADfxyiuiDXjGCA2wwggNoAgEBMIGTMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMN
U3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmlu
ZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMyBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQg
Q0ECAgC7MAkGBSsOAwIaBQCgggGtMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcN
AQkFMQ8XDTEyMDEwNjE3MzEzNVowIwYJKoZIhvcNAQkEMRYEFBGnT2qu3rgXWSkX+2+FMnwQsMoZ
MIGkBgkrBgEEAYI3EAQxgZYwgZMwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBM
dGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQD
Ey9TdGFydENvbSBDbGFzcyAzIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQICALswgaYG
CyqGSIb3DQEJEAILMYGWoIGTMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRk
LjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMv
U3RhcnRDb20gQ2xhc3MgMyBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAgC7MA0GCSqG
SIb3DQEBAQUABIIBACgxT9PMrihevKKrxoyQ61CiCiEn2ArKWrN4zi7kmc2uwQSFkuitTjIC0mqu
gbCPj/648VZtHDPySQ10S0/CsLVJXh+06xkZoZopc6yiOOFEn83jn+Ymt5E+MsMjU0D1UbpNw05R
mkEELjfGyX9M8DW2jHDru7mJIXkA2s/gy32gCU9Sbp/kgUpl+5v0oYGZpGqgWwBUc83AdML2w3Fs
T1Z14zBglV3z5m0T2axpTXJcL/Be/8pOspL1l3moaf1DobUlM1mmsUQLC6Oz1n0ujOwuaKwndlW6
aPJdshi04+zo5TIdPxn7d0rSTjBKc2QtAGVz3EufQC1dL12UX8X/kRwAAAAAAAA=

--Apple-Mail-65462CE5-42F6-47A8-8F19-8E32853CCBD1--

From report at bugs.python.org  Fri Jan  6 18:38:32 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Fri, 06 Jan 2012 17:38:32 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
Message-ID: <1325871512.71.0.286774610121.issue13721@psf.upfronthosting.co.za>


New submission from Mads Kiilerich <mads at kiilerich.com>:

According to http://docs.python.org/release/2.7.2/library/ssl wrap_socket can be used either on connected sockets or on un-connected sockets which then can be .connect'ed.

In the latter case SSLSocket.connect() will (AFAIK) always raise a nice exception if the connection fails before the ssl negotiation has completed.

But when an existing connected but failing socket is wrapped then SSLSocket.__init__ will create an instance with self._sslobj = None without negotiating ssl and without raising any exception. Many SSLSocket methods (such as .cipher) checks for self._sslobj, but for example .getpeercert doesn't and will derefence None. That can lead to spurious crashes in applications.

This problem showed up with Mercurial and connections from China to code.google.com being blocked by the Chinese firewall - see for example https://bugzilla.redhat.com/show_bug.cgi?id=771691 .

In that case

  import socket, ssl, time
  s = socket.create_connection(('code.google.com', 443))
  time.sleep(1)
  ssl_sock = ssl.wrap_socket(s)
  ssl_sock.getpeercert(True)

would fail with
  ...
      ssl_sock.getpeercert(True)
    File "/usr/lib64/python2.7/ssl.py", line 172, in getpeercert
      return self._sslobj.peer_certificate(binary_form)
  AttributeError: 'NoneType' object has no attribute 'peer_certificate'

The problem occurs in the case where The Chinese Wall responds correctly to the SYN with SYN+ACK but soon after sends a RST. The sleep is necessary to reproduce it consistently; that makes sure the RST has been received and getpeername fails. Otherwise getpeername succeeds and the connection reset is only seen later on while negotiation ssl, and socket errors there are handled 'correctly'.

The problem can be reproduced on Linux with
  iptables -t filter -A FORWARD -p tcp --dport 443 ! --tcp-flags SYN SYN -j REJECT --reject-with tcp-reset

I would expect that wrap_socket / SSLSocket.__init__ raised an exception if the wrapped socket has been connected but failed. Calling getpeername is insufficient to detect that (and it is too racy to be reliable).

Alternatively all SSLSocket methods should take care not to dereference self._sslobj and they should respond properly - preferably with a socket/ssl exception.

Alternatively the documentation should describe how all applications that wraps connected sockets has to verify that it actually was connected. Checking .cipher() is apparently currently the least ugly way to do that?

One good(?) reason to wrap connected sockets is to be able to use socket.create_connection which tries all IP adresses of a fqdn before it fails. (Btw: That isn't described in the documentation! That confused me while debugging this.)

I guess applications (like Mercurial) that for that reason wants to use create_connection with 2.7.2 and older should check .cipher() as a workaround?

----------
components: Library (Lib)
messages: 150754
nosy: kiilerix
priority: normal
severity: normal
status: open
title: ssl.wrap_socket on a connected but failed connection succeeds and .peer_certificate gives AttributeError
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:52:01 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 17:52:01 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1325871512.71.0.286774610121.issue13721@psf.upfronthosting.co.za>
Message-ID: <1325872321.78.0.377841113146.issue13721@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Alternatively all SSLSocket methods should take care not to dereference 
> self._sslobj and they should respond properly - preferably with a
> socket/ssl exception.

In getpeercert()'s case, I think None would be the right thing to return (as cipher() and compression() already do).
But, yes, this deserves fixing.

> One good(?) reason to wrap connected sockets is to be able to use
> socket.create_connection which tries all IP adresses of a fqdn before
> it fails. (Btw: That isn't described in the documentation! That
> confused me while debugging this.)

Which documentation? create_connection's? Feel free to propose an improved wording.

----------
nosy: +pitrou
stage:  -> needs patch
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Fri Jan  6 18:59:40 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 06 Jan 2012 17:59:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F072943.8090603@egenix.com>
Message-ID: <4F073682.6050005@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Here's an example of hash-attack.patch finding an on-purpose
programming error (hashing all objects to the same value):

http://stackoverflow.com/questions/4865325/counting-collisions-in-a-python-dictionary
(see the second example on the page for @Winston Ewert's solution)

With the patch you get:

Traceback (most recent call last):
  File "testcollisons.py", line 20, in <module>
    d[o] = 1
KeyError: 'too many hash collisions'

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:08:37 2012
From: report at bugs.python.org (Benjamin West)
Date: Fri, 06 Jan 2012 18:08:37 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325873317.72.0.00507269429834.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Ok, Steven, that sounds reasonable.

I checked out git-svn python and started comparing diffs... I'm a little confused.  What version of argparse should be patched to provide this feature?

My HG version from https://code.google.com/p/argparse/ seems to contain a version of argparse 1.2 while, my git-svn checkout of python seems to contain an argparse 1.1.  Should I attempt to bring cpython's version up to date as well, or attempt to strip out the version bump changes?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:08:38 2012
From: report at bugs.python.org (Benjamin West)
Date: Fri, 06 Jan 2012 18:08:38 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325873318.71.0.539001779089.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Ok, Steven, that sounds reasonable.

I checked out git-svn python and started comparing diffs... I'm a little confused.  What version of argparse should be patched to provide this feature?

My HG version from https://code.google.com/p/argparse/ seems to contain a version of argparse 1.2 while, my git-svn checkout of python seems to contain an argparse 1.1.  Should I attempt to bring cpython's version up to date as well, or attempt to strip out the version bump changes?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:27:48 2012
From: report at bugs.python.org (Eric Snow)
Date: Fri, 06 Jan 2012 18:27:48 +0000
Subject: [issue13562] Notes about module load path
In-Reply-To: <1323397324.61.0.220189805252.issue13562@psf.upfronthosting.co.za>
Message-ID: <1325874468.57.0.328276313978.issue13562@psf.upfronthosting.co.za>


Changes by Eric Snow <ericsnowcurrently at gmail.com>:


----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13562>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:40:10 2012
From: report at bugs.python.org (Brian Curtin)
Date: Fri, 06 Jan 2012 18:40:10 +0000
Subject: [issue13719] bdist_msi upload fails
In-Reply-To: <1325855918.9.0.207720990162.issue13719@psf.upfronthosting.co.za>
Message-ID: <1325875210.31.0.751491770392.issue13719@psf.upfronthosting.co.za>


Changes by Brian Curtin <brian at python.org>:


----------
components: +Windows
nosy: +brian.curtin
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13719>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:55:09 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 18:55:09 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325876109.99.0.362089733965.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> Some remarks on the Windows implementation in termsize.diff.4:

> - On Windows, the C runtime always sets fileno(stdout) to 1, so
> hardcoded values are OK.
> But on Unix, I'm quite sure that embedded interpreters (mod_python?) 
> sometimes close the standard descriptor, so fd=1 can refer to 
> something entirely different.
> Does it makes sense to initialize fd=fileno(stdout) (this is C code) 
> instead?
OK, I agree that fd=fileno(stdout) is better. Now it matches the docstring which says "defaults to stdout" better.

> - When GetStdHandle() returns INVALID_HANDLE_VALUE, 
> PyErr_SetFromWindowsErr(0) should be used.
OK.
> And it's not necessary to use GetLastError(), 0 is enough.
OK, good no know.

> - GetStdHandle will return NULL in a non-console application (try 
> with pythonw.exe or IDLE), I think a specific error message should be 
> raised in this case.
OK. I tried and I see that the GetStdHandle documentation says so, but
I couldn't get it to return NULL. Nevertheless, I added a check:
+        if (handle == NULL)
+            return PyErr_Format(PyExc_OSError, "stdout not connected");

Thanks for the review!

Updated: termsize.diff.5

----------
Added file: http://bugs.python.org/file24157/termsize.diff.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:55:11 2012
From: report at bugs.python.org (Nam Nguyen)
Date: Fri, 06 Jan 2012 18:55:11 +0000
Subject: [issue13562] Notes about module load path
In-Reply-To: <1323397324.61.0.220189805252.issue13562@psf.upfronthosting.co.za>
Message-ID: <1325876111.51.0.368815795165.issue13562@psf.upfronthosting.co.za>


Nam Nguyen <bitsink at gmail.com> added the comment:

Please do keep the warning though. It is best to warn in both Python-C Py_Initialize() document and the Embedding tutorial.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13562>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:57:04 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Fri, 06 Jan 2012 18:57:04 +0000
Subject: [issue1521950] shlex.split() does not tokenize like the shell
Message-ID: <1325876224.24.0.86081172412.issue1521950@psf.upfronthosting.co.za>


Vinay Sajip <vinay_sajip at yahoo.co.uk> added the comment:

I've made a patch which implements this functionality, together with docs and tests. Please review.

----------
hgrepos: +99
nosy: +vinay.sajip
stage: test needed -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1521950>
_______________________________________

From report at bugs.python.org  Fri Jan  6 19:57:51 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Fri, 06 Jan 2012 18:57:51 +0000
Subject: [issue1521950] shlex.split() does not tokenize like the shell
Message-ID: <1325876271.14.0.751577514426.issue1521950@psf.upfronthosting.co.za>


Changes by Vinay Sajip <vinay_sajip at yahoo.co.uk>:


Added file: http://bugs.python.org/file24158/9e12275eec25.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1521950>
_______________________________________

From report at bugs.python.org  Fri Jan  6 20:16:59 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 06 Jan 2012 19:16:59 +0000
Subject: [issue13555] cPickle MemoryError when loading large file (while
	pickle works)
In-Reply-To: <CAH_1eM2LY6fncukVuO9r+LaZTpZPBnhE+aMfO5Ot8GrDRzR+sg@mail.gmail.com>
Message-ID: <CAH_1eM07eBxEZa0jPDt+CVPviSsW_hZzUNaE2Bup=TStXZHLJQ@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Antoine, could you test the last version (test_pickle and if possible
with the OP testcase)?
I can't test it myself (32-bit machine with 1 GB).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13555>
_______________________________________

From report at bugs.python.org  Fri Jan  6 20:17:45 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 06 Jan 2012 19:17:45 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <E1RjFIN-0005Ix-Ta@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1ea8b7233fd7 by Antoine Pitrou in branch 'default':
Issue #9993: When the source and destination are on different filesystems,
http://hg.python.org/cpython/rev/1ea8b7233fd7

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Fri Jan  6 20:18:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 19:18:32 +0000
Subject: [issue9993] shutil.move fails on symlink source
In-Reply-To: <1285801392.69.0.814963089991.issue9993@psf.upfronthosting.co.za>
Message-ID: <1325877512.26.0.972930792492.issue9993@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Patch now committed to 3.3. Thank you Jonathan and Hynek!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9993>
_______________________________________

From report at bugs.python.org  Fri Jan  6 20:33:29 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 19:33:29 +0000
Subject: [issue13555] cPickle MemoryError when loading large file (while
	pickle works)
In-Reply-To: <CAH_1eM07eBxEZa0jPDt+CVPviSsW_hZzUNaE2Bup=TStXZHLJQ@mail.gmail.com>
Message-ID: <1325878328.3341.14.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

Le vendredi 06 janvier 2012 ? 19:17 +0000, Charles-Fran?ois Natali a
?crit :
> Charles-Fran?ois Natali <neologix at free.fr> added the comment:
> 
> Antoine, could you test the last version (test_pickle and if possible
> with the OP testcase)?
> I can't test it myself (32-bit machine with 1 GB).

Well, first, there are compilation warnings on a 64-bit box:

gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/usr/local/include -I/home/antoine/cpython/27/Include -I/home/antoine/cpython/27 -c /home/antoine/cpython/27/Modules/cPickle.c -o build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o
/home/antoine/cpython/27/Modules/cPickle.c: In function ?put2?:
/home/antoine/cpython/27/Modules/cPickle.c:810:9: attention : format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t?
/home/antoine/cpython/27/Modules/cPickle.c: In function ?load_mark?:
/home/antoine/cpython/27/Modules/cPickle.c:4610:21: attention : assignment from incompatible pointer type
gcc -pthread -shared build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o -L/usr/local/lib -o build/lib.linux-x86_64-2.7-pydebug/cPickle.so

Second, I can't seem to get the test to work with 8GB RAM (approximately
6.5GB free according to "free"). The MemoryError is quite expectable for
test_pickle, though, since the code there doesn't try to conserve memory
at all:

test test_pickle failed -- Traceback (most recent call last):
  File "/home/antoine/cpython/27/Lib/test/test_support.py", line 983, in wrapper
    return f(self, maxsize)
  File "/home/antoine/cpython/27/Lib/test/pickletester.py", line 1298, in test_huge_str_32b
    pickled = self.dumps(data, proto)
  File "/home/antoine/cpython/27/Lib/test/test_pickle.py", line 74, in dumps
    return pickle.dumps(arg, proto)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/cpython/27/Lib/pickle.py", line 488, in save_string
    self.write(STRING + repr(obj) + '\n')
MemoryError

I would therefore suggest to enable the test only for cPickle.
For test_cpickle the behaviour is different:
- for protocol 0, I get a MemoryError (which may be expected, if the
test truly needs more than 6.5GB, for example because of suboptimal
buffer management)
- for protocol 1 and 2, I get "SystemError: error return without
exception set"

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13555>
_______________________________________

From report at bugs.python.org  Fri Jan  6 20:53:31 2012
From: report at bugs.python.org (Paul McMillan)
Date: Fri, 06 Jan 2012 19:53:31 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325818660.63.0.112419628547.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRXt71NjO_kjO086Uv508LpF_PGxJDDJgf_+1Q5sm_vc1w@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> Those who use or advocate a simple randomized starting hash (Perl, Ruby, perhaps MS, and the CCC presenters) are presuming that the randomized hash values are kept private. Indeed, they should be (and the docs could note this) unless an attacker has direct access to the interpreter.

Except that this is patently untrue. Anytime any programmer iterates
over a dictionary and returns the ordered result to the user in some
form, they're leaking information about the hash value. I hope you're
not suggesting that any programmer who is concerned about security
will make sure to sort the results of every iteration before making it
public in some fashion.

> I do not think we, as Python developers, should be concerned about esoteric timing attacks.

Timing attacks are less esoteric than you think they are. This issue
gets worse, not better, as the internet moves (for better or worse)
towards virtualized computing.

> And if hashing were randomized per process, and probes were randomly distributed among processes, and processes were periodically killed and restarted with new seeds, could such an attack get anywhere...

You're suggesting that in order for a Python application to be secure,
it's a requirement that we randomly kill and restart processes from
time to time? I thought we were going for a good solution here, not a
hacky workaround.

> We could also consider, for 3.3, making the output of hash() be different from the internal values used for dicts, perhaps by switching random seeds in hash(). So even if someone does return hash(x) values to potential attackers, they are not the values used in dicts. (This would require a slight change in the doc.)

This isn't a bad idea, but I'd be fine documenting that the output of
hash() shouldn't be made public.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 21:32:22 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 20:32:22 +0000
Subject: [issue13722] "distributions can disable the encodings package"
Message-ID: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

In _PyCodecRegistry_Init() (in Python/codecs.c), it is attempted to import the encodings module (so that the default search function gets registered) and failures get ignored following the same reasoning:

            /* Ignore ImportErrors... this is done so that
               distributions can disable the encodings package. Note
               that other errors are not masked, e.g. SystemErrors
               raised to inform the user of an error in the Python
               configuration are still reported back to the user. */

However, it is unlikely for Python 3 to start up at all without an encodings package (we needs some codecs to initialize stdio). Also, I'm not sure what the point would be. Distributions can simply distribute a dummy encodings module if they really want to disable the default encodings.
Not silencing the error would probably make it easier to diagnose early import issues.

----------
components: Interpreter Core
messages: 150767
nosy: amaury.forgeotdarc, loewis, pitrou
priority: low
severity: normal
status: open
title: "distributions can disable the encodings package"
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Fri Jan  6 21:48:08 2012
From: report at bugs.python.org (Pavel Labushev)
Date: Fri, 06 Jan 2012 20:48:08 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325882888.55.0.689088634411.issue13703@psf.upfronthosting.co.za>


Changes by Pavel Labushev <p.labushev at gmail.com>:


----------
nosy: +Arach

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 21:50:22 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 06 Jan 2012 20:50:22 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325883022.76.0.107819272479.issue13703@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

"You're suggesting that in order for a Python application to be secure,
it's a requirement that we randomly kill and restart processes from
time to time?"

No, that is not what I said.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 21:56:41 2012
From: report at bugs.python.org (Paul McMillan)
Date: Fri, 06 Jan 2012 20:56:41 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F06EDC3.7040600@egenix.com>
Message-ID: <CAO_YWRWerZ-cn_89wCdXu_oouiMTxRMphi0jgR-3GQE0cEGbUg@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> An attack can be based on trying to find many objects with the same
> hash value, or trying to find many objects that, as they get inserted
> into a dictionary, very often cause collisions due to the collision
> resolution algorithm not finding a free slot.

Yep. Allowing an attacker to produce very large dictionaries is also bad.

> if the application
> puts too much trust into large blobs of input data - which is
> the actual security issues we're trying to work around here...

To be very clear the issue is ANY large blob of data anywhere in the
application, not just on input. An attack could happen after whatever
transform your application runs on the data before returning it.

> I'll upload a patch that demonstrates the collisions counting
> strategy to show that detecting the problem is easy. Whether
> just raising an exception is a good idea, is another issue.

I'm in cautious agreement that collision counting is a better
strategy. The dict implementation performance would suffer from
randomization.

> The dict implementation could then alter the hash parameter
> and recreate the dict table in case the number of collisions
> exceeds a certain limit, thereby actively taking action
> instead of just relying on randomness solving the issue in
> most cases.

This is clever. You basically neuter the attack as you notice it but
everything else is business as usual. I'm concerned that this may end
up being costly in some edge cases (e.g. look up how many collisions
it takes to force the recreation, and then aim for just that many
collisions many times). Unfortunately, each dict object has to
discover for itself that it's full of offending hashes. Another
approach would be to neuter the offending object by changing its hash,
but this would require either returning multiple values, or fixing up
existing dictionaries, neither of which seems feasible.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 22:43:56 2012
From: report at bugs.python.org (Eric Promislow)
Date: Fri, 06 Jan 2012 21:43:56 +0000
Subject: [issue13723] Regular expressions: (?:X|\s+)*$ takes a long time
Message-ID: <1325886236.37.0.0463179039095.issue13723@psf.upfronthosting.co.za>


New submission from Eric Promislow <ericp at activestate.com>:

This regular expression takes a few seconds to be evaluated
against any text:

(.*?)((?:X|\s+)*)$

This reg ex is much faster:

(.*?)((?:X|\s)*)$

To be fair, Ruby's performance with the first regex is the same as Python's. PHP and JavaScript both fail to match the first regex
at all.  Only Perl evaluates both regexes nearly instantly.

----------
messages: 150770
nosy: ericp
priority: normal
severity: normal
status: open
title: Regular expressions: (?:X|\s+)*$ takes a long time
type: performance
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13723>
_______________________________________

From report at bugs.python.org  Fri Jan  6 22:46:33 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Fri, 06 Jan 2012 21:46:33 +0000
Subject: [issue13723] Regular expressions: (?:X|\s+)*$ takes a long time
In-Reply-To: <1325886236.37.0.0463179039095.issue13723@psf.upfronthosting.co.za>
Message-ID: <1325886393.81.0.675246751954.issue13723@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13723>
_______________________________________

From report at bugs.python.org  Fri Jan  6 22:53:35 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 06 Jan 2012 21:53:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325886815.25.0.739422477639.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I'm in cautious agreement that collision counting is a better
> strategy.

Disagreed. Raising randomly is unacceptable (false positives), especially in a bugfix release.

> The dict implementation performance would suffer from
> randomization.

Benchmarks please. http://hg.python.org/benchmarks/ for example.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 23:03:46 2012
From: report at bugs.python.org (Stefan Krah)
Date: Fri, 06 Jan 2012 22:03:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325887426.14.0.791207757465.issue13703@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan  6 23:13:50 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Fri, 06 Jan 2012 22:13:50 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
Message-ID: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>


New submission from Mads Kiilerich <mads at kiilerich.com>:

Forked from issue13721 where I was too lazy to report it separately:

http://docs.python.org/release/2.7.2/library/socket#socket.create_connection doesn't describe how it loops over all IP addresses. That seems to be the functions main advantage (and a gotcha) compared to creating the socket and connecting directly.

I propose to warn that it might "hang" up to n*timeout before anything happens with something like this in documentation and docstring:

... and return the socket object.
If the host resolves to multiple IP addresses then they will all be tried in turn until one of them succeeds.
Passing the optional timeout parameter will set the timeout on the socket instance before each attempt to connect.
If no ...

----------
assignee: docs at python
components: Documentation
messages: 150772
nosy: docs at python, kiilerix, pitrou
priority: normal
severity: normal
status: open
title: socket.create_connection and multiple IP addresses
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Fri Jan  6 23:14:35 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Fri, 06 Jan 2012 22:14:35 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1325871512.71.0.286774610121.issue13721@psf.upfronthosting.co.za>
Message-ID: <1325888075.0.0.00318737341897.issue13721@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

I think it would be confusing if getpeercert returned None both for valid connections without certificates and also for invalid connections. I would almost prefer the current behaviour (AttributeError) if just it was documented and there was a documented way to check if the connection actually was alive. Do you agree that checking .cipher() is the recommended way to do that in a way that is compatible with past and future 2.x versions?

I hope the proper fix will ensure that an exception always is raised if the ssl handshake fails - and that a successful wrap_socket means that the ssl negotiation did succeed with the given constraints. It might however only be feasible to fix that for 3.x.

I filed Issue13724 for the create_socket documentation.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Fri Jan  6 23:16:18 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 06 Jan 2012 22:16:18 +0000
Subject: [issue13511] Specifying multiple lib and include directories on linux
In-Reply-To: <1322683661.17.0.138081414237.issue13511@psf.upfronthosting.co.za>
Message-ID: <1325888178.93.0.601454512236.issue13511@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

No, I only read the original message. I find it too tedious to read the entire set of messages just to find out what the issue may be. If an issue has sufficiently deviated from the original report, it is better closed, and a new issue is opened.

Even after scanning the messages, I still cannot figure out what the issue is. Your last message is particularly mysterious to me.

So if you think there still is an issue, please report a new one. Make sure to include reproducable test cases, and a clear statement of what you expected to happen (in addition to reporting what actually happened).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13511>
_______________________________________

From report at bugs.python.org  Fri Jan  6 23:44:31 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 06 Jan 2012 22:44:31 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1325853017.71.0.936491533882.issue13609@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwb+sajBWpKVe4wpSD0ahJgu=b60UOJ7_=VRhPCixYQP-Q@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> - fd argument is retained, because we might want to test terminals
> ?opened with openpty.

You mean a terminal different than the one used for stdin, stdout and stderr?

> - two functions: still there. I think that get_terminal_size() should
> ?provide an easy-to-use, even trivial-to-use, way to get a sensible
> ?value without writing a wrapper. In the _majority_ of cases the
> ?wrapper would be something like get_terminal_size() currently.

get_terminal_size() looks like reliable than raw_get_terminal_size()
because environment variables are not updated when the terminal is
resized.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan  7 00:07:01 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 06 Jan 2012 23:07:01 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325891221.45.0.805076589081.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

>> - fd argument is retained, because we might want to test terminals
>>  opened with openpty.
>
> You mean a terminal different than the one used for stdin, stdout and 
> stderr?

For example, let's see what is the size of my two xterms:

>>> os.get_terminal_size_raw()
os.terminal_size(columns=125, rows=39)
>>> fd = os.open('/proc/22736/fd/1', os.O_RDONLY)
>>> fd
6
>>> os.get_terminal_size_raw(6)
os.terminal_size(columns=95, rows=33)

I'm not saying that this serves a clear purpose, but it is possible
and it is good not the restrain the API. I'm sure somebody could find
a use for it.

>> - two functions: still there. I think that get_terminal_size() should
>>  provide an easy-to-use, even trivial-to-use, way to get a sensible
>>  value without writing a wrapper. In the _majority_ of cases the
>>  wrapper would be something like get_terminal_size() currently.
>
> get_terminal_size() looks like [less?] reliable than raw_get_terminal_size()
> because environment variables are not updated when the terminal is
> resized.
Please look at my comment above: http://bugs.python.org/issue13609#msg149620
$COLUMNS is only used for overriding -- it normally _isn't_ set.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan  7 00:23:39 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Fri, 06 Jan 2012 23:23:39 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <1325892219.47.0.345876767403.issue13722@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Agreed.  This behavior probably comes from the times when unicode was an optional feature.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:05:11 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 00:05:11 +0000
Subject: [issue13685] argparse does not sanitize help strings for % signs
In-Reply-To: <1325284167.16.0.584942168313.issue13685@psf.upfronthosting.co.za>
Message-ID: <1325894711.92.0.452226022923.issue13685@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +bethard

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13685>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:08:50 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 00:08:50 +0000
Subject: [issue13686] Some notes on the docs of multiprocessing
In-Reply-To: <1325307613.69.0.259805833126.issue13686@psf.upfronthosting.co.za>
Message-ID: <1325894930.0.0.972424900095.issue13686@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +jnoller

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13686>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:22:01 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 00:22:01 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1325895721.19.0.959705773375.issue13691@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I agree. It should explain the three options:
help(object): help on object or class of object, except
help('name'): help on object/module named 'name'
help(): run utility, which starts with utility help

help(help) prints unhelpful
"Help on _Helper in module site object:

class _Helper(builtins.object)
 |  Define the builtin 'help'.
 |  This is a wrapper around pydoc.help (with a twist).
 |  
 |  Methods defined here:
...
"
which says not at all how to use help().

I have the feeling that there is a related issue on the tracker, but searching for 'help', 'help()', or 'help(help)' all return 620 hits.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:23:47 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 00:23:47 +0000
Subject: [issue13692] 2to3 mangles from . import frobnitz
In-Reply-To: <1325419411.86.0.637139477325.issue13692@psf.upfronthosting.co.za>
Message-ID: <1325895827.54.0.778657414464.issue13692@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

If this is fixed in 3.2, can this be closed?

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13692>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:26:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 07 Jan 2012 00:26:53 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1325888075.0.0.00318737341897.issue13721@psf.upfronthosting.co.za>
Message-ID: <1325895934.3341.16.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I hope the proper fix will ensure that an exception always is raised
> if the ssl handshake fails - and that a successful wrap_socket means
> that the ssl negotiation did succeed with the given constraints. It
> might however only be feasible to fix that for 3.x.

Actually, the handshake is not even attempted because getpeername()
returns ENOTCONN, which is interpreted as meaning "the socket hasn't
been connected yet". Any idea how to improve that?

(I find it strange that ENOTCONN is returned rather than say ECONNRESET)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:28:31 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 07 Jan 2012 00:28:31 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
In-Reply-To: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>
Message-ID: <1325896033.3341.18.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> http://docs.python.org/release/2.7.2/library/socket#socket.create_connection doesn't describe how it loops over all IP addresses. That seems to be the functions main advantage (and a gotcha) compared to creating the socket and connecting directly.

Well the main advantage is that it makes it much easier to write clients
supporting both IPv4 and IPv6.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Sat Jan  7 01:57:30 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 00:57:30 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
In-Reply-To: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>
Message-ID: <1325897850.79.0.49155697002.issue13720@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The code works fine on 3.2.2, Win7, IDLE, narrowest window possible (about 14 chars), which actually wraps to the window width. (In command window, lines are fixed length and scroll bar is added if window is narrowed.) What system and version are you running? 

The error directly comes from textwrap. In the other hand, textwrap.wrap works with widths down to 1 (on 3.2.2), which suggests that argparse is calling it wrong. Except that it is not on my system. 

Could you add 'print(width)' before the call to textwrap
    return _textwrap.wrap(text, width)
to see if -1 is being passed?

----------
nosy: +bethard, georg.brandl, terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Sat Jan  7 03:17:18 2012
From: report at bugs.python.org (Devin Jeanpierre)
Date: Sat, 07 Jan 2012 02:17:18 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1325902638.0.0.159576945123.issue13691@psf.upfronthosting.co.za>


Devin Jeanpierre <jeanpierreda at gmail.com> added the comment:

> IMO, help('help') should document the help function, not start an interactive help session (that?d be help()).

Ahhh, that explains it. help('help') isn't ever meant to be called; it's supposed to be:

>>> help()
...
help> help
...

the call to "help" at the help> prompt is equivalent to help('help').


help('help') should do something different, such as say how to use the help function, as you say. Whereas "help> help" makes sense.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Sat Jan  7 04:47:47 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 03:47:47 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325908067.57.0.0428089777491.issue9253@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

You should work in the 3.3 standard library, i.e. on Lib/argparse.py in the default branch of the CPython Mercurial repository.  See the devguide for more info.  Thanks!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Sat Jan  7 04:52:34 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sat, 07 Jan 2012 03:52:34 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1325908354.21.0.187120799869.issue13691@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
stage:  -> needs patch
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Sat Jan  7 09:50:30 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Sat, 07 Jan 2012 08:50:30 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
In-Reply-To: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>
Message-ID: <1325926230.36.0.475345768198.issue13720@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> What system and version are you running? 
Linux (debian amd64), Python is compiled from hg (1ea8b7233fd7).

> The error directly comes from textwrap. In the other hand, 
> textwrap.wrap works with widths down to 1 (on 3.2.2), which suggests 
> that argparse is calling it wrong. Except that it is not on my system. 
That's really surprising, because it is all pure Python code and don't
really see how it _could_ be right: take $COLUMNS, subtract, subtract,
and sooner or later _width will go below 0.

> Could you add 'print(width)' before the call to textwrap
>     return _textwrap.wrap(text, width)
> to see if -1 is being passed?
Prints -1.

> The code works fine on 3.2.2, Win7, IDLE, narrowest window possible (about 14 
> chars), which actually wraps to the window width. 
Oh, I just tried it in IDLE and it prints:
  64
  64
  64
  64
  usage: ...
in a very small window (30 cells wide). So IDLE is just doesn't allow you to
go below a certain size.

See also #13107.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Sat Jan  7 10:14:29 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 09:14:29 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
In-Reply-To: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>
Message-ID: <1325927669.1.0.136557162965.issue13720@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I am not setting columns, so that might be the important difference.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Sat Jan  7 10:34:23 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Sat, 07 Jan 2012 09:34:23 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
In-Reply-To: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>
Message-ID: <1325928863.14.0.02604240404.issue13720@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> I am not setting columns, so that might be the important difference.
Yeah, the whole example with IDLE is moot: argparse only checks
$COLUMNS and defaults to 80, so if COLUMNS is not set, you are only
checking if the code works with 80 columns.

Please try my commandline example or set os.environ['COLUMNS'] = 15.
I get the exception in IDLE too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Sat Jan  7 11:52:34 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Sat, 07 Jan 2012 10:52:34 +0000
Subject: [issue13511] Specifying multiple lib and include directories on linux
In-Reply-To: <1322683661.17.0.138081414237.issue13511@psf.upfronthosting.co.za>
Message-ID: <1325933554.73.0.0307543541216.issue13511@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

He's probably referring to this fragment in setup.py:


        if platform == 'darwin':
            # This should work on any unixy platform ;-)
            # If the user has bothered specifying additional -I and -L flags
            # in OPT and LDFLAGS we might as well use them here.
            #   NOTE: using shlex.split would technically be more correct, but
            # also gives a bootstrap problem. Let's hope nobody uses directories
            # with whitespace in the name to store libraries.
            cflags, ldflags = sysconfig.get_config_vars(
                    'CFLAGS', 'LDFLAGS')
            for item in cflags.split():
                if item.startswith('-I'):
                    inc_dirs.append(item[2:])
            for item in ldflags.split():
                if item.startswith('-L'):
                    lib_dirs.append(item[2:])


The entire block was added when I worked on the Mac port with Jack Jansen.  It ensures that additional include and lib directories specified through the 'OPT' and 'LDFLAGS' shell variables get added to the search path used by distutils.

As the comment says this should be safe all unity platforms, not just OSX.  IIRC the platform test is there because we couldn't easily test on other platforms and didn't want to accidentally break stuff (this was before we had a good buildbot setup).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13511>
_______________________________________

From report at bugs.python.org  Sat Jan  7 12:31:19 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Sat, 07 Jan 2012 11:31:19 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
In-Reply-To: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>
Message-ID: <1325935879.49.0.265199435171.issue13724@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Sat Jan  7 12:38:08 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 07 Jan 2012 11:38:08 +0000
Subject: [issue13720] argparse print_help() fails if COLUMNS is set to a low
	value
In-Reply-To: <1325858148.7.0.947093232457.issue13720@psf.upfronthosting.co.za>
Message-ID: <1325936288.19.0.497309102415.issue13720@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Now I get error, even with Window actually about 100 columns wide.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13720>
_______________________________________

From report at bugs.python.org  Sat Jan  7 12:51:48 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 07 Jan 2012 11:51:48 +0000
Subject: [issue13725] regrtest does not recognize -d flag
Message-ID: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>


New submission from Erno Tukia <erno.tukia at iki.fi>:

./python -m test --help
-d/--debug      -- print traceback for failed tests

./python -m test -d?test_imaplib
option -d not recognized
Use --help for usage

Patch included.

----------
components: Tests
files: regrtest-debug.patch
keywords: patch
messages: 150790
nosy: etukia
priority: normal
severity: normal
status: open
title: regrtest does not recognize -d flag
type: behavior
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24159/regrtest-debug.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan  7 13:12:38 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 07 Jan 2012 12:12:38 +0000
Subject: [issue13033] Add shutil.chowntree
In-Reply-To: <1316769849.87.0.0298236424523.issue13033@psf.upfronthosting.co.za>
Message-ID: <1325938358.16.0.531249909328.issue13033@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I believe the current "check_chown" could be passed by a no-op (since the file will be owned by the current user even *before* the call to chowntree). Testing this properly is actually rather difficult (since the only uid and gid we can rely on are those of the current process).

More significantly, I don't agree with the proposed error handling (i.e. attempting to roll back to the original state). Instead, I think it would be more appropriate to follow the rmtree ignore_errors/onerror style so that uses can either unconditionally ignore errors (including dir listing errors) or else tailor the error handling themselves. Any custom error handling should also cover the actual chown operation, not just directory listing errors inside os.walk.

I think, like walkdir itself, there's enough under the hood here that the idea requires some baking time outside the standard library. How do you feel about migrating this discussion over to the walkdir issue tracker as a higher level API proposal there? (https://bitbucket.org/ncoghlan/walkdir/issues).

I had a couple of other minor comments, although they're largely irrelevant given the more significant comments above:

There's a gratuitous inconsistency in the type-checking for uid/gid (one uses "isinstance(uid, str)", the other "not isinstance(gid, int)". Neither is a particular good check for the "None, integer or string" case anyway. It would be better to just try the following in order:

- "is None"
- operator.index
- _get_uid/gid (as appropriate)

The dict initialisation and error handler definition may as well move inside the if statement.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13033>
_______________________________________

From report at bugs.python.org  Sat Jan  7 13:14:48 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 07 Jan 2012 12:14:48 +0000
Subject: [issue13726] regrtest ambiguous -S flag
Message-ID: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>


New submission from Erno Tukia <erno.tukia at iki.fi>:

./python -m test --help
-S/--slow       -- print the slowest 10 tests

-S is used to continue running tests after an aborted run.  It will
maintain the order a standard run (ie, this assumes -r is not used).
This is useful after the tests have prematurely stopped for some external
reason and you want to start running from where you left off rather
than starting from the beginning.

in Lib/test/regrtest.py

opts, args = getopt.getopt(sys.argv[1:], '...S...',
            [..., 'slow', ... , 'start=', ...])

for o, a in opts:
    elif o in ('-S', '--start'):
        start = a
    elif o in ('-S', '--slow'):
        print_slow = True

At the moment -S (no args) and --slow (no args) are the same, not what the documentation says and not how the code executes (-S goes with --start). Help says nothing about --start.

--slow or --start needs a new short opt, and corrected documentation.

----------
assignee: docs at python
components: Documentation, Tests
messages: 150792
nosy: docs at python, etukia
priority: normal
severity: normal
status: open
title: regrtest ambiguous -S flag
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan  7 13:57:39 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sat, 07 Jan 2012 12:57:39 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325941059.9.0.853319373491.issue13716@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

I would make comments out of the XXX, and if a whole section is just that XXX, remove the section as well.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Sat Jan  7 14:13:07 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sat, 07 Jan 2012 13:13:07 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1228232522.58.0.0992151848245.issue4489@psf.upfronthosting.co.za>
Message-ID: <1325941987.22.0.377706666189.issue4489@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

What's the current state here? Anyone working on a solution or are we waiting how http://hg.python.org/features/pathlib/ will work out?

If the consensus is to add a generic walker method, wouldn't be appropriate to open a new bug and add it as dependency? Or is there one I've missed?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Sat Jan  7 14:17:34 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Sat, 07 Jan 2012 13:17:34 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAO_YWRWerZ-cn_89wCdXu_oouiMTxRMphi0jgR-3GQE0cEGbUg@mail.gmail.com>
Message-ID: <4F0845E2.5040304@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Paul McMillan wrote:
> 
>> I'll upload a patch that demonstrates the collisions counting
>> strategy to show that detecting the problem is easy. Whether
>> just raising an exception is a good idea, is another issue.
> 
> I'm in cautious agreement that collision counting is a better
> strategy. The dict implementation performance would suffer from
> randomization.
> 
>> The dict implementation could then alter the hash parameter
>> and recreate the dict table in case the number of collisions
>> exceeds a certain limit, thereby actively taking action
>> instead of just relying on randomness solving the issue in
>> most cases.
> 
> This is clever. You basically neuter the attack as you notice it but
> everything else is business as usual. I'm concerned that this may end
> up being costly in some edge cases (e.g. look up how many collisions
> it takes to force the recreation, and then aim for just that many
> collisions many times). Unfortunately, each dict object has to
> discover for itself that it's full of offending hashes. Another
> approach would be to neuter the offending object by changing its hash,
> but this would require either returning multiple values, or fixing up
> existing dictionaries, neither of which seems feasible.

I ran some experiments with the collision counting patch and
could not trigger it in normal applications, not even in cases
that are documented in the dict implementation to have a poor
collision resolution behavior (integers with zeros the the low bits).
The probability of having to deal with dictionaries that create
over a thousand collisions for one of the key objects in a
real life application appears to be very very low.

Still, it may cause problems with existing applications for the
Python dot releases, so it's probably safer to add it in a
disabled-per-default form there (using an environment variable
to adjust the setting). For 3.3 it could be enabled per default
and it would also make sense to allow customizing the limit
using a sys module setting.

The idea with adding a parameter to the hash method/slot in order
to have objects provide a hash family function instead of a fixed
unparametrized hash function would probably have to be implemented
as additional hash method, e.g. .__uhash__() and tp_uhash ("u"
for universal).

The builtin types should then grow such methods
in order to make hashing safe against such attacks. For objects
defined in 3rd party extensions, we would need to encourage
implementing the slot/method as well. If it's not implemented,
the dict implementation would have to fallback to raising an
exception.

Please note that I'm just sketching things here. I don't have
time to work on a full-blown patch, just wanted to show what
I meant with the collision counting idea and demonstrate that
it actually works as intended.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan  7 14:50:36 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Sat, 07 Jan 2012 13:50:36 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
Message-ID: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>


New submission from Amaury Forgeot d'Arc <amauryfa at gmail.com>:

All objects of the datetime module have macros to access their properties, except timedelta.
This simple patch adds the macros PyDateTime_DELTA_GET_DAYS, PyDateTime_DELTA_GET_SECONDS, PyDateTime_DELTA_GET_MICROSECONDS; module developers are encouraged to use these instead of obj->seconds for example.

Motivation:
PyPy cannot easily expose PyDateTime_Delta fields because datetime is implemented as a pure Python module and it's difficult to rebuild a C structure from a heap type.
In PyPy these macros are actually functions, which do something similar to PyLong_AsLong(PyObject_GetAttrString(obj, "seconds")):
https://bitbucket.org/pypy/pypy/src/b67e65d709e1/pypy/module/cpyext/cdatetime.py#cl-235

----------
files: timedelta_macros.patch
keywords: patch
messages: 150796
nosy: amaury.forgeotdarc
priority: normal
severity: normal
stage: patch review
status: open
title: Accessor macros for PyDateTime_Delta members
versions: Python 3.3
Added file: http://bugs.python.org/file24160/timedelta_macros.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Sat Jan  7 15:42:31 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 14:42:31 +0000
Subject: [issue13692] 2to3 mangles from . import frobnitz
In-Reply-To: <1325419411.86.0.637139477325.issue13692@psf.upfronthosting.co.za>
Message-ID: <1325947351.29.0.788620376177.issue13692@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the report, but 3.1 does not get bug fixes any more.  Maybe Benjamin would like to add a regression test for this, so I?m leaving the decision to close this report to him.

----------
nosy: +benjamin.peterson, eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13692>
_______________________________________

From report at bugs.python.org  Sat Jan  7 15:47:06 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 14:47:06 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325947626.08.0.26602034733.issue13716@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

These are the XXX:

  XXX true? does ANSI say anything about this?
  XXX defaults to what?
  XXX see also.
  XXX see also.
  XXX see also.
  XXX see also.
  XXX see also.

The first two are questions that can be answered, and when I research them for the packaging docs I could also backport the changes to the distutils docs; what to do for the other instances is less clear, so just deleting may be better than turning them into comments that will never get solved.

----------
assignee: docs at python -> eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Sat Jan  7 15:48:21 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 14:48:21 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1325947701.26.0.241083419964.issue13691@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I?m interested in this, so I?m taking assignment.  If a contributor is interested in making a patch I?ll review it, otherwise I?ll say when I start on a patch.

----------
assignee:  -> eric.araujo
keywords: +easy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Sat Jan  7 15:51:36 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sat, 07 Jan 2012 14:51:36 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325947896.77.0.585739886067.issue13716@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Agreed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Sat Jan  7 16:06:50 2012
From: report at bugs.python.org (Florent Xicluna)
Date: Sat, 07 Jan 2012 15:06:50 +0000
Subject: [issue13716] distutils doc contains lots of XXX
In-Reply-To: <1325790136.17.0.881224521536.issue13716@psf.upfronthosting.co.za>
Message-ID: <1325948810.43.0.964659516993.issue13716@psf.upfronthosting.co.za>


Florent Xicluna <florent.xicluna at gmail.com> added the comment:

The proposed solution is ok.

It sounds like unfinished documentation when you hit an "XXX".
For the "/dev/" documentation, it's OK, but for the released version, we should avoid it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13716>
_______________________________________

From report at bugs.python.org  Sat Jan  7 16:36:53 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sat, 07 Jan 2012 15:36:53 +0000
Subject: [issue13728] Description of -m and -c cli options wrong?
Message-ID: <1325950613.83.0.912226487635.issue13728@psf.upfronthosting.co.za>


New submission from Sandro Tosi <sandro.tosi at gmail.com>:

Hi,
looking at http://mail.python.org/pipermail/docs/2011-December/006672.html I tried to replicate what the user said, and *it seems* that -c and -m doesn't add the current directory to the start of sys.path:

$ echo -e "import sys\nprint (sys.path)" > dummy.py ; ./python -mdummy
['', '/usr/local/lib/python27.zip', '/home/morph/cpython/py27/Lib', '/home/morph/cpython/py27/Lib/plat-linux2', '/home/morph/cpython/py27/Lib/lib-tk', '/home/morph/cpython/py27/Lib/lib-old', '/home/morph/cpython/py27/build/lib.linux-x86_64-2.7-pydebug', '/usr/local/lib/python2.7/site-packages']

and 

$ ./python -c "import dummy"
['', '/usr/local/lib/python27.zip', '/home/morph/cpython/py27/Lib', '/home/morph/cpython/py27/Lib/plat-linux2', '/home/morph/cpython/py27/Lib/lib-tk', '/home/morph/cpython/py27/Lib/lib-old', '/home/morph/cpython/py27/build/lib.linux-x86_64-2.7-pydebug', '/usr/local/lib/python2.7/site-packages']

the same sys.path as from the interactive shell:

$ ./python 
Python 2.7.2+ (2.7:f0666e56a552, Jan  7 2012, 16:31:06) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
[38650 refs]
>>> sys.path
['', '/usr/local/lib/python27.zip', '/home/morph/cpython/py27/Lib', '/home/morph/cpython/py27/Lib/plat-linux2', '/home/morph/cpython/py27/Lib/lib-tk', '/home/morph/cpython/py27/Lib/lib-old', '/home/morph/cpython/py27/build/lib.linux-x86_64-2.7-pydebug', '/usr/local/lib/python2.7/site-packages']
[38657 refs]

So, is http://docs.python.org/using/cmdline.html#interface-options correct in saying "he current directory will be added to the start of sys.path (allowing modules in that directory to be imported as top level modules)."? (and so forth for -m option?) am I confused?

----------
assignee: docs at python
components: Documentation
messages: 150802
nosy: docs at python, sandro.tosi
priority: normal
severity: normal
stage: needs patch
status: open
title: Description of -m and -c cli options wrong?
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13728>
_______________________________________

From report at bugs.python.org  Sat Jan  7 16:40:49 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 15:40:49 +0000
Subject: [issue12989] Consistently handle path separator in Py_GetPath on
	Windows
In-Reply-To: <1316131696.55.0.834479609561.issue12989@psf.upfronthosting.co.za>
Message-ID: <1325950849.59.0.177899043577.issue12989@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12989>
_______________________________________

From report at bugs.python.org  Sat Jan  7 16:40:51 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sat, 07 Jan 2012 15:40:51 +0000
Subject: [issue13692] 2to3 mangles from . import frobnitz
In-Reply-To: <1325419411.86.0.637139477325.issue13692@psf.upfronthosting.co.za>
Message-ID: <1325950851.92.0.498310355901.issue13692@psf.upfronthosting.co.za>


Changes by Benjamin Peterson <benjamin at python.org>:


----------
resolution:  -> out of date
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13692>
_______________________________________

From report at bugs.python.org  Sat Jan  7 16:55:46 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 07 Jan 2012 15:55:46 +0000
Subject: [issue13550] Rewrite logging hack of the threading module
In-Reply-To: <1323297626.55.0.00727033353184.issue13550@psf.upfronthosting.co.za>
Message-ID: <1325951746.8.0.444281864633.issue13550@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Alright, Nick agreed on python-dev to remove the logging hack.

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13550>
_______________________________________

From report at bugs.python.org  Sat Jan  7 17:05:32 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sat, 07 Jan 2012 16:05:32 +0000
Subject: [issue13729] Evaluation order for dics key/value
Message-ID: <1325952332.37.0.223922022645.issue13729@psf.upfronthosting.co.za>


New submission from Sandro Tosi <sandro.tosi at gmail.com>:

Hello, following up http://mail.python.org/pipermail/docs/2011-December/006782.html , I'm opening this issue to have others confirmation of what's happening:

echo -e "def ev(r):\n    print(r)\n    return r\n\n{ev(1): ev(2), ev(3): ev(4)}" | ./python 
2
1
4
3

(this applies for 2.7/3.2/default) so it seems the value expressions are evaluated before the key expressions, so the

{expr1: expr2, expr3: expr4}

in http://docs.python.org/reference/expressions.html#evaluation-order has to be updated in

{expr2: expr1, expr4: expr3}

Is that correct/can someone confirm this is the expected behavior?

----------
assignee: docs at python
components: Documentation
messages: 150804
nosy: docs at python, sandro.tosi
priority: normal
severity: normal
stage: needs patch
status: open
title: Evaluation order for dics key/value
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13729>
_______________________________________

From report at bugs.python.org  Sat Jan  7 17:09:18 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 16:09:18 +0000
Subject: [issue13550] Rewrite logging hack of the threading module
In-Reply-To: <1323297626.55.0.00727033353184.issue13550@psf.upfronthosting.co.za>
Message-ID: <1325952558.03.0.0679575502002.issue13550@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

haypo?s threading_note_global looks good to me.  The only thing I?m not sure about is the signature change from X(verbose, *args, **kwargs) to X(*args, **kwargs): is it okay?

(BTW you probably want to delete the obsolete reference to ihooks in a comment before committing.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13550>
_______________________________________

From report at bugs.python.org  Sat Jan  7 17:20:12 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 07 Jan 2012 16:20:12 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1325953212.19.0.577281094492.issue13609@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan  7 17:57:51 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 16:57:51 +0000
Subject: [issue13666] datetime documentation typos
In-Reply-To: <1324936913.94.0.649454454071.issue13666@psf.upfronthosting.co.za>
Message-ID: <1325955471.28.0.0936271571621.issue13666@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

This patch fixes the rzinfo typo as well as the GMT2 issue (GMT +2 should behave exactly the same as GMT +1 with regards to DST, it's base offset should simply be +2 hours instead of +1).

This does not; however, address the comment about the first line of the tzinfo.utcoffset(). The fact that tzinfo.utcoffset() should return a timedelta or None is addressed later in the same paragraph, as such I'm not sure the proposed change is an improvement.

----------
keywords: +patch
nosy: +zacherates
Added file: http://bugs.python.org/file24161/issue13666.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13666>
_______________________________________

From report at bugs.python.org  Sat Jan  7 18:15:09 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 17:15:09 +0000
Subject: [issue13666] datetime documentation typos
In-Reply-To: <1324936913.94.0.649454454071.issue13666@psf.upfronthosting.co.za>
Message-ID: <1325956509.43.0.618394244929.issue13666@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

Looks like the issue of the first line of utcoffsect was also raised in issue 8810.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13666>
_______________________________________

From report at bugs.python.org  Sat Jan  7 18:26:39 2012
From: report at bugs.python.org (Stephen Kelly)
Date: Sat, 07 Jan 2012 17:26:39 +0000
Subject: [issue13666] datetime documentation typos
In-Reply-To: <1324936913.94.0.649454454071.issue13666@psf.upfronthosting.co.za>
Message-ID: <1325957199.78.0.346936826069.issue13666@psf.upfronthosting.co.za>


Stephen Kelly <steveire at gmail.com> added the comment:

Patch looks good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13666>
_______________________________________

From report at bugs.python.org  Sat Jan  7 18:27:07 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 07 Jan 2012 17:27:07 +0000
Subject: [issue13502] Documentation for Event.wait return value is either
	wrong or incomplete
In-Reply-To: <1322599516.14.0.0239672713193.issue13502@psf.upfronthosting.co.za>
Message-ID: <E1Rja2r-0006HZ-TS@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset eb39d862a250 by Charles-Fran?ois Natali in branch '3.2':
Issue #13502: threading: Fix a race condition in Event.wait() that made it
http://hg.python.org/cpython/rev/eb39d862a250

New changeset 0fe63bb20e74 by Charles-Fran?ois Natali in branch 'default':
Issue #13502: threading: Fix a race condition in Event.wait() that made it
http://hg.python.org/cpython/rev/0fe63bb20e74

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13502>
_______________________________________

From report at bugs.python.org  Sat Jan  7 18:56:23 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 07 Jan 2012 17:56:23 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1325941987.22.0.377706666189.issue4489@psf.upfronthosting.co.za>
Message-ID: <1325958902.3435.1.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> What's the current state here? Anyone working on a solution or are we
> waiting how http://hg.python.org/features/pathlib/ will work out?

Well, I am not working on that one, so waiting for it to work out might
be optimistic :)
I don't know what to do with it (the pathlib): is such a feature
desireable enough?

> If the consensus is to add a generic walker method, wouldn't be
> appropriate to open a new bug and add it as dependency?

Agreed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Sat Jan  7 19:44:11 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 18:44:11 +0000
Subject: [issue12005] modulo result of Decimal differs from float/int
In-Reply-To: <1304584519.64.0.99823949629.issue12005@psf.upfronthosting.co.za>
Message-ID: <1325961851.47.0.660888855659.issue12005@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

Here is a patch that adds an explination for the difference in the behaviour to the FAQ section of the Decimal documentation.

----------
keywords: +patch
nosy: +zacherates
Added file: http://bugs.python.org/file24162/issue12005.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12005>
_______________________________________

From report at bugs.python.org  Sat Jan  7 19:49:24 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 07 Jan 2012 18:49:24 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1325855998.02.0.814986313066.issue8184@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2Dyn5BUEZmnHX+1p7x=B73cF9LNGbtjz_i+0aVV3Hrow@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I noticed that if bind() fails (in this case with EADDRINUSE), the
socket isn't closed (FD leak).
Here's a patch.

----------
keywords: +patch
Added file: http://bugs.python.org/file24163/connection_error.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________
-------------- next part --------------
diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -575,10 +575,14 @@
     '''
     def __init__(self, address, family, backlog=1):
         self._socket = socket.socket(getattr(socket, family))
-        self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-        self._socket.bind(address)
-        self._socket.listen(backlog)
-        self._address = self._socket.getsockname()
+        try:
+            self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            self._socket.bind(address)
+            self._socket.listen(backlog)
+            self._address = self._socket.getsockname()
+        except OSError:
+            self._socket.close()
+            raise
         self._family = family
         self._last_accepted = None
 

From report at bugs.python.org  Sat Jan  7 20:09:30 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 19:09:30 +0000
Subject: [issue13730] Grammar mistake in Decimal documentation
Message-ID: <1325963370.31.0.599534414571.issue13730@psf.upfronthosting.co.za>


New submission from Aaron Maenpaa <aaron at maenpaa.ca>:

In the sentance: "In contrast, numbers like 1.1 and 2.2 do not have an exact representations in binary floating point." there is a mismatch in number between "an" and "representations". I suggest removing "an" to make the whole thing plural.

A patch is attached.

----------
assignee: docs at python
components: Documentation
files: plural.diff
keywords: patch
messages: 150813
nosy: docs at python, zacherates
priority: normal
severity: normal
status: open
title: Grammar mistake in Decimal documentation
versions: Python 3.3
Added file: http://bugs.python.org/file24164/plural.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13730>
_______________________________________

From report at bugs.python.org  Sat Jan  7 20:16:46 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 19:16:46 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
Message-ID: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>


New submission from Aaron Maenpaa <aaron at maenpaa.ca>:

The paragraph: "The exactness carries over into arithmetic. In decimal floating point, 0.1 + 0.1 + 0.1 - 0.3 is exactly equal to zero. In binary floating point, the result is 5.5511151231257827e-017. While near to zero, the differences prevent reliable equality testing and differences can accumulate. For this reason, decimal is preferred in accounting applications which have strict equality invariants."

... has some awkward phrasing to my ear. I've attached a patch with a proposed alternative.

----------
assignee: docs at python
components: Documentation
files: rephrase.diff
keywords: patch
messages: 150814
nosy: docs at python, zacherates
priority: normal
severity: normal
status: open
title: Awkward phrasing in Decimal documentation
versions: Python 3.3
Added file: http://bugs.python.org/file24165/rephrase.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 20:40:56 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 07 Jan 2012 19:40:56 +0000
Subject: [issue13502] Documentation for Event.wait return value is either
	wrong or incomplete
In-Reply-To: <1322599516.14.0.0239672713193.issue13502@psf.upfronthosting.co.za>
Message-ID: <1325965256.46.0.765879635726.issue13502@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13502>
_______________________________________

From report at bugs.python.org  Sat Jan  7 20:52:20 2012
From: report at bugs.python.org (Benjamin West)
Date: Sat, 07 Jan 2012 19:52:20 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325965940.99.0.481997517441.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Thanks Eric.  I was thrown by this document: http://wiki.python.org/moin/Git which describes fetching the sources from SVN using git.  I'm comfortable doing either, but it doesn't resolve my confusion.

The version of argparse in the python checkout is 1.1: http://hg.python.org/cpython/file/default/Lib/argparse.py
64 __version__ = '1.1' but differs from the SVN version.

whereas the argparse version available via google code is 1.2.  The diffs indicate several changes not related to the change I'm attempting to make, which prevent my patch from applying cleanly.  Looks like the HG version includes the 1.2 code... but I'm not sure why it would differ from SVN's trunk.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Sat Jan  7 20:52:21 2012
From: report at bugs.python.org (Benjamin West)
Date: Sat, 07 Jan 2012 19:52:21 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325965941.89.0.365502484117.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Thanks Eric.  I was thrown by this document: http://wiki.python.org/moin/Git which describes fetching the sources from SVN using git.  I'm comfortable doing either, but it doesn't resolve my confusion.

The version of argparse in the python checkout is 1.1: http://hg.python.org/cpython/file/default/Lib/argparse.py
64 __version__ = '1.1' but differs from the SVN version.

whereas the argparse version available via google code is 1.2.  The diffs indicate several changes not related to the change I'm attempting to make, which prevent my patch from applying cleanly.  Looks like the HG version includes the 1.2 code... but I'm not sure why it would differ from SVN's trunk.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Sat Jan  7 21:06:27 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 07 Jan 2012 20:06:27 +0000
Subject: [issue13732] test_logging failure on Windows buildbots
Message-ID: <1325966787.14.0.961413341663.issue13732@psf.upfronthosting.co.za>


New submission from Charles-Fran?ois Natali <neologix at free.fr>:

Commit 57295c4d81ac879dd2d804190b38b2e91f934acd broke Windows buildbots:

"""
======================================================================
ERROR: test_rotator (test.test_logging.RotatingFileHandlerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_logging.py", line 3543, in tearDown
    os.unlink(fn)
PermissionError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\db3l\\appdata\\local\\temp\\test_logging-2-9ozjk9.log'

======================================================================
FAIL: test_rotator (test.test_logging.RotatingFileHandlerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_logging.py", line 3631, in test_rotator
    self.assertEqual(data.decode("ascii"), m1.msg + "\n")
AssertionError: '1\r\n' != '1\n'
- 1
?  -
+ 1


======================================================================
FAIL: test_should_not_rollover (test.test_logging.RotatingFileHandlerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_logging.py", line 3536, in setUp
    BaseTest.setUp(self)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_logging.py", line 114, in setUp
    raise AssertionError('Unexpected handlers: %s' % hlist)
AssertionError: Unexpected handlers: [<logging.StreamHandler object at 0x0EBAC540>]
"""

See http://python.org/dev/buildbot/all/builders/x86%20Windows7%203.x/builds/4180/steps/test/logs/stdio and http://python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5780/steps/test/logs/stdio

----------
assignee: vinay.sajip
components: Tests
messages: 150817
nosy: neologix, vinay.sajip
priority: normal
severity: normal
stage: needs patch
status: open
title: test_logging failure on Windows buildbots
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13732>
_______________________________________

From report at bugs.python.org  Sat Jan  7 21:07:24 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 20:07:24 +0000
Subject: [issue13587] Correcting the typos error in Doc/howto/urllib2.rst
In-Reply-To: <1323693654.01.0.260479490427.issue13587@psf.upfronthosting.co.za>
Message-ID: <1325966844.41.0.859284995677.issue13587@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

Here's a patch that makes the WWW-Authenticate headers in howto/urllib2 agree with rfc2617.

----------
keywords: +patch
nosy: +zacherates
Added file: http://bugs.python.org/file24166/issue13587.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13587>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:08:37 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Sat, 07 Jan 2012 21:08:37 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325970517.16.0.644338117514.issue13731@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee: docs at python -> rhettinger
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:12:22 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Sat, 07 Jan 2012 21:12:22 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325970742.89.0.731074427481.issue13731@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

I'm sorry but I think the current wording is better that your proposed revision.

When I get a chance, I'll revisit it to see if I can find another way to improve the text.

----------
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:28:06 2012
From: report at bugs.python.org (Benjamin West)
Date: Sat, 07 Jan 2012 21:28:06 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325971686.02.0.425236620249.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Ok, here's a rough attempt at stubbing this out against a python checkout.  Will try to look at adding tests.

(BTW, subsequent GETs should not modify the bug tracker... this seems like a bug since GET should be idempotent, but SFTN from the double posting.)

----------
hgrepos: +100

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:28:07 2012
From: report at bugs.python.org (Benjamin West)
Date: Sat, 07 Jan 2012 21:28:07 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1325971687.04.0.253680930142.issue9253@psf.upfronthosting.co.za>


Benjamin West <bewest at gmail.com> added the comment:

Ok, here's a rough attempt at stubbing this out against a python checkout.  Will try to look at adding tests.

(BTW, subsequent GETs should not modify the bug tracker... this seems like a bug since GET should be idempotent, but SFTN from the double posting.)

----------
hgrepos: +101

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:34:39 2012
From: report at bugs.python.org (Eric Snow)
Date: Sat, 07 Jan 2012 21:34:39 +0000
Subject: [issue13728] Description of -m and -c cli options wrong?
In-Reply-To: <1325950613.83.0.912226487635.issue13728@psf.upfronthosting.co.za>
Message-ID: <1325972079.46.0.745351783397.issue13728@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

see http://docs.python.org/library/sys.html#sys.path

"As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first."

So the empty string here is the current working directory (at the time the script is run?).

----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13728>
_______________________________________

From report at bugs.python.org  Sat Jan  7 22:39:13 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 21:39:13 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325972353.16.0.612346880682.issue13731@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

That's fine. I'm not particularly attached to that phrasing.

The one thing I would push for is to add a comma to "... decimal is preferred in accounting applications which have strict equality invariants."

... since, as far as I can tell, "which have strict equality invariants" is supposed to be a parenthetical statement explaining why accounting applications prefer to use decimal arithmetic, rather than a constraints on the preference for decimal arithmetic to only those accounting applications that have "strict equality invariants".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 23:32:26 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 22:32:26 +0000
Subject: [issue13050] RLock support the context manager protocol but this is
	not documented
In-Reply-To: <1318172715.77.0.360303466185.issue13050@psf.upfronthosting.co.za>
Message-ID: <1325975546.91.0.804213636309.issue13050@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

Here is a patch that adds an note about using Locks, RLocks, Conditions, and Semaphores as context managers to each of their descriptions as well as a link to the "Using locks, conditions, and semaphores in the with statement" section.

----------
keywords: +patch
nosy: +zacherates
versions: +Python 3.3
Added file: http://bugs.python.org/file24167/issue13050.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13050>
_______________________________________

From report at bugs.python.org  Sat Jan  7 23:42:38 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Sat, 07 Jan 2012 22:42:38 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325976158.92.0.651444069723.issue13731@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

If you can't ascertain the meaning of the sentence, I'll consider making a change.  Itherwise, this appears to have degenerated into trivial micro-wordsmithing and I'll close this as not being worth consuming any more of my time.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 23:47:33 2012
From: report at bugs.python.org (Aaron Maenpaa)
Date: Sat, 07 Jan 2012 22:47:33 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325976453.57.0.311234657811.issue13731@psf.upfronthosting.co.za>


Aaron Maenpaa <aaron at maenpaa.ca> added the comment:

I can understand what was meant. You're welcome to close the issue.

Sorry for the nitpick.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sat Jan  7 23:48:26 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Sat, 07 Jan 2012 22:48:26 +0000
Subject: [issue13731] Awkward phrasing in Decimal documentation
In-Reply-To: <1325963806.67.0.168164997308.issue13731@psf.upfronthosting.co.za>
Message-ID: <1325976506.72.0.468000029578.issue13731@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

No problem.  Thanks for showing an interest in the quality of the documentation.

----------
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13731>
_______________________________________

From report at bugs.python.org  Sun Jan  8 00:13:45 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Sat, 07 Jan 2012 23:13:45 +0000
Subject: [issue13050] RLock support the context manager protocol but this is
	not documented
In-Reply-To: <1318172715.77.0.360303466185.issue13050@psf.upfronthosting.co.za>
Message-ID: <1325978025.43.0.76903837517.issue13050@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

This edit looks reasonable.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13050>
_______________________________________

From report at bugs.python.org  Sun Jan  8 00:24:49 2012
From: report at bugs.python.org (Tim Peters)
Date: Sat, 07 Jan 2012 23:24:49 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325978689.22.0.415333730698.issue13703@psf.upfronthosting.co.za>


Tim Peters <tim.peters at gmail.com> added the comment:

[Marc-Andre]
> BTW: I wonder how long it's going to take before
> someone figures out that our merge sort based
> list.sort() is vulnerable as well... its worst-
> case performance is O(n log n), making attacks
> somewhat harder.

I wouldn't worry about that, because nobody could stir up anguish
about it by writing a paper ;-)

1. O(n log n) is enormously more forgiving than O(n**2).

2. An attacker need not be clever at all:  O(n log n) is not only
sort()'s worst case, it's also its _expected_ case when fed randomly
ordered data.

3. It's provable that no comparison-based sorting algorithm can have
better worst-case asymptotic behavior when fed randomly ordered data.

So if anyone whines about this, tell 'em to go do something useful instead :-)

----------
nosy: +tim_one

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 00:26:25 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 07 Jan 2012 23:26:25 +0000
Subject: [issue13732] test_logging failure on Windows buildbots
In-Reply-To: <1325966787.14.0.961413341663.issue13732@psf.upfronthosting.co.za>
Message-ID: <E1RjfeZ-0002zB-Pu@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 870f8fdfbada by Vinay Sajip in branch 'default':
Closes #13732: now use os.linesep instead of a literal newline.
http://hg.python.org/cpython/rev/870f8fdfbada

----------
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13732>
_______________________________________

From report at bugs.python.org  Sun Jan  8 00:53:22 2012
From: report at bugs.python.org (Paul Smedley)
Date: Sat, 07 Jan 2012 23:53:22 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
Message-ID: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>


New submission from Paul Smedley <paul at smedley.id.au>:

Python 2.7.2 fails during compilation on OS/2  with:
 File "./setup.py", line 1154, in detect_modules
   for arg in sysconfig.get_config_var("CONFIG_ARGS").split()]
AttributeError: 'NoneType' object has no attribute 'split'
make: *** [sharedmods] Error 1

Attached diff for sysconfig.py corrects the problem

----------
components: Build
files: sysconfig.py.diff
keywords: patch
messages: 150831
nosy: Paul.Smedley
priority: normal
severity: normal
status: open
title: Change required to sysconfig.py for Python 2.7.2 on OS/2
type: compile error
versions: Python 2.7
Added file: http://bugs.python.org/file24168/sysconfig.py.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 00:53:45 2012
From: report at bugs.python.org (Martin)
Date: Sat, 07 Jan 2012 23:53:45 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325980425.1.0.333063906681.issue13703@psf.upfronthosting.co.za>


Martin <gzlist at googlemail.com> added the comment:

I built random-2.patch on my windows xp box (updating the project and fixing some compile errors in random.c required), and initialising crypto has a noticeable impact on startup time. The numbers vary a fair bit naturally, two representative runs are as follows:

changeset 52796:1ea8b7233fd7 on default branch:

    >timeit %PY3K% -c "import sys;print(sys.version)"
    3.3.0a0 (default, Jan  7 2012, 00:12:45) [MSC v.1500 32 bit (Intel)]

    Version Number:   Windows NT 5.1 (Build 2600)
    Exit Time:        0:16 am, Saturday, January 7 2012
    Elapsed Time:     0:00:00.218
    Process Time:     0:00:00.187
    System Calls:     4193
    Context Switches: 445
    Page Faults:      1886
    Bytes Read:       642542
    Bytes Written:    272
    Bytes Other:      31896

with random-2.patch and fixes applied:

    >timeit %PY3K% -c "import sys;print(sys.version)"
    3.3.0a0 (default, Jan  7 2012, 00:58:32) [MSC v.1500 32 bit (Intel)]

    Version Number:   Windows NT 5.1 (Build 2600)
    Exit Time:        0:59 am, Saturday, January 7 2012
    Elapsed Time:     0:00:00.296
    Process Time:     0:00:00.234
    System Calls:     4712
    Context Switches: 642
    Page Faults:      2049
    Bytes Read:       1059381
    Bytes Written:    272
    Bytes Other:      34544

This is with hot caches, cold will likely be worse, but a smaller percentage change. On a faster box, or with an SSD, or win 7, the delta will likely be smaller too.

A 50-100ms slow down is consistent with the difference on Python 2.7 between calling `os.urandom(1)` or not. However, the baseline is faster with Python 2, frequently dipping under 100ms, so there this change could double the runtime of trivial scripts.

----------
nosy: +gz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 01:14:03 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sun, 08 Jan 2012 00:14:03 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
Message-ID: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>


New submission from Hynek Schlawack <hs at ox.cx>:

This is an offspring of #4489 (which is a security bug). The method is AFAIU intended to be private. 

As shown in the discussion of the mentioned #4489, there is a whole family of attacks that exploit the time window between gathering path names and executing a function on them. A general description of this problem can be found in: https://www.securecoding.cert.org/confluence/display/seccode/POS35-C.+Avoid+race+conditions+while+checking+for+the+existence+of+a+symbolic+link

While the consequences in rmtree() are probably most dramatic, other recursive functions could benefit too (chmodtree() and chowntree() were mentioned) so Charles-Fran?ois suggested to write a "generic walker method that would take as argument the methods to call on a directory and on a file (or link)".

Some (probably) necessary helper functions has been already implemented in #4761 (*at()) and #10755 (fdlistdir()).

Has there already been done any work? Ross mentioned he wanted to take a stab?

----------
components: Library (Lib)
messages: 150833
nosy: hynek.schlawack, neologix, pitrou, rosslagerwall, tarek
priority: normal
severity: normal
status: open
title: Add a generic directory walker method to avoid symlink attacks
type: security
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 01:15:51 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sun, 08 Jan 2012 00:15:51 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1325958902.3435.1.camel@localhost.localdomain>
Message-ID: <607C351611E54406973EB6E489589DF1@gmail.com>


Hynek Schlawack <hs at ox.cx> added the comment:

> > What's the current state here? Anyone working on a solution or are we
> > waiting how http://hg.python.org/features/pathlib/ will work out?
>  
> Well, I am not working on that one, so waiting for it to work out might
> be optimistic :)
> I don't know what to do with it (the pathlib): is such a feature
> desireable enough?

Independently from this bug, I'd say it would be a good thing.

Proof: http://twistedmatrix.com/documents/current/api/twisted.python.filepath.html ? Twisted already implemented something similar for themselves.

> > If the consensus is to add a generic walker method, wouldn't be
> > appropriate to open a new bug and add it as dependency?
>  
> Agreed.

See #13734

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Sun Jan  8 01:19:16 2012
From: report at bugs.python.org (Glenn Linderman)
Date: Sun, 08 Jan 2012 00:19:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325981956.53.0.572573613251.issue13703@psf.upfronthosting.co.za>


Glenn Linderman <v+python at g.nevcal.com> added the comment:

Given Martin's comment (msg150832) I guess I should add my suggestion to this issue, at least for the record.

Rather than change hash functions, randomization could be added to those dicts that are subject to attack by wanting to store user-supplied key values.  The list so far seems to be   urllib.parse, cgi, json  Some have claimed there are many more, but without enumeration.  These three are clearly related to the documented issue.

The technique would be to wrap dict and add a short random prefix to each key value, preventing the attacker from supplier keys that are known to collide... and even if he successfully stumbles on a set that does collide on one request, it is unlikely to collide on a subsequent request with a different prefix string.

The technique is fully backward compatible with all applications except those that contain potential vulnerabilities as described by the researchers. The technique adds no startup or runtime overhead to any application that doesn't contain the potential vulnerabilities.  Due to the per-request randomization, the complexity of creating a sequence of sets of keys that may collide is enormous, and requires that such a set of keys happen to arrive on a request in the right sequence where the predicted prefix randomization would be used to cause the collisions to occur.  This might be possible on a lightly loaded system, but is less likely on a system with heavy load, which are more interesting to attack.

Serhiy Storchaka provided a sample implementation on the python-dev, copied below, and attached as a file (but is not a patch).

# -*- coding: utf-8 -*-
from collections import MutableMapping
import random


class SafeDict(dict, MutableMapping):

    def __init__(self, *args, **kwds):
        dict.__init__(self)
        self._prefix = str(random.getrandbits(64))
        self.update(*args, **kwds)

    def clear(self):
        dict.clear(self)
        self._prefix = str(random.getrandbits(64))

    def _safe_key(self, key):
        return self._prefix + repr(key), key

    def __getitem__(self, key):
        try:
            return dict.__getitem__(self, self._safe_key(key))
        except KeyError as e:
            e.args = (key,)
            raise e

    def __setitem__(self, key, value):
        dict.__setitem__(self, self._safe_key(key), value)

    def __delitem__(self, key):
        try:
            dict.__delitem__(self, self._safe_key(key))
        except KeyError as e:
            e.args = (key,)
            raise e

    def __iter__(self):
        for skey, key in dict.__iter__(self):
            yield key

    def __contains__(self, key):
        return dict.__contains__(self, self._safe_key(key))

    setdefault = MutableMapping.setdefault
    update = MutableMapping.update
    pop = MutableMapping.pop
    popitem = MutableMapping.popitem
    keys = MutableMapping.keys
    values = MutableMapping.values
    items = MutableMapping.items

    def __repr__(self):
        return '{%s}' % ', '.join('%s: %s' % (repr(k), repr(v))
            for k, v in self.items())

    def copy(self):
        return self.__class__(self)

    @classmethod
    def fromkeys(cls, iterable, value=None):
        d = cls()
        for key in iterable:
            d[key] = value
        return d

    def __eq__(self, other):
        return all(k in other and other[k] == v for k, v in self.items()) and \
            all(k in self and self[k] == v for k, v in other.items())

    def __ne__(self, other):
        return not self == other

----------
Added file: http://bugs.python.org/file24169/SafeDict.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 01:21:48 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Sun, 08 Jan 2012 00:21:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325982108.66.0.329382432453.issue13703@psf.upfronthosting.co.za>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

You're seriously underestimating the number of vulnerable dicts.  It has nothing to do with the module, and everything to do with the origin of the data.  There's tons of user code that's vulnerable too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 01:33:00 2012
From: report at bugs.python.org (Glenn Linderman)
Date: Sun, 08 Jan 2012 00:33:00 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1325982780.36.0.940971927339.issue13703@psf.upfronthosting.co.za>


Glenn Linderman <v+python at g.nevcal.com> added the comment:

Alex, I agree the issue has to do with the origin of the data, but the modules listed are the ones that deal with the data supplied by this particular attack.

Note that changing the hash algorithm for a persistent process, even though each process may have a different seed or randomized source, allows attacks for the life of that process, if an attack vector can be created during its lifetime. This is not a problem for systems where each request is handled by a different process, but is a problem for systems where processes are long-running and handle many requests.

Regarding vulnerable user code, supplying SafeDict (or something similar) in the stdlib or as sample code for use in such cases allows user code to be fixed also.

You have entered the class of people that claim lots of vulnerabilities, without enumeration.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 02:08:23 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 08 Jan 2012 01:08:23 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1325984903.99.0.190030725288.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I'm working on a library of general directory walking tools that will hopefully make their way back into the stdlib at some point (http://walkdir.readthedocs.org).

They're designed to filter and transform the output of os.walk (and similar iterators) in various ways.

It may provide a good environment for prototyping a general purpose "tree_map" for applying an operation to a filesystem tree without being vulnerable to symlink attacks.

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 02:23:24 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 01:23:24 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1325985804.92.0.415565716053.issue13733@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think OS/2 portability fixes should probably target 3.3 instead.

----------
nosy: +eric.araujo, loewis, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 03:40:42 2012
From: report at bugs.python.org (Paul McMillan)
Date: Sun, 08 Jan 2012 02:40:42 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325982780.36.0.940971927339.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRUVxD8Sw148ezEfxqV2b-0hhMqYDC8MdjU3pAjh0NxK2w@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> Alex, I agree the issue has to do with the origin of the data, but the modules listed are the ones that deal with the data supplied by this particular attack.

They deal directly with the data. Do any of them pass the data
further, or does the data stop with them? A short and very incomplete
list of vulnerable standard lib modules includes: every single parsing
library (json, xml, html, plus all the third party libraries that do
that), all of numpy (because it processes data which probably came
from a user [yes, integers can trigger the vulnerability]), difflib,
the math module, most database adaptors, anything that parses metadata
(including commonly used third party libs like PIL), the tarfile lib
along with other compressed format handlers, the csv module,
robotparser, plistlib, argparse, pretty much everything under the
heading of "18. Internet Data Handling" (email, mailbox, mimetypes,
etc.), "19. Structured Markup Processing Tools", "20. Internet
Protocols and Support", "21. Multimedia Services", "22.
Internationalization", TKinter, and all the os calls that handle
filenames. The list is impossibly large, even if we completely ignore
user code. This MUST be fixed at a language level.

I challenge you to find me 15 standard lib components that are certain
to never handle user-controlled input.

> Note that changing the hash algorithm for a persistent process, even though each process may have a different seed or randomized source, allows attacks for the life of that process, if an attack vector can be created during its lifetime. This is not a problem for systems where each request is handled by a different process, but is a problem for systems where processes are long-running and handle many requests.

This point has been made many times now. I urge you to read the entire
thread on the mailing list. Your implementation is impractical because
your "safe" implementation completely ignores all hash caching (each
entry must be re-hashed for that dict). Your implementation is still
vulnerable in exactly the way you mentioned if you ever have any kind
of long-lived dict in your program thread.

> You have entered the class of people that claim lots of vulnerabilities, without enumeration.

I have enumerated. Stop making this argument.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 04:38:30 2012
From: report at bugs.python.org (Kay Hayen)
Date: Sun, 08 Jan 2012 03:38:30 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
Message-ID: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>


New submission from Kay Hayen <kayhayen at gmx.de>:

Hello,

I am implementing a Python compiler (Nuitka) that is testing if when it compiles itself, it gives the same output. 

I have been using "protocol = 0" ever since with "pickle" module for historic reasons (gcc bug with raw strings lead me to believe it's better) and lately, I have changed to "protocol = 2" and cPickle. But then I noticed that my compile itself test now fail to give same code from pickling of dictionary constants.

Imanaged and isolated the issue, and it's a Python2.7 regression, Python2.6 is fine:

Observe this output from "cPickle.dumps" for a constant dictionary with one element:

Protocol 0 :
Dumping read const const stream "(dp1\nS'modules'\np2\nNs."
Dumping load const const stream "(dp1\nS'modules'\np2\nNs."
Dumping load const const stream "(dp1\nS'modules'\np2\nNs."
Protocol 1 :
Dumping read const const stream '}q\x01U\x07modulesq\x02Ns.'
Dumping load const const stream '}q\x01U\x07modulesNs.'
Dumping load const const stream '}q\x01U\x07modulesNs.'
Protocol 2 :
Dumping read const const stream '\x80\x02}q\x01U\x07modulesq\x02Ns.'
Dumping load const const stream '\x80\x02}q\x01U\x07modulesNs.'
Dumping load const const stream '\x80\x02}q\x01U\x07modulesNs.'

It seems that cPickle as of CPython2.7 does give a better stream for dictionaries it itself emitted. With CPython2.6 I observe no difference.

My work-around is to "re-stream", "dumps" -> "loads" -> "dumps" with CPython2.7 for the time being.

Can you either: Fix cPickle to treat the dictionaries the same, or enhance to core to produce the same dict as cPickle does? It appears at least some kind of efficiency might be missed out for marshall as well.

----------
components: Interpreter Core, Library (Lib)
files: stream.py
messages: 150841
nosy: kayhayen
priority: normal
severity: normal
status: open
title: The protocol > 0 of cPickle does not given stable dictionary values
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file24170/stream.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 04:47:36 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 08 Jan 2012 03:47:36 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1325994456.89.0.346556764801.issue13733@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Paul, can you please elaborate on the patch? The original code looks better to me than the new code. Why do you think this change is right?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 04:49:11 2012
From: report at bugs.python.org (Paul Smedley)
Date: Sun, 08 Jan 2012 03:49:11 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1325994551.09.0.8089016253.issue13733@psf.upfronthosting.co.za>


Paul Smedley <paul at smedley.id.au> added the comment:

I'm building OS/2 python here using configure, etc - without the changes attached, python would not complete the build.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 04:54:30 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sun, 08 Jan 2012 03:54:30 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1325994870.55.0.413369657712.issue12736@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Here is a patch. I only dealt with case mappings and not titlecase. Doing titlecase properly requires word segmentation, which I think should be another patch/issue. This patch fixes swapcase(), capitalize(), upper(), and lower(). It does not include the changes to Objects/unicodetype_db.h because those are huge. Regenerate the database if you want to test it. Please review.

----------
keywords: +patch
nosy: +benjamin.peterson
Added file: http://bugs.python.org/file24171/full-casemapping.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Sun Jan  8 05:03:02 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 08 Jan 2012 04:03:02 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1325995382.65.0.668552628343.issue13734@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

> Has there already been done any work? Ross mentioned he wanted to take a stab?

Unfortunately, I'm rather busy at the moment but when I get some free time and if no one else wants to work on it then I'll take a look.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 06:18:56 2012
From: report at bugs.python.org (Glenn Linderman)
Date: Sun, 08 Jan 2012 05:18:56 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAO_YWRUVxD8Sw148ezEfxqV2b-0hhMqYDC8MdjU3pAjh0NxK2w@mail.gmail.com>
Message-ID: <4F092727.2010200@g.nevcal.com>


Glenn Linderman <v+python at g.nevcal.com> added the comment:

[offlist]
Paul, thanks for the enumeration and response.  Some folks have more 
experience, but the rest of us need to learn.  Having the proposal in 
the ticket, with an explanation of its deficiencies is not all bad, 
however, others can learn, perhaps.  On the other hand, I'm willing to 
learn more, if you are willing to address my concerns below.

I had read the whole thread and issue, but it still seemed like a leap 
of faith to conclude that the only, or at least best, solution is 
changing the hash.  Yet, changing the hash still doesn't seem like a 
sufficient solution, due to long-lived processes.

On 1/7/2012 6:40 PM, Paul McMillan wrote:
> Paul McMillan<paul at mcmillan.ws>  added the comment:
>
>> Alex, I agree the issue has to do with the origin of the data, but the modules listed are the ones that deal with the data supplied by this particular attack.
> They deal directly with the data. Do any of them pass the data
> further, or does the data stop with them?

For web forms and requests, which is the claimed vulnerability, I would 
expect that most of them do not pass the data further, without 
validation or selection, and it is unlikely that the form is actually 
expecting data with colliding strings, so it seems very unlikely that 
they would be passed on. At least that is how I code my web apps: just 
select the data I expect from my form.  At present I do not reject data 
I do not expect, but I'll have to consider either using SafeDict (which 
I can start using ASAP, not waiting for a new release of Perl to be 
installed on my Web Server (currently running Perl 2.4), or rejecting 
data I do not expect prior to putting it in a dict.  That might require 
tweaking urllib.parse a bit, or cgi, or both.

> A short and very incomplete
> list of vulnerable standard lib modules includes: every single parsing
> library (json, xml, html, plus all the third party libraries that do
> that), all of numpy (because it processes data which probably came
> from a user [yes, integers can trigger the vulnerability]), difflib,
> the math module, most database adaptors, anything that parses metadata
> (including commonly used third party libs like PIL), the tarfile lib
> along with other compressed format handlers, the csv module,
> robotparser, plistlib, argparse, pretty much everything under the
> heading of "18. Internet Data Handling" (email, mailbox, mimetypes,
> etc.), "19. Structured Markup Processing Tools", "20. Internet
> Protocols and Support", "21. Multimedia Services", "22.
> Internationalization", TKinter, and all the os calls that handle
> filenames. The list is impossibly large, even if we completely ignore
> user code. This MUST be fixed at a language level.
>
> I challenge you to find me 15 standard lib components that are certain
> to never handle user-controlled input.

I do appreciate your enumeration, but I'll decline the challenge.  While 
all of them can be interesting exploits of na?ve applications (written 
by programmers who may be quite experienced in some things, but can 
na?vely overlook other things), most of them probably do not apply to 
the documented vulnerability. Many I had thought of, but rejected for 
this context; some I had not.  So while there are many possible 
situations where happily stuffing things into a dict may be an easy 
solution, there are many possible cases where it should be prechecked on 
the way in.  And there is another restriction: if the user-controlled 
input enters a user-run program, it is unlikely to be attacked in the 
same manner than web servers are attacked.  A user, for example, is 
unlikely to contrive colliding file names for the purpose of making his 
file listing program run slow.

So it is really system services and web services that need to be 
particularly careful. Randomizing the hash seed might reduce the problem 
from any system/web services to only long-running system/web services, 
but doesn't really solve the complete problem, as far as I can tell... 
only proper care in writing the application (and the stdlib code) will 
solve the complete problem.  Sadly, beefing up the stdlib code will 
probably reduce performance for things that will not be exploited to be 
careful enough in the cases that could be exploited.

>> Note that changing the hash algorithm for a persistent process, even though each process may have a different seed or randomized source, allows attacks for the life of that process, if an attack vector can be created during its lifetime. This is not a problem for systems where each request is handled by a different process, but is a problem for systems where processes are long-running and handle many requests.
> This point has been made many times now. I urge you to read the entire
> thread on the mailing list. Your implementation is impractical because
> your "safe" implementation completely ignores all hash caching (each
> entry must be re-hashed for that dict). Your implementation is still
> vulnerable in exactly the way you mentioned if you ever have any kind
> of long-lived dict in your program thread.

I had read the entire thread, or at least the part that made it to my 
inbox. It is a very interesting discussion.

I have to admit I was aware that hashes got cached, but the fact that 
strings keep getting recreated causes the cache to be sidestepped did 
slip past me. On the other hand, the only strings that will get 
recreated in a CGI situation are a few FORM parameters per request... 
the expected ones.  And only if the application keeps referring to the 
CGI dict to find them, instead of binding the values locally.  For JSON 
data structures, it seems more likely that they would be iterated, 
rather than point lookups being done.  But it does depend on the 
application whether missing the hash caching would be severe.

A long-lived dict is only vulnerable if it continues to accept updates 
during its lifetime.  For web requests, the vulnerability of concern 
here, that is generally not the case... web requests are generally 
discarded at completion.

>> You have entered the class of people that claim lots of vulnerabilities, without enumeration.
> I have enumerated. Stop making this argument.

I appreciate the enumeration, even though you were not the person to 
whom that statement was addressed.

And, I see that a randomized hash per process helps reduce a large class 
of potential vulnerabilities for short-lived processes, but I still have 
concerns that simply randomizing the hash per process adds startup 
overhead, gratuitous incompatibility, and doesn't fully solve the 
problem, as collisions are still possible, and attackers are clever.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 06:37:01 2012
From: report at bugs.python.org (Christian Heimes)
Date: Sun, 08 Jan 2012 05:37:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326001021.01.0.669536670395.issue13703@psf.upfronthosting.co.za>


Christian Heimes <lists at cheimes.de> added the comment:

Glenn, you have reached a point where you stop bike-shedding and start to troll by attacking people. Please calm down. I'm sure that you are just worried about the future of Python and all the bad things, that might be introduced by a fix for the issue.

Please trust us! Paul, Victor, Antoine and several more involved developers are professional Python devs and have been for years. Most of them do Python development for a living. We won't kill the snake that pays our bills. ;) Ultimately it's Guido's choice, too. 

Martin:
Ouch, the startup impact is large! Have we reached a point where "one size fits all" doesn't work any longer? It's getting harder to have just one executable for 500ms scripts and server processes that last for weeks.

Marc-Andre:
Have you profiled your suggestion? I'm interested in the speed implications. My gut feeling is that your idea could be slower, since you have added more instructions to a tight loop, that is execute on every lookup, insert, update and deletion of a dict key. The hash modification could have a smaller impact, since the hash is cached. I'm merely speculating here until we have some numbers to compare.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 06:55:11 2012
From: report at bugs.python.org (Glenn Linderman)
Date: Sun, 08 Jan 2012 05:55:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326001021.01.0.669536670395.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F092FAA.2080902@g.nevcal.com>


Glenn Linderman <v+python at g.nevcal.com> added the comment:

I don't find a way to delete my prior comment, so I'll add one more 
(only). The prior comment was intended to go to one person, but I didn't 
notice the From, having one person's name, actually went back to the 
ticket (the email address not being for that individual), now I do, so 
I've learned that.

My prior comment was a request for further explanation of things I still 
don't understand, not intended to be an attack.  If someone can delete 
both this and my prior comment from the issue, or tell me how, feel free.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 07:20:24 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 08 Jan 2012 06:20:24 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1228232522.58.0.0992151848245.issue4489@psf.upfronthosting.co.za>
Message-ID: <1326003624.19.0.0308971487042.issue4489@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
dependencies: +Add a generic directory walker method to avoid symlink attacks

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Sun Jan  8 07:22:20 2012
From: report at bugs.python.org (Jyotirmoy Bhattacharya)
Date: Sun, 08 Jan 2012 06:22:20 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
Message-ID: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>


New submission from Jyotirmoy Bhattacharya <jyotirmoy at jyotirmoy.net>:

The documentation for urlopen says that it raises URLError on error. 
But there exist error conditions such as a socket timeout or a bad HTTP status line under which the exception from the underlying library leaks through urllib, thus breaking the promise in the documentation.

I am attaching a test program that demonstrates this bug.

----------
components: Library (Lib)
files: test_urllib_except.py
messages: 150849
nosy: jmoy, orsenthil
priority: normal
severity: normal
status: open
title: urllib.request.urlopen leaks exceptions from socket and httplib.client
versions: Python 3.3
Added file: http://bugs.python.org/file24172/test_urllib_except.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Sun Jan  8 07:26:05 2012
From: report at bugs.python.org (Jyotirmoy Bhattacharya)
Date: Sun, 08 Jan 2012 06:26:05 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>
Message-ID: <1326003965.78.0.823131019371.issue13736@psf.upfronthosting.co.za>


Jyotirmoy Bhattacharya <jyotirmoy at jyotirmoy.net> added the comment:

A patch to fix this issue. Catches exceptions from underlying libraries and reraises them as URLError.

I put the class name of the underlying exception in the reason to make it more descriptive.

----------
keywords: +patch
type:  -> behavior
Added file: http://bugs.python.org/file24173/urllib_exception_leak.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Sun Jan  8 07:43:56 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 08 Jan 2012 06:43:56 +0000
Subject: [issue12364] Deadlock in test_concurrent_futures
In-Reply-To: <1308504315.81.0.454658658034.issue12364@psf.upfronthosting.co.za>
Message-ID: <E1RjmTw-0006QQ-Rl@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 26389e9efa9c by Ross Lagerwall in branch '3.2':
Issue #12364: Fix a hang in concurrent.futures.ProcessPoolExecutor.
http://hg.python.org/cpython/rev/26389e9efa9c

New changeset 25f879011102 by Ross Lagerwall in branch 'default':
Merge with 3.2 for #12364.
http://hg.python.org/cpython/rev/25f879011102

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12364>
_______________________________________

From report at bugs.python.org  Sun Jan  8 08:52:07 2012
From: report at bugs.python.org (Bithin A)
Date: Sun, 08 Jan 2012 07:52:07 +0000
Subject: [issue13737] bugs.python.org's Django settings file DEBUG=True
Message-ID: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>


New submission from Bithin A <bithin2007 at gmail.com>:

I am getting an Django error when I was trying to review a patch. This error has occurred in the site as the DEBUG mode in the Django settings file is set to True. It is a security issue and should be set to false. I am attaching the screen shot of the error which I have come across.

----------
components: None
files: Screenshot at 2012-01-08 13:11:43.png
messages: 150852
nosy: Bithin.A
priority: normal
severity: normal
status: open
title: bugs.python.org's Django settings file DEBUG=True
type: security
Added file: http://bugs.python.org/file24174/Screenshot at 2012-01-08 13:11:43.png

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Sun Jan  8 10:35:50 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 08 Jan 2012 09:35:50 +0000
Subject: [issue12364] Deadlock in test_concurrent_futures
In-Reply-To: <1308504315.81.0.454658658034.issue12364@psf.upfronthosting.co.za>
Message-ID: <1326015350.1.0.42981293874.issue12364@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

Thanks!

----------
assignee:  -> rosslagerwall
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12364>
_______________________________________

From report at bugs.python.org  Sun Jan  8 10:40:25 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 08 Jan 2012 09:40:25 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1326015625.89.0.207334721149.issue13733@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Please consider this: the code that is there was specifically added by somebody porting Python to OS/2. So if it doesn't work for you, either OS/2 has changed (which I don't think is the case), or you are doing something wrong (such as using the wrong tool chain).

If the change you propose was carried out, it would likely break what works currently. So everybody is hesitant to add this change, unless you can explain it to us: line-by-line, character-by-character. E.g. why are you proposing that OS/2 is a POSIX system? Treating it similar to NT is much more plausible, given that NT was designed as the OS/2 successor originally.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 11:13:34 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 08 Jan 2012 10:13:34 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326017614.67.0.222812768652.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Since walkdir is currently entirely based on returning filesystem paths as strings (just like os.walk()) and hence shares the pervasive symlink attack vulnerability, I'm particularly interested in the question of whether or not the various *at APIs can be used to avoid symlink attacks if we just have a os.walkfd() API that emits a (dirfd, subdirs, files) triple instead of the os.walk style (dirpath, subdirs, files).

The reason I'd find that interesting is that many of walkdir's filtering steps (notably those for including and excluding directories) don't care about the value of dirpath, so they could still be used with such an API.

Thoughts?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 11:14:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 08 Jan 2012 10:14:57 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1326017697.31.0.773046635524.issue13737@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
assignee:  -> loewis
nosy: +loewis
priority: normal -> high
title: bugs.python.org's Django settings file DEBUG=True -> bugs.python.org/review's Django settings file DEBUG=True

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Sun Jan  8 11:20:28 2012
From: report at bugs.python.org (Paul McMillan)
Date: Sun, 08 Jan 2012 10:20:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326001021.01.0.669536670395.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAO_YWRU_97bwyCgYVK54ucdVj82Bcyb=nu6E2OANvar8Me1xrQ@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> Christian Heimes added the comment:
> Ouch, the startup impact is large! Have we reached a point where "one size fits all" doesn't work any longer? It's getting harder to have just one executable for 500ms scripts and server processes that last for weeks.

This concerns me too, and is one reason I think the collision counting
code might be the winning solution. Randomness is hard to do correctly
and is expensive. If we can avoid it, we should try very hard to do
so...

> Christian Heimes said to Marc-Andre:
> Have you profiled your suggestion? I'm interested in the speed implications. My gut feeling is that your idea could be slower, since you have added more instructions to a tight loop, that is execute on every lookup, insert, update and deletion of a dict key. The hash modification could have a smaller impact, since the hash is cached. I'm merely speculating here until we have some numbers to compare.

Interesting point, though I think we might be able to work it out so
that we're only adding instructions when there's actually a detected
collision. I'll be interested to see what the benchmarks (and real
world) have to say about the impacts of randomization as compared to
the existing black-magic optimization of the hash function.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 12:33:28 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Sun, 08 Jan 2012 11:33:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325978689.22.0.415333730698.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F097F81.1050103@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Tim Peters wrote:
> 
> Tim Peters <tim.peters at gmail.com> added the comment:
> 
> [Marc-Andre]
>> BTW: I wonder how long it's going to take before
>> someone figures out that our merge sort based
>> list.sort() is vulnerable as well... its worst-
>> case performance is O(n log n), making attacks
>> somewhat harder.
> 
> I wouldn't worry about that, because nobody could stir up anguish
> about it by writing a paper ;-)
> 
> 1. O(n log n) is enormously more forgiving than O(n**2).
> 
> 2. An attacker need not be clever at all:  O(n log n) is not only
> sort()'s worst case, it's also its _expected_ case when fed randomly
> ordered data.
> 
> 3. It's provable that no comparison-based sorting algorithm can have
> better worst-case asymptotic behavior when fed randomly ordered data.
> 
> So if anyone whines about this, tell 'em to go do something useful instead :-)

Right on all accounts :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 12:44:27 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 08 Jan 2012 11:44:27 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326023067.89.0.361519615891.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Another, possibly better, alternative would be to produce a tuple-subclass that adds a separate "dirfd" attribute to the (dirpath, subdirs, files) triple.

I'll stop talking about the walkdir implications here. Instead, I've created a corresponding issue on walkdir's own tracker:
https://bitbucket.org/ncoghlan/walkdir/issue/8/add-walkfd

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 12:47:18 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Sun, 08 Jan 2012 11:47:18 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326001021.01.0.669536670395.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0982C2.5040606@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Christian Heimes wrote:
> Marc-Andre:
> Have you profiled your suggestion? I'm interested in the speed implications. My gut feeling is that your idea could be slower, since you have added more instructions to a tight loop, that is execute on every lookup, insert, update and deletion of a dict key. The hash modification could have a smaller impact, since the hash is cached. I'm merely speculating here until we have some numbers to compare.

I haven't done any profiling on this yet, but will run some
tests.

The lookup functions in the dict implementation are optimized
to make the first non-collision case fast. The patch doesn't touch this
loop. The only change is in the collision case, where an increment
and comparison is added (and then only after the comparison which
is the real cost factor in the loop). I did add a printf() to
see how often this case occurs - it's a surprisingly rare case,
which suggests that Tim, Christian and all the others that have
invested considerable time into the implementation have done
a really good job here.

BTW: I noticed that a rather obvious optimization appears to be
missing from the Python dict initialization code: when passing in
a list of (key, value) pairs, the implementation doesn't make
use of the available length information and still starts with an
empty (small) dict table and then iterates over the pairs, increasing
the table size as necessary. It would be better to start with a
table that is presized to O(len(data)). The dict implementation
already provides such a function, but it's not being used
in the case dict(pair_list). Anyway, just an aside.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 12:59:59 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 11:59:59 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326023999.1.0.920150282309.issue12760@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I intend to commit this patch within a couple days (unless anyone objects of course).

----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sun Jan  8 13:02:08 2012
From: report at bugs.python.org (Matthias Klose)
Date: Sun, 08 Jan 2012 12:02:08 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326024128.27.0.371906597641.issue10278@psf.upfronthosting.co.za>


Matthias Klose <doko at debian.org> added the comment:

on linux the underlying functionality is implemented in librt; the extension doesn't check for this or links with -lrt.

----------
nosy: +doko

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Sun Jan  8 13:03:20 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 08 Jan 2012 12:03:20 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326024200.39.0.853101399157.issue13122@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi all, how can we fix it? or better, should we fix it? From a user POV, it is a weird to see 3.1 doc referring to 3.2 doc as "in development" then clicking on that link, being redirect to 3.3a0 doc and see there that 3.2 is stable (with no reference back to 3.1).

Maybe we can adjust the sidebar in 3.1 references and do a manual rebuild or just wait for another security release? maybe we can discuss a policy for such things and write it in stone (aka devguide).

----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 13:35:48 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 12:35:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326026148.75.0.646196413571.issue13703@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
Removed message: http://bugs.python.org/msg150848

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 13:36:35 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 12:36:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326026195.88.0.185307734054.issue13703@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
Removed message: http://bugs.python.org/msg150837

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 14:34:07 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 13:34:07 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1326029647.49.0.202665757317.issue13733@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well, the answer has probably been already given on python-dev:

> I've been building Python 2.x for a while, and currently have binaries 
> of 2.6.5 available from http://os2ports.smedley.info

> Unlike Andrew Mcintyre, I'm using libc for development 
> (http://svn.netlabs.org/libc) rather than emx.  libc is still being 
> developed whereas emx hasn't been updated in about 10 years.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Sun Jan  8 14:55:21 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 13:55:21 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1326017614.67.0.222812768652.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326030840.3374.5.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Since walkdir is currently entirely based on returning filesystem
> paths as strings (just like os.walk()) and hence shares the pervasive
> symlink attack vulnerability, I'm particularly interested in the
> question of whether or not the various *at APIs can be used to avoid
> symlink attacks if we just have a os.walkfd() API that emits a (dirfd,
> subdirs, files) triple instead of the os.walk style (dirpath, subdirs,
> files).

Be aware that you have to manage dirfd's lifetime, which can make things
interesting.
Also be aware that symlinks mean sometimes you won't have a dirfd: if
you have a symlink that points to another directory, you can't open that
directory using openat from the symlink's directory. So if you follow
symlinks (or have an option to do so) you must also take that case into
account.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 15:23:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 14:23:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAO_YWRU_97bwyCgYVK54ucdVj82Bcyb=nu6E2OANvar8Me1xrQ@mail.gmail.com>
Message-ID: <1326032510.3374.14.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Randomness is hard to do correctly
> and is expensive. If we can avoid it, we should try very hard to do
> so...

os.urandom() is actually cheaper on Windows 7 here:

1000000 loops, best of 3: 1.78 usec per loop

than on Linux:

$ ./python -m timeit -s "import os" "os.urandom(16)"
100000 loops, best of 3: 4.85 usec per loop
$ ./python -m timeit -s "import os; f=os.open('/dev/urandom', os.O_RDONLY)" "os.read(f, 16)"
100000 loops, best of 3: 2.35 usec per loop

(note that the os.read timing is optimistic since I'm not checking the
return value!)

I don't know if the patch's startup overhead has to do with initializing
the crypo context or simply with looking up the symbols in advapi32.dll.
Perhaps we should link explicitly against advapi32.dll as suggested by
Martin?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 15:26:13 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 14:26:13 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326032773.69.0.823320063087.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Again, Roundup ate up some of the text:

>PCbuild\amd64\python.exe  -m timeit -s "import os" "os.urandom(16)"
1000000 loops, best of 3: 1.81 usec per loop

(for the record, the Roundup issue is at http://psf.upfronthosting.co.za/roundup/meta/issue264 )

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan  8 15:36:58 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 14:36:58 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326033418.04.0.645986399942.issue12760@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I don't think the "created()" method has to be exposed. People can inspect the "mode" attribute if they want to have that information.
(besides, the semantics are misleading since a new file opened with "w" has also be created, but created() would return False)

There's some bogus indentation in the patch (it uses tab characters in some places).

Also:

+    if not (creating + reading or writing or appending):

Not sure why the "+". Shouldn't it be "or" instead?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sun Jan  8 15:49:19 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 14:49:19 +0000
Subject: [issue13738] Optimize bytes.upper() and lower()
Message-ID: <1326034158.97.0.725746984408.issue13738@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

The current implementation has useless testing and copying.

----------
components: Interpreter Core
files: bytesupper.patch
keywords: patch
messages: 150868
nosy: benjamin.peterson, flox, haypo, pitrou
priority: low
severity: normal
stage: patch review
status: open
title: Optimize bytes.upper() and lower()
type: performance
versions: Python 3.3
Added file: http://bugs.python.org/file24175/bytesupper.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13738>
_______________________________________

From report at bugs.python.org  Sun Jan  8 16:11:18 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sun, 08 Jan 2012 15:11:18 +0000
Subject: [issue13738] Optimize bytes.upper() and lower()
In-Reply-To: <1326034158.97.0.725746984408.issue13738@psf.upfronthosting.co.za>
Message-ID: <1326035478.95.0.83768711727.issue13738@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

LGTM.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13738>
_______________________________________

From report at bugs.python.org  Sun Jan  8 16:25:49 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 15:25:49 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1326030840.3374.5.camel@localhost.localdomain>
Message-ID: <CAH_1eM0_NTJ_NUmoWMBGG8qx095bN8PmjvsoKFVy8SQbF-59=Q@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's a possible walkfd() implementation.

Example:
"""
$ cat /home/cf/testwalkfd.py
import os
import sys

topfd = os.open(sys.argv[1], os.O_RDONLY)

for rootfd, dirs, files in os.walkfd(topfd):
    print(rootfd, dirs, files)
$ ./python ~/testwalkfd.py /etc/apt/
3 ['sources.list.d', 'preferences.d', 'trusted.gpg.d', 'apt.conf.d']
['trustdb.gpg', 'trusted.gpg~', 'sources.list', 'trusted.gpg']
4 [] []
4 [] []
4 [] []
4 [] ['70debconf', '01autoremove', '00trustcdrom']
[44194 refs]
"""

AFAICT, a safe rmtree could be implemented simply with
walkfd(topdown=False), but Antoine's remarks make me thing I missed
something.

> Be aware that you have to manage dirfd's lifetime, which can make things
> interesting.

Basically, this means that doing:
for rootfd, dirs, files in walkfd(topfd):
    print(fstat(rootfd), dirs, files))

is valid whereas

print([(fstat(rootfd), dirs, files) for (rootfd, dirs, files) in
walkfd(topfd)]) isn't.

> Also be aware that symlinks mean sometimes you won't have a dirfd: if
> you have a symlink that points to another directory, you can't open that
> directory using openat from the symlink's directory. So if you follow
> symlinks (or have an option to do so) you must also take that case into
> account.

I'm not sure I understand this. Why "you can't open that directory
using openat from the symlink's directory". Could you elaborate?

----------
keywords: +patch
Added file: http://bugs.python.org/file24176/walkfd.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________
-------------- next part --------------
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -24,6 +24,7 @@
 #'
 
 import sys, errno
+import stat as st
 
 _names = sys.builtin_module_names
 
@@ -151,7 +152,6 @@
     try:
         mkdir(name, mode)
     except OSError as e:
-        import stat as st
         if not (e.errno == errno.EEXIST and exist_ok and path.isdir(name) and
                 st.S_IMODE(lstat(name).st_mode) == _get_masked_mode(mode)):
             raise
@@ -298,6 +298,78 @@
 
 __all__.append("walk")
 
+def _are_same_file(stat1, stat2):
+    """Helper function that checks whether two stat results refer to the same
+    file.
+    """
+    return (stat1.st_mode == stat2.st_mode and stat1.st_ino == stat2.st_ino and
+            stat1.st_dev == stat2.st_dev)
+
+def walkfd(topfd, topdown=True, onerror=None, followlinks=False):
+    """Directory tree generator.
+
+    This behaves exactly like walk(), except that it accepts a file descriptor
+    as top directory, and yields a 3-tuple
+
+        dirfd, dirnames, filenames
+
+    dirfd is a file descriptor referring to the directory.  dirnames is a list
+    of the names of the subdirectories in dirpath (excluding '.' and '..').
+    filenames is a list of the names of the non-directory files in dirpath.
+
+    The advantage of walkfd() over walk() is that it's safe against symlink
+    races (when followlinks is False).
+    """
+    # fdlistdir() closes the passed FD, hence the dup()
+    fd = dup(topfd)
+    try:
+        names = fdlistdir(fd)
+    except error as err:
+        if onerror is not None:
+            onerror(err)
+        return
+
+    # whether to follow symlinks
+    flag = 0 if followlinks else AT_SYMLINK_NOFOLLOW
+
+    dirs, dirmodes, nondirs = [], [], []
+    for name in names:
+        try:
+            orig_st = fstatat(topfd, name, flag)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return
+        if st.S_ISDIR(orig_st.st_mode):
+            # Store the result of stat to check that the file hasn't been
+            # modified when we call walkfd() recursively (symlink race).
+            dirs.append(name)
+            dirmodes.append(orig_st)
+        else:
+            nondirs.append(name)
+
+    if topdown:
+        yield topfd, dirs, nondirs
+    for name, orig_st in zip(dirs, dirmodes):
+        try:
+            dirfd = openat(topfd, name, O_RDONLY)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return
+        try:
+            # To guard against symlinks race, compare with the original stat
+            # result.
+            if followlinks or _are_same_file(orig_st, fstat(dirfd)):
+                for x in walkfd(dirfd, topdown, onerror, followlinks):
+                    yield x
+        finally:
+            close(dirfd)
+    if not topdown:
+        yield topfd, dirs, nondirs
+
+__all__.append("walkfd")
+
 # Make sure os.environ exists, at least
 try:
     environ

From report at bugs.python.org  Sun Jan  8 16:28:31 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 15:28:31 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326036511.09.0.157149720436.issue13735@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

- Can you use pickletools.dis to examine what differs in the pickles' bytecode?
- Does it also apply to 3.3?

> It appears at least some kind of efficiency might be missed out for
> marshall as well.

marshal doesn't use the pickle protocols, so it's unlikely to be affected.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 16:44:56 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 08 Jan 2012 15:44:56 +0000
Subject: [issue13738] Optimize bytes.upper() and lower()
In-Reply-To: <1326034158.97.0.725746984408.issue13738@psf.upfronthosting.co.za>
Message-ID: <E1RjuvX-00066T-Bu@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9683d59170ee by Antoine Pitrou in branch 'default':
Issue #13738: Simplify implementation of bytes.lower() and bytes.upper().
http://hg.python.org/cpython/rev/9683d59170ee

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13738>
_______________________________________

From report at bugs.python.org  Sun Jan  8 16:48:24 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 15:48:24 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <CAH_1eM0_NTJ_NUmoWMBGG8qx095bN8PmjvsoKFVy8SQbF-59=Q@mail.gmail.com>
Message-ID: <1326037622.3374.23.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > Also be aware that symlinks mean sometimes you won't have a dirfd: if
> > you have a symlink that points to another directory, you can't open that
> > directory using openat from the symlink's directory. So if you follow
> > symlinks (or have an option to do so) you must also take that case into
> > account.
> 
> I'm not sure I understand this. Why "you can't open that directory
> using openat from the symlink's directory". Could you elaborate?

Hmm, sorry, I must have misremembered. I thought openat didn't follow
symlinks.

As for the patch, I think there's a problem with the API:

+    This behaves exactly like walk(), except that it accepts a file descriptor
+    as top directory, and yields a 3-tuple
+
+        dirfd, dirnames, filenames

It doesn't tell you to which dirname corresponds dirfd, so you don't
know the path of the directory you are handed (which can be useful for
progress report, error report, or anything else where you need the name
- e.g. making a zip archive of the directory). Also giving the dirnames
without their fds encourages using them by name, not by fd ;-)

Also, walkfd would be easier to use if callable with a str or bytes path
rather than an int fd.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Sun Jan  8 16:48:35 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 15:48:35 +0000
Subject: [issue13738] Optimize bytes.upper() and lower()
In-Reply-To: <1326034158.97.0.725746984408.issue13738@psf.upfronthosting.co.za>
Message-ID: <1326037715.78.0.958515930964.issue13738@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13738>
_______________________________________

From report at bugs.python.org  Sun Jan  8 17:07:58 2012
From: report at bugs.python.org (sbt)
Date: Sun, 08 Jan 2012 16:07:58 +0000
Subject: [issue13673] PyTraceBack_Print() fails if signal received but
	PyErr_CheckSignals() not called
In-Reply-To: <1325108100.49.0.901203078622.issue13673@psf.upfronthosting.co.za>
Message-ID: <1326038878.02.0.912122508184.issue13673@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Trivial 3 lines patch.

I guess there is still a race: if Ctrl-C is pressed after PyErr_CheckSignals() is called but before PyObject_Str() then the printing of any exception can still be suppressed.

----------
Added file: http://bugs.python.org/file24177/traceback_checksignals_2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13673>
_______________________________________

From report at bugs.python.org  Sun Jan  8 17:08:42 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 08 Jan 2012 16:08:42 +0000
Subject: [issue13026] Dis module - documentation of MAKE_FUNCTION
In-Reply-To: <1316642558.64.0.601129278503.issue13026@psf.upfronthosting.co.za>
Message-ID: <1326038922.48.0.151518607059.issue13026@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi Arnaud,
would you like to provide a patch to update the MAKE_FUNCTION opcode description? That would speed up a bit the fix-up.

Also, I don't have that clear what "For each keyword only default, ..." mean: are those the default values for keyword arguments? something else?

Thanks in advance,
Sandro

----------
nosy: +sandro.tosi
stage:  -> needs patch
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13026>
_______________________________________

From report at bugs.python.org  Sun Jan  8 17:26:26 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 16:26:26 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
Message-ID: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>


New submission from Charles-Fran?ois Natali <neologix at free.fr>:

After a call to fdlistdir(), another call to fdlistdir() on the same file handle (but using a different FD, since the FD passed to fdlistdir() is closed) will return an empty list:

"""
$ cat ~/test_fdlistdir.py 
import os
import sys


fd = os.open(sys.argv[1], os.O_RDONLY)

print(os.fdlistdir(os.dup(fd)))
print(os.fdlistdir(os.dup(fd)))

os.close(fd)
$ ./python ~/test_fdlistdir.py /tmp/
['pulse-B1FebW397VI5', 'ksocket-kdm', 'etc', 'kde-cf', 'ksocket-cf', 'test_posix.py', '.X0-lock', 'kde-kdm', 'akonadi-cf.k6y52j', 'ssh-iSFleEAS1243', '.ICE-unix', '.X11-unix']
[]
"""

That's because fdopendir()/readdir doesn't reset the FD offset.
It's documented by POSIX:
"""
The file offset associated with the file descriptor at the time of the call determines which entries are returned.
"""

That's rather suprising (I got bitten while trying to write a test for #13734).
I see two options:
1. rewind the directory stream in fdlistdir()
2. document this

Here's a patch for option 1.

----------
components: None
files: fdlistdir.diff
keywords: patch
messages: 150877
nosy: neologix, pitrou, rosslagerwall
priority: normal
severity: normal
status: open
title: os.fdlistdir() is not idempotent
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24178/fdlistdir.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 17:50:40 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 08 Jan 2012 16:50:40 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1322872228.97.0.525171057572.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326041440.0.0.506595255244.issue13521@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

Bump! There was no activity here for two weeks. Is my patch for 2.7 ok or should I do something more about it?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:18:31 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 17:18:31 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <1326043111.81.0.746626888231.issue13739@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I see two options:
> 1. rewind the directory stream in fdlistdir()
> 2. document this
> 
> Here's a patch for option 1.

Agreed with that, and ok with the patch :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:21:54 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Sun, 08 Jan 2012 17:21:54 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1325871512.71.0.286774610121.issue13721@psf.upfronthosting.co.za>
Message-ID: <1326043314.86.0.512929381574.issue13721@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

I won't claim to know more about socket error codes than what the Linux man pages says. According to them only send() can fail with ECONNRESET, even though POSIX Programmer's Manual man pages mentions many others. getpeername() is however not documented as being able to return ECONNRESET, and ENOTCONN is apparently the most appropriate error code for getpeername() both on linux (3) and posix (3p).

So: ENOTCONN from getpeername just means that the connection isn't connected. It doesn't indicate that no connection attempt has been mode, and the use of getpeername in SSLSocket.__init__ is thus not usable for checking "if the underlying socket isn?t connected yet".

The wrap_socket API promises to be able to wrap both connected and unconnected sockets without being told explicit what the programmer intended. A system network socket knows if it is unused or failed, but I am not aware of any reliable cross-platform way to observe that (but that doesn't mean that none exist). The only way to reliably implement the documented wrap_socket API might thus be to maintain a flag in PySocketSockObject.

Introducing a new and more explicit way of wrapping connected sockets might be a simpler and more stable solution.

>From another perspective: Any user of sockets must be aware that socket operations can fail at any time. It might thus not be a problem that wrap_socket fails to fail, as long as the programmer knows how to catch the failure in the next operation. From that point of view the problem is that it is surprising and undocumented how getpeercert can fail.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:25:08 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 17:25:08 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326043508.95.0.025794266015.issue12760@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's a new version of the patch that should address all the comments.

----------
Added file: http://bugs.python.org/file24179/open_create_x-3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:32:07 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 08 Jan 2012 17:32:07 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <1326043927.92.0.78979483432.issue13739@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

> I see two options:
> 1. rewind the directory stream in fdlistdir()
> 2. document this
> 
> Here's a patch for option 1.

Yeah, looks good.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:34:44 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 08 Jan 2012 17:34:44 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <E1Rjwdm-0005Nf-0o@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 7b2a178c028b by Charles-Fran?ois Natali in branch 'default':
Issue #13739: In os.listdir(), rewind the directory stream (so that listdir()
http://hg.python.org/cpython/rev/7b2a178c028b

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:39:12 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 17:39:12 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1326043508.95.0.025794266015.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326044272.3374.24.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Here's a new version of the patch that should address all the comments.

Just a small note: FileExistsError is raised, not exactly OSError, when
the file exists.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:51:00 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 17:51:00 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1326043314.86.0.512929381574.issue13721@psf.upfronthosting.co.za>
Message-ID: <1326044979.3374.27.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> The only way to reliably implement the documented wrap_socket API
> might thus be to maintain a flag in PySocketSockObject.

Agreed. With the annoyance that the flag must be exposed to Python code,
since ssl's wrap_socket is written in Python. It may be private, though.

> Introducing a new and more explicit way of wrapping connected sockets
> might be a simpler and more stable solution.

I'm a bit wary of API bloat here.

> From another perspective: Any user of sockets must be aware that
> socket operations can fail at any time. It might thus not be a problem
> that wrap_socket fails to fail, as long as the programmer knows how to
> catch the failure in the next operation. From that point of view the
> problem is that it is surprising and undocumented how getpeercert can
> fail.

Thanks. So fixing how getpeercert behaves and either raise a dedicated
error or return None would improve things here, right?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Sun Jan  8 18:58:44 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 17:58:44 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1322872228.97.0.525171057572.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326045524.37.0.428771253512.issue13521@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Looking at the patch again, I think this isn't enough.
setdefault() will still call the lookup routine twice which, in the general case (i.e. lookdict() not lookdict_unicode()), can call arbitrary Python code through e.g. __eq__ methods.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:07:42 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 08 Jan 2012 18:07:42 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <E1Rjx9h-0007lj-F8@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 36f2e236c601 by Charles-Fran?ois Natali in branch 'default':
Issue #13739: It's simpler and more direct to call rewinddir() at the
http://hg.python.org/cpython/rev/36f2e236c601

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:18:53 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 08 Jan 2012 18:18:53 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326046733.87.0.172346677278.issue10278@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> on linux the underlying functionality is implemented in librt; the extension doesn't check for this or links with -lrt.

The changeset 35e4b7c4bafa changed configure.in to check clock_gettime(). It checks without and with librt:

     7.7 +AC_CHECK_FUNCS(clock_gettime, [], [
     7.8 +    AC_CHECK_LIB(rt, clock_gettime, [
     7.9 +        AC_DEFINE(HAVE_CLOCK_GETTIME, 1)
    7.10 +        AC_DEFINE(TIMEMODULE_LIB, [rt],
    7.11 +                  [Library needed by timemodule.c: librt may be needed for clock_gettime()])
    7.12 +    ])
    7.13 +])

wallclock-3.patch checks for HAVE_CLOCK_GETTIME, which does use librt in its test (see above).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:19:32 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Sun, 08 Jan 2012 18:19:32 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1326044979.3374.27.camel@localhost.localdomain>
Message-ID: <4F09DE2F.4060304@kiilerich.com>


Mads Kiilerich <mads at kiilerich.com> added the comment:

> I'm a bit wary of API bloat here.

Yes, but explicit is better than magic ...

> Thanks. So fixing how getpeercert behaves and either raise a dedicated
> error or return None would improve things here, right?

Well ... that would at least make it theoretically possible to claim 
that it works as intended ;-)

A counter argument could be that retrieving the certificate that already 
has been used for negotiation isn't a socket operation. It would make 
sense to be able to look at it even after the socket has been closed. 
 From that point of view _sslobj should be kept "forever".

A return value of None would still not indicate if we had a working 
connection without certificate or a failed connection. That would be 
annoying.

My primary concern with my Mercurial hat on is to get the documentation 
updated so we know how to write code that works correctly also with 
previous Python versions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:32:41 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 18:32:41 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <4F09DE2F.4060304@kiilerich.com>
Message-ID: <1326047480.3374.28.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> A return value of None would still not indicate if we had a working 
> connection without certificate or a failed connection. That would be 
> annoying.

Ah, right. Then raising e.g. a ValueError would be better.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:41:41 2012
From: report at bugs.python.org (Kay Hayen)
Date: Sun, 08 Jan 2012 18:41:41 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326048101.95.0.077244290503.issue13735@psf.upfronthosting.co.za>


Kay Hayen <kayhayen at gmx.de> added the comment:

It seems that there is an extra "BINPUT 2", whatever it does. I am attaching a variant that does pickletools.dis on the 3 dumps.

Protocol 2 :
Dumping read const const stream '\x80\x02}q\x01U\x07modulesq\x02Ns.'
    0: \x80 PROTO      2
    2: }    EMPTY_DICT
    3: q    BINPUT     1
    5: U    SHORT_BINSTRING 'modules'
   14: q    BINPUT     2
   16: N    NONE
   17: s    SETITEM
   18: .    STOP
highest protocol among opcodes = 2
Dumping load const const stream '\x80\x02}q\x01U\x07modulesNs.'
    0: \x80 PROTO      2
    2: }    EMPTY_DICT
    3: q    BINPUT     1
    5: U    SHORT_BINSTRING 'modules'
   14: N    NONE
   15: s    SETITEM
   16: .    STOP
highest protocol among opcodes = 2
Dumping load const const stream '\x80\x02}q\x01U\x07modulesNs.'
    0: \x80 PROTO      2
    2: }    EMPTY_DICT
    3: q    BINPUT     1
    5: U    SHORT_BINSTRING 'modules'
   14: N    NONE
   15: s    SETITEM
   16: .    STOP
highest protocol among opcodes = 2

----------
Added file: http://bugs.python.org/file24180/stream.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:46:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 18:46:53 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326048413.52.0.0841153417296.issue13735@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Ah, right. BINPUT is used to memoize objects in case a restructive structure happens. You can use pickletools.optimize() to remove the useless BINPUTs out of a pickle stream.

Note that 3.x is more consistent and always emits the BINPUTs. It seems 2.x's cPickle (which is really a weird piece of code) may do some premature optimization here.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:46:59 2012
From: report at bugs.python.org (Kay Hayen)
Date: Sun, 08 Jan 2012 18:46:59 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326048419.41.0.519131718607.issue13735@psf.upfronthosting.co.za>


Kay Hayen <kayhayen at gmx.de> added the comment:

Sending my attached file "stream.py" through "2to3.py" it shows that CPython 3.2 doesn't exihibit the issue for either protocol, which may be because it's now "unicode" key, but as it's the only value I tried, I can't tell.

Hope this helps.

Regarding the "marshal", I presume, that somehow the dictionary when created via "marshal" (or compile, if no ".pyc" is involved?) first time is somehow less efficient to determine/stream that the one "cPickle" created.

My assumption is that somehow, when creating the dictionary from cPickle, it is different (has to be), and that's interesting. And of course the easier to stream dictionary may be the nicer one at runtime as well. But that is just my speculation.

Yours,
Kay

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 19:49:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 18:49:25 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326048565.45.0.0213523607706.issue13735@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think the "premature" optimization comes down to that code:

"""
static int
put(Picklerobject *self, PyObject *ob)
{
    if (Py_REFCNT(ob) < 2 || self->fast)
        return 0;

    return put2(self, ob);
}
"""

(put2() being the function which emits BINPUT)

When you unpickle a dict, its contents are created anew and therefore the refcount is 1 => next pickling avoids emitting a BINPUT.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:02:54 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 19:02:54 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1326048419.41.0.519131718607.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326049293.3374.31.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Regarding the "marshal", I presume, that somehow the dictionary when
> created via "marshal" (or compile, if no ".pyc" is involved?) first
> time is somehow less efficient to determine/stream that the one
> "cPickle" created.

I don't think so.  marshal uses a much simpler and less general
serialization protocol, so it should theoretically (!) be faster as
well.
For example, I don't think marshal supports recursive structures.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:03:17 2012
From: report at bugs.python.org (Kay Hayen)
Date: Sun, 08 Jan 2012 19:03:17 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326049397.34.0.277245786524.issue13735@psf.upfronthosting.co.za>


Kay Hayen <kayhayen at gmx.de> added the comment:

I see, if it's refcount dependent, that explains why it changes from interpreter provided dictionary and self-created one.

So, I take, I should always call "pickletools.optimize( cPickle.dumps( value ))" then.

Thanks,
Kay

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:05:30 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 19:05:30 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326049530.31.0.188433876417.issue13735@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Closing the issue as it's really a quirk rather than a bug. Thanks for reporting, though.

----------
resolution:  -> wont fix
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:14:47 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 19:14:47 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <1326050087.71.0.970632965681.issue13739@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

For some reason, the second changeset broke the OpenIndiana buildbots:
http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/builds/2485

Also, wouldn't it be better to call rewinddir with the GIL released?
(although I agree rewinddir shouldn't be expensive)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:32:12 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 19:32:12 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326050087.71.0.970632965681.issue13739@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2y8AfhHjtA4N87itVYcx0GTrDdHpxTC-gqQiDi6Ga1Hw@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> For some reason, the second changeset broke the OpenIndiana buildbots:
>

I have absolutely no idea of why this doesn't work. I suspect
rewinddir() is a noop on OpenIndiana if readdir() hasn't been called.
I'll revert this commit.

> Also, wouldn't it be better to call rewinddir with the GIL released?
> (although I agree rewinddir shouldn't be expensive)

rewinddir() just changes a pointer, and calls - or ought to call -
lseek() on the FD. This should be fast, since no I/O is involved
(lseek() is not documented to return EINTR, for example). Releasing
the GIL has a cost :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:43:14 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 19:43:14 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <1269104932.24.0.802901286813.issue8184@psf.upfronthosting.co.za>
Message-ID: <1326051794.53.0.921653759311.issue8184@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's a patch addressing the multiple bind() problem on Windows.

Note that this problem also affects other parts of the stdlib, which use SO_REUSEADDR when available.

Also, there's an rather confusing comment in support.find_unused_port():

"""
    (This is easy to reproduce on Windows, unfortunately, and can be traced to
    the SO_REUSEADDR socket option having different semantics on Windows versus
    Unix/Linux.  On Unix, you can't have two AF_INET SOCK_STREAM sockets bind,
    listen and then accept connections on identical host/ports.  An EADDRINUSE
    socket.error will be raised at some point (depending on the platform and
    the order bind and listen were called on each socket).

    However, on Windows, if SO_REUSEADDR is set on the sockets, no EADDRINUSE
    will ever be raised when attempting to bind two identical host/ports. When
    accept() is called on each socket, the second caller's process will steal
    the port from the first caller, leaving them both in an awkwardly wedged
    state where they'll no longer respond to any signals or graceful kills, and
    must be forcibly killed via OpenProcess()/TerminateProcess().

    The solution on Windows is to use the SO_EXCLUSIVEADDRUSE socket option
    instead of SO_REUSEADDR, which effectively affords the same semantics as
    SO_REUSEADDR on Unix.  Given the propensity of Unix developers in the Open
    Source world compared to Windows ones, this is a common mistake.  A quick
    look over OpenSSL's 0.9.8g source shows that they use SO_REUSEADDR when
    openssl.exe is called with the 's_server' option, for example. See
    http://bugs.python.org/issue2550 for more info.  The following site also
    has a very thorough description about the implications of both REUSEADDR
    and EXCLUSIVEADDRUSE on Windows:
    http://msdn2.microsoft.com/en-us/library/ms740621(VS.85).aspx)
"""

I have no idea why it uses SO_REUSEADDR + SO_EXCLUSIVEADDRUSE instead of just keeping the default setting, since we don't want to allow multiple bind(), but bypass the TIME-WAIT lingering.

----------
Added file: http://bugs.python.org/file24181/connection_multiple_bind.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Sun Jan  8 20:44:27 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 08 Jan 2012 19:44:27 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326051867.92.0.677106025512.issue12760@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Just a small note: FileExistsError is raised, not exactly OSError,
> when the file exists.

I've updated the doc accordingly.

----------
stage: patch review -> commit review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sun Jan  8 21:40:38 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 20:40:38 +0000
Subject: [issue13026] Dis module - documentation of MAKE_FUNCTION
In-Reply-To: <1316642558.64.0.601129278503.issue13026@psf.upfronthosting.co.za>
Message-ID: <1326055238.02.0.19378700044.issue13026@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

"For each keyword only default" is not in my text, but if you are referring to "keyword only name, default parameter object pairs ", yes. Here is a revised suggested wording:

MAKE_FUNCTION(argc)
Pushes a new function object on the stack. From bottom to top, the consumed stack must have have (argc & 0xFF) default argument objects in positional order, ((argc >> 8) & 0xFF) (name, default argument) pairs, with name just below the object on the stack, for keyword-only parameters, ((argc >> 16) & 0x7FFF) parameter annotation objects, a tuple listing the parameter names for the annotations, and finally a code object as TOS.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13026>
_______________________________________

From report at bugs.python.org  Sun Jan  8 21:51:46 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 08 Jan 2012 20:51:46 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1326055906.96.0.795836129307.issue13737@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I disagree that this is a security issue, or an issue at all. All source code of the site is in a public subversion repository, available for review to any attacker (as well as any security review) - and that is deliberately so because we fundamentally believe in openness of source code.

I fail to see why making the traceback available would pose any additional threat. Having the traceback is lightly helpful when people actually do encounter bugs and report them.

----------
priority: high -> normal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:12:59 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 21:12:59 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326057179.03.0.679480715571.issue13122@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

http://docs.python.org/release/2.6.7/
has the same problem that the 3.x 'in development' version is called 3.2, while there is no link to current stable 3.2.2 except indirectly by going to 'old versions' (which perhaps should be 'other versions'). I do not know enough about frames to know how to future-proof the sidebar better.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:15:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 21:15:39 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326057339.71.0.836605649843.issue13122@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Nobody said 3.2 was not stable...

Well, the sidebar says it's in development while 2.7 is stable :)

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:18:46 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 21:18:46 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326057526.33.0.736124726566.issue13122@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Well, actually I do have an idea. When 3.3 comes out, I think docs.python.org should point to that, and another url should point to the most recent 2.7.x release. The url for all versions is already stable. Then the sidebar could have entries for most recent 3.x and 2.x releases (without specifying) and other versions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:29:02 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 21:29:02 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326058142.09.0.395885901793.issue13122@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

My specific suggestion is that the sidebar be time-independent (for at least a decade, until Python 4 ;-) and say

Docs for other versions
  Current Python 3
  Current Python 2
  In development
  Everything else

with stable links for each.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:31:04 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 08 Jan 2012 21:31:04 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1322872228.97.0.525171057572.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326058263.99.0.416110838489.issue13521@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

I understand you are talking about this call:

mp->ma_lookup(mp, key, hash);

I haven't noticed that earlier. I'll try to provide a better fix (for 2.7 first, after we agree it's good enough, I will provide one for 3.3).

Do you have any idea, how I might this part? I mean how to check, that this is called only once?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:33:48 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 08 Jan 2012 21:33:48 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326058428.01.0.475679882678.issue13122@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Sounds good.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:42:17 2012
From: report at bugs.python.org (Atle Pedersen)
Date: Sun, 08 Jan 2012 21:42:17 +0000
Subject: [issue13717] print fails on unicode '\udce5' surrogates not allowed
In-Reply-To: <1325794375.32.0.250961574863.issue13717@psf.upfronthosting.co.za>
Message-ID: <1326058937.65.0.667624750854.issue13717@psf.upfronthosting.co.za>


Atle Pedersen <atle.pedersen at gmail.com> added the comment:

Just wanted to say thanks for very fast response, and informative information.

I respect your decision to close the bug as invalid. But my five cent is that it still feels like a bug, something that shouldn't happen. Especially since it's part of a very basic function, and very unpredictable for inexperienced Python programmers.

I do understand your headache. I've had my share of character set issues in my time.

But thanks again for the quick reply, and suggested workarounds, which will work well for me and my situation.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13717>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:42:51 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 08 Jan 2012 21:42:51 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326058971.11.0.934785873807.issue13122@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

To make my idea work both now and after we switch docs.python.org, that url should not be used as either of the first two links. Rather there should be a docs.python.org/py2k (or whatever) that is the permanent 'latest Python 2 release' docs link.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:44:53 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Sun, 08 Jan 2012 21:44:53 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326059093.43.0.30779146856.issue13642@psf.upfronthosting.co.za>


Michele Orr? <maker.py at gmail.com> added the comment:

There's no need to port your patch over python3k, since urllib behaves differently with http passwords - as you can see in the doc http://docs.python.org/dev/py3k/library/urllib.request.html#examples 

I would be glad to finish your password on 2.7 as soon as possible, joneskoo. Thanks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:45:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 08 Jan 2012 21:45:28 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1326058263.99.0.416110838489.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326058858.3374.34.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Do you have any idea, how I might this part? I mean how to check, that
> this is called only once?

Checking that __eq__ is called only once should be a good proxy.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:47:34 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 08 Jan 2012 21:47:34 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1322872228.97.0.525171057572.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326059254.65.0.533948618961.issue13521@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

Thanks, I'll try that. Like before I will probably find time next weekend.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan  8 22:47:45 2012
From: report at bugs.python.org (Jelle Geerts)
Date: Sun, 08 Jan 2012 21:47:45 +0000
Subject: [issue13740] winsound.SND_NOWAIT ignored on modern Windows platforms
Message-ID: <1326059265.01.0.0567556069676.issue13740@psf.upfronthosting.co.za>


New submission from Jelle Geerts <jellegeerts at gmail.com>:

As the attached example Python script explains, the winsound.SND_NOWAIT flag doesn't do anything, at least not on modern Windows platforms.

The updated MSDN documentation for PlaySound() states that SND_NOWAIT is not supported (it is ignored).
Link: http://msdn.microsoft.com/en-us/library/windows/desktop/dd743680%28v=vs.85%29.aspx

Assuming that the flag is supported on older Windows versions, I propose that the Python documentation is updated to state the same thing for SND_NOWAIT that it states for the SND_PURGE flag, which is the following:
"Note: This flag is not supported on modern Windows platforms."

Even though it might be that SND_NOWAIT was in fact never supported on any Windows version, the proposed change is at least better than not doing anything about it.

----------
components: Windows
files: example.py
messages: 150915
nosy: bughunter2
priority: normal
severity: normal
status: open
title: winsound.SND_NOWAIT ignored on modern Windows platforms
type: behavior
Added file: http://bugs.python.org/file24182/example.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13740>
_______________________________________

From report at bugs.python.org  Mon Jan  9 00:41:21 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 08 Jan 2012 23:41:21 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326066081.89.0.0296311740486.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Thanks for that Charles-Fran?ois - do you mind if I adapt that for walkdir?

The changes I would make are basically those that Antoine pointed out:
- rather than replacing the dirpath entry, instead yield a 4-tuple that appends the dirfd attribute at the end*
- if the dup(fd) fails, fall back to assuming top is a string or bytes path

*I'm still interested in opinions on this aspect. I see 5 main possibilities:
- (dirfd, subdirs, files) triple (problematic for the reasons Antoine pointed out)
- (dirpath, subdirs, files, dirfd) 4-tuple
- ((dirpath, dirfd), subdirs, files) nested tuple
- (dirpath, subdirs, files) tuple subclass with separate dirfd attribute
- (dirpath, subdirs, files) triple with dirpath as a str subclass with a separate fd attribute

I'm currently leaning towards the simple 4-tuple approach - it's simple, explicit and the walkdir pipeline operations can easily accept either underlying iterable by using indexing operations rather than tuple unpacking (a change I already planned to make so that the pipeline passed along the objects from the underlying iterable, anyway)

(I'll also need to create ctypes-based variants of all the relevant *at functions, since the stdlib wrappers won't be available in existing versions of Python)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Mon Jan  9 01:47:15 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Mon, 09 Jan 2012 00:47:15 +0000
Subject: [issue13721] ssl.wrap_socket on a connected but failed connection
	succeeds and .peer_certificate gives AttributeError
In-Reply-To: <1325871512.71.0.286774610121.issue13721@psf.upfronthosting.co.za>
Message-ID: <1326070035.91.0.353106595321.issue13721@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

I would find ValueError surprising here. socket.error or SSLError would be less surprising ... even though it technically isn't completely true. But isn't it more like a kind of RuntimeError to call getpeercert after the socket has been closed?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13721>
_______________________________________

From report at bugs.python.org  Mon Jan  9 04:29:39 2012
From: report at bugs.python.org (Bithin A)
Date: Mon, 09 Jan 2012 03:29:39 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1326079779.19.0.121458241643.issue13737@psf.upfronthosting.co.za>


Bithin A <bithin2007 at gmail.com> added the comment:

The bugs.python.org/review is a running application and it is very bad to see debug error messages.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Mon Jan  9 04:54:38 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Mon, 09 Jan 2012 03:54:38 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326081278.99.0.32586243583.issue13734@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

> I'm currently leaning towards the simple 4-tuple approach

I would also take that approach. It seems simplest to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Mon Jan  9 06:22:34 2012
From: report at bugs.python.org (Andrew)
Date: Mon, 09 Jan 2012 05:22:34 +0000
Subject: [issue13741] *** glibc detected *** python: double free or corruption
	(!prev): 0x0000000001d53ad0 ***
Message-ID: <1326086552.75.0.47475679765.issue13741@psf.upfronthosting.co.za>


New submission from Andrew <awnystrom at gmail.com>:

Hello,
This program crashes after 12-24 hours of running.  My OS is Ubuntu 11.10, I'm using Python 2.7.2, and gcc 4.6.1.

Here's the error:

*** glibc detected *** python: double free or corruption (!prev): 0x0000000001d53ad0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x78a96)[0x7fbd28c21a96]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fbd28c25d7c]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0xd6aa)[0x7fbd27c806aa]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0xde51)[0x7fbd27c80e51]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0x13f9b)[0x7fbd27c86f9b]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0x146c6)[0x7fbd27c876c6]
python(PyEval_EvalFrameEx+0x2f9)[0x4b6569]
python(PyEval_EvalFrameEx+0xb07)[0x4b6d77]
======= Memory map: ========
00400000-00633000 r-xp 00000000 08:06 1901460                            /usr/bin/python2.7
00832000-00833000 r--p 00232000 08:06 1901460                            /usr/bin/python2.7
00833000-0089c000 rw-p 00233000 08:06 1901460                            /usr/bin/python2.7
0089c000-008ae000 rw-p 00000000 00:00 0 
01b30000-01f26000 rw-p 00000000 00:00 0                                  [heap]
7fbd1dffc000-7fbd1dffd000 ---p 00000000 00:00 0 
7fbd1dffd000-7fbd1e7fd000 rw-p 00000000 00:00 0 
7fbd1e7fd000-7fbd1e7fe000 ---p 00000000 00:00 0 
7fbd1e7fe000-7fbd1effe000 rw-p 00000000 00:00 0 
7fbd1effe000-7fbd1efff000 ---p 00000000 00:00 0 
7fbd1efff000-7fbd1f7ff000 rw-p 00000000 00:00 0 
7fbd1f7ff000-7fbd1f800000 ---p 00000000 00:00 0 
7fbd1f800000-7fbd20000000 rw-p 00000000 00:00 0 
7fbd20000000-7fbd2011e000 rw-p 00000000 00:00 0 
7fbd2011e000-7fbd24000000 ---p 00000000 00:00 0 
7fbd24411000-7fbd24426000 r-xp 00000000 08:06 1754089                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fbd24426000-7fbd24625000 ---p 00015000 08:06 1754089                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fbd24625000-7fbd24626000 r--p 00014000 08:06 1754089                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fbd24626000-7fbd24627000 rw-p 00015000 08:06 1754089                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fbd24627000-7fbd24628000 ---p 00000000 00:00 0 
7fbd24628000-7fbd24e28000 rw-p 00000000 00:00 0 
7fbd24e28000-7fbd24e34000 r-xp 00000000 08:06 1754109                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fbd24e34000-7fbd25033000 ---p 0000c000 08:06 1754109                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fbd25033000-7fbd25034000 r--p 0000b000 08:06 1754109                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fbd25034000-7fbd25035000 rw-p 0000c000 08:06 1754109                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
7fbd25035000-7fbd2503f000 r-xp 00000000 08:06 1754113                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fbd2503f000-7fbd2523f000 ---p 0000a000 08:06 1754113                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fbd2523f000-7fbd25240000 r--p 0000a000 08:06 1754113                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fbd25240000-7fbd25241000 rw-p 0000b000 08:06 1754113                    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
7fbd25241000-7fbd25258000 r-xp 00000000 08:06 1754103                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7fbd25258000-7fbd25457000 ---p 00017000 08:06 1754103                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7fbd25457000-7fbd25458000 r--p 00016000 08:06 1754103                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7fbd25458000-7fbd25459000 rw-p 00017000 08:06 1754103                    /lib/x86_64-linux-gnu/libnsl-2.13.so
7fbd25459000-7fbd2545b000 rw-p 00000000 00:00 0 
7fbd2545b000-7fbd25463000 r-xp 00000000 08:06 1754105                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fbd25463000-7fbd25662000 ---p 00008000 08:06 1754105                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fbd25662000-7fbd25663000 r--p 00007000 08:06 1754105                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fbd25663000-7fbd25664000 rw-p 00008000 08:06 1754105                    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
7fbd25664000-7fbd2566b000 r-xp 00000000 08:06 1754132                    /lib/x86_64-linux-gnu/librt-2.13.so
7fbd2566b000-7fbd2586a000 ---p 00007000 08:06 1754132                    /lib/x86_64-linux-gnu/librt-2.13.so
7fbd2586a000-7fbd2586b000 r--p 00006000 08:06 1754132                    /lib/x86_64-linux-gnu/librt-2.13.so
7fbd2586b000-7fbd2586c000 rw-p 00007000 08:06 1754132                    /lib/x86_64-linux-gnu/librt-2.13.so
7fbd2586c000-7fbd2587a000 r-xp 00000000 08:06 1982536                    /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
7fbd2587a000-7fbd25a7a000 ---p 0000e000 08:06 1982536                    /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
7fbd25a7a000-7fbd25a7b000 r--p 0000e000 08:06 1982536                    /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
7fbd25a7b000-7fbd25a7c000 rw-p 0000f000 08:06 1982536                    /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
7fbd25a7c000-7fbd25b3d000 rw-p 00000000 00:00 0 
7fbd25b3d000-7fbd25b40000 r-xp 00000000 08:06 1754095                    /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fbd25b40000-7fbd25d3f000 ---p 00003000 08:06 1754095                    /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fbd25d3f000-7fbd25d40000 r--p 00002000 08:06 1754095                    /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fbd25d40000-7fbd25d41000 rw-p 00003000 08:06 1754095                    /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fbd25d41000-7fbd25d51000 r-xp 00000000 08:06 8518                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.11
7fbd25d51000-7fbd25f50000 ---p 00010000 08:06 8518                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.11
7fbd25f50000-7fbd25f51000 r--p 0000f000 08:06 8518                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.11
7fbd25f51000-7fbd25f52000 rw-p 00010000 08:06 8518                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.11
7fbd25f52000-7fbd25fcc000 r-xp 00000000 08:06 1754091                    /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fbd25fcc000-7fbd261cc000 ---p 0007a000 08:06 1754091                    /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fbd261cc000-7fbd261cd000 r--p 0007a000 08:06 1754091                    /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fbd261cd000-7fbd261d0000 rw-p 0007b000 08:06 1754091                    /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fbd261d0000-7fbd26271000 r-xp 00000000 08:06 8379                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.16.14
7fbd26271000-7fbd26470000 ---p 000a1000 08:06 8379                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.16.14
7fbd26470000-7fbd26477000 r--p 000a0000 08:06 8379                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.16.14
7fbd26477000-7fbd26478000 Aborted

----------
files: server.py
messages: 150920
nosy: BlendsInWell
priority: normal
severity: normal
status: open
title: *** glibc detected *** python: double free or corruption (!prev): 0x0000000001d53ad0 ***
type: crash
versions: Python 2.7
Added file: http://bugs.python.org/file24183/server.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13741>
_______________________________________

From report at bugs.python.org  Mon Jan  9 08:12:53 2012
From: report at bugs.python.org (Peter Csapo)
Date: Mon, 09 Jan 2012 07:12:53 +0000
Subject: [issue11990] redirected output - stdout writes newline as \n in
	windows
In-Reply-To: <1304458030.29.0.258748957891.issue11990@psf.upfronthosting.co.za>
Message-ID: <1326093173.73.0.463717512877.issue11990@psf.upfronthosting.co.za>


Peter Csapo <astrobuf at gmail.com> added the comment:

I have having the same issues as Jimbofbx. This seems to stem from changes due to issue 10841. All stdio is now opened in binary mode, in consideration that it is the TextIOWrapper's job to do endline translation.

The problem here is that the newline mode = '\n' for the TextIOWrapper created for stdout. ( see pythonrun.c: create_stdio() ). For windows, the newline mode for stdin is already set to null enabling universal newline translation on input, and it should be set to null for newline transation on output as well.

OLD CODE
newline = "\n";
#ifdef MS_WINDOWS
if (!write_mode) {
    /* translate \r\n to \n for sys.stdin on Windows */
    newline = NULL;
}
#endif


FIXED??? CODE
newline = "\n";
#ifdef MS_WINDOWS
/* translate \r\n to \n for sys.stdin on Windows */
/* translate \n to \r\n for sys.stdout on Windows */
newline = NULL;
#endif

----------
nosy: +astrobuf

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11990>
_______________________________________

From report at bugs.python.org  Mon Jan  9 08:15:27 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 07:15:27 +0000
Subject: [issue9116] test_capi.test_no_FatalError_infinite_loop crash on
	Windows
In-Reply-To: <1277827107.42.0.0793977492373.issue9116@psf.upfronthosting.co.za>
Message-ID: <1326093327.44.0.234841267348.issue9116@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I just got 'fatal error' running python_d 3.3.a0 checked out yesterday. When I closed message, test continued (but really crashed later).

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9116>
_______________________________________

From report at bugs.python.org  Mon Jan  9 08:16:16 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 07:16:16 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <1326093376.25.0.228636710257.issue11906@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

http://docs.python.org/devguide/runtests.html
"If you don?t have easy access to a command line, you can run the test suite from a Python or IDLE shell:
>>> from test import autotest"

However, argparse is the least of the test suite problems on Windows.
(#9116, #11732, other problems)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Mon Jan  9 09:02:41 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Mon, 09 Jan 2012 08:02:41 +0000
Subject: [issue11633] regression: print buffers output when end=''
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326096161.29.0.146492820244.issue11633@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

I've tried to switch to Python 3 once more and stumbled upon this problem once more.

Seems like this regression got stale. Last Victor's proposal seems reasonable for me. Should we open a new, more clear bug report and close this one?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 09:05:31 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 09 Jan 2012 08:05:31 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1326081278.99.0.32586243583.issue13734@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2EZawpBrzwzk+LAEek3CqEHOiTJhYeXzaaT2vE-xA1ug@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Hmm, sorry, I must have misremembered. I thought openat didn't follow
> symlinks.

OK, I was afraid I had missed something.

 > As for the patch, I think there's a problem with the API

Yes, it was really a proof-of-concept, the directory names are missing.

> Also, walkfd would be easier to use if callable with a str or bytes path
> rather than an int fd.

Agreed.

> Also giving the dirnames without their fds encourages using them
> by name, not by fd ;-)

Well, that's not easy:
- right now, the code uses O(depth of directory hierarchy) FDs
- returning FDs for sub-directories would require O(number of nodes in
the hierarchy), or at least O(max number of child nodes): I fear we'll
run out of FDs quite easily
Also, I don't think it's really a problem, since you have to use the
names anyway. The *at() family accepts a FD as a pointer to the
containing directory, but the target entries are accessed by name.
For example, to perform a safe rmtree, you  would do something like:

for dirfd, dirs, files in os.walkfd(topfd, topdown=False):
    for file in files:
        os.unlinkat(dirfd, file)
    for dir in dirs:
        os.unlinkat(dirfd, dir, os.AT_REMOVEDIR)

> Thanks for that Charles-Fran?ois - do you mind if I adapt that for walkdir?

Of course not, go ahead. I'll update walkfd() accordingly, and write
doc and test for it.
By the way, do you plan to get walkdir merged in 3.3?
I've been doing a lot of sys-admin scripts lately, and this would be
really helpful.

> I'm currently leaning towards the simple 4-tuple approach

Sounds good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Mon Jan  9 09:26:35 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 09 Jan 2012 08:26:35 +0000
Subject: [issue13741] *** glibc detected *** python: double free or corruption
	(!prev): 0x0000000001d53ad0 ***
In-Reply-To: <1326086552.75.0.47475679765.issue13741@psf.upfronthosting.co.za>
Message-ID: <1326097595.33.0.5222669788.issue13741@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0xd6aa)[0x7fbd27c806aa]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0xde51)[0x7fbd27c80e51]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0x13f9b)[0x7fbd27c86f9b]
/usr/lib/python2.7/dist-packages/psycopg2/_psycopg.so(+0x146c6)[0x7fbd27c876c6]

Looks like you should report this to Psyco, no?

Also, you can read this on the project's main page:
"""
16 July 2010

Python 2.7 is unsupported so far. Anyone interested in porting Psyco to it is welcome. I started the work in a branch but it needs finishing.
"""

----------
nosy: +neologix
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13741>
_______________________________________

From report at bugs.python.org  Mon Jan  9 10:35:36 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 09 Jan 2012 09:35:36 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326101736.61.0.543922607751.issue13703@psf.upfronthosting.co.za>


Changes by Zbyszek Szmek <zbyszek at in.waw.pl>:


----------
nosy: +zbysz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan  9 11:35:42 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 09 Jan 2012 10:35:42 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
Message-ID: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>


New submission from Simon Sapin <simon.sapin at kozea.fr>:

Hi,

The attached patch adds a 'key' optional parameter to the heapq.merge function that behaves as in sorted().

Related discussion: http://mail.python.org/pipermail/python-ideas/2012-January/013295.html

This is my first contribution to CPython.

----------
components: Library (Lib)
files: heapq_merge_key.patch
keywords: patch
messages: 150927
nosy: SimonSapin, rhettinger
priority: normal
severity: normal
status: open
title: Add a key parameter (like sorted) to heapq.merge
type: enhancement
versions: Python 3.3, Python 3.4
Added file: http://bugs.python.org/file24184/heapq_merge_key.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 11:43:36 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 09 Jan 2012 10:43:36 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326105816.66.0.903535644472.issue13742@psf.upfronthosting.co.za>


Simon Sapin <simon.sapin at kozea.fr> added the comment:

The attached script benchmarks the basline (current implementation) against 3 new implementations, as suggested on http://mail.python.org/pipermail/python-ideas/2012-January/013296.html

On my machine, the output is:

    merge_baseline
    per run, min of 3 = 7.527 ms
    
    merge_1
    per run, min of 3 = 9.894 ms
    131.449 % of baseline
    
    merge_2
    per run, min of 3 = 7.948 ms
    105.594 % of baseline
    
    merge_3
    per run, min of 3 = 7.581 ms
    100.716 % of baseline

On this particular input, merge_2 adds 6% of overhead when the key parameter is not used. While merge_3 only adds 1% of overhead, it almost doubles the amount of code. (Which was admittedly not that long to begin with.)

The patch in the previous message is with the merge_2 implementation, which seemed like the best compromise to me.

----------
Added file: http://bugs.python.org/file24185/benchmark_heapq_merge.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 11:46:59 2012
From: report at bugs.python.org (Andrew)
Date: Mon, 09 Jan 2012 10:46:59 +0000
Subject: [issue13741] *** glibc detected *** python: double free or corruption
	(!prev): 0x0000000001d53ad0 ***
In-Reply-To: <1326086552.75.0.47475679765.issue13741@psf.upfronthosting.co.za>
Message-ID: <1326106019.13.0.81141525454.issue13741@psf.upfronthosting.co.za>


Andrew <awnystrom at gmail.com> added the comment:

I'll try Python 2.6, but this says 2.7 is supported: http://initd.org/psycopg/features/

Thanks for the suggestion!

----------
status: pending -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13741>
_______________________________________

From report at bugs.python.org  Mon Jan  9 11:51:00 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Mon, 09 Jan 2012 10:51:00 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326106260.67.0.237708362039.issue13742@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
assignee:  -> rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 12:03:04 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 11:03:04 +0000
Subject: [issue11633] regression: print buffers output when end=''
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326106984.18.0.918234186257.issue11633@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

No, I don't think so. Another issue will not magically create more time for anyone.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 12:10:14 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 09 Jan 2012 11:10:14 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326107414.59.0.459466148011.issue13742@psf.upfronthosting.co.za>


Simon Sapin <simon.sapin at kozea.fr> added the comment:

Oops, the patch to the documentation would also need 'New in 3.3: the key parameter', with the right Sphinx directive. But that depends on whether this change ends up in 3.3 or 3.4.

Does 3.3 still get new features?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 12:11:15 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 09 Jan 2012 11:11:15 +0000
Subject: [issue5441] Convenience API for timeit.main
In-Reply-To: <1236492146.81.0.454674408295.issue5441@psf.upfronthosting.co.za>
Message-ID: <1326107475.32.0.586117645597.issue5441@psf.upfronthosting.co.za>


Changes by Simon Sapin <simon.sapin at kozea.fr>:


----------
nosy: +ssapin

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5441>
_______________________________________

From report at bugs.python.org  Mon Jan  9 12:36:00 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Mon, 09 Jan 2012 11:36:00 +0000
Subject: [issue11633] regression: print buffers output when end=''
In-Reply-To: <1326106984.18.0.918234186257.issue11633@psf.upfronthosting.co.za>
Message-ID: <CAPkN8xJ4jefvbQjjjiLAHgPydjcsxk8QR_55Fq8d4TUcVxLJ6Q@mail.gmail.com>


anatoly techtonik <techtonik at gmail.com> added the comment:

On Mon, Jan 9, 2012 at 2:03 PM, Terry J. Reedy <report at bugs.python.org>wrote:

>
> Terry J. Reedy <tjreedy at udel.edu> added the comment:
>
> No, I don't think so. Another issue will not magically create more time
> for anyone.
>

But anyone will waste less time to get to the outcome of discussion.
-- 
anatoly t.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 12:45:16 2012
From: report at bugs.python.org (Neil Muller)
Date: Mon, 09 Jan 2012 11:45:16 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <1326109516.38.0.959596347911.issue9637@psf.upfronthosting.co.za>


Neil Muller <drnlmuller+bugs at gmail.com> added the comment:

This patch is now more than a year old with no comment. What needs to happen to get this bug fixed?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Mon Jan  9 13:16:13 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 09 Jan 2012 12:16:13 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0982C2.5040606@egenix.com>
Message-ID: <4F0ADA89.6070309@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg <mal at egenix.com> added the comment:
> 
> Christian Heimes wrote:
>> Marc-Andre:
>> Have you profiled your suggestion? I'm interested in the speed implications. My gut feeling is that your idea could be slower, since you have added more instructions to a tight loop, that is execute on every lookup, insert, update and deletion of a dict key. The hash modification could have a smaller impact, since the hash is cached. I'm merely speculating here until we have some numbers to compare.
> 
> I haven't done any profiling on this yet, but will run some
> tests.

I ran pybench and pystone: neither shows a significant change.

I wish we had a simple to run benchmark based on Django to allow
checking such changes against real world applications. Not that I
expect different results from such a benchmark...

To check the real world impact, I guess it would be best to
run a few websites with the patch for a week and see whether the
collision exception gets raised.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan  9 14:06:18 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 09 Jan 2012 13:06:18 +0000
Subject: [issue13743] xml.dom.minidom.Document class is not documented
Message-ID: <1326114378.06.0.424770823698.issue13743@psf.upfronthosting.co.za>


New submission from Sandro Tosi <sandro.tosi at gmail.com>:

Hello, following up http://mail.python.org/pipermail/docs/2012-January/007026.html I noticed the whole Document class (and relative methods) is not documented.

Maybe it is a choice to not document it, or maybe it's just a missing doc, dunno, the's why I'm opening this :)

----------
assignee: docs at python
components: Documentation
messages: 150935
nosy: docs at python, sandro.tosi
priority: normal
severity: normal
stage: needs patch
status: open
title: xml.dom.minidom.Document class is not documented
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13743>
_______________________________________

From report at bugs.python.org  Mon Jan  9 14:34:22 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Mon, 09 Jan 2012 13:34:22 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
Message-ID: <1326116062.09.0.84789277908.issue13744@psf.upfronthosting.co.za>


New submission from Barry A. Warsaw <barry at python.org>:

The lexical analysis documentation says this:

http://docs.python.org/py3k/reference/lexical_analysis.html?highlight=raw%20bytes

"Bytes literals are always prefixed with 'b' or 'B';..."

"Both string and bytes literals may optionally be prefixed with a letter 'r' or 'R';..."

But that would lead you to believe that to get raw byte strings you should use rb"foo".  In fact, that's a SyntaxError in Python 2.6+ and Python 3.  What *does* work though is br"foo".

Either Python should accept both spellings (harder) or the documentation should make it clear that the 'b' must preceded the 'r'.

----------
assignee: docs at python
components: Documentation
messages: 150936
nosy: barry, docs at python
priority: normal
severity: normal
status: open
title: raw byte strings are described in a confusing way
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Mon Jan  9 15:23:22 2012
From: report at bugs.python.org (Jelle Geerts)
Date: Mon, 09 Jan 2012 14:23:22 +0000
Subject: [issue4630] IDLE no longer respects .Xdefaults insertOffTime
In-Reply-To: <1228982754.08.0.523311497339.issue4630@psf.upfronthosting.co.za>
Message-ID: <1326119002.88.0.0873462981832.issue4630@psf.upfronthosting.co.za>


Jelle Geerts <jellegeerts at gmail.com> added the comment:

I, too, would like to see this as an option in IDLE, as I also find blinking cursors distracting.

I can confirm that tkinter.Text's "insertofftime" option works on Windows as well, so the proposed patch should work on Windows.

----------
nosy: +bughunter2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4630>
_______________________________________

From report at bugs.python.org  Mon Jan  9 15:25:47 2012
From: report at bugs.python.org (Matthias Klose)
Date: Mon, 09 Jan 2012 14:25:47 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326119147.48.0.265997401773.issue10278@psf.upfronthosting.co.za>


Matthias Klose <doko at debian.org> added the comment:

then something like this is missing?

diff -r ca2a35140e6a Modules/Setup.dist
--- a/Modules/Setup.dist	Mon Jan 09 06:17:39 2012 +0000
+++ b/Modules/Setup.dist	Mon Jan 09 15:25:05 2012 +0100
@@ -166,7 +166,7 @@
 #cmath cmathmodule.c _math.c # -lm # complex math library functions
 #math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
 #_struct _struct.c	# binary structure packing/unpacking
-#time timemodule.c # -lm # time operations and variables
+#time timemodule.c $(TIMEMODULE_LIB) # -lm # time operations and variables
 #_weakref _weakref.c	# basic weak reference support
 #_testcapi _testcapimodule.c    # Python C API test module
 #_random _randommodule.c	# Random number generator

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Mon Jan  9 15:28:05 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Mon, 09 Jan 2012 14:28:05 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326119285.71.0.168082706657.issue13734@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Mon Jan  9 15:29:21 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Mon, 09 Jan 2012 14:29:21 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1228232522.58.0.0992151848245.issue4489@psf.upfronthosting.co.za>
Message-ID: <1326119361.65.0.989692987035.issue4489@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Mon Jan  9 15:37:14 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Mon, 09 Jan 2012 14:37:14 +0000
Subject: [issue13735] The protocol > 0 of cPickle does not given stable
	dictionary values
In-Reply-To: <1325993910.77.0.693345658912.issue13735@psf.upfronthosting.co.za>
Message-ID: <1326119834.31.0.651719942219.issue13735@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13735>
_______________________________________

From report at bugs.python.org  Mon Jan  9 16:05:12 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Mon, 09 Jan 2012 15:05:12 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326121512.04.0.115071303322.issue13642@psf.upfronthosting.co.za>


Michele Orr? <maker.py at gmail.com> added the comment:

Patch attached. Note that now unquote is called with host using map(), and b64 encoded strings are no more hardcoded. Please tell me if those changes are acceptable - anyway they don't break any other unit tests.

----------
nosy: +ezio.melotti
Added file: http://bugs.python.org/file24186/issue13642.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From mal at egenix.com  Mon Jan  9 13:16:09 2012
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 09 Jan 2012 13:16:09 +0100
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0982C2.5040606@egenix.com>
References: <4F0982C2.5040606@egenix.com>
Message-ID: <4F0ADA89.6070309@egenix.com>

Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg <mal at egenix.com> added the comment:
> 
> Christian Heimes wrote:
>> Marc-Andre:
>> Have you profiled your suggestion? I'm interested in the speed implications. My gut feeling is that your idea could be slower, since you have added more instructions to a tight loop, that is execute on every lookup, insert, update and deletion of a dict key. The hash modification could have a smaller impact, since the hash is cached. I'm merely speculating here until we have some numbers to compare.
> 
> I haven't done any profiling on this yet, but will run some
> tests.

I ran pybench and pystone: neither shows a significant change.

I wish we had a simple to run benchmark based on Django to allow
checking such changes against real world applications. Not that I
expect different results from such a benchmark...

To check the real world impact, I guess it would be best to
run a few websites with the patch for a week and see whether the
collision exception gets raised.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 09 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From report at bugs.python.org  Mon Jan  9 16:15:21 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 15:15:21 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
In-Reply-To: <1326116062.09.0.84789277908.issue13744@psf.upfronthosting.co.za>
Message-ID: <1326122121.12.0.749802610433.issue13744@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Either Python should accept both spellings

+1. Been annoyed several times by this.

----------
nosy: +gvanrossum, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Mon Jan  9 16:18:47 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Mon, 09 Jan 2012 15:18:47 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
In-Reply-To: <1326122121.12.0.749802610433.issue13744@psf.upfronthosting.co.za>
Message-ID: <20120109161837.13e4a570@rivendell>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 09, 2012, at 03:15 PM, Antoine Pitrou wrote:

>
>Antoine Pitrou <pitrou at free.fr> added the comment:
>
>> Either Python should accept both spellings
>
>+1. Been annoyed several times by this.

The $64k question: is this a new feature or a bug? :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Mon Jan  9 16:35:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 15:35:50 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
In-Reply-To: <1326116062.09.0.84789277908.issue13744@psf.upfronthosting.co.za>
Message-ID: <1326123350.78.0.424322163056.issue13744@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> The $64k question: is this a new feature or a bug? :)

Most certainly a feature...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Mon Jan  9 16:40:10 2012
From: report at bugs.python.org (Matthias Klose)
Date: Mon, 09 Jan 2012 15:40:10 +0000
Subject: [issue13745] configuring --with-dbmliborder=bdb doesn't build the
	gdbm extension
Message-ID: <1326123610.21.0.125848045882.issue13745@psf.upfronthosting.co.za>


New submission from Matthias Klose <doko at debian.org>:

configuring --with-dbmliborder=bdb doesn't build the gdbm extension. this option should only affect the build of the dbm extension, not the build of the gdbm extension.

----------
messages: 150943
nosy: doko
priority: low
severity: normal
status: open
title: configuring --with-dbmliborder=bdb doesn't build the gdbm extension
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13745>
_______________________________________

From report at bugs.python.org  Mon Jan  9 16:44:35 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Mon, 09 Jan 2012 15:44:35 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
In-Reply-To: <1326123350.78.0.424322163056.issue13744@psf.upfronthosting.co.za>
Message-ID: <20120109164429.43ef6842@rivendell>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 09, 2012, at 03:35 PM, Antoine Pitrou wrote:

>
>Antoine Pitrou <pitrou at free.fr> added the comment:
>
>> The $64k question: is this a new feature or a bug? :)
>
>Most certainly a feature...

In that case, since we can only add the new prefixes to 3.3, I still think we
need to fix the documentation to remove the confusion for stable releases.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:01:38 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:01:38 +0000
Subject: [issue13682] Documentation of os.fdopen() refers to non-existing
	bufsize argument of builtin open()
In-Reply-To: <1325244968.02.0.142917484563.issue13682@psf.upfronthosting.co.za>
Message-ID: <1326124898.79.0.863830564443.issue13682@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13682>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:09:09 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:09:09 +0000
Subject: [issue13631] readline fails to parse some forms of .editrc under
	editline (libedit) emulation on Mac OS X
In-Reply-To: <1324267597.43.0.334578978638.issue13631@psf.upfronthosting.co.za>
Message-ID: <1326125349.97.0.774500443614.issue13631@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the analysis and patch.  Have you tested that your patch does not break the readline module when linking against GNU readline?  If not, I can do it.

----------
keywords: +needs review
nosy: +eric.araujo, r.david.murray
stage:  -> commit review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13631>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:13:23 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:13:23 +0000
Subject: [issue12207] Document ast.PyCF_ONLY_AST
In-Reply-To: <1306689830.34.0.885575322408.issue12207@psf.upfronthosting.co.za>
Message-ID: <1326125603.57.0.82154058401.issue12207@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

FWIW this issue was also noted by Terry in #4118, but there was no reply.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12207>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:13:43 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:13:43 +0000
Subject: [issue1612012] builtin compile() doc needs PyCF_DONT_IMPLY_DEDENT
Message-ID: <1326125623.62.0.336684119123.issue1612012@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1612012>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:22:02 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:22:02 +0000
Subject: [issue13677] correct docstring for builtin compile
In-Reply-To: <1325180963.04.0.479072583529.issue13677@psf.upfronthosting.co.za>
Message-ID: <1326126122.82.0.96653779604.issue13677@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

The compiler flags are not really documented in the ast module doc or anywhere: see #12207 and #1612012.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13677>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:26:27 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:26:27 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <1326126387.79.0.436589384326.issue11906@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> "If you don?t have easy access to a command line, you can run the test
> suite from a Python or IDLE shell:
> >>> from test import autotest"

I discovered that after our discussion in this report and added it to the devguide in c18fd0ee23ed.

BTW I agree with Ezio that the fix you proposed should be committed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:31:01 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:31:01 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1326126661.65.0.68841754415.issue13733@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

The first diff hunk looks a clear typo fix for all versions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:33:56 2012
From: report at bugs.python.org (Igor Bronshteyn)
Date: Mon, 09 Jan 2012 16:33:56 +0000
Subject: [issue13746] ast.Tuple's have an inconsistent "col_offset" value
Message-ID: <1326126836.81.0.857723956921.issue13746@psf.upfronthosting.co.za>


New submission from Igor Bronshteyn <igor.bronstein at gmail.com>:

Consider the following snippet (the file is attached):

==== code starts ====
a = [1, 3.14, 'abc', u'XYZ']
b = (1, 3.14, 'abc', u'XYZ')
c = {1 : 3.14, 'abc' : u'XYZ'}
===== code ends =====

The list has correct position: (1,4), the dict has correct position too: (3,4). But the position of tuple node (ast.Tuple) has wrong or inconsistent "col_offset" = 5: (2,5).

----------
files: sample01.py
messages: 150950
nosy: bronikkk
priority: normal
severity: normal
status: open
title: ast.Tuple's have an inconsistent "col_offset" value
type: behavior
versions: Python 2.6, Python 2.7
Added file: http://bugs.python.org/file24187/sample01.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13746>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:36:53 2012
From: report at bugs.python.org (Igor Bronshteyn)
Date: Mon, 09 Jan 2012 16:36:53 +0000
Subject: [issue13746] ast.Tuple's have an inconsistent "col_offset" value
In-Reply-To: <1326126836.81.0.857723956921.issue13746@psf.upfronthosting.co.za>
Message-ID: <1326127013.55.0.143445660511.issue13746@psf.upfronthosting.co.za>


Igor Bronshteyn <igor.bronstein at gmail.com> added the comment:

I mean, that AST generated with standard "ast.parse" has nodes with confusing positions. Sorry for the first indistinct message.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13746>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:40:02 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:40:02 +0000
Subject: [issue4489] shutil.rmtree is vulnerable to a symlink attack
In-Reply-To: <1228232522.58.0.0992151848245.issue4489@psf.upfronthosting.co.za>
Message-ID: <1326127202.41.0.40656725109.issue4489@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

In case you want opinions on pathlib: I, for one, disliked Jason Orendorff?s path module, because it did not distinguish between pure string operations and I/O-inducing operations, and also because it duplicated os/os.path functions.  Your API doesn?t have these issues, so for my taste it?s conceptually better.  I should clone your repo and play with the module a bit to see if I like it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4489>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:40:54 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:40:54 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326127254.18.0.371314937285.issue9253@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150758

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:40:57 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:40:57 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326127257.22.0.945973168854.issue9253@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150815

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:41:00 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:41:00 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326127260.48.0.157612915074.issue9253@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150820

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:47:18 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:47:18 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326127638.48.0.14401113995.issue9253@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for persevering in the face of VCS complications :)  I have added a warning to the obsolete Git wiki page; I can?t do anything for the argparse Google code page.  Anyway, trust us that argparse in the 3.3 stdlib is the place where development happens.  (The Python Subversion repository is now dead.)

As for the tracker, well, its use of HTTP and URIs is somewhat idiosyncratic.  It?s far from perfect and definitely not as elegant or REST-compliant that one could wish for.  Anyway, it?s just a tool that serves us rather well; I?ve never seen a double submission issues like here before.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:53:31 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:53:31 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326128011.11.0.0813960447711.issue13742@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Yes, 3.3 is still in the early development stage, and new features will be accepted until the first beta (in June, see PEP 398).  ?.. versionadded:: 3.3 The *key* parameter? will do.

----------
nosy: +eric.araujo
versions:  -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 17:59:10 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 16:59:10 +0000
Subject: [issue11633] Document that print buffers output when end=''
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326128350.73.0.726188257152.issue11633@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> You must realize that the most common use case for print(..., end!='\n') is when you want
> to notify user about intermediate progress of a very long operation.

References needed.

----------
keywords:  -3.2regression, patch
nosy: +eric.araujo
title: regression: print buffers output when end='' -> Document that print buffers output when end=''

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 18:00:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 09 Jan 2012 17:00:55 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
In-Reply-To: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>
Message-ID: <1326128455.53.0.540099740808.issue13718@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

FTR the doc for ',' was added in a6e7ed52df9b and fa16acb2ffdd.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Mon Jan  9 18:39:03 2012
From: report at bugs.python.org (Zvezdan Petkovic)
Date: Mon, 09 Jan 2012 17:39:03 +0000
Subject: [issue13631] readline fails to parse some forms of .editrc under
	editline (libedit) emulation on Mac OS X
In-Reply-To: <1324267597.43.0.334578978638.issue13631@psf.upfronthosting.co.za>
Message-ID: <1326130743.96.0.191308727947.issue13631@psf.upfronthosting.co.za>


Zvezdan Petkovic <zvezdan at zope.com> added the comment:

I did not test against a readline build.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13631>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:18:15 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 09 Jan 2012 18:18:15 +0000
Subject: [issue13741] *** glibc detected *** python: double free or corruption
	(!prev): 0x0000000001d53ad0 ***
In-Reply-To: <1326086552.75.0.47475679765.issue13741@psf.upfronthosting.co.za>
Message-ID: <1326133095.71.0.83958856554.issue13741@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> I'll try Python 2.6, but this says 2.7 is supported

(OOps, mixed up psyco and psycopg).
Anyway, if you look at the backtrace, you'll see it segfaults inside psycopg.
You should report this crash to this project.
Closing.

----------
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13741>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:21:49 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 18:21:49 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326133309.88.0.492423931809.issue13703@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
Removed message: http://bugs.python.org/msg150846

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:29:09 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 18:29:09 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <1326133749.88.0.694456054214.issue9637@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Senthil, any comment on this?

----------
nosy: +orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:38:39 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Mon, 09 Jan 2012 18:38:39 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <1326134319.94.0.738113011092.issue9637@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Sorry, this escaped for so long. First thing is, having different values for HTTP_PROXY and HTTP_proxy and http_proxy is plain wrong at the user side. They all should be same and in that case the possible problem which Kirikaza  and Neil Muller state is extremely unlikely (and if the user faces any issue, it is due to improper HTTP configuration).

The reason that lowering of environ vars is performed is similar to any string match with input from user where we want to ignore the case sensitivity.

Having said that, I think, I shall just add teh info on lowering and verification of proxy environment to the docs. I am -1 on changing any behavior for this issue.

----------
assignee: docs at python -> orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:41:29 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Mon, 09 Jan 2012 18:41:29 +0000
Subject: [issue11633] Document that print buffers output when end=''
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326134489.17.0.307039997227.issue11633@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

> Making documentation for simple print() statement overloaded with low level buffering details makes language seem overly complicated for new users.

Why don't anybody require references for that?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:43:16 2012
From: report at bugs.python.org (Stefan Krah)
Date: Mon, 09 Jan 2012 18:43:16 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
In-Reply-To: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>
Message-ID: <1326134596.86.0.513049763559.issue13718@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

[[fill]align][sign][#][0][width][,][.precision][type]

The ',' must be before [.precision]:

>>> '{:,.2%}'.format(55.537568)
'5,553.76%'


In my opinion this is not a bug.

----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Mon Jan  9 19:44:52 2012
From: report at bugs.python.org (Daniel Stutzbach)
Date: Mon, 09 Jan 2012 18:44:52 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326134692.83.0.710357608255.issue13742@psf.upfronthosting.co.za>


Changes by Daniel Stutzbach <stutzbach at google.com>:


----------
nosy: +stutzbach

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:10:42 2012
From: report at bugs.python.org (Ben Darnell)
Date: Mon, 09 Jan 2012 19:10:42 +0000
Subject: [issue13747] ssl_version documentation error
Message-ID: <1326136242.52.0.747462245395.issue13747@psf.upfronthosting.co.za>


New submission from Ben Darnell <ben.darnell at gmail.com>:

The ssl module docs claim that the default ssl_version for client-side operation is SSLv3, but it is actually SSLv23.  The exact behavior depends on the version of openssl:  starting in 1.0 the connection is limited by default to SSLv3 or TLSv1 (as documented in the note below the compatibility table), but in older versions of openssl SSLv2 is allowed by default. 

This is just a documentation error if you've got a recent version of openssl, but it's also a security problem with older versions, since people may have been unknowingly using the weaker SSLv2 protocol.  (I don't know how widespread pre-1.0 versions of openssl are these days, but OSX Lion still ships with 0.9.8)  It would be nice if the default mode were SSLv23 with SSL_OP_NO_SSLv2 set so the defaults would be safe even with older versions of openssl (there's no way to set this configuration from python code before py3.2)

Also, the compatibility table claims that an SSLv3 client can talk to an SSLv2 server, which is incorrect.  SSLv23 clients can talk to SSLv3 and TLSv1 servers if openssl is at least version 1.0 and SSLv2 ciphers are not explicitly enabled.

----------
messages: 150963
nosy: Ben.Darnell
priority: normal
severity: normal
status: open
title: ssl_version documentation error
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13747>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:14:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 19:14:10 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
Message-ID: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

As mentioned in issue13744, it is not easy to remember that br"" is an allowed bytes literal but not rb"". Here is a patch to allow both forms.

----------
components: Interpreter Core
files: rb.patch
keywords: patch
messages: 150964
nosy: barry, benjamin.peterson, gvanrossum, ncoghlan, pitrou
priority: normal
severity: normal
stage: patch review
status: open
title: Allow rb"" literals as an equivalent to br""
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24188/rb.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:19:29 2012
From: report at bugs.python.org (K Richard Pixley)
Date: Mon, 09 Jan 2012 19:19:29 +0000
Subject: [issue13749] socketserver can't stop
Message-ID: <1326136769.56.0.386749729049.issue13749@psf.upfronthosting.co.za>


New submission from K Richard Pixley <rich at noir.com>:

Once I've instantiated my server class, along with a handler class, called server.serve_forever(), handler.handle() has been called, I've done my work, and I'm ready to shut the whole thing down...

How do I do that?

The doc says server.shutdown(), but if I call self.server.shutdown() from within handler.handle(), I seem to get a deadlock, which is exactly what I'd expect in a single threaded system with no way to "signal" the server.server_forever() loop which is several frames up the stack.

I've also tried sys.exit() but it seems that the server object is catching that as an exception.

How is this expected to work?  How do I terminate the server.serve_forever() loop?

Both 3.2 and 2.7 appear to behave the same way.  The documentation is confusing here as it doesn't explain what is expected to happen in this case.

----------
components: Library (Lib)
messages: 150965
nosy: teamnoir
priority: normal
severity: normal
status: open
title: socketserver can't stop
type: behavior
versions: Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13749>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:21:38 2012
From: report at bugs.python.org (K Richard Pixley)
Date: Mon, 09 Jan 2012 19:21:38 +0000
Subject: [issue13749] socketserver can't stop
In-Reply-To: <1326136769.56.0.386749729049.issue13749@psf.upfronthosting.co.za>
Message-ID: <1326136898.29.0.227846676231.issue13749@psf.upfronthosting.co.za>


K Richard Pixley <rich at noir.com> added the comment:

It appears as though the problem is that shutdown() blocks waiting for the serve_forever loop to terminate, which won't happen as long as the process is blocked on shutdown.

I'd like to propose that the library be changed to eliminate the block.  Shutdown() can set the flag and then return.  This should allow the handler to return and the serve_forever loop to notice that it has been asked to cease operations.

Failing that, I think the library needs some other way to exit a socketserver.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13749>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:27:37 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 19:27:37 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>
Message-ID: <1326137257.59.0.691762076356.issue13748@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Updated patch with doc.

----------
Added file: http://bugs.python.org/file24189/rb2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:28:07 2012
From: report at bugs.python.org (K Richard Pixley)
Date: Mon, 09 Jan 2012 19:28:07 +0000
Subject: [issue13749] socketserver can't stop
In-Reply-To: <1326136769.56.0.386749729049.issue13749@psf.upfronthosting.co.za>
Message-ID: <1326137287.78.0.651126445227.issue13749@psf.upfronthosting.co.za>


K Richard Pixley <rich at noir.com> added the comment:

On second thought, my proposal is likely to break existing code, so I withdraw it.

I don't know how to exit the server in a way that both works in all conditions and also continues to support existing semantics.

I expect we'll need to create a new call.  Perhaps "request_shutdown" which simply sets the flag without waiting?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13749>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:33:36 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 09 Jan 2012 19:33:36 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326137616.96.0.861227889639.issue13742@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Simon, please keep the original version fast by creating two code paths:

   if key is None:
      original_code
   else:
      new_code using the key_function

----------
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:42:29 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 09 Jan 2012 19:42:29 +0000
Subject: [issue13739] os.fdlistdir() is not idempotent
In-Reply-To: <1326039986.48.0.995838424483.issue13739@psf.upfronthosting.co.za>
Message-ID: <1326138149.8.0.467142909794.issue13739@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13739>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:45:10 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Mon, 09 Jan 2012 19:45:10 +0000
Subject: [issue13750] queue broken when built without-thread
Message-ID: <1326138310.33.0.521077885473.issue13750@psf.upfronthosting.co.za>


New submission from Charles-Fran?ois Natali <neologix at free.fr>:

Commit 190ad17f5a87481a006434a2a3d3a8e1e954a6db broke the fedora without-threads buildbot:
"""
./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/queue.py", line 4, in <module>
    import threading
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/threading.py", line 4, in <module>
    import _thread
ImportError: No module named '_thread'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./Tools/scripts/run_tests.py", line 12, in <module>
    import test.support
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/support.py", line 25, in <module>
    import logging.handlers
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/logging/handlers.py", line 30, in <module>
    import queue
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/queue.py", line 6, in <module>
    import dummythreading as threading
ImportError: No module named 'dummythreading'
[76890 refs]
make: *** [buildbottest] Error 1
"""

See http://python.org/dev/buildbot/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/1333/steps/test/logs/stdio

----------
components: Library (Lib)
messages: 150970
nosy: neologix, rhettinger
priority: normal
severity: normal
stage: needs patch
status: open
title: queue broken when built without-thread
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13750>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:58:22 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 09 Jan 2012 19:58:22 +0000
Subject: [issue13750] queue broken when built without-thread
In-Reply-To: <1326138310.33.0.521077885473.issue13750@psf.upfronthosting.co.za>
Message-ID: <1326139102.97.0.478527766481.issue13750@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee:  -> rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13750>
_______________________________________

From report at bugs.python.org  Mon Jan  9 20:59:39 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 09 Jan 2012 19:59:39 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326139179.84.0.49246565533.issue11633@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The current doc says

"print([object, ...], *, sep=' ', end='\n', file=sys.stdout) 
Print object(s) to the stream file, separated by sep and followed by end. sep, end and file, if present, must be given as keyword arguments.

All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no object is given, print() will just write end.

The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used."

(The bit about None, said twice, could be factored out and said once after the second sentence.)

This is exactly what print does and Guido today (Python ideas) said that is what it should do and that "Apps that need flushing should call flush()." So a code change is rejected.

The issue title was incorrect. The print function does not do any buffering. The file object it writes to may. Even sys.stdout may or may not. 

We could add at the end a sentence or two something like

"Output buffering is determined by *file*. Call file.flush() to ensure, for instance, immediate appearance on a screen."

----------
assignee:  -> docs at python
components: +Documentation -Interpreter Core
nosy: +docs at python
stage:  -> needs patch
title: Document that print buffers output when end='' -> Document that print may need explicit flushing

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:02:43 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Mon, 09 Jan 2012 20:02:43 +0000
Subject: [issue13729] Evaluation order for dics key/value
In-Reply-To: <1325952332.37.0.223922022645.issue13729@psf.upfronthosting.co.za>
Message-ID: <1326139363.13.0.742830698319.issue13729@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

This is a duplicate of issue11205.

----------
nosy: +amaury.forgeotdarc
resolution:  -> duplicate
status: open -> closed
superseder:  -> Evaluation order of dictionary display is different from reference manual.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13729>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:03:50 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 09 Jan 2012 20:03:50 +0000
Subject: [issue13750] queue broken when built without-thread
In-Reply-To: <1326138310.33.0.521077885473.issue13750@psf.upfronthosting.co.za>
Message-ID: <1326139430.12.0.550608993526.issue13750@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Fixed in changeset 32eae3c48631.
Thanks for the report.

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13750>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:09:44 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Mon, 09 Jan 2012 20:09:44 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326137257.59.0.691762076356.issue13748@psf.upfronthosting.co.za>
Message-ID: <20120109210926.08b725e6@rivendell>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 09, 2012, at 07:27 PM, Antoine Pitrou wrote:

>Updated patch with doc.

Nicely done.  +1

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:23:58 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Mon, 09 Jan 2012 20:23:58 +0000
Subject: [issue13751] multiprocessing.pool hangs if any worker raises an
	Exception whose constructor requires a parameter
Message-ID: <1326140638.43.0.449072803971.issue13751@psf.upfronthosting.co.za>


New submission from Faheem Mitha <faheem at faheem.info>:

See my question at http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing

I can't improve on the analysis by Lorenzo Bolla,
so I reproduce his example below. This example hangs if
BadExc is thrown, but not if GoodExc is thrown.
The only difference between these is that the GoodExc does
not require an argument be passed to its constructor, while
BadExc does.

This looks like a bug to me, though I suppose it might
be some pickling limitation.

I have confirmed this behavior is present in 2.6.6, 2.7.2,
and 3.1.3, all tested on Debian squeeze.

                                       Regards, Faheem

#################################################

import multiprocessing

class BadExc(Exception):
    def __init__(self, a):
        '''Non-optional param in the constructor.'''
        self.a = a

class GoodExc(Exception):
    def __init__(self, a=None):
        '''Optional param in the constructor.'''
        self.a = a

def do(kwargs):
    i = kwargs['i']
    print i
    raise BadExc('a')
    # raise GoodExc('a')
    return i

pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
    arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
    # set a timeout in order to be able to catch C-c
    r.get(1e100)
except KeyboardInterrupt:
    pass
print results

----------
components: Library (Lib)
messages: 150975
nosy: fmitha
priority: normal
severity: normal
status: open
title: multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
type: behavior
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:44:58 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 09 Jan 2012 20:44:58 +0000
Subject: [issue13747] ssl_version documentation error
In-Reply-To: <1326136242.52.0.747462245395.issue13747@psf.upfronthosting.co.za>
Message-ID: <E1RkM5Q-0001sh-GZ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3db0abf3058b by Antoine Pitrou in branch '2.7':
Issue #13747: fix documentation error about the default SSL version.
http://hg.python.org/cpython/rev/3db0abf3058b

New changeset 4f14c249f3de by Antoine Pitrou in branch '2.7':
Issue #13747: fix SSL compatibility table.
http://hg.python.org/cpython/rev/4f14c249f3de

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13747>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:47:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 09 Jan 2012 20:47:39 +0000
Subject: [issue13747] ssl_version documentation error
In-Reply-To: <1326136242.52.0.747462245395.issue13747@psf.upfronthosting.co.za>
Message-ID: <E1RkM81-0002DE-PN@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 7ae0f71862f9 by Antoine Pitrou in branch '3.2':
Issue #13747: fix documentation error about the default SSL version.
http://hg.python.org/cpython/rev/7ae0f71862f9

New changeset b4194af97948 by Antoine Pitrou in branch '3.2':
Issue #13747: fix SSL compatibility table.
http://hg.python.org/cpython/rev/b4194af97948

New changeset d2a47650031a by Antoine Pitrou in branch 'default':
Merge SSL doc fixes (issue #13747).
http://hg.python.org/cpython/rev/d2a47650031a

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13747>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:53:34 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 09 Jan 2012 20:53:34 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>
Message-ID: <1326142414.65.0.504379753122.issue13748@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I would be happier if there was precedent for this with "ur" = "ru" in 2.x, which is not true.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Mon Jan  9 21:57:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 20:57:50 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326142414.65.0.504379753122.issue13748@psf.upfronthosting.co.za>
Message-ID: <1326142585.4158.0.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I would be happier if there was precedent for this with "ur" = "ru" in
> 2.x, which is not true.

In my experience, raw unicode literals were very rare while raw bytes
literals are quite frequent (mainly for regular expressions).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Mon Jan  9 22:03:17 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 21:03:17 +0000
Subject: [issue13747] ssl_version documentation error
In-Reply-To: <1326136242.52.0.747462245395.issue13747@psf.upfronthosting.co.za>
Message-ID: <1326142997.43.0.995384822952.issue13747@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks for noticing. I've now fixed the docs.

> It would be nice if the default mode were SSLv23 with SSL_OP_NO_SSLv2
> set so the defaults would be safe even with older versions of openssl

Mmmh, perhaps, although wouldn't someone deploying a new version of Python also deploy a new version of OpenSSL?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13747>
_______________________________________

From report at bugs.python.org  Mon Jan  9 22:24:11 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 09 Jan 2012 21:24:11 +0000
Subject: [issue11205] Evaluation order of dictionary display is different from
	reference manual.
In-Reply-To: <1297589953.69.0.371220549392.issue11205@psf.upfronthosting.co.za>
Message-ID: <1326144251.6.0.951667210147.issue11205@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11205>
_______________________________________

From report at bugs.python.org  Mon Jan  9 22:27:58 2012
From: report at bugs.python.org (sbt)
Date: Mon, 09 Jan 2012 21:27:58 +0000
Subject: [issue13751] multiprocessing.pool hangs if any worker raises an
	Exception whose constructor requires a parameter
In-Reply-To: <1326140638.43.0.449072803971.issue13751@psf.upfronthosting.co.za>
Message-ID: <1326144478.74.0.962463624744.issue13751@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

This is not specific to multiprocessing.  It is really an issue with the pickling of exceptions:

  >>> import cPickle
  >>> class BadExc(Exception):
  ...     def __init__(self, a):
  ...         '''Non-optional param in the constructor.'''
  ...         self.a = a
  ...
  >>> a = cPickle.dumps(BadExc(1))
  >>> cPickle.loads(a)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.BadExc'>, ())

I think that when you create a new exception class with an __init__() method, you need to make sure that self.args is set properly by calling the __init__() method of the parent class using the same arguments.  So you can instead do

  class BadExc(Exception):
      def __init__(self, a):
          '''Non-optional param in the constructor.'''
          Exception.__init__(self, a)
          self.a = a

----------
nosy: +sbt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________

From report at bugs.python.org  Mon Jan  9 22:40:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 09 Jan 2012 21:40:39 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <E1RkMxK-0000EL-3B@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset bf609baff4d3 by Charles-Fran?ois Natali in branch 'default':
Issue #12760: Add a create mode to open(). Patch by David Townshend.
http://hg.python.org/cpython/rev/bf609baff4d3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Mon Jan  9 23:13:45 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 09 Jan 2012 22:13:45 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326147225.62.0.0604442878452.issue13742@psf.upfronthosting.co.za>


Simon Sapin <simon.sapin at kozea.fr> added the comment:

Raymond, please have a look at merge_3 in benchmark_heapq_merge.py. It is implemented as you say.

Do you think the speed is worth the code duplication?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan  9 23:18:09 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 09 Jan 2012 22:18:09 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326147489.6.0.0802188395507.issue13645@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

I'm not suggesting two stat calls (in the general case); you would call one or the other depending on the magic number of the pyc file.

Anyway, it would probably be best to have some method that is expected to return a specific object which embodies all the desired information for bytecode generation (and if you encompass source code with this object then you can get rid of get_source() as well). But it shouldn't be a raw stat object since not all bits of information will come from a stat call (eg. storing bytecode in a sqlite3 database) and thus require bogus data. If you want to move towards that kind of API I can support that.

As for path_mtime() returning an int instead of some number that can be converted to an int, that's because I didn't plan for the "Antoine wants to muck with the .pyc format" contingency (IOW I just didn't think about it). =) It wouldn't be a big deal to change the API to take a keyword-only argument specifying you want the highest resolution number instead of specifically an int.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Tue Jan 10 00:05:07 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 09 Jan 2012 23:05:07 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1326147489.6.0.0802188395507.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326150221.4158.2.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I'm not suggesting two stat calls (in the general case); you would
> call one or the other depending on the magic number of the pyc file.

The proposal is to store both mtime and size, actually, to make false
positives less likely.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Tue Jan 10 00:28:33 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 09 Jan 2012 23:28:33 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326151713.64.0.369678294185.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I've filed a support incident with the Microsoft Support team to try to push this issue determine a fix or workaround.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Tue Jan 10 00:29:47 2012
From: report at bugs.python.org (Alex Regueiro)
Date: Mon, 09 Jan 2012 23:29:47 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326151787.59.0.282086360666.issue6727@psf.upfronthosting.co.za>


Alex Regueiro <alexreg at gmail.com> added the comment:

That's great, thanks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Tue Jan 10 00:34:30 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 09 Jan 2012 23:34:30 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <E1RkOjV-0005uf-Aa@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b950267efd59 by Terry Jan Reedy in branch '3.2':
#11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/b950267efd59

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:09:41 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Tue, 10 Jan 2012 00:09:41 +0000
Subject: [issue13718] Format Specification Mini-Language does not accept comma
	for percent value
In-Reply-To: <1325849041.09.0.966587471813.issue13718@psf.upfronthosting.co.za>
Message-ID: <1326154181.68.0.740791803602.issue13718@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

Good point. I hadn't looked at the string closely enough. Closing.

----------
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13718>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:20:22 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 10 Jan 2012 00:20:22 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <E1RkPRt-00057j-ET@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset ec32e6ec16fc by Terry Jan Reedy in branch '2.7':
#11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/ec32e6ec16fc

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:22:47 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 10 Jan 2012 00:22:47 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <1326154967.27.0.341922377633.issue11906@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
assignee:  -> terry.reedy
resolution:  -> fixed
stage:  -> committed/rejected

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:33:49 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 10 Jan 2012 00:33:49 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326155629.34.0.182190024692.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

OK, os.walkfd is sounding good:

- accepts a file descriptor, byte sequence or string for "top"
- produces 4-tuples, with the dirfd added at the end
- documents clearly that the dirfd is normally only valid until the next iteration step, so you need to call os.dup() if you want to hang onto it

As far as walkdir integration goes, I currently plan to add it as a "Directory Walking" subsection in shutil before the first alpha. However, it needs a few more updates in PyPI first (e.g. preserving the tuples produced by the underlying iterators, making sure it behaves itself when handed binary paths). I'll post to python-dev about it before I actually commit anything.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:34:28 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Tue, 10 Jan 2012 00:34:28 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326155668.47.0.0436583280877.issue13241@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

On my system (Snow Leopard OSX 10.6.8, Xcode 4.1) there are several different versions of gcc available in /usr/bin/. Some are symlinks to others; in particular gcc links to llvm-gccc-4.2. By using "./configure CC=gcc-4.2" I get a working config file. I ran "make clean" to avoid reusing possibly corrupt .o files compiled with the default.

----------
nosy: +gvanrossum

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Tue Jan 10 01:48:43 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 10 Jan 2012 00:48:43 +0000
Subject: [issue11948] Tutorial/Modules - small fix to better clarify the
	modules search path
In-Reply-To: <1304011111.24.0.834772686878.issue11948@psf.upfronthosting.co.za>
Message-ID: <1326156523.03.0.112485567014.issue11948@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Sandro, I think you can apply this.

----------
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11948>
_______________________________________

From report at bugs.python.org  Tue Jan 10 02:36:03 2012
From: report at bugs.python.org (Ned Deily)
Date: Tue, 10 Jan 2012 01:36:03 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326159363.85.0.794055662236.issue13241@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

That works for Xcode 4.1, which is now out-of-date. The current releases of Xcode, 4.2.x for 10.7 Lion and optionally for 10.6 Snow Leopard, no longer include gcc-4.2, only llvm-gcc and clang.  I am working on completing an extensive set of regression tests for all of the development branches versus the most important set of Xcode/platform options and should have results here later this week.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Tue Jan 10 02:38:36 2012
From: report at bugs.python.org (Ben Darnell)
Date: Tue, 10 Jan 2012 01:38:36 +0000
Subject: [issue13747] ssl_version documentation error
In-Reply-To: <1326136242.52.0.747462245395.issue13747@psf.upfronthosting.co.za>
Message-ID: <1326159516.89.0.443574013086.issue13747@psf.upfronthosting.co.za>


Ben Darnell <ben.darnell at gmail.com> added the comment:

Not necessarily.  If I want to run python 2.7 or 3.x on an older linux distribution (e.g. Ubuntu 10.04 LTS, which has python 2.6 and openssl 0.9.8), I need to build from source, but I wouldn't think to update/rebuild all the dependencies from the ground up.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13747>
_______________________________________

From report at bugs.python.org  Tue Jan 10 02:40:24 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Tue, 10 Jan 2012 01:40:24 +0000
Subject: [issue13752] add a casefold() method
Message-ID: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>


New submission from Benjamin Peterson <benjamin at python.org>:

See http://mail.python.org/pipermail/python-ideas/2012-January/013292.html

----------
components: Unicode
messages: 150996
nosy: benjamin.peterson, ezio.melotti
priority: low
severity: normal
status: open
title: add a casefold() method
type: enhancement
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Tue Jan 10 04:11:59 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 10 Jan 2012 03:11:59 +0000
Subject: [issue11418] Method's global scope is module containing function
	definition, not class.
In-Reply-To: <1299454101.64.0.841920247259.issue11418@psf.upfronthosting.co.za>
Message-ID: <1326165119.96.0.122243773085.issue11418@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

This paragraph follows a discussion and example of the fact that methods do *not* have to be defined within a class statement.

Any objections to changing

"The global scope associated with a method is the module containing the class definition. (The class itself is never used as a global scope.)"

to 

"The global scope associated with a method is the module containing its definition. (A class is never used as a global scope.)"

?

----------
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11418>
_______________________________________

From report at bugs.python.org  Tue Jan 10 04:45:26 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 10 Jan 2012 03:45:26 +0000
Subject: [issue13168] Python 2.6 having trouble finding modules when invoked
	via a symlink
In-Reply-To: <1318522134.74.0.524944339938.issue13168@psf.upfronthosting.co.za>
Message-ID: <1326167126.26.0.94153907213.issue13168@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
resolution:  -> out of date
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13168>
_______________________________________

From report at bugs.python.org  Tue Jan 10 04:49:32 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Tue, 10 Jan 2012 03:49:32 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326167372.55.0.325653271103.issue12736@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

New patch. I implemented it the way Antoine desired. It seems rather inefficient to be copying around so much data...

----------
Added file: http://bugs.python.org/file24190/full-casemapping.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Tue Jan 10 04:52:41 2012
From: report at bugs.python.org (py.user)
Date: Tue, 10 Jan 2012 03:52:41 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
Message-ID: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

http://docs.python.org/py3k/library/stdtypes.html#str.join

str.join(iterable)?

    Return a string which is the concatenation of the strings in the iterable iterable. A TypeError will be raised if there are any non-string values in seq, including bytes objects. The separator between elements is the string providing this method.


"non-string values in seq" -> "non-string values in iterable"

----------
assignee: docs at python
components: Documentation
messages: 150999
nosy: docs at python, py.user
priority: normal
severity: normal
status: open
title: str.join description contains an incorrect reference to argument
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Tue Jan 10 04:59:55 2012
From: report at bugs.python.org (py.user)
Date: Tue, 10 Jan 2012 03:59:55 +0000
Subject: [issue13754] str.ljust and str.rjust do not exactly describes
	original string return
Message-ID: <1326167995.85.0.0875969790457.issue13754@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

http://docs.python.org/py3k/library/stdtypes.html#str.ljust

str.ljust(width[, fillchar])?

    Return the string left justified in a string of length width. Padding is done using the specified fillchar (default is a space). The original string is returned if width is less than len(s).

str.rjust(width[, fillchar])?

    Return the string right justified in a string of length width. Padding is done using the specified fillchar (default is a space). The original string is returned if width is less than len(s).



"less than len(s)" -> "less than or equal to len(s)"

----------
assignee: docs at python
components: Documentation
messages: 151000
nosy: docs at python, py.user
priority: normal
severity: normal
status: open
title: str.ljust and str.rjust do not exactly describes original string return
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13754>
_______________________________________

From report at bugs.python.org  Tue Jan 10 05:04:53 2012
From: report at bugs.python.org (py.user)
Date: Tue, 10 Jan 2012 04:04:53 +0000
Subject: [issue13754] str.ljust and str.rjust do not exactly describes
	original string return
In-Reply-To: <1326167995.85.0.0875969790457.issue13754@psf.upfronthosting.co.za>
Message-ID: <1326168293.73.0.739369650668.issue13754@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

str.zfill also

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13754>
_______________________________________

From report at bugs.python.org  Tue Jan 10 05:09:29 2012
From: report at bugs.python.org (py.user)
Date: Tue, 10 Jan 2012 04:09:29 +0000
Subject: [issue13755] str.endswith and str.startswith do not take lists of
	strings
Message-ID: <1326168569.6.0.720816160315.issue13755@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

>>> 'abcd'.endswith(['a', 'b'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'list' object to str implicitly
>>>

it would be nice like in str.join
>>> ''.join(('a', 'b'))
'ab'
>>> ''.join(['a', 'b'])
'ab'
>>> ''.join({'a', 'b'})
'ab'
>>> ''.join({'a': 1, 'b': 2})
'ab'
>>>

----------
components: Interpreter Core
messages: 151002
nosy: py.user
priority: normal
severity: normal
status: open
title: str.endswith and str.startswith do not take lists of strings
type: behavior
versions: Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13755>
_______________________________________

From report at bugs.python.org  Tue Jan 10 06:22:57 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Tue, 10 Jan 2012 05:22:57 +0000
Subject: [issue13751] multiprocessing.pool hangs if any worker raises an
	Exception whose constructor requires a parameter
In-Reply-To: <1326140638.43.0.449072803971.issue13751@psf.upfronthosting.co.za>
Message-ID: <1326172977.71.0.337532512047.issue13751@psf.upfronthosting.co.za>


Faheem Mitha <faheem at faheem.info> added the comment:

Thanks to sbt for his helpful and clear explanation.
The following bug report appears relevant, http://bugs.python.org/issue1692335. It seems the proposed fix
was never applied because it caused breakage to existing gcode.
It is not clear to me whether this behavior is considered a bug
or a feature. :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________

From report at bugs.python.org  Tue Jan 10 06:45:11 2012
From: report at bugs.python.org (INADA Naoki)
Date: Tue, 10 Jan 2012 05:45:11 +0000
Subject: [issue11418] Method's global scope is module containing function
	definition, not class.
In-Reply-To: <1326165119.96.0.122243773085.issue11418@psf.upfronthosting.co.za>
Message-ID: <CAEfz+Tx9_N_CWtwKkBh1jXjAFn3qbyaZxY8td-jRg5pfe6Z0PA@mail.gmail.com>


INADA Naoki <songofacandy at gmail.com> added the comment:

> Any objections to changing
>
> "The global scope associated with a method is the module containing the class definition. (The class itself is never used as a global scope.)"
>
> to
>
> "The global scope associated with a method is the module containing its definition. (A class is never used as a global scope.)"
>
> ?

+1

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11418>
_______________________________________

From report at bugs.python.org  Tue Jan 10 08:01:48 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Tue, 10 Jan 2012 07:01:48 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326178908.28.0.554576587376.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

Michele, in your patch:

+            authorization = ("Authorization: Basic %s\r\n" %
+                             b64encode('a%20b:c%20d'))

This is wrong. See the original report by me and RFC 2617. The username and password MUST NOT be url encoded before base64. That is the original problem. The point is that this test should fail with urllib before the change and not fail with the fix.

Secondly, I think unquote will fail when given a None. For me, some other unit tests caught this when I had the unquote where the splituser is called. I didn't run your code but are other urllib tests ok for you?

I like your change of having the base64 explicitly there and not as a magic string is a good idea.

Senthil, could you provide the better alternative for the class field abuse, please?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 08:17:23 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Tue, 10 Jan 2012 07:17:23 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326179843.7.0.894688026318.issue13241@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

The version of clang included in Xcode 4.2.1 (the current release) still miscompiles Python 3.3 and both C files included in this issue.

The same is true for the Xcode 4.3 beta.

Python 2.7 seems to work fine with the same compiler, either because we don't trigger the bug or it fails in a more subtle way than crashing.

Has anyone filed a bug with Apple and/or the clang developers?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Tue Jan 10 08:44:24 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Tue, 10 Jan 2012 07:44:24 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326181464.89.0.44060731539.issue13241@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

I've filed an issue in Apple's tracker, #10667893. Filing more issues would be helpful, if more issues in their tracker mention this bug it is more likely to get fixed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Tue Jan 10 08:57:39 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 10 Jan 2012 07:57:39 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326182259.36.0.557156764656.issue12760@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Committed.
David, thanks for the patch!

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Tue Jan 10 09:41:11 2012
From: report at bugs.python.org (Paul Smedley)
Date: Tue, 10 Jan 2012 08:41:11 +0000
Subject: [issue13733] Change required to sysconfig.py for Python 2.7.2 on OS/2
In-Reply-To: <1325980402.33.0.867781342076.issue13733@psf.upfronthosting.co.za>
Message-ID: <1326184871.45.0.262313933792.issue13733@psf.upfronthosting.co.za>


Paul Smedley <paul at smedley.id.au> added the comment:

When I get some spare time, I'll test the default code using Andy Mac's last build of Python - which afaik was 2.4.x and see if it works correctly

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13733>
_______________________________________

From report at bugs.python.org  Tue Jan 10 11:47:37 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Tue, 10 Jan 2012 10:47:37 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326192457.63.0.0942668754243.issue13642@psf.upfronthosting.co.za>


Michele Orr? <maker.py at gmail.com> added the comment:

Whoops, probably I tested using $ python instead of $ ./python.exe - 
Attaching two patches, one keeps using map(), but definitely changes unquote() behavior; the other simply asserts user_passwd exists before using unquote().

Well, concerning the class field abuse, HTTPConnection._buffer attribute might help? The point is that I can't find an easy way to get the HTTPConnection instance from an urlopen().

----------
Added file: http://bugs.python.org/file24191/issue13642.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 11:47:52 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Tue, 10 Jan 2012 10:47:52 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326192472.04.0.433051048874.issue13642@psf.upfronthosting.co.za>


Changes by Michele Orr? <maker.py at gmail.com>:


Removed file: http://bugs.python.org/file24186/issue13642.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 11:48:07 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Tue, 10 Jan 2012 10:48:07 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326192487.79.0.877433652301.issue13642@psf.upfronthosting.co.za>


Changes by Michele Orr? <maker.py at gmail.com>:


Added file: http://bugs.python.org/file24192/issue13642_with_map.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 12:21:43 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Tue, 10 Jan 2012 11:21:43 +0000
Subject: [issue13755] str.endswith and str.startswith do not take lists of
	strings
In-Reply-To: <1326168569.6.0.720816160315.issue13755@psf.upfronthosting.co.za>
Message-ID: <1326194503.44.0.525153535741.issue13755@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

No, but they take tuples:

>>> 'abcd'.endswith(('a', 'b'))
False
>>> 'abcd'.endswith(('c', 'd'))
True

Suggest closing as out of date.

----------
nosy: +mark.dickinson
type: behavior -> enhancement
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13755>
_______________________________________

From report at bugs.python.org  Tue Jan 10 12:38:00 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 11:38:00 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326133309.95.0.224363958602.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwYxhDta2TZLrsyqSTgiJRea+0xFf5EqRLkecunT5dWMNg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Version 3 of my patch:
 - Add PYTHONHASHSEED environment variable to get a fixed seed or to
disable the randomized hash function (PYTHONHASHSEED=0)
 - Add tests on the randomized hash function
 - Add more tests on os.urandom()

----------
Added file: http://bugs.python.org/file24193/random-3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -460,6 +460,13 @@ These environment variables influence Py
    option.
 
 
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for the Unicode randomized hash:
+   number in range [0; 4294967295]. The value 0 disables the Unicode randomized
+   hash.
+
+
 .. envvar:: PYTHONIOENCODING
 
    If this is set before running the interpreter, it overrides the encoding used
diff --git a/Include/pythonrun.h b/Include/pythonrun.h
--- a/Include/pythonrun.h
+++ b/Include/pythonrun.h
@@ -246,6 +246,8 @@ typedef void (*PyOS_sighandler_t)(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int PyOS_URandom) (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -376,6 +376,18 @@ typedef struct {
 PyAPI_DATA(PyTypeObject) PyUnicode_Type;
 PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type;
 
+/* Issue #13703: add 2 x sizeof(Py_hash_t) random bytes to the output of
+   hash(str) to make the dict hash attack more complex. The attack computes N
+   strings with the same hash value to exploit to worst case of a dict lookup:
+   O(n^2) complexity. Without these random bytes, an attacker can precompute
+   the N keys only once to cause a denial of service. See oCERT advisory for the
+   details: http://www.ocert.org/advisories/ocert-2011-003.html */
+typedef struct {
+    Py_hash_t prefix;
+    Py_hash_t suffix;
+} _Py_unicode_hash_secret_t;
+PyAPI_DATA(_Py_unicode_hash_secret_t) _Py_unicode_hash_secret;
+
 #define PyUnicode_Check(op) \
                  PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
 #define PyUnicode_CheckExact(op) (Py_TYPE(op) == &PyUnicode_Type)
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -761,23 +761,6 @@ try:
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -12,6 +12,7 @@ import subprocess
 import time
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 import contextlib
 import mmap
 import platform
@@ -630,14 +631,32 @@ class DevNullTests(unittest.TestCase):
             self.assertEqual(f.read(), b'')
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 @contextlib.contextmanager
 def _execvpe_mockup(defpath=None):
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -11,6 +11,7 @@ import struct
 import sys
 import unittest
 import warnings
+from test.script_helper import assert_python_ok
 from test import support, string_tests
 
 # Error handling (bad decoder return)
@@ -1891,6 +1892,34 @@ class StringModuleTest(unittest.TestCase
         self.assertRaises(TypeError, _string.formatter_field_name_split, 1)
 
 
+class HashTest(unittest.TestCase):
+    def get_hash(self, text, seed=None):
+        env = {}
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'print(hash(%r))' % text,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        self.assertEqual(self.get_hash("abc", 0), -1600925533)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        self.assertEqual(self.get_hash("abc", 42), -206076799)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        run1 = self.get_hash("abc")
+        run2 = self.get_hash("abc")
+        self.assertNotEqual(run1, run2)
+
 def test_main():
     support.run_unittest(__name__)
 
diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -322,6 +322,7 @@ PYTHON_OBJS=	\
 		Python/pystate.o \
 		Python/pythonrun.o \
 		Python/pytime.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -9326,81 +9326,37 @@ posix_getloadavg(PyObject *self, PyObjec
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
+PyDoc_STRVAR(posix_urandom__doc__,
+"urandom(n) -> bytes\n\n\
 Return n random bytes suitable for cryptographic use.");
 
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
 static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t howMany;
+    PyObject *result;
+    int ret;
 
     /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
+    if (! PyArg_ParseTuple(args, "n:urandom", &howMany))
         return NULL;
     if (howMany < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
 
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
     result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
+    if (result == NULL)
+        return NULL;
+
+    ret = PyOS_URandom(PyBytes_AS_STRING(result),
+                       PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        PyErr_SetString(PyExc_RuntimeError, "Fail to generate random bytes");
+        return NULL;
     }
     return result;
 }
-#endif
 
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
@@ -9442,42 +9398,6 @@ device_encoding(PyObject *self, PyObject
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
-    }
-    return result;
-}
-#endif
-
 #ifdef HAVE_SETRESUID
 PyDoc_STRVAR(posix_setresuid__doc__,
 "setresuid(ruid, euid, suid)\n\n\
@@ -10884,12 +10804,7 @@ static PyMethodDef posix_methods[] = {
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
 #ifdef HAVE_SETRESUID
     {"setresuid",       posix_setresuid, METH_VARARGS, posix_setresuid__doc__},
 #endif
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -165,6 +165,8 @@ extern "C" {
             *_to++ = (to_type) *_iter++;                \
     } while (0)
 
+_Py_unicode_hash_secret_t _Py_unicode_hash_secret;
+
 /* This dictionary holds all interned unicode strings.  Note that references
    to strings in this dictionary are *not* counted in the string's ob_refcnt.
    When the interned string reaches a refcnt of 0 the string deallocation
@@ -11143,8 +11145,6 @@ unicode_getitem(PyObject *self, Py_ssize
     return PyUnicode_FromOrdinal(ch);
 }
 
-/* Believe it or not, this produces the same value for ASCII strings
-   as bytes_hash(). */
 static Py_hash_t
 unicode_hash(PyObject *self)
 {
@@ -11156,13 +11156,19 @@ unicode_hash(PyObject *self)
     if (PyUnicode_READY(self) == -1)
         return -1;
     len = PyUnicode_GET_LENGTH(self);
+    if (len == 0) {
+        _PyUnicode_HASH(self) = 0;
+        return 0;
+    }
 
     /* The hash function as a macro, gets expanded three times below. */
 #define HASH(P) \
-    x = (Py_uhash_t)*P << 7; \
+    x ^= (Py_uhash_t)*P << 7; \
     while (--len >= 0) \
         x = (1000003*x) ^ (Py_uhash_t)*P++;
 
+    x = _Py_unicode_hash_secret.prefix;
+
     switch (PyUnicode_KIND(self)) {
     case PyUnicode_1BYTE_KIND: {
         const unsigned char *c = PyUnicode_1BYTE_DATA(self);
@@ -11184,6 +11190,7 @@ unicode_hash(PyObject *self)
     }
     }
     x ^= (Py_uhash_t)PyUnicode_GET_LENGTH(self);
+    x ^= _Py_unicode_hash_secret.suffix;
 
     if (x == -1)
         x = -2;
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -71,6 +71,7 @@ extern int _PyUnicode_Init(void);
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 extern int _PyFaulthandler_Init(void);
 extern void _PyFaulthandler_Fini(void);
 
@@ -219,6 +220,8 @@ Py_InitializeEx(int install_sigs)
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
 
+    _PyRandom_Init();
+
     interp = PyInterpreterState_New();
     if (interp == NULL)
         Py_FatalError("Py_Initialize: can't make first interpreter");

From report at bugs.python.org  Tue Jan 10 13:34:47 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Tue, 10 Jan 2012 12:34:47 +0000
Subject: [issue13755] str.endswith and str.startswith do not take lists of
	strings
In-Reply-To: <1326168569.6.0.720816160315.issue13755@psf.upfronthosting.co.za>
Message-ID: <1326198887.07.0.373195082421.issue13755@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

It seems like a set would make more sense than a tuple. And if tuples, why not lists?

Not that it matters much, since I doubt it's worth changing in either case. It's easy enough for the caller to convert.

----------
nosy: +eric.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13755>
_______________________________________

From report at bugs.python.org  Tue Jan 10 13:42:06 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Tue, 10 Jan 2012 12:42:06 +0000
Subject: [issue13755] str.endswith and str.startswith do not take lists of
	strings
In-Reply-To: <1326168569.6.0.720816160315.issue13755@psf.upfronthosting.co.za>
Message-ID: <1326199326.98.0.104684509559.issue13755@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13755>
_______________________________________

From report at bugs.python.org  Tue Jan 10 13:44:19 2012
From: report at bugs.python.org (Holger)
Date: Tue, 10 Jan 2012 12:44:19 +0000
Subject: [issue13756] Python3.2.2 make fail on cygwin
Message-ID: <1326199459.04.0.575648132034.issue13756@psf.upfronthosting.co.za>


New submission from Holger <holger at bille.dk>:

Downloaded Python-3.2.2.tar
cd Python-3.2.2
./configure
make
....
Fail on:
--------------------
.....
ranlib libpython3.2m.a
make: *** No rule to make target `libpython3.2m.dll.a', needed by `python.exe'.  Stop.
--------------------

libpython3.2m.a was built successfully:
--------------------
$ ls -l libpython3.2m.a
-rw-r--r--+ 1 hbille Domain Users 7182050 Jan 10 13:31 libpython3.2m.a
--------------------

I am not sure which part is incorrect. The name with or without "dll"

----------
components: Installation
messages: 151014
nosy: holgerd00d
priority: normal
severity: normal
status: open
title: Python3.2.2 make fail on cygwin
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13756>
_______________________________________

From report at bugs.python.org  Tue Jan 10 14:49:56 2012
From: report at bugs.python.org (Florian Berger)
Date: Tue, 10 Jan 2012 13:49:56 +0000
Subject: [issue5998] Add __bool__ to threading.Event and multiprocessing.Event
In-Reply-To: <1242077059.81.0.31819803999.issue5998@psf.upfronthosting.co.za>
Message-ID: <1326203396.69.0.131233438105.issue5998@psf.upfronthosting.co.za>


Florian Berger <fberger at florian-berger.de> added the comment:

Voting for re-opening.

I am currently porting a non-threaded function for use within a threaded application. If threading.Event had __bool__, it would be a drop-in replacement for simple True/False flags that can not be used in multithreaded code.

To me, it is actually surprising that I can not do tests like "if event: ...". IMHO, "if event.is_set(): ..." is unnecessarily complicated, especially with the documentation speaking of a true/false flag all the time.

I will subclass threading.Event now and add __bool__, but I don't feel this is a nice solution.

----------
nosy: +fberger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5998>
_______________________________________

From report at bugs.python.org  Tue Jan 10 15:03:40 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Tue, 10 Jan 2012 14:03:40 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326204220.17.0.652276522373.issue12736@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

__ap__'s implementation method is about 2x faster than mine.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Tue Jan 10 15:26:57 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 10 Jan 2012 14:26:57 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwYxhDta2TZLrsyqSTgiJRea+0xFf5EqRLkecunT5dWMNg@mail.gmail.com>
Message-ID: <1326205530.3391.2.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Version 3 of my patch:
>  - Add PYTHONHASHSEED environment variable to get a fixed seed or to
> disable the randomized hash function (PYTHONHASHSEED=0)
>  - Add tests on the randomized hash function
>  - Add more tests on os.urandom()

You forgot random.c.

+        PyErr_SetString(PyExc_RuntimeError, "Fail to generate random
bytes");

I would put an OSError and preserve the errno.

+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        self.assertEqual(self.get_hash("abc", 0), -1600925533)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        self.assertEqual(self.get_hash("abc", 42), -206076799)

This is portable on both 32-bit and 64-bit builds?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 10 17:09:54 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 10 Jan 2012 16:09:54 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <E1RkeGm-0002Wb-FK@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 01ef13e9e225 by Senthil Kumaran in branch '2.7':
- Issue #13642: Unquote before b64encoding user:password during Basic
http://hg.python.org/cpython/rev/01ef13e9e225

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 17:47:20 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Tue, 10 Jan 2012 16:47:20 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326214040.68.0.449277164688.issue13642@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Joonas and Michele - The fix along with the tests is in for 2.7 line. W.r.t to tests having it in the class level buf seems to be only easy way, for other it seemed to be that FakeSocket and FakeConnection stuff need some major change (and resulted in breaking of some tests). I thought it is better to push it in the current form as we use the buf for temporary storage and verification.
I think, those tests can be brought into 3.x line as well.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Tue Jan 10 18:30:40 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 10 Jan 2012 17:30:40 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <E1RkfWx-0000ou-2L@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3370fa13ed73 by Senthil Kumaran in branch '3.2':
Issue9637 - Explain in getproxies_environment that <scheme>_proxy environ variable case does not matter.
http://hg.python.org/cpython/rev/3370fa13ed73

New changeset eb028b3c62c9 by Senthil Kumaran in branch 'default':
Merge from 3.2 - Issue9637 - Explain in getproxies_environment that <scheme>_proxy environ variable case does not matter.
http://hg.python.org/cpython/rev/eb028b3c62c9

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Tue Jan 10 18:35:22 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 10 Jan 2012 17:35:22 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <E1RkfbV-0001hL-Ae@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4a30eae3b945 by Senthil Kumaran in branch '2.7':
port to 2.7 - Issue9637 - Details that case of Proxy env var does not matter.
http://hg.python.org/cpython/rev/4a30eae3b945

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Tue Jan 10 18:39:26 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Tue, 10 Jan 2012 17:39:26 +0000
Subject: [issue9637] docs do not say that urllib uses HTTP_PROXY
In-Reply-To: <1282196669.74.0.629180904202.issue9637@psf.upfronthosting.co.za>
Message-ID: <1326217166.7.0.980087775753.issue9637@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

This is fixed. Thanks for your contribution.

----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9637>
_______________________________________

From report at bugs.python.org  Tue Jan 10 19:56:52 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 10 Jan 2012 18:56:52 +0000
Subject: [issue13757] os.fdlistdir() should not close the file descriptor
	given in argument
Message-ID: <1326221812.3.0.905477293867.issue13757@psf.upfronthosting.co.za>


New submission from Charles-Fran?ois Natali <neologix at free.fr>:

os.fdlistdir() closes the FD passed as argument.
This is annoying, since in 99% of the cases you'd like to keep FD intact, so you end up doing os.fdlistdir(os.dup(fd)).
Here's a patch that duplicates the FD in fdlistdir(), so that the original FD is kept intact, which is much more natural (at least to me :).
That's an API change, but since fdlistdir() has been introduced in 3.3 which hasn't been released yet, I think it should be acceptable.

----------
components: Extension Modules
files: fdlistdir_noclose.diff
keywords: patch
messages: 151023
nosy: neologix, pitrou, rosslagerwall
priority: normal
severity: normal
status: open
title: os.fdlistdir() should not close the file descriptor given in argument
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24194/fdlistdir_noclose.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13757>
_______________________________________

From report at bugs.python.org  Tue Jan 10 19:58:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 10 Jan 2012 18:58:46 +0000
Subject: [issue13757] os.fdlistdir() should not close the file descriptor
	given in argument
In-Reply-To: <1326221812.3.0.905477293867.issue13757@psf.upfronthosting.co.za>
Message-ID: <1326221926.7.0.7349736831.issue13757@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Agreed with the change.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13757>
_______________________________________

From report at bugs.python.org  Tue Jan 10 20:00:28 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Tue, 10 Jan 2012 19:00:28 +0000
Subject: [issue13757] os.fdlistdir() should not close the file descriptor
	given in argument
In-Reply-To: <1326221812.3.0.905477293867.issue13757@psf.upfronthosting.co.za>
Message-ID: <1326222028.6.0.227207461978.issue13757@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

The reason I made it like that was that it seemed "closer" to the fdopendir() function which steals the fd for internal use.

However, I agree that it makes more sense to dup() it.

Patch looks good.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13757>
_______________________________________

From report at bugs.python.org  Tue Jan 10 20:13:58 2012
From: report at bugs.python.org (Steven Bethard)
Date: Tue, 10 Jan 2012 19:13:58 +0000
Subject: [issue13685] argparse does not sanitize help strings for % signs
In-Reply-To: <1325284167.16.0.584942168313.issue13685@psf.upfronthosting.co.za>
Message-ID: <1326222838.88.0.836045249714.issue13685@psf.upfronthosting.co.za>


Steven Bethard <steven.bethard at gmail.com> added the comment:

Eric's suggested doc fix looks good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13685>
_______________________________________

From report at bugs.python.org  Tue Jan 10 20:25:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 10 Jan 2012 19:25:47 +0000
Subject: [issue13757] os.fdlistdir() should not close the file descriptor
	given in argument
In-Reply-To: <1326221812.3.0.905477293867.issue13757@psf.upfronthosting.co.za>
Message-ID: <E1RkhKL-000591-Es@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 6d395ed03f95 by Charles-Fran?ois Natali in branch 'default':
Issue #13757: Change os.fdlistdir() so that it duplicates the passed file
http://hg.python.org/cpython/rev/6d395ed03f95

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13757>
_______________________________________

From report at bugs.python.org  Tue Jan 10 20:37:33 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 10 Jan 2012 19:37:33 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326224253.42.0.267965751848.issue12760@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Nick suggested to call the new flag "exclusive create" in the doc (and explain in whatsnew that it's based C11 new 'x' flag).
Could someone please check the attached patch?
My wording sounds really clumsy, so I'd prefer if a native speaker could review it.

----------
nosy: +ncoghlan
Added file: http://bugs.python.org/file24195/x_flag.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Tue Jan 10 21:40:20 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 10 Jan 2012 20:40:20 +0000
Subject: [issue13757] os.fdlistdir() should not close the file descriptor
	given in argument
In-Reply-To: <1326221812.3.0.905477293867.issue13757@psf.upfronthosting.co.za>
Message-ID: <1326228020.19.0.494370742494.issue13757@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Thanks!

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13757>
_______________________________________

From report at bugs.python.org  Tue Jan 10 22:40:23 2012
From: report at bugs.python.org (Brett Cannon)
Date: Tue, 10 Jan 2012 21:40:23 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1326150221.4158.2.camel@localhost.localdomain>
Message-ID: <CAP1=2W6iqKrvPmt7tY5VBmbjGLissJCZAxPOz02eNvPBe+z9Mg@mail.gmail.com>


Brett Cannon <brett at python.org> added the comment:

On Mon, Jan 9, 2012 at 18:05, Antoine Pitrou <report at bugs.python.org> wrote:

>
> Antoine Pitrou <pitrou at free.fr> added the comment:
>
> > I'm not suggesting two stat calls (in the general case); you would
> > call one or the other depending on the magic number of the pyc file.
>
> The proposal is to store both mtime and size, actually, to make false
> positives less likely.

Then a method that returns some object representing all of the needed info
on the source file is needed, but that does NOT blindly return the stat
info (eg. modification date, "file" size, and even source as bytes can be
in this object).  It could even have methods to generate the bytecode, etc.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Tue Jan 10 23:15:05 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 22:15:05 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326233705.64.0.0592118091471.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 4:
 - os.urandom() raises again exceptions on failure
 - drop support of VMS (which used RAND_pseudo_bytes from OpenSSL): I don't see how to link Python/random.c to libcrypto on VMS, I don't have VMS, and it don't see how it was working because posixmodule.c was neither linked to libcrypto !?
 - fix test_dict, test_gdb, test_builtin
 - win32_urandom() handles size bigger than INT_MAX using a loop (it may be DWORD max instead?)
 - _PyRandom_Init() does nothing it is called twice to fix a _testembed failure (don't change the Unicode secret because Python stores some strings somewhere and never destroy them)

----------
Added file: http://bugs.python.org/file24196/random-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 00:06:22 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 10 Jan 2012 23:06:22 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1326155629.34.0.182190024692.issue13734@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3u2gcQzvW42oq2ARB2GXwS3aOzFQNAYU0Y-oVc03qRGQ@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's a patch with tests and documentation.
I noticed something surprising:
walk() with followlinks=False returns links to directories as
directories (in dirnames).
I find this surprising, since if you don't follow symlinks, those are
just files (and you don't recurse into it). Also, it's a pain when you
want to remove dirnames, since you have to distinguish between a link
and a directory (unlink()/rmdir() or unlinkat() without/with
AT_REMOVEDIR)
To be consistent with this behavior, I had to change fdwalk() (I
renamed it to be consistent with fdlistdir()) to perform a call to
fstatat() without AT_SYMLINK_NOFOLLOW, since otherwise it would report
such links as files.
So the bottom line is that because of this, you can have up to 3
stat() calls per entry:
- fstatat(rootfd, name)
- fstatat(rootfd, name, AT_SYMLINK_NOFOLLOW) right before opening the directory
- fstat(dirfd) right after open to check that we're dealing with the same file
(walk() currently uses two stat() per entry, so it's not too bad).

----------
Added file: http://bugs.python.org/file24197/fdwalk.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2240,6 +2240,58 @@
               os.rmdir(os.path.join(root, name))
 
 
+.. function:: fdwalk(top, topdown=True, onerror=None, followlinks=False)
+
+   .. index::
+      single: directory; walking
+      single: directory; traversal
+
+    This behaves exactly like :func:`walk`, except that it yields a 4-tuple
+    ``(dirpath, dirnames, filenames, dirfd)``.
+
+   *dirpath*, *dirnames* and *filenames* are identical to :func:`walk` output,
+   and *dirfd* is an open file descriptor to the directory.
+
+   .. note::
+
+      Since :func:`fdwalk` yields file descriptors, those are only valid until
+      the next iteration step, so you should duplicate them (e.g. with
+      :func:`dup`) if you want to keep them longer.
+
+   .. note::
+
+      Contrarily to :func:`walk`, modifying the dirnames list in-place won't
+      affect the directories traversed.
+
+   This example displays the number of bytes taken by non-directory files in each
+   directory under the starting directory::
+
+      import os
+      for root, dirs, files, rootfd in os.fdwalk('python/Lib/email'):
+          print(root, "consumes", end="")
+          print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                end="")
+          print("bytes in", len(files), "non-directory files")
+
+   In the next example, walking the tree bottom-up is essential:
+   :func:`unlinkat` doesn't allow deleting a directory before the directory is
+   empty::
+
+      # Delete everything reachable from the directory named in "top",
+      # assuming there are no symbolic links.
+      # CAUTION:  This is dangerous!  For example, if top == '/', it
+      # could delete all your disk files.
+      import os
+      for root, dirs, files, rootfd in os.fdwalk(top, topdown=False):
+          for name in files:
+              os.unlinkat(rootfd, name)
+          for name in dirs:
+              os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+
+   Availability: Unix.
+
+   .. versionadded:: 3.3
+
 .. _os-process:
 
 Process Management
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -478,6 +478,10 @@
 
   (Patch submitted by Giampaolo Rodol?? in :issue:`10784`.)
 
+* The :mod:`os` module has a new :func:`~os.fdwalk` function similar to
+  :func:`~os.walk` excepts that it also yields an open file descriptor to the
+  directories visited. This is especially useful to avoid symlink races.
+
 * "at" functions (:issue:`4761`):
 
   * :func:`~os.faccessat`
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -24,6 +24,7 @@
 #'
 
 import sys, errno
+import stat as st
 
 _names = sys.builtin_module_names
 
@@ -32,6 +33,9 @@
            "defpath", "name", "path", "devnull",
            "SEEK_SET", "SEEK_CUR", "SEEK_END"]
 
+def _exists(name):
+    return name in globals()
+
 def _get_exports_list(module):
     try:
         return list(module.__all__)
@@ -120,7 +124,13 @@
     umask(mask)
     return mode & ~mask
 
-#'
+def _are_same_file(stat1, stat2):
+    """Helper function that checks whether two stat results refer to the same
+    file.
+    """
+    return (stat1.st_mode == stat2.st_mode and stat1.st_ino == stat2.st_ino and
+            stat1.st_dev == stat2.st_dev)
+#
 
 # Super directory utilities.
 # (Inspired by Eric Raymond; the doc strings are mostly his)
@@ -151,7 +161,6 @@
     try:
         mkdir(name, mode)
     except OSError as e:
-        import stat as st
         if not (e.errno == errno.EEXIST and exist_ok and path.isdir(name) and
                 st.S_IMODE(lstat(name).st_mode) == _get_masked_mode(mode)):
             raise
@@ -298,6 +307,94 @@
 
 __all__.append("walk")
 
+if _exists("openat"):
+
+    def fdwalk(top, topdown=True, onerror=None, followlinks=False):
+        """Directory tree generator.
+
+        This behaves exactly like walk(), except that it yields a 4-tuple
+
+            dirpath, dirnames, filenames, dirfd
+
+        dirpath, dirnames and filenames are identical to walk() output, and
+        dirfd is an open file descriptor to dirpath.
+
+        The advantage of walkfd() over walk() is that it's safe against symlink
+        races (when followlinks is False).
+
+        Caution:
+        Since fdwalk() yields file descriptors, those are only valid until the
+        next iteration step, so you should dup() them if you want to keep them
+        for a longer period.
+        Also, contrarily to walk(), modifying the dirnames list in-place won't
+        affect the directories traversed.
+
+        Example:
+
+        import os
+        for root, dirs, files, rootfd in os.fdwalk('python/Lib/email'):
+            print(root, "consumes", end="")
+            print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                  end="")
+            print("bytes in", len(files), "non-directory files")
+        """
+        # Note: To guard against symlink races, we use the standard
+        # lstat()/open()/fstat() trick.
+        orig_st = lstat(top)
+        topfd = open(top, O_RDONLY)
+        try:
+            if (followlinks or (st.S_ISDIR(orig_st.st_mode) and
+                               _are_same_file(orig_st, fstat(topfd)))):
+                for x in _fdwalk(topfd, top, topdown, onerror, followlinks):
+                    yield x
+        finally:
+            close(topfd)
+
+    def _fdwalk(topfd, toppath, topdown, onerror, followlinks):
+        try:
+            names = fdlistdir(topfd)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return
+
+        dirs, nondirs = [], []
+        for name in names:
+            # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
+            # walk() which reports symlinks to directories as directories. We do
+            # however check for symlinks before recursing into a subdirectory.
+            if st.S_ISDIR(fstatat(topfd, name).st_mode):
+                dirs.append(name)
+            else:
+                nondirs.append(name)
+
+        # whether to follow symlinks
+        flag = 0 if followlinks else AT_SYMLINK_NOFOLLOW
+
+        if topdown:
+            yield toppath, dirs, nondirs, topfd
+
+        for name in dirs:
+            try:
+                orig_st = fstatat(topfd, name, flag)
+                dirfd = openat(topfd, name, O_RDONLY)
+            except error as err:
+                if onerror is not None:
+                    onerror(err)
+                return
+            try:
+                if followlinks or _are_same_file(orig_st, fstat(dirfd)):
+                    dirpath = path.join(toppath, name)
+                    for x in _fdwalk(dirfd, dirpath, topdown, onerror, followlinks):
+                        yield x
+            finally:
+                close(dirfd)
+
+        if not topdown:
+            yield toppath, dirs, nondirs, topfd
+
+    __all__.append("fdwalk")
+
 # Make sure os.environ exists, at least
 try:
     environ
@@ -598,9 +695,6 @@
 fsencode, fsdecode = _fscodec()
 del _fscodec
 
-def _exists(name):
-    return name in globals()
-
 # Supply spawn*() (probably only for Unix)
 if _exists("fork") and not _exists("spawnv") and _exists("execv"):
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -20,6 +20,8 @@
 import asyncore
 import asynchat
 import socket
+import itertools
+import stat
 try:
     import threading
 except ImportError:
@@ -147,7 +149,6 @@
         if not hasattr(os, "stat"):
             return
 
-        import stat
         result = os.stat(fname)
 
         # Make sure direct access works
@@ -464,7 +465,7 @@
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
-    def test_traversal(self):
+    def setUp(self):
         import os
         from os.path import join
 
@@ -569,6 +570,57 @@
                     os.remove(dirname)
         os.rmdir(support.TESTFN)
 
+ at unittest.skipUnless(hasattr(os, 'fdwalk'), "Test needs os.fdwalk()")
+class FdWalkTests(WalkTests):
+    """Tests for os.fdwalk()."""
+
+    def test_compare_to_walk(self):
+        # compare with walk() results
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            expected = {}
+            for root, dirs, files in os.walk(*args):
+                expected[root] = (set(dirs), set(files))
+
+            for root, dirs, files, rootfd in os.fdwalk(*args):
+                self.assertEqual(expected[root], (set(dirs), set(files)))
+
+    def test_dir_fd(self):
+        # check returned file descriptors
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            for root, dirs, files, rootfd in os.fdwalk(*args):
+                # check that the FD is valid
+                os.fstat(rootfd)
+                # check that fdlistdir() returns consistent information
+                self.assertEqual(set(os.fdlistdir(rootfd)), set(dirs) | set(files))
+
+    def test_fd_leak(self):
+        # Since we're opening a lot of FDs, we must be careful to avoid leaks:
+        # we both check that calling fdwalk() a large number of times doesn't
+        # yield EMFILE, and that the minimum allocated FD hasn't changed.
+        minfd = os.dup(1)
+        os.close(minfd)
+        for i in range(1024):
+            for x in os.fdwalk(support.TESTFN):
+                pass
+        newfd = os.dup(1)
+        self.addCleanup(os.close, newfd)
+        self.assertEqual(newfd, minfd)
+
+    def tearDown(self):
+        # cleanup
+        for root, dirs, files, rootfd in os.fdwalk(support.TESTFN, topdown=False):
+            for name in files:
+                os.unlinkat(rootfd, name)
+            for name in dirs:
+                st = os.fstatat(rootfd, name, os.AT_SYMLINK_NOFOLLOW)
+                if stat.S_ISDIR(st.st_mode):
+                    os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+                else:
+                    os.unlinkat(rootfd, name)
+        os.rmdir(support.TESTFN)
+
 class MakedirTests(unittest.TestCase):
     def setUp(self):
         os.mkdir(support.TESTFN)
@@ -1683,6 +1735,7 @@
         StatAttributeTests,
         EnvironTests,
         WalkTests,
+        FdWalkTests,
         MakedirTests,
         DevNullTests,
         URandomTests,

From report at bugs.python.org  Wed Jan 11 00:07:08 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 23:07:08 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326236828.91.0.989657506926.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 5 fixes test_unicode for 64-bit system.

----------
Added file: http://bugs.python.org/file24198/random-5.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 00:07:37 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 23:07:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326236857.92.0.553180631047.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24142/random.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 00:07:40 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 23:07:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326236860.58.0.291675364375.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24143/random-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 00:07:43 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 23:07:43 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326236863.05.0.277715578351.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24193/random-3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 00:07:45 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 10 Jan 2012 23:07:45 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326236865.89.0.919431552313.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24196/random-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 04:38:21 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Wed, 11 Jan 2012 03:38:21 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326253101.47.0.305140356526.issue12736@psf.upfronthosting.co.za>


Changes by Benjamin Peterson <benjamin at python.org>:


Added file: http://bugs.python.org/file24199/full-casemapping.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Wed Jan 11 04:46:44 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 03:46:44 +0000
Subject: [issue13758] compile() should not encode 'filename' (at least on
	Windows)
Message-ID: <1326253604.11.0.305060780894.issue13758@psf.upfronthosting.co.za>


New submission from Terry J. Reedy <tjreedy at udel.edu>:

The 3.2.2 doc for compile() says "The filename argument should give the file from which the code was read; pass some recognizable value if it wasn?t read from a file ('<string>' is commonly used)."

I am not sure what 'recognizable' is supposed to mean, but as I understand it, it would be user-specific and any string containing a fake 'filename' should be accepted and attached to the output code object as the .co_filename attribute. (At least on Windows.)

In fact, compile() has a hidden restriction: it encodes 'filename' with the local filesystem encoding. It tosses the bytes result (at least on Windows) but lets a UnicodeEncodeError terminate compilation. The effect is to add an undocumented and spurious dependency to code that has nothing to do with real files or the local machine.

In #10114, msg118845, Victor Stinner justified this with 
"co_filename attribute is used to display the traceback: Python opens the related file, read the source code line and display it."
If the filename is fake, it cannot do that. (Perhaps the doc should warn users to make sure that fake filenames do not match any possibly real filenames ;-). The traceback mechanism could ignore UnicodeEncodeErrors just as well as it now ignores IO(?)Errors when open('fakename') does not not work.

Victor continues "On Windows, co_filename is directly used because Windows accepts unicode for filenames." This is not true in that on at least some Windows, compile tries to encode with the mbcs codec, which in turn uses the hidden local codepage. I believe that for most or all codepages, this will even raise errors for some valid Unicode filenames.

I do not know whether the stored .co_filename attribute type for *nix is str, as on Windows, or bytes. If the latter, the doc should say so.
If compile() continues to filter fake filenames, which I oppose, the doc should also say so and say what it does.

This issue came up on python-list when someone used a Chinese filename and mbcs rejected it.

----------
components: Interpreter Core
messages: 151034
nosy: terry.reedy
priority: normal
severity: normal
stage: test needed
status: open
title: compile() should not encode 'filename' (at least on Windows)
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13758>
_______________________________________

From report at bugs.python.org  Wed Jan 11 08:36:56 2012
From: report at bugs.python.org (Atsushi Shibata)
Date: Wed, 11 Jan 2012 07:36:56 +0000
Subject: [issue13759] Python 3.2.2 Mac installer version doesn't accept
	multibyte character in interactive mode
Message-ID: <1326267416.18.0.541225422796.issue13759@psf.upfronthosting.co.za>


New submission from Atsushi Shibata <shibata at webcore.co.jp>:

I installed Python 3.2.2 on Macintosh(Snow Leopard), by using installer on python.org.
In interactive mode, it does not accept multibyte character. When I type Japanese character on it, it changes to ?.
I tried to make Python 3.2.2 by using tar ball on the same machine. I can type Japanese character in interactive mode.
I guess it's because of some minor issue, such as compile option is different etc.

Thanks in advance.

----------
assignee: ronaldoussoren
components: Macintosh
messages: 151035
nosy: ats, ronaldoussoren
priority: normal
severity: normal
status: open
title: Python 3.2.2 Mac installer version doesn't accept multibyte character in interactive mode
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13759>
_______________________________________

From report at bugs.python.org  Wed Jan 11 08:50:12 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Wed, 11 Jan 2012 07:50:12 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
Message-ID: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>


New submission from Faheem Mitha <faheem at faheem.info>:

I have not experienced this myself, but see
http://stackoverflow.com/questions/2246384/multiprocessing-pool-hangs-when-there-is-a-exception-in-any-of-the-thread

This appears to be another case of http://bugs.python.org/issue1692335

I also recently reported a similar problem at http://bugs.python.org/issue13751

Looking at the code for `NoOptionError` (see below) in 2.6 at least it would indeed be subject to the same problem. Perhaps some attention could be paid to 
resolving 1692335, which would presumably make this vexing problem go away?

class NoOptionError(Error):
    """A requested option was not found."""

    def __init__(self, option, section):
        Error.__init__(self, "No option %r in section: %r" %
                       (option, section))
        self.option = option
        self.section = section

----------
components: Library (Lib)
messages: 151036
nosy: fmitha
priority: normal
severity: normal
status: open
title: ConfigParser exceptions are not pickleable
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:00:29 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 08:00:29 +0000
Subject: [issue13754] str.ljust and str.rjust do not exactly describes
	original string return
In-Reply-To: <1326167995.85.0.0875969790457.issue13754@psf.upfronthosting.co.za>
Message-ID: <1326268829.46.0.0773778486757.issue13754@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The short doc strings do not mention the issue either way. I think it ok to leave them as they are. While the claim is how the functions *should* operate, I verified that it is how they *do* operate in all three branches.

----------
assignee: docs at python -> terry.reedy
keywords: +easy, patch
nosy: +terry.reedy
stage:  -> needs patch
versions: +Python 2.7, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13754>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:00:47 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 08:00:47 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
In-Reply-To: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>
Message-ID: <1326268847.77.0.0787941824235.issue13753@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
assignee: docs at python -> terry.reedy
keywords: +easy, patch
nosy: +terry.reedy
versions: +Python 2.7, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:01:03 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 11 Jan 2012 08:01:03 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>
Message-ID: <1326268863.04.0.646820429045.issue13748@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

+1 for "raw bytes", which reads much nicer to me than "bytes raw".

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:01:59 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Wed, 11 Jan 2012 08:01:59 +0000
Subject: [issue1692335] Fix exception pickling: Move initial args assignment
	to BaseException.__new__
Message-ID: <1326268919.16.0.0686305821532.issue1692335@psf.upfronthosting.co.za>


Faheem Mitha <faheem at faheem.info> added the comment:

What is the status on this? It contains to be an
issue. See http://bugs.python.org/issue13751 and
http://bugs.python.org/issue13760

----------
nosy: +fmitha

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1692335>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:08:24 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 08:08:24 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
In-Reply-To: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>
Message-ID: <1326269304.41.0.160845261594.issue13753@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
versions:  -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:11:46 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 11 Jan 2012 08:11:46 +0000
Subject: [issue13759] Python 3.2.2 Mac installer version doesn't accept
	multibyte character in interactive mode
In-Reply-To: <1326267416.18.0.541225422796.issue13759@psf.upfronthosting.co.za>
Message-ID: <1326269506.05.0.759957575321.issue13759@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

Can you give the exact steps of what you did to demonstrate the problem?  Also, there are two OS X installers for Python 3.2.2.  Which one did you install?  (The two lines of version information when starting Python is enough to identify which one it is.)

----------
nosy: +ned.deily

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13759>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:13:48 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 11 Jan 2012 08:13:48 +0000
Subject: [issue13752] add a unicode.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326269628.23.0.176089817414.issue13752@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
stage:  -> needs patch
title: add a casefold() method -> add a unicode.casefold() method

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:24:42 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 11 Jan 2012 08:24:42 +0000
Subject: [issue13761] Add flush keyword to print()
Message-ID: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>


New submission from Georg Brandl <georg at python.org>:

Add a flush keyword argument to print(), defaulting to False. If true, output will be flushed immediately.

See http://mail.python.org/pipermail/python-ideas/2012-January/013340.html

----------
assignee: georg.brandl
components: Interpreter Core
keywords: easy
messages: 151041
nosy: georg.brandl
priority: normal
severity: normal
stage: needs patch
status: open
title: Add flush keyword to print()
type: enhancement
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:33:44 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 11 Jan 2012 08:33:44 +0000
Subject: [issue13754] str.ljust and str.rjust do not exactly describes
	original string return
In-Reply-To: <1326167995.85.0.0875969790457.issue13754@psf.upfronthosting.co.za>
Message-ID: <E1Rktcs-0002yl-HQ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3aed8787ce5c by Terry Jan Reedy in branch '2.7':
Closes #13754 String returned if less than *or equal to* x 3
http://hg.python.org/cpython/rev/3aed8787ce5c

New changeset 3f2e5fd17b76 by Terry Jan Reedy in branch '3.2':
#13754 String returned if less than *or equal to* x 3
http://hg.python.org/cpython/rev/3f2e5fd17b76

New changeset 0df85a2ebe4d by Terry Jan Reedy in branch 'default':
Merge with 3.2
http://hg.python.org/cpython/rev/0df85a2ebe4d

----------
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13754>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:42:19 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 08:42:19 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
In-Reply-To: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>
Message-ID: <1326271339.42.0.620718785927.issue13753@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

See #13754 for commits (not here because 'close' != 'closes' or closed')

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:47:31 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Wed, 11 Jan 2012 08:47:31 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326271651.17.0.80053617573.issue13761@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 09:53:21 2012
From: report at bugs.python.org (Atsushi Shibata)
Date: Wed, 11 Jan 2012 08:53:21 +0000
Subject: [issue13759] Python 3.2.2 Mac installer version doesn't accept
	multibyte character in interactive mode
In-Reply-To: <1326267416.18.0.541225422796.issue13759@psf.upfronthosting.co.za>
Message-ID: <1326272001.44.0.352989412117.issue13759@psf.upfronthosting.co.za>


Atsushi Shibata <shibata at webcore.co.jp> added the comment:

The procedure is following.

1. type "python" in the terminal.app to launch interactive mode.
2. type '?' in it.

And I use Mac OS X 64-bit/32-bit Installer (3.2.2) for Mac OS X 10.6 and 10.7.
Here is the first 2 lines of initial message.

Python 3.2.2 (v3.2.2:137e45f15c0b, Sep  3 2011, 17:28:59) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

thanks :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13759>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:01:41 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 09:01:41 +0000
Subject: [issue4630] IDLE no longer respects .Xdefaults insertOffTime
In-Reply-To: <1228982754.08.0.523311497339.issue4630@psf.upfronthosting.co.za>
Message-ID: <1326272501.73.0.484326455253.issue4630@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The idea and design look good to me. I will test on Windows when I am able to do so in development builds (unless someone beats me ;-).

----------
nosy: +terry.reedy
versions: +Python 2.7, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4630>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:06:21 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 09:06:21 +0000
Subject: [issue4630] IDLE: add cursor noblink option
In-Reply-To: <1228982754.08.0.523311497339.issue4630@psf.upfronthosting.co.za>
Message-ID: <1326272781.48.0.472207917339.issue4630@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
title: IDLE no longer respects .Xdefaults insertOffTime -> IDLE: add cursor noblink option

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4630>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:17:37 2012
From: report at bugs.python.org (Ned Deily)
Date: Wed, 11 Jan 2012 09:17:37 +0000
Subject: [issue13759] Python 3.2.2 Mac installer version doesn't accept
	multibyte character in interactive mode
In-Reply-To: <1326267416.18.0.541225422796.issue13759@psf.upfronthosting.co.za>
Message-ID: <1326273457.11.0.241907209658.issue13759@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

I am not familiar with the Japanese input methods available in OS X. But using copy and paste with an US UTF-8 locale, it seems to work.  Perhaps there is a different locale in effect?

Python 3.2.2 (v3.2.2:137e45f15c0b, Sep  3 2011, 17:28:59) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x = '?'
>>> print(x)
?
>>> print(ord(x))
12354
>>> import unicodedata
>>> unicodedata.name(x)
'HIRAGANA LETTER A'
>>> print('%x' % ord(x))
3042
>>> import locale
>>> locale.getlocale()
('en_US', 'UTF-8')

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13759>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:28:31 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 09:28:31 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326236828.91.0.989657506926.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0D5638.7090903@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> 
> Patch version 5 fixes test_unicode for 64-bit system.

Victor, I don't think the randomization idea is going anywhere. The
code has many issues:

 * it is exceedingly complex
 * the method would need to be implemented for all hashable
   Python types
 * it causes startup time to increase (you need urandom data for
   every single hashable Python data type)
 * it causes run-time to increase due to changes in the hash
   algorithm (more operations in the tight loop)
 * causes different processes in a multi-process setup to use different
   hashes for the same object
 * doesn't appear to work well in embedded interpreters that
   regularly restarted interpreters (AFAIK, some objects persist across
   restarts and those will have wrong hash values in the newly started
   instances)

The most important issue, though, is that it doesn't really
protect Python against the attack - it only makes it less
likely that an adversary will find the init vector (or a way
around having to find it via crypt analysis).

OTOH, the collision counting patch is very simple, doesn't have
the performance issues and provides real protection against the
attack. Even better still, it can detect programming errors in
hash method implementations.

IMO, it would be better to put efforts into refining the collision
detection patch (perhaps adding support for the universal hash
method slot I mentioned) and run some real life tests with it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:47:25 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 09:47:25 +0000
Subject: =?utf-8?q?=5Bissue13472=5D_devguide_doesn=E2=80=99t_list_all_build_depend?=
	=?utf-8?q?encies?=
In-Reply-To: <1322142769.6.0.687458451697.issue13472@psf.upfronthosting.co.za>
Message-ID: <1326275245.63.0.140418582549.issue13472@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13472>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:48:42 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 09:48:42 +0000
Subject: [issue13440] Explain the "status quo wins a stalemate" principle in
	the devguide
In-Reply-To: <1321833284.75.0.438515825131.issue13440@psf.upfronthosting.co.za>
Message-ID: <1326275322.22.0.724753902281.issue13440@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13440>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:56:12 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 11 Jan 2012 09:56:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0D5638.7090903@egenix.com>
Message-ID: <CAMpsgwZraWfhT_bXn2F-BmYUWkPKf5Gg1evRrijffn2xmTXjzg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> ?* it is exceedingly complex

Which part exactly? For hash(str), it just add two extra XOR.

> ?* the method would need to be implemented for all hashable Python types

It was already discussed, and it was said that only hash(str) need to
be modified.

> ?* it causes startup time to increase (you need urandom data for
> ? every single hashable Python data type)

My patch reads 8 or 16 bytes from /dev/urandom which doesn't block. Do
you have a benchmark showing a difference?

I didn't try my patch on Windows yet.

> ?* it causes run-time to increase due to changes in the hash
> ? algorithm (more operations in the tight loop)

I posted a micro-benchmark on hash(str) on python-dev: the overhead is
nul. Did you have numbers showing that the overhead is not nul?

> ?* causes different processes in a multi-process setup to use different
> ? hashes for the same object

Correct. If you need to get the same hash, you can disable the
randomized hash (PYTHONHASHSEED=0) or use a fixed seed (e.g.
PYTHONHASHSEED=42).

> ?* doesn't appear to work well in embedded interpreters that
> ? regularly restarted interpreters (AFAIK, some objects persist across
> ? restarts and those will have wrong hash values in the newly started
> ? instances)

test_capi runs _testembed which restarts a embedded interpreters 3
times, and the test pass (with my patch version 5). Can you write a
script showing the problem if there is a real problem?

In an older version of my patch, the hash secret was recreated at each
initiliazation. I changed my patch to only generate the secret once.

> The most important issue, though, is that it doesn't really
> protect Python against the attack - it only makes it less
> likely that an adversary will find the init vector (or a way
> around having to find it via crypt analysis).

I agree that the patch is not perfect. As written in the patch, it
just makes the attack more complex. I consider that it is enough.

Perl has a simpler protection than the one proposed in my patch. Is
Perl vulnerable to the hash collision vulnerability?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 10:58:18 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 09:58:18 +0000
Subject: [issue12907] Update test coverage devguide page
In-Reply-To: <1315284413.51.0.757189280849.issue12907@psf.upfronthosting.co.za>
Message-ID: <1326275898.16.0.375197842439.issue12907@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12907>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:02:18 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:02:18 +0000
Subject: [issue12296] Minor clarification in devguide
In-Reply-To: <1307634327.28.0.988674687108.issue12296@psf.upfronthosting.co.za>
Message-ID: <1326276138.47.0.656949295957.issue12296@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12296>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:08:17 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:08:17 +0000
Subject: [issue13762] missing section: how to contribute to devguide
Message-ID: <1326276497.12.0.812256237013.issue13762@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

I struggled to find how to contribute to the devguide. I even struggled to find where the VCS is. I think it would be kool if there was a section for this info (it's not always fun to search the intertubes).

----------
components: Devguide
messages: 151049
nosy: ezio.melotti, tshepang
priority: normal
severity: normal
status: open
title: missing section: how to contribute to devguide
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13762>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:11:18 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:11:18 +0000
Subject: [issue13124] Add "Running a Build Slave" page to the devguide
In-Reply-To: <1318011517.56.0.343886781977.issue13124@psf.upfronthosting.co.za>
Message-ID: <1326276678.58.0.677888434077.issue13124@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13124>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:15:46 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:15:46 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326276946.66.0.248180713999.issue12409@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:26:12 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:26:12 +0000
Subject: [issue13275] Recommend xml.etree for XML processing
In-Reply-To: <1319709959.84.0.984784684468.issue13275@psf.upfronthosting.co.za>
Message-ID: <1326277572.18.0.830020410279.issue13275@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13275>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:27:23 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:27:23 +0000
Subject: [issue13275] Recommend xml.etree for XML processing
In-Reply-To: <1319709959.84.0.984784684468.issue13275@psf.upfronthosting.co.za>
Message-ID: <1326277643.83.0.675397721942.issue13275@psf.upfronthosting.co.za>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

@Araujo bit offtopic, but which XML library have you found most elegant? Is it available in the stdlib?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13275>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:41:27 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:41:27 +0000
Subject: [issue13763] rm obsolete reference in devguide
Message-ID: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

this line:

"commonly abbreviated svn, after the program name"

was once changed to:

"commonly abbreviated hg, after the program name"

and it no longer makes sense

----------
components: Devguide
files: rm-obsolete-reference.patch
keywords: patch
messages: 151051
nosy: ezio.melotti, tshepang
priority: normal
severity: normal
status: open
title: rm obsolete reference in devguide
Added file: http://bugs.python.org/file24200/rm-obsolete-reference.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:48:22 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:48:22 +0000
Subject: [issue13764] Misc/build.sh is outdated... talks about svn
Message-ID: <1326278902.44.0.401175449456.issue13764@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

I am wondering what else is outdated in that file. Is it still useful?

----------
components: Build
messages: 151052
nosy: tshepang
priority: normal
severity: normal
status: open
title: Misc/build.sh is outdated... talks about svn

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13764>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:50:11 2012
From: report at bugs.python.org (Atsushi Shibata)
Date: Wed, 11 Jan 2012 10:50:11 +0000
Subject: [issue13759] Python 3.2.2 Mac installer version doesn't accept
	multibyte character in interactive mode
In-Reply-To: <1326267416.18.0.541225422796.issue13759@psf.upfronthosting.co.za>
Message-ID: <1326279011.79.0.107931096001.issue13759@psf.upfronthosting.co.za>


Atsushi Shibata <shibata at webcore.co.jp> added the comment:

Thank you for testing on your environment :-).
I'll make sure the difference that caused the issue.

I changed the status to pending. I'll come back to this issue when I find the crue.

Thanks anyway !

----------
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13759>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:53:28 2012
From: report at bugs.python.org (Almar Klein)
Date: Wed, 11 Jan 2012 10:53:28 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
Message-ID: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>


New submission from Almar Klein <almar.klein at gmail.com>:

I found an easy to solve bug in distutils, which is causing problems with compiling Cython code on Windows. I have reproduced this on Python 2.6 and Python 3.2 (32 bit).

The problem occurs with the native msvc compiler. Using gcc (MinGW) works fine.

The problem is that the command to link the libraries does not put double quotes around paths that have spaces in them. Unfortunately, the path where I have Python installed has spaces in it ("c:/program files/python26"). Small example of part of a link command: 

/LIBPATH:C:\Program Files (x86)\python32\libs. 

Note that the include_dirs DO have double quotes around them.

The problem is easily solved (I confirmed this) by a small change in msvc9compiler.py and msvccompiler.py (see also the patch):

def library_dir_option(self, dir):  # OLD VERSION
     return "/LIBPATH:" + dir

def library_dir_option(self, dir): # FIXED VERSION
    if ' ' in dir and not dir.startswith('"'):
        dir = '"%s"' % dir
    return "/LIBPATH:" + dir

I tried to see if it would be nicer to apply a change elsewhere, e.g. where library_dir_option() is called. However, it is called nowhere from within the distutils package. In my case I suspect numpy.distutils or a part of Cython calls it. Anyway, in my opinion you should be able to pass this function a dirname that has a space in it, and get the right (partial) command. For the record, this change is quite safe because it checks whether quotes are already present. 


===== Below follows a minimal Cython example and traceback =====

===== test_.pyx
def foo():
print('hello')


===== setup.py
import os, sys
from Cython.Distutils import build_ext
from distutils.core import setup
from distutils.extension import Extension
from numpy.distutils.misc_util import get_numpy_include_dirs

# Ugly hack so I can run setup.py in my IDE
sys.argv = ['setup.py', 'build_ext', '--inplace']

# Init include dirs
include_dirs = ['.']
include_dirs.extend(get_numpy_include_dirs())

# Creat Extensions
ext_modules = [
     Extension('test_', ['test_.pyx'],
        include_dirs=include_dirs,
        ),
     ]

# Compile
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules,
    )

print('Successfully compiled cython file: test_')


===== output when running setup.py
running build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
cythoning test_.pyx to test_.c
building 'test_' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Program Files (x86)\python32\libs /LIBPATH:C:\Program Files (x86)\python32\PCbuild /EXPORT:PyInit_test_ build\temp.win32-3.2\Release\test_.obj /OUT:C:\almar\projects\py\cmu1394\test_.pyd /IMPLIB:build\temp.win32-3.2\Release\test_.lib /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'Files.obj'

----------
assignee: tarek
components: Distutils
files: distutils_spaces_in_libdir.patch
keywords: patch
messages: 151054
nosy: almar, eric.araujo, mhammond, tarek
priority: normal
severity: normal
status: open
title: Distutils does not put quotes around paths that contain spaces when compiling with MSVC
type: behavior
versions: Python 2.6, Python 3.2
Added file: http://bugs.python.org/file24201/distutils_spaces_in_libdir.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Wed Jan 11 11:59:14 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 10:59:14 +0000
Subject: [issue13766] explain the relationship between Lib/lib2to3/Grammar.txt
	and Grammar/Grammar
Message-ID: <1326279554.15.0.713951578682.issue13766@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

Lib/lib2to3/Grammar.txt seems like a modified copy of Grammar/Grammar. Can you state so somewhere, maybe in the beginning of Lib/lib2to3/Grammar.txt (the Note section).

One problem with the way it is now, is that there are changes done to Grammar/Grammar that are not put into Lib/lib2to3/Grammar.txt. Maybe they are minor, but I think we don't want them diverging too much.

Another problem is that the Note on Lib/lib2to3/Grammar.txt maybe irrelevant, since this is not the authoritative Grammar file.

----------
assignee: docs at python
components: Documentation
messages: 151055
nosy: docs at python, tshepang
priority: normal
severity: normal
status: open
title: explain the relationship between Lib/lib2to3/Grammar.txt and Grammar/Grammar
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13766>
_______________________________________

From report at bugs.python.org  Wed Jan 11 12:22:30 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 11 Jan 2012 11:22:30 +0000
Subject: [issue13762] missing section: how to contribute to devguide
In-Reply-To: <1326276497.12.0.812256237013.issue13762@psf.upfronthosting.co.za>
Message-ID: <1326280950.36.0.362947562444.issue13762@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hello Tshepang,
The current section is at: http://docs.python.org/devguide/docquality.html#helping-with-the-developer-s-guide reachable from the mainpage looking for helping with documentation.

it may be a little to condense but it is a good start (and it seems to cover the points you made in the issue): do you find it useful as it is? would you like to add something?

----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13762>
_______________________________________

From report at bugs.python.org  Wed Jan 11 13:06:55 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 12:06:55 +0000
Subject: [issue13762] missing section: how to contribute to devguide
In-Reply-To: <1326276497.12.0.812256237013.issue13762@psf.upfronthosting.co.za>
Message-ID: <1326283615.37.0.361221574737.issue13762@psf.upfronthosting.co.za>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

I guess I should have looked more carefully. Thanks and sorry for the noise.

----------
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13762>
_______________________________________

From report at bugs.python.org  Wed Jan 11 13:31:16 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 12:31:16 +0000
Subject: [issue2175] Expat sax parser silently ignores the InputSource protocol
In-Reply-To: <1203861783.69.0.636987169656.issue2175@psf.upfronthosting.co.za>
Message-ID: <1326285076.11.0.837383709248.issue2175@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2175>
_______________________________________

From report at bugs.python.org  Wed Jan 11 13:29:46 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 12:29:46 +0000
Subject: [issue12779] Update packaging documentation
In-Reply-To: <1313686169.0.0.406072932055.issue12779@psf.upfronthosting.co.za>
Message-ID: <1326284986.67.0.790902982651.issue12779@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: +tshepang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12779>
_______________________________________

From report at bugs.python.org  Wed Jan 11 13:44:08 2012
From: report at bugs.python.org (Floris Bruynooghe)
Date: Wed, 11 Jan 2012 12:44:08 +0000
Subject: [issue1975] signals not always delivered to main thread,
	since other threads have the signal unmasked
In-Reply-To: <1201711085.9.0.419365608938.issue1975@psf.upfronthosting.co.za>
Message-ID: <1326285848.91.0.621653458344.issue1975@psf.upfronthosting.co.za>


Changes by Floris Bruynooghe <floris.bruynooghe at gmail.com>:


----------
nosy: +flub

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1975>
_______________________________________

From report at bugs.python.org  Wed Jan 11 14:24:52 2012
From: report at bugs.python.org (Zaheer Merali)
Date: Wed, 11 Jan 2012 13:24:52 +0000
Subject: [issue13767] Would be nice to have a future import that turned off
	old except style
Message-ID: <1326288292.54.0.187974704865.issue13767@psf.upfronthosting.co.za>


New submission from Zaheer Merali <zaheermerali at gmail.com>:

It would be nice to have a future import to be able to prevent people in code doing:

try:
    ....
except Exception, e:
    ....

and make people do:

try:
    ....
except Exception as e:
    ....

----------
components: Interpreter Core
messages: 151058
nosy: Zaheer.Merali
priority: normal
severity: normal
status: open
title: Would be nice to have a future import that turned off old except style
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13767>
_______________________________________

From report at bugs.python.org  Wed Jan 11 13:17:04 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 11 Jan 2012 12:17:04 +0000
Subject: [issue13762] missing section: how to contribute to devguide
In-Reply-To: <1326276497.12.0.812256237013.issue13762@psf.upfronthosting.co.za>
Message-ID: <1326284224.73.0.19328666737.issue13762@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
resolution:  -> invalid

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13762>
_______________________________________

From report at bugs.python.org  Wed Jan 11 15:00:30 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Wed, 11 Jan 2012 14:00:30 +0000
Subject: [issue13767] Would be nice to have a future import that turned off
	old except style
In-Reply-To: <1326288292.54.0.187974704865.issue13767@psf.upfronthosting.co.za>
Message-ID: <1326290430.87.0.43959264113.issue13767@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

If you run python with the -3 flag you should get a warning about that.  If you combine it with -We you should be able to get an error instead of a warning.  Note however that both the flags are not specific for the "except" but also generate/affect other warnings as well.

----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13767>
_______________________________________

From report at bugs.python.org  Wed Jan 11 15:05:38 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Wed, 11 Jan 2012 14:05:38 +0000
Subject: [issue13767] Would be nice to have a future import that turned off
	old except style
In-Reply-To: <1326288292.54.0.187974704865.issue13767@psf.upfronthosting.co.za>
Message-ID: <1326290738.88.0.0660540545788.issue13767@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Unfortunately, that would be a new feature, so it is not acceptable to Python 2.7.

----------
nosy: +benjamin.peterson
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13767>
_______________________________________

From report at bugs.python.org  Wed Jan 11 15:34:18 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 14:34:18 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwZraWfhT_bXn2F-BmYUWkPKf5Gg1evRrijffn2xmTXjzg@mail.gmail.com>
Message-ID: <4F0D9DE3.6010509@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner at haypocalc.com> added the comment:
> 
>>  * it is exceedingly complex
> 
> Which part exactly? For hash(str), it just add two extra XOR.

I'm not talking specifically about your patch, but the whole idea
and the needed changes in general.

>>  * the method would need to be implemented for all hashable Python types
> 
> It was already discussed, and it was said that only hash(str) need to
> be modified.

Really ? What about the much simpler attack on integer hash values ?

You only have to send a specially crafted JSON dictionary with integer
keys to a Python web server providing JSON interfaces in order to
trigger the integer hash attack.

The same goes for the other Python data types.

>>  * it causes startup time to increase (you need urandom data for
>>   every single hashable Python data type)
> 
> My patch reads 8 or 16 bytes from /dev/urandom which doesn't block. Do
> you have a benchmark showing a difference?
> 
> I didn't try my patch on Windows yet.

Your patch only implements the simple idea of adding an init
vector and a fixed suffix vector (which you don't need since
it doesn't prevent hash collisions).

I don't think that's good enough, since
it doesn't change how the hash algorithm works on the actual
data, but instead just shifts the algorithm to a different
sequence. If you apply the same logic to the integer hash
function, you'll see that more clearly.

Paul's algorithm is much more secure in this respect, but it
requires more random startup data.

>>  * it causes run-time to increase due to changes in the hash
>>   algorithm (more operations in the tight loop)
> 
> I posted a micro-benchmark on hash(str) on python-dev: the overhead is
> nul. Did you have numbers showing that the overhead is not nul?

For the simple solution, that's an expected result, but if you want
more safety, then you'll see a hit due to the random data getting
XOR'ed in every single loop.

>>  * causes different processes in a multi-process setup to use different
>>   hashes for the same object
> 
> Correct. If you need to get the same hash, you can disable the
> randomized hash (PYTHONHASHSEED=0) or use a fixed seed (e.g.
> PYTHONHASHSEED=42).

So you have the choice of being able to work in a multi-process
environment and be vulnerable to the attack or not. I think we
can do better :-)

Note that web servers written in Python tend to be long running
processes, so an attacker has lots of time to test various
seeds.

>>  * doesn't appear to work well in embedded interpreters that
>>   regularly restarted interpreters (AFAIK, some objects persist across
>>   restarts and those will have wrong hash values in the newly started
>>   instances)
> 
> test_capi runs _testembed which restarts a embedded interpreters 3
> times, and the test pass (with my patch version 5). Can you write a
> script showing the problem if there is a real problem?
> 
> In an older version of my patch, the hash secret was recreated at each
> initiliazation. I changed my patch to only generate the secret once.

Ok, that should fix the case.

Two more issue that I forgot:

 * enabling randomized hashing can make debugging a lot harder, since
   it's rather difficult to reproduce the same state in a controlled
   way (unless you record the hash seed somewhere in the logs)

and even though applications should not rely on the order of dict
repr()s or str()s, they do often enough:

 * randomized hashing will result in repr() and str() of dictionaries
   to be random as well

>> The most important issue, though, is that it doesn't really
>> protect Python against the attack - it only makes it less
>> likely that an adversary will find the init vector (or a way
>> around having to find it via crypt analysis).
> 
> I agree that the patch is not perfect. As written in the patch, it
> just makes the attack more complex. I consider that it is enough.

Wouldn't you rather see a fix that works for all hash functions
and Python objects ? One that doesn't cause performance
issues ?

The collision counting idea has this potential.

> Perl has a simpler protection than the one proposed in my patch. Is
> Perl vulnerable to the hash collision vulnerability?

I don't know what Perl did or how hashing works in Perl, so cannot
comment on the effect of their fix. FWIW, I don't think that we
should use Perl or Java as reference here.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 15:45:35 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 14:45:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0D9DE3.6010509@egenix.com>
Message-ID: <1326293048.3531.6.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> OTOH, the collision counting patch is very simple, doesn't have
> the performance issues and provides real protection against the
> attack.

I don't know about real protection: you can still slow down dict
construction by 1000x (the number of allowed collisions per lookup),
which can be enough combined with a brute-force DOS.

Also, how about false positives? Having legitimate programs break
because of legitimate data would be a disaster.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 15:55:55 2012
From: report at bugs.python.org (Mark Shannon)
Date: Wed, 11 Jan 2012 14:55:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0D9DE3.6010509@egenix.com>
Message-ID: <4F0DA2FD.9060209@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

>>>  * the method would need to be implemented for all hashable Python types
>> It was already discussed, and it was said that only hash(str) need to
>> be modified.
> 
> Really ? What about the much simpler attack on integer hash values ?
> 
> You only have to send a specially crafted JSON dictionary with integer
> keys to a Python web server providing JSON interfaces in order to
> trigger the integer hash attack.

JSON objects are decoded as dicts with string keys, integers keys are 
not possible.

 >>> json.loads(json.dumps({1:2}))
{'1': 2}

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 16:12:50 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 11 Jan 2012 15:12:50 +0000
Subject: [issue13768] Doc/tools/dailybuild.py available only on 2.7 branch
Message-ID: <1326294770.64.0.354971567958.issue13768@psf.upfronthosting.co.za>


Changes by Tshepang Lekhonkhobe <tshepang at gmail.com>:


----------
nosy: tshepang
priority: normal
severity: normal
status: open
title: Doc/tools/dailybuild.py available only on 2.7 branch
versions: Python 2.6, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13768>
_______________________________________

From report at bugs.python.org  Wed Jan 11 16:41:09 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 15:41:09 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0DA2FD.9060209@hotpy.org>
Message-ID: <4F0DAD8F.1020808@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Mark Shannon wrote:
> 
> Mark Shannon <mark at hotpy.org> added the comment:
> 
>>>>  * the method would need to be implemented for all hashable Python types
>>> It was already discussed, and it was said that only hash(str) need to
>>> be modified.
>>
>> Really ? What about the much simpler attack on integer hash values ?
>>
>> You only have to send a specially crafted JSON dictionary with integer
>> keys to a Python web server providing JSON interfaces in order to
>> trigger the integer hash attack.
> 
> JSON objects are decoded as dicts with string keys, integers keys are 
> not possible.
> 
>  >>> json.loads(json.dumps({1:2}))
> {'1': 2}

Thanks for the correction. Looks like XML-RPC also doesn't accept
integers as dict keys. That's good :-)

However, as Paul already noted, such attacks can also occur in other
places or parsers in an application, e.g. when decoding FORM parameters
that use integers to signal a line or parameter position (example:
value_1=2&value_2=3...) which are then converted into a dictionary
mapping the position integer to the data.

marshal and pickle are vulnerable, but then you normally don't expose
those to untrusted data.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 17:03:19 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 16:03:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326293048.3531.6.camel@localhost.localdomain>
Message-ID: <4F0DB2C1.6010308@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou at free.fr> added the comment:
> 
>> OTOH, the collision counting patch is very simple, doesn't have
>> the performance issues and provides real protection against the
>> attack.
> 
> I don't know about real protection: you can still slow down dict
> construction by 1000x (the number of allowed collisions per lookup),
> which can be enough combined with a brute-force DOS.

On my slow dev machine 1000 collisions run in around 22ms:

python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 1000))"
100 loops, best of 3: 22.4 msec per loop

Using this for a DOS attack would be rather noisy, much unlike
sending a single POST.

Note that the choice of 1000 as limit is rather arbitrary. I just
chose it because it's high enough because it's very unlikely to be
hit by an application that is not written to trigger it and it's low
enough to still provide a good run-time behavior. Perhaps an
even lower figure would be better.

> Also, how about false positives? Having legitimate programs break
> because of legitimate data would be a disaster.

Yes, which is why the patch should be disabled by default (using
an env var) in dot-releases. It's probably also a good idea to
make the limit configurable to adjust to ones needs.

Still, it is *very* unlikely that you run into real data causing
more than 1000 collisions for a single insert.

For full protection the universal hash method idea would have
to be implemented (adding a parameter to the hash methods, so
that they can be parametrized). This would then allow switching
the dict to an alternative hash implementation resolving the collision
problem, in case the implementation detects high number of
collisions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 17:41:19 2012
From: report at bugs.python.org (=?utf-8?b?0JzQsNGA0Log0JrQvtGA0LXQvdCx0LXRgNCz?=)
Date: Wed, 11 Jan 2012 16:41:19 +0000
Subject: [issue13769] json.dump(ensure_ascii=False) return str instead of
	unicode
Message-ID: <1326300079.82.0.279410736545.issue13769@psf.upfronthosting.co.za>


New submission from ???? ????????? <socketpair at gmail.com>:

$ ipython
In [1]: type(json.dumps({'a':'b'}, ensure_ascii=False))
Out[1]: <type 'str'>

In [2]: type(json.dumps({'a':u'b'}, ensure_ascii=False))
Out[2]: <type 'unicode'>
-----------------------
Documentation:
If ensure_ascii is False, then the return value will be a unicode instance.
--------------------------------

Not applicable to python3

----------
assignee: docs at python
components: Documentation, Library (Lib)
messages: 151066
nosy: docs at python, mmarkk
priority: normal
severity: normal
status: open
title: json.dump(ensure_ascii=False) return str instead of unicode
type: behavior
versions: Python 2.6, Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13769>
_______________________________________

From report at bugs.python.org  Wed Jan 11 17:42:11 2012
From: report at bugs.python.org (=?utf-8?b?0JzQsNGA0Log0JrQvtGA0LXQvdCx0LXRgNCz?=)
Date: Wed, 11 Jan 2012 16:42:11 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
Message-ID: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>


New submission from ???? ????????? <socketpair at gmail.com>:

Please fix documentation about ensure_ascii. Does it have any meaning in python3?

----------
assignee: docs at python
components: Documentation
messages: 151067
nosy: docs at python, mmarkk
priority: normal
severity: normal
status: open
title: python3 & json: add ensure_ascii documentation
versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Wed Jan 11 17:42:40 2012
From: report at bugs.python.org (=?utf-8?b?0JzQsNGA0Log0JrQvtGA0LXQvdCx0LXRgNCz?=)
Date: Wed, 11 Jan 2012 16:42:40 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
In-Reply-To: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>
Message-ID: <1326300160.28.0.935080447763.issue13770@psf.upfronthosting.co.za>


Changes by ???? ????????? <socketpair at gmail.com>:


----------
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Wed Jan 11 17:57:10 2012
From: report at bugs.python.org (Tal Einat)
Date: Wed, 11 Jan 2012 16:57:10 +0000
Subject: [issue1692335] Fix exception pickling: Move initial args assignment
	to BaseException.__new__
Message-ID: <1326301030.41.0.790370225151.issue1692335@psf.upfronthosting.co.za>


Tal Einat <taleinat at gmail.com> added the comment:

I'd just like to weigh in and say that this is a major issue for me at the moment. Not being able to indiscriminately pickle/unpickle exceptions is making my parallel-processing work very painful, because of problematic stdlib exceptions.

I'm surprised this hasn't been fixed way back in 2.x. FWIW, for my project having this fixed in 3.x could be a significant incentive to finally ditch 2.x.

----------
nosy: +taleinat

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1692335>
_______________________________________

From report at bugs.python.org  Wed Jan 11 18:28:01 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 17:28:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F0DB2C1.6010308@egenix.com>
Message-ID: <1326302790.3531.16.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> On my slow dev machine 1000 collisions run in around 22ms:
> 
> python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 1000))"
> 100 loops, best of 3: 22.4 msec per loop
> 
> Using this for a DOS attack would be rather noisy, much unlike
> sending a single POST.

Note that sending one POST is not enough, unless the attacker is content
with blocking *one* worker process for a couple of seconds or minutes
(which is a rather tiny attack if you ask me :-)). Also, you can combine
many dicts in a single JSON list, so that the 1000 limit isn't
overreached for any of the dicts.

So in all cases the attacker would have to send many of these POST
requests in order to overwhelm the target machine. That's how DOS
attacks work AFAIK.

> Yes, which is why the patch should be disabled by default (using
> an env var) in dot-releases. It's probably also a good idea to
> make the limit configurable to adjust to ones needs.

Agreed if it's disabled by default then it's not a problem, but then
Python is vulnerable by default...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 18:34:33 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Wed, 11 Jan 2012 17:34:33 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326303273.14.0.0974032239331.issue13703@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

[Antoine]
> Also, how about false positives? Having legitimate programs break
> because of legitimate data would be a disaster.

This worries me, too.

[MAL]
> Yes, which is why the patch should be disabled by default (using
> an env var) in dot-releases.

Are you proposing having it enabled by default in Python 3.3?

----------
nosy: +mark.dickinson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 18:38:10 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 17:38:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326303273.14.0.0974032239331.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0DC8FB.8050909@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Mark Dickinson wrote:
> 
> Mark Dickinson <dickinsm at gmail.com> added the comment:
> 
> [Antoine]
>> Also, how about false positives? Having legitimate programs break
>> because of legitimate data would be a disaster.
> 
> This worries me, too.
> 
> [MAL]
>> Yes, which is why the patch should be disabled by default (using
>> an env var) in dot-releases.
> 
> Are you proposing having it enabled by default in Python 3.3?

Possibly, yes. Depends on whether anyone comes up with a problem in
the alpha, beta, RC release cycle.

It would be great to have the universal hash method approach for
Python 3.3. That way Python could self-heal itself in case it
finds too many collisions. My guess is that it's still better
to raise an exception, though, since it would uncover either
attacks or programming errors.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 18:53:11 2012
From: report at bugs.python.org (Michael Mulich)
Date: Wed, 11 Jan 2012 17:53:11 +0000
Subject: [issue13771] HTTPSConnection __init__ super implementation causes
	recursion error
Message-ID: <1326304391.56.0.972578337686.issue13771@psf.upfronthosting.co.za>


New submission from Michael Mulich <michael.mulich at gmail.com>:

While working on porting wsgi_intercept to Python 3 I came across a recursion issue with http.client.HTTPSConnection. The following is an lesser extraction of the traceback:


Traceback (most recent call last):
  File ".../wsgi_intercept/test/test_httplib.py", line 28, in test_success
    http = self.make_one(self.domain)
  File ".../wsgi_intercept/test/test_httplib.py", line 40, in make_one
    return http.client.HTTPSConnection(*args)
  File ".../lib/python3.2/http/client.py", line 1075, in __init__
    source_address)
  ...
  File ".../lib/python3.2/http/client.py", line 1075, in __init__
    source_address)
  File ".../lib/python3.2/http/client.py", line 1074, in __init__
    super(HTTPSConnection, self).__init__(host, port, strict, timeout,
RuntimeError: maximum recursion depth exceeded while calling a Python object

Some background information is necessary to explain what is happening here. One, this is a traceback from a test method (make_one) which makes and https connection. Two, wsgi_intercept has overridden http.client.HTTPSConnection with a class that subclasses HTTPSConnection and overrides a few methonds. For more general information, see the PyPI page (http://pypi.python.org/pypi/wsgi_intercept). 

After the wsgi_intercept behavior is invoked the __mro__ of http.client.HTTPSConnection becomes: (<class 'wsgi_intercept.WSGI_HTTPSConnection'>, <class 'http.client.HTTPSConnection'>, <class 'http.client.HTTPConnection'>, <class 'object'>). Because of this we end up recursively running the __init__.

Possible solutions:

1) Fix the issue in http/client.py:1074 by replacing "super(HTTPSConnection, self)" with "super()", which if I'm not mistaken is the recommended usage of super in Python 3.
2) Fix the issue in the wsgi_intercept package.

I was successful with both approaches, but applying the second fix would make the maintenance of wsgi_intercept slightly harder because of the code duplication and round-about-way of calling its parent classes.

----------
components: None
messages: 151072
nosy: michael.mulich
priority: normal
severity: normal
status: open
title: HTTPSConnection __init__ super implementation causes recursion error
type: behavior
versions: Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13771>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:03:00 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 18:03:00 +0000
Subject: [issue13771] HTTPSConnection __init__ super implementation causes
	recursion error
In-Reply-To: <1326304391.56.0.972578337686.issue13771@psf.upfronthosting.co.za>
Message-ID: <1326304980.43.0.326084039627.issue13771@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
components: +Library (Lib) -None
nosy: +orsenthil
versions:  -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13771>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:05:30 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 11 Jan 2012 18:05:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326302790.3531.16.camel@localhost.localdomain>
Message-ID: <4F0DCF62.5020209@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou at free.fr> added the comment:
> 
>> On my slow dev machine 1000 collisions run in around 22ms:
>>
>> python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 1000))"
>> 100 loops, best of 3: 22.4 msec per loop
>>
>> Using this for a DOS attack would be rather noisy, much unlike
>> sending a single POST.
> 
> Note that sending one POST is not enough, unless the attacker is content
> with blocking *one* worker process for a couple of seconds or minutes
> (which is a rather tiny attack if you ask me :-)). Also, you can combine
> many dicts in a single JSON list, so that the 1000 limit isn't
> overreached for any of the dicts.

Right, but such an approach only scales linearly and doesn't
exhibit the quadric nature of the collision resolution.

The above with 10000 items takes 5 seconds on my machine.
The same with 100000 items is still running after 16 minutes.

> So in all cases the attacker would have to send many of these POST
> requests in order to overwhelm the target machine. That's how DOS
> attacks work AFAIK.

Depends :-) Hiding a few tens of such requests in the input stream
of a busy server is easy. Doing the same with thousands of requests
is a lot harder.

FWIW: The above dict string version just has some 263kB for the 100000
case, 114kB if gzip compressed.

>> Yes, which is why the patch should be disabled by default (using
>> an env var) in dot-releases. It's probably also a good idea to
>> make the limit configurable to adjust to ones needs.
> 
> Agreed if it's disabled by default then it's not a problem, but then
> Python is vulnerable by default...

Yes, but at least the user has an option to switch on the added
protection. We'd need some field data to come to a decision.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:18:16 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 18:18:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326303273.14.0.0974032239331.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326305809.3531.23.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> [MAL]
> > Yes, which is why the patch should be disabled by default (using
> > an env var) in dot-releases.
> 
> Are you proposing having it enabled by default in Python 3.3?

I would personally prefer 3.3 and even 3.2 to have proper randomization
(either Paul's or Victor's or another proposal). Victor's proposal makes
fixing other hash functions very simple (there could even be helper
macros). The only serious concern IMO is startup time under Windows;
someone with Windows-fu should investigate that.

2.x maintainers might want to be more conservative, although disabling a
fix (the collision counter) by default doesn't sound very wise or
helpful to me.
(for completeness, the collision counter must also be added to sets,
btw)

It would be nice to hear from distro maintainers here.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:27:35 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Wed, 11 Jan 2012 18:27:35 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326306455.25.0.0505251792324.issue13761@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

In the python-ideas discussion people have argued that flush=False should or could be interpreted as "definitely do not flush" which is unimplementable (the buffer may be full, or the stream may be unbuffered, and there is no way to tell a write() call to skip the flushing if the stream's policy would be to flush). Sticklers have proposed to name the flag "force_flush" to avoid this ambiguity, or to pass None instead of False.

I think that's all being hypercorrect -- first of all, nobody is going to explicitly write flush=False since that is the default, and second of all, who could possibly care about not flushing on a per-call basis? The flag should have a short name and simple semantics. flush=True/False does this: if flush is true, an explicit flush() call is made, if it is false, flush() is not called. What the stream does is its business.

----------
nosy: +gvanrossum

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:41:30 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 18:41:30 +0000
Subject: [issue13758] compile() should not encode 'filename' (at least on
	Windows)
In-Reply-To: <1326253604.11.0.305060780894.issue13758@psf.upfronthosting.co.za>
Message-ID: <1326307290.94.0.0230588172486.issue13758@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

My supposition that compile() rejects some real file names appears correct: from python-list
ME: Is this a filename that could be an actual, valid filename on your system?
OP: Yes it is. open works on that file.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13758>
_______________________________________

From report at bugs.python.org  Wed Jan 11 19:58:53 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Wed, 11 Jan 2012 18:58:53 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <CAH_1eM3u2gcQzvW42oq2ARB2GXwS3aOzFQNAYU0Y-oVc03qRGQ@mail.gmail.com>
Message-ID: <CAH_1eM2H0EAW7GfH078+97y35W1ey5hH1djugmCmnXDF39x8Ow@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's an updated version.

Note that I'm not pushing towards changing the current behavior
pertaining to symlinks to directories, because if we change this, this
will break code.
For example to count the number of lines of all the files under a
directory, a code could go like this:

for root, dirs, files in os.walk(top):
    for file in files:
        f = open(file)
        for n, l in enumerate(f, 1):
            pass
        print(file, n)

If, suddently, a symlink to a directory appeared in files, this will
break. So I'm not convinced it's worth changing this. A symlink to a
directory is not much closer to a file than to a directory, it really
depends on the use case.
I'm also fine with keeping fdwalk() consistent with this to make
porting easier (and also because it makes it easy to test, I just have
to compare fdwlak()'s output to walk()'s output).

----------
Added file: http://bugs.python.org/file24202/fdwalk-1.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2240,6 +2240,58 @@
               os.rmdir(os.path.join(root, name))
 
 
+.. function:: fdwalk(top, topdown=True, onerror=None, followlinks=False)
+
+   .. index::
+      single: directory; walking
+      single: directory; traversal
+
+    This behaves exactly like :func:`walk`, except that it yields a 4-tuple
+    ``(dirpath, dirnames, filenames, dirfd)``.
+
+   *dirpath*, *dirnames* and *filenames* are identical to :func:`walk` output,
+   and *dirfd* is a file descriptor referring to the directory *dirpath*.
+
+   .. note::
+
+      Since :func:`fdwalk` yields file descriptors, those are only valid until
+      the next iteration step, so you should duplicate them (e.g. with
+      :func:`dup`) if you want to keep them longer.
+
+   .. note::
+
+      Contrarily to :func:`walk`, modifying the dirnames list in-place won't
+      affect the directories traversed.
+
+   This example displays the number of bytes taken by non-directory files in each
+   directory under the starting directory::
+
+      import os
+      for root, dirs, files, rootfd in os.fdwalk('python/Lib/email'):
+          print(root, "consumes", end="")
+          print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                end="")
+          print("bytes in", len(files), "non-directory files")
+
+   In the next example, walking the tree bottom-up is essential:
+   :func:`unlinkat` doesn't allow deleting a directory before the directory is
+   empty::
+
+      # Delete everything reachable from the directory named in "top",
+      # assuming there are no symbolic links.
+      # CAUTION:  This is dangerous!  For example, if top == '/', it
+      # could delete all your disk files.
+      import os
+      for root, dirs, files, rootfd in os.fdwalk(top, topdown=False):
+          for name in files:
+              os.unlinkat(rootfd, name)
+          for name in dirs:
+              os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+
+   Availability: Unix.
+
+   .. versionadded:: 3.3
+
 .. _os-process:
 
 Process Management
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -478,6 +478,10 @@
 
   (Patch submitted by Giampaolo Rodol?? in :issue:`10784`.)
 
+* The :mod:`os` module has a new :func:`~os.fdwalk` function similar to
+  :func:`~os.walk` except that it also yields file descriptors referring to the
+  directories visited. This is especially useful to avoid symlink races.
+
 * "at" functions (:issue:`4761`):
 
   * :func:`~os.faccessat`
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -24,6 +24,7 @@
 #'
 
 import sys, errno
+import stat as st
 
 _names = sys.builtin_module_names
 
@@ -32,6 +33,9 @@
            "defpath", "name", "path", "devnull",
            "SEEK_SET", "SEEK_CUR", "SEEK_END"]
 
+def _exists(name):
+    return name in globals()
+
 def _get_exports_list(module):
     try:
         return list(module.__all__)
@@ -120,7 +124,13 @@
     umask(mask)
     return mode & ~mask
 
-#'
+def _are_same_file(stat1, stat2):
+    """Helper function that checks whether two stat results refer to the same
+    file.
+    """
+    return (stat1.st_mode == stat2.st_mode and stat1.st_ino == stat2.st_ino and
+            stat1.st_dev == stat2.st_dev)
+#
 
 # Super directory utilities.
 # (Inspired by Eric Raymond; the doc strings are mostly his)
@@ -151,7 +161,6 @@
     try:
         mkdir(name, mode)
     except OSError as e:
-        import stat as st
         if not (e.errno == errno.EEXIST and exist_ok and path.isdir(name) and
                 st.S_IMODE(lstat(name).st_mode) == _get_masked_mode(mode)):
             raise
@@ -298,6 +307,94 @@
 
 __all__.append("walk")
 
+if _exists("openat"):
+
+    def fdwalk(top, topdown=True, onerror=None, followlinks=False):
+        """Directory tree generator.
+
+        This behaves exactly like walk(), except that it yields a 4-tuple
+
+            dirpath, dirnames, filenames, dirfd
+
+        `dirpath`, `dirnames` and `filenames` are identical to walk() output,
+        and `dirfd` is a file descriptor referring to the directory `dirpath`.
+
+        The advantage of walkfd() over walk() is that it's safe against symlink
+        races (when followlinks is False).
+
+        Caution:
+        Since fdwalk() yields file descriptors, those are only valid until the
+        next iteration step, so you should dup() them if you want to keep them
+        for a longer period.
+        Also, contrarily to walk(), modifying the `dirnames` list in-place won't
+        affect the directories traversed.
+
+        Example:
+
+        import os
+        for root, dirs, files, rootfd in os.fdwalk('python/Lib/email'):
+            print(root, "consumes", end="")
+            print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                  end="")
+            print("bytes in", len(files), "non-directory files")
+        """
+        # Note: To guard against symlink races, we use the standard
+        # lstat()/open()/fstat() trick.
+        orig_st = lstat(top)
+        topfd = open(top, O_RDONLY)
+        try:
+            if (followlinks or (st.S_ISDIR(orig_st.st_mode) and
+                               _are_same_file(orig_st, fstat(topfd)))):
+                for x in _fdwalk(topfd, top, topdown, onerror, followlinks):
+                    yield x
+        finally:
+            close(topfd)
+
+    def _fdwalk(topfd, toppath, topdown, onerror, followlinks):
+        try:
+            names = fdlistdir(topfd)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return
+
+        dirs, nondirs = [], []
+        for name in names:
+            # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
+            # walk() which reports symlinks to directories as directories. We do
+            # however check for symlinks before recursing into a subdirectory.
+            if st.S_ISDIR(fstatat(topfd, name).st_mode):
+                dirs.append(name)
+            else:
+                nondirs.append(name)
+
+        # whether to follow symlinks
+        flag = 0 if followlinks else AT_SYMLINK_NOFOLLOW
+
+        if topdown:
+            yield toppath, dirs, nondirs, topfd
+
+        for name in dirs:
+            try:
+                orig_st = fstatat(topfd, name, flag)
+                dirfd = openat(topfd, name, O_RDONLY)
+            except error as err:
+                if onerror is not None:
+                    onerror(err)
+                return
+            try:
+                if followlinks or _are_same_file(orig_st, fstat(dirfd)):
+                    dirpath = path.join(toppath, name)
+                    for x in _fdwalk(dirfd, dirpath, topdown, onerror, followlinks):
+                        yield x
+            finally:
+                close(dirfd)
+
+        if not topdown:
+            yield toppath, dirs, nondirs, topfd
+
+    __all__.append("fdwalk")
+
 # Make sure os.environ exists, at least
 try:
     environ
@@ -598,9 +695,6 @@
 fsencode, fsdecode = _fscodec()
 del _fscodec
 
-def _exists(name):
-    return name in globals()
-
 # Supply spawn*() (probably only for Unix)
 if _exists("fork") and not _exists("spawnv") and _exists("execv"):
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -20,6 +20,8 @@
 import asyncore
 import asynchat
 import socket
+import itertools
+import stat
 try:
     import threading
 except ImportError:
@@ -147,7 +149,6 @@
         if not hasattr(os, "stat"):
             return
 
-        import stat
         result = os.stat(fname)
 
         # Make sure direct access works
@@ -464,7 +465,7 @@
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
-    def test_traversal(self):
+    def setUp(self):
         import os
         from os.path import join
 
@@ -569,6 +570,58 @@
                     os.remove(dirname)
         os.rmdir(support.TESTFN)
 
+ at unittest.skipUnless(hasattr(os, 'fdwalk'), "Test needs os.fdwalk()")
+class FdWalkTests(WalkTests):
+    """Tests for os.fdwalk()."""
+
+    def test_compare_to_walk(self):
+        # compare with walk() results
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            expected = {}
+            for root, dirs, files in os.walk(*args):
+                expected[root] = (set(dirs), set(files))
+
+            for root, dirs, files, rootfd in os.fdwalk(*args):
+                self.assertIn(root, expected)
+                self.assertEqual(expected[root], (set(dirs), set(files)))
+
+    def test_dir_fd(self):
+        # check returned file descriptors
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            for root, dirs, files, rootfd in os.fdwalk(*args):
+                # check that the FD is valid
+                os.fstat(rootfd)
+                # check that fdlistdir() returns consistent information
+                self.assertEqual(set(os.fdlistdir(rootfd)), set(dirs) | set(files))
+
+    def test_fd_leak(self):
+        # Since we're opening a lot of FDs, we must be careful to avoid leaks:
+        # we both check that calling fdwalk() a large number of times doesn't
+        # yield EMFILE, and that the minimum allocated FD hasn't changed.
+        minfd = os.dup(1)
+        os.close(minfd)
+        for i in range(512):
+            for x in os.fdwalk(support.TESTFN):
+                pass
+        newfd = os.dup(1)
+        self.addCleanup(os.close, newfd)
+        self.assertEqual(newfd, minfd)
+
+    def tearDown(self):
+        # cleanup
+        for root, dirs, files, rootfd in os.fdwalk(support.TESTFN, topdown=False):
+            for name in files:
+                os.unlinkat(rootfd, name)
+            for name in dirs:
+                st = os.fstatat(rootfd, name, os.AT_SYMLINK_NOFOLLOW)
+                if stat.S_ISDIR(st.st_mode):
+                    os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+                else:
+                    os.unlinkat(rootfd, name)
+        os.rmdir(support.TESTFN)
+
 class MakedirTests(unittest.TestCase):
     def setUp(self):
         os.mkdir(support.TESTFN)
@@ -1683,6 +1736,7 @@
         StatAttributeTests,
         EnvironTests,
         WalkTests,
+        FdWalkTests,
         MakedirTests,
         DevNullTests,
         URandomTests,

From report at bugs.python.org  Wed Jan 11 20:07:03 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 19:07:03 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326308823.86.0.673278558448.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I've benchmarked Victor's patch and got the following results:

Report on Linux localhost.localdomain 2.6.38.8-desktop-9.mga #1 SMP Tue Dec 20 09:45:44 UTC 2011 x86_64 x86_64
Total CPU cores: 4

### call_simple ###
Min: 0.223778 -> 0.209204: 1.07x faster
Avg: 0.227634 -> 0.212437: 1.07x faster
Significant (t=15.40)
Stddev: 0.00291 -> 0.00248: 1.1768x smaller
Timeline: http://tinyurl.com/87vkdps

### fastpickle ###
Min: 0.484052 -> 0.499832: 1.03x slower
Avg: 0.487370 -> 0.507909: 1.04x slower
Significant (t=-8.40)
Stddev: 0.00261 -> 0.00481: 1.8446x larger
Timeline: http://tinyurl.com/7ntcudz

### float ###
Min: 0.052819 -> 0.051540: 1.02x faster
Avg: 0.054304 -> 0.052922: 1.03x faster
Significant (t=3.89)
Stddev: 0.00125 -> 0.00126: 1.0101x larger
Timeline: http://tinyurl.com/7rqfurw

### formatted_logging ###
Min: 0.252709 -> 0.257303: 1.02x slower
Avg: 0.254741 -> 0.259967: 1.02x slower
Significant (t=-4.90)
Stddev: 0.00155 -> 0.00181: 1.1733x larger
Timeline: http://tinyurl.com/8xu2zdt

### normal_startup ###
Min: 0.450661 -> 0.435943: 1.03x faster
Avg: 0.454536 -> 0.438212: 1.04x faster
Significant (t=9.41)
Stddev: 0.00327 -> 0.00209: 1.5661x smaller
Timeline: http://tinyurl.com/8ygw272

### nqueens ###
Min: 0.269426 -> 0.255306: 1.06x faster
Avg: 0.270105 -> 0.255844: 1.06x faster
Significant (t=28.63)
Stddev: 0.00071 -> 0.00086: 1.2219x larger
Timeline: http://tinyurl.com/823dwzo

### regex_compile ###
Min: 0.390307 -> 0.380736: 1.03x faster
Avg: 0.391959 -> 0.382025: 1.03x faster
Significant (t=8.93)
Stddev: 0.00194 -> 0.00156: 1.2395x smaller
Timeline: http://tinyurl.com/72shbzh

### silent_logging ###
Min: 0.060115 -> 0.057777: 1.04x faster
Avg: 0.060241 -> 0.058019: 1.04x faster
Significant (t=13.29)
Stddev: 0.00010 -> 0.00036: 3.4695x larger
Timeline: http://tinyurl.com/76bfguf

### unpack_sequence ###
Min: 0.000043 -> 0.000046: 1.07x slower
Avg: 0.000044 -> 0.000047: 1.06x slower
Significant (t=-107.47)
Stddev: 0.00000 -> 0.00000: 1.1231x larger
Timeline: http://tinyurl.com/6us6yys

The following not significant results are hidden, use -v to show them:
call_method, call_method_slots, call_method_unknown, fastunpickle, iterative_count, json_dump, json_load, nbody, pidigits, regex_effbot, regex_v8, richards, simple_logging, startup_nosite, threaded_count.


In short, any difference is in the noise.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:10:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 19:10:25 +0000
Subject: [issue8184] multiprocessing.managers will not fail if listening ocket
	already in use
In-Reply-To: <CAH_1eM2Dyn5BUEZmnHX+1p7x=B73cF9LNGbtjz_i+0aVV3Hrow@mail.gmail.com>
Message-ID: <1326308937.3531.24.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I noticed that if bind() fails (in this case with EADDRINUSE), the
> socket isn't closed (FD leak).

Well it would probably be closed when the connection object is
destroyed, but the patch looks ok anyway.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8184>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:15:16 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 11 Jan 2012 19:15:16 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <E1Rl3dj-0007NH-57@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset bc043cef94f2 by Terry Jan Reedy in branch '3.2':
Closes #11633 Clarify print buffering.
http://hg.python.org/cpython/rev/bc043cef94f2

New changeset fb0d61fd1753 by Terry Jan Reedy in branch 'default':
Merge with 3.2
http://hg.python.org/cpython/rev/fb0d61fd1753

----------
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:16:36 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 19:16:36 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326309396.58.0.792430967995.issue11633@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

#13761 proposes to add flush=False param with option for True.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:22:01 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Wed, 11 Jan 2012 19:22:01 +0000
Subject: [issue13769] json.dump(ensure_ascii=False) return str instead of
	unicode
In-Reply-To: <1326300079.82.0.279410736545.issue13769@psf.upfronthosting.co.za>
Message-ID: <1326309721.87.0.820373228893.issue13769@psf.upfronthosting.co.za>


Changes by Martin v. L?wis <martin at v.loewis.de>:


----------
versions:  -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13769>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:28:07 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 11 Jan 2012 19:28:07 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326310087.73.0.0689228614333.issue13761@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I added a couple of lines to the 3.2, 3.3 print doc in #11633. When a change is made here, they could be updated to something like:
"Output buffering is normally determined by *file*. Use flush=True to force immediate output to a device such as a screen."

Unless there is a good reason, I prefer, unlike the other params, *not* allowing flush=None as a synonym for flush=<the default, ie False>.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:38:58 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 11 Jan 2012 19:38:58 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326310738.49.0.107429821933.issue13761@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

There is a reason: we don't usually check the type of flag arguments; just their truth value is used.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:50:21 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 11 Jan 2012 19:50:21 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326311421.19.0.287587222236.issue13761@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Here is a patch.

One open question is whether failure in flush() should be reraised (as implemented in the patch), or ignored (as in input()).

----------
keywords: +patch
Added file: http://bugs.python.org/file24203/flush.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:57:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 11 Jan 2012 19:57:47 +0000
Subject: [issue11418] Method's global scope is module containing function
	definition, not class.
In-Reply-To: <1299454101.64.0.841920247259.issue11418@psf.upfronthosting.co.za>
Message-ID: <E1Rl4Ir-00033k-PA@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 10a5165103f9 by Terry Jan Reedy in branch '2.7':
Minor correction. Closes #11418
http://hg.python.org/cpython/rev/10a5165103f9

New changeset 32ea3675fba2 by Terry Jan Reedy in branch '3.2':
Minor correction. #11418
http://hg.python.org/cpython/rev/32ea3675fba2

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11418>
_______________________________________

From report at bugs.python.org  Wed Jan 11 20:59:16 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 11 Jan 2012 19:59:16 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326311956.18.0.211337489029.issue12409@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi,
i'm at last back with some work to show; I've integrated all the "Documenting python" doc into devguide, and the commits are in this sandbox: http://hg.python.org/sandbox/morph-devguide/shortlog/a92d431ef749 (`documenting` branch).

I took the files from default branch and merged all of them in a single page. It might be a bit log, but it has the advantage (at least for me) that there you can find all the info you need for documenting python, a ctrl+f works and you don't need to follow tons of links to find what you're looking for.

then I've removed all the info from default branch (here: http://hg.python.org/sandbox/morph/shortlog/e9a4276b19d4 in default-documenting branch) which left only documenting/index.rst as mainly a reference to the devguide.

I don't know what would be the best way to handle this transition on the python doc side, maybe converting that to a simple Doc/documenting.rst as it's now index.rst? but what text to left?

Comments are very welcome!

----------
stage: needs patch -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Wed Jan 11 21:01:15 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 11 Jan 2012 20:01:15 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
Message-ID: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

Note how _getfinalpathname works and calling listdir on the final path name also works:

>>> os.symlink(".\\test", "Lib\\bar")
>>> os.listdir("Lib\\bar")[:4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotADirectoryError: [Error 267] The directory name is invalid: 'Lib\\bar\\*.*'

>>> nt._getfinalpathname("Lib\\bar")
'\\\\?\\C:\\t\\pathlib\\Lib\\test'
>>> os.listdir(nt._getfinalpathname("Lib\\bar"))[:4]
['185test.db', 'audiotest.au', 'autotest.py', 'badcert.pem']

Calling listdir on the non-final extended path doesn't work:

>>> os.listdir('\\\\?\\C:\\t\\pathlib\\Lib\\bar')[:4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotADirectoryError: [Error 267] The directory name is invalid: '\\\\?\\C:\\t\\pa
thlib\\Lib\\bar\\*.*'

But open() works:

>>> open('Lib\\bar\\regrtest.py')
<_io.TextIOWrapper name='Lib\\bar\\regrtest.py' mode='r' encoding='cp1252'>

----------
components: Library (Lib), Windows
messages: 151087
nosy: amaury.forgeotdarc, brian.curtin, pitrou, tim.golden
priority: normal
severity: normal
status: open
title: listdir() doesn't work with non-trivial symlinks
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Wed Jan 11 21:20:09 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Wed, 11 Jan 2012 20:20:09 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326313209.66.0.271597035462.issue12736@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

New patch with title casing mappings added.

----------
Added file: http://bugs.python.org/file24204/full-casemapping.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Wed Jan 11 21:33:18 2012
From: report at bugs.python.org (poq)
Date: Wed, 11 Jan 2012 20:33:18 +0000
Subject: [issue13773] Support sqlite3 uri filenames
Message-ID: <1326313998.63.0.583537561643.issue13773@psf.upfronthosting.co.za>


New submission from poq <poq at gmx.com>:

URIs are an extensible way to pass options to SQLite. See:
http://www.sqlite.org/uri.html

Patch adds a keyword argument "uri" to sqlite3.connect which causes the filename to be parsed as a URI if set to True.

----------
components: Extension Modules
files: sqlite-uri.patch
keywords: patch
messages: 151089
nosy: poq
priority: normal
severity: normal
status: open
title: Support sqlite3 uri filenames
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24205/sqlite-uri.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13773>
_______________________________________

From report at bugs.python.org  Wed Jan 11 22:07:34 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Wed, 11 Jan 2012 21:07:34 +0000
Subject: [issue13773] Support sqlite3 uri filenames
In-Reply-To: <1326313998.63.0.583537561643.issue13773@psf.upfronthosting.co.za>
Message-ID: <1326316054.58.0.0883961191253.issue13773@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Hi, I will let others discuss the feature itself, here are already some comments about your patch:

- In module_connect(), the ability to pass "factory" as a positional argument is broken, please restore the previous code; I'm afraid there is no better implementation, even if it's a copy of connection.__init__.
- I suggest to allow "uri" in all builds, and fail only when it's True on a non supported platform.
- In general, try to wrap C code in 80 columns (except for the ".. function::" directive in .rst files)
- IIRC a correct URI is file:///path/to/file but sqlite probably allows file:/path/to/file as well.
- The test function should use "from test.support import TESTFN" to get a temporary file name (but I agree that sqlite tests don't seem to follow Python standards)

----------
nosy: +amaury.forgeotdarc
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13773>
_______________________________________

From report at bugs.python.org  Wed Jan 11 22:14:28 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Wed, 11 Jan 2012 21:14:28 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326316468.04.0.683689988597.issue13761@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

I'd ignore the failure. It could easily mask some other more interesting error.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Wed Jan 11 22:46:12 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Wed, 11 Jan 2012 21:46:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326318372.91.0.960872424612.issue13703@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I must be missing something, but how is raising an exception when a collision threshold is reached a good thing?
Basically, we're just exchanging a DoS for another (just feed the server process with ad-hoc data and he'll commit suicide). Sure, the caller can catch the exception to detect this, but what for? Restart the process, so that the attacker can just try again?
Also, there's the potential of perfectly legit applications breaking.
IMHO, randomization is the way to go, so that an attacker cannot generate a set of colliding values beforehand, which renders the attack impracticle. The same idea is behind ASLR used in modern kernels, and AFAICT, has been chosen by other implementations.
If a such patch has a negligible performance impact, then it should definitely be enabled by default. People who want deterministic hashing (maybe to bypass an application bug, or just because the want determinism) can disable it if they really want to.

----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 11 22:56:05 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Wed, 11 Jan 2012 21:56:05 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326318965.43.0.28657050808.issue13241@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

I've found a bug in the curses module that is causing problems:


diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2056,7 +2056,8 @@
     {"encoding",
      (getter)PyCursesWindow_get_encoding,
      (setter)PyCursesWindow_set_encoding,
-     "the typecode character used to create the array"}
+     "the typecode character used to create the array"},
+    {NULL, NULL, NULL, NULL}
 };
 
 /* -------------------------------------------------------*/

That is, there is no sentinel in the PyCursesWindow_getsets list. Shouldn't this cause failures on more platforms than just OSX?

With this patch I can build python 3.3 with clang (as included in Xcode 4.2.1). 

BTW. clang as included with Xcode 4.2.1 seems to build the attached copy of unicode.c without problems, the gcc command still fails though.

I did run into problems when building outside of the source directory, but that seems to be an unrelated issue.

----------
keywords: +needs review, patch
Added file: http://bugs.python.org/file24206/issue13241.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Wed Jan 11 23:01:03 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 11 Jan 2012 22:01:03 +0000
Subject: [issue13681] Aifc read compressed frames fix
In-Reply-To: <1325243830.12.0.0659769703479.issue13681@psf.upfronthosting.co.za>
Message-ID: <1326319263.1.0.547562367756.issue13681@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi Oleg,
thanks for your interest in aifc module!

I think that if the fix for audioop has a reason for itself to exists, then another issue (for that only) has to be filed and set the accordingly dependency on this one.

----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13681>
_______________________________________

From report at bugs.python.org  Wed Jan 11 23:12:38 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 11 Jan 2012 22:12:38 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326319958.33.0.359533944214.issue13763@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

But we may explain what's behind: "after the mercury chemical element symbol" - what do you think?

----------
nosy: +sandro.tosi
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 11 23:19:57 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Wed, 11 Jan 2012 22:19:57 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326320397.61.0.555869264614.issue13763@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

If 'program' refers to the executable, the sentence is still valid.

----------
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 11 23:44:09 2012
From: report at bugs.python.org (poq)
Date: Wed, 11 Jan 2012 22:44:09 +0000
Subject: [issue13773] Support sqlite3 uri filenames
In-Reply-To: <1326313998.63.0.583537561643.issue13773@psf.upfronthosting.co.za>
Message-ID: <1326321849.19.0.650085999435.issue13773@psf.upfronthosting.co.za>


poq <poq at gmx.com> added the comment:

Thanks for your comments. You're right, I didn't consider positional arguments. Here's a patch that addresses your comments.

Should I also rewrap modified lines that were already much too long?

I also noticed & fixed an unrelated typo in Lib/sqlite3/test/hooks.py...

----------
Added file: http://bugs.python.org/file24207/sqlite-uri.v2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13773>
_______________________________________

From report at bugs.python.org  Thu Jan 12 00:17:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 11 Jan 2012 23:17:47 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <E1Rl7QP-0005o8-MC@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f7e05d205a52 by Benjamin Peterson in branch 'default':
use full unicode mappings for upper/lower/title case (#12736)
http://hg.python.org/cpython/rev/f7e05d205a52

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Thu Jan 12 00:23:51 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Wed, 11 Jan 2012 23:23:51 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326324231.35.0.0487701613455.issue12736@psf.upfronthosting.co.za>


Changes by Benjamin Peterson <benjamin at python.org>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Thu Jan 12 00:56:28 2012
From: report at bugs.python.org (Cameron Simpson)
Date: Wed, 11 Jan 2012 23:56:28 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326326188.47.0.945168203136.issue13761@psf.upfronthosting.co.za>


Changes by Cameron Simpson <cs at zip.com.au>:


----------
nosy: +cameron

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 01:07:02 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 00:07:02 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1326326822.0.0.505634238951.issue13772@psf.upfronthosting.co.za>


Changes by Santoso Wijaya <santoso.wijaya at gmail.com>:


----------
nosy: +santa4nt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 01:08:25 2012
From: report at bugs.python.org (Nam Nguyen)
Date: Thu, 12 Jan 2012 00:08:25 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326326905.59.0.984127685914.issue13241@psf.upfronthosting.co.za>


Nam Nguyen <bitsink at gmail.com> added the comment:

Ronald's patch does fix the segmentation fault with clang-compiled Python. But this bug report seems to be about LLVM-GCC not compiling correctly.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Thu Jan 12 01:48:10 2012
From: report at bugs.python.org (Julian Berman)
Date: Thu, 12 Jan 2012 00:48:10 +0000
Subject: [issue13774] json.loads raises a SystemError for invalid encoding on
	2.7.2
Message-ID: <1326329290.58.0.508357217155.issue13774@psf.upfronthosting.co.za>


New submission from Julian Berman <Julian+Python.org at GrayVines.com>:

>>> import json
>>> json.loads("{}", [1, 2, 3])
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/json/__init__.py", line 339, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/json/decoder.py", line 359, in __init__
    self.scan_once = scanner.make_scanner(self)
SystemError: NULL result without error in PyObject_Call

Python 3.2 and 2.6 are not affected it'd seem, so it looks to be 2.7 only.

----------
components: Library (Lib)
messages: 151100
nosy: Julian
priority: normal
severity: normal
status: open
title: json.loads raises a SystemError for invalid encoding on 2.7.2
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:06:49 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 01:06:49 +0000
Subject: [issue13775] Access Denied message on symlink creation misleading for
	an existing file/directory target.
Message-ID: <1326330408.97.0.484457419227.issue13775@psf.upfronthosting.co.za>


New submission from Santoso Wijaya <santoso.wijaya at gmail.com>:

Consider:

>>> os.symlink('.\\test', 'Lib\\bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
WindowsError: [Error 5] Access is denied: '.\\test'

Where Lib\\bar is previously created. The symlink creation is rightly rejected, but with a misleading message. The failure is because 'Lib\\bar' already exists, not because of '.\\test'.

----------
components: Windows
messages: 151101
nosy: santa4nt
priority: normal
severity: normal
status: open
title: Access Denied message on symlink creation misleading for an existing file/directory target.
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13775>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:09:26 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 01:09:26 +0000
Subject: [issue13775] Access Denied message on symlink creation misleading for
	an existing file/directory target.
In-Reply-To: <1326330408.97.0.484457419227.issue13775@psf.upfronthosting.co.za>
Message-ID: <1326330566.52.0.198391176586.issue13775@psf.upfronthosting.co.za>


Santoso Wijaya <santoso.wijaya at gmail.com> added the comment:

Simple patch.

----------
keywords: +patch
Added file: http://bugs.python.org/file24208/issue13775_py33.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13775>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:11:55 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 01:11:55 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1326330715.94.0.871113429912.issue13772@psf.upfronthosting.co.za>


Santoso Wijaya <santoso.wijaya at gmail.com> added the comment:

I think this is because "Lib\\bar" is NOT being created as a directory symlink, but rather as a regular one.

As such, the documentation for symlink where it states the optional `target_is_directory=False` argument should be automatically detect whether the source is a file or directory does not hold true.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:12:36 2012
From: report at bugs.python.org (Cameron Simpson)
Date: Thu, 12 Jan 2012 01:12:36 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326330756.01.0.841407514223.issue13761@psf.upfronthosting.co.za>


Cameron Simpson <cs at zip.com.au> added the comment:

I'm against ignoring a flush failure. What happened to "errors should never pass silently"? IMO, if we get as far as calling flush and having an exception occur, a "more interesting error" hasn't yet occurred.

I really dislike things that fail silently. If the caller asks print to flush, and the flush fails, the caller's request has not been met. The caller needs to know or incorrect behaviour can ensue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:13:28 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 01:13:28 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1326330808.57.0.00696759333452.issue13772@psf.upfronthosting.co.za>


Santoso Wijaya <santoso.wijaya at gmail.com> added the comment:

Confirmed (on latest py33 build):

>>> os.listdir('Lib\\bar')[:4]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotADirectoryError: [Error 267] The directory name is invalid: 'Lib\\bar\\*.*'
[61658 refs]

... after manually deleting the file-symlink `bar` ...

>>> os.symlink('.\\test', 'Lib\\bar', target_is_directory=True)
[61658 refs]
>>> os.listdir('Lib\\bar')[:4]
['185test.db', 'audiotest.au', 'autotest.py', 'badcert.pem']
[61666 refs]

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:26:40 2012
From: report at bugs.python.org (Cameron Simpson)
Date: Thu, 12 Jan 2012 01:26:40 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326331600.18.0.459636158754.issue11633@psf.upfronthosting.co.za>


Changes by Cameron Simpson <cs at zip.com.au>:


----------
nosy: +cameron

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:43:05 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 01:43:05 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326330715.94.0.871113429912.issue13772@psf.upfronthosting.co.za>
Message-ID: <1326332493.3531.29.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I think this is because "Lib\\bar" is NOT being created as a directory
> symlink, but rather as a regular one.

Ha! I didn't even know about that option. Thanks for noticing.

> As such, the documentation for symlink where it states the optional
> `target_is_directory=False` argument should be automatically detect
> whether the source is a file or directory does not hold true.

I don't know if auto-detection is a good idea. Of a course, from a Unix
user's perspective, Windows' behaviour doesn't make a lot of sense.
Especially when functions other than listdir() work fine anyway.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 02:51:24 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 01:51:24 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326333084.28.0.102447209945.issue13761@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think it should be re-raised (or rather, let pass through as in the patch). If you are asking for a flush you want to know why it failed.

(it shouldn't ever fail on "normal" files, anyway)

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 03:02:18 2012
From: report at bugs.python.org (Cameron Simpson)
Date: Thu, 12 Jan 2012 02:02:18 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326333738.1.0.706798422587.issue13761@psf.upfronthosting.co.za>


Cameron Simpson <cs at zip.com.au> added the comment:

Flush can fail of disc full or any number of low level things that prevent the OS getting the data into the on-disc file.

Speaking for myself, I certainly want to know if that happens.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 03:57:30 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Thu, 12 Jan 2012 02:57:30 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326337050.57.0.206224076238.issue13761@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

Ok, I'm fine with passing through the exception from flush().

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 05:04:16 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Thu, 12 Jan 2012 04:04:16 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326341056.33.0.208973516405.issue13761@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 05:40:48 2012
From: report at bugs.python.org (Martin Panter)
Date: Thu, 12 Jan 2012 04:40:48 +0000
Subject: [issue4806] Function calls taking a generator as star argument can
	mask TypeErrors in the generator
In-Reply-To: <1230900364.95.0.0532833034138.issue4806@psf.upfronthosting.co.za>
Message-ID: <1326343248.72.0.419002881577.issue4806@psf.upfronthosting.co.za>


Changes by Martin Panter <vadmium+py at gmail.com>:


----------
nosy: +vadmium

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4806>
_______________________________________

From report at bugs.python.org  Thu Jan 12 05:41:00 2012
From: report at bugs.python.org (Martin Panter)
Date: Thu, 12 Jan 2012 04:41:00 +0000
Subject: [issue11944] Function call with * and generator hide exception raised
	by generator.
In-Reply-To: <1303952447.32.0.22528411021.issue11944@psf.upfronthosting.co.za>
Message-ID: <1326343260.69.0.0679040158041.issue11944@psf.upfronthosting.co.za>


Changes by Martin Panter <vadmium+py at gmail.com>:


----------
nosy: +vadmium

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11944>
_______________________________________

From report at bugs.python.org  Thu Jan 12 05:51:33 2012
From: report at bugs.python.org (Martin Panter)
Date: Thu, 12 Jan 2012 04:51:33 +0000
Subject: [issue4806] Function calls taking a generator as star argument can
	mask TypeErrors in the generator
In-Reply-To: <1230900364.95.0.0532833034138.issue4806@psf.upfronthosting.co.za>
Message-ID: <1326343893.7.0.997168215817.issue4806@psf.upfronthosting.co.za>


Martin Panter <vadmium+py at gmail.com> added the comment:

I haven?t tried to understand what the patches do, but Issue 5218 looks like a very similar problem with a patch including a test case.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4806>
_______________________________________

From report at bugs.python.org  Thu Jan 12 05:52:44 2012
From: report at bugs.python.org (Martin Panter)
Date: Thu, 12 Jan 2012 04:52:44 +0000
Subject: [issue5218] Check for tp_iter in ceval:ext_do_call before overriding
	exception message
In-Reply-To: <1234380763.79.0.974439208897.issue5218@psf.upfronthosting.co.za>
Message-ID: <1326343964.29.0.947768968836.issue5218@psf.upfronthosting.co.za>


Martin Panter <vadmium+py at gmail.com> added the comment:

See also Issue 4806

----------
nosy: +vadmium

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5218>
_______________________________________

From report at bugs.python.org  Thu Jan 12 06:27:29 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 05:27:29 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326346049.88.0.979703983612.issue13645@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Then a method that returns some object representing all of the needed info
> on the source file is needed, but that does NOT blindly return the stat
info

Then it's simpler for the object to be a dict, and backwards compatibility is straightforward (by ignoring absent as well as unknown keys). Patch attached.

----------
Added file: http://bugs.python.org/file24209/impsize2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:04:54 2012
From: report at bugs.python.org (David Townshend)
Date: Thu, 12 Jan 2012 06:04:54 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <1326348294.26.0.290797017439.issue12760@psf.upfronthosting.co.za>


David Townshend <aquavitae69 at gmail.com> added the comment:

I've done a bit or rewording in the io docs, but I honestly don't know if this is any better that what you already had!

----------
Added file: http://bugs.python.org/file24210/x_diff2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:42:16 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Thu, 12 Jan 2012 06:42:16 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1326350536.87.0.61980118752.issue13772@psf.upfronthosting.co.za>


Santoso Wijaya <santoso.wijaya at gmail.com> added the comment:

I agree that it's probably not a good idea. Windows users should be aware of symlink limitation on the platform to begin with, IMHO.

Besides, keeping the detection behaviour in light of this defect would be adding unnecessary complexity to the code. Currently, the `src` argument (=".\\test") is simply given to a Windows function to see if it is a directory, from the currently working directory.

To make it aware of its path relativity, in the context of the target, is much harder than just changing the documentation. ;-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:42:59 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 06:42:59 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326332493.3531.29.camel@localhost.localdomain>
Message-ID: <1326350489.3374.20.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > As such, the documentation for symlink where it states the optional
> > `target_is_directory=False` argument should be automatically detect
> > whether the source is a file or directory does not hold true.
> 
> I don't know if auto-detection is a good idea. Of a course, from a Unix
> user's perspective, Windows' behaviour doesn't make a lot of sense.
> Especially when functions other than listdir() work fine anyway.

Ah, sorry, I had misunderstood your comment. Indeed, os.symlink
*already* tries to autodetect the target's file type, but the detection
is broken with relative symlinks: it doesn't try to join them with the
link's directory and so it thinks the target doesn't exist.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:51:58 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 12 Jan 2012 06:51:58 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326351118.72.0.0954765885771.issue13734@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


Removed file: http://bugs.python.org/file24176/walkfd.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:52:09 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 12 Jan 2012 06:52:09 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326351129.84.0.205909316826.issue13734@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


Removed file: http://bugs.python.org/file24197/fdwalk.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:52:27 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 12 Jan 2012 06:52:27 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326351147.51.0.291420261445.issue13734@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


Removed file: http://bugs.python.org/file24202/fdwalk-1.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Thu Jan 12 07:52:47 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 12 Jan 2012 06:52:47 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1326351167.28.0.270188231171.issue13734@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


Added file: http://bugs.python.org/file24211/fdwalk-2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Thu Jan 12 08:03:53 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Thu, 12 Jan 2012 07:03:53 +0000
Subject: [issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS
	10.7)
In-Reply-To: <1319226560.18.0.474501275115.issue13241@psf.upfronthosting.co.za>
Message-ID: <1326351833.84.0.262544836538.issue13241@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

Both clang and llvm-gcc had problems, the llvm-gcc problem is still there and is caused by a compiler bug. There is nothing we can do about that beyond warning Apple.

It's also pretty clear that llvm-gcc (and gcc in general) is on the way out as the system compiler on OSX. It's therefore unlikely that the compiler bug we ran into will get fixed.

I'm leaning towards adding code to configure that either tests for this issue and refers to documentation in Mac/README (also to be added) when this bug is found, or just switches to clang instead of gcc as the default compiler when a recent version of Xcode is detected (and 'CC' is not set in the environment).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13241>
_______________________________________

From report at bugs.python.org  Thu Jan 12 08:11:24 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 12 Jan 2012 07:11:24 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
In-Reply-To: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>
Message-ID: <E1RlEok-0002HP-A8@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5594cd88bfed by Antoine Pitrou in branch '3.2':
Issue #13724: improve documentation for socket.create_connection.
http://hg.python.org/cpython/rev/5594cd88bfed

New changeset 43d08528223c by Antoine Pitrou in branch 'default':
Issue #13724: improve documentation for socket.create_connection.
http://hg.python.org/cpython/rev/43d08528223c

New changeset 60e77810a47e by Antoine Pitrou in branch '2.7':
Issue #13724: improve documentation for socket.create_connection.
http://hg.python.org/cpython/rev/60e77810a47e

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Thu Jan 12 08:11:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 07:11:53 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
In-Reply-To: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>
Message-ID: <1326352313.66.0.148088686141.issue13724@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Wording fixed, thank you.

----------
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Thu Jan 12 08:12:00 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 07:12:00 +0000
Subject: [issue13724] socket.create_connection and multiple IP addresses
In-Reply-To: <1325888030.3.0.0416342115288.issue13724@psf.upfronthosting.co.za>
Message-ID: <1326352320.52.0.977069850914.issue13724@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13724>
_______________________________________

From report at bugs.python.org  Thu Jan 12 09:24:56 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Thu, 12 Jan 2012 08:24:56 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326356696.77.0.0528592500709.issue13761@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

I am not discussing "printing to file", so my 0.02 is that "flush=True" should be made default for print() with sys.stdout, because it is probably what users expect when calling this function.

If somebody needs buffering/cache or more fine-grained control over output, they are free to use sys.stdout.write/flush directly.

And users won't be happy to wrap every print() into try/catch to guard from the flush exceptions.

----------
nosy: +techtonik

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 09:53:24 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Thu, 12 Jan 2012 08:53:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326358404.42.0.384907026194.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

I'd like to add a few notes:

1. both 32-bit and 64-bit python are vulnerable
2. collision-counting will break other things
3. imho, randomization is the way to go, enabled by default.
4. do we need a steady hash-function later again?

I created ~500KB of colliding strings for both 32-bit and 64-bit python.
It works impressively good:

32bit: ~500KB payload keep django busy for >30 minutes.
64bit: ~500KB payload keep django busy for 5 minutes.

Django is more vulnerable than python-dict alone, because it
* converts the strings to unicode first, making the comparision more expensive
* does 5 dict-lookups per key.

So Python's dict of str alone is probably ~10x faster. Of course it's much harder to create the payload for 64-bit python than for 32-bit, but it works for both.

The collision-counting idea makes some sense in the web environment, but for other software types it can causes serious problems.

I don't want my software to stop working because someone managed to enter 1000 bad strings into it. Think of a software that handles names of customers or filenames. We don't want it to break completely just because someone entered a few clever names.

Randomization fixes most of these problems.

However, it breaks the steadiness of hash(X) between two runs of the same software. There's probably code out there that assumes that hash(X) always returns the same value: database- or serialization-modules, for example.

There might be good reasons to also have a steady hash-function available. The broken code is hard to fix if no such a function is available at all. Maybe it's possible to add a second steady hash-functions later again?

For the moment I think the best way is to turn on randomization of hash() by default, but having a way to turn it off.

----------
nosy: +fx5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 12 10:27:36 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 12 Jan 2012 09:27:36 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326358404.42.0.384907026194.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F0EA780.8020007@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Frank Sievertsen wrote:
> 
> I don't want my software to stop working because someone managed to enter 1000 bad strings into it. Think of a software that handles names of customers or filenames. We don't want it to break completely just because someone entered a few clever names.

Collision counting is just a simple way to trigger an action. As I mentioned
in my proposal on this ticket, raising an exception is just one way to deal
with the problem in case excessive collisions are found. A better way is to
add a universal hash method, so that the dict can adapt to the data and
modify the hash functions for just that dict (without breaking other
dicts or changing the standard hash functions).

Note that raising an exception doesn't completely break your software.
It just signals a severe problem with the input data and a likely
attack on your software. As such, it's no different than turning on DOS
attack prevention in your router.

In case you do get an exception, a web server will simply return a 500 error
and continue working normally.

For other applications, you may see a failure notice in your logs. If
you're sure that there are no possible ways to attack the application using
such data, then you can simply disable the feature to prevent such
exceptions.

> Randomization fixes most of these problems.

See my list of issues with this approach (further up on this ticket).

> However, it breaks the steadiness of hash(X) between two runs of the same software. There's probably code out there that assumes that hash(X) always returns the same value: database- or serialization-modules, for example.
> 
> There might be good reasons to also have a steady hash-function available. The broken code is hard to fix if no such a function is available at all. Maybe it's possible to add a second steady hash-functions later again?

This is one of the issues I mentioned.

> For the moment I think the best way is to turn on randomization of hash() by default, but having a way to turn it off.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 12 11:02:06 2012
From: report at bugs.python.org (Graham Dumpleton)
Date: Thu, 12 Jan 2012 10:02:06 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326362526.75.0.345257374046.issue13703@psf.upfronthosting.co.za>


Graham Dumpleton <Graham.Dumpleton at gmail.com> added the comment:

Right back at the start it was said:

"""
We haven't agreed whether the randomization should be enabled by default or disabled by default. IMHO it should be disabled for all releases except for the upcoming 3.3 release. The env var PYTHONRANDOMHASH=1 would enable the randomization. It's simple to set the env var in e.g. Apache for mod_python and mod_wsgi.
"""

with a environment variable PYTHONHASHSEED still being mentioned towards the end.

Be aware that a user being able to set an environment variable which is used on Python interpreter initialisation when using mod_python or mod_wsgi is not as trivial as made out in leading comment.

To set an environment variable would require the setting of the environment variable to be done in the Apache etc init.d scripts, or if the Apache distro still follows Apache Software Foundation conventions, in the 'envvars' file.

Having to do this requires root access and is inconvenient, especially since where it needs to be done differs between every distro.

Where there are other environment variables that are useful to set for interpreter initialisation, mod_wsgi has been changed in the past to add specific directives for the Apache configuration file to set them prior to interpreter initialisation. This at least makes it somewhat easier, but still only of help where you are the admin of the server.

If that approach is necessary, then although mod_wsgi could eventually add such a directive, as mod_python is dead it will never happen for it.

As to another question posed about whether mod_wsgi itself is doing anything to combat this, the answer is no as don't believe there is anything it can do. Values like the query string or post data is simply passed through as is and always pulled apart by the application.

----------
nosy: +grahamd

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 12 11:16:48 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Thu, 12 Jan 2012 10:16:48 +0000
Subject: [issue13774] json.loads raises a SystemError for invalid encoding on
	2.7.2
In-Reply-To: <1326329290.58.0.508357217155.issue13774@psf.upfronthosting.co.za>
Message-ID: <1326363408.51.0.471147624037.issue13774@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Here is a patch

----------
nosy: +amaury.forgeotdarc
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 11:17:09 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Thu, 12 Jan 2012 10:17:09 +0000
Subject: [issue13774] json.loads raises a SystemError for invalid encoding on
	2.7.2
In-Reply-To: <1326329290.58.0.508357217155.issue13774@psf.upfronthosting.co.za>
Message-ID: <1326363429.16.0.942261698254.issue13774@psf.upfronthosting.co.za>


Changes by Amaury Forgeot d'Arc <amauryfa at gmail.com>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24212/json_badencoding.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 14:19:26 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Thu, 12 Jan 2012 13:19:26 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <1326374366.74.0.206883959365.issue6774@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

I was scared by the note in the documentation and wondered if the socket Python API was completely incapable of handling half-closed connections cross platform. pitrou helped me on IRC to track the note down to this issue.


IMO the bug report should have been rejected and the documentation patch should be removed. It shouldn't be that surprising that shutting something down that already has been shutdown (by the peer) will fail.

I don't see any indication that a "shutdown call closes the connection on the other half". It makes it half-closed as it should - and if it did anything else (which the note indicates) then it would be a big violation of BSD TCP API.

Ok, it might be slightly surprising that the next shutdown on the other end fails, but that is fully covered by "Note Some behavior may be platform dependent, since calls are made to the operating system socket APIs." It is not specific to Python in any way, AFAICT.

If anything it could just say something like "Note that shutdown of a socket that already has been shut down by the peer is platform dependent and might fail."

----------
nosy: +kiilerix, pitrou
title: socket.shudown documentation: on some platforms, closing one half closes the other half -> socket.shutdown documentation: on some platforms, closing one half closes the other half

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 14:24:13 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 13:24:13 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <1326374653.6.0.738140024583.issue6774@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 15:21:17 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Thu, 12 Jan 2012 14:21:17 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326378077.48.0.61977985061.issue11682@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


Added file: http://bugs.python.org/file24213/494c976c41c4.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 15:27:02 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Thu, 12 Jan 2012 14:27:02 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326378422.48.0.548465449791.issue11682@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


Removed file: http://bugs.python.org/file23774/0d1d76f68750.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 15:27:11 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Thu, 12 Jan 2012 14:27:11 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326378431.58.0.31792854726.issue11682@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


Removed file: http://bugs.python.org/file24213/494c976c41c4.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 15:28:04 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Thu, 12 Jan 2012 14:28:04 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326378484.61.0.0256645584253.issue11682@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


Added file: http://bugs.python.org/file24214/f8349cbc1b26.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 15:33:11 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Thu, 12 Jan 2012 14:33:11 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326378791.67.0.170426662389.issue11682@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Nice milestone this evening: by incorporating doc updates based on Zbysek's efforts and dropping the explicit bytecode generation tests, I now have something that appears ready to commit *without* a dependency on the proposed dis module updates.

So, assuming nobody spots any major problems between now and then, the PEP 380 implementation should be landing in the main repo some time this weekend :)

----------
dependencies:  -Refactor the dis module to provide better building blocks for bytecode analysis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:04:34 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Thu, 12 Jan 2012 15:04:34 +0000
Subject: [issue13359] urllib2 doesn't escape spaces in http requests
In-Reply-To: <1320610427.07.0.982938376676.issue13359@psf.upfronthosting.co.za>
Message-ID: <1326380674.53.0.227797946409.issue13359@psf.upfronthosting.co.za>


Michele Orr? <maker.py at gmail.com> added the comment:

Patch attached for python3, with unit tests.

----------
nosy: +maker
Added file: http://bugs.python.org/file24215/issue13359.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13359>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:10:58 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Thu, 12 Jan 2012 15:10:58 +0000
Subject: [issue13359] urllib2 doesn't escape spaces in http requests
In-Reply-To: <1320610427.07.0.982938376676.issue13359@psf.upfronthosting.co.za>
Message-ID: <1326381058.67.0.491463265511.issue13359@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

FWIW, I don't think it is a good idea to escape automatically. It will change the behaviour in a non-backward compatible way for existing applications that pass encoded urls to this function.

I think the existing behaviour is better. The documentation and the failure mode for passing URLs with spaces could however be improved.

----------
nosy: +kiilerix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13359>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:15:18 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 12 Jan 2012 15:15:18 +0000
Subject: [issue13776] formatter_unicode.c still assumes ASCII
Message-ID: <1326381318.07.0.368656319142.issue13776@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

http://docs.python.org/library/string.html#format-specification-mini-language defines

    fill        ::=  <a character other than '}'>

and the text also excludes '{'.  It does not require that the fill character be ASCII.

However, function parse_internal_render_format_spec 
http://hg.python.org/cpython/file/c2153ce1b5dd/Python/formatter_unicode.c#l277 raises a ValueError if fill_char > 127.

I'm honestly not certain which of the three is correct, but they should be consistent, and if anything but '{' is excluded, it would be best to explain why.

----------
components: Unicode
messages: 151128
nosy: Jim.Jewett, ezio.melotti
priority: normal
severity: normal
status: open
title: formatter_unicode.c still assumes ASCII
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13776>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:30:04 2012
From: report at bugs.python.org (=?utf-8?q?Michele_Orr=C3=B9?=)
Date: Thu, 12 Jan 2012 15:30:04 +0000
Subject: [issue13359] urllib2 doesn't escape spaces in http requests
In-Reply-To: <1320610427.07.0.982938376676.issue13359@psf.upfronthosting.co.za>
Message-ID: <1326382204.67.0.49776875462.issue13359@psf.upfronthosting.co.za>


Michele Orr? <maker.py at gmail.com> added the comment:

Here the patch for python2.


kiilerix, RFC 1738 explicitly says that the space character shall not be used.

----------
Added file: http://bugs.python.org/file24216/issue13359_py2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13359>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:30:15 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Thu, 12 Jan 2012 15:30:15 +0000
Subject: [issue13776] formatter_unicode.c still assumes ASCII
In-Reply-To: <1326381318.07.0.368656319142.issue13776@psf.upfronthosting.co.za>
Message-ID: <1326382215.3.0.150581028329.issue13776@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

This is a duplicate of issue 13706.

----------
nosy: +eric.smith
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13776>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:35:57 2012
From: report at bugs.python.org (Mads Kiilerich)
Date: Thu, 12 Jan 2012 15:35:57 +0000
Subject: [issue13359] urllib2 doesn't escape spaces in http requests
In-Reply-To: <1320610427.07.0.982938376676.issue13359@psf.upfronthosting.co.za>
Message-ID: <1326382557.86.0.37686987905.issue13359@psf.upfronthosting.co.za>


Mads Kiilerich <mads at kiilerich.com> added the comment:

Yes, the url sent by urllib2 must not contain spaces. In my opinion the only way to handle that correctly is to not pass urls with spaces to urlopen. Escaping the urls is not a good solution - even if the API was to be designed from scratch. It would be better to raise an exception if it is passed an invalid url.

Note for example that '/' and the %-encoding of '/' are different, and it must thus be possible to pass an url containing both to urlopen. That is not possible if it automically escapes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13359>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:55:07 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Thu, 12 Jan 2012 15:55:07 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326383707.31.0.255579630459.issue13761@psf.upfronthosting.co.za>


Guido van Rossum <guido at python.org> added the comment:

Anatoly, duly noted, and disagreed with.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 16:58:22 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 15:58:22 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326383902.31.0.13940511826.issue9253@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
hgrepos: +102

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:02:48 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:02:48 +0000
Subject: [issue9253] argparse: optional subparsers
In-Reply-To: <1279056589.03.0.566167994161.issue9253@psf.upfronthosting.co.za>
Message-ID: <1326384168.43.0.71362040782.issue9253@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24217/ed0fce615582.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9253>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:04:49 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:04:49 +0000
Subject: [issue13631] readline fails to parse some forms of .editrc under
	editline (libedit) emulation on Mac OS X
In-Reply-To: <1324267597.43.0.334578978638.issue13631@psf.upfronthosting.co.za>
Message-ID: <1326384289.08.0.466960070462.issue13631@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I applied your patch and built the readline module against libreadline and imported it successfully.  (I did not try the special parsing feature, but I don?t think it?s needed.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13631>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:08:28 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:08:28 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <1326384508.18.0.300550053652.issue13761@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Patch LGTM.  (s/assertEquals/assertEqual/ though, or you?ll get a warning)

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:26:00 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:26:00 +0000
Subject: [issue6095] os.curdir as the default argument for os.listdir
In-Reply-To: <1243122886.86.0.888741323334.issue6095@psf.upfronthosting.co.za>
Message-ID: <1326385560.78.0.929777907576.issue6095@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +michael.foord
resolution: accepted -> fixed
stage: patch review -> committed/rejected

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6095>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:29:11 2012
From: report at bugs.python.org (Vadim Fint)
Date: Thu, 12 Jan 2012 16:29:11 +0000
Subject: [issue11397] os.path.realpath() may produce incorrect results
In-Reply-To: <1299259038.05.0.675723191983.issue11397@psf.upfronthosting.co.za>
Message-ID: <1326385751.1.0.339142218657.issue11397@psf.upfronthosting.co.za>


Changes by Vadim Fint <mocksoul at gmail.com>:


----------
nosy: +Vadim.Fint

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11397>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:29:20 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:29:20 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326385760.41.0.108517356995.issue13765@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the report and patch.  How can we test it?  Do we need to install a Python on a path with embedded spaces?

----------
keywords: +needs review
stage:  -> patch review
versions: +Python 2.7, Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:30:24 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:30:24 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326385824.05.0.0923499301076.issue13765@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Oh and BTW it would be very helpful if you could adapt your test script to be pure-distutils, without Cython or Numpy extensions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:33:06 2012
From: report at bugs.python.org (Konstantin Nikitin)
Date: Thu, 12 Jan 2012 16:33:06 +0000
Subject: [issue11397] os.path.realpath() may produce incorrect results
In-Reply-To: <1299259038.05.0.675723191983.issue11397@psf.upfronthosting.co.za>
Message-ID: <1326385986.71.0.597454380215.issue11397@psf.upfronthosting.co.za>


Changes by Konstantin Nikitin <nk.0 at ya.ru>:


----------
nosy: +stromsund

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11397>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:35:26 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 16:35:26 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326386126.16.0.595709523629.issue12409@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Could you provide a single patch or publish your version somewhere to help reviews?

For the CPython docs, I think just removing the whole of Doc/documenting would be okay, or if we expect that people will go to that URI we may leave a placeholder page with only links.  Georg?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Thu Jan 12 17:41:41 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Thu, 12 Jan 2012 16:41:41 +0000
Subject: [issue13706] non-ascii fill characters no longer work in formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1326386501.64.0.362338726418.issue13706@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

Sorry for the off-the-cuff diagnosis. I had assumed this was the unintended result of the conversion, but of course I'm wrong.

I'd like to fix this.

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Thu Jan 12 18:03:38 2012
From: report at bugs.python.org (Brett Cannon)
Date: Thu, 12 Jan 2012 17:03:38 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326387818.1.0.0490538643018.issue13645@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

LGTM (although I didn't run the unit tests and focused mainly on the importlib._bootstrap and abc changes). Only two things I would change. One is possibly deprecating path_mtime() so people don't waste time implementing it (we actually never need to remove it thanks to the ABC; otherwise we need to make sure the docs strongly state to only bother with path_stats()). The other is to say the mtime key should contain a value that is a real number (ie. float and any other numeric type that can cast to an integer). That way you get your higher resolution in the dict while still being able to use the value in writing the bytecode.

And is there any efficient way to get the stat info on a file AND its contents in a single call? If so we might want a method to support that (eg. add a 'source_bytes' key or something), but I can't think of any low-level call that supports that kind of thing.

----------
assignee:  -> pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Thu Jan 12 18:10:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 12 Jan 2012 17:10:55 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326388255.6.0.573752741873.issue11633@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I agree with the python-ideas message that ``sys.stdout.flush()`` is surprising / possibly misleading and should be ``file.flush()``.  If the other bug report about adding a flush argument is rejected, please consider this.  Thanks :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Thu Jan 12 18:17:25 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 12 Jan 2012 17:17:25 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326388645.13.0.653867487918.issue12736@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

The currently applied patch ( http://hg.python.org/cpython/rev/f7e05d205a52 ) left some dead code in unicodeobject.c

function fixup ( http://hg.python.org/cpython/file/f7e05d205a52/Objects/unicodeobject.c#l9386 ) has a shortcut for when the fixer doesn't make any actual changes.  The removed fixers (like fixupper ) returned 0 rather than maxchar to indicate that.  The only remaining fixer, fix_decimal_and_space_to_ascii (line 8839), does not.  (I think fix_decimal_and_space_to_ascii *should* add a touched flag, but until it does, the shortcut dedup code is dead.)

Also, around line 10502, there is an #if 0 section with code that relied on one of the removed fixers; is it time to remove that section?

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Thu Jan 12 18:27:27 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 12 Jan 2012 17:27:27 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326389247.43.0.923309309968.issue12409@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Sure, sorry I didn't make it before; here attached the 2 patches, one against devguide, the other to default cpython repo.

----------
keywords: +patch
Added file: http://bugs.python.org/file24218/doc2devguide-devguide.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Thu Jan 12 18:27:36 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 12 Jan 2012 17:27:36 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326389256.65.0.348448873287.issue12409@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


Added file: http://bugs.python.org/file24219/doc2devguide-cpython.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:03:35 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Thu, 12 Jan 2012 18:03:35 +0000
Subject: [issue2292] Missing *-unpacking generalizations
In-Reply-To: <1205595680.3.0.859525264867.issue2292@psf.upfronthosting.co.za>
Message-ID: <1326391415.14.0.231202943168.issue2292@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

#11682 will likely be merged. The part of this patch about "yielding everything from an iterator" becomes obsolete:
>>> def flatten(iterables):
...    for it in iterables:
...      yield from it
... 
>>> L = [ [0,1,2], (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> list(flatten(L))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The rest is of course still valid and useful.

----------
nosy: +zbysz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2292>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:07:06 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Thu, 12 Jan 2012 18:07:06 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326391626.92.0.183840488127.issue11682@psf.upfronthosting.co.za>


Changes by Zbyszek Szmek <zbyszek at in.waw.pl>:


----------
hgrepos:  -66

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:10:42 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Thu, 12 Jan 2012 18:10:42 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326391842.76.0.847649199495.issue11682@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Some minor comments in http://bugs.python.org/review/11682/show.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:16:32 2012
From: report at bugs.python.org (Michael Goderbauer)
Date: Thu, 12 Jan 2012 18:16:32 +0000
Subject: [issue13777] socket: communicating with Mac OS X KEXT controls
Message-ID: <1326392192.39.0.824287439397.issue13777@psf.upfronthosting.co.za>


New submission from Michael Goderbauer <contact at m-goderbauer.de>:

Mac OS X provides a socket-based API to communicate with Kernel Extensions (KEXTs) called "KEXT Controls". For this, Mac OS X defines PF_SYSTEM as a new socket domain which supports the SYSPROTO_CONTROL protocol.

Right now the PF_SYSTEM domain and the SYSPROTO_CONTROL protocol are not supported by Python's socket module. I am attaching a patch that introduces support for both.

More information on KEXT Controls can be found here:
http://developer.apple.com/library/mac/documentation/Darwin/Conceptual/NKEConceptual/control/control.html

----------
components: Extension Modules
files: kext.patch
keywords: patch
messages: 151145
nosy: goderbauer
priority: normal
severity: normal
status: open
title: socket: communicating with Mac OS X KEXT controls
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24220/kext.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13777>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:18:54 2012
From: report at bugs.python.org (Jan Kratochvil)
Date: Thu, 12 Jan 2012 18:18:54 +0000
Subject: [issue10112] Use -Wl, --dynamic-list=x.list,
	not -Xlinker -export-dynamic
In-Reply-To: <1287137785.61.0.962664170792.issue10112@psf.upfronthosting.co.za>
Message-ID: <1326392334.27.0.239764566327.issue10112@psf.upfronthosting.co.za>


Jan Kratochvil <jan.kratochvil at redhat.com> added the comment:

Here is the implementation.

Python/getargs.c was modified for:
ImportError: /usr/lib64/python2.7/lib-dynload/fcntlmodule.so: undefined symbol: _PyArg_ParseTuple_SizeT
but I guess that patch part should be different.  There is no need to do #ifdef HAVE_DECLSPEC_DLL as PyAPI_FUNC is properly defined even without it.  So the declaration+export block should be unified there.

The Modules/ files were modified for:
*** WARNING: renaming "_ctypes_test" since importing it failed: dynamic module does not define init function (init_ctypes_test)
*** WARNING: renaming "_elementtree" since importing it failed: dynamic module does not define init function (init_elementtree)
*** WARNING: renaming "_hotshot" since importing it failed: dynamic module does not define init function (init_hotshot)
*** WARNING: renaming "_json" since importing it failed: dynamic module does not define init function (init_json)
*** WARNING: renaming "ossaudiodev" since importing it failed: dynamic module does not define init function (initossaudiodev)

without this patch:
-rw-r--r-- 1 jkratoch jkratoch 5775829 Jan 12 17:42 python-libs-2.7.2-18.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1738264 Jan 12 17:44 ./clean/python-libs-2.7.2-18.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1226 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1986161 Jan 12 17:48 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 5018800 Jan 12 19:13 clean/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*

with this patch:
-rw-r--r-- 1 jkratoch jkratoch 5762537 Jan 12 19:04 python-libs-2.7.2-18hidden3.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1720920 Jan 12 19:06 ./hidden3/python-libs-2.7.2-18hidden3.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1046 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1886781 Jan 12 19:11 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 4732080 Jan 12 19:13 hidden3/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*

----------
keywords: +patch
versions: +Python 2.7 -Python 3.2
Added file: http://bugs.python.org/file24221/hidden.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10112>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:45:41 2012
From: report at bugs.python.org (Meador Inge)
Date: Thu, 12 Jan 2012 18:45:41 +0000
Subject: [issue10112] Use -Wl, --dynamic-list=x.list,
	not -Xlinker -export-dynamic
In-Reply-To: <1287137785.61.0.962664170792.issue10112@psf.upfronthosting.co.za>
Message-ID: <1326393941.37.0.992065226481.issue10112@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10112>
_______________________________________

From report at bugs.python.org  Thu Jan 12 19:52:13 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 12 Jan 2012 18:52:13 +0000
Subject: [issue13775] Access Denied message on symlink creation misleading for
	an existing file/directory target.
In-Reply-To: <1326330408.97.0.484457419227.issue13775@psf.upfronthosting.co.za>
Message-ID: <1326394333.81.0.945982449003.issue13775@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I fail to see the bug. The output is correct, you are just misinterpreting it. It gives you the error code, error message, and original filename. It doesn't *actually* claim that the access to test is denied.

Now, changing the file in the message to the dest file doesn't really improve things. It fixes your case, but breaks cases involving problems with the source file.

So if any change is made, it should include both source and dest in the exception.

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13775>
_______________________________________

From report at bugs.python.org  Thu Jan 12 20:11:56 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Thu, 12 Jan 2012 19:11:56 +0000
Subject: [issue13777] socket: communicating with Mac OS X KEXT controls
In-Reply-To: <1326392192.39.0.824287439397.issue13777@psf.upfronthosting.co.za>
Message-ID: <1326395516.52.0.923782290543.issue13777@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +ned.deily, ronaldoussoren

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13777>
_______________________________________

From report at bugs.python.org  Thu Jan 12 20:40:43 2012
From: report at bugs.python.org (Meador Inge)
Date: Thu, 12 Jan 2012 19:40:43 +0000
Subject: [issue13709] Capitalization mistakes in the documentation for ctypes
In-Reply-To: <1325645203.91.0.562347938508.issue13709@psf.upfronthosting.co.za>
Message-ID: <1326397243.28.0.532545897273.issue13709@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
components: +ctypes
nosy: +meador.inge
stage:  -> needs patch
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13709>
_______________________________________

From report at bugs.python.org  Thu Jan 12 20:48:01 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 12 Jan 2012 19:48:01 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <1326397681.31.0.398313708103.issue6774@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> I was scared by the note in the documentation and wondered if the
> socket Python API was completely incapable of handling half-closed
> connections cross platform.
[...]
> It makes it half-closed as it should

Indeed. Calling shutdown(SHUT_WR) doesn't close the other end (which doesn't make much sense), it just sends a FIN (or RST depending on the context). It's the other end which decides to return ENOTCONN upon shutdown(SHUT_RD) on OS X, which is questionable (not sure it's against the BSD socket API, since shutdown(SHUT_RD) doesn't have any counterpart in the TCP layer).

I also find this note confusing and scary for no good reason, and since I don't think we should document every OS idiosyncrasies, it would probably be better to revert it.

I'll leave this open for a couple days to see if anyone objects, otherwise I'll revert it.

----------
resolution: accepted -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 20:52:07 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 12 Jan 2012 19:52:07 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <E1RlQgw-0007HB-2j@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4a767054551b by Terry Jan Reedy in branch '3.2':
#11633 At least 2 people prefer earlier revision.
http://hg.python.org/cpython/rev/4a767054551b

New changeset 22688f5f9d0f by Terry Jan Reedy in branch 'default':
Merge #11633 At least 2 people prefer earlier revision.
http://hg.python.org/cpython/rev/22688f5f9d0f

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Thu Jan 12 22:16:22 2012
From: report at bugs.python.org (Nicolas Dumazet)
Date: Thu, 12 Jan 2012 21:16:22 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <1326402982.11.0.0757175288221.issue6774@psf.upfronthosting.co.za>


Nicolas Dumazet <nicdumz at gmail.com> added the comment:

> It's the other end which decides to return ENOTCONN upon shutdown(SHUT_RD) on OS X, which is questionable
> (not sure it's against the BSD socket API, since shutdown(SHUT_RD) doesn't have any counterpart in the TCP layer).

Exactly. The same code raises a socket.error in one platform (mac os) and not on another (linux). Why not document this questionable behavior?

I'm sorry, I realize that my original patch was imprecise. I'm not an expert here, and I simply read http://svn.python.org/view/python/trunk/Lib/test/test_socket.py?r1=64125&r2=68611&pathrev=68611 . Ok, fine -- it doesn't close the other end per se, but shutdown(SH_RD) after a FIN on MacOS raises a socket.error . This is questionable, unexpected, and should be documented.


If possible, I'd like to push for a rewording instead of a revert.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Thu Jan 12 22:36:52 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Thu, 12 Jan 2012 21:36:52 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326319958.33.0.359533944214.issue13763@psf.upfronthosting.co.za>
Message-ID: <CAA77j2DdTR4wLv=5mOEc6zfx_mASk31z++r3+v_VOf91vC0BJg@mail.gmail.com>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

On Thu, Jan 12, 2012 at 00:12, Sandro Tosi <report at bugs.python.org> wrote:
>
> Sandro Tosi <sandro.tosi at gmail.com> added the comment:
>
> But we may explain what's behind: "after the mercury chemical element symbol" - what do you think?

Good idea. That's actually what I thought it meant initially before
noticing it was (probably) left there accidentally.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Thu Jan 12 22:46:41 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Thu, 12 Jan 2012 21:46:41 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326320397.61.0.555869264614.issue13763@psf.upfronthosting.co.za>
Message-ID: <CAA77j2BByM9DzZCPb=hyKpywYSYA6VEWzbX3XPq3fgH1tcD=jg@mail.gmail.com>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

> Ezio Melotti <ezio.melotti at gmail.com> added the comment:
>
> If 'program' refers to the executable, the sentence is still valid.

Interesting. It didn't appear like that until you mentioned it. It was
far more clear with subversion:svn than mercurial:hg. Sandro's
suggestion removes the ambiguity, although it would be kool to also
mention that it's also the program's name, since it seems to always be
used in that context.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Thu Jan 12 22:47:55 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 12 Jan 2012 21:47:55 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>
Message-ID: <E1RlSUz-00020D-Qu@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset bbed36370b08 by Antoine Pitrou in branch 'default':
Issue #13748: Raw bytes literals can now be written with the `rb` prefix as well as `br`.
http://hg.python.org/cpython/rev/bbed36370b08

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Thu Jan 12 22:48:48 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 12 Jan 2012 21:48:48 +0000
Subject: [issue13748] Allow rb"" literals as an equivalent to br""
In-Reply-To: <1326136450.88.0.500129181902.issue13748@psf.upfronthosting.co.za>
Message-ID: <1326404928.09.0.875158272698.issue13748@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Committed, thanks.

----------
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13748>
_______________________________________

From report at bugs.python.org  Thu Jan 12 23:18:22 2012
From: report at bugs.python.org (Ethan Furman)
Date: Thu, 12 Jan 2012 22:18:22 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1326406702.21.0.349004764605.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Okay, I like Matthew Barnett's idea of 

    except SomeError [as e]:
        raise as SomeOtherError('...')

This would require a change to the grammer as well, yes?  From:

    raise_stmt: 'raise' [test ['from' test]]

to

    raise_stmt: 'raise' [test ['from' test]] | 'raise' 'as' [test ['from' test]]

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Thu Jan 12 23:39:55 2012
From: report at bugs.python.org (Patrick Westerhoff)
Date: Thu, 12 Jan 2012 22:39:55 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1326407995.38.0.860930973923.issue6210@psf.upfronthosting.co.za>


Patrick Westerhoff <PatrickWesterhoff at gmail.com> added the comment:

Yeah, I would really like that solution, especially as it separates from the `from X` syntax that sets the exception?s cause.

Also I would prefer a syntax solution over a class method simply because the exception context is something that is implicitely set by Python with the `raise` statement. So it would make sense to overwrite that behaviour with a syntax construct as well.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 13 01:08:23 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 13 Jan 2012 00:08:23 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326413303.75.0.515940055297.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 6:

 - remove a debug code in dev_urandom() (did always raise an exception for testing)
 - dev_urandom() raises an exception if open() fails
 - os.urandom() uses again the right exception type and message (instead of a generic exception)
 - os.urandom() is not more linked to PYTHONHASHSEED
 - replace uint32_t by unsigned int in lcg_urandom() because Visual Studio 8 doesn't provide this type. "unsigned __int32" is available but I prefer to use a more common type. 32 or 64-bit types are supposed to generate the same sequence number (I didn't test).
 - fix more tests
 - regrtest.py restarts the process with PYTHONHASHSEED=randomseed if -r --randomseed=SEED is used
 - fix compilation on Windows (add random.c to the Visual Studio project file)

----------
Added file: http://bugs.python.org/file24222/random-6.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 13 01:36:07 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 13 Jan 2012 00:36:07 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326414967.11.0.0501104505783.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I wrote bench_startup.py to measure the startup time on Windows. The precision of the script is quite bad because Windows timer has a bad resolution (e.g. 15.6 ms on Windows 7) :-/

In release mode, the best startup time is 45.2 ms without random, 50.9 ms with random: an overhead of 5.6 ms (12%).

My script uses PYTHONHASHSEED=0 to disable the initialization of CryptoGen. You may modify the script to compare an unpatched Python with a patched Python for better numbers.

An overhead 12% is important. random-6.patch contains a faster (but also weaker) RNG on Windows, disable at compilation time. Search "#if 1" at the end of random.c. It uses my linear congruential generator (LCG) initialized with gettimeofday() and getpid() (which are known to be weak) instead of CryptoGen. Using the LCG, the startup overhead is between 1 and 2%.

----------
Added file: http://bugs.python.org/file24223/bench_startup.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 13 01:48:55 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 13 Jan 2012 00:48:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326415735.65.0.0333292674812.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

SafeDict.py: with this solution, the hash of key has to be recomputed at each access to the dict (creation, get, set), the hash is not cached in the string object.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 13 03:09:54 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 13 Jan 2012 02:09:54 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326420594.96.0.81764930328.issue13752@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Patch for review.

----------
keywords: +patch
title: add a unicode.casefold() method -> add a str.casefold() method
Added file: http://bugs.python.org/file24224/casefolding.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Fri Jan 13 04:53:43 2012
From: report at bugs.python.org (Meador Inge)
Date: Fri, 13 Jan 2012 03:53:43 +0000
Subject: [issue13629] _PyParser_TokenNames does not match up with the token.h
	numbers
In-Reply-To: <1324232482.43.0.0474353637736.issue13629@psf.upfronthosting.co.za>
Message-ID: <1326426823.87.0.109548890974.issue13629@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

Here is a patch that renumber the tokens.  I also regenerated token.py.

----------
nosy: +georg.brandl
Added file: http://bugs.python.org/file24225/renumber-tokens.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13629>
_______________________________________

From report at bugs.python.org  Fri Jan 13 05:00:27 2012
From: report at bugs.python.org (Meador Inge)
Date: Fri, 13 Jan 2012 04:00:27 +0000
Subject: [issue13746] ast.Tuple's have an inconsistent "col_offset" value
In-Reply-To: <1326126836.81.0.857723956921.issue13746@psf.upfronthosting.co.za>
Message-ID: <1326427227.65.0.853433015725.issue13746@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

I can reproduce this in tip as well:

>>> ast.dump(ast.parse('a = (1,2)'), include_attributes=True)
"Module(body=[Assign(targets=[Name(id='a', ctx=Store(), lineno=1, col_offset=0)], value=Tuple(elts=[Num(n=1, lineno=1, col_offset=5), Num(n=2, lineno=1, col_offset=7)], ctx=Load(), lineno=1, col_offset=5), lineno=1, col_offset=0)])"
>>> ast.dump(ast.parse('a = (1 + 2)'), include_attributes=True)
"Module(body=[Assign(targets=[Name(id='a', ctx=Store(), lineno=1, col_offset=0)], value=BinOp(left=Num(n=1, lineno=1, col_offset=5), op=Add(), right=Num(n=2, lineno=1, col_offset=9), lineno=1, col_offset=5), lineno=1, col_offset=0)])"

The actual column number is lost when calling into 'ast_for_testlist'
from the 'LPAR' case of 'ast_for_atom'.

----------
nosy: +benjamin.peterson, meador.inge
stage:  -> needs patch
versions: +Python 3.2, Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13746>
_______________________________________

From report at bugs.python.org  Fri Jan 13 05:34:27 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 13 Jan 2012 04:34:27 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326429267.57.0.881893192721.issue13752@psf.upfronthosting.co.za>


Changes by Benjamin Peterson <benjamin at python.org>:


Added file: http://bugs.python.org/file24226/casefolding.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Fri Jan 13 10:08:08 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 13 Jan 2012 09:08:08 +0000
Subject: [issue10344] codecs.readline doesn't care buffering=0
In-Reply-To: <1289084418.05.0.231852584468.issue10344@psf.upfronthosting.co.za>
Message-ID: <1326445688.11.0.3915279075.issue10344@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Something seems wrong somewhere. First,
codecs.open(filename, mode[, encoding[, errors[, buffering]]]) 
in the doc, should be, to match the code, in the current sytle
codecs.open(filename, mode='rb', encoding=None, errors='strict', buffering=1)
The other entries below follow this style.

The Note says "Files are always opened in binary mode, even if no binary mode was specified.". However, the code is
    if encoding is not None and \
       'b' not in mode:
        # Force opening of the file in binary mode
        mode = mode + 'b'
so the forcing only happens when an encoding is given. Since the intent is that codecs.open == open when no encoding is given, I believe the Note should be revised rather than the code.

(buffering=1) means line buffered. However, the doc for builtin open() says about buffering "1 to select line buffering (only usable in text mode)" So the default buffering is one that is not usable in the normal forced binary mode. Marc-Andre, can you explain this? (The doc for open() does not specify what happens when the buffering conflicts with the mode.)

The doc for StreamReader.readline() says ""size, if given, is passed as size argument to the stream?s readline() method.". If that were true, size would the max bytes to read. However, the docstring for the same in codecs.py says "size, if given, is passed as size argument to the read() method.", and that is what the code does. If not given, 72 is used as the default. (Why not 80?)

So, while the doc needs a minor tweak, I do not see what the OP's posted original result has to do with buffering. .readline does not have a fixed internal buffer of 72 chars that I can see. Rather, that is the default number of chars to read. So that is what it read, given that the file is longer than that.

I believe this is what Marc-Andre said, in different words, in his first post, in between the distraction of whether to remove open.

Santiago, yes, there is a difference between open.readline and codecs.readline. It will be more obvious when the codecs.readline size doc is corrected to specify that it is passed to read(), not readline(), and that it defaults to 72.

----------
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10344>
_______________________________________

From report at bugs.python.org  Fri Jan 13 10:16:55 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 13 Jan 2012 09:16:55 +0000
Subject: [issue10344] codecs.StreamReader.readline doc needs fix
In-Reply-To: <1289084418.05.0.231852584468.issue10344@psf.upfronthosting.co.za>
Message-ID: <1326446215.36.0.890871257557.issue10344@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

What I described is the behavior of codecs.StreamReader. However, the streamreader associated with a particular encoding(codec) might do differently. My understanding is that StreamReader is an example that a particular codec can use, derive from, or merely mimic the interface of.

----------
title: codecs.readline doesn't care buffering=0 -> codecs.StreamReader.readline doc needs fix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10344>
_______________________________________

From report at bugs.python.org  Fri Jan 13 10:24:58 2012
From: report at bugs.python.org (lesha)
Date: Fri, 13 Jan 2012 09:24:58 +0000
Subject: [issue13778] Python should invalidate all non-owned 'thread.lock'
	objects when forking
Message-ID: <1326446698.28.0.760510427025.issue13778@psf.upfronthosting.co.za>


New submission from lesha <pybug.20.lesha at xoxy.net>:

Here is a great description of the issue:

http://docs.oracle.com/cd/E19683-01/806-6867/gen-1/index.html


This enhancement proposes a way to make Python more resistant to this kind of deadlock.


Consider this program:


import threading
import subprocess
import time

l = threading.Lock() 

def f():
    l.acquire()  
    time.sleep(1)
    l.release()
 
t = threading.Thread(target=f)
t.start()

def g(l):
    l.acquire()
    l.release() 
    print 'ohai'


subprocess.Popen(['ls'], preexec_fn=lambda: g(l))



g() gets called in the forked process, which means that it's waiting on a *copy* of the lock, which can never get released.


This, in turn, means that the main thread will forever wait for the Popen to finish.



The program above incorrectly assumes that a threading lock can be shared across fork() parent and child.

I suspect adding such sharing is impractical, requiring OS support or excessive complexity. If the sharing could be had cheaply, it would be great -- programs like this would work as intended, but no other programs would break. 

Crazy idea: free the locks. Sadly, that is not safe! The ones that are currently locked by other threads might be protecting some network resource, and allowing the fork child to access them would result in a logical error.

However, it is always a bad idea for a fork() child to access a lock that is held by a thread that is not its fork() parent. That lock was locked at the time of the fork(), and will stay locked, because the child process will not get updated by the lock-holding threads.

So, it is always invalid to access that type of lock. Currently, you are guaranteed a deadlock.

Proposal: trying to acquire such a lock should crash the forked child with a nice, detailed error message (including the offending lock), rather than hang the entire program.

Sample steps to implement:

1) Store the process ID on each lock instance.
2) Acquire/release should crash if the lock does not belong to the current thread AND has a different process ID from the current one.

There are other potential implementations, such as explicitly enumerating such locks at the time of fork, and invalidating them.

This crash cannot be an exception in the child, because lock methods must not throw. However, it can and should be an exception in the fork() parent.

I think this enhancement would make it much easier to debug this kind of problem. It's an easy mistake to make, because preexec_fn or fork docs do not warn you of the danger, and locks can be acquired quite implicitly by innocent-looking code.

----------
components: Library (Lib)
messages: 151165
nosy: lesha
priority: normal
severity: normal
status: open
title: Python should invalidate all non-owned 'thread.lock' objects when forking
type: enhancement
versions: Python 2.6, Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13778>
_______________________________________

From report at bugs.python.org  Fri Jan 13 10:33:04 2012
From: report at bugs.python.org (lesha)
Date: Fri, 13 Jan 2012 09:33:04 +0000
Subject: [issue13778] Python should invalidate all non-owned 'thread.lock'
	objects when forking
In-Reply-To: <1326446698.28.0.760510427025.issue13778@psf.upfronthosting.co.za>
Message-ID: <1326447184.81.0.483004190361.issue13778@psf.upfronthosting.co.za>


lesha <pybug.20.lesha at xoxy.net> added the comment:

Actually, I think it does not matter which thread owns the lock, it is still invalid to try to acquire a lock that was grabbed by the fork() parent. Why? Because the fork() parent cannot free the child's copy of the lock anyway, and it's guaranteed to be dead also.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13778>
_______________________________________

From report at bugs.python.org  Fri Jan 13 11:17:28 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Fri, 13 Jan 2012 10:17:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326449848.75.0.36679779906.issue13703@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Added some small comments in http://bugs.python.org/review/13703/show.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 13 11:42:12 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 13 Jan 2012 10:42:12 +0000
Subject: [issue13778] Python should invalidate all non-owned 'thread.lock'
	objects when forking
In-Reply-To: <1326446698.28.0.760510427025.issue13778@psf.upfronthosting.co.za>
Message-ID: <1326451332.88.0.0408809846338.issue13778@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> Locks in python standard library should be sanitized on fork

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13778>
_______________________________________

From report at bugs.python.org  Fri Jan 13 12:02:11 2012
From: report at bugs.python.org (lesha)
Date: Fri, 13 Jan 2012 11:02:11 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1250550378.97.0.072881968798.issue6721@psf.upfronthosting.co.za>
Message-ID: <1326452531.03.0.423724201086.issue6721@psf.upfronthosting.co.za>


lesha <pybug.20.lesha at xoxy.net> added the comment:

Just wanted to say that I spent something like 8 hours debugging a subprocess + threading + logging deadlock on a real production system. 

I suspected one of my locks at first, but I couldn't find any. The post-fork code was very simple, and I didn't suspect that logging would be subject to the same issue.

The good news that I see a very clean solution for fixing this.

We can't free all locks across fork -- that is unsafe and mad, because the child might end up corrupting some shared (network) resource, for example/

However, extending RLock to provide ForkClearedRLock (this would be used by logging, i.e.) is quite straighforward.

The extended class would simply need to record the process ID, in which the lock was created, and the process ID, which is trying to acquire it.  Done!

----------
nosy: +lesha

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Fri Jan 13 12:03:09 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Fri, 13 Jan 2012 11:03:09 +0000
Subject: [issue13779] os.walk: bottom-up
Message-ID: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>


New submission from patrick vrijlandt <patrick.vrijlandt at gmail.com>:

PythonWin 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32.
Portions Copyright 1994-2008 Mark Hammond - see 'Help/About PythonWin' for further copyright information.
>>> import os
>>> os.makedirs("g:/a/b/c")
>>> os.listdir("g:/a")
['b']
>>> for root, dirs, files in os.walk("g:/a", topdown = False):
... 	print(root, dirs, files, os.listdir(root))
... 	os.rmdir(root)
... 	
g:/a\b\c [] [] []
g:/a\b ['c'] [] []
g:/a ['b'] [] []
>>> 

>From the documentation of os.walk:
If topdown is False, the triple for a directory is generated after the triples for all of its subdirectories (directories are generated bottom-up).

As the above example shows, the directories are generated in the correct order, "generated" referring to yield from generator os.walk. However, the generated (files? and) dirs do not necessarily reflect the current situation as produced by os.listdir.

Therefore, this does not clear the entire directory tree as I would expect.

>>> os.makedirs("g:/a/b/c")
>>> for root, dirs, files in os.walk("g:/a", topdown = False):
... 	print(root, dirs, files, os.listdir(root))	
... 	if not (files + dirs):
... 	    os.rmdir(root)
... 
g:/a\b\c [] [] []
g:/a\b ['c'] [] []
g:/a ['b'] [] ['b']

I think that at least the documentation should be more clear on this issue. I would like even better, if files + dirs would match os.listdir on the moment they are generated (=yielded).

----------
assignee: docs at python
components: Documentation
messages: 151169
nosy: docs at python, patrick.vrijlandt
priority: normal
severity: normal
status: open
title: os.walk: bottom-up
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Fri Jan 13 13:02:19 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 13 Jan 2012 12:02:19 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326456139.9.0.000706462181716.issue11682@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Committed for 3.3: http://hg.python.org/cpython/rev/d64ac9ab4cd0

Thanks to Greg for the initial effort on the PEP and reference implementation and to all involved in updating the original patch for 3.3 and getting the tests and documentation to an acceptable state.

Any issues discovered after this can be given a new tracker entry :)

----------
components: +Interpreter Core
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Fri Jan 13 14:48:28 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 13 Jan 2012 13:48:28 +0000
Subject: [issue13780] make YieldFrom its own node
Message-ID: <1326462508.45.0.280954945804.issue13780@psf.upfronthosting.co.za>


New submission from Benjamin Peterson <benjamin at python.org>:

As promised in a review a while ago of PEP 380.

----------
files: yieldfromnode.patch
keywords: patch
messages: 151172
nosy: benjamin.peterson, ncoghlan
priority: normal
severity: normal
status: open
title: make YieldFrom its own node
Added file: http://bugs.python.org/file24227/yieldfromnode.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13780>
_______________________________________

From report at bugs.python.org  Fri Jan 13 15:38:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 14:38:57 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326465537.33.0.411758780831.issue12409@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

We'll probably have to redirect all /documenting in Apache anyway.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Fri Jan 13 15:53:49 2012
From: report at bugs.python.org (Almar Klein)
Date: Fri, 13 Jan 2012 14:53:49 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326466429.66.0.916767920992.issue13765@psf.upfronthosting.co.za>


Almar Klein <almar.klein at gmail.com> added the comment:

Ok, I went to prepare a minimal example that does not use Cython nor Numpy. And then the problem was gone. Even more so, my fix would cause a problem, because somewhere quotes are placed around the entire command:

...link.exe /DLL "/LIBPATH:C:\Program Files (x86)\python32\libs" etc.

This somewhere is in spawn(), which calls nt_quote_args().

So I went on to search for the cause in Cython, and after that in Numpy. I have no traced it to Numpy, because it does NOT use the nt_quote_args() function of distutils.

So I think we can close the issue. I will open an issue at numpy.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Fri Jan 13 16:11:53 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 15:11:53 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326467513.68.0.228765182346.issue13765@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

OK, thanks.  Please add a link to the Numpy or Cython bug report here.

----------
assignee: tarek -> eric.araujo
resolution:  -> invalid
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Fri Jan 13 16:15:54 2012
From: report at bugs.python.org (Almar Klein)
Date: Fri, 13 Jan 2012 15:15:54 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326467754.12.0.908979529332.issue13765@psf.upfronthosting.co.za>


Almar Klein <almar.klein at gmail.com> added the comment:

This issue is posted at http://projects.scipy.org/numpy/ticket/2018

----------
resolution: invalid -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Fri Jan 13 16:19:37 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 15:19:37 +0000
Subject: [issue13765] Distutils does not put quotes around paths that contain
	spaces when compiling with MSVC
In-Reply-To: <1326279207.66.0.00870443648881.issue13765@psf.upfronthosting.co.za>
Message-ID: <1326467977.36.0.233353962646.issue13765@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13765>
_______________________________________

From report at bugs.python.org  Fri Jan 13 16:57:30 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 15:57:30 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1326470250.87.0.807748762379.issue11682@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Kudos!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Fri Jan 13 17:00:30 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 16:00:30 +0000
Subject: [issue13473] Add tests for files byte-compiled by distutils[2]
In-Reply-To: <1322145911.61.0.528651769931.issue13473@psf.upfronthosting.co.za>
Message-ID: <1326470430.11.0.0702245605537.issue13473@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Nick, would you have a bit of time to read my OP and reply?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13473>
_______________________________________

From report at bugs.python.org  Fri Jan 13 17:31:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 13 Jan 2012 16:31:28 +0000
Subject: [issue13764] Misc/build.sh is outdated... talks about svn
In-Reply-To: <1326278902.44.0.401175449456.issue13764@psf.upfronthosting.co.za>
Message-ID: <1326472288.31.0.137244527075.issue13764@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think the script was used by the "period regression test" crontask that used to send emails to python-checkins. The crontask is offline and the script probably hasn't been used by anyone else, so we could indeed remove it.

----------
components: +Demos and Tools -Build
nosy: +brett.cannon, nnorwitz, pitrou
priority: normal -> low
versions: +Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13764>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:09:16 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 13 Jan 2012 17:09:16 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1326387818.1.0.0490538643018.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326474245.3359.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> One is possibly deprecating path_mtime() so people don't waste time
> implementing it (we actually never need to remove it thanks to the
> ABC; otherwise we need to make sure the docs strongly state to only
> bother with path_stats()).

Ok, I saw I also forgot to update some importlib docs.

> The other is to say the mtime key should contain a value that is a
> real number (ie. float and any other numeric type that can cast to an
> integer).

Ok.

> And is there any efficient way to get the stat info on a file AND its
> contents in a single call?

I don't think so.  os.fstat() on an open fd looks minimally faster than
os.stat() on the filename (0.5?s faster here on Linux), but opening the
file has its own cost.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:13:20 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 17:13:20 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326474800.4.0.966807788542.issue12415@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
dependencies: +Moving "Documenting Python" to Devguide

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:13:37 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 17:13:37 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326474817.8.0.144200216803.issue12409@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

About the devguide patch:

a) The part about C roles and directives should probably mention version specifics (:cmacro: for 2.7, :c:macro: for 3.x) ? unless you make this bug dependent on updating Sphinx to 1.0 for 2.7 and then all versions use new-style :c:.

b) You probably want to use ref:documenting instead of :doc:documenting (we always use ref)

Not related to your patch, but it made me think about them so I?m just putting it out there for comments: Not sure we should keep the reST basics or just redirect to docutils and Sphinx docs.  Likewise, maybe it?s time to stop mentioning the old Latex-based docs.  (A nit: there are a few instances of `markup` that should be :file:`etc` or ``code``.)

About the cpython patch: You could just have said ?rm Doc/documenting? :)  Georg, will you be the one to set up the docs.python.org redirects?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:17:09 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 17:17:09 +0000
Subject: [issue13447] Add tests for some scripts in Tools/scripts
In-Reply-To: <1321895284.05.0.389852057258.issue13447@psf.upfronthosting.co.za>
Message-ID: <1326475029.75.0.000741475665093.issue13447@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
title: Add tests for Tools/scripts/reindent.py -> Add tests for some scripts in Tools/scripts

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13447>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:19:58 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 17:19:58 +0000
Subject: [issue1475523] gettext breaks on plural-forms header
Message-ID: <1326475198.31.0.20260419186.issue1475523@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
dependencies: +Add tests for some scripts in Tools/scripts

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1475523>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:20:13 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Fri, 13 Jan 2012 17:20:13 +0000
Subject: [issue1475523] gettext breaks on plural-forms header
Message-ID: <1326475213.01.0.920431140383.issue1475523@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
dependencies:  -Add tests for some scripts in Tools/scripts

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1475523>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:47:31 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Fri, 13 Jan 2012 17:47:31 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1326474817.8.0.144200216803.issue12409@psf.upfronthosting.co.za>
Message-ID: <CAPdtAj0AQSEyN62eyNwunTy_hEeNhwLJ6mxc=7tUxcsPhQGBTw@mail.gmail.com>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi ?ric,
thanks for the review.

On Fri, Jan 13, 2012 at 18:13, ?ric Araujo <report at bugs.python.org> wrote:
>
> ?ric Araujo <merwok at netwok.org> added the comment:
>
> About the devguide patch:
>
> a) The part about C roles and directives should probably mention version specifics (:cmacro: for 2.7, :c:macro: for 3.x) ? unless you make this bug dependent on updating Sphinx to 1.0 for 2.7 and then all versions use new-style :c:.

yeah, I'd love to not write both rules/directives formats to later
revert this addition, so ok, let's make this bug depending on the
migration to sphinx 1.0 of the 2.7 version. I've sent a follow-up of
that task to python-dev some days ago, and comments (and help!) would
be welcome there too.

> b) You probably want to use ref:documenting instead of :doc:documenting (we always use ref)

ah ok, I just used the same role that was there; updated to use :ref:
- fixed that (just pushed on my repo, not updated patch yet)

> Not related to your patch, but it made me think about them so I?m just putting it out there for comments: Not sure we should keep the reST basics or just redirect to docutils and Sphinx docs.

as a matter of personal taste, I like small lists of commonly used
commands/roles/directives/whatever in the doc, with additional links
as needed, this give a quick idea (that's usually enough) on what to
do but also the possibility for further digging (if one wants it).

>?Likewise, maybe it?s time to stop mentioning the old Latex-based docs.

we can do that, but maybe at a later step? so now just put the doc in
the canonical place, later let's refactor it, removing dead parts and
so on? i'm fine anyhow

> ?(A nit: there are a few instances of `markup` that should be :file:`etc` or ``code``.)

I'll fix them in the mean time.

> About the cpython patch: You could just have said ?rm Doc/documenting? :)

eheh, well but a tiny tiny part remains :) if we go for a redirect to
the devguide, I'm all for removing all that section from the doc; I
left it there in case someone still refers directly to documenting/ or
so (but indeed, all the sub-pages are no more reachable).

>?Georg, will you be the one to set up the docs.python.org redirects?

I can provide the configuration, but I don't have access to the
machine, so some admins must be there as well.

Cheers,
Sandro

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:53:53 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 13 Jan 2012 17:53:53 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <E1RllK4-0003TW-2I@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 87331661042b by Antoine Pitrou in branch 'default':
Issue #13645: pyc files now contain the size of the corresponding source
http://hg.python.org/cpython/rev/87331661042b

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Fri Jan 13 18:56:59 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 13 Jan 2012 17:56:59 +0000
Subject: [issue13645] import machinery vulnerable to timestamp collisions
In-Reply-To: <1324470738.36.0.917497036729.issue13645@psf.upfronthosting.co.za>
Message-ID: <1326477419.95.0.227294590286.issue13645@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Now pushed in. Thanks for the reviews!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13645>
_______________________________________

From report at bugs.python.org  Fri Jan 13 19:17:06 2012
From: report at bugs.python.org (Neal Norwitz)
Date: Fri, 13 Jan 2012 18:17:06 +0000
Subject: [issue13764] Misc/build.sh is outdated... talks about svn
In-Reply-To: <1326472288.31.0.137244527075.issue13764@psf.upfronthosting.co.za>
Message-ID: <CAFr+w9Gup6Affiey1kG=wfK0j9GPkVKWcQfLLC5x272YVGfSVA@mail.gmail.com>


Neal Norwitz <nnorwitz at gmail.com> added the comment:

If this script isn't used any more, it should be removed.

On Fri, Jan 13, 2012 at 8:31 AM, Antoine Pitrou <report at bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou at free.fr> added the comment:
>
> I think the script was used by the "period regression test" crontask that
> used to send emails to python-checkins. The crontask is offline and the
> script probably hasn't been used by anyone else, so we could indeed remove
> it.
>
> ----------
> components: +Demos and Tools -Build
> nosy: +brett.cannon, nnorwitz, pitrou
> priority: normal -> low
> versions: +Python 2.7, Python 3.2, Python 3.3
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13764>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13764>
_______________________________________

From report at bugs.python.org  Fri Jan 13 19:27:24 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Fri, 13 Jan 2012 18:27:24 +0000
Subject: [issue13728] Description of -m and -c cli options wrong?
In-Reply-To: <1325950613.83.0.912226487635.issue13728@psf.upfronthosting.co.za>
Message-ID: <1326479244.21.0.248950765122.issue13728@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Ah indeed, I could have looked at sys.path doc myself after all.. sorry for the noise.

----------
resolution:  -> invalid
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13728>
_______________________________________

From report at bugs.python.org  Fri Jan 13 19:42:18 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 13 Jan 2012 18:42:18 +0000
Subject: [issue13761] Add flush keyword to print()
In-Reply-To: <1326270282.71.0.886382293128.issue13761@psf.upfronthosting.co.za>
Message-ID: <E1Rlm4u-0008Eu-Ty@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3120a988a1a3 by Georg Brandl in branch 'default':
Closes #13761: add a "flush" keyword argument to print().
http://hg.python.org/cpython/rev/3120a988a1a3

----------
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13761>
_______________________________________

From report at bugs.python.org  Fri Jan 13 19:45:34 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 13 Jan 2012 18:45:34 +0000
Subject: [issue13764] Misc/build.sh is outdated... talks about svn
In-Reply-To: <1326278902.44.0.401175449456.issue13764@psf.upfronthosting.co.za>
Message-ID: <E1Rlm84-0000AP-QR@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f36c6f5f9d61 by Antoine Pitrou in branch '3.2':
Issue #13764: remove outdated script Misc/build.sh
http://hg.python.org/cpython/rev/f36c6f5f9d61

New changeset 609482c6710e by Antoine Pitrou in branch 'default':
Issue #13764: remove outdated script Misc/build.sh
http://hg.python.org/cpython/rev/609482c6710e

New changeset 7fcfbaad75ee by Antoine Pitrou in branch '2.7':
Issue #13764: remove outdated script Misc/build.sh
http://hg.python.org/cpython/rev/7fcfbaad75ee

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13764>
_______________________________________

From report at bugs.python.org  Fri Jan 13 19:47:30 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 13 Jan 2012 18:47:30 +0000
Subject: [issue13764] Misc/build.sh is outdated... talks about svn
In-Reply-To: <1326278902.44.0.401175449456.issue13764@psf.upfronthosting.co.za>
Message-ID: <1326480450.81.0.838962986091.issue13764@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Removed then, thank you.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13764>
_______________________________________

From report at bugs.python.org  Fri Jan 13 20:47:04 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 19:47:04 +0000
Subject: [issue13746] ast.Tuple's have an inconsistent "col_offset" value
In-Reply-To: <1326126836.81.0.857723956921.issue13746@psf.upfronthosting.co.za>
Message-ID: <1326484024.27.0.588295255137.issue13746@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

This is because the parentheses don't really belong to the tuple literal.

You could just as well write

b = 1, 3.14, 'abc', u'XYZ'

In other cases, the parentheses may be needed for grouping purposes (e.g. in function calls), but they still are only for grouping, just as in (a + b) * c.

For the empty tuple, where the parentheses actually are part of the literal, the col_offset is correct.

----------
nosy: +georg.brandl
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13746>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:26:51 2012
From: report at bugs.python.org (Brian Visel)
Date: Fri, 13 Jan 2012 20:26:51 +0000
Subject: [issue2124] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <1203086575.38.0.209227458245.issue2124@psf.upfronthosting.co.za>
Message-ID: <1326486411.37.0.685974316732.issue2124@psf.upfronthosting.co.za>


Brian Visel <aeon.descriptor at gmail.com> added the comment:

..still an issue.

----------
nosy: +Brian.Visel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2124>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:39:33 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 20:39:33 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1326487173.31.0.155471858538.issue13737@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

While the actual code may be accessible to everyone, the server configuration (paths etc. -- just look at the page; at least the session secret key and passwords are masked by Django) are not, and exposing that can be a security problem as well.  And while I agree that this possibility appears remote, just look at the current discussion about hashing attacks.  Running in debug mode also simply looks bad to just about every web programmer, which explains this bug report.

Lastly, setting DEBUG to true also has other consequences, like the possibility to "leak" memory for long-running processes, as explained here: https://docs.djangoproject.com/en/1.3/faq/models/

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:45:47 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 20:45:47 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
In-Reply-To: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>
Message-ID: <1326487547.39.0.724130323968.issue13753@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

You put '*' instead of '#' in the commit message.

Also, I don't think you should close more than one issue in one commit.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:52:54 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 20:52:54 +0000
Subject: [issue13768] Doc/tools/dailybuild.py available only on 2.7 branch
Message-ID: <1326487974.89.0.656724547294.issue13768@psf.upfronthosting.co.za>


New submission from Georg Brandl <georg at python.org>:

Why is that a concern?  It is not needed for the doc build and intended to be used on python.org only.

----------
nosy: +georg.brandl
resolution:  -> wont fix
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13768>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:53:28 2012
From: report at bugs.python.org (Georg Brandl)
Date: Fri, 13 Jan 2012 20:53:28 +0000
Subject: [issue1692335] Fix exception pickling: Move initial args assignment
	to BaseException.__new__
Message-ID: <1326488008.48.0.700204503598.issue1692335@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
assignee: georg.brandl -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1692335>
_______________________________________

From report at bugs.python.org  Fri Jan 13 21:55:54 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 13 Jan 2012 20:55:54 +0000
Subject: [issue2124] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <1203086575.38.0.209227458245.issue2124@psf.upfronthosting.co.za>
Message-ID: <1326488154.89.0.564654888514.issue2124@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

And my position still remains the same: this is not a bug. Applications affected by this need to use the APIs that are in place precisely to deal with this issue.

So I propose to close this report as invalid.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2124>
_______________________________________

From report at bugs.python.org  Fri Jan 13 22:12:05 2012
From: report at bugs.python.org (Brian Visel)
Date: Fri, 13 Jan 2012 21:12:05 +0000
Subject: [issue2124] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <1203086575.38.0.209227458245.issue2124@psf.upfronthosting.co.za>
Message-ID: <1326489125.74.0.319208512829.issue2124@psf.upfronthosting.co.za>


Brian Visel <aeon.descriptor at gmail.com> added the comment:

Of course, you can do as you like.

http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2124>
_______________________________________

From report at bugs.python.org  Fri Jan 13 22:29:50 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 13 Jan 2012 21:29:50 +0000
Subject: [issue2124] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <1203086575.38.0.209227458245.issue2124@psf.upfronthosting.co.za>
Message-ID: <1326490190.75.0.232657257328.issue2124@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Well, the issue is clearly underspecified, and different people read different things into it. I take your citation of the W3C blog entry that you are asking that caching should be employed. I read the issue entirely different, namely that by default no attempt to download the DTD should be made, or that the DOM loaders should provide better customization in that matter, or that catalogs shall be used.

Given that the issue was underspecified to begin with, I'm now closing it. Anybody who still has an issue here, please open a new issue and report your specific problem, preferably also proposing a solution.

If you need to follow up to this message, please do so in private email (martin at v.loewis.de).

----------
resolution:  -> rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2124>
_______________________________________

From report at bugs.python.org  Fri Jan 13 22:54:23 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 13 Jan 2012 21:54:23 +0000
Subject: [issue13774] json.loads raises a SystemError for invalid encoding on
	2.7.2
In-Reply-To: <1326329290.58.0.508357217155.issue13774@psf.upfronthosting.co.za>
Message-ID: <E1Rlp4m-0001L5-IC@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a9680746ae4a by Amaury Forgeot d'Arc in branch '2.7':
Issue #13774: json: Fix a SystemError when a bogus encoding is passed to
http://hg.python.org/cpython/rev/a9680746ae4a

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13774>
_______________________________________

From report at bugs.python.org  Fri Jan 13 22:56:01 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Fri, 13 Jan 2012 21:56:01 +0000
Subject: [issue13774] json.loads raises a SystemError for invalid encoding on
	2.7.2
In-Reply-To: <1326329290.58.0.508357217155.issue13774@psf.upfronthosting.co.za>
Message-ID: <1326491761.87.0.0259186498188.issue13774@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

This will be fixed in the next 2.7.3, thanks for the report!

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13774>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:14:38 2012
From: report at bugs.python.org (Floris Bruynooghe)
Date: Fri, 13 Jan 2012 22:14:38 +0000
Subject: [issue8881] socket.getaddrinfo() should return named tuples
In-Reply-To: <1275514288.42.0.225628200657.issue8881@psf.upfronthosting.co.za>
Message-ID: <1326492878.86.0.541024466115.issue8881@psf.upfronthosting.co.za>


Floris Bruynooghe <floris.bruynooghe at gmail.com> added the comment:

I think the part which could possibly a problem is addressed in http://hg.python.org/cpython/rev/384f73a104e9/.  Bearing in mind that direct usage for string interpolation is a pretty strange use for the result of getaddrinfo.

----------
nosy: +flub

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8881>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:15:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 13 Jan 2012 22:15:50 +0000
Subject: [issue8881] socket.getaddrinfo() should return named tuples
In-Reply-To: <1275514288.42.0.225628200657.issue8881@psf.upfronthosting.co.za>
Message-ID: <1326492950.45.0.868419986369.issue8881@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

A reasonable request indeed.

----------
stage:  -> needs patch
type:  -> enhancement
versions: +Python 3.3 -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8881>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:19:19 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 13 Jan 2012 22:19:19 +0000
Subject: [issue13753] str.join description contains an incorrect reference to
	argument
In-Reply-To: <1326167561.64.0.37328706988.issue13753@psf.upfronthosting.co.za>
Message-ID: <1326493159.1.0.146796418507.issue13753@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I see the two as really one issue -- minor corrections to the string section of stdtype.rst -- that py.user happened to have filed as two.
But I could have left out the closes and done at least one explicitly.
Or I could have closed this and said I was consolidating with #13754.
Thanks for pointing out the typo.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13753>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:31:56 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Fri, 13 Jan 2012 22:31:56 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
Message-ID: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>


New submission from Gregory P. Smith <greg at krypto.org>:

gzip.GzipFile accepts a fileobj parameter with an open file object.

Unfortunately gzip requires a filename be embedded in the gzip file and the gzip module code uses fileobj.name to get that.

This results in the fake "<fdopen>" name from posixmodule.c being embedded in the output gzipped file when using Python 2.x.  This causes problems when ungzipping these files with gzip -d or ungzip implementations that always rely on the embedded filename when writing their output file rather than stripping a suffix from the input filename as they cannot open a file called "<fdopen>" or if they do, each successive ungzip overwrites the previous...


On Python 3.x the problem is different, the gzip module fails entirely when given an os.fdopen()'ed file object:


$ ./python gzip_fdopen_prob.py 
out_file <_io.BufferedWriter name='FOO.gz'>
out_fd 3
fd_out_file <_io.BufferedWriter name=3>
fd_out_file.name 3
Traceback (most recent call last):
  File "gzip_fdopen_prob.py", line 13, in <module>
    gz_out_file = gzip.GzipFile(fileobj=fd_out_file)
  File "/home/gps/oss/cpython/default/Lib/gzip.py", line 184, in __init__
    self._write_gzip_header()
  File "/home/gps/oss/cpython/default/Lib/gzip.py", line 221, in _write_gzip_header
    fname = os.path.basename(self.name)
  File "/home/gps/oss/cpython/default/Lib/posixpath.py", line 132, in basename
    i = p.rfind(sep) + 1
AttributeError: 'int' object has no attribute 'rfind'

(code attached)

The os.fdopen()'ed file object is kindly using the integer file descriptor as its .name attribute.  That might or might not be an issue, but regardless of that:

1) GzipFile should not fail in this case.
2) GzipFile should never embed a fake made up filename in its output.

Fixing the gzip module to catch errors and use an empty b'' filename for the gzip code in the above error is easy.

What should be done about the .name attribute on fake file objects?  I don't think it should exist at all.

(another quick test shows that gzip in python 3.x can't output to a BytesIO fileobj at all, it thinks it is readonly)

----------
files: gzip_fdopen_prob.py
messages: 151203
nosy: gregory.p.smith
priority: normal
severity: normal
status: open
title: gzip module does the wrong thing with an os.fdopen()'ed fileobj
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24228/gzip_fdopen_prob.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:40:44 2012
From: report at bugs.python.org (Paul Boddie)
Date: Fri, 13 Jan 2012 22:40:44 +0000
Subject: [issue2124] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <1326490190.75.0.232657257328.issue2124@psf.upfronthosting.co.za>
Message-ID: <201201132345.45247.paul@boddie.org.uk>


Paul Boddie <paul at boddie.org.uk> added the comment:

Note that Python 3 provided a good opportunity for doing the minimal amount of 
work here - just stop things from accessing remote DTDs - but I imagine that 
even elementary standard library improvements of this kind weren't made (let 
alone the more extensive standard library changes I advocated), so there's 
going to be a backwards compatibility situation regardless of which Python 
series is involved now, unfortunately.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2124>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:53:00 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 13 Jan 2012 22:53:00 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495180.19.0.71945830473.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

The MS support engineer has come back confirming the bug and suggesting using the MS C API instead of wstat for doing directory detection. It's still possible they'll fix the bug in an upcoming update, but even if they do, it will leave a window of broken systems, so I'm inclined to implement a workaround (i.e. not use stat for directory detection in import.c).

I started looking at implementing this for the Python 2.7 branch and I found that I could drastically simplify the code by extracting an 'isdir' function.

Consider the two changesets in this patch. I've confirmed this fixes the issue. Would somebody review this patch? If it's deemed acceptable, I'll consider something similar for Python 3.x.

----------
hgrepos: +103

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:53:38 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 13 Jan 2012 22:53:38 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495218.86.0.325498174876.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24229/f3c7f4243a04.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:55:21 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Fri, 13 Jan 2012 22:55:21 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495321.29.0.613550247457.issue6727@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Please regenerate your diff file, it undoes an unrelated fix :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:55:55 2012
From: report at bugs.python.org (Alex Regueiro)
Date: Fri, 13 Jan 2012 22:55:55 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495355.09.0.111600360395.issue6727@psf.upfronthosting.co.za>


Alex Regueiro <alexreg at gmail.com> added the comment:

That's very good news. I suspected MS has written a wrapper over this somewhere to accommodate for this bug, and it was just a matter of using the right API.

I look forward to seeing this in the next 2.7 release.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Fri Jan 13 23:57:54 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 13 Jan 2012 22:57:54 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495474.03.0.172646144993.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24230/55d164f36389.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sat Jan 14 00:01:21 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 13 Jan 2012 23:01:21 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495681.7.0.493230650918.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24231/577b717055bc.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sat Jan 14 00:02:59 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Fri, 13 Jan 2012 23:02:59 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326495779.13.0.677265281489.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I've regenerated the diff a couple of times (once after rebasing to not undo Amaury's changes and another after correcting the whitespace).

----------
keywords: +needs review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sat Jan 14 01:07:06 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 14 Jan 2012 00:07:06 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326499626.33.0.127795338963.issue13781@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +nadeem.vawda
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Sat Jan 14 01:46:49 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sat, 14 Jan 2012 00:46:49 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326502009.98.0.299900111439.issue13752@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Fixed patch.

----------
Added file: http://bugs.python.org/file24232/casefolding.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Sat Jan 14 01:54:13 2012
From: report at bugs.python.org (John Machin)
Date: Sat, 14 Jan 2012 00:54:13 +0000
Subject: [issue13782] xml.etree.ElementTree: Element.append doesn't type-check
	its argument
Message-ID: <1326502453.32.0.206779556991.issue13782@psf.upfronthosting.co.za>


New submission from John Machin <sjmachin at lexicon.net>:

import xml.etree.ElementTree as et
node = et.Element('x')
node.append(not_an_Element_instance)

2.7 and 3.2 produce no complaint at all.
2.6 and 3.1 produce an AssertionError.

However cElementTree in all 4 versions produces a TypeError.

Please fix 2.7 and 3.2 ElementTree to produce a TypeError.

----------
messages: 151210
nosy: sjmachin
priority: normal
severity: normal
status: open
title: xml.etree.ElementTree: Element.append doesn't type-check its argument
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13782>
_______________________________________

From report at bugs.python.org  Sat Jan 14 01:54:52 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 00:54:52 +0000
Subject: [issue1397474] timeit execution enviroment
Message-ID: <1326502492.28.0.503234139282.issue1397474@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
stage: needs patch -> patch review
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1397474>
_______________________________________

From report at bugs.python.org  Sat Jan 14 02:29:48 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 01:29:48 +0000
Subject: [issue8473] doctest fails if you have inconsistent lineendings
In-Reply-To: <1271781617.9.0.374951389596.issue8473@psf.upfronthosting.co.za>
Message-ID: <1326504588.4.0.581015911772.issue8473@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

By 'update', do you mean to change

+Due to the way releases are made on different platforms, we sometimes test
+files on a *nix system with Windows file endings. Unfortunately, that leaves
+some of the test files broken:

to something like

+Issue8473: Make sure doctest works with mixed line endings.
+When this test is run on *nix, it has the side effect of making sure
+that doctest can handle Windows line endings on *nix.

Or just leave the last two lines off?

And also change '/r/n/r/n' x 2 to '/r/n/n' and '/n/r/n'.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8473>
_______________________________________

From report at bugs.python.org  Sat Jan 14 02:45:21 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 14 Jan 2012 01:45:21 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326505521.99.0.694305947571.issue13752@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +pitrou
stage: needs patch -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Sat Jan 14 03:47:05 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 02:47:05 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <1326509225.91.0.763437727452.issue13722@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:18:41 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:18:41 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326511121.84.0.264877709802.issue11633@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thank you sir.  Should the doc edit be backported to the 2.7 docs, with a mention that it?s only on unix?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:33:16 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Sat, 14 Jan 2012 03:33:16 +0000
Subject: [issue13782] xml.etree.ElementTree: Element.append doesn't type-check
	its argument
In-Reply-To: <1326502453.32.0.206779556991.issue13782@psf.upfronthosting.co.za>
Message-ID: <1326511996.29.0.629356613693.issue13782@psf.upfronthosting.co.za>


Changes by Santoso Wijaya <santoso.wijaya at gmail.com>:


----------
components: +Library (Lib), XML
nosy: +santa4nt
versions: +Python 3.3 -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13782>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:35:00 2012
From: report at bugs.python.org (Cameron Simpson)
Date: Sat, 14 Jan 2012 03:35:00 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326512100.05.0.755740049401.issue11633@psf.upfronthosting.co.za>


Cameron Simpson <cs at zip.com.au> added the comment:

Putting the wording into 2.7 might be nice, but I thought it was in bugfix only mode.

Regarding UNIX only, I'd avoid it; any file may be buffered in almost any way on any platform. Saying an explicit flush call may be necessary for immediate output is _not_ UNIX only and would be very misleading. Remembering that ~UNIX != Windows.

Telling users to explicitly call flush to ensure immediate output where that is necessary ensures portable coding (or ought to, user pigheadedness discounted:-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:36:56 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:36:56 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326512216.4.0.465598540602.issue12409@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I just replied to your python-dev email.  I think you can update 2.7 to use Sphinx 1.0 as soon as you?re down to a handful of warnings.

When the migration is done, you can add the documenting doc to the devguide and send a message to pydev to tell people to not commit changes to Doc/documenting in CPython clones anymore.

In the same email you can ask who to ask for setting up redirects (BTW, will redirecting paths to fragments work?  e.g. docs.py.o/documenting/style to devguide/documenting#style-and-blah).  When the redirects are up you?ll be able to rm Doc/documenting.

Then we?ll open other reports to talk about the unrelated I mentioned and to see how to stop downloading Sphinx from svn once per clone :)

----------
versions: +Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:41:41 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:41:41 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326512501.72.0.0593888752921.issue13725@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the report and patch; I can?t easily push these days but if nobody gets to it today I will commit this.

BTW, are you finding these regrtest bugs while using it to run the CPython test suite or to run your tests?

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:45:04 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:45:04 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1326512704.22.0.85184210566.issue13737@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

(FWIW this tracker is used for what is distributed as CPython, please use the metatracker (link on the left ?Report Tracker Problem?) for future reports.  Thanks)

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:46:02 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:46:02 +0000
Subject: [issue13756] Python3.2.2 make fail on cygwin
In-Reply-To: <1326199459.04.0.575648132034.issue13756@psf.upfronthosting.co.za>
Message-ID: <1326512762.39.0.546616530976.issue13756@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I am not sure building CPython with Cygwin is supported.

----------
nosy: +eric.araujo, loewis, mhammond

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13756>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:49:38 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:49:38 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1326512978.65.0.385819721103.issue13760@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
assignee:  -> lukasz.langa
nosy: +lukasz.langa
stage:  -> test needed
versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:52:52 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:52:52 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326513172.57.0.322175691967.issue13763@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I understand the phrasing this way: The system is called Mercurial and commonly abbreviated ?hg?, after the name of its main executable.  So we say ?the hg server?, ?I like the branches concept in hg?, etc.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:54:20 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:54:20 +0000
Subject: [issue13766] explain the relationship between Lib/lib2to3/Grammar.txt
	and Grammar/Grammar
In-Reply-To: <1326279554.15.0.713951578682.issue13766@psf.upfronthosting.co.za>
Message-ID: <1326513260.27.0.344096474072.issue13766@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +benjamin.peterson, eric.araujo, gvanrossum

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13766>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:56:25 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:56:25 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
In-Reply-To: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>
Message-ID: <1326513385.53.0.355823398946.issue13770@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Most probably.  The text model of Python has no bearing on the JSON spec, and 2.x as well as 3.x Pythons may want to output UTF-8 JSON or JavaScript-escaped ASCII-only JSON.  What specific wording is an issue?

----------
nosy: +eric.araujo
versions:  -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:57:21 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 03:57:21 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <1326513441.24.0.0215158508807.issue13722@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Those lines were whitespace reformed by Antoine in 61466/61463.
They previously came from 28325 which fixed #663074 (2003).
I do not see any discussion in that issue of making the import optional.
I suspect there is no test of Python working with no encodings ;-)

Just for fun, I commented out the import and error checks. There is an error message during the build about python_d.exe not working and python_d.exe crashes (ungracefully) on explicit startup, as you expected. I presume the patch is to delete the entire block added in 28325:
        if (PyErr_ExceptionMatches(PyExc_ImportError)) {
            /* Ignore ImportErrors... this is done so that
               distributions can disable the encodings package. Note
               that other errors are not masked, e.g. SystemErrors
               raised to inform the user of an error in the Python
               configuration are still reported back to the user. */
            PyErr_Clear();
            return 0;

Only in 3.3 or further back? I can do it if you like.

----------
nosy: +terry.reedy
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Sat Jan 14 04:57:45 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 03:57:45 +0000
Subject: [issue13773] Support sqlite3 uri filenames
In-Reply-To: <1326313998.63.0.583537561643.issue13773@psf.upfronthosting.co.za>
Message-ID: <1326513465.7.0.150700002795.issue13773@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
assignee:  -> ghaering
nosy: +eric.araujo, ghaering

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13773>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:01:31 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 04:01:31 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>
Message-ID: <1326513691.57.0.765114931739.issue13736@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
nosy: +eric.araujo, ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:10:05 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 04:10:05 +0000
Subject: [issue13723] Regular expressions: (?:X|\s+)*$ takes a long time
In-Reply-To: <1325886236.37.0.0463179039095.issue13723@psf.upfronthosting.co.za>
Message-ID: <1326514205.75.0.0718099729581.issue13723@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I believe it is a known fact that repeated repeats, like (...+)*, make for slow matching (if they work at all) with the current re engine.
[I would not be surprised if Perl does some special casing to (in effect at least) rewrite the re to your second version.] This is not going to be improved in 2.7, nor immediately in 3.x. You can try the regex module on pypi, but it may act the same. I suspect there are similar issues like this on the tracker. Best to write the re properly.

[Antoine or Ezio: If you think I am mistaken in closing this, please reopen.]

----------
nosy: +ezio.melotti, pitrou, terry.reedy
resolution:  -> wont fix
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13723>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:13:41 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sat, 14 Jan 2012 04:13:41 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
In-Reply-To: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>
Message-ID: <1326514421.77.0.737560642763.issue13770@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:35:00 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 04:35:00 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <1326515700.8.0.688139926442.issue13726@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +ezio.melotti, michael.foord
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:35:11 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 04:35:11 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326515711.55.0.100900760151.issue13725@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

unittest.test.test_program.TestCommandLineArgs has a few test functions that do at least a minimal test of help and verbosity options and command line args. Should more be added for other options, like this and #13726?

----------
nosy: +ezio.melotti, michael.foord, terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:36:00 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 04:36:00 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <1326515760.81.0.121405601427.issue11633@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Bug fixes include doc improvements, so 2.7 is fair game.

Thanks for your suggestion to not mention specific platforms.  Let?s just backport the 3.2 text.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:48:28 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 04:48:28 +0000
Subject: [issue13730] Grammar mistake in Decimal documentation
In-Reply-To: <1325963370.31.0.599534414571.issue13730@psf.upfronthosting.co.za>
Message-ID: <E1RlvXW-0001ja-Hm@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1e65183337ff by Terry Jan Reedy in branch '2.7':
closes #13730
http://hg.python.org/cpython/rev/1e65183337ff

New changeset bac18092c1dc by Terry Jan Reedy in branch '3.2':
#13730 grammar fix
http://hg.python.org/cpython/rev/bac18092c1dc

New changeset e6b0d9d209f2 by Terry Jan Reedy in branch 'default':
Merge with 3.2 #13730 grammar fix
http://hg.python.org/cpython/rev/e6b0d9d209f2

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13730>
_______________________________________

From report at bugs.python.org  Sat Jan 14 05:50:08 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 04:50:08 +0000
Subject: [issue13730] Grammar mistake in Decimal documentation
In-Reply-To: <1325963370.31.0.599534414571.issue13730@psf.upfronthosting.co.za>
Message-ID: <1326516608.56.0.0628951746495.issue13730@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Thanks for the report.

----------
nosy: +terry.reedy
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13730>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:05:05 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 14 Jan 2012 05:05:05 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326517505.2.0.734768003358.issue13725@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

@?ric I just tried to fix, with tests, the imaplib bug (#13700) and I found this bug. And fixing this bug I happened to notice in the source code another bug (#13726). No problems with CPython test suite relating to these regrtest bugs.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:07:43 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 05:07:43 +0000
Subject: [issue11633] Document that print may need explicit flushing
In-Reply-To: <1300790242.96.0.630463287349.issue11633@psf.upfronthosting.co.za>
Message-ID: <E1Rlvq8-0004UN-CW@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8935a33773b9 by Terry Jan Reedy in branch '2.7':
#11633 about buffering of print
http://hg.python.org/cpython/rev/8935a33773b9

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11633>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:13:01 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 05:13:01 +0000
Subject: [issue13744] raw byte strings are described in a confusing way
In-Reply-To: <1326116062.09.0.84789277908.issue13744@psf.upfronthosting.co.za>
Message-ID: <1326517981.26.0.0340462221198.issue13744@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +terry.reedy
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13744>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:17:43 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 05:17:43 +0000
Subject: [issue13749] socketserver can't stop
In-Reply-To: <1326136769.56.0.386749729049.issue13749@psf.upfronthosting.co.za>
Message-ID: <1326518263.83.0.55793537143.issue13749@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
stage:  -> test needed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13749>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:23:48 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 05:23:48 +0000
Subject: [issue13751] multiprocessing.pool hangs if any worker raises an
	Exception whose constructor requires a parameter
In-Reply-To: <1326140638.43.0.449072803971.issue13751@psf.upfronthosting.co.za>
Message-ID: <1326518628.38.0.143210189534.issue13751@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

2.6 and 3.1 only get security fixes.
I am only guessing that this is still an issue for 3.2. A test with 3.2.2 would be good. If this is really about pickling behavior w/r/t exceptions that cannot be changed then this should be closed.

----------
nosy: +alexandre.vassalotti, jnoller, pitrou, terry.reedy
versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:25:38 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sat, 14 Jan 2012 05:25:38 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326518738.31.0.276489922112.issue13779@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:28:43 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 05:28:43 +0000
Subject: [issue13769] json.dump(ensure_ascii=False) return str instead of
	unicode
In-Reply-To: <1326300079.82.0.279410736545.issue13769@psf.upfronthosting.co.za>
Message-ID: <1326518923.47.0.567787965131.issue13769@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Ezio, Raymond: is it the doc that is wrong?

----------
nosy: +ezio.melotti, rhettinger, terry.reedy
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13769>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:31:40 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sat, 14 Jan 2012 05:31:40 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326519100.0.0.954570450406.issue13725@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Erno: Okay, I just wanted to make sure you weren?t using it for your own projects, as regrtest is not an official public module.  Thanks for your patches!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:32:46 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 14 Jan 2012 05:32:46 +0000
Subject: [issue13780] make YieldFrom its own node
In-Reply-To: <1326462508.45.0.280954945804.issue13780@psf.upfronthosting.co.za>
Message-ID: <1326519166.41.0.656521649361.issue13780@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
components: +Interpreter Core
stage:  -> patch review
type:  -> enhancement
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13780>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:40:22 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sat, 14 Jan 2012 05:40:22 +0000
Subject: [issue13769] json.dump(ensure_ascii=False) return str instead of
	unicode
In-Reply-To: <1326300079.82.0.279410736545.issue13769@psf.upfronthosting.co.za>
Message-ID: <1326519622.35.0.498745272965.issue13769@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

The docstring says:
"""
    If ``ensure_ascii`` is false, then the return value will be a
    ``unicode`` instance subject to normal Python ``str`` to ``unicode``
    coercion rules instead of being escaped to an ASCII ``str``.
"""

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13769>
_______________________________________

From report at bugs.python.org  Sat Jan 14 06:40:43 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Sat, 14 Jan 2012 05:40:43 +0000
Subject: [issue13751] multiprocessing.pool hangs if any worker raises an
	Exception whose constructor requires a parameter
In-Reply-To: <1326140638.43.0.449072803971.issue13751@psf.upfronthosting.co.za>
Message-ID: <1326519643.87.0.951292745214.issue13751@psf.upfronthosting.co.za>


Faheem Mitha <faheem at faheem.info> added the comment:

This is an issue with SQLAlchemy exceptions, and has been worked around by Mike Bayer in http://www.sqlalchemy.org/trac/ticket/2371

For the record, I think the real problem is that Python exception pickling is broken, see http://bugs.python.org/issue1692335

It would be nice to see this fixed, otherwise this issue will continue to plague the Python standard libraries as well as other libraries. Library writers don't seem to be aware of the issue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________

From report at bugs.python.org  Sat Jan 14 07:06:41 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Sat, 14 Jan 2012 06:06:41 +0000
Subject: [issue13782] xml.etree.ElementTree: Element.append doesn't type-check
	its argument
In-Reply-To: <1326502453.32.0.206779556991.issue13782@psf.upfronthosting.co.za>
Message-ID: <1326521201.47.0.337192194689.issue13782@psf.upfronthosting.co.za>


Santoso Wijaya <santoso.wijaya at gmail.com> added the comment:

This does not only affect `append`, but also, `insert`, `extend`, etc.

In fact, the list-like operations on Element are merely forwarded to its `_children` (a list) field. Should we really type check these methods' arguments each? Doesn't strike as Pythonic to me...

OTOH, in cElementTree, by virtue of C-API's tuple unpacking convention, the argument's type is always strictly checked prior to accessing the children list.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13782>
_______________________________________

From report at bugs.python.org  Sat Jan 14 07:15:43 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sat, 14 Jan 2012 06:15:43 +0000
Subject: [issue13780] make YieldFrom its own node
In-Reply-To: <1326462508.45.0.280954945804.issue13780@psf.upfronthosting.co.za>
Message-ID: <1326521743.72.0.592405225569.issue13780@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Thanks for the review, Nick.

----------
Added file: http://bugs.python.org/file24233/yieldfromnode.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13780>
_______________________________________

From report at bugs.python.org  Sat Jan 14 07:19:47 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 14 Jan 2012 06:19:47 +0000
Subject: [issue13783] Clean up PEP 380 C API additions
Message-ID: <1326521987.14.0.4759346281.issue13783@psf.upfronthosting.co.za>


New submission from Nick Coghlan <ncoghlan at gmail.com>:

Georg noted that the PEP 380 patch added a couple of new C interfaces without documenting them. These interfaces need to be either:
1. Documented;
2. Made private; or
3. Removed (if they're completely trivial)

PyGen_FetchStopIterationValue: probably document, explicitly noting that it also clears the StopIteration exception from the thread state

PyStopIteration_Create: probably remove

----------
assignee: docs at python
components: Documentation, Interpreter Core
messages: 151235
nosy: docs at python, ncoghlan
priority: normal
severity: normal
status: open
title: Clean up PEP 380 C API additions
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13783>
_______________________________________

From report at bugs.python.org  Sat Jan 14 07:23:54 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 14 Jan 2012 06:23:54 +0000
Subject: [issue13783] Clean up PEP 380 C API additions
In-Reply-To: <1326521987.14.0.4759346281.issue13783@psf.upfronthosting.co.za>
Message-ID: <1326522234.38.0.356263445228.issue13783@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

The current use of PyStopIteration_Create in genobject.c can probably be replaced with PyErr_SetObject(PyExc_StopIteration, value) anyway.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13783>
_______________________________________

From report at bugs.python.org  Sat Jan 14 07:53:30 2012
From: report at bugs.python.org (Yaniv Aknin)
Date: Sat, 14 Jan 2012 06:53:30 +0000
Subject: [issue4106] multiprocessing occasionally spits out exception during
	shutdown
In-Reply-To: <1223695822.85.0.145044917765.issue4106@psf.upfronthosting.co.za>
Message-ID: <1326524010.42.0.928265413029.issue4106@psf.upfronthosting.co.za>


Yaniv Aknin <yaniv.aknin at gmail.com> added the comment:

Ugh. Not 100% sure it's related, but I've been getting a similar traceback when running pip's test suite (python setup.py test) on OSX 10.6.8 with Python 2.7.2.

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
    info('process shutting down')
TypeError: 'NoneType' object is not callable

Obviously it's not the exact same bug as fixed here, but Googling the traceback led me here and I do think it's the same genre of bug, i.e., multiprocessing's use of forking leads to issues when atexit is called (wasn't sure whether to open it here or #9207). Also, see https://groups.google.com/forum/#!topic/nose-users/fnJ-kAUbYHQ, it seems other users of the nose testsuite ran into this.

I'm afraid I won't have time to look much further into this (the reason I'm running pip's testsuite is that I'm already trying to make a contribution to pip...), but I thought it's best to at least mention it somewhere.

----------
nosy: +Yaniv.Aknin

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4106>
_______________________________________

From report at bugs.python.org  Sat Jan 14 08:08:02 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 07:08:02 +0000
Subject: [issue13294] http.server: minor code style changes.
In-Reply-To: <1319985642.06.0.787750282348.issue13294@psf.upfronthosting.co.za>
Message-ID: <1326524882.21.0.830375262268.issue13294@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13294>
_______________________________________

From report at bugs.python.org  Sat Jan 14 08:16:53 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 07:16:53 +0000
Subject: [issue13294] http.server: minor code style changes.
In-Reply-To: <1319985642.06.0.787750282348.issue13294@psf.upfronthosting.co.za>
Message-ID: <1326525413.73.0.414924684466.issue13294@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Eric - noted the point. :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13294>
_______________________________________

From report at bugs.python.org  Sat Jan 14 08:44:27 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Sat, 14 Jan 2012 07:44:27 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326527067.01.0.748043918173.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

Senthil, I ported the tests to 3.2. The quoting problem seems to be the same in 3.2 and the new test fails. I don't know how the password managers handle the usernames and passwords in python3 urllib so I did not look at that.

Could you please also port the fix to relevant python3 branches? I think you can do it much quicker since you are already familiar with both code bases and the fix.

----------
Added file: http://bugs.python.org/file24234/urllib-userpass-w-spaces.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sat Jan 14 09:09:32 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Sat, 14 Jan 2012 08:09:32 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326528572.67.0.343400738016.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

Also adding a patch that may be enough to fix the problem in python3.2. Review needed, did not test more than passing the previously failed unit test.

----------
Added file: http://bugs.python.org/file24235/urllib-userpass-w-spaces-fix.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sat Jan 14 11:29:42 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 14 Jan 2012 10:29:42 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1326402982.11.0.0757175288221.issue6774@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3xmAn_cNogpvDHZS2Zt5b13aE8utpo_5218dxt-P1sYw@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> This is questionable, unexpected, and should be documented.

There's already this note at the top of the socket module documentation:
"""
Note

Some behavior may be platform dependent, since calls are made to the
operating system socket APIs.
"""

There are other such subtleties with the socket API, e.g. SO_REUSEADDR
which doesn't have the same semantics on Windows. As I said earlier, I
don't think we should document every platform quirks: it will make
users worry for no reason, and as time passes, there's a chance that
the documentation doesn't match the actual behavior.
For example, this specific behavior might very well be a bug, and it's
not our responsibility to document this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Sat Jan 14 11:39:41 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 10:39:41 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <E1Rm11M-0000u5-TZ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4fc5dfad766a by Senthil Kumaran in branch '3.2':
Issue13726 - Fix the  ambiguous -S flag in regrtest. It is -o/--slow for printing the 10 slowest test.
http://hg.python.org/cpython/rev/4fc5dfad766a

New changeset 9a0efac5bbcb by Senthil Kumaran in branch '3.2':
Adding the -o option for Issue13726
http://hg.python.org/cpython/rev/9a0efac5bbcb

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 11:42:21 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 10:42:21 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <1326537741.43.0.936880410903.issue13726@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

I add -o as a short form for --slow. regrtest was using getopt and this conflicting option was not caught. I see it is a bug than a feature so the change is made in 3.2 and 3.3
default: b84f61fdcb4c and 6797e7458ad0

Thanks for catching this bug, Erno Tukia.

----------
nosy: +orsenthil
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 11:53:46 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 10:53:46 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <1313502559.06.0.415498401391.issue12760@psf.upfronthosting.co.za>
Message-ID: <E1Rm1F2-0005t6-R0@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8bcbe2dc3835 by Charles-Fran?ois Natali in branch 'default':
Issue #12760: Refer to the new 'x' open mode as "exclusive creation" mode.
http://hg.python.org/cpython/rev/8bcbe2dc3835

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sat Jan 14 12:13:22 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 11:13:22 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <E1Rm1Xz-0003lL-5u@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 80e3b8de4edd by Senthil Kumaran in branch '3.2':
Fix Issue #13642: Unquote before b64encoding user:password during Basic Authentication.
http://hg.python.org/cpython/rev/80e3b8de4edd

New changeset 4b4029fc8cf2 by Senthil Kumaran in branch 'default':
merge from 3.2 - Fix Issue #13642: Unquote before b64encoding user:password during Basic Authentication.
http://hg.python.org/cpython/rev/4b4029fc8cf2

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sat Jan 14 12:15:55 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 11:15:55 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326539755.86.0.590420062086.issue13642@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Here we go! I thought the problem did not exist in py3k, but good that the tests caught them and we have a fix now.

Thanks for the complete patch, Joonas. I hope it was easy to port the patch to 3k. The encoding part may perhaps be the only thing to careful with and use of the new string format feature was good one.  Thanks!

----------
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sat Jan 14 12:36:56 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 11:36:56 +0000
Subject: [issue13766] explain the relationship between Lib/lib2to3/Grammar.txt
	and Grammar/Grammar
In-Reply-To: <1326279554.15.0.713951578682.issue13766@psf.upfronthosting.co.za>
Message-ID: <1326541016.32.0.772875145453.issue13766@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
nosy: +orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13766>
_______________________________________

From report at bugs.python.org  Sat Jan 14 13:02:15 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Sat, 14 Jan 2012 12:02:15 +0000
Subject: [issue13784] Documentation of xml.sax.xmlreader:
	Locator.getLineNumber() and Locator.getColumnNumber()
Message-ID: <1326542534.95.0.205057982858.issue13784@psf.upfronthosting.co.za>


New submission from patrick vrijlandt <patrick.vrijlandt at gmail.com>:

Problem:
Locator methods return the location where the event starts, not where it ends.
Locator line numbers start at 1, Locator column numbers can be 0.

Proposal:
Adapt documentation.

>From the docs:
Instances of Locator provide these methods:

Locator.getColumnNumber() 
Return the column number where the current event ends.

Locator.getLineNumber() 
Return the line number where the current event ends

My Test:

import xml.sax

data = b"""<main>
    <sub
        attr="1"
        id="name"
        >
        <subsub />
    </sub>
</main>"""


class MyHandler(xml.sax.handler.ContentHandler):

    def startElement(self, name, attrs):
        if name == "sub":
            print("open", name, self._locator.getLineNumber(), self._locator.getColumnNumber())
            
    def endElement(self, name):
        if name == "sub":
            print("close", name, self._locator.getLineNumber(), self._locator.getColumnNumber())

xml.sax.parseString(data, MyHandler())

Output:

open sub 2 4
close sub 7 4

----------
assignee: docs at python
components: Documentation, XML
messages: 151247
nosy: docs at python, patrick.vrijlandt
priority: normal
severity: normal
status: open
title: Documentation of  xml.sax.xmlreader: Locator.getLineNumber() and Locator.getColumnNumber()
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13784>
_______________________________________

From report at bugs.python.org  Sat Jan 14 13:10:41 2012
From: report at bugs.python.org (Joonas Kuorilehto)
Date: Sat, 14 Jan 2012 12:10:41 +0000
Subject: [issue13642] urllib incorrectly quotes username and password in https
	basic auth
In-Reply-To: <1324393425.24.0.243026081887.issue13642@psf.upfronthosting.co.za>
Message-ID: <1326543041.79.0.297250222974.issue13642@psf.upfronthosting.co.za>


Joonas Kuorilehto <joneskoo at derbian.fi> added the comment:

Updating the issue with version 3.2 tag since it was fixed there as well. Still fixed, of course.

You are correct that the encodings can be tricky. Luckily I only added coding to tests. But you're right, I would consider very carefully before using similar code outside tests.

I just realized what's the impact of this change on python3.2 really. Since urllib.request.urlopen (for some reason) does not allow username and password on the URI, it is not possible to hit this with that, I think. But FancyURLopener does allow using user-pass in url, so this bug was reachable. I just verified that and the fix :)

Fix does the trick for FancyURLopener when the username and password are passed in the URL.

----------
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13642>
_______________________________________

From report at bugs.python.org  Sat Jan 14 13:30:00 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 14 Jan 2012 12:30:00 +0000
Subject: [issue12760] Add create mode to open()
In-Reply-To: <E1Rm1F2-0005t6-R0@dinsdale.python.org>
Message-ID: <CAH_1eM0+zO1LPCX5D4u+-PmRa_mxfTs8xBSPT+TL=ySfMEqK7g@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Thanks, I've committed your version.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12760>
_______________________________________

From report at bugs.python.org  Sat Jan 14 13:45:45 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 14 Jan 2012 12:45:45 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <1326545145.45.0.932173694809.issue13726@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

--start requires an argument but short opt -S does not.

in Lib/test/regrtest.py

opts, args = getopt.getopt(sys.argv[1:], '...S...',
            [..., 'start=', ...])

Patch included.

----------
keywords: +patch
Added file: http://bugs.python.org/file24236/issue13726-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 13:53:11 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 14 Jan 2012 12:53:11 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <1326545591.32.0.818794063062.issue13726@psf.upfronthosting.co.za>


Changes by Erno Tukia <erno.tukia at iki.fi>:


----------
resolution: fixed -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 14:08:28 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 13:08:28 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <E1Rm3LP-00011i-4T@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset c4919642fd25 by Senthil Kumaran in branch '3.2':
Fix issue13726: -S accepts an argument just as --start.
http://hg.python.org/cpython/rev/c4919642fd25

New changeset 347f6305bd26 by Senthil Kumaran in branch 'default':
merge from 3.2 - Fix issue13726: -S accepts an argument just as --start.
http://hg.python.org/cpython/rev/347f6305bd26

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 14:09:11 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 14 Jan 2012 13:09:11 +0000
Subject: [issue13726] regrtest ambiguous -S flag
In-Reply-To: <1325938488.7.0.238559424001.issue13726@psf.upfronthosting.co.za>
Message-ID: <1326546551.73.0.29429029337.issue13726@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Fixed now. Thanks!

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13726>
_______________________________________

From report at bugs.python.org  Sat Jan 14 14:58:29 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 13:58:29 +0000
Subject: [issue13780] make YieldFrom its own node
In-Reply-To: <1326462508.45.0.280954945804.issue13780@psf.upfronthosting.co.za>
Message-ID: <E1Rm47o-000470-4a@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset e1177b9167ce by Benjamin Peterson in branch 'default':
make YieldFrom its own distinct from Yield (closes #13780)
http://hg.python.org/cpython/rev/e1177b9167ce

----------
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13780>
_______________________________________

From report at bugs.python.org  Sat Jan 14 15:02:16 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 14 Jan 2012 14:02:16 +0000
Subject: [issue13783] Clean up PEP 380 C API additions
In-Reply-To: <1326521987.14.0.4759346281.issue13783@psf.upfronthosting.co.za>
Message-ID: <1326549736.11.0.132360659843.issue13783@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Also check any GetAttr and CallMethod operations to see if they should be moved over to the new *Id variants.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13783>
_______________________________________

From report at bugs.python.org  Sat Jan 14 16:19:49 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 15:19:49 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <E1Rm5OV-00013y-Gw@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f80f9eb5df02 by Meador Inge in branch '3.2':
Issue #13725: regrtest does not recognize -d flag.
http://hg.python.org/cpython/rev/f80f9eb5df02

New changeset f2b764dd0851 by Meador Inge in branch 'default':
Issue #13725: regrtest does not recognize -d flag.
http://hg.python.org/cpython/rev/f2b764dd0851

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 16:21:33 2012
From: report at bugs.python.org (Meador Inge)
Date: Sat, 14 Jan 2012 15:21:33 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326554493.34.0.644506596065.issue13725@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

?ric, I got it.

Erno, thanks for the patch.

----------
nosy: +meador.inge
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 16:57:37 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 15:57:37 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <E1Rm5z5-000435-LL@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 3e9b107fb5da by Meador Inge in branch '3.2':
Issue #13725: Add a NEWS entry.
http://hg.python.org/cpython/rev/3e9b107fb5da

New changeset 7332e97587da by Meador Inge in branch 'default':
Issue #13725: Add a NEWS entry.
http://hg.python.org/cpython/rev/7332e97587da

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 17:41:06 2012
From: report at bugs.python.org (=?utf-8?b?0JzQsNGA0Log0JrQvtGA0LXQvdCx0LXRgNCz?=)
Date: Sat, 14 Jan 2012 16:41:06 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
In-Reply-To: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>
Message-ID: <1326559266.35.0.286584474759.issue13770@psf.upfronthosting.co.za>


???? ????????? <socketpair at gmail.com> added the comment:

ensure_ascii is documented only in Jsonencoder class. Other functions like dumps() should refer to this class description for that parameter.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Sat Jan 14 18:07:15 2012
From: report at bugs.python.org (Juan Javier)
Date: Sat, 14 Jan 2012 17:07:15 +0000
Subject: [issue13785] Make concurrent.futures.Future state public
Message-ID: <1326560835.55.0.282912980665.issue13785@psf.upfronthosting.co.za>


New submission from Juan Javier <jjdominguezm at gmail.com>:

Hello,

This is a proposal to make the state of Future objects public.

The idea is to have access to the current state of the Future using a property instead of calling several methods (done, cancelled, etc.).

Also, a history property that returns a list of Event(state, timestamp) objects is written, the list stores the timestamp every time the state of a future changes.

There is a patch attached to the issue.

Regards.

----------
components: Library (Lib)
files: concurrent.futures.Future.state_public.patch
keywords: patch
messages: 151259
nosy: bquinlan, jjdominguezm
priority: normal
severity: normal
status: open
title: Make concurrent.futures.Future state public
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24237/concurrent.futures.Future.state_public.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13785>
_______________________________________

From report at bugs.python.org  Sat Jan 14 18:42:52 2012
From: report at bugs.python.org (Meador Inge)
Date: Sat, 14 Jan 2012 17:42:52 +0000
Subject: [issue13786] regrtest.py does not handle --trace
Message-ID: <1326562972.28.0.512194843439.issue13786@psf.upfronthosting.co.za>


New submission from Meador Inge <meadori at gmail.com>:

While fixing issue13725 I noticed that there is a long option for 'trace', but it is not actually handled:

[meadori at motherbrain cpython]$ ./python -m test --trace
No handler for option --trace.  Please report this as a bug at http://bugs.python.org.

I will commit the attached patch shortly.

----------
assignee: meador.inge
components: Tests
files: regrtest-remove-unhandled-trace-longoption.patch
keywords: easy, patch
messages: 151260
nosy: meador.inge
priority: low
severity: normal
stage: commit review
status: open
title: regrtest.py does not handle --trace
type: behavior
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24238/regrtest-remove-unhandled-trace-longoption.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13786>
_______________________________________

From report at bugs.python.org  Sat Jan 14 18:47:47 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 14 Jan 2012 17:47:47 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1250550378.97.0.072881968798.issue6721@psf.upfronthosting.co.za>
Message-ID: <1326563267.89.0.572431244449.issue6721@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Sat Jan 14 18:56:19 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 17:56:19 +0000
Subject: [issue13786] regrtest.py does not handle --trace
In-Reply-To: <1326562972.28.0.512194843439.issue13786@psf.upfronthosting.co.za>
Message-ID: <E1Rm7py-0003bo-28@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a32bc62b8772 by Meador Inge in branch '3.2':
Issue #13786: Remove unimplemented 'trace' long option from regrtest.py.
http://hg.python.org/cpython/rev/a32bc62b8772

New changeset 9b14b04a0cdd by Meador Inge in branch 'default':
Issue #13786: Remove unimplemented 'trace' long option from regrtest.py.
http://hg.python.org/cpython/rev/9b14b04a0cdd

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13786>
_______________________________________

From report at bugs.python.org  Sat Jan 14 18:56:49 2012
From: report at bugs.python.org (Meador Inge)
Date: Sat, 14 Jan 2012 17:56:49 +0000
Subject: [issue13786] regrtest.py does not handle --trace
In-Reply-To: <1326562972.28.0.512194843439.issue13786@psf.upfronthosting.co.za>
Message-ID: <1326563809.9.0.938758223859.issue13786@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

Fixed.

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13786>
_______________________________________

From report at bugs.python.org  Sat Jan 14 19:11:01 2012
From: report at bugs.python.org (Erno Tukia)
Date: Sat, 14 Jan 2012 18:11:01 +0000
Subject: [issue13725] regrtest does not recognize -d flag
In-Reply-To: <1325937108.85.0.410316805004.issue13725@psf.upfronthosting.co.za>
Message-ID: <1326564661.38.0.316636356715.issue13725@psf.upfronthosting.co.za>


Erno Tukia <erno.tukia at iki.fi> added the comment:

Meador, thanks for the acknowledgement.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13725>
_______________________________________

From report at bugs.python.org  Sat Jan 14 19:13:53 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sat, 14 Jan 2012 18:13:53 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1326512216.4.0.465598540602.issue12409@psf.upfronthosting.co.za>
Message-ID: <CAPdtAj0JrhurNdB=u=UQURhC4PKTAcFAVVtErKMkJG9m2M4urQ@mail.gmail.com>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

On Sat, Jan 14, 2012 at 04:36, ?ric Araujo <report at bugs.python.org> wrote:
>
> ?ric Araujo <merwok at netwok.org> added the comment:
>
> I just replied to your python-dev email. ?I think you can update 2.7 to use Sphinx 1.0 as soon as you?re down to a handful of warnings.

thanks! I've just updated 2.7 to sphinx 1.0 - yay

> When the migration is done, you can add the documenting doc to the devguide and send a message to pydev to tell people to not commit changes to Doc/documenting in CPython clones anymore.
>
> In the same email you can ask who to ask for setting up redirects (BTW, will redirecting paths to fragments work? ?e.g. docs.py.o/documenting/style to devguide/documenting#style-and-blah). ?When the redirects are up you?ll be able to rm Doc/documenting.

I just did that with:
http://mail.python.org/pipermail/python-dev/2012-January/115520.html

> Then we?ll open other reports to talk about the unrelated I mentioned and to see how to stop downloading Sphinx from svn once per clone :)

Ah yes, sure!

Cheers,
Sandro

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Sat Jan 14 19:23:41 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 14 Jan 2012 18:23:41 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <E1Rm8GR-0005Z0-JI@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset d4669f43d05f by Benjamin Peterson in branch 'default':
add str.casefold() (closes #13752)
http://hg.python.org/cpython/rev/d4669f43d05f

----------
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Sat Jan 14 20:32:38 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 14 Jan 2012 19:32:38 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1326452531.03.0.423724201086.issue6721@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3HuQRAbhSisEzRSXPty4mr32edRoeJ3kPj6PF9f1Mjig@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> However, extending RLock to provide ForkClearedRLock (this would be used by logging, i.e.) is quite straighforward.
>
> The extended class would simply need to record the process ID, in which the lock was created, and the process ID, which is trying to acquire it. ?Done!

There are at least two problems with this approach.
- with LinuxThreads, threads have different PIDs, so this would break.
 LinuxThreads have now been replaced with NPTL, so this may not be a
showstopper, though
- however, the other problem is more serious: it has the exact same
problem as the lock reinitialization upon fork(): locks are used to
protect critical sections, to make sure that threads see a consistent
state: if you simply proceed and reset/acquire the lock when the
process is not the last one that owned it, the invariants protected by
the lock will be broken.
The proper solution is to setup handlers to be called upon fork, that
not only reset locks, but also internal state of objects they protect.
However, this is a dull and boring task, and would require patching
dozens of different places. It's been on my todo list for some time...
Another "solution" would be to remove the only place in the standard
library where a bare fork() is used, in multiprocessing (issue #8713).
Then, it's the user's problem if he calls fork() :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Sat Jan 14 20:40:52 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Sat, 14 Jan 2012 19:40:52 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1250550378.97.0.072881968798.issue6721@psf.upfronthosting.co.za>
Message-ID: <1326570052.5.0.861056478517.issue6721@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

A new lock type will NOT solve this.  It is ALWAYS okay to clear all thread/threading module locks after a fork.

They are and always have been process-local by definition so they are also by definition 100% invalid to any child process.

Anyone who has written code using them to "lock" an out-of-process resource has written code that is already broken today. Thread locks can't guard network resources.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Sat Jan 14 22:00:16 2012
From: report at bugs.python.org (Brian Quinlan)
Date: Sat, 14 Jan 2012 21:00:16 +0000
Subject: [issue13785] Make concurrent.futures.Future state public
In-Reply-To: <1326560835.55.0.282912980665.issue13785@psf.upfronthosting.co.za>
Message-ID: <1326574816.27.0.645210553898.issue13785@psf.upfronthosting.co.za>


Brian Quinlan <brian at sweetapp.com> added the comment:

Thanks for the patch!

1. The fetching the state feature seems reasonable but I think that explaining the difference between CANCELLED and CANCELLED_AND_NOTIFIED is going to be hard. Maybe you could look at how Doc/library/concurrent.futures.rst would need to be updated to see if we can provide a reasonable description of the different states?

2. Having the future record the history of its state transitions seems potentially useful but there is no precedent for this in other Python objects where it would also be useful. Maybe you could make the executors take a Futures factory and then provide a subclass that does that? Not sure that I like that either but...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13785>
_______________________________________

From report at bugs.python.org  Sat Jan 14 23:55:51 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sat, 14 Jan 2012 22:55:51 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326581751.26.0.103089673712.issue12409@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

"Documenting Python" has been:

- moved to devguide; http://hg.python.org/devguide/rev/5d4a90e568ed
- removed from 2.7, 3.2 and default;
    http://hg.python.org/cpython/rev/8975f7d0fbf7
    http://hg.python.org/cpython/rev/70c46599fb0e
    http://hg.python.org/cpython/rev/900c01e643a7

Thanks everyone!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Sun Jan 15 00:47:04 2012
From: report at bugs.python.org (Mahmoud Hashemi)
Date: Sat, 14 Jan 2012 23:47:04 +0000
Subject: [issue12949] Documentation of PyCode_New() lacks kwonlyargcount
	argument
In-Reply-To: <1315592789.75.0.632556595518.issue12949@psf.upfronthosting.co.za>
Message-ID: <1326584824.7.0.725469469119.issue12949@psf.upfronthosting.co.za>


Changes by Mahmoud Hashemi <makuro at gmail.com>:


----------
nosy: +mahmoud

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12949>
_______________________________________

From report at bugs.python.org  Sun Jan 15 01:33:11 2012
From: report at bugs.python.org (Mahmoud Hashemi)
Date: Sun, 15 Jan 2012 00:33:11 +0000
Subject: [issue13787] PyCode_New not round-trippable (TypeError)
Message-ID: <1326587591.5.0.14837599409.issue13787@psf.upfronthosting.co.za>


New submission from Mahmoud Hashemi <makuro at gmail.com>:

On Python 3.1.4, attempting to create a code object will apparently result in a TypeError (must be str, not tuple), even when you're creating a code object from another, working code object:

# co_what.py
def foo():
    return 'bar'

co = foo.__code__

co_copy = type(co)(co.co_argcount,
                   co.co_kwonlyargcount,
                   co.co_nlocals,
                   co.co_stacksize,
                   co.co_flags,
                   co.co_code,
                   co.co_consts,
                   co.co_names,
                   co.co_varnames,
                   co.co_freevars,
                   co.co_cellvars,
                   co.co_filename,
                   co.co_name,
                   co.co_firstlineno,
                   co.co_lnotab)
# EOF
$ python3 co_what.py
Traceback (most recent call last):
  File "co_what.py", line 20, in <module>
    co.co_lnotab)
TypeError: must be str, not tuple

Looking at the PyCode_New function, all the arguments look correctly matched up according to the signature in my Python 3.1.4 build source (looks identical to the trunk source):

# Objects/codeobject.c

PyCode_New(int argcount, int kwonlyargcount,
           int nlocals, int stacksize, int flags,
           PyObject *code, PyObject *consts, PyObject *names,
           PyObject *varnames, PyObject *freevars, PyObject *cellvars,
           PyObject *filename, PyObject *name, int firstlineno,
           PyObject *lnotab)
{
    PyCodeObject *co;
    Py_ssize_t i;

    /* Check argument types */
    if (argcount < 0 || nlocals < 0 ||
        code == NULL ||
        consts == NULL || !PyTuple_Check(consts) ||
        names == NULL || !PyTuple_Check(names) ||
        varnames == NULL || !PyTuple_Check(varnames) ||
        freevars == NULL || !PyTuple_Check(freevars) ||
        cellvars == NULL || !PyTuple_Check(cellvars) ||
        name == NULL || !PyUnicode_Check(name) ||
        filename == NULL || !PyUnicode_Check(filename) ||
        lnotab == NULL || !PyBytes_Check(lnotab) ||
        !PyObject_CheckReadBuffer(code)) {
        PyErr_BadInternalCall();
        return NULL;
    }

And, for the record, this same behavior works just fine in the equivalent Python 2.

----------
components: Interpreter Core
files: co_what.py
messages: 151270
nosy: mahmoud
priority: normal
severity: normal
status: open
title: PyCode_New not round-trippable (TypeError)
type: behavior
versions: Python 3.1, Python 3.2
Added file: http://bugs.python.org/file24239/co_what.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13787>
_______________________________________

From report at bugs.python.org  Sun Jan 15 01:34:35 2012
From: report at bugs.python.org (Mahmoud Hashemi)
Date: Sun, 15 Jan 2012 00:34:35 +0000
Subject: [issue13787] PyCode_New not round-trippable (TypeError)
In-Reply-To: <1326587591.5.0.14837599409.issue13787@psf.upfronthosting.co.za>
Message-ID: <1326587675.34.0.519525942308.issue13787@psf.upfronthosting.co.za>


Mahmoud Hashemi <makuro at gmail.com> added the comment:

And here's the working Python 2 version (works fine on Python 2.7, and likely a few versions prior).

----------
Added file: http://bugs.python.org/file24240/co_what2.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13787>
_______________________________________

From report at bugs.python.org  Sun Jan 15 01:41:48 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Sun, 15 Jan 2012 00:41:48 +0000
Subject: [issue13770] python3 & json: add ensure_ascii documentation
In-Reply-To: <1326300131.17.0.750918490995.issue13770@psf.upfronthosting.co.za>
Message-ID: <1326588108.36.0.897609302866.issue13770@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

All right.  I?ll do like other parameters (see check_circular for an example) and just duplicate the two-line doc.

----------
assignee: docs at python -> eric.araujo
versions: +Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13770>
_______________________________________

From report at bugs.python.org  Sun Jan 15 05:29:25 2012
From: report at bugs.python.org (Ferringb)
Date: Sun, 15 Jan 2012 04:29:25 +0000
Subject: [issue13788] os.closerange optimization
Message-ID: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>


New submission from Ferringb <ferringb at gmail.com>:

The current implementation of closerange essentially is a bruteforce invocation of close for every integer in the range.

While this works, it's rather noisy for stracing, and for most invocations, is near a thousand close invocations more than needed.

As such it should be aware of /proc/${PID}/fd, and use that to isolate down just what is actually open, and close that.

----------
components: Extension Modules
files: closerange-optimization.patch
keywords: patch
messages: 151273
nosy: ferringb
priority: normal
severity: normal
status: open
title: os.closerange optimization
type: performance
Added file: http://bugs.python.org/file24241/closerange-optimization.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 05:43:26 2012
From: report at bugs.python.org (Ferringb)
Date: Sun, 15 Jan 2012 04:43:26 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326602606.95.0.107599723199.issue13788@psf.upfronthosting.co.za>


Ferringb <ferringb at gmail.com> added the comment:

Fixed tabs/spaces...

----------
Added file: http://bugs.python.org/file24242/closerange-optimization.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 05:43:56 2012
From: report at bugs.python.org (Ferringb)
Date: Sun, 15 Jan 2012 04:43:56 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326602636.83.0.450050251875.issue13788@psf.upfronthosting.co.za>


Changes by Ferringb <ferringb at gmail.com>:


Removed file: http://bugs.python.org/file24241/closerange-optimization.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 05:48:53 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 15 Jan 2012 04:48:53 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326602933.87.0.593829590986.issue13788@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

Thanks for the patch.

However, this cannot as far as I understand be used for the subprocess implementation due to the limitation of what can be called after a fork() and before an exec().

Take a look at #8052 for some more discussion of this.

----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 05:59:16 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 15 Jan 2012 04:59:16 +0000
Subject: [issue13789] _tkinter does not build on Windows 7
Message-ID: <1326603556.49.0.117233546142.issue13789@psf.upfronthosting.co.za>


New submission from Terry J. Reedy <tjreedy at udel.edu>:

Modules/_tkinter.py #includes Modules/tkinter.h, tcl.h, and tk.h. The latter two #include tclDecls.h, tclPlatDecls.h, and tkDecls.h. I obtained 8.5.9 versions of all 5 and added them to Include/. Although they came from Ubuntu, they appear to be cross-platform in that they look for Mac and Win defines. They make the corresponding build errors go away. However, one remains: a failure to open X11/Xlib.h.

This must come from the following in tk.h:
#ifndef RC_INVOKED
#ifndef _XLIB_H
#   if defined(MAC_OSX_TK)
#    include <X11/Xlib.h>
#    include <X11/X.h>
#   else
#    include <X11/Xlib.h>
#   endif
#endif
#ifdef __STDC__
#   include <stddef.h>
#endif

It seems this tk.h header, at least, expects RC_INVOKED to be defined on non-X11 platforms. Something here is crucially different than on the machine that builds Python for windows. The version of tk.h? the absence of X11/xlib.h? the pcbuild stuff? or the Visual Studio version?

I tried to compile PCbuild/pcbuild.sln with VC2008 Express in the 3.3, 3.2, and 2.7 repositories. For 3.3, I tried both debug and release builds.

----------
components: Build, Windows
messages: 151276
nosy: brian.curtin, loewis, terry.reedy, tim.golden
priority: normal
severity: normal
status: open
title: _tkinter does not build on Windows 7
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13789>
_______________________________________

From report at bugs.python.org  Sun Jan 15 06:31:19 2012
From: report at bugs.python.org (py.user)
Date: Sun, 15 Jan 2012 05:31:19 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
Message-ID: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

>>> '{0:d}'.format('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
>>> '{0:d}'.format(1+1j)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'complex'
>>> '{0:d}'.format([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
>>>

also strange behavior:
>>> '{0:s}'.format((1, 2, 3))
'(1, 2, 3)'
>>> '{0:10.5s}'.format([1, 2, 3])
'[1, 2     '
>>>

----------
components: Interpreter Core
messages: 151277
nosy: py.user
priority: normal
severity: normal
status: open
title: In str.format an incorrect error message for list, tuple, dict, set
type: behavior
versions: Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sun Jan 15 06:51:12 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sun, 15 Jan 2012 05:51:12 +0000
Subject: [issue13495] IDLE: Regressions - Two ColorDelegator instances loaded
	and -e no longer edits new files.
In-Reply-To: <1322510218.36.0.927250562508.issue13495@psf.upfronthosting.co.za>
Message-ID: <1326606672.71.0.877560879814.issue13495@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

There was another regression introduced by (a4bd8a4805a8). IDLE 3 no longer allows for editing of new files from the command line. 

For example:

   idle -e /tmp/newfile.py 

will momentarily flash an editor window before it is closed. This is due to "good_load" not being set. What's worse is that the IDLE process will continue running without having any visible windows open. 

The rev2 patch removes the "good_load" flag and fixes the ColorDelegator issue.

I understand the purpose of the original code is to avoid displaying a blank editor in case of a decode error. The rev2 patch reintroduces this behavior. In my opinion, that behavior is not a serious problem.

----------
nosy: +terry.reedy
title: IDLE: Regression - Two ColorDelegator instances loaded -> IDLE: Regressions - Two ColorDelegator instances loaded and -e no longer edits new files.
Added file: http://bugs.python.org/file24243/issue13495_rev2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13495>
_______________________________________

From report at bugs.python.org  Sun Jan 15 06:57:35 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sun, 15 Jan 2012 05:57:35 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326607055.16.0.76433371333.issue13790@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

I agree it's not the best error message. What's happening is that these types (list, tuple, etc.) do not implement __format__, so object.__format__ is used. It returns str(self). Then the resulting string is formatted with the given format_spec. Since str does not support the 'd' format type, the error you see is raised.

I'm open to suggestions on how to improve this, but I don't see how it's possible given what str.__format__ knows when it generates the error.

----------
assignee:  -> eric.smith
nosy: +eric.smith
versions: +Python 3.2, Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sun Jan 15 07:52:23 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 15 Jan 2012 06:52:23 +0000
Subject: [issue13495] IDLE: Regressions - Two ColorDelegator instances loaded
	and -e no longer edits new files.
In-Reply-To: <1322510218.36.0.927250562508.issue13495@psf.upfronthosting.co.za>
Message-ID: <1326610343.16.0.536540741859.issue13495@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

If you are saying that it is better to open a blank window than to not open a filled window, I agree.

----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13495>
_______________________________________

From report at bugs.python.org  Sun Jan 15 10:06:53 2012
From: report at bugs.python.org (Matt Joiner)
Date: Sun, 15 Jan 2012 09:06:53 +0000
Subject: [issue13791] Other versions
Message-ID: <1326618413.55.0.73666208075.issue13791@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

The navigation region at http://docs.python.org/dev/ should list "Other versions" instead of "Old versions" as the in development docs are also available here.

Docs for other versions
Python 2.7 (stable)
Python 3.2 (stable)
Old versions

Unreleased, development versions of the documentation. This may contain more information than the current released documentation.

http://www.python.org/doc/versions/

----------
assignee: docs at python
components: Documentation
messages: 151281
nosy: anacrolix, docs at python
priority: normal
severity: normal
status: open
title: Other versions
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13791>
_______________________________________

From report at bugs.python.org  Sun Jan 15 10:15:30 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 15 Jan 2012 09:15:30 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326618930.46.0.526232544232.issue13788@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> subprocess close_fds behavior should only close open fds

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 10:58:09 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Sun, 15 Jan 2012 09:58:09 +0000
Subject: [issue13768] Doc/tools/dailybuild.py available only on 2.7 branch
In-Reply-To: <1326487974.89.0.656724547294.issue13768@psf.upfronthosting.co.za>
Message-ID: <CAA77j2CVvH=8qyzhXUp=0i7CqJn3B4YWKNjtdFid5HZ1rXs91w@mail.gmail.com>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

> New submission from Georg Brandl <georg at python.org>:
>
> Why is that a concern? ?It is not needed for the doc build and intended to be used on python.org only.

It's mentioned in
http://hg.python.org/devguide/file/5d4a90e568ed/docquality.rst, so I
got curious and found it missing in my 'default' branch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13768>
_______________________________________

From report at bugs.python.org  Sun Jan 15 11:00:28 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Sun, 15 Jan 2012 10:00:28 +0000
Subject: [issue13768] Doc/tools/dailybuild.py available only on 2.7 branch
In-Reply-To: <1326487974.89.0.656724547294.issue13768@psf.upfronthosting.co.za>
Message-ID: <1326621628.64.0.479822600614.issue13768@psf.upfronthosting.co.za>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

ok, the url is http://hg.python.org/devguide/file/5d4a90e568ed/docquality.rst

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13768>
_______________________________________

From report at bugs.python.org  Sun Jan 15 11:04:40 2012
From: report at bugs.python.org (Jed Davis)
Date: Sun, 15 Jan 2012 10:04:40 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326621880.8.0.928304183001.issue13781@psf.upfronthosting.co.za>


Changes by Jed Davis <jld at panix.com>:


----------
nosy: +jld

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Sun Jan 15 11:17:26 2012
From: report at bugs.python.org (Ferringb)
Date: Sun, 15 Jan 2012 10:17:26 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1326622646.58.0.214299064907.issue8052@psf.upfronthosting.co.za>


Ferringb <ferringb at gmail.com> added the comment:

In #13788, I've uploaded a patch modifying closerange along the same lines as this discussion; someone w/ appropriate rights should set dependencies as needed.

Either way, here's a question: does anyone actually know of a unix that does procfs, and has a daft opendir implementation as described below?  Aka, are we actually worrying about something relevant, or just hypotheticals?

Strikes me any scenario where this actually would occur, we're already in hell from broken implementations- meaning we probably don't support them.  In the same angle, has anyone asked opengroup (forums, ml, etc), or moreso figured out *where* to ask for the given reasoning here?

Regardless, if we're dead set on adhering to the standards there (and using re-entrant readdir_r and friends isn't enough to make people happy), a few hacks come to mind:

1) in the child (child1), split a pipe, fork/exec (child2) an ls -l (or equivalent) of /proc/$PID/fd, feeding it back to child1 which then acts on it.
2) grab the fd list pre-fork along w/ the link count for /proc/$PID/fd; child re-stats /proc/$PID/fd, if link count is the same, the results should be able to be acted upon.  I'm *reasonably* sure there is a retarded FS or two out there that doesn't set link count for a directory correctly, but I've never heard of it for a procfs.  Regardless, should be detectable- nlinks would be 0.  If it is, and len(fds) != 0, then you know you can't trust the results and have to fallback to brute force close the range.  Additionally, we ought to be able to test for this... so... score.

Addressing: "signal handlers can open files".  Yes, they can, and our current implementation doesn't handle that case /anyways/, so it's orthogonal to speeding up closerange.

Finally, doing some codesearching, here's the rough list of heavy hitters spotted using this:
*) java (as mentioned)
*) chrome's seccomp sandbox uses it
*) glibc's nscd uses it (pretty strong indication this is safe in that case to say the least)
*) gdm
*) pulseaudio (limited to linux)
*) opensolaris actually does this, although from reading the code it sounds as if there is an issue w/ vfork- thus they use getdents directly.  Look for spawn_closefrom for details.

So.. seems a bit linuxy.  Could possibly enable it just there (whitelist).

----------
nosy: +ferringb

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 15 12:09:51 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 15 Jan 2012 11:09:51 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1326622646.58.0.214299064907.issue8052@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM0AU9f80e_g6L8X9p-BmYnV3-a0jECz6XOyRvwfrbULGQ@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Either way, here's a question: does anyone actually know of a unix that does procfs, and has a daft opendir implementation as described below?  Aka, are we actually worrying about something relevant, or just hypotheticals?

I think it's more theoretical.
Since dirent have per-struct locks, the only reason why
opendir/readdir would not be async-safe would be because malloc() is
not async-safe. Since we already allow running Python code after
fork(), we implicitely assume that malloc() (and actually most of the
libc) is async-safe, which is true in practice because malloc() uses
pthread_atfork to reset its internal locks after fork().
So IMHO, calling opendir() should be safe (and as noted, many code out
there already does this).
The only question is: do other Unix also have /proc/<pid>/fd? e.g.
FreeBSD, OpenBSD. That's especially important because FreeBSD can have
a huge RLIMIT_NOFILE by default.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 15 13:02:30 2012
From: report at bugs.python.org (Ferringb)
Date: Sun, 15 Jan 2012 12:02:30 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1326628950.32.0.331676886965.issue8052@psf.upfronthosting.co.za>


Ferringb <ferringb at gmail.com> added the comment:

>The only question is: do other Unix also have /proc/<pid>/fd? e.g.
>FreeBSD, OpenBSD. That's especially important because FreeBSD can have
>a huge RLIMIT_NOFILE by default.

Unless the OS gives some way to optimize the process (whether inferring from procfs, or making use of spawn_closefrom), there really isn't anything we can do.  O_CLOEXEC is one option, but that's basically the same as the close loop in terms of syscalls- specifically post fork looping over the range and setting it.  Beyond that, it's linux specific, so only would be used if the root python was invoked from lacked procfs.

I'm willing to extend my original patch to handle alternate OS hints as needed; in the same way, the nlinks trick I can implement although I'd be more inclined to just limit my original closerange patch to OSs that have a sane opendir and procfs.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 15 13:38:01 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 15 Jan 2012 12:38:01 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326631081.57.0.51393340183.issue13788@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

fwiw, s/MSDOS_WINDOWS/MS_WINDOWS/.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 15:01:30 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 15 Jan 2012 14:01:30 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1322872228.97.0.525171057572.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326636090.29.0.839986828946.issue13521@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

No more double lookup.

----------
Added file: http://bugs.python.org/file24244/13521_27_3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan 15 16:25:49 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Sun, 15 Jan 2012 15:25:49 +0000
Subject: [issue13788] os.closerange optimization
In-Reply-To: <1326601765.34.0.310008166629.issue13788@psf.upfronthosting.co.za>
Message-ID: <1326641149.93.0.121930195835.issue13788@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

Reopening.  Comments added to the code review.

This issue is independent of the subprocess module issue in #8052.  The _posixsubprocess.c has its own fd closing loop.

 http://hg.python.org/cpython/file/050c07b31192/Modules/_posixsubprocess.c#l118

----------
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith
resolution: duplicate -> 
status: closed -> open
superseder: subprocess close_fds behavior should only close open fds -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13788>
_______________________________________

From report at bugs.python.org  Sun Jan 15 16:44:40 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 15 Jan 2012 15:44:40 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <1326642280.15.0.300969104042.issue13722@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Actually, that code is from bc861add5d71.
But the error was also muted in the initial checkin in d0e06efb3165.
In any case, the silencing should be removed, both in 3.2 and 3.3 IMO.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Sun Jan 15 16:50:39 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 15 Jan 2012 15:50:39 +0000
Subject: [issue11694] xdrlib raises ConversionError in inconsistent way
In-Reply-To: <1301224336.99.0.411299821684.issue11694@psf.upfronthosting.co.za>
Message-ID: <1326642639.65.0.0204558178612.issue11694@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

Bump! It's almost 3 months since I posted the patch, so I would like to remind about this bug.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11694>
_______________________________________

From report at bugs.python.org  Sun Jan 15 17:46:26 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 15 Jan 2012 16:46:26 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1326645986.2.0.491747897938.issue8052@psf.upfronthosting.co.za>


Ross Lagerwall <rosslagerwall at gmail.com> added the comment:

FreeBSD has a /dev/fd as well as a procfs (deprecated AFAIK).
However, both may not be mounted so a patch would *need* to at least fallback to the current functionality.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 15 17:54:22 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 15 Jan 2012 16:54:22 +0000
Subject: [issue11694] xdrlib raises ConversionError in inconsistent way
In-Reply-To: <1301224336.99.0.411299821684.issue11694@psf.upfronthosting.co.za>
Message-ID: <1326646462.34.0.584562176818.issue11694@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Looks good to me.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11694>
_______________________________________

From report at bugs.python.org  Sun Jan 15 17:54:32 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 15 Jan 2012 16:54:32 +0000
Subject: [issue11694] xdrlib raises ConversionError in inconsistent way
In-Reply-To: <1301224336.99.0.411299821684.issue11694@psf.upfronthosting.co.za>
Message-ID: <1326646472.71.0.674741971666.issue11694@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
stage: test needed -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11694>
_______________________________________

From report at bugs.python.org  Sun Jan 15 17:56:26 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Sun, 15 Jan 2012 16:56:26 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326646586.06.0.132541491357.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24245/2b7bf4e5cb9f.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sun Jan 15 18:02:35 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Sun, 15 Jan 2012 17:02:35 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326646955.8.0.789135054157.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24246/1cdb64480494.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sun Jan 15 18:04:02 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Sun, 15 Jan 2012 17:04:02 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326647042.74.0.1589948706.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I've created yet another patch (1cdb64480494) which adds a regression test under Python 2.7 to demonstrate that the fix works (based on test_import_symlink_package.py).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sun Jan 15 18:04:11 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Sun, 15 Jan 2012 17:04:11 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326647051.21.0.071648580613.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I've created yet another patch (1cdb64480494) which adds a regression test under Python 2.7 to demonstrate that the fix works (based on test_import_symlink_package.py).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sun Jan 15 18:20:07 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 15 Jan 2012 17:20:07 +0000
Subject: [issue13521] Make dict.setdefault() atomic
In-Reply-To: <1326636090.29.0.839986828946.issue13521@psf.upfronthosting.co.za>
Message-ID: <1326647904.3396.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> No more double lookup.

Your patch doesn't check hashed2.eq_count.
Since the dict specification doesn't say on which instance __eq__ will
be called when doing a lookup, the patch should either check
``hashed1.eq_count + hashed2.eq_count``, or make eq_count a class
attribute.

A nit: be careful not to use tabs in C files.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13521>
_______________________________________

From report at bugs.python.org  Sun Jan 15 19:23:22 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 15 Jan 2012 18:23:22 +0000
Subject: [issue13789] _tkinter does not build on Windows 7
In-Reply-To: <1326603556.49.0.117233546142.issue13789@psf.upfronthosting.co.za>
Message-ID: <1326651802.87.0.0911494324693.issue13789@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

See PCbuild/readme.txt, and follow its instructions scrupulously. If you use a different build procedure, or different Tcl sources, you are on your own.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13789>
_______________________________________

From report at bugs.python.org  Sun Jan 15 19:40:44 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 15 Jan 2012 18:40:44 +0000
Subject: [issue13589] Aifc low level serialization primitives fix
In-Reply-To: <1323714686.93.0.781541085274.issue13589@psf.upfronthosting.co.za>
Message-ID: <1326652844.09.0.413661877628.issue13589@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

For the record, can you explain why infinity and NaN can end up in AIFC files?
That said, the patch looks ok to me.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13589>
_______________________________________

From report at bugs.python.org  Sun Jan 15 22:02:24 2012
From: report at bugs.python.org (Juan Javier)
Date: Sun, 15 Jan 2012 21:02:24 +0000
Subject: [issue13785] Make concurrent.futures.Future state public
In-Reply-To: <1326560835.55.0.282912980665.issue13785@psf.upfronthosting.co.za>
Message-ID: <1326661344.17.0.708068246784.issue13785@psf.upfronthosting.co.za>


Juan Javier <jjdominguezm at gmail.com> added the comment:

Hello,

You're right, explaining the difference between CANCELLED and CANCELLED_AND_NOTIFIED is gong to be hard and might be confusing. I also agree that there is no precedent for storing the history of something, and I don't like either the idea of having a futures factory (that was my first idea).

But, what about using callbacks? it is possible to add done callbacks, why can't we have a list of callbacks attached to each "public" state.

Something like:

Future.append_callback(self, state: "One of PENDING, RUNNING, CANCELLED, FINISHED", fn)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13785>
_______________________________________

From report at bugs.python.org  Sun Jan 15 23:46:03 2012
From: report at bugs.python.org (py.user)
Date: Sun, 15 Jan 2012 22:46:03 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326667563.92.0.782142630532.issue13790@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

also strange(unobvious) behavior:
>>> '{0:.3s}'.format((i for i in (1, 2, 3)))
'<ge'
>>> '{0:.3s}'.format(range(10))
'ran'
>>> '{0:.3s}'.format(None)
'Non'
>>>

it would be better to print an error:
ValueError: Unknown format code 's' for object of type 'generator'

like in this:
>>> '{0:d}'.format(4.5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'float'
>>>

in the documentation there is nothing about it

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sun Jan 15 23:54:16 2012
From: report at bugs.python.org (R. David Murray)
Date: Sun, 15 Jan 2012 22:54:16 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326668056.56.0.934515329859.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

No, it wouldn't.  I expect

  "{}".format(x)

to produce something for an arbitrary x.  Breaking that would break a fundamental Python contract.

Improving the error message for 'd' is more possible.  Perhaps "the format code 'd' is not implemented by objects of type <type>"?

----------
nosy: +r.david.murray

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:00:50 2012
From: report at bugs.python.org (R. David Murray)
Date: Sun, 15 Jan 2012 23:00:50 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326668450.29.0.650021644047.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Oh, and when you say there is nothing in the documentation about the 's' case for arbitrary objects, it is made clear in various places that every object has an str, which defaults to its repr if it has no specific __str__.  Combine that with the description of what happens when you use a fixed field length for 's', and you get the results you see.  There should be nothing surprising about this to anyone who has read the tutorial, I think.  (But specific suggestions for improving the docs are always welcome.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:04:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 15 Jan 2012 23:04:50 +0000
Subject: [issue13785] Make concurrent.futures.Future state public
In-Reply-To: <1326560835.55.0.282912980665.issue13785@psf.upfronthosting.co.za>
Message-ID: <1326668690.72.0.343029110531.issue13785@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> The idea is to have access to the current state of the Future using a property instead of calling several methods (done, cancelled, etc.).

I think one point of having methods is that querying is decoupled from implementation. The internal states could for example be finer-grained than what is exposed by the API.

> Also, a history property that returns a list of Event(state, timestamp) objects is written, the list stores the timestamp every time the state of a future changes.

Uh, what is the use case exactly?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13785>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:05:04 2012
From: report at bugs.python.org (Kay Hayen)
Date: Sun, 15 Jan 2012 23:05:04 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
Message-ID: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>


New submission from Kay Hayen <kayhayen at gmx.de>:

Hello,

I am the author of the Python compiler Nuitka. It has the ability to immediately execute the created executable file. For that I am using "os.execl" to immediately replace the compiler and run the freshly created binary instead.

This worked well so far, but as of late, I am also checking the exit codes, and it turns out that even for failing programs, the exit code is "0" on Windows, even though the compiled binary is exiting with "1".

Investigating further, I made a simple program:

-------
import os
os.execl( "FailingProgram.exe", "lala" )
-------

And it turns out, it's giving me "0", whereas when executed directly "FailingProgram.exe" gives "1". Checking %errorlevel% manually that is, my test framework uses "subprocess" module and gets "0".

The same code works fine (preserves exit code) under Linux. I didn't find the windows specific code responsible for implementing "os.execv" under Win32. I am suspecting that somehow "cmd.exe" may not be propagating the error code, but for that to confirm I would need pointers.

Thanks in advance,
Kay

----------
components: Library (Lib)
messages: 151304
nosy: kayhayen
priority: normal
severity: normal
status: open
title: The "os.execl" call doesn't give programs exit code
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:13:27 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 15 Jan 2012 23:13:27 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <1326669207.3.0.581903822038.issue13792@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Python's os.execl simply calls Windows' execv() function, which AFAIK has nothing to do with cmd.exe.

----------
nosy: +brian.curtin, pitrou, tim.golden

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:26:53 2012
From: report at bugs.python.org (py.user)
Date: Sun, 15 Jan 2012 23:26:53 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326670013.23.0.475182473087.issue13790@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

R. David Murray wrote:
> it is made clear in various places that every object has an str

here:
http://docs.python.org/py3k/library/string.html#format-specification-mini-language

3rd paragraph:
"A general convention is that an empty format string ("") produces the same result as if you had called str() on the value. A non-empty format string typically modifies the result."

"an empty format string ("")" what does it mean ?

"".format(value) or "{}".format(value) or "{0}".format(value) ?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:29:52 2012
From: report at bugs.python.org (py.user)
Date: Sun, 15 Jan 2012 23:29:52 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326670192.88.0.27867390147.issue13790@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

also here:
http://docs.python.org/py3k/library/string.html#format-examples

there is no example with list or tuple to know exactly how they are formatted

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 00:49:49 2012
From: report at bugs.python.org (R. David Murray)
Date: Sun, 15 Jan 2012 23:49:49 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326671389.04.0.998420710174.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

"an empty format string" is exactly what I was talking about.  Putting nothing between the {}'s is an empty format string.  I can't think of any way to make that wording clearer.

The format docs should not contains examples of the repr of all possible python objects.  The examples of what tuples and lists and dicts &c look like are shown in the docs for those objects.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 01:09:07 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 00:09:07 +0000
Subject: [issue13039] IDLE editor: shell-like behaviour on line starting with
	">>>"
In-Reply-To: <1316816819.73.0.347712729805.issue13039@psf.upfronthosting.co.za>
Message-ID: <E1Rma8F-00062i-4r@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 95b704cb7f7c by Terry Jan Reedy in branch '2.7':
#13039 allow proper deletion of '>>> ' in IDLE editor windows.
http://hg.python.org/cpython/rev/95b704cb7f7c

New changeset c6e7473b1fb5 by Terry Jan Reedy in branch '3.2':
#13039 allow proper deletion of '>>> ' in IDLE editor windows.
http://hg.python.org/cpython/rev/c6e7473b1fb5

New changeset 6099d9dd0c26 by Terry Jan Reedy in branch 'default':
Merge with 3.2 #13039 allow proper deletion of '>>> ' in IDLE editor windows.
http://hg.python.org/cpython/rev/6099d9dd0c26

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13039>
_______________________________________

From report at bugs.python.org  Mon Jan 16 01:14:49 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 00:14:49 +0000
Subject: [issue13039] IDLE editor: shell-like behaviour on line starting with
	">>>"
In-Reply-To: <1316816819.73.0.347712729805.issue13039@psf.upfronthosting.co.za>
Message-ID: <1326672889.07.0.9914689336.issue13039@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I tested with 3.2.2 on Win7. Deletes prompt in editor window with both backspace and delete keys, but not prompt in shell. EditorWindow.py is identical in all three branches, at least in this area, so pushed. One nuisance gone. Thanks Roger.

----------
assignee:  -> terry.reedy
resolution:  -> fixed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13039>
_______________________________________

From report at bugs.python.org  Mon Jan 16 01:15:36 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 00:15:36 +0000
Subject: [issue13039] IDLE editor: shell-like behaviour on line starting with
	">>>"
In-Reply-To: <1316816819.73.0.347712729805.issue13039@psf.upfronthosting.co.za>
Message-ID: <1326672936.25.0.890852202886.issue13039@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13039>
_______________________________________

From report at bugs.python.org  Mon Jan 16 01:24:47 2012
From: report at bugs.python.org (Jim Jewett)
Date: Mon, 16 Jan 2012 00:24:47 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <1326673487.26.0.947477273583.issue12736@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Why was the delta-processing removed from the casing functions?

As best I can tell, the whole point of going through multiple levels of indirection (courtesy splitbins) is to maximize compression and minimize the amount of cache that unicode might occupy.

By using deltas, only one record is needed for each combination of (upper - lower, upper - title), which is generally only one or two combinations per script.  

Without deltas, nearly every cased letter needs its own record, and the index tables also get bigger. (It seems to be about 2.6 times as large, but cache effects may be worse, since letters from the same script will no longer be in the same record or the same index chain.)

If it is a concern about not enough room for flags, then the decimal/digit chars could be combined.  They are always the same, unless the number isn't decimal (in which case the flag is enough).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Mon Jan 16 01:58:50 2012
From: report at bugs.python.org (Berker Peksag)
Date: Mon, 16 Jan 2012 00:58:50 +0000
Subject: [issue13588] Change name of internal closure functions in importlib
In-Reply-To: <1323703954.36.0.434643027987.issue13588@psf.upfronthosting.co.za>
Message-ID: <1326675530.38.0.844012535759.issue13588@psf.upfronthosting.co.za>


Berker Peksag <berker.peksag at gmail.com> added the comment:

Hi Brett, did you have a chance to review the patch I submitted?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13588>
_______________________________________

From report at bugs.python.org  Mon Jan 16 02:34:27 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 01:34:27 +0000
Subject: [issue13629] _PyParser_TokenNames does not match up with the token.h
	numbers
In-Reply-To: <1324232482.43.0.0474353637736.issue13629@psf.upfronthosting.co.za>
Message-ID: <E1RmbSp-0002gu-Vo@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1c0c6fa7341c by Meador Inge in branch '3.2':
Issue #13629: Renumber the tokens in token.h to match the _PyParser_TokenNames indexes.
http://hg.python.org/cpython/rev/1c0c6fa7341c

New changeset c0660d7cc1fe by Meador Inge in branch 'default':
Issue #13629: Renumber the tokens in token.h to match the _PyParser_TokenNames indexes.
http://hg.python.org/cpython/rev/c0660d7cc1fe

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13629>
_______________________________________

From report at bugs.python.org  Mon Jan 16 02:35:52 2012
From: report at bugs.python.org (Meador Inge)
Date: Mon, 16 Jan 2012 01:35:52 +0000
Subject: [issue13629] _PyParser_TokenNames does not match up with the token.h
	numbers
In-Reply-To: <1324232482.43.0.0474353637736.issue13629@psf.upfronthosting.co.za>
Message-ID: <1326677752.95.0.389223333861.issue13629@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13629>
_______________________________________

From report at bugs.python.org  Mon Jan 16 03:19:31 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 02:19:31 +0000
Subject: [issue12736] Request for python casemapping functions to use full not
	simple casemaps per Unicode's recommendation
In-Reply-To: <1313098784.96.0.996919907442.issue12736@psf.upfronthosting.co.za>
Message-ID: <E1RmcAU-0005bt-Dv@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 03ea95e3b497 by Benjamin Peterson in branch 'default':
delta encoding of upper/lower/title makes a glorious return (#12736)
http://hg.python.org/cpython/rev/03ea95e3b497

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12736>
_______________________________________

From report at bugs.python.org  Mon Jan 16 03:28:55 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 02:28:55 +0000
Subject: [issue13766] explain the relationship between Lib/lib2to3/Grammar.txt
	and Grammar/Grammar
In-Reply-To: <1326279554.15.0.713951578682.issue13766@psf.upfronthosting.co.za>
Message-ID: <E1RmcJZ-0006EB-VG@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8f7e707ab729 by Benjamin Peterson in branch '3.2':
explain why we need this grammar file (closes #13766)
http://hg.python.org/cpython/rev/8f7e707ab729

New changeset ecc998ec8c6f by Benjamin Peterson in branch 'default':
merge 3.2 (#13766)
http://hg.python.org/cpython/rev/ecc998ec8c6f

New changeset 5d1c177c585b by Benjamin Peterson in branch '2.7':
explain why we need this grammar file (closes #13766)
http://hg.python.org/cpython/rev/5d1c177c585b

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13766>
_______________________________________

From report at bugs.python.org  Mon Jan 16 05:25:48 2012
From: report at bugs.python.org (py.user)
Date: Mon, 16 Jan 2012 04:25:48 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326687948.4.0.191093126307.issue13790@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

R. David Murray wrote:
> Putting nothing between the {}'s is an empty format string.

this is an empty replacement field

here:
http://docs.python.org/py3k/library/string.html#format-string-syntax

the definition of format string:
"Format strings contain ?replacement fields? surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output."

"The grammar for a replacement field is as follows:"
replacement_field ::=  "{" [field_name] ["!" conversion] [":" format_spec] "}"

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 05:39:06 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 04:39:06 +0000
Subject: =?utf-8?q?=5Bissue13791=5D_Reword_=E2=80=9COld_versions=E2=80=9D_in_the_d?=
	=?utf-8?q?oc_sidebar?=
In-Reply-To: <1326618413.55.0.73666208075.issue13791@psf.upfronthosting.co.za>
Message-ID: <1326688746.3.0.756838284261.issue13791@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

To avoid repetition (?Docs for other versions? ? ?Other versions?), I?d propose changing ?Old versions? to ?All versions?.

----------
nosy: +eric.araujo, ezio.melotti, sandro.tosi, terry.reedy
title: Other versions -> Reword ?Old versions? in the doc sidebar
versions:  -Python 2.6, Python 3.1, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13791>
_______________________________________

From report at bugs.python.org  Mon Jan 16 05:50:43 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 04:50:43 +0000
Subject: =?utf-8?q?=5Bissue13791=5D_Reword_=E2=80=9COld_versions=E2=80=9D_in_the_d?=
	=?utf-8?q?oc_sidebar?=
In-Reply-To: <1326618413.55.0.73666208075.issue13791@psf.upfronthosting.co.za>
Message-ID: <1326689443.14.0.132961932376.issue13791@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I believe this should be closed in favor of #13122. The sidebar needs more change than just this one thing. See my proposal for a version-independent sidebar in msg150907, which Georg B. approved of.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13791>
_______________________________________

From report at bugs.python.org  Mon Jan 16 05:56:42 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 04:56:42 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326689802.75.0.21635275736.issue12705@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

IMO this is a bug and should be fixed in stable versions too.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:05:25 2012
From: report at bugs.python.org (Jim Jewett)
Date: Mon, 16 Jan 2012 05:05:25 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
Message-ID: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

The documentation for hasattr, getattr, and delattr state that they are equivalent to object.attribute access; this isn't quite true, because object.attribute uses a NFKC-normalized version of the string as only the secondary location, while hasattr, getattr, and delattr (assuming an object rather than an Identifier or string) don't seem to do the normalization at all.

I think the simplest fix would be to normalize and retry when hasattr, getattr, and delattr fail with a string, but I'm not sure that normalization shouldn't be the only string tried. 

>>> o.?
Traceback (most recent call last):
  File "<pyshell#820>", line 1, in <module>
    o.?
AttributeError: 'Object' object has no attribute 'o'
>>> o.o
Traceback (most recent call last):
  File "<pyshell#821>", line 1, in <module>
    o.o
AttributeError: 'Object' object has no attribute 'o'
>>> o.?=[]
>>> hasattr(o, "?")
False
>>> getattr(o, "?")
Traceback (most recent call last):
  File "<pyshell#824>", line 1, in <module>
    getattr(o, "?")
AttributeError: 'Object' object has no attribute '?'
>>> delattr(o, "?")
Traceback (most recent call last):
  File "<pyshell#825>", line 1, in <module>
    delattr(o, "?")
AttributeError: ?
>>> o.?
[]
>>> o.? is o.o
True
>>> o.o
[]
>>> del o.?
>>> o.o
Traceback (most recent call last):
  File "<pyshell#830>", line 1, in <module>
    o.o
AttributeError: 'Object' object has no attribute 'o'

>>> o.? = 5
>>> hasattr(o, "?")
False
>>> hasattr(o, "o")
True
>>> hasattr(o, "o")
True
>>> o.?
5
>>> delattr(o, "o")
>>> o.?

----------
components: Unicode
messages: 151320
nosy: Jim.Jewett, ezio.melotti
priority: normal
severity: normal
status: open
title: hasattr, delattr, getattr fail with unnormalized names

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:21:44 2012
From: report at bugs.python.org (AJ)
Date: Mon, 16 Jan 2012 05:21:44 +0000
Subject: [issue13794] Copyright Year - Chnage it to 2012 please
Message-ID: <1326691304.18.0.453476801273.issue13794@psf.upfronthosting.co.za>


New submission from AJ <amanjeev at gmail.com>:

Copyright in the footer on *.python.org says -2011. I believe this can be changed to 2012?

----------
components: None
messages: 151321
nosy: amanjeev
priority: normal
severity: normal
status: open
title: Copyright Year - Chnage it to 2012 please

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13794>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:32:11 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:32:11 +0000
Subject: [issue6528] builtins colored as keyword at beginning of line
In-Reply-To: <1248125971.41.0.191814152476.issue6528@psf.upfronthosting.co.za>
Message-ID: <1326691931.74.0.984275189567.issue6528@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +terry.reedy
priority: low -> normal
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6528>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:33:20 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:33:20 +0000
Subject: [issue13078] IDLE: Python Crashes When Saving Or Opening
In-Reply-To: <1317404515.12.0.258020695982.issue13078@psf.upfronthosting.co.za>
Message-ID: <1326692000.34.0.88806063354.issue13078@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13078>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:39:38 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:39:38 +0000
Subject: [issue13052] IDLE: replace ending with '\' causes crash
In-Reply-To: <1317177843.04.0.691154940707.issue13052@psf.upfronthosting.co.za>
Message-ID: <1326692378.95.0.0815302312148.issue13052@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

How difficult would it be to add tests for this?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13052>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:42:46 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:42:46 +0000
Subject: [issue2134] Add new attribute to TokenInfo to report specific token
	IDs
In-Reply-To: <1203289230.78.0.790200923315.issue2134@psf.upfronthosting.co.za>
Message-ID: <1326692566.15.0.900750388436.issue2134@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
stage: needs patch -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:44:14 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:44:14 +0000
Subject: [issue13465] A Jython section in the dev guide would be great
In-Reply-To: <1322074015.2.0.311635199294.issue13465@psf.upfronthosting.co.za>
Message-ID: <1326692654.52.0.572998294271.issue13465@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
stage:  -> committed/rejected

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13465>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:45:51 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 05:45:51 +0000
Subject: =?utf-8?q?=5Bissue13791=5D_Reword_=E2=80=9COld_versions=E2=80=9D_in_the_d?=
	=?utf-8?q?oc_sidebar?=
In-Reply-To: <1326618413.55.0.73666208075.issue13791@psf.upfronthosting.co.za>
Message-ID: <1326692751.38.0.0539353364306.issue13791@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Agreed (I?d just suggest finding something better than ?Everything else? :-)

----------
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> Out of date links in the sidebar of the documentation index of versions 3.1 and 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13791>
_______________________________________

From report at bugs.python.org  Mon Jan 16 06:53:50 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 05:53:50 +0000
Subject: [issue13794] Copyright Year - Change it to 2012 please
In-Reply-To: <1326691304.18.0.453476801273.issue13794@psf.upfronthosting.co.za>
Message-ID: <1326693230.05.0.569157955859.issue13794@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
assignee:  -> ezio.melotti
components:  -None
nosy: +ezio.melotti
stage:  -> needs patch
title: Copyright Year - Chnage it to 2012 please -> Copyright Year - Change it to 2012 please
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13794>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:00:52 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 06:00:52 +0000
Subject: [issue13665] TypeError: string or integer address expected instead of
	str instance
In-Reply-To: <1324921834.92.0.644281007494.issue13665@psf.upfronthosting.co.za>
Message-ID: <1326693652.59.0.646557535654.issue13665@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Here's a patch.
I found a similar instance that says "unicode string or integer address expected instead of %s instance", but it's inside an "if (!PyUnicode_Check(value)) {}" so it should be ok ("unicode string" could be replaced to 'str' though).

----------
keywords: +patch
stage: needs patch -> commit review
Added file: http://bugs.python.org/file24247/issue13665.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13665>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:08:49 2012
From: report at bugs.python.org (Kay Hayen)
Date: Mon, 16 Jan 2012 06:08:49 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <1326694129.84.0.0913527023733.issue13792@psf.upfronthosting.co.za>


Kay Hayen <kayhayen at gmx.de> added the comment:

Well, I saw that code, but expected that there must be more to it. But I found out, the bug is actually caused by at least MinGW. See below how I build a program with it, that does "execl" on an error exiting program and then the "errorlevel" variable is "0", whereas direct execution gives "1".

I don't have MSVC installed, so I cannot tell if it is affected as well. I will report this as a bug to MinGW then.

c:\Users\hayen\Nuitka>gcc -v
Es werden eingebaute Spezifikationen verwendet.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=c:/mingw/lib/../libexec/gcc/mingw32/4.6.2/lto-wrapper.exe
Ziel: mingw32
Konfiguriert mit: ../gcc-4.6.2/configure --enable-languages=c,c++,ada,fortran,ob
jc,obj-c++ --disable-sjlj-exceptions --with-dwarf2 --enable-shared --enable-libg
omp --disable-win32-registry --enable-libstdcxx-debug --enable-version-specific-
runtime-libs --build=mingw32 --prefix=/mingw
Thread-Modell: win32
gcc-Version 4.6.2 (GCC)

c:\Users\hayen\Nuitka>gcc exec_sample.cpp

c:\Users\hayen\Nuitka>type exec_sample.cpp

#include <unistd.h>
#include <stdio.h>


int main()
{
    puts( "Hello bad world!" );

    execl( "badprogram.exe", "badprogram", "what" );

    puts( "Look, this is not happening!" );
    return 2;
}


c:\Users\hayen\Nuitka>.\a.exe
Hello bad world!

c:\Users\hayen\Nuitka>Traceback (most recent call last):
  File "tests\syntax\RelativeNonPackageImport.py", line 20, in <module>
    from . import whatever
ValueError: Attempted relative import in non-package

c:\Users\hayen\Nuitka>echo %errorlevel%
0

c:\Users\hayen\Nuitka>.\badprogram.exe
Traceback (most recent call last):
  File "tests\syntax\RelativeNonPackageImport.py", line 20, in <module>
    from . import whatever
ValueError: Attempted relative import in non-package

c:\Users\hayen\Nuitka>echo %errorlevel%
1

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:29:52 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 06:29:52 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <E1Rmg4k-0006nX-Rp@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2dd96cf324ee by Ezio Melotti in branch '2.7':
#13695: fix a couple of typos in the doc.
http://hg.python.org/cpython/rev/2dd96cf324ee

New changeset bff9ab281385 by Ezio Melotti in branch '3.2':
#13695: fix a couple of typos in the doc.
http://hg.python.org/cpython/rev/bff9ab281385

New changeset 3b097e0892cf by Ezio Melotti in branch 'default':
#13695: merge with 3.2.
http://hg.python.org/cpython/rev/3b097e0892cf

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:32:04 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 06:32:04 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326695524.24.0.88195008216.issue13695@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

I fixed both 'type-specific' and 'Python 3'.  The python-distribute link is not part of our documentation so I can't fix it.

----------
assignee: docs at python -> ezio.melotti
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:36:29 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 06:36:29 +0000
Subject: [issue13443] wrong links and examples in the functional HOWTO
In-Reply-To: <1321850404.45.0.767133152656.issue13443@psf.upfronthosting.co.za>
Message-ID: <1326695789.6.0.790328187749.issue13443@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150397

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13443>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:37:04 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 06:37:04 +0000
Subject: [issue13443] wrong links and examples in the functional HOWTO
In-Reply-To: <1321850404.45.0.767133152656.issue13443@psf.upfronthosting.co.za>
Message-ID: <1326695824.38.0.335548816713.issue13443@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
Removed message: http://bugs.python.org/msg150399

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13443>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:37:58 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 06:37:58 +0000
Subject: [issue13789] _tkinter does not build on Windows 7
In-Reply-To: <1326603556.49.0.117233546142.issue13789@psf.upfronthosting.co.za>
Message-ID: <1326695878.18.0.108610243264.issue13789@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I have read readme.txt, more than once. It seems a bit out-of-date. Parts of it I have trouble understanding, or at least, being sure I have understood. It seems to disagree with the devguide about whether to build debug or release builds. I though of opening an issue for possible readme.txt revisions.

However, my goal at the moment is to be able to test patches to IDLE .py files in repository builds on Win7, with the least fuss necessary. I have several questions:

About _tkinter, readme.txt says:
"Wraps the Tk windowing system.  Unlike _sqlite3, there's no    corresponding tcltk.vcproj-type project that builds Tcl/Tk from    vcproj's within our pcbuild.sln, which means this module expects to find a pre-built Tcl/Tk in either ..\..\tcltk for 32-bit or   ..\..\tcltk64 for 64-bit (relative to this directory).  See below for instructions to build Tcl/Tk."

I interpret 'this module expects to find' to mean 'after _tkinter is 
built'. If building _tkinter requires tcltk first, it should says so. (And I did try to build it, see below.)

I interpret 'this directory' to mean, on my system, '.../dev/cpython/pcbuild'. I therefore interpret '../../tcltk' to mean '.../dev/tcltk'. Experiments in Command Prompt support this.

I interpret 'pre-built Tcl/Tk' to include the pre-built Tcl/Tk already installed on my system, just as it does on *nix. I tried copying the contents of /tcl and dlls/t*.pyd from installed 3.2.2 into dev/tcltk, but that does not work. Is there any way to use the existing tcl/tk build?

As for rebuilding, the readme says "run either external.bat or 
external-amd64.bat in the ..\Tools\buildbot directory from ..\, i.e.:

    C:\..\svn.python.org\projects\python\trunk\PCbuild>cd ..
    C:\..\svn.python.org\projects\python\trunk>Tools\buildbot\external.bat"

That exact file builds 8.5.2 instead of 8.5.9, so the above could be updated to refer to the current hg version: 'whatever/cpython> Tools...'.

external.bat runs external-common.bat. That brings all the external dependencies, which is more than I need or really want. Can I comment out the other dependencies if I want?

In any case, the file has near the top '''
@rem XXX: If you need to force the buildbots to start from a fresh environment, uncomment
@rem the following, check it in, then check it out, comment it out, then check it back in.
@rem if exist tcltk rd /s/q tcltk
@rem if exist tcltk64 rd /s/q tcltk64
'''
I do not really understand this. Does 'rd' mean 'remove directory? Can I just remove the tcltk directory I made and ignore this?

Each dependency has a line like
"   svn export http://svn.python.org/projects/external/tcl-8.5.9.0"
Can I use hg export or do I need to find and install (tortoise)svn?
Are the svn urls still correct or is there an hg equivalent?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13789>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:46:19 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 06:46:19 +0000
Subject: [issue13715] typo in unicodedata documentation
In-Reply-To: <1325782041.56.0.783502480563.issue13715@psf.upfronthosting.co.za>
Message-ID: <E1RmgKg-00081d-DJ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b4dac315feef by Ezio Melotti in branch '2.7':
#13715: fix typo in unicodedata doc.  Patch by Eli Collins.
http://hg.python.org/cpython/rev/b4dac315feef

New changeset f50ff6dd6b41 by Ezio Melotti in branch '3.2':
#13715: fix typo in unicodedata doc.  Patch by Eli Collins.
http://hg.python.org/cpython/rev/f50ff6dd6b41

New changeset f1408e41e306 by Ezio Melotti in branch 'default':
#13715: merge with 3.2.
http://hg.python.org/cpython/rev/f1408e41e306

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13715>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:47:31 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 06:47:31 +0000
Subject: [issue13715] typo in unicodedata documentation
In-Reply-To: <1325782041.56.0.783502480563.issue13715@psf.upfronthosting.co.za>
Message-ID: <1326696451.33.0.561937491971.issue13715@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Fixed, thanks for the report and the patch!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13715>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:52:19 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 06:52:19 +0000
Subject: [issue13443] wrong links and examples in the functional HOWTO
In-Reply-To: <1321850404.45.0.767133152656.issue13443@psf.upfronthosting.co.za>
Message-ID: <1326696739.78.0.500023966133.issue13443@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Sorry, my messages could be confusing.  What I meant is: While I think that Antoine?s doc changes were clear improvements, in the specific case of the functional howto there were good arguments in favor of keeping the part about the functional module.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13443>
_______________________________________

From report at bugs.python.org  Mon Jan 16 07:57:20 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 16 Jan 2012 06:57:20 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326697040.72.0.956820988554.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

There's also one more typo in the doc for cmp_to_key() function. Fix the first sentence "Transform an old-style comparison function to a key-function." to "Transform an old-style comparison function to a key function." (note the removal of the hyphen between words "key" and "function")

http://docs.python.org/library/functools.html#functools.cmp_to_key

----------
resolution: fixed -> remind
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:06:11 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 16 Jan 2012 07:06:11 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326697571.13.0.98341992147.issue13695@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
assignee: ezio.melotti -> rhettinger
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:12:53 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 07:12:53 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <1326697973.26.0.962849753388.issue11906@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

This was committed on py3k in 4f8c24830a5c. Terry, can the issue be closed?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:16:44 2012
From: report at bugs.python.org (Pedro Andres Aranda Gutierrez)
Date: Mon, 16 Jan 2012 07:16:44 +0000
Subject: [issue13795] CDATA Element missing
Message-ID: <1326698204.07.0.970512725442.issue13795@psf.upfronthosting.co.za>


New submission from Pedro Andres Aranda Gutierrez <paaguti at gmail.com>:

When creating ElementTree objects that hold SVG drawings, I need a CDATA object similar to the ProcessingInstruction object. There was a circumvention of the problem for Python 2.6:

http://stackoverflow.com/questions/174890/how-to-output-cdata-using-elementtree

This workaround doesn't work in Python 2.7 and above

----------
messages: 151334
nosy: paaguti
priority: normal
severity: normal
status: open
title: CDATA Element missing
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13795>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:20:36 2012
From: report at bugs.python.org (jbitcm-)
Date: Mon, 16 Jan 2012 07:20:36 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1326698436.61.0.973555602684.issue13691@psf.upfronthosting.co.za>


jbitcm- <chakrapackager at gmail.com> added the comment:

I am working on it

----------
nosy: +jbitcm-

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:21:27 2012
From: report at bugs.python.org (Pedro Andres Aranda Gutierrez)
Date: Mon, 16 Jan 2012 07:21:27 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
Message-ID: <1326698487.2.0.0622950382199.issue13796@psf.upfronthosting.co.za>


New submission from Pedro Andres Aranda Gutierrez <paaguti at gmail.com>:

I have extended the xml.etree.ElementTree.Element class and pass the text attribute in the arguments. This creates much more compact code:

import xml.etree.ElementTree as xml
        
        
class Element(xml.Element):
    def __init__(self,tag,attrib={},**attrs):
        super(xml.Element,self).__init__()
        self.tag = tag
        self.attrib = attrib
        self.attrib.update(attrs)
        self.text = self.attrib.pop('text',None)
        self.tail = self.attrib.pop('tail',None)
        self._children = []

if __name__ == '__main__':
    from sys import stdout

    test = Element('Hello',)
    test2 = Element('World',{'humour':'excelent'},text = 'How do you do', tail="Fine")
    test.append(test2)    
    xml.ElementTree(test).write(stdout,encoding="utf-8",xml_declaration="yes",method="xml")

----------
messages: 151336
nosy: paaguti
priority: normal
severity: normal
status: open
title: use 'text=...' to define the text attribute of and xml.etree.ElementTree.Element
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:31:48 2012
From: report at bugs.python.org (Mark Diekhans)
Date: Mon, 16 Jan 2012 07:31:48 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326699108.79.0.0187448644639.issue12600@psf.upfronthosting.co.za>


Mark Diekhans <markd at kermodei.com> added the comment:

The lack of the ability to pass a parameter to a test case is a very
frustrating restriction with unittest. The frequent need if for a database
connection for testing database related classes.

Yes, there are lots of other ways to work around it, but they tend to involved need to understand and subclass several pieces of the unittest
framework.

An enthusiastic yes on this.

----------
nosy: +diekhans

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 08:52:14 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 07:52:14 +0000
Subject: [issue13275] Recommend xml.etree for XML processing
In-Reply-To: <1319709959.84.0.984784684468.issue13275@psf.upfronthosting.co.za>
Message-ID: <1326700334.37.0.0570524650088.issue13275@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

(I haven?t found an elegant XML lib, and have not yet needed to look for one as I touch very little XML.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13275>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:23:04 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 08:23:04 +0000
Subject: [issue6528] builtins colored as keyword at beginning of line
In-Reply-To: <1248125971.41.0.191814152476.issue6528@psf.upfronthosting.co.za>
Message-ID: <E1RmhqI-0006dL-Tf@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2d4ce2cdd6d4 by Terry Jan Reedy in branch '3.2':
#6528 None, True, False are keywords in 3.x. Patch by Roger Serwy.
http://hg.python.org/cpython/rev/2d4ce2cdd6d4

New changeset 522f07bfa067 by Terry Jan Reedy in branch 'default':
Merge with 3.2 #6528 None, True, False are keywords in 3.x. Patch by R. Serwy.
http://hg.python.org/cpython/rev/522f07bfa067

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6528>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:25:37 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 08:25:37 +0000
Subject: [issue6528] builtins colored as keyword at beginning of line
In-Reply-To: <1248125971.41.0.191814152476.issue6528@psf.upfronthosting.co.za>
Message-ID: <1326702337.36.0.372435287057.issue6528@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I just realized that there is 'another' reason not to reuse builtin names (other than those usually given): they will be colorized as builtins even if they have no relation to the builtin. Make_pat uses the function any() defined just above (before the builtin, I am sure), so 'any' is colored as if it were the builtin. Within ColorDelegator.py, it is only used within make_pat, so I considered renaming it to anyp or any_pat, but it might be used by some other module that imports ColorDelegator. Another issue.

"name not in keyword.kwlist" could be "name not in {None, True, False}", but those happen to be the first three items in the kwlist, so the time difference would be nil.

Looks good. Tested on 3.2.2 Win7. Applied.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6528>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:38:59 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 08:38:59 +0000
Subject: [issue13052] IDLE: replace ending with '\' causes crash
In-Reply-To: <1317177843.04.0.691154940707.issue13052@psf.upfronthosting.co.za>
Message-ID: <1326703139.26.0.495893206206.issue13052@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

If there are any tests for Idle, I have not found them. No idlelib/test/, no test/test_idle. There is a testcode.py with random code. I want to look at tkinter/test/ someday to see if it has any models for testing Idle. Roger, do you know of anything?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13052>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:40:06 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 08:40:06 +0000
Subject: [issue13435] Copybutton does not hide tracebacks
In-Reply-To: <1321724543.66.0.854276029391.issue13435@psf.upfronthosting.co.za>
Message-ID: <1326703206.15.0.040454295725.issue13435@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Now 2.7 uses Sphinx 1.0, but I think a full rebuild should be triggered in order to update all the files.
Georg, can you do something about it?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13435>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:42:17 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 08:42:17 +0000
Subject: [issue13052] IDLE: replace ending with '\' causes crash
In-Reply-To: <1317177843.04.0.691154940707.issue13052@psf.upfronthosting.co.za>
Message-ID: <1326703337.66.0.40988975504.issue13052@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

If the patch works for you, I guess you can commit it; if tests can be added too, that's even better.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13052>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:51:44 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 16 Jan 2012 08:51:44 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326703904.22.0.323036366874.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Found one little bug again in http://docs.python.org/library/stdtypes.html#sequence-types-str-unicode-list-tuple-bytearray-buffer-xrange

Look at the table with the fields Operation, Result, Notes, and fix the fifth record of the Result field from "i'th item of s, origin 0" to "i-th item of s, origin 0".

So change only that apostrophe into a hyphen, so "i'th" to "i-th".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:54:29 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 08:54:29 +0000
Subject: [issue6528] builtins colored as keyword at beginning of line
In-Reply-To: <1248125971.41.0.191814152476.issue6528@psf.upfronthosting.co.za>
Message-ID: <1326704069.93.0.0268518542989.issue6528@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
assignee:  -> terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6528>
_______________________________________

From report at bugs.python.org  Mon Jan 16 09:55:42 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 08:55:42 +0000
Subject: [issue2292] Missing *-unpacking generalizations
In-Reply-To: <1205595680.3.0.859525264867.issue2292@psf.upfronthosting.co.za>
Message-ID: <1326704142.63.0.344432888456.issue2292@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2292>
_______________________________________

From report at bugs.python.org  Mon Jan 16 10:09:36 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 09:09:36 +0000
Subject: [issue11906] test_argparse failure in interactive mode
In-Reply-To: <1303442771.01.0.0867760921949.issue11906@psf.upfronthosting.co.za>
Message-ID: <1326704976.56.0.629556713871.issue11906@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11906>
_______________________________________

From report at bugs.python.org  Mon Jan 16 10:17:54 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 09:17:54 +0000
Subject: [issue13122] Out of date links in the sidebar of the documentation
	index of versions 3.1 and 3.2
In-Reply-To: <1317999740.46.0.922190010194.issue13122@psf.upfronthosting.co.za>
Message-ID: <1326705474.75.0.713539920992.issue13122@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

'All Releases' rather than 'Everything else' is both more accurate and to me, more graceful.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13122>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:12:02 2012
From: report at bugs.python.org (Luis Marsano)
Date: Mon, 16 Jan 2012 10:12:02 +0000
Subject: [issue13756] Python3.2.2 make fail on cygwin
In-Reply-To: <1326199459.04.0.575648132034.issue13756@psf.upfronthosting.co.za>
Message-ID: <1326708722.3.0.634893071348.issue13756@psf.upfronthosting.co.za>


Luis Marsano <luis.marsano at gmail.com> added the comment:

The README file implies support:
[?]
Build Instructions
------------------

On Unix, Linux, BSD, OSX, and Cygwin:
[?]

----------
components: +Build -Installation
nosy: +Luis.Marsano

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13756>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:29:11 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 10:29:11 +0000
Subject: [issue13797] Add a __buffer__ special method to retrieve a PEP 3118
	object
Message-ID: <1326709751.32.0.965094376059.issue13797@psf.upfronthosting.co.za>


New submission from Nick Coghlan <ncoghlan at gmail.com>:

Currently, there's no straightforward way to create new classes in Python that delegate the PEP 3118 buffer APIs to another object that supports them (e.g. bytes, bytearray, memoryview, array.array, a NumPy array).

I see a few possible ways to go about this:

1. Add a single new special method, __buffer__. This creates an entry in the underlying getbuffer slot, but leaves releasebuffer empty.

The getbuffer wrapper then calls __buffer__ and delegates the getbuffer call to the returned object.

Any created Py_buffer instances refer directly to the underlying PEP 3118 supporting object, not the original object that defines __buffer__

The downside of this approach is that you can't implement a completely new PEP 3118 supporting object in pure Python (since you don't get access to the flags)

2. As above, but any Py_buffer structs returned are updated to refer to the original object, not the underlying one. A releasebuffer wrapper is also created, that calls back into __buffer__ to retrieve the object again. This approach has problems if __buffer__ can change to refer to different objects over time.

3. Define separate __getbuffer__ and __releasebuffer__ special methods. All C types implementing the PEP 3118 slots get these special methods automatically (as with any other slot). The flag variables are passed into __getbuffer__, and it is expected to return a memoryview object defining the view. The underlying getbuffer slot wrapper stores the current object in the Py_buffer "obj" slot, and the returned memoryview in "internal". The rest of the Py_buffer fields are populated based on the memoryview contents.

The releasebuffer wrapper then reverses this process, by passing the memoryview instance from the internal slot into the __releasebuffer__ call.

----------
components: Interpreter Core
messages: 151347
nosy: ncoghlan, pitrou, skrah
priority: normal
severity: normal
status: open
title: Add a __buffer__ special method to retrieve a PEP 3118 object
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13797>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:29:30 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 10:29:30 +0000
Subject: [issue13797] Add a Python level special method to retrieve a PEP 3118
	object
In-Reply-To: <1326709751.32.0.965094376059.issue13797@psf.upfronthosting.co.za>
Message-ID: <1326709770.16.0.573571792995.issue13797@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


----------
dependencies: +Problems with Py_buffer management in memoryobject.c (and elsewhere?)
title: Add a __buffer__ special method to retrieve a PEP 3118 object -> Add a Python level special method to retrieve a PEP 3118 object

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13797>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:30:14 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 10:30:14 +0000
Subject: [issue13797] Allow objects implemented in pure Python to export PEP
	3118 buffers
In-Reply-To: <1326709751.32.0.965094376059.issue13797@psf.upfronthosting.co.za>
Message-ID: <1326709814.26.0.954979663914.issue13797@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


----------
title: Add a Python level special method to retrieve a PEP 3118 object -> Allow objects implemented in pure Python to export PEP 3118 buffers

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13797>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:38:26 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 16 Jan 2012 10:38:26 +0000
Subject: [issue13435] Copybutton does not hide tracebacks
In-Reply-To: <1321724543.66.0.854276029391.issue13435@psf.upfronthosting.co.za>
Message-ID: <1326710306.96.0.368118762257.issue13435@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

A full rebuild should be done anyway next time the script runs.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13435>
_______________________________________

From report at bugs.python.org  Mon Jan 16 11:39:10 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 16 Jan 2012 10:39:10 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326710350.13.0.650175731174.issue13695@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Would you care to explain why that change needs to be made?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 12:14:10 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Mon, 16 Jan 2012 11:14:10 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
In-Reply-To: <1326698487.2.0.0622950382199.issue13796@psf.upfronthosting.co.za>
Message-ID: <1326712450.91.0.336594921639.issue13796@psf.upfronthosting.co.za>


patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:

I agree the Element syntax is sometimes awkward.

But how would you represent text or tail attributes within this enhanced element?
<animal name="cat" tail="yes"> comes to mind ...

----------
nosy: +patrick.vrijlandt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Mon Jan 16 12:47:46 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 11:47:46 +0000
Subject: [issue13797] Allow objects implemented in pure Python to export PEP
	3118 buffers
In-Reply-To: <1326709751.32.0.965094376059.issue13797@psf.upfronthosting.co.za>
Message-ID: <1326714466.61.0.307345341911.issue13797@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Reviewing Stefan's work on #10181 suggests to me that option 3 is the only feasible approach.

(Allowing memoryview subclasses will permit types to pass their own state between __getbuffer__ and __releasebuffer__)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13797>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:22:51 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 12:22:51 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1326716571.88.0.358826772709.issue13793@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +benjamin.peterson
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:23:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 12:23:25 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326716605.32.0.205734212454.issue12705@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +benjamin.peterson
stage: test needed -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:44:44 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Mon, 16 Jan 2012 12:44:44 +0000
Subject: [issue13798] Pasting and then running code doesn't work in the IDLE
	Shell
Message-ID: <1326717884.33.0.870144380911.issue13798@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

If you paste code with multiple lines which are not in a block of code into the IDLE Shell and press enter to run the code, only the first line of the code gets executed and the rest of the code is ignored.

You can even put garbage such as "$$$$" in the lines after the first line.

----------
components: IDLE
messages: 151352
nosy: ramchandra.apte
priority: normal
severity: normal
status: open
title: Pasting and then running code doesn't work in the IDLE Shell
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13798>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:45:17 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 16 Jan 2012 12:45:17 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326449848.75.0.36679779906.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwYM=Wi1fnfh3opoiUsZzLoMna2Z35kbudk2rtdCF3x-Lw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

The vulnerability is known since 2003 (Usenix 2003): read "Denial of
Service via Algorithmic Complexity Attacks" by Scott A. Crosby and Dan
S. Wallach.
http://www.cs.rice.edu/~scrosby/hash/CrosbyWallach_UsenixSec2003.pdf

This paper compares Perl 5.8 hash function, MD5, UHASH (UMAC
universal), CW (Carter-Wegman) and XOR12. Read more about UMAC:
http://en.wikipedia.org/wiki/UMAC
"A UMAC has provable cryptographic strength and is usually a lot less
computationally intensive than other MACs."

oCERT advisory #2011-003: multiple implementations denial-of-service
via hash algorithm collision
http://www.ocert.org/advisories/ocert-2011-003.html

nRuns advisory:
http://www.nruns.com/_downloads/advisory28122011.pdf

CRuby 1.8.7 fix (use a randomized hash function):
http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/branches/ruby_1_8_7/string.c?r1=34151&r2=34150&pathrev=34151
http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=34151

JRuby uses Murmurhash and a hash (random) "seed" since JRuby 1.6.5.1:
https://github.com/jruby/jruby/commit/c1c9f95ed29cb93806fbc90e9eaabb9c406581e5
https://github.com/jruby/jruby/commit/2fc3a13c4af99be7f25f7dfb6ae3459505bb7c61
http://jruby.org/2011/12/27/jruby-1-6-5-1
JRUBY-6324: random seed for srand is not initialized properly:
https://github.com/jruby/jruby/commit/f7041c2636f46e398e3994fba2045e14a890fc14

Murmurhash:
https://sites.google.com/site/murmurhash/
pyhash implements Murmurhash:
http://code.google.com/p/pyfasthash/

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:50:46 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Mon, 16 Jan 2012 12:50:46 +0000
Subject: [issue13799] Base 16 should be hexadecimal in Unicode HOWTO
Message-ID: <1326718246.49.0.980962494093.issue13799@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

"base 16" should be "hexadecimal" in the Unicode HOWTO in both Python 2 and 3 docs.

----------
assignee: docs at python
components: Documentation
messages: 151354
nosy: docs at python, ramchandra.apte
priority: normal
severity: normal
status: open
title: Base 16 should be hexadecimal in Unicode HOWTO
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13799>
_______________________________________

From report at bugs.python.org  Mon Jan 16 13:53:05 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 16 Jan 2012 12:53:05 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326718385.35.0.615353168056.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Good point.  That should be fixed.  It should be "empty format specification".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:09:28 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 13:09:28 +0000
Subject: [issue12834] memoryview.tobytes() incorrect for non-contiguous arrays
In-Reply-To: <1314211505.01.0.756597830367.issue12834@psf.upfronthosting.co.za>
Message-ID: <1326719368.29.0.590882299651.issue12834@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Added 10181 as a dependency - as noted in my review comments on that issue, I think this becomes fairly trivial to fix (and test) given Stefan's other improvements.

----------
dependencies: +Problems with Py_buffer management in memoryobject.c (and elsewhere?)
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12834>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:10:28 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 16 Jan 2012 13:10:28 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326719428.79.0.682772790109.issue12600@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Meaning you want to run the same test suite with a variety of different DB connections?  That seems like a reasonable use case.

Personally I find that while I sometimes create subclasses to adjust certain class parameters (thus creating a parameterized test case), more often I also have test support methods that are different between the subclasses, which would be more clumsy to do via non-subclass parameterization.

Also, I do have to admit that for the case of just adjusting class constants, it would probably be a fair bit more readable to use a class parameterization scheme to express the testing variations, if only because it would pull the variations into a single cohesive block of code.

So I guess I'm a +0 on this given a concrete use case.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:23:45 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 16 Jan 2012 13:23:45 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326720225.98.0.599117190399.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Finally reviewed Stefan's latest patch. It mostly looks great to me. Aside from a few minor cosmetic quibbles, the only substantive change I would like is to expose the Py_buffer len field as "memoryview.size" instead of "memoryview.len" (to reduce confusion with "len(memoryview)" and to encourage a conceptual link with sys.getsizeof()).

As noted in the review, I also think fixing #12384 should be fairly straightforward and it would be nice to have that working when the change is applied.

(Kristjan: I added you to the nosy list, since you commented on another issue that you had been poking around in the memoryview internals and noticed a few dodgy things. Stefan's work on this issue is aimed at comprehensively addressing those problems).

----------
nosy: +krisvale
versions: +Python 3.3 -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:26:18 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 16 Jan 2012 13:26:18 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326720378.48.0.410229540086.issue12600@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Drat, the tracker lost my post.  In summary, given a concrete use case (running a test case with a variety of different DB connections) and the improved readablility for the common case of just changing class constants in the 'parameterized' subclasses, I'd change to being a +0 on this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:28:05 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Mon, 16 Jan 2012 13:28:05 +0000
Subject: [issue13801] The Python 3 Docs don't highlight nonlocal
Message-ID: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>


New submission from Ramchandra Apte <maniandram01 at gmail.com>:

The Python 3 Docs don't highlight nonlocal such as in the code example in the Python 3 documentation for itertools.zip_longest() (http://docs.python.org/py3k/library/itertools.html)

----------
messages: 151362
nosy: ramchandra.apte
priority: normal
severity: normal
status: open
title: The Python 3 Docs don't highlight nonlocal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:28:29 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Mon, 16 Jan 2012 13:28:29 +0000
Subject: [issue13801] The Python 3 Docs doesn't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326720509.32.0.0789240723663.issue13801@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
title: The Python 3 Docs don't highlight nonlocal -> The Python 3 Docs doesn't highlight nonlocal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:28:54 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Mon, 16 Jan 2012 13:28:54 +0000
Subject: [issue13801] The Python 3 Docs doesn't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326720534.35.0.172237912689.issue13801@psf.upfronthosting.co.za>


Changes by Ramchandra Apte <maniandram01 at gmail.com>:


----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python
versions: +Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:31:11 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 16 Jan 2012 13:31:11 +0000
Subject: [issue13801] The Python 3 Docs doesn't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326720671.93.0.493642254401.issue13801@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti, georg.brandl
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 14:31:39 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 13:31:39 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326720699.82.0.268426133832.issue12600@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Mark, would you like to work on a patch for this?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:09:38 2012
From: report at bugs.python.org (Michael Foord)
Date: Mon, 16 Jan 2012 14:09:38 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326722978.37.0.853404273675.issue12600@psf.upfronthosting.co.za>


Michael Foord <michael at voidspace.org.uk> added the comment:

Why not create a simple TestCase factory in load_tests?

Before a patch can be produced a clean api that offers a clear benefit over the TestCase factory needs to be proposed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:10:18 2012
From: report at bugs.python.org (=?utf-8?b?Q2hyaXN0aWFuIEjDpGdnc3Ryw7Zt?=)
Date: Mon, 16 Jan 2012 14:10:18 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326723018.45.0.875034472255.issue6531@psf.upfronthosting.co.za>


Changes by Christian H?ggstr?m <pajton at kalvdans.no-ip.org>:


----------
nosy: +chn

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:31:40 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Mon, 16 Jan 2012 14:31:40 +0000
Subject: [issue13801] The Python 3 Docs doesn't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326724300.95.0.723663719739.issue13801@psf.upfronthosting.co.za>


Justin Wehnes <jwehnes at gmail.com> added the comment:

Nonlocal is highlighted under Language Reference under simple statements section 6.13 (http://docs.python.org/dev/reference/index.html).  help("nonlocal") will also bring up the documentation (works for me in 3.3).

----------
nosy: +jwehnes

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:34:46 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 16 Jan 2012 14:34:46 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326724486.65.0.853796969906.issue13779@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Hi,
I think that the documentation is pretty clear ("[if topdown=False] ...  the directories in dirnames have already been generated by the time dirnames itself is generated"). And such behaviour is what one would expect, since it is the result of the simplest implementation (listdir(), yield <subdir>, yield <subdir>, yeild <subdir>, yeild <dir>).

I don't think that the implementation will be changed, since it is pretty to do listdir() yourself if needed, and it would make the function slower for the common use case.

Improving the documentation is always possible, what sentence would you see added (what would make the behaviour clearer to you?) ?

----------
nosy: +zbysz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:41:29 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 14:41:29 +0000
Subject: [issue13801] The Python 3 Docs don't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326724889.21.0.094375559507.issue13801@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Not really.  If you follow the link given in the first message and compare how ?is? and ?nonlocal? are styled, you will see the bug.  :)

I hope that Sphinx configures Pygments to use a Python3Lexer, not Python.  Georg?

----------
nosy: +eric.araujo
stage:  -> needs patch
title: The Python 3 Docs doesn't highlight nonlocal -> The Python 3 Docs don't highlight nonlocal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:48:46 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Mon, 16 Jan 2012 14:48:46 +0000
Subject: [issue13801] The Python 3 Docs don't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326725326.74.0.0941466963721.issue13801@psf.upfronthosting.co.za>


Justin Wehnes <jwehnes at gmail.com> added the comment:

Ahh yes I see it.  Sorry about that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 15:58:54 2012
From: report at bugs.python.org (Simon Sapin)
Date: Mon, 16 Jan 2012 14:58:54 +0000
Subject: [issue13742] Add a key parameter (like sorted) to heapq.merge
In-Reply-To: <1326105342.42.0.0197399269422.issue13742@psf.upfronthosting.co.za>
Message-ID: <1326725934.92.0.652939141479.issue13742@psf.upfronthosting.co.za>


Simon Sapin <simon.sapin at kozea.fr> added the comment:

heapq_merge_key_duplicate.patch is a new patch with two code path. It also updates the function?s docstring (which the previous patch did not). Raymond, do you think the speed is worth the DRY violation?

----------
Added file: http://bugs.python.org/file24248/heapq_merge_key_duplicate.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13742>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:03:22 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 15:03:22 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326726202.79.0.567652085423.issue13779@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> However, the generated (files? and) dirs do not necessarily reflect the current situation as produced by os.listdir.

What do you mean exactly? Another process has re-created "b" in parallel?
This race condition is pretty impossible to solve in the general case (unless the filesystem is transactional): even if you call os.listdir() again, a new file or dir could appear just before the following call to os.rmdir().

(just for the record, shutil.rmtree already allows you to delete a filesystem tree)

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:10:30 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 16 Jan 2012 15:10:30 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1326726630.8.0.765737781098.issue13793@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Normalizing the string in getattr() is unacceptable. We'll just have to document, that all identifiers are in NFKC.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:14:37 2012
From: report at bugs.python.org (Meador Inge)
Date: Mon, 16 Jan 2012 15:14:37 +0000
Subject: [issue13665] TypeError: string or integer address expected instead of
	str instance
In-Reply-To: <1324921834.92.0.644281007494.issue13665@psf.upfronthosting.co.za>
Message-ID: <1326726877.1.0.851589740032.issue13665@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

LGTM.  Thanks for fixing this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13665>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:17:59 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 16 Jan 2012 15:17:59 +0000
Subject: [issue1692335] Fix exception pickling: Move initial args assignment
	to BaseException.__new__
Message-ID: <1326727079.01.0.0337454686765.issue1692335@psf.upfronthosting.co.za>


Changes by Zbyszek Szmek <zbyszek at in.waw.pl>:


----------
nosy: +zbysz

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1692335>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:25:13 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 16 Jan 2012 15:25:13 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326727513.7.0.302802454422.issue12705@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I think an exception is fine, but it should only happen in 3.3.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:36:08 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 16 Jan 2012 15:36:08 +0000
Subject: [issue13801] The Python 3 Docs don't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326728168.03.0.0808217436734.issue13801@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

The problem was until very recently, pygments omitted the nonlocal keyword. We probably need to wait for another pygments release.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 16:56:27 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Mon, 16 Jan 2012 15:56:27 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326729387.1.0.786726344314.issue13779@psf.upfronthosting.co.za>


patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:

The documentation of this function is generally ambiguous, because os.walk is a generator. Thus "generate" means (1) yielded from a generator and (2) prepared for later use within the generator. To avoid the ambiguity, "generate" should be avoided if possible. (1) might be replaced by "return" and (2) by "prepare".

@zbysz
The paragraph you cite is about "Modifying dirnames ". If you are not "Modifying dirnames" (like me) this section easily skips your attention.

My problem was, of course, that I had myself changed the directory structure - It's not a race condition with another process but an effect  of the loop os.walk was managing.

@pitrou
shutil.rmtree cannot help me, because I'm only deleting empty dirs.

Proposal (very verbose I'm afraid):
If you change the directory structure below dirpath while topdown=True, you can modify dirnames in-place so that walk will visit the right directories. If you change the directory structure below dirpath while topdown=False (maybe while you where there), dirnames and filenames can still reflect the old situation and it might be necessary to call os.listdir again.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:13:45 2012
From: report at bugs.python.org (Pedro Andres Aranda Gutierrez)
Date: Mon, 16 Jan 2012 16:13:45 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
In-Reply-To: <1326712450.91.0.336594921639.issue13796@psf.upfronthosting.co.za>
Message-ID: <CAO48Bk-ewBnvWp5Y_YCwiBgcjnUVZOWOQRCfYCpt__YpvZ-vcg@mail.gmail.com>


Pedro Andres Aranda Gutierrez <paaguti at gmail.com> added the comment:

Touch? :-)
I was just frustrated because my XMLs never have tail or text as
attributes and I wanted to have more compact code...

On Mon, Jan 16, 2012 at 12:14 PM, patrick vrijlandt
<report at bugs.python.org> wrote:
>
> patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:
>
> I agree the Element syntax is sometimes awkward.
>
> But how would you represent text or tail attributes within this enhanced element?
> <animal name="cat" tail="yes"> comes to mind ...
>
> ----------
> nosy: +patrick.vrijlandt
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13796>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:16:48 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:16:48 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326730608.44.0.996261336039.issue12415@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Please review.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:17:07 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:17:07 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326730627.34.0.0502568229771.issue12415@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


Added file: http://bugs.python.org/file24249/metadoc-where-are-the-sources.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:19:27 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Mon, 16 Jan 2012 16:19:27 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1326730767.06.0.750792428598.issue13790@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

Changing to a documentation issue.

----------
assignee: eric.smith -> docs at python
components: +Documentation -Interpreter Core
keywords: +easy
nosy: +docs at python
versions: +Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:23:34 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:23:34 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326731014.26.0.269495487099.issue11805@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Copied from #13712:

[I]
Problem is that the setup.cfg syntax does not define how to give more than one value.  If it?s judged acceptable to disallow paths with embedded spaces, we could do something like this:

[files]
package_data =
    spam = first second third

Otherwise we?d need to use multiple lines (requested in #5302) [actually no, it was this bug #11805]

[files]
package_data =
    spam = first
    spam = second
    spam = third

We probably don?t want that.  An intermediate idea:

[files]
package_data =
    spam = first
           second
           third

Not sure this would be the nicest thing for people to write, and for us (me) to extend the setup.cfg parser for.


[Erik Bray]
FWIW, I'm for the first option for specifying package_data:

[files]
package_data =
    spam = first second third

I'm pretty sure this is how I ended up implementing it in d2to1, since I needed this functionality.

Theoretically spaces could be supported with an escape sequence, but I don't think that's worth complicating things for if package_data is deprecated anyways.  I'm all for making it difficult for anyone trying to include filenames with spaces in their source code.


So, I think allowing spaces and newlines to separate multiple values will be the nicest (spaces only would require users to cram all there file specs on one line).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:27:47 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:27:47 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326731267.28.0.130764341966.issue12409@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Glad my action plan worked :)

There are now obsolete entries in the Doc/tools/sphinxext/susp-ignored.csv file: All lines starting with ?documenting? can be removed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:29:45 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:29:45 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326731385.14.0.115169684476.issue11805@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

(To clarify quotes in my message: First is a copied message from me, marked [I], then one form Erik, and the last paragraph is me again.  Sleep good, should get more of that.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:30:53 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 16 Jan 2012 16:30:53 +0000
Subject: [issue13588] Change name of internal closure functions in importlib
In-Reply-To: <1326675530.38.0.844012535759.issue13588@psf.upfronthosting.co.za>
Message-ID: <CAP1=2W40FaRKd8WgrhazVbFFpTRG1bNCWNKiVdBeZPuoucmNBA@mail.gmail.com>


Brett Cannon <brett at python.org> added the comment:

Sorry, been busy. I will definitely get to it this week.

On Sun, Jan 15, 2012 at 19:58, Berker Peksag <report at bugs.python.org> wrote:

>
> Berker Peksag <berker.peksag at gmail.com> added the comment:
>
> Hi Brett, did you have a chance to review the patch I submitted?
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13588>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13588>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:31:32 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 16:31:32 +0000
Subject: [issue13412] Symbolic links omitted by os.listdir on some systems
In-Reply-To: <1321396602.6.0.583941827853.issue13412@psf.upfronthosting.co.za>
Message-ID: <1326731492.04.0.672361562997.issue13412@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

While this may in fact be related to #6727, I don't believe it's a proper duplicate. I believe there's a more fundamental issue that's causing both #6727 and #13412, and that's that stat/wstat is broken on Windows due to a regression in the MS C runtime: http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/93ebb061-d952-4650-b15c-30548a6649a8/

I mention that here because a fix for #6727 if localized to import.c may not address this issue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13412>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:33:21 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:33:21 +0000
Subject: [issue5300] Distutils ignores file permissions
In-Reply-To: <1234907694.25.0.317480697226.issue5300@psf.upfronthosting.co.za>
Message-ID: <1326731601.24.0.7843712458.issue5300@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

FTR it looks like http://mail.python.org/pipermail/python-list/2009-January/1188084.html was the original report for this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5300>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:34:13 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 16 Jan 2012 16:34:13 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326731653.56.0.736520747715.issue13779@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> The documentation of this function is generally ambiguous, because
> os.walk is a generator. Thus "generate" means (1) yielded from a 
> generator and (2) prepared for later use within the generator. To avoid 
> the ambiguity, "generate" should be avoided if possible. (1) might be 
> replaced by "return" and (2) by "prepare".
It think that you are wrong here: generate is consistently used in
meaning (1) in the docstring. Also "return" means something completely different. "generate" is better than "prepare", because it is more specific,
especially from the POV of the user of the function, who only cares about
when something is yielded, not when it is prepared.

> Proposal (very verbose I'm afraid):
> If you change the directory structure below dirpath while topdown=True, 
> you can modify dirnames in-place so that walk will visit the right 
> directories. If you change the directory structure below dirpath while 
> topdown=False (maybe while you where there), dirnames and filenames can 
> still reflect the old situation and it might be necessary to call 
> os.listdir again.
Hm, I think that the difference in behaviour between topdown=False and topdown=True is smaller then this proposal implies.
When topdown=True, the list is not updated after changes on disk. So both removing and adding directories does _not_ cause them to added or removed from the list of subdirectories to visit. Nevertheless, the default behaviour on
error is to do nothing, so it _looks_ like they were skipped.

So I think that the documentation could only clarify that the list of subdirectories is queried first, before the dir or the subdirectories are "generated".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:36:39 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 16 Jan 2012 16:36:39 +0000
Subject: [issue13779] os.walk: bottom-up
In-Reply-To: <1326452589.73.0.0617213643117.issue13779@psf.upfronthosting.co.za>
Message-ID: <1326731799.92.0.876645927.issue13779@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

>>> os.makedirs('/tmp/a/b/c')
>>> gen  = os.walk('/tmp/a')
>>> next(gen)
('/tmp/a', ['b'], [])
>>> os.makedirs('/tmp/a/b2')
>>> next(gen)
('/tmp/a/b', ['c'], [])
>>> next(gen)
('/tmp/a/b/c', [], [])

>>> gen  = os.walk('/tmp/a', onerror=print)
>>> next(gen)
('/tmp/a', ['b2', 'b'], [])
>>> os.rmdir('/tmp/a/b2')
>>> next(gen)
[Errno 2] No such file or directory: '/tmp/a/b2'
('/tmp/a/b', ['c'], [])

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13779>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:36:44 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:36:44 +0000
Subject: [issue8502] support plurals in pygettext
In-Reply-To: <1271985007.3.0.675755685137.issue8502@psf.upfronthosting.co.za>
Message-ID: <1326731804.97.0.903423511419.issue8502@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thank you for making a patch.  I?ve been busy and could not find time to look at it, but after I add a testing machinery for scripts like pygettext I will look into this.

----------
assignee: docs at python -> 
dependencies: +Add tests for some scripts in Tools/scripts
nosy:  -georg.brandl
stage:  -> needs patch
title: proposal: encourage xgettext rather than pygettext.py in gettext docs -> support plurals in pygettext

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8502>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:39:37 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 16 Jan 2012 16:39:37 +0000
Subject: [issue8913] Document that datetime.__format__ is datetime.strftime
In-Reply-To: <1275794779.99.0.00462382173098.issue8913@psf.upfronthosting.co.za>
Message-ID: <1326731977.01.0.78435634329.issue8913@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

IMO, now that two official releases of Python 3 have shipped with this behavior (3.1 and 3.2), it is too late to think about changing what datetime.__format__ does, and we should merely document that it is the same as strftime.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8913>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:46:38 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 16:46:38 +0000
Subject: [issue13588] Change name of internal closure functions in importlib
In-Reply-To: <1323703954.36.0.434643027987.issue13588@psf.upfronthosting.co.za>
Message-ID: <E1Rmphc-0003HJ-Ll@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9935f7837c4b by Brett Cannon in branch 'default':
Issue #13588: Rename decorators in importlib.
http://hg.python.org/cpython/rev/9935f7837c4b

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13588>
_______________________________________

From report at bugs.python.org  Mon Jan 16 17:47:05 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 16 Jan 2012 16:47:05 +0000
Subject: [issue13588] Change name of internal closure functions in importlib
In-Reply-To: <1323703954.36.0.434643027987.issue13588@psf.upfronthosting.co.za>
Message-ID: <1326732425.51.0.449735592915.issue13588@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

Thanks for the patch, Berker! It's all committed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13588>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:03:26 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 16 Jan 2012 17:03:26 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1326733406.57.0.214031410044.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Does this need need more discussion, code review, testing, or just more time?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:06:36 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 16 Jan 2012 17:06:36 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <E1Rmq0w-0005Ek-ST@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 7167c04780ed by Sandro Tosi in branch '2.7':
Issue: #12409: remove obsolete susp-entries
http://hg.python.org/cpython/rev/7167c04780ed

New changeset 153e32333aaa by Sandro Tosi in branch '3.2':
Issue: #12409: remove obsolete susp-entries
http://hg.python.org/cpython/rev/153e32333aaa

New changeset a6bf08666da2 by Sandro Tosi in branch 'default':
Issue: #12409: merge with 3.2
http://hg.python.org/cpython/rev/a6bf08666da2

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:07:29 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 16 Jan 2012 17:07:29 +0000
Subject: [issue12409] Moving "Documenting Python" to Devguide
In-Reply-To: <1309008968.0.0.644952507743.issue12409@psf.upfronthosting.co.za>
Message-ID: <1326733649.74.0.593986908406.issue12409@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

?ric, thanks for spotting it: removed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12409>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:11:26 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 17:11:26 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326733886.5.0.551456863818.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


----------
hgrepos: +104

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:12:13 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 17:12:13 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326733933.34.0.653312818345.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24250/4aaf78f0dd10.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:13:31 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 17:13:31 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326734011.68.0.542512193917.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

4aaf78f0dd10.diff is the same patch as that for 2.7 but ported to 3.1.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 18:52:33 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 17:52:33 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326736353.14.0.235134366039.issue6531@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

There seem to be two issues at play here:

- the atexit module (and its companion helper _Py_PyAtExit()) doesn't know about sub-interpreters.

- PyState_FindModule() doesn't know about sub-interpreters either, because the m_index field (which records the module's index in an interpreter's module list (PyInterpreterState.modules_by_index)) is recorded in the PyModuleDef structure rather than the module instance: it is therefore global to all interpreters

Having atexit work properly with sub-interpreters would require fixing these two issues AFAICT.

----------
nosy: +loewis, pitrou
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:00:05 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 18:00:05 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326736805.08.0.562667868792.issue6531@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Hmm, it seems I may be mistaken about the second point. m_index is actually generated such that all modules end up in the same position in the interpreters' respective modules_by_index lists. Sorry.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:00:09 2012
From: report at bugs.python.org (Meador Inge)
Date: Mon, 16 Jan 2012 18:00:09 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326736809.36.0.870440161983.issue12705@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:04:46 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 16 Jan 2012 18:04:46 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326737086.88.0.00137550738699.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I guess you're reffering to the  "i'th" --> "i-th"  thing.

I found numerous occasions on the Internet that say like "n-th element blah blah blah", not "n'th element..."

So I guess writting this thing as "i-th" would be correct here, too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:11:44 2012
From: report at bugs.python.org (Roger Serwy)
Date: Mon, 16 Jan 2012 18:11:44 +0000
Subject: [issue13798] Pasting and then running code doesn't work in the IDLE
	Shell
In-Reply-To: <1326717884.33.0.870144380911.issue13798@psf.upfronthosting.co.za>
Message-ID: <1326737504.27.0.9255719011.issue13798@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

This is a duplicate of #3559.

----------
nosy: +serwy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13798>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:14:11 2012
From: report at bugs.python.org (Roger Serwy)
Date: Mon, 16 Jan 2012 18:14:11 +0000
Subject: [issue13052] IDLE: replace ending with '\' causes crash
In-Reply-To: <1317177843.04.0.691154940707.issue13052@psf.upfronthosting.co.za>
Message-ID: <1326737651.46.0.442286245008.issue13052@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

I'm not aware of any formal tests for IDLE. Can you show me the docs for writing proper unit tests? I'll see if I can write one for this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13052>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:16:52 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 16 Jan 2012 18:16:52 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326737812.66.0.739021077379.issue12600@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Maybe we could add a recipe for doing this to the load_tests docs?

I don't think that load_tests is going to be more readable, though, since it doesn't allow you to put the parameterization next to the class you are parameterizing (unless you do some additional hackery).

But yes, if anything else is done a concrete API proposal is the first requirement.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:29:01 2012
From: report at bugs.python.org (Eric Snow)
Date: Mon, 16 Jan 2012 18:29:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326738541.06.0.558956564364.issue13703@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

> The vulnerability is known since 2003 (Usenix 2003): read "Denial of
> Service via Algorithmic Complexity Attacks" by Scott A. Crosby and Dan
> S. Wallach.

Crosby started a meaningful thread on python-dev at that time similar to the current one:

  http://mail.python.org/pipermail/python-dev/2003-May/035874.html

It includes a some good insight into the problem.

----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:52:09 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 16 Jan 2012 18:52:09 +0000
Subject: [issue13801] The Python 3 Docs don't highlight nonlocal
In-Reply-To: <1326720484.97.0.730349515473.issue13801@psf.upfronthosting.co.za>
Message-ID: <1326739929.73.0.865616095543.issue13801@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13801>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:52:25 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 18:52:25 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326739945.7.0.775212809371.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


----------
hgrepos: +105

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:52:55 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 18:52:55 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326739975.53.0.610921751028.issue6727@psf.upfronthosting.co.za>


Changes by Jason R. Coombs <jaraco at jaraco.com>:


Added file: http://bugs.python.org/file24251/4fdbc9f74235.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 19:58:52 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 16 Jan 2012 18:58:52 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326738541.06.0.558956564364.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F147368.3000205@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Eric Snow wrote:
> 
> Eric Snow <ericsnowcurrently at gmail.com> added the comment:
> 
>> The vulnerability is known since 2003 (Usenix 2003): read "Denial of
>> Service via Algorithmic Complexity Attacks" by Scott A. Crosby and Dan
>> S. Wallach.
> 
> Crosby started a meaningful thread on python-dev at that time similar to the current one:
> 
>   http://mail.python.org/pipermail/python-dev/2003-May/035874.html
> 
> It includes a some good insight into the problem.

Thanks for the pointer. Some interesting postings...

Vulnerability of applications:
http://mail.python.org/pipermail/python-dev/2003-May/035887.html

Speed of hashing, portability and practical aspects:
http://mail.python.org/pipermail/python-dev/2003-May/035902.html

Changing the hash function:
http://mail.python.org/pipermail/python-dev/2003-May/035911.html
http://mail.python.org/pipermail/python-dev/2003-May/035915.html

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 16 20:00:03 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 16 Jan 2012 19:00:03 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326740403.82.0.00266680723774.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

I ported the patch to the 3.2 branch. This port required more extensive adjustments. In particular, the NullImporter no longer used (char *) for directory detection, but instead used platform-specific PyObjects.

Therefore, the 3.2 diff creates a new function, Py_PathIsDir, which takes a PyUnicode on Windows and PyBytes on other platforms and calculates the directory state of the parameter.

I'm fairly unhappy with the way the patch for 2.7 and 3.1 translated to 3.2.

So, I'm seeking review and approval of the 2.7 and 3.1 patches, and comments on the 3.2 patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 21:00:26 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 16 Jan 2012 20:00:26 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326744026.8.0.673125966561.issue13695@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

"numerous occasions" is not good enough. Why not let a native speaker decide?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 21:14:14 2012
From: report at bugs.python.org (Jim Jewett)
Date: Mon, 16 Jan 2012 20:14:14 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1326744854.6.0.779616841754.issue13793@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Why is normalization in getattr unacceptable?  I won't pretend to *like* it, but the difference between two canonically equal strings really is (by definition) just a representational issue.

Would it be OK to normalize in object's own implementation, so that custom classes could avoid the normalization, but it would happen by default?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 16 21:19:32 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 16 Jan 2012 20:19:32 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326745172.05.0.832356161607.issue12415@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

that looks ok to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Mon Jan 16 21:21:20 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 16 Jan 2012 20:21:20 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1326745280.57.0.466017651813.issue13793@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Because it's (very) expensive and the method you propose would fail in the case that someone had put the normalized and unnormalized string in the object's __dict__.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 16 21:55:42 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 20:55:42 +0000
Subject: [issue13052] IDLE: replace ending with '\' causes crash
In-Reply-To: <1317177843.04.0.691154940707.issue13052@psf.upfronthosting.co.za>
Message-ID: <1326747342.15.0.228834163033.issue13052@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The library manual has chapters for

unittest: the generic testing framework (package) that any app can use

test: a package with the Python test suite; test_x tests module x
  It has other modules and subpackages, probably not all documented.
  The test modules use unittest, doctest, and other code.

test.support: some of the other code

AS it says in the test chapter, the test suite imports each test_x and runs its test_main function. The tkinter package includes a test subpackage. test_tk checks that _tkinter and tk are available and if so, runs the tests in tkinter.test. (But I am not sure it really does what it should ;-).

My thought is that we might want to do something similar with idle. However, there needs to be a test_idle module in any case, so we can start with a few tests there and migrate them to a separate idlelib.test subpackage later if we want.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13052>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:06:45 2012
From: report at bugs.python.org (Mark Diekhans)
Date: Mon, 16 Jan 2012 21:06:45 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1326719428.79.0.682772790109.issue12600@psf.upfronthosting.co.za>
Message-ID: <20244.37217.392842.295095@osprey.kermodei.com>


Mark Diekhans <markd at kermodei.com> added the comment:

> R. David Murray <rdmurray at bitdance.com> added the comment:
> 
> Meaning you want to run the same test suite with a variety of
> different DB connections?  That seems like a reasonable use
> case.

This is for external parameterization of a test to run in a
different environment.  Internal reuse of test code is usually
better done inside of the code using standard OOP approaches.

The different in database connections is due to wanting to run
the tests on different systems with different database users,
passwords and database names.  Normally we have an object that
reads this information from file specified on the command line,
the object is passed to code that creates the connections.

A similar problem exists when tests must be run against a server
and require a host name.  External parameterization to specify
the host name is required.

While it's certainly possible to come up with says to pass this
setting things in globals (including environment variable), this
does lead to confusing code.

Mark

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:11:29 2012
From: report at bugs.python.org (Berker Peksag)
Date: Mon, 16 Jan 2012 21:11:29 +0000
Subject: [issue13641] decoding functions in the base64 module could accept
	unicode strings
In-Reply-To: <1324386392.48.0.968435833624.issue13641@psf.upfronthosting.co.za>
Message-ID: <1326748289.33.0.29176428656.issue13641@psf.upfronthosting.co.za>


Changes by Berker Peksag <berker.peksag at gmail.com>:


Added file: http://bugs.python.org/file24252/issue13641_v3_with_tests.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13641>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:34:41 2012
From: report at bugs.python.org (Berker Peksag)
Date: Mon, 16 Jan 2012 21:34:41 +0000
Subject: [issue13165] Integrate stringbench in the Tools directory
In-Reply-To: <1318520246.46.0.148882944484.issue13165@psf.upfronthosting.co.za>
Message-ID: <1326749681.14.0.445311738585.issue13165@psf.upfronthosting.co.za>


Changes by Berker Peksag <berker.peksag at gmail.com>:


----------
nosy: +berkerpeksag

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13165>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:40:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 21:40:46 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
In-Reply-To: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>
Message-ID: <1326750046.11.0.794975032314.issue13727@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The macros should be named PyDelta_XXX, as PyDelta_Check and friend.
Otherwise, makes sense.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:43:32 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Mon, 16 Jan 2012 21:43:32 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
In-Reply-To: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>
Message-ID: <1326750212.94.0.563670433643.issue13727@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

I named them following the other accessor macros: PyDateTime_TIME_GET_HOUR(), even though the check function is named PyTime_Check().
Which inconsistency do you prefer? :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 22:51:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 16 Jan 2012 21:51:39 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
In-Reply-To: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>
Message-ID: <1326750699.97.0.331048867989.issue13727@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Ow. Then I don't mind :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Mon Jan 16 23:01:45 2012
From: report at bugs.python.org (Berker Peksag)
Date: Mon, 16 Jan 2012 22:01:45 +0000
Subject: [issue11695] Improve argparse usage/help customization
In-Reply-To: <1301234745.11.0.232414036754.issue11695@psf.upfronthosting.co.za>
Message-ID: <1326751305.87.0.512698561014.issue11695@psf.upfronthosting.co.za>


Changes by Berker Peksag <berker.peksag at gmail.com>:


----------
nosy: +berkerpeksag

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11695>
_______________________________________

From report at bugs.python.org  Mon Jan 16 23:32:42 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 22:32:42 +0000
Subject: [issue8285] IDLE not smart indenting correctly in nested statements
In-Reply-To: <1270157080.33.0.1290049865.issue8285@psf.upfronthosting.co.za>
Message-ID: <1326753162.55.0.890777732169.issue8285@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The current indentation is a consequent of using tabs for indents.
Changing that is the subject of #7676.
#1196946 is a similar issue.

----------
nosy: +terry.reedy
resolution:  -> duplicate
status: open -> closed
superseder:  -> IDLE shell shouldn't use TABs
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8285>
_______________________________________

From report at bugs.python.org  Mon Jan 16 23:37:58 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Mon, 16 Jan 2012 22:37:58 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326753478.5.0.440050257285.issue13781@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

For 3.x, I think that ignoring non-string names is a reasonable fix. The docs
for io.FileIO specify that its name attribute can be either a path or an integer
file descriptor, and changing this doesn't seem to serve any purpose.

As for the case of 2.7's bogus "<fdopen>" name attribute, I'm not sure what the
best course of action is. I agree that ideally we would want to get rid of the
attribute altogether (for objects returned by fdopen), or change the semantics
to those used by FileIO in 3.x, but making that sort of change in a bugfix
release seems unwise.

One alternative would be for GzipFile to specifically check whether a file
object was returned by fdopen(), and if so ignore the fake name. I'm not sure
how this could be accomplished, though - just checking for name == "<fdopen>" is
too fragile for my liking, and I can't see any other obvious way of
distinguishing objects created by fdopen() from those created by open().

> (another quick test shows that gzip in python 3.x can't output to a BytesIO
> fileobj at all, it thinks it is readonly)

Are you sure about this? I can't reproduce the problem. Running this script:

    import gzip, io

    b = io.BytesIO()
    with gzip.GzipFile(fileobj=b, mode="w") as g:
        g.write(b"asdf ghjk")
    print(b.getvalue())
    b.seek(0)
    with gzip.GzipFile(fileobj=b, mode="r") as g:
        print(g.read())

I get the following output:

    b'\x1f\x8b\x08\x00\xe1\xa4\x14O\x02\xffK,NISH\xcf\xc8\xca\x06\x00P\xd2\x1cJ\t\x00\x00\x00'
    b'asdf ghjk'

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Tue Jan 17 00:22:20 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 16 Jan 2012 23:22:20 +0000
Subject: [issue13802] IDLE Prefernces/Fonts: use multiple alphabets in examples
Message-ID: <1326756140.29.0.0714084996967.issue13802@psf.upfronthosting.co.za>


New submission from Terry J. Reedy <tjreedy at udel.edu>:

In the Fonts/Tabs tab of the IDLE Preference dialog, the large box for examples of the font selected shows a small square of ascii chars. I think the box should also show 1 char for each of several alphabets so the consequence of choosing various fonts will be more evident. I am thinking of adding several line with the format
Alphabet \uXXXX <char>

----------
components: IDLE
messages: 151415
nosy: terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: IDLE Prefernces/Fonts: use multiple alphabets in examples
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13802>
_______________________________________

From report at bugs.python.org  Tue Jan 17 00:32:03 2012
From: report at bugs.python.org (Jim Jewett)
Date: Mon, 16 Jan 2012 23:32:03 +0000
Subject: [issue13165] Integrate stringbench in the Tools directory
In-Reply-To: <1318520246.46.0.148882944484.issue13165@psf.upfronthosting.co.za>
Message-ID: <1326756723.52.0.603041665137.issue13165@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

The URL got mangled in at least my browser, so I'm repasting it on its own line:

http://svn.python.org/projects/sandbox/trunk/stringbench

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13165>
_______________________________________

From report at bugs.python.org  Tue Jan 17 01:14:58 2012
From: report at bugs.python.org (Eric Snow)
Date: Tue, 17 Jan 2012 00:14:58 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326759298.21.0.7980056459.issue6531@psf.upfronthosting.co.za>


Changes by Eric Snow <ericsnowcurrently at gmail.com>:


----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Tue Jan 17 01:15:25 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 17 Jan 2012 00:15:25 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326759325.64.0.382083780395.issue12600@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

That's not the kind of parameterization this ticket is about, though.  You are talking about passing data in to a test run from the command line (or other source), which is a different issue (though the implementations might share some common infrastructure).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 01:55:04 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 17 Jan 2012 00:55:04 +0000
Subject: [issue7676] IDLE shell shouldn't use TABs
In-Reply-To: <1263213381.47.0.181451927584.issue7676@psf.upfronthosting.co.za>
Message-ID: <1326761704.54.0.654362793391.issue7676@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I closed #8285 as it merely reported a consequence of using tabs.

The Command Prompt window used for interactive Python on Windows only uses fixed-pitch fonts. Idle (tk) allows any font. I happen to use Lucida Sans Unicode (variable pitch) for what I think is the largest set of glyphs. (Perhaps Lucida Console (fixed) has the same set, I don't know. See #13802 for an idea to improve font selection in this regard.) Whatever we do should work well with either fixed or variable pitches. (The current patch does not -- see below.) Or we should prohibit variable pitch fonts and only allow fixed pitch fonts

Selection in CP selects a rectangle, so one can select code with or without the prompts. This is only possible with fixed pitch. Tk text boxes are limited to selecting a continuous stream of text, without excluding a margin.

This contrasts with, for example Notepad++. Its (tabbed) edit window has a gray left margin with line numbers. Then it has a narrow lighter gray column with [-] and [+] boxes to collapse and expand indented regions. Then it has a white text box for user entry and only user entry. Selections ignore the gray margins and only select within the white user text box. Perhaps this is done by tying three borderless windows to one vertical scroll bar, all inside one frame. I don't know. If we could do the same with tkinter, we could put prompts in a grayed margin, but I do not know if we can.

CP has tab stops at every 8 chars. So if one indents with tabs, the first indent is 4 spaces past the prompt and further indents are 8 spaces beyond the last. Not good. The alternative is hand spacing. Also not good if you do very much. Idle (Tk) appears to have tabs at fixed positions, regardless of the font. (Notepad++ acts like IDLE's edit windows -- a tab in converted to 4 spaces.) In Lucida Sans, a tab is physically the same as 16 spaces. Since each '>' is 3 spaces, an indent of 8 spaces instead of a tab is a dedent on the screen. So the patch does not work for me.

My conclusion is that having the shell write a prompt onto the user entry line, in the tk environment, is incompatible with the behavior we want. Fix 1 would be to put the prompt on a line by itself ('>>>\n' or 'input>\n'), use the same indents as the editor, and add an similar output marker ('###\n' or '<output\n'). Option 2 is to omit prompts, use editor indents, and prefix every output line with '#'. Fix 3 would be to put the prompt in a grayed margin. (If this can be done, we could also add line #s as an editor option.)

A separate idea is to give output a colored background.

Thoughts?

----------
nosy: +terry.reedy
stage: needs patch -> patch review
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7676>
_______________________________________

From report at bugs.python.org  Tue Jan 17 02:53:51 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 01:53:51 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326765231.61.0.190596727505.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 7:
 - Make PyOS_URandom() private (renamed to _PyOS_URandom)
 - os.urandom() releases the GIL for I/O operation for its implementation reading /dev/urandom
 - move _Py_unicode_hash_secret_t documentation into unicode_hash()

I moved also fixes for tests in a separated patch: random_fix-tests.patch.

----------
Added file: http://bugs.python.org/file24253/random-7.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 02:57:17 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 01:57:17 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326765437.69.0.904869746925.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Added file: http://bugs.python.org/file24254/random-fix_tests.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:02:38 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Tue, 17 Jan 2012 02:02:38 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1326765758.73.0.0983667064741.issue8052@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

attaching a patch that implements this safely.

----------
keywords: +patch
Added file: http://bugs.python.org/file24255/subprocess-close-open-fds-gps01.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:08:58 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Tue, 17 Jan 2012 02:08:58 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1326766138.55.0.555801130124.issue8052@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

attaching a better formatted one for review.

----------
Added file: http://bugs.python.org/file24256/subprocess-close-open-fds-gps02.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:10:42 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 02:10:42 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326766242.22.0.74313150893.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Some tests are still failing with my 2 patches:
 - test_dis
 - test_inspect
 - test_json
 - test_packaging
 - test_ttk_textonly
 - test_urllib

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:17:07 2012
From: report at bugs.python.org (Ferringb)
Date: Tue, 17 Jan 2012 02:17:07 +0000
Subject: [issue6559] add pass_fds paramter to subprocess.Popen()
In-Reply-To: <1248422197.05.0.784368367425.issue6559@psf.upfronthosting.co.za>
Message-ID: <1326766627.8.0.791474824618.issue6559@psf.upfronthosting.co.za>


Ferringb <ferringb at gmail.com> added the comment:

Just noticed this patch... aside from liking the intention, the api for this is going to grow tiresome quick since it expects the FDs to already be in place; is there any reasons a mapping wasn't used here, specifically of (src_fd|src_fileobj) -> target_fd ?

If that was fed in, post fork the client can shuffle around the fd's into appropriate positions- something the parent may not be able to do (threaded environment for example, or async in some respect).

I've had similar functionality in my own process code for a while, and have found it to be generally pretty easy to deal with- for subprocess it has the added benefit that the existing stdin/stdout/stderr bits could be pretty easily folded directly into it.

So... any reason this route wasn't considered, or just wasn't thought about?

----------
nosy: +ferringb

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6559>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:20:53 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Tue, 17 Jan 2012 02:20:53 +0000
Subject: [issue6559] add pass_fds paramter to subprocess.Popen()
In-Reply-To: <1248422197.05.0.784368367425.issue6559@psf.upfronthosting.co.za>
Message-ID: <1326766853.51.0.63549393377.issue6559@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

Wasn't thought about.  I have seen something similar to that done in another c++ subprocess implementation since.  If you have suggestions for a more useful API, feel free to propose them in a new issue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6559>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:46:25 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 17 Jan 2012 02:46:25 +0000
Subject: [issue964437] idle help is modal
Message-ID: <1326768385.23.0.155372318553.issue964437@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I agree that modal is a nuisance. Another problem with the help text window is that, on my system, it opens too narrow -- about 2/3rds the width of a shell or edit window. So many line are wrapped. Is this easy to fix and include with this patch, or does it need a new issue?

This patch fails to apply to 2.7 or 3.2: (t)hg messages are:
applying F:\Python\dev\help_nonmodal.diff
unable to find 'idlelib/textView.py' for patching
patching file idlelib/textView.py
6 out of 6 hunks FAILED -- saving rejects to file idlelib/textView.py.rej
abort: patch failed to apply
I redownloaded and tried again with same result.

----------
nosy: +serwy
priority: low -> normal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue964437>
_______________________________________

From report at bugs.python.org  Tue Jan 17 03:54:23 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 17 Jan 2012 02:54:23 +0000
Subject: [issue6698] IDLE no longer opens only an edit window when configured
	to do so
In-Reply-To: <1250200991.43.0.141209205867.issue6698@psf.upfronthosting.co.za>
Message-ID: <1326768863.12.0.516988571815.issue6698@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

When I click r71126 it is mapped to bc084a97318b for #1529142
KBK msg85379 says "The -e option now only opens an editor window,"

I think I agree with Tal's suggestion as long as thing still work ok on Mac. It is trivial to open a new empty window from the shell.

I have not looked at the current patch yet.

----------
nosy: +terry.reedy
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6698>
_______________________________________

From report at bugs.python.org  Tue Jan 17 05:08:39 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 17 Jan 2012 04:08:39 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
Message-ID: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>


New submission from Jes?s Cea Avi?n <jcea at jcea.es>:

When compiling modules under Solaris, distutils generates directories like "build/lib.solaris-2.10-i86pc-3.2". The "i86pc" part is the same both in 32 and 64 bits.

So when building 32 and 64 bit C code, the binaries are mixed and the best result you can get is something like:

"""
ImportError: ld.so.1: python: fatal: /home/pybsddb/build/lib.solaris-2.10-i86pc-2.7/bsddb3/_pybsddb.so: wrong ELF class: ELFCLASS64
"""

In other platforms, bitness is correctly detected and included in the directory name.

Since Solaris is explicitly managed in the sourcecode ("distutils.util.get_platform()"), adding "platform.architecture()[0]" in the directory name seems trivial.

I think this patch should be applied to 2.6 and 3.1, unless they are open only for security fixes.

----------
assignee: tarek
components: Distutils
keywords: easy
messages: 151427
nosy: eric.araujo, jcea, tarek
priority: normal
severity: normal
status: open
title: Under Solaris, distutils doesn't include bitness in the directory name
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Tue Jan 17 06:49:53 2012
From: report at bugs.python.org (akira)
Date: Tue, 17 Jan 2012 05:49:53 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326779393.49.0.875397219459.issue10278@psf.upfronthosting.co.za>


Changes by akira <4kir4.1i at gmail.com>:


----------
nosy: +akira

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Tue Jan 17 08:08:16 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 07:08:16 +0000
Subject: [issue13794] Copyright Year - Change it to 2012 please
In-Reply-To: <1326691304.18.0.453476801273.issue13794@psf.upfronthosting.co.za>
Message-ID: <E1Rn39S-0007GF-Fn@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8f3a3247972b by Ezio Melotti in branch 'default':
#13794: Update copyright year to 2012 in the devguide.
http://hg.python.org/devguide/rev/8f3a3247972b

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13794>
_______________________________________

From report at bugs.python.org  Tue Jan 17 08:12:56 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 07:12:56 +0000
Subject: [issue13794] Copyright Year - Change it to 2012 please
In-Reply-To: <1326691304.18.0.453476801273.issue13794@psf.upfronthosting.co.za>
Message-ID: <1326784376.6.0.0510144553548.issue13794@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

I updated the tracker instances in http://svn.python.org/view?view=revision&revision=88930 and the devguide.
docs.python.org is already ok, but www.python.org still needs to be updated.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13794>
_______________________________________

From amauryfa at gmail.com  Tue Jan 17 08:26:38 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 08:26:38 +0100
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
References: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <CAGmFidYg-YEKjbeGgZK5L0+4KnKk7gAX2f4Z2ShBd1T=WAW32A@mail.gmail.com>

Le 16 janv. 2012 00:09, "Kay Hayen" <report at bugs.python.org> a ?crit :
> I am the author of the Python compiler Nuitka. It has the ability to
immediately execute the created executable file. For that I am using
"os.execl" to immediately replace the compiler and run the freshly created
binary instead.
>
> This worked well so far, but as of late, I am also checking the exit
codes, and it turns out that even for failing programs, the exit code is
"0" on Windows, even though the compiled binary is exiting with "1".

This is expected behavior on Windows: exec does not replace the existing
process like with Unix, it creates a new process and exit(0) the caller.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-bugs-list/attachments/20120117/b83f63fd/attachment.html>

From report at bugs.python.org  Tue Jan 17 08:26:41 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 07:26:41 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <CAGmFidYg-YEKjbeGgZK5L0+4KnKk7gAX2f4Z2ShBd1T=WAW32A@mail.gmail.com>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

immediately execute the created executable file. For that I am using
"os.execl" to immediately replace the compiler and run the freshly created
binary instead.
>
> This worked well so far, but as of late, I am also checking the exit
codes, and it turns out that even for failing programs, the exit code is
"0" on Windows, even though the compiled binary is exiting with "1".

This is expected behavior on Windows: exec does not replace the existing
process like with Unix, it creates a new process and exit(0) the caller.

----------
nosy: +Amaury.Forgeot.d'Arc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Tue Jan 17 09:07:04 2012
From: report at bugs.python.org (Garrett Cooper)
Date: Tue, 17 Jan 2012 08:07:04 +0000
Subject: [issue13500] Hitting EOF gets cmd.py into a infinite EOF on return
	loop
In-Reply-To: <1322578601.01.0.207872314852.issue13500@psf.upfronthosting.co.za>
Message-ID: <1326787624.96.0.385803871303.issue13500@psf.upfronthosting.co.za>


Garrett Cooper <yanegomi at gmail.com> added the comment:

Here's a unittest patch for the py3k branch.

{{{
1 items passed all tests:
  32 tests in test.test_cmd.samplecmdclass
32 tests in 19 items.
32 passed and 0 failed.
Test passed.
doctest (test.test_cmd) ... 32 tests with zero failures
test_file_with_missing_final_nl (__main__.TestAlternateInput) ... ok
test_input_reset_at_EOF (__main__.TestAlternateInput) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
}}}

----------
Added file: http://bugs.python.org/file24257/python-issue13500-test.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13500>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:00:50 2012
From: report at bugs.python.org (Mark Diekhans)
Date: Tue, 17 Jan 2012 09:00:50 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1326737812.66.0.739021077379.issue12600@psf.upfronthosting.co.za>
Message-ID: <20245.14525.902239.978965@osprey.kermodei.com>


Mark Diekhans <markd at kermodei.com> added the comment:

Allowing loadTestsFromTestCase() to take either a testCaseClass
or a (testCaseClass, param) tuple, where the param is then past
to the __init__ function might do the trick.  One param is sufficient, since it can
be a container for any number of params. This allows more fields
to be added to the tuple for some future, unforeseen need.

An container object for class and parameters would be a bit more
structured than a tuple.

A factory function would be the most flexible, however it seems
to go against the class introspection for discovering tests.
Then again, I don't know the code very well.

R. David Murray <report at bugs.python.org> writes:
> 
> R. David Murray <rdmurray at bitdance.com> added the comment:
> 
> Maybe we could add a recipe for doing this to the load_tests docs?
> 
> I don't think that load_tests is going to be more readable, though, since it doesn't allow you to put the parameterization next to the class you are parameterizing (unless you do some additional hackery).
> 
> But yes, if anything else is done a concrete API proposal is the first requirement.
> 
> ----------
> 
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue12600>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:07:17 2012
From: report at bugs.python.org (Cherniavsky Beni)
Date: Tue, 17 Jan 2012 09:07:17 +0000
Subject: [issue8285] IDLE not smart indenting correctly in nested statements
In-Reply-To: <1270157080.33.0.1290049865.issue8285@psf.upfronthosting.co.za>
Message-ID: <1326791237.5.0.236131485944.issue8285@psf.upfronthosting.co.za>


Cherniavsky Beni <beni at google.com> added the comment:

Mark: customizing tabs to be anything but 8 spaces is inadvisable with Python, because Python always parses them as 8.
Sooner or later one would mix tabs and spaces and the result would be really painful to debug.

----------
nosy: +cben

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8285>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:28:41 2012
From: report at bugs.python.org (Kay Hayen)
Date: Tue, 17 Jan 2012 09:28:41 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <1326792521.14.0.829984655277.issue13792@psf.upfronthosting.co.za>


Kay Hayen <kayhayen at gmx.de> added the comment:

Does the Python standard library not offer anything that does replace with current process code with another? I checked with subprocess, and admittedly it's not that. Does Win32 API offer nothing for that?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:44:03 2012
From: report at bugs.python.org (Tal Einat)
Date: Tue, 17 Jan 2012 09:44:03 +0000
Subject: [issue8285] IDLE not smart indenting correctly in nested statements
In-Reply-To: <1270157080.33.0.1290049865.issue8285@psf.upfronthosting.co.za>
Message-ID: <1326793443.36.0.887849395836.issue8285@psf.upfronthosting.co.za>


Changes by Tal Einat <taleinat at gmail.com>:


----------
nosy:  -taleinat

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8285>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:48:37 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Tue, 17 Jan 2012 09:48:37 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
Message-ID: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>


New submission from Martin H?cker <spamfaenger at gmx.de>:

Code that uses higher order methods is often the clearest description of what you want to do. However since the higher order methods in python (filter, map, reduce) are free functions and aren't available on collection classes as methods, using them creates really hard to read code.

For example, if I want to get an attribute out of an array of objects and concatenate the results it's two steps, get the attributes out of the object, then concatenate the results.

Without higher order methods its like this. Uses three lines, intermediate variable is quite clear, but hard to compose and hard to abstract out parts of it.

 self.questions = []
 for topic in self.topics:
   self.questions.append(topic.questions)

if I want to do it with higher order functions there's really two ways, do it in one step with reduce:
 self.questions = reduce(lambda memo, topic: memo + topic.questions, self.topics, [])

Or use two steps with the operator module
 self.questions = reduce(operator.add, map(operator.attrgetter('questions'), self.topics), [])

Of these thee first still couples two steps into one lambda, while the second one decoples everything nicely but is a total train wreck to read, as you need to constantly jump back and forth in the line to understand what it does.

Having map and reduce defined on collections would not only make it drastically shorter but also allows you to read it from front to back in one go. (Ok, there is still the caveat that the assignment is in the front instead of at the end, but hey)

 self.questions = self.topics.map(attrgetter('questions')).reduce(add, [])

That would be nicely separated into individual steps that exactly describe what each step is actually doing, is easy to abstract over and actually very succinct.

----------
messages: 151435
nosy: dwt
priority: normal
severity: normal
status: open
title: Python library structure creates hard to read code when using higher order functions
type: enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Tue Jan 17 10:55:38 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 09:55:38 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326794138.45.0.157088847775.issue12415@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

+The documentation sources are part of the main :ref:`CPython Mercurial
+repository <setup>`.

I think "documentation sources" is a bit vague.  If the goal is enable people to find those files, IMHO it would be better to state explicitly that there are bunch of rst files in the Doc/ directory that comes with cpython and that they are used to build the online documentation.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:00:56 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 10:00:56 +0000
Subject: [issue13799] Base 16 should be hexadecimal in Unicode HOWTO
In-Reply-To: <1326718246.49.0.980962494093.issue13799@psf.upfronthosting.co.za>
Message-ID: <1326794456.65.0.679062743257.issue13799@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Do you mean the "base 16" in this sentence: "A code point is an integer value, usually denoted in base 16."?
Why would "hexadecimal" be better than "base 16"?

----------
assignee: docs at python -> ezio.melotti
nosy: +ezio.melotti
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13799>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:04:29 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 10:04:29 +0000
Subject: [issue13792] The "os.execl" call doesn't give programs exit code
In-Reply-To: <1326668704.55.0.266446288638.issue13792@psf.upfronthosting.co.za>
Message-ID: <1326794669.4.0.279471404011.issue13792@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

No. On Windows the only way to start a new executable is to create a new process (with the CreateProcess function, which all spawn* and exec* functions ultimately call), and this yields a new PID.
This is a fundamental difference with unix, where the only way to create a process is to clone the current one with fork().

And yes, this makes launchers more difficult to write on Windows.  For a discussion see the paragraph "Process Launching" in http://www.python.org/dev/peps/pep-0397/

----------
nosy: +amaury.forgeotdarc
resolution:  -> invalid
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13792>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:21:52 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Tue, 17 Jan 2012 10:21:52 +0000
Subject: [issue13805] [].sort() should return self
Message-ID: <1326795711.98.0.609923971725.issue13805@psf.upfronthosting.co.za>


New submission from Martin H?cker <spamfaenger at gmx.de>:

[].sort() returns None which means you can't chain it.

So for example someDict.keys().sort()[0] doesn't work but you have to use sorted(someDict.keys())[0] instead which is harder to read as you have to read the line not from the beginning to the end but jump back and forth in it to understand it (which gets progressively harder as the individual parts of it get longer / more complex).

----------
messages: 151439
nosy: dwt
priority: normal
severity: normal
status: open
title: [].sort() should return self
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:22:08 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Tue, 17 Jan 2012 10:22:08 +0000
Subject: [issue13805] [].sort() should return self
In-Reply-To: <1326795711.98.0.609923971725.issue13805@psf.upfronthosting.co.za>
Message-ID: <1326795728.41.0.898580924928.issue13805@psf.upfronthosting.co.za>


Martin H?cker <spamfaenger at gmx.de> added the comment:

It really should return self.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:24:36 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 10:24:36 +0000
Subject: [issue13805] [].sort() should return self
In-Reply-To: <1326795711.98.0.609923971725.issue13805@psf.upfronthosting.co.za>
Message-ID: <1326795876.54.0.311177490071.issue13805@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

This is by design.
Methods that mutate the object return None.
Methods that create a new object return the new object.
list.sort sorts the list in place, so it returns None.

----------
nosy: +ezio.melotti
resolution:  -> rejected
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 11:28:59 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 10:28:59 +0000
Subject: [issue13805] [].sort() should return self
In-Reply-To: <1326795711.98.0.609923971725.issue13805@psf.upfronthosting.co.za>
Message-ID: <1326796139.97.0.395182807055.issue13805@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

See also http://docs.python.org/faq/design.html#why-doesn-t-list-sort-return-the-sorted-list

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 12:28:09 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 17 Jan 2012 11:28:09 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326799689.31.0.093883939386.issue12600@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Mark, please stop discussing per-run parameters in this issue. Those are NOT the kind of parameters we're talking about (and are easily handled via a global settings module, anyway, the exact same way you can handle process global settings for *any* kind of application). If you want to continue discussing that topic, please create a new issue.

This issue is about parameterisation of a TestCase *within* a run, where the same set of tests is run against multiple sets of parameters, and the test framework provides good support for collating and presenting those results in a meaningful fashion.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 12:33:16 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 17 Jan 2012 11:33:16 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326799996.01.0.606666939523.issue12600@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Back on topic...

While I can see the advantage of parameterisation at the level of individual tests, I'm not at all clear on the benefits at the TestCase level.

For CPython's own test suite, if we want to share tests amongst multiple test cases, we just use ordinary inheritance. You get parameterisation pretty much for free with that approach:

class _BaseTest(object):
    # Tests go here
    # setUp and tearDown often go here, too

class FooTestCase(_BaseTest, TestCase):
    # Parameter settings go here

class BarTestCase(_BaseTest, TestCase):
    # Parameter settings go here

If you want to get data-driven about it, you can also do dynamic TestCase creation based on a sequence of parameter sets.

So, absent a compelling explanation for why the ordinary inheritance mechanisms aren't adequate, I'd be in favour of closing this one.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 12:41:55 2012
From: report at bugs.python.org (=?utf-8?q?Kristj=C3=A1n_Valur_J=C3=B3nsson?=)
Date: Tue, 17 Jan 2012 11:41:55 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326800515.64.0.41109182013.issue10181@psf.upfronthosting.co.za>


Kristj?n Valur J?nsson <kristjan at ccpgames.com> added the comment:

Thanks Nick.  Looking through this discussion it looks as though you encountered all the confusing bits that I ran into (dup_buffer, ownership issues, reference counting and so forth.) Good work.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Tue Jan 17 12:41:58 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Tue, 17 Jan 2012 11:41:58 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326800518.94.0.242988370916.issue13781@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Attached is a fix for 3.x.

----------
keywords: +patch
Added file: http://bugs.python.org/file24258/gzip-fdopen.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Tue Jan 17 12:53:19 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 17 Jan 2012 11:53:19 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1326720225.98.0.599117190399.issue10181@psf.upfronthosting.co.za>
Message-ID: <20120117115317.GA9679@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Thanks for the comments. Most of them should be easy to fix.

Nick Coghlan <report at bugs.python.org> wrote:
> [...] expose the Py_buffer len field as "memoryview.size" instead of
> "memoryview.len" (to reduce confusion with "len(memoryview)" and
> to encourage a conceptual link with sys.getsizeof()).

I agree with this. The best thing is probably to have both versions
work in 3.3 and remove memoryview.len in 3.4 (or later).

> As noted in the review, I also think fixing #12384 should be fairly
> straightforward and it would be nice to have that working when the
> change is applied.

tobytes() currently calls PyBuffer_ToContiguous(..., 'C') from
Objects/abstract.c. Since the function repeatedly calls
PyBuffer_GetPointer(), it has this comment:

    /* XXX : This is not going to be the fastest code in the world
             several optimizations are possible.
     */

So if the fix for tobytes() should go in, I'd like to use the
copy_buffer() function from _testbuffer.c. PyBuffer_ToContiguous()
would need to be fixed separately.

test_buffer.py already asserts that for each result, result.tolist()
and result.tobytes() match the results obtained from PyBuffer_GetPointer(indices)
(See: test_buffer.py:779), so I'm not worried about using the same
implementation in the tests as in the production code.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Tue Jan 17 13:21:33 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 12:21:33 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326766242.22.0.74313150893.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwaB1ETQOZiLXodJf=5DmPaHcm5FuAkKRuCp6XsPcLfEbw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 8: the whole test suite now pass successfully.

The remaining question is if CryptoGen should be used instead of the
weak LCG initialized by gettimeofday() and getpid(). According to
Martin von Loewis, we must link statically Python to advapi32.dll. It
should speed up the startup.

----------
Added file: http://bugs.python.org/file24259/random-8.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -460,6 +460,13 @@ These environment variables influence Py
    option.
 
 
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for the Unicode randomized hash:
+   number in range [0; 4294967295]. The value 0 disables the Unicode randomized
+   hash.
+
+
 .. envvar:: PYTHONIOENCODING
 
    If this is set before running the interpreter, it overrides the encoding used
diff --git a/Include/pythonrun.h b/Include/pythonrun.h
--- a/Include/pythonrun.h
+++ b/Include/pythonrun.h
@@ -246,6 +246,8 @@ typedef void (*PyOS_sighandler_t)(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -376,6 +376,12 @@ typedef struct {
 PyAPI_DATA(PyTypeObject) PyUnicode_Type;
 PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type;
 
+typedef struct {
+    Py_hash_t prefix;
+    Py_hash_t suffix;
+} _Py_unicode_hash_secret_t;
+PyAPI_DATA(_Py_unicode_hash_secret_t) _Py_unicode_hash_secret;
+
 #define PyUnicode_Check(op) \
                  PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_UNICODE_SUBCLASS)
 #define PyUnicode_CheckExact(op) (Py_TYPE(op) == &PyUnicode_Type)
diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py
--- a/Lib/json/__init__.py
+++ b/Lib/json/__init__.py
@@ -31,7 +31,9 @@ Encoding basic Python object hierarchies
 Compact encoding::
 
     >>> import json
-    >>> json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',', ':'))
+    >>> from collections import OrderedDict
+    >>> mydict = OrderedDict([('4', 5), ('6', 7)])
+    >>> json.dumps([1,2,3,mydict], separators=(',', ':'))
     '[1,2,3,{"4":5,"6":7}]'
 
 Pretty printing::
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -761,23 +761,6 @@ try:
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -311,8 +311,11 @@ class MainProgram:
                          'package_data', 'extra_files'):
                 if not(name in self.data and self.data[name]):
                     continue
-                fp.write('%s = %s\n'
-                         % (name, '\n    '.join(self.data[name]).strip()))
+                data = self.data[name]
+                if name == "extra_files":
+                    data.sort()
+                data = '\n    '.join(data).strip()
+                fp.write('%s = %s\n' % (name, data))
             fp.write('\nresources =\n')
             for src, dest in self.data['resources']:
                 fp.write('    %s = %s\n' % (src, dest))
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -150,16 +150,16 @@ class CreateTestCase(support.TempdirMana
                 mymodule
             scripts = my_script
                 bin/run
-            extra_files = Martinique/Lamentin/dady
+            extra_files = Alexander
+                Flora
+                Martinique/Lamentin/bro
+                Martinique/Lamentin/dady
                 Martinique/Lamentin/mumy
                 Martinique/Lamentin/sys
-                Martinique/Lamentin/bro
+                Pom
+                README
+                pyxfoil/fengine.so
                 setup.py
-                README
-                Pom
-                Flora
-                Alexander
-                pyxfoil/fengine.so
 
             resources =
                 README.rst = {doc}
@@ -217,8 +217,8 @@ ho, baby!
 
             [files]
             packages = pyxfoil
-            extra_files = pyxfoil/fengine.so
-                pyxfoil/babar.so
+            extra_files = pyxfoil/babar.so
+                pyxfoil/fengine.so
 
             resources =
                 README.rst = {doc}
diff --git a/Lib/test/mapping_tests.py b/Lib/test/mapping_tests.py
--- a/Lib/test/mapping_tests.py
+++ b/Lib/test/mapping_tests.py
@@ -14,7 +14,7 @@ class BasicTestMappingProtocol(unittest.
     def _reference(self):
         """Return a dictionary of values which are invariant by storage
         in the object under test."""
-        return {1:2, "key1":"value1", "key2":(1,2,3)}
+        return {"1": "2", "key1":"value1", "key2":(1,2,3)}
     def _empty_mapping(self):
         """Return an empty mapping object"""
         return self.type2test()
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -559,6 +559,11 @@ def main(tests=None, testdir=None, verbo
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(selected)
diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -597,7 +597,6 @@ class BuiltinTest(unittest.TestCase):
         self.assertEqual(hash(1), hash(1))
         self.assertEqual(hash(1), hash(1.0))
         hash('spam')
-        self.assertEqual(hash('spam'), hash(b'spam'))
         hash((0,1,2,3))
         def f(): pass
         self.assertRaises(TypeError, hash, [])
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -6,6 +6,7 @@ import unittest
 import sys
 import dis
 import io
+from test.script_helper import assert_python_ok
 
 class _C:
     def __init__(self, x):
@@ -322,12 +323,6 @@ Variable names:
    0: x""" % (('Formatted details of methods, functions, or code.', '   6: None\n')
               if sys.flags.optimize < 2 else (None, ''))
 
- at staticmethod
-def tricky(x, y, z=True, *args, c, d, e=[], **kwds):
-    def f(c=c):
-        print(x, y, z, c, d, e, f)
-    yield x, y, z, c, d, e, f
-
 code_info_tricky = """\
 Name:              tricky
 Filename:          (.*)
@@ -357,8 +352,6 @@ Cell variables:
    4: x
    5: z"""
 
-co_tricky_nested_f = tricky.__func__.__code__.co_consts[1]
-
 code_info_tricky_nested_f = """\
 Name:              f
 Filename:          (.*)
@@ -426,13 +419,36 @@ Names:
 class CodeInfoTests(unittest.TestCase):
     test_pairs = [
       (dis.code_info, code_info_code_info),
-      (tricky, code_info_tricky),
-      (co_tricky_nested_f, code_info_tricky_nested_f),
       (expr_str, code_info_expr_str),
       (simple_stmt_str, code_info_simple_stmt_str),
       (compound_stmt_str, code_info_compound_stmt_str),
     ]
 
+    def get_tricky_output(self, func):
+        code = """
+import dis
+
+ at staticmethod
+def tricky(x, y, z=True, *args, c, d, e=[], **kwds):
+    def f(c=c):
+        print(x, y, z, c, d, e, f)
+    yield x, y, z, c, d, e, f
+
+%s
+        """ % func
+        err, stdout, stderr = assert_python_ok('-c', code, PYTHONHASHSEED='0')
+        return (stdout + stderr).decode('ascii', 'surrogateescape')
+
+    def test_tricky(self):
+        stdout = self.get_tricky_output('dis.show_code(tricky)')
+        self.assertRegex(stdout, code_info_tricky)
+
+    def test_tricky_nested(self):
+        stdout = self.get_tricky_output(
+            'co_tricky_nested_f = tricky.__func__.__code__.co_consts[1]; '
+            'dis.show_code(co_tricky_nested_f)')
+        self.assertRegex(stdout, code_info_tricky_nested_f)
+
     def test_code_info(self):
         self.maxDiff = 1000
         for x, expected in self.test_pairs:
diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py
--- a/Lib/test/test_gdb.py
+++ b/Lib/test/test_gdb.py
@@ -52,13 +52,15 @@ class DebuggerTests(unittest.TestCase):
 
     """Test that the debugger can debug Python."""
 
-    def run_gdb(self, *args):
+    def run_gdb(self, *args, **env):
         """Runs gdb with the command line given by *args.
 
         Returns its stdout, stderr
         """
+        if not env:
+            env = None
         out, err = subprocess.Popen(
-            args, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+            args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env,
             ).communicate()
         return out.decode('utf-8', 'replace'), err.decode('utf-8', 'replace')
 
@@ -118,7 +120,7 @@ class DebuggerTests(unittest.TestCase):
         # print ' '.join(args)
 
         # Use "args" to invoke gdb, capturing stdout, stderr:
-        out, err = self.run_gdb(*args)
+        out, err = self.run_gdb(*args, PYTHONHASHSEED='0')
 
         # Ignore some noise on stderr due to the pending breakpoint:
         err = err.replace('Function "%s" not defined.\n' % breakpoint, '')
@@ -207,7 +209,8 @@ class PrettyPrintTests(DebuggerTests):
         'Verify the pretty-printing of dictionaries'
         self.assertGdbRepr({})
         self.assertGdbRepr({'foo': 'bar'})
-        self.assertGdbRepr({'foo': 'bar', 'douglas':42})
+        self.assertGdbRepr({'foo': 'bar', 'douglas': 42},
+                           "{'foo': 'bar', 'douglas': 42}")
 
     def test_lists(self):
         'Verify the pretty-printing of lists'
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -799,7 +799,6 @@ class TestGetcallargsFunctions(unittest.
             self.assertEqualException(f, '2, 3, 4')
             self.assertEqualException(f, '1, 2, 3, a=1')
             self.assertEqualException(f, '2, 3, 4, c=5')
-            self.assertEqualException(f, '2, 3, 4, a=1, c=5')
             # f got an unexpected keyword argument
             self.assertEqualException(f, 'c=2')
             self.assertEqualException(f, '2, c=3')
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -12,6 +12,7 @@ import subprocess
 import time
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 import contextlib
 import mmap
 import platform
@@ -630,14 +631,32 @@ class DevNullTests(unittest.TestCase):
             self.assertEqual(f.read(), b'')
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 @contextlib.contextmanager
 def _execvpe_mockup(defpath=None):
diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py
--- a/Lib/test/test_set.py
+++ b/Lib/test/test_set.py
@@ -891,7 +891,17 @@ class TestBasicOpsString(TestBasicOps):
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +926,22 @@ class TestBasicOpsMixedStringBytes(TestB
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
 
     def tearDown(self):
         self._warning_filters.__exit__(None, None, None)
 
+    def test_repr(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
+
 #==============================================================================
 
 def baditer():
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -11,8 +11,11 @@ import struct
 import sys
 import unittest
 import warnings
+from test.script_helper import assert_python_ok
 from test import support, string_tests
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 # Error handling (bad decoder return)
 def search_function(encoding):
     def decode1(input, errors="strict"):
@@ -1943,6 +1946,42 @@ class StringModuleTest(unittest.TestCase
         self.assertRaises(TypeError, _string.formatter_field_name_split, 1)
 
 
+class HashTest(unittest.TestCase):
+    def get_hash(self, text, seed=None):
+        env = {}
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'print(hash(%r))' % text,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            hash_empty = 1453079729188098211
+        else:
+            hash_empty = -1600925533
+        self.assertEqual(self.get_hash("abc", 0), hash_empty)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        self.assertEqual(self.get_hash("abc", 42), h)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        run1 = self.get_hash("abc")
+        run2 = self.get_hash("abc")
+        self.assertNotEqual(run1, run2)
+
 def test_main():
     support.run_unittest(__name__)
 
diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -1,16 +1,17 @@
 """Regresssion tests for urllib"""
 
-import urllib.parse
-import urllib.request
-import urllib.error
+import collections
+import email.message
 import http.client
-import email.message
 import io
-import unittest
-from test import support
 import os
 import sys
 import tempfile
+from test import support
+import unittest
+import urllib.error
+import urllib.parse
+import urllib.request
 
 from base64 import b64encode
 
@@ -950,8 +951,9 @@ class urlencode_Tests(unittest.TestCase)
         self.assertEqual("a=1&a=2", urllib.parse.urlencode({"a": [1, 2]}, True))
         self.assertEqual("a=None&a=a",
                          urllib.parse.urlencode({"a": [None, "a"]}, True))
+        data = collections.OrderedDict([("a", 1), ("b", 1)])
         self.assertEqual("a=a&a=b",
-                         urllib.parse.urlencode({"a": {"a": 1, "b": 1}}, True))
+                         urllib.parse.urlencode({"a": data}, True))
 
     def test_urlencode_encoding(self):
         # ASCII encoding. Expect %3F with errors="replace'
diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py
--- a/Lib/test/test_urlparse.py
+++ b/Lib/test/test_urlparse.py
@@ -1,6 +1,7 @@
 #! /usr/bin/env python3
 
 from test import support
+import collections
 import unittest
 import urllib.parse
 
@@ -768,7 +769,10 @@ class UrlParseTestCase(unittest.TestCase
     def test_urlencode_sequences(self):
         # Other tests incidentally urlencode things; test non-covered cases:
         # Sequence and object values.
-        result = urllib.parse.urlencode({'a': [1, 2], 'b': (3, 4, 5)}, True)
+        data = collections.OrderedDict()
+        data['a'] = [1, 2]
+        data['b'] = (3, 4, 5)
+        result = urllib.parse.urlencode(data, True)
         self.assertEqual(result, 'a=1&a=2&b=3&b=4&b=5')
 
         class Trivial:
diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py
--- a/Lib/tkinter/test/test_ttk/test_functions.py
+++ b/Lib/tkinter/test/test_ttk/test_functions.py
@@ -143,7 +143,7 @@ class InternalFunctionsTest(unittest.Tes
             ('a', 'b', 'c')), ("test {a b} c", ()))
         # state spec and options
         self.assertEqual(ttk._format_elemcreate('image', False, 'test',
-            ('a', 'b'), a='x', b='y'), ("test a b", ("-a", "x", "-b", "y")))
+            ('a', 'b'), a='x'), ("test a b", ("-a", "x")))
         # format returned values as a tcl script
         # state spec with multiple states and an option with a multivalue
         self.assertEqual(ttk._format_elemcreate('image', True, 'test',
diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -322,6 +322,7 @@ PYTHON_OBJS=	\
 		Python/pystate.o \
 		Python/pythonrun.o \
 		Python/pytime.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -9329,81 +9329,36 @@ posix_getloadavg(PyObject *self, PyObjec
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
+Return n pseudo-random bytes.");
 
 static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
 
     /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
 
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandleW(L"advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
@@ -9445,42 +9400,6 @@ device_encoding(PyObject *self, PyObject
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
-    }
-    return result;
-}
-#endif
-
 #ifdef HAVE_SETRESUID
 PyDoc_STRVAR(posix_setresuid__doc__,
 "setresuid(ruid, euid, suid)\n\n\
@@ -10887,12 +10806,7 @@ static PyMethodDef posix_methods[] = {
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
 #ifdef HAVE_SETRESUID
     {"setresuid",       posix_setresuid, METH_VARARGS, posix_setresuid__doc__},
 #endif
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -166,6 +166,8 @@ extern "C" {
             *_to++ = (to_type) *_iter++;                \
     } while (0)
 
+_Py_unicode_hash_secret_t _Py_unicode_hash_secret;
+
 /* This dictionary holds all interned unicode strings.  Note that references
    to strings in this dictionary are *not* counted in the string's ob_refcnt.
    When the interned string reaches a refcnt of 0 the string deallocation
@@ -11193,8 +11195,6 @@ unicode_getitem(PyObject *self, Py_ssize
     return PyUnicode_FromOrdinal(ch);
 }
 
-/* Believe it or not, this produces the same value for ASCII strings
-   as bytes_hash(). */
 static Py_hash_t
 unicode_hash(PyObject *self)
 {
@@ -11206,10 +11206,23 @@ unicode_hash(PyObject *self)
     if (PyUnicode_READY(self) == -1)
         return -1;
     len = PyUnicode_GET_LENGTH(self);
+    if (len == 0) {
+        _PyUnicode_HASH(self) = 0;
+        return 0;
+    }
+
+    /* Issue #13703: add 2 x sizeof(Py_hash_t) random bytes (prefix and suffix)
+       to the output of hash(str) to protect Python against the hash collision
+       attack on dictionary. Without these random bytes, an attacker can
+       computes N strings with the same hash value to exploit to worst case of
+       a dict lookup, O(n^2) complexity, to cause a denial of service. See
+       oCERT advisory for the details:
+       http://www.ocert.org/advisories/ocert-2011-003.html */
+    x = _Py_unicode_hash_secret.prefix;
 
     /* The hash function as a macro, gets expanded three times below. */
 #define HASH(P) \
-    x = (Py_uhash_t)*P << 7; \
+    x ^= (Py_uhash_t)*P << 7; \
     while (--len >= 0) \
         x = (_PyHASH_MULTIPLIER*x) ^ (Py_uhash_t)*P++;
 
@@ -11234,6 +11247,7 @@ unicode_hash(PyObject *self)
     }
     }
     x ^= (Py_uhash_t)PyUnicode_GET_LENGTH(self);
+    x ^= _Py_unicode_hash_secret.suffix;
 
     if (x == -1)
         x = -2;
diff --git a/PCbuild/pythoncore.vcproj b/PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj
+++ b/PCbuild/pythoncore.vcproj
@@ -1891,6 +1891,10 @@
 				>
 			</File>
 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
+			<File
 				RelativePath="..\Python\structmember.c"
 				>
 			</File>
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -71,6 +71,7 @@ extern int _PyUnicode_Init(void);
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 extern int _PyFaulthandler_Init(void);
 extern void _PyFaulthandler_Fini(void);
 
@@ -219,6 +220,8 @@ Py_InitializeEx(int install_sigs)
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
 
+    _PyRandom_Init();
+
     interp = PyInterpreterState_New();
     if (interp == NULL)
         Py_FatalError("Py_Initialize: can't make first interpreter");
diff --git a/Python/random.c b/Python/random.c
new file mode 100644
--- /dev/null
+++ b/Python/random.c
@@ -0,0 +1,268 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Fail to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Fail to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Fail to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Fail to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Fail to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_unicode_hash_secret;
+    Py_ssize_t secret_size = sizeof _Py_unicode_hash_secret;
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized Unicode hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+

From report at bugs.python.org  Tue Jan 17 13:36:34 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 12:36:34 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwaB1ETQOZiLXodJf=5DmPaHcm5FuAkKRuCp6XsPcLfEbw@mail.gmail.com>
Message-ID: <CAMpsgwZsh1Nnhcaau8TNHcHPkjpxbjRvXeDbzAd9CXRZF=17DQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Hum, test_runpy fails something with a segfault and/or a recursion
limit because of my hack to rerun regrtest.py to set PYTHONHASHSEED
environment variable. The fork should be defined if main() of
regrtest.py is called directly. Example:

diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -258,7 +258,7 @@ def main(tests=None, testdir=None, verbo
          findleaks=False, use_resources=None, trace=False, coverdir='coverage',
          runleaks=False, huntrleaks=False, verbose2=False, print_slow=False,
          random_seed=None, use_mp=None, verbose3=False, forever=False,
-         header=False, failfast=False, match_tests=None):
+         header=False, failfast=False, match_tests=None, allow_fork=False):
     """Execute a test suite.

     This also parses command-line options and modifies its behavior
@@ -559,6 +559,11 @@ def main(tests=None, testdir=None, verbo
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if (not hashseed and allow_fork):
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(selected)
@@ -1809,4 +1814,4 @@ if __name__ == '__main__':
     # change the CWD, the original CWD will be used. The original CWD is
     # available from support.SAVEDCWD.
     with support.temp_cwd(TESTCWD, quiet=True):
-        main()
+        main(allow_fork=True)

As Antoine wrote on IRC, regrtest.py should be changed later.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 13:57:53 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 17 Jan 2012 12:57:53 +0000
Subject: [issue12600] Support parameterized TestCases in unittest
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326805073.07.0.696404690579.issue12600@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

I'd still like to see a recipe for creating parameterized test cases via load_tests added to the docs.  It may be relatively obvious how to do it once you think of it, but it isn't obvious to a relative newcomer that you *can* do it, and it would make a great example of how load_tests can be used.  (The current example is very trivial since it just re-implements the default behavior, and while that's useful, it doesn't really demonstrate the power of load_tests).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 13:58:03 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 17 Jan 2012 12:58:03 +0000
Subject: [issue13411] Hashable memoryviews
In-Reply-To: <1321395643.62.0.259606214561.issue13411@psf.upfronthosting.co.za>
Message-ID: <1326805083.48.0.643606185973.issue13411@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
dependencies: +Problems with Py_buffer management in memoryobject.c (and elsewhere?)
resolution: fixed -> remind
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13411>
_______________________________________

From report at bugs.python.org  Tue Jan 17 13:59:40 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 17 Jan 2012 12:59:40 +0000
Subject: [issue13411] Hashable memoryviews
In-Reply-To: <1321395643.62.0.259606214561.issue13411@psf.upfronthosting.co.za>
Message-ID: <1326805180.96.0.176793203491.issue13411@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Reopening as a reminder that it isn't fixed yet in http://hg.python.org/features/pep-3118 .

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13411>
_______________________________________

From report at bugs.python.org  Tue Jan 17 14:01:28 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 17 Jan 2012 13:01:28 +0000
Subject: [issue12600] Add example of using load_tests to parameterise Test
	Cases
In-Reply-To: <1311226769.48.0.369455670372.issue12600@psf.upfronthosting.co.za>
Message-ID: <1326805288.32.0.965098042065.issue12600@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I agree with David, so switching this over to a docs enhancement request.

----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python
title: Support parameterized TestCases in unittest -> Add example of using load_tests to parameterise Test Cases
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12600>
_______________________________________

From report at bugs.python.org  Tue Jan 17 14:58:46 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Tue, 17 Jan 2012 13:58:46 +0000
Subject: [issue13589] Aifc low level serialization primitives fix
In-Reply-To: <1323714686.93.0.781541085274.issue13589@psf.upfronthosting.co.za>
Message-ID: <1326808726.41.0.489440102278.issue13589@psf.upfronthosting.co.za>


Oleg Plakhotnyuk <olegus8 at gmail.com> added the comment:

I have absolutely no idea :-)
I just covered every line of code with tests. Some bugs prevented me from do it, so I fixed them.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13589>
_______________________________________

From report at bugs.python.org  Tue Jan 17 15:01:39 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 14:01:39 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326808899.31.0.950084712506.issue13804@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Did you consider list comprehension?

self.questions = sum((topic.questions for topic in self.topics), [])

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Tue Jan 17 15:18:45 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 17 Jan 2012 14:18:45 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <1326809925.49.0.460193305693.issue13803@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Proposed patch for Python 2.7:

"""
--- util.py.old 2011-12-12 01:34:04.412234183 +0100
+++ util.py     2012-01-17 15:15:23.262257886 +0100
@@ -12,6 +12,7 @@
 from distutils.spawn import spawn
 from distutils import log
 from distutils.errors import DistutilsByteCompileError
+import platform
 
 def get_platform ():
     """Return a string that identifies the current platform.  This is used
@@ -76,6 +77,7 @@
         if release[0] >= "5":           # SunOS 5 == Solaris 2
             osname = "solaris"
             release = "%d.%s" % (int(release[0]) - 3, release[2:])
+            machine += ".%s" %platform.architecture()[0]
         # fall through to standard osname-release-machine representation
     elif osname[:4] == "irix":              # could be "irix64"!
         return "%s-%s" % (osname, release)
"""

So now the directory is named like "lib.solaris-2.10-i86pc.32bit-2.7".

Please, review.

I will commit it to 2.7, 3.1, 3.2 and 3.3 in a few days.

----------
assignee: tarek -> jcea
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Tue Jan 17 15:22:24 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 17 Jan 2012 14:22:24 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326810144.2.0.698362369605.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Well, actually, it's the only correct way to write it (i-th). This is a simple orthographical error that was made. Ezio, please fix those two additional typos. And Georg, stop being such a wise-ass.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Tue Jan 17 15:40:38 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Tue, 17 Jan 2012 14:40:38 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326811238.91.0.875562490994.issue13804@psf.upfronthosting.co.za>


Martin H?cker <spamfaenger at gmx.de> added the comment:

Yes - however it has the same problem as the higher order version in the python libraries that to read it you need to constantly jump back and forth in the line.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Tue Jan 17 16:11:25 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Tue, 17 Jan 2012 15:11:25 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
In-Reply-To: <CAO48Bk-ewBnvWp5Y_YCwiBgcjnUVZOWOQRCfYCpt__YpvZ-vcg@mail.gmail.com>
Message-ID: <CAHRxO8irhc-vMHx5oXQo+w+sSo6J+Kao2O0YeTGJNv9nER0eOQ@mail.gmail.com>


patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:

Hi,

Did you look at lxml (http://lxml.de)?

from lxml.builder import E
from lxml import etree

tree = etree.ElementTree(
    E.Hello(
        "Good morning!",
        E.World("How do you do", humour = "excellent"),
        "Fine",
        E.Goodbye(),
        ),
    )

print(etree.tostring(tree, pretty_print=True).decode())

# output, even more prettified

<Hello>
    Good morning!
    <World humour="excellent">
          How do you do
    </World>
          Fine
    <Goodbye/>
</Hello>

By the way, your Element enhancement is buggy, because all newly create
elements will share the same attrib dictionary (if attrib is not given).
Notice that Donald Duck will be sad; by the time we print even Hello is sad.

import xml.etree.ElementTree as etree

class Element(etree.Element):
    def __init__(self, tag, attrib={}, **extra):
        super().__init__(tag)
        self.tag = tag
        self.attrib = attrib
        self.attrib.update(extra)
        self.text = self.attrib.pop('text', None)
        self.tail = self.attrib.pop('tail', None)
        self._children = []

if __name__ == '__main__':

    test = Element('Hello',)
    test2 = Element('World',{'humour':'excelent'},text = 'How do you do',
tail="Fine")
    test3 = Element('Goodbye', humour='sad')
    test4 = Element('Donaldduck')
    test.append(test2)
    test.append(test3)
    test.append(test4)
    tree = etree.ElementTree(test)
    print(etree.tostring(test, encoding="utf-8", method="xml"))

<Hello humour="sad">
    <World humour="excelent">How do you do</World>Fine
    <Goodbye humour="sad" />
    <Donaldduck humour="sad" />
</Hello>'

The correct idiom would be:

    def __init__(self, tag, attrib=None, **extra):
        if attrib is None:
            attrib = {}
        super().__init__(tag)

Cheers,

Patrick

2012/1/16 Pedro Andres Aranda Gutierrez <report at bugs.python.org>

>
> Pedro Andres Aranda Gutierrez <paaguti at gmail.com> added the comment:
>
> Touch? :-)
> I was just frustrated because my XMLs never have tail or text as
> attributes and I wanted to have more compact code...
>
> On Mon, Jan 16, 2012 at 12:14 PM, patrick vrijlandt
> <report at bugs.python.org> wrote:
> >
> > patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:
> >
> > I agree the Element syntax is sometimes awkward.
> >
> > But how would you represent text or tail attributes within this enhanced
> element?
> > <animal name="cat" tail="yes"> comes to mind ...
> >
> > ----------
> > nosy: +patrick.vrijlandt
> >
> > _______________________________________
> > Python tracker <report at bugs.python.org>
> > <http://bugs.python.org/issue13796>
> > _______________________________________
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13796>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Tue Jan 17 16:23:45 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Tue, 17 Jan 2012 15:23:45 +0000
Subject: [issue13806] Audioop decompression frames size check fix
Message-ID: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>


New submission from Oleg Plakhotnyuk <olegus8 at gmail.com>:

According to documentation (http://docs.python.org/library/audioop.html), adpcm2lin, alaw2lin and ulaw2lin are using 'width' argument to represent output frames width. However, in audioop.c module there are checks that are raising exceptions if input frames length is not multiple of 'width'. I have replaced checking of 'len' to match 'size' with checking of 'len*size' to match 'size' in order to retain only basic length validity checks.

----------
components: Library (Lib)
files: audioop_size_check.patch
keywords: patch
messages: 151459
nosy: Oleg.Plakhotnyuk, ezio.melotti, sandro.tosi
priority: normal
severity: normal
status: open
title: Audioop decompression frames size check fix
type: behavior
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24260/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Tue Jan 17 16:23:59 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Tue, 17 Jan 2012 15:23:59 +0000
Subject: [issue13681] Aifc read compressed frames fix
In-Reply-To: <1325243830.12.0.0659769703479.issue13681@psf.upfronthosting.co.za>
Message-ID: <1326813839.94.0.912020420889.issue13681@psf.upfronthosting.co.za>


Changes by Oleg Plakhotnyuk <olegus8 at gmail.com>:


Removed file: http://bugs.python.org/file24112/aifc_compression.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13681>
_______________________________________

From report at bugs.python.org  Tue Jan 17 16:29:34 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Tue, 17 Jan 2012 15:29:34 +0000
Subject: [issue13681] Aifc read compressed frames fix
In-Reply-To: <1325243830.12.0.0659769703479.issue13681@psf.upfronthosting.co.za>
Message-ID: <1326814174.58.0.980843292383.issue13681@psf.upfronthosting.co.za>


Oleg Plakhotnyuk <olegus8 at gmail.com> added the comment:

Ok, I have opened issue 13806 with audioop fix alone. However, I cannot change current issue's dependencies to reflect that current issue depends on issue 13806. Could anyone do this for me please?

----------
Added file: http://bugs.python.org/file24261/aifc_compression.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13681>
_______________________________________

From report at bugs.python.org  Tue Jan 17 16:52:07 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 17 Jan 2012 15:52:07 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326815527.92.0.678129151244.issue12705@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

The attached patch fixes this be checking what is left in the input
buffer after parsing.  Anything but trailing whitespace and comments
triggers the exception.

Ignoring trailing whitespace and comments makes sense, but it is also 
somewhat required.  Trailing comments are common in doctests:

    >>> doctest.testfile('test_doctest.txt', raise_on_error=True)
    ... # doctest: +ELLIPSIS

and trailing whitespace is used by the codeop heuristics.

Here is an example of the new exception:

>>> compile('1 + 2\n3 + 4\n', '','single')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 1
    1 + 2
        ^
SyntaxError: multiple statements found while compiling a single statement

Tested on Fedora 15; no regressions.

----------
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24262/issue12705-0.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:00:14 2012
From: report at bugs.python.org (Thomas Ryschawy)
Date: Tue, 17 Jan 2012 16:00:14 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
Message-ID: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>


New submission from Thomas Ryschawy <ThomasOtto.Ryschawy at Emerson.com>:

It seems to be known that in case of a Windows GUI app that isn?t connected to a console sys.stderr can be None. See the Note on http://docs.python.org/py3k/library/sys.html: Under some conditions stdin, stdout and stderr as well as the original values __stdin__, __stdout__ and __stderr__ can be None. It is usually the case for Windows GUI apps that aren?t connected to a console and Python apps started with pythonw.
In combination with logging this leads to an issue similar to issue #5971. In my case sys.stderr is None and File: lib\logging\__init__.py, Class: Handler, Method: handleError raises AttributeError: 'NoneType' object has no attribute 'write'.
A 'simple' solution would be to check if sys.stderr is not None. In case it is None handleError must not call traceback.print_exception() and not directly write to sys.stderr.

----------
components: Library (Lib)
files: traceback.txt
messages: 151462
nosy: ThomasRyschawy
priority: normal
severity: normal
status: open
title: logging.Handler.handlerError() may raise AttributeError in traceback.print_exception()
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file24263/traceback.txt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:03:46 2012
From: report at bugs.python.org (Thomas Ryschawy)
Date: Tue, 17 Jan 2012 16:03:46 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
In-Reply-To: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>
Message-ID: <1326816226.09.0.918107291207.issue13807@psf.upfronthosting.co.za>


Changes by Thomas Ryschawy <ThomasOtto.Ryschawy at Emerson.com>:


Removed file: http://bugs.python.org/file24263/traceback.txt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:07:04 2012
From: report at bugs.python.org (Thomas Ryschawy)
Date: Tue, 17 Jan 2012 16:07:04 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
In-Reply-To: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>
Message-ID: <1326816424.8.0.739469974919.issue13807@psf.upfronthosting.co.za>


Changes by Thomas Ryschawy <ThomasOtto.Ryschawy at Emerson.com>:


Added file: http://bugs.python.org/file24264/traceback.txt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:13:10 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:13:10 +0000
Subject: [issue13691] pydoc help (or help('help')) claims to run a help
	utility; does nothing
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1326816790.22.0.213840241107.issue13691@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Just a heads-up: I?ll be offline between January 19 and the end of the month, so don?t worry if you make a patch and it?s not reviewed immediately (at least not by me, other developers may do it :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:13:50 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:13:50 +0000
Subject: [issue13691] pydoc help (or help('help')) should show the doc for help
In-Reply-To: <1325406152.03.0.315132806833.issue13691@psf.upfronthosting.co.za>
Message-ID: <1326816830.68.0.0949116809824.issue13691@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
title: pydoc help (or help('help')) claims to run a help utility; does nothing -> pydoc help (or help('help')) should show the doc for help

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13691>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:19:35 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 16:19:35 +0000
Subject: [issue13589] Aifc low level serialization primitives fix
In-Reply-To: <1323714686.93.0.781541085274.issue13589@psf.upfronthosting.co.za>
Message-ID: <E1RnBl0-00021h-AI@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f715c4a5a107 by Antoine Pitrou in branch '3.2':
Issue #13589: Fix some serialization primitives in the aifc module.
http://hg.python.org/cpython/rev/f715c4a5a107

New changeset b039965b0066 by Antoine Pitrou in branch 'default':
Issue #13589: Fix some serialization primitives in the aifc module.
http://hg.python.org/cpython/rev/b039965b0066

New changeset 8fac90d0f4cd by Antoine Pitrou in branch '2.7':
Issue #13589: Fix some serialization primitives in the aifc module.
http://hg.python.org/cpython/rev/8fac90d0f4cd

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13589>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:20:07 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:20:07 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326817207.7.0.988384894284.issue13804@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I think this discussion would be more useful on the python-ideas mailing list.  The request (adding map to sequences) will probably be rejected*, but you have a good chance to get an explanation for this design choice by Guido van Rossum or one of the old-timer core devs.

*I think so because of str.join: people have said that they would prefer a list operation, but it?s always been denied because it would put the burden on any sequency type to implement the method, whereas with a str method then all iterables and iterators are supported for free.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:21:18 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 16:21:18 +0000
Subject: [issue13589] Aifc low level serialization primitives fix
In-Reply-To: <1323714686.93.0.781541085274.issue13589@psf.upfronthosting.co.za>
Message-ID: <1326817278.66.0.165724158274.issue13589@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I've finally committed the patch, thank you!

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13589>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:21:36 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:21:36 +0000
Subject: [issue13794] Copyright Year - Change it to 2012 please
In-Reply-To: <1326691304.18.0.453476801273.issue13794@psf.upfronthosting.co.za>
Message-ID: <1326817296.96.0.101424527322.issue13794@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Websites that Python devs can work on are updated, now the request should go to the pydotorg mailing list or webmaster email address.

----------
nosy: +eric.araujo
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13794>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:22:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 16:22:32 +0000
Subject: [issue13681] Aifc read compressed frames fix
In-Reply-To: <1325243830.12.0.0659769703479.issue13681@psf.upfronthosting.co.za>
Message-ID: <1326817352.11.0.287277364491.issue13681@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
dependencies: +Audioop decompression frames size check fix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13681>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:23:23 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:23:23 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326817403.4.0.313782111026.issue13703@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

#13712 contains a patch for test_packaging.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:26:09 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:26:09 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <1326817569.76.0.701557294697.issue13803@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

OK.

 from distutils.spawn import spawn
 from distutils import log
 from distutils.errors import DistutilsByteCompileError
+import platform

Please put that import higher up (with the other ?import X?, before the ?from X import Y?).
 
+            machine += ".%s" %platform.architecture()[0]

One space after the % operator please.

> I will commit it to 2.7, 3.1, 3.2 and 3.3 in a few days.

Not to 3.1, this is not a security issue.

----------
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:28:14 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:28:14 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1326817694.45.0.224138585192.issue6727@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

(3.1 doesn?t get non-security bug fixes either)

----------
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:35:08 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:35:08 +0000
Subject: [issue12415] Missing: How to checkout the Doc sources
In-Reply-To: <1309069778.19.0.06342861361.issue12415@psf.upfronthosting.co.za>
Message-ID: <1326818108.28.0.770940129847.issue12415@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> I think "documentation sources" is a bit vague.
I don?t know.  I probably wrote it under the assumption that the audience of this doc was developers, who already know that programmers don?t write HTML manually but generate it, and just need to know where to find the sources.  To a C programmer wanting to fix a bug, we just say ?the sources for X are in Include and Modules?, and it?s enough.

Thinking about it, the assumption is probably bad.  Sometimes we have to explain to new contributors where to find a Python module in the repo and how to write a test, so it would probably be better to explain more.  Here?s my draft, feel free to improve my patch with it (there may be duplication) and commit the result when you two are in agreement while I?ll be offline.

  The documentation in HTML, PDF or epub format is generated from text files
  written using the :ref:`reStructuredText format <markup> and contained in
  the :ref:`CPython Mercurial repository <setup>`.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12415>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:35:50 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 17 Jan 2012 16:35:50 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326817403.4.0.313782111026.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwYnWxXYO7ui72TzJYKk5toGVB6gVCYDYC=DC+-RK87vVg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> #13712 contains a patch for test_packaging.

It doesn't look related to randomized hash function. random-8.patch
contains a fix to test_packaging.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:37:45 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 16:37:45 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1326818265.28.0.885924158401.issue13806@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well, you should just replace these calls with audioop_check_size() instead. Apparently the checks were blindly added in issue7673.

By the way, I'm surprised audioop accepts unicode strings... :/

----------
nosy: +haypo, pitrou
versions: +Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:46:05 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:46:05 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326818765.89.0.036426786423.issue13703@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

>> #13712 contains a patch for test_packaging.
> It doesn't look related to randomized hash function.
Trust me.  (If you read the whole report you?ll see why it looks unrelated: instead of sorting things like your patch does mine addresses a more serious behavior bug).

> random-8.patch contains a fix to test_packaging.
I know, but mine is a bit better.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 17:56:51 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 16:56:51 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326819411.67.0.187811015451.issue12705@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I don?t understand why some two-liners are allowed (like "class X:\n pass").  The doc says ?a single interactive statement?.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:01:03 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 17:01:03 +0000
Subject: [issue13787] PyCode_New not round-trippable (TypeError)
In-Reply-To: <1326587591.5.0.14837599409.issue13787@psf.upfronthosting.co.za>
Message-ID: <1326819663.19.0.0542752349058.issue13787@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

co_freevars and co_cellvars are the last arguments of the function.  Your "co_what2.py" version of the script is correct, but "co_what.py" has a different order.

----------
nosy: +amaury.forgeotdarc
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13787>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:02:02 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 17:02:02 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326819722.31.0.501027133624.issue11805@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Here are patches for CPython and the d2 repo.  There is no doc update as the setupscript page still talks about setup.py and the setupcfg spec does not mention package_data at all (the negationists resources-promoters at work :), so this can wait for my big doc updates.  The changes to test_command_{build_py,sdist} are unrelated, I just found some fixes to do after grepping for package_data and I?ll commit them separately.

Please review.  (I?ll be offline until the end of the month FYI, but I?ll commit after that.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:02:38 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 17:02:38 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326819758.88.0.0776680231127.issue11805@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24265/fix-package_data-multivalue-cpy33.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:02:55 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 17:02:55 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326819775.93.0.245250933908.issue11805@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


Added file: http://bugs.python.org/file24266/fix-package_data-multivalue-d2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:09:46 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 17:09:46 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326820186.94.0.403302720539.issue11805@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Note that my proposed syntax does not allow something equivalent to {'': [etc.]} in distutils, i.e. package data for top-level modules.  I think this is okay: modules should not install data in their installation dir.  I don?t think it was widely used, but maybe I should ask on distutils-sig or survey setup.py scripts on PyPI to be sure.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Tue Jan 17 18:58:48 2012
From: report at bugs.python.org (Chris Jones)
Date: Tue, 17 Jan 2012 17:58:48 +0000
Subject: [issue11240] Running unit tests in a command line tool leads to
	infinite loop with multiprocessing on Windows
In-Reply-To: <1297987020.73.0.563526251198.issue11240@psf.upfronthosting.co.za>
Message-ID: <1326823128.33.0.685962744983.issue11240@psf.upfronthosting.co.za>


Chris Jones <chris at chrisejones.com> added the comment:

You can work around this issue by using:
python.exe -c "import nose; nose.main()"
instead of nosetests

Note that nose.main() with no args parses sys.argv

----------
nosy: +Chris.Jones

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11240>
_______________________________________

From report at bugs.python.org  Tue Jan 17 20:14:15 2012
From: report at bugs.python.org (Georg Brandl)
Date: Tue, 17 Jan 2012 19:14:15 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326827655.81.0.956506140682.issue13695@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Once you start contributing anything useful, I will start treating it as such.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Tue Jan 17 20:20:34 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 17 Jan 2012 19:20:34 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1326819411.67.0.187811015451.issue12705@psf.upfronthosting.co.za>
Message-ID: <CAK1QoorsqA8QAmVy4ea=WSb3dPAMMztW9229xaPcjYf1tXXRdQ@mail.gmail.com>


Meador Inge <meadori at gmail.com> added the comment:

On Tue, Jan 17, 2012 at 10:56 AM, ?ric Araujo <report at bugs.python.org> wrote:

> I don?t understand why some two-liners are allowed (like "class X:\n pass"). ?The doc says ?a single interactive statement?.

Because a single statement can be multiple lines (as is the case for
compound statements).  Look at the grammar for a single input
statement
(interactive_input):
http://docs.python.org/dev/reference/toplevel_components.html#interactive-input.
 This issue is really about multiple statements
and not multiple lines.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Tue Jan 17 20:25:20 2012
From: report at bugs.python.org (Georg Brandl)
Date: Tue, 17 Jan 2012 19:25:20 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326828320.61.0.316615814734.issue12705@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Because a class statement is one statement (it is a compound statement, but still just one).  The docs don't talk about lines :)

A "single interactive statement" is what you can enter at the interactive prompt in one line, or more than one line with only secondary prompts from the second line on.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Tue Jan 17 20:42:34 2012
From: report at bugs.python.org (Daniel Stutzbach)
Date: Tue, 17 Jan 2012 19:42:34 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326829354.98.0.476997337245.issue13804@psf.upfronthosting.co.za>


Daniel Stutzbach <stutzbach at google.com> added the comment:

If I'm understanding Martin H?cker's code correctly, the list comprehension equivalent is:

self.questions = [topic.questions for topic in self.topics]

The reduce() variants are not only much harder to read, but they will take O(n**2) operations because they create an O(n) list O(n) times.

----------
nosy: +stutzbach

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Tue Jan 17 20:59:51 2012
From: report at bugs.python.org (Jim Jewett)
Date: Tue, 17 Jan 2012 19:59:51 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326830391.84.0.596881406431.issue13703@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

To be more explicit about Martin A. Lemburg's msg151121 (which I agree with):

Count the collisions on a single lookup. 
If they exceed a threshhold, do something different.

Martin's strawman proposal was threshhold=1000, and raise.  It would be just as easy to say "whoa!  5 collisions -- time to use the alternative hash instead" (and, possibly, to issue a warning).  

Even that slight tuning removes the biggest objection, because it won't ever actually fail.

Note that the use of a (presumably stronger 2nd) hash wouldn't come into play until (and unless) there was a problem for that specific key in that specific dictionary.  For the normal case, nothing changes -- unless we take advantage of the existence of a 2nd hash to simplify the first few rounds of collision resolution.  (Linear probing is more cache-friendly, but also more vulnerable to worst-case behavior -- but if probing stops at 4 or 8, that may not matter much.)  For quick scripts, the 2nd hash will almost certainly never be needed, so startup won't pay the penalty.

The only down side I see is that the 2nd (presumably randomized) hash won't be cached without another slot, which takes more memory and shouldn't be done in a bugfix release.

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 17 21:36:51 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 20:36:51 +0000
Subject: [issue5689] Support xz compression in tarfile module
In-Reply-To: <1238861315.81.0.348941469726.issue5689@psf.upfronthosting.co.za>
Message-ID: <1326832611.76.0.290000503756.issue5689@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Ping.  Windows buildbots are still failing with MemoryError because of this preset=9.
The patch looks good to me as well.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5689>
_______________________________________

From report at bugs.python.org  Tue Jan 17 21:39:44 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 20:39:44 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
In-Reply-To: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>
Message-ID: <E1RnFok-0000xD-VM@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 463acb73fd79 by Amaury Forgeot d'Arc in branch 'default':
Issue #13727: Add 3 macros to access PyDateTime_Delta members:
http://hg.python.org/cpython/rev/463acb73fd79

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Tue Jan 17 21:40:18 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 20:40:18 +0000
Subject: [issue13727] Accessor macros for PyDateTime_Delta members
In-Reply-To: <1325944236.57.0.0169968212645.issue13727@psf.upfronthosting.co.za>
Message-ID: <1326832818.63.0.861903441368.issue13727@psf.upfronthosting.co.za>


Changes by Amaury Forgeot d'Arc <amauryfa at gmail.com>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13727>
_______________________________________

From report at bugs.python.org  Tue Jan 17 22:16:36 2012
From: report at bugs.python.org (Mahmoud Hashemi)
Date: Tue, 17 Jan 2012 21:16:36 +0000
Subject: [issue13787] PyCode_New not round-trippable (TypeError)
In-Reply-To: <1326587591.5.0.14837599409.issue13787@psf.upfronthosting.co.za>
Message-ID: <1326834996.09.0.713702778449.issue13787@psf.upfronthosting.co.za>


Mahmoud Hashemi <makuro at gmail.com> added the comment:

Yes, I knew it was an issue with crossed wires somewhere. The Python 2 code doesn't translate well to Python 3 because the function signature changed to add kwargonlycount. And I guess the argument order is substantially different, too, as described in Objects/codeobject.c#l291.

Thanks for clearing that up, though,

Mahmoud

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13787>
_______________________________________

From report at bugs.python.org  Tue Jan 17 22:45:28 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Tue, 17 Jan 2012 21:45:28 +0000
Subject: [issue13808] url for Tutor mailing list is broken
Message-ID: <1326836728.79.0.892184596442.issue13808@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

faq.rst: correct url is http://mail.python.org/mailman/listinfo/tutor

----------
components: Devguide
messages: 151488
nosy: ezio.melotti, tshepang
priority: normal
severity: normal
status: open
title: url for Tutor mailing list is broken

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13808>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:05:53 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 22:05:53 +0000
Subject: [issue13808] url for Tutor mailing list is broken
In-Reply-To: <1326836728.79.0.892184596442.issue13808@psf.upfronthosting.co.za>
Message-ID: <E1RnHA8-0000wo-I4@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 471f70b0b6b0 by Ezio Melotti in branch 'default':
#13808: fix a link and specify the IRC server.
http://hg.python.org/devguide/rev/471f70b0b6b0

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13808>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:06:50 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 17 Jan 2012 22:06:50 +0000
Subject: [issue13808] url for Tutor mailing list is broken
In-Reply-To: <1326836728.79.0.892184596442.issue13808@psf.upfronthosting.co.za>
Message-ID: <1326838010.01.0.575694586541.issue13808@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Fixed, thanks for the report!

----------
assignee:  -> ezio.melotti
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13808>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:20:57 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 17 Jan 2012 22:20:57 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
Message-ID: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>


New submission from Amaury Forgeot d'Arc <amauryfa at gmail.com>:

In bz2.py, "import threading" prevents the bz2 module from working when threads are not enabled.  The attached patch removes the limitation and provides a fake lock object.

I don't know if this should be backported to 3.2.

----------
components: Library (Lib)
files: bz2_nothread.patch
keywords: patch
messages: 151491
nosy: amaury.forgeotdarc
priority: normal
severity: normal
stage: patch review
status: open
title: bz2 does not work when threads are disabled
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24267/bz2_nothread.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:21:23 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 22:21:23 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
In-Reply-To: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>
Message-ID: <1326838883.57.0.468665525404.issue13809@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:49:04 2012
From: report at bugs.python.org (Georg Brandl)
Date: Tue, 17 Jan 2012 22:49:04 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
In-Reply-To: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>
Message-ID: <1326840544.76.0.370527514731.issue13809@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Isn't there already a dummy lock in dummy_threading?

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Tue Jan 17 23:55:38 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Tue, 17 Jan 2012 22:55:38 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
In-Reply-To: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>
Message-ID: <1326840938.99.0.0701572427307.issue13809@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

As Georg suggested, it would be better to use dummy_threading.RLock,
rather than providing our own implementation.

The test in the patch fails when I try to run it on a no-thread build.
support.import_fresh_module seems to treat the absence of the threading
module as an error, and returns None instead of allowing the bz2 module
to recover from the ImportError.

We needn't worry about 3.2. It still uses the old all-C implementation,
which has its threading dependencies protected by #ifdefs.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Wed Jan 18 00:12:41 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 23:12:41 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326841961.02.0.175983041316.issue6531@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Hmm something else: currently the atexit funcs are only called when the main interpreter exits, but that doesn't really make sense: if I register a function from a sub-interpreter, why would it execute correctly from another interpreter? All kind of fun things can happen...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Wed Jan 18 00:24:10 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 23:24:10 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <E1RnINs-0007tU-An@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset eb47af6e9e22 by Antoine Pitrou in branch '3.2':
Test running of code in a sub-interpreter
http://hg.python.org/cpython/rev/eb47af6e9e22

New changeset a108818aaa0d by Antoine Pitrou in branch 'default':
Test running of code in a sub-interpreter
http://hg.python.org/cpython/rev/a108818aaa0d

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Wed Jan 18 00:29:20 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 17 Jan 2012 23:29:20 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326842960.82.0.900923880596.issue6531@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Here is a patch for subinterp-wise atexit functions.
(I haven't got any specific test for the crash but the patch adds a test and it works)

----------
keywords: +patch
stage:  -> patch review
Added file: http://bugs.python.org/file24268/atexitsubinterps.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Wed Jan 18 00:57:43 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 17 Jan 2012 23:57:43 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
In-Reply-To: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>
Message-ID: <E1RnIuM-0001wz-5C@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2fb93282887a by Nadeem Vawda in branch 'default':
Issue #13809: Make bz2 module work with threads disabled.
http://hg.python.org/cpython/rev/2fb93282887a

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Wed Jan 18 00:59:04 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Tue, 17 Jan 2012 23:59:04 +0000
Subject: [issue13809] bz2 does not work when threads are disabled
In-Reply-To: <1326838856.97.0.0714966673096.issue13809@psf.upfronthosting.co.za>
Message-ID: <1326844744.87.0.979113389735.issue13809@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Fix committed. For the test, it turns out we can get the desired behavior
by telling import_fresh_module to block the threading module directly,
instead of blocking _thread.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13809>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:02:02 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 18 Jan 2012 00:02:02 +0000
Subject: [issue13810] refer people to Doc/Makefile when not using 'make' to
	build main documentation
Message-ID: <1326844922.3.0.806900590875.issue13810@psf.upfronthosting.co.za>


New submission from Tshepang Lekhonkhobe <tshepang at gmail.com>:

this is regarding "documenting.rst":

The url for checking out Sphinx from svn has been outdated for a while, so I think it would be better to refer people to Doc/Makefile for the correct urls, instead of having to keep updating this file on each change. This applies to the other modules in that section (which are currently up-to-date):

* docutils
* jinja2
* Pygments

----------
components: Devguide
messages: 151499
nosy: ezio.melotti, tshepang
priority: normal
severity: normal
status: open
title: refer people to Doc/Makefile when not using 'make' to build main documentation

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13810>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:06:12 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 00:06:12 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326845172.41.0.863393797813.issue12705@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thank you both for fixing my incomplete understanding.

Meador, unless you want a review from another core dev, I think you can just go ahead.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:18:32 2012
From: report at bugs.python.org (Tshepang Lekhonkhobe)
Date: Wed, 18 Jan 2012 00:18:32 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326845912.6.0.361033748204.issue13763@psf.upfronthosting.co.za>


Tshepang Lekhonkhobe <tshepang at gmail.com> added the comment:

I included both suggestions in the latest patch.

----------
Added file: http://bugs.python.org/file24269/rm-obsolete-reference2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:30:33 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 00:30:33 +0000
Subject: [issue13763] rm obsolete reference in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326846633.19.0.327889487929.issue13763@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I really think there is no bug.  We don?t have to explain why Mercurial or hg are named so.  The current text merely says that ?hg?, which is the name of the executable, is commonly used as a short form of ?Mercurial?.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:31:33 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 00:31:33 +0000
Subject: [issue13763] Potentially hard to understand wording in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326846693.91.0.6043549032.issue13763@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Adding a native speaker for confirmation.

----------
nosy: +terry.reedy
title: rm obsolete reference in devguide -> Potentially hard to understand wording in devguide

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:53:16 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 00:53:16 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <E1RnJm6-0006BX-N9@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset bb10cd354e49 by Victor Stinner in branch 'default':
Close #10278: Add time.wallclock() function, monotonic clock.
http://hg.python.org/cpython/rev/bb10cd354e49

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:54:04 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 00:54:04 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
Message-ID: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

http://docs.python.org/py3k/library/string.html#format-specification-mini-language

"The fill character can be any character other than ?{? or ?}?. The presence of a fill character is signaled by the character following it, which must be one of the alignment options. If the second character of format_spec is not a valid alignment option, then it is assumed that both the fill character and the alignment option are absent."


>>> '{0:x>10d}'.format(1)
'xxxxxxxxx1'
>>> '{0:xx10d}'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid conversion specification
>>>

----------
messages: 151505
nosy: py.user
priority: normal
severity: normal
status: open
title: In str.format an incorrect alignment option doesn't make fill char and onself absent

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:54:22 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 00:54:22 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326848062.28.0.691114349007.issue13811@psf.upfronthosting.co.za>


Changes by py.user <port139 at yandex.ru>:


----------
components: +Interpreter Core
type:  -> behavior
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 01:57:38 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 00:57:38 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <E1RnJqK-0006WO-BQ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1de276420470 by Victor Stinner in branch 'default':
Issue #10278: fix a typo in the doc
http://hg.python.org/cpython/rev/1de276420470

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Wed Jan 18 02:15:09 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Wed, 18 Jan 2012 01:15:09 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326849309.38.0.120196996235.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

What is the expected output, and why?

I think the error message might be incorrect, possibly it should be "invalid format specifier".

----------
nosy: +eric.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 02:15:46 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Wed, 18 Jan 2012 01:15:46 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
Message-ID: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>


New submission from Jon Brandvein <jon.brandvein at gmail.com>:

When a child process exits due to an exception, a traceback is written, but stderr is not flushed. Thus I see a header like "Process 1:\n", but no traceback.

I don't have a development environment or any experience with Mecurial, so I'm afraid there's no patch, but it's a one-liner.

In   /Lib/multiprocess/process.py :: Process._bootstrap

    except:
        exitcode = 1
        import traceback
        sys.stderr.write('Process %s:\n' % self.name)
        sys.stderr.flush()
        traceback.print_exc()

Append a "sys.stderr.flush()" to the suite.

It surprised me that flushing was even necessary. I would've thought that the standard streams would all be closed just before the process terminated, regardless of exit status. But I observe that unless I explicitly flush stdout and stderr before terminating, the output is lost entirely, even if the exit is not abnormal. This isn't the desired behavior, is it?

----------
components: Library (Lib)
messages: 151508
nosy: brandj
priority: normal
severity: normal
status: open
title: multiprocessing package doesn't flush stderr on child exception
type: behavior
versions: Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Wed Jan 18 02:17:53 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Wed, 18 Jan 2012 01:17:53 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1326849473.41.0.34988517386.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

(Er, that should be /Lib/multiprocessing/process.py :: Process._bootstrap of course.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Wed Jan 18 03:06:08 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 02:06:08 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326852368.64.0.994938423564.issue13811@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

absent fill char and align option:

>>> '{0:10d}'.format(1)
'         1'
>>>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 03:26:09 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Wed, 18 Jan 2012 02:26:09 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326853569.33.0.597778682391.issue13781@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

thanks that looks good.

As far as fixing this for 2.7 goes, i don't like the _sound_ of it because it is gross... But i'm actually okay with having special case code in the gzip module that rejects '<fdopen>' as an actual filename and uses '' instead in that case.  It is VERY unlikely that anyone ever intentionally wants to use that as a filename.

Anything more than that (changing the actual '<fdopen>' string for example) seems too invasive and might break someone's doctests and does genuinely make it more difficult to see what a fdopened file object is from its repr.

----------
assignee:  -> gregory.p.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:01:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 03:01:39 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <E1RnLmL-0006s1-Gc@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 284550d0d8ae by Jesus Cea in branch '2.7':
Closes #13803: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/284550d0d8ae

New changeset eed73b16e71f by Jesus Cea in branch '3.2':
Closes #13803: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/eed73b16e71f

New changeset 8d0ccb4ad206 by Jesus Cea in branch 'default':
MERGE: Closes #13803: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/8d0ccb4ad206

----------
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:28:14 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 03:28:14 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <E1RnMC4-0000rK-Ch@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2ec4ab2a6f65 by Jesus Cea in branch '2.7':
Emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/2ec4ab2a6f65

New changeset 9d62f5aa35ff by Jesus Cea in branch '3.2':
Emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/9d62f5aa35ff

New changeset 4a6e0b6d493b by Jesus Cea in branch 'default':
MERGE: Emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/4a6e0b6d493b

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:43:37 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 03:43:37 +0000
Subject: [issue13665] TypeError: string or integer address expected instead of
	str instance
In-Reply-To: <1324921834.92.0.644281007494.issue13665@psf.upfronthosting.co.za>
Message-ID: <E1RnMQy-00024N-8B@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b4d9243d16c9 by Ezio Melotti in branch '3.2':
#13665: s/string/bytes/ in error message.
http://hg.python.org/cpython/rev/b4d9243d16c9

New changeset 0c0ffebfccb0 by Ezio Melotti in branch 'default':
#13665: merge with 3.2.
http://hg.python.org/cpython/rev/0c0ffebfccb0

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13665>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:44:15 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Wed, 18 Jan 2012 03:44:15 +0000
Subject: [issue13665] TypeError: string or integer address expected instead of
	str instance
In-Reply-To: <1324921834.92.0.644281007494.issue13665@psf.upfronthosting.co.za>
Message-ID: <1326858255.53.0.517280649041.issue13665@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
assignee:  -> ezio.melotti
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13665>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:50:01 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 03:50:01 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <E1RnMX9-0002XB-QQ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4074439c3894 by Jesus Cea in branch '2.7':
Yet another emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/4074439c3894

New changeset 37efae3bf912 by Jesus Cea in branch '3.2':
Yet another emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/37efae3bf912

New changeset afdce2e2f98d by Jesus Cea in branch 'default':
MERGE: Yet another emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/afdce2e2f98d

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 04:58:38 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 18 Jan 2012 03:58:38 +0000
Subject: [issue13813] "sysconfig.py" and "distutils/util.py" redundancy
Message-ID: <1326859118.01.0.584725172013.issue13813@psf.upfronthosting.co.za>


New submission from Jes?s Cea Avi?n <jcea at jcea.es>:

During development of patch for issue #13803, I found redundancy between "sysconfig.py" and "distutils/util.py". The functions are the same, and the return values must be the same too. So programmers *MUST* remember to change both sources in sync.

This redundancy must be trimmed.

This is not strictly a bugfix, so could be not adequate for 2.7/3.2, but applying for 3.2 at least would easy future merges.

----------
assignee: tarek
components: Distutils
keywords: easy
messages: 151516
nosy: eric.araujo, jcea, tarek
priority: normal
severity: normal
status: open
title: "sysconfig.py" and "distutils/util.py" redundancy
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13813>
_______________________________________

From report at bugs.python.org  Wed Jan 18 05:00:14 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 18 Jan 2012 04:00:14 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <1326859214.67.0.146980860865.issue13803@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

A trivial check-in causing a nightmare at 5AM just hours before a day-long offline trip :-).

Eric, could you possibly review the committed version?. I can't use "platform.architecture()" because bootstrap issues. For instance: http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%202.7/builds/828/steps/compile/logs/stdio

Moreover, there are function redundancy between "distutils/util.py" and "sysconfig.py", and the return value must be consistent. This is just created issue13813.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 05:05:48 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 04:05:48 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <E1RnMmQ-0003sI-OO@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 147ad02875fa by Jesus Cea in branch '3.2':
And yet another emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/147ad02875fa

New changeset 582274636446 by Jesus Cea in branch 'default':
MERGE: And yet another emergency fix for #13803 bootstrap issue: Under Solaris, distutils doesn't include bitness in the directory name
http://hg.python.org/cpython/rev/582274636446

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 07:16:55 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Wed, 18 Jan 2012 06:16:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326867415.98.0.792717979563.issue13703@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

I like what you've done in #13704 better than what I see in random-8.patch so far.  see the code review comments i've left on both issues.

----------
nosy: +gregory.p.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 08:32:33 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 07:32:33 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <E1RnQ0V-00021c-Nf@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 7d405058e458 by Nadeem Vawda in branch '3.2':
Issue #13781: Fix GzipFile to work with os.fdopen()'d file objects.
http://hg.python.org/cpython/rev/7d405058e458

New changeset fe36edf3a341 by Nadeem Vawda in branch 'default':
Merge: #13781: Fix GzipFile to work with os.fdopen()'d file objects.
http://hg.python.org/cpython/rev/fe36edf3a341

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Wed Jan 18 08:55:59 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 18 Jan 2012 07:55:59 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326873359.65.0.118956465184.issue13781@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

> As far as fixing this for 2.7 goes, i don't like the _sound_ of it
> because it is gross... But i'm actually okay with having special case
> code in the gzip module that rejects '<fdopen>' as an actual filename
> and uses '' instead in that case.  It is VERY unlikely that anyone ever
> intentionally wants to use that as a filename.

I agree - it sounds ugly, but pragmatically it seems like the best option.
Given that the output will still be a valid gzip file even in this rare
case, it seems unlikely to cause trouble even then.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Wed Jan 18 09:31:33 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 08:31:33 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326875493.14.0.0844277039798.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Shut?up,?Georg!?Ezio,?please?fix?this?two?additional?typos?to?close?this?bug?report?for?good.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Wed Jan 18 09:45:09 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Wed, 18 Jan 2012 08:45:09 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326876309.84.0.153896166793.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

I'm not sure what you're saying here. Is it that 'xx' should be ignored? The documentation says that 'xx' isn't fill and alignment, not that they don't exist. If they're not fill and alignment, then the format string is in error.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 09:45:34 2012
From: report at bugs.python.org (Chris Withers)
Date: Wed, 18 Jan 2012 08:45:34 +0000
Subject: [issue9208] SMTPHandler in the logging module does not handle unicode
	strings
In-Reply-To: <1278627371.74.0.0567748809415.issue9208@psf.upfronthosting.co.za>
Message-ID: <1326876334.72.0.10365804667.issue9208@psf.upfronthosting.co.za>


Chris Withers <chris at simplistix.co.uk> added the comment:

Just as a post-fix to this, the email handlers for the python logging framework that I maintain as a package on PyPI now handle unicode email correctly:

http://pypi.python.org/pypi/mailinglogger/3.7.0

I'd suggest people looking for fully-featured email log handlers use mailinglogger...

----------
nosy: +cjw296

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9208>
_______________________________________

From report at bugs.python.org  Wed Jan 18 10:26:57 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 09:26:57 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326878817.22.0.879965568767.issue13811@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Please?fix the error message?to?"invalid format specifier"

----------
nosy: +Retro

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 10:31:52 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Wed, 18 Jan 2012 09:31:52 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326879112.46.0.0668241183029.issue13804@psf.upfronthosting.co.za>


Martin H?cker <spamfaenger at gmx.de> added the comment:

@stutzbach: I believe you got me wrong, as the example topic.questions is meant to return a list of questions that need concatenating - thus you can't save the second step.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Wed Jan 18 10:36:31 2012
From: report at bugs.python.org (Austin Bingham)
Date: Wed, 18 Jan 2012 09:36:31 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
In-Reply-To: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>
Message-ID: <1326879391.92.0.789738489713.issue13807@psf.upfronthosting.co.za>


Changes by Austin Bingham <austin.bingham at gmail.com>:


----------
nosy: +abingham

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Wed Jan 18 10:49:15 2012
From: report at bugs.python.org (Graham Dumpleton)
Date: Wed, 18 Jan 2012 09:49:15 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1248175169.79.0.15962102552.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326880155.54.0.953824884852.issue6531@psf.upfronthosting.co.za>


Graham Dumpleton <Graham.Dumpleton at gmail.com> added the comment:

What are the intentions with respect to atexit and sub interpreters?

The original report was only about ensuring that the main interpreter doesn't crash if an atexit function was registered in a sub interpreter. So, was not expecting a change to sub interpreters in submitting this report, in as much as atexit callbacks for sub interpreters are never invoked in Python 2.X.

That said, for mod_wsgi I have extended sub interpreter destruction so that atexit callbacks registered in sub interpreters are called. For mod_wsgi though, sub interpreters are only destroyed on process shutdown. For the general case, a sub interpreter could be destroyed at any time during the life of the process. If one called atexit callbacks on such sub interpreter destruction, it notionally changes the meaning of atexit, which is in process exit and not really sub interpreter exit.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Wed Jan 18 11:01:42 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 18 Jan 2012 10:01:42 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326867415.98.0.792717979563.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwY2fmpHGBKCw9AVWBHJBfZeVq61iOP5sz+KwC-myQz5kg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I like what you've done in #13704 better than what I see in random-8.patch so far. ?see the code review comments i've left on both issues.

I didn't write "3106cc0a2024.diff" patch attached to #13704, I just
clicked on the button to generate a patch from the repository.
Christian Heimes wrote the patch.

I don't really like "3106cc0a2024.diff", we don't need Mersenne
Twister to initialize the hash secret. The patch doesn't allow to set
a fixed secret if you need the same secret for a group of processes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 11:07:54 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Wed, 18 Jan 2012 10:07:54 +0000
Subject: [issue13810] refer people to Doc/Makefile when not using 'make' to
	build main documentation
In-Reply-To: <1326844922.3.0.806900590875.issue13810@psf.upfronthosting.co.za>
Message-ID: <1326881274.49.0.999246538033.issue13810@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

The outdated command is addressed in issue#12415, and I think it's better to provide a precise command in devguide, so that if you don't use make you don't even need to understand where to grab the information to checkout third-party tools.

Additionally, the whole way we're fetching those tools will be revisited soon.

----------
nosy: +sandro.tosi
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> Missing: How to checkout the Doc sources

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13810>
_______________________________________

From report at bugs.python.org  Wed Jan 18 11:43:16 2012
From: report at bugs.python.org (Arkadiusz Wahlig)
Date: Wed, 18 Jan 2012 10:43:16 +0000
Subject: [issue13814] Generators as context managers.
Message-ID: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>


New submission from Arkadiusz Wahlig <arkadiusz.wahlig at gmail.com>:

Generators should support the with statement with __exit__ calling self.close().

with genfunc() as g:
    for item in g:
        print(item)

----------
messages: 151530
nosy: yak
priority: normal
severity: normal
status: open
title: Generators as context managers.
type: enhancement
versions: Python 2.7, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Wed Jan 18 11:56:49 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 10:56:49 +0000
Subject: [issue6531] atexit_callfuncs() crashing within Py_Finalize() when
	using multiple interpreters.
In-Reply-To: <1326880155.54.0.953824884852.issue6531@psf.upfronthosting.co.za>
Message-ID: <1326884100.3395.7.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> That said, for mod_wsgi I have extended sub interpreter destruction so
> that atexit callbacks registered in sub interpreters are called. For
> mod_wsgi though, sub interpreters are only destroyed on process
> shutdown. For the general case, a sub interpreter could be destroyed
> at any time during the life of the process. If one called atexit
> callbacks on such sub interpreter destruction, it notionally changes
> the meaning of atexit, which is in process exit and not really sub
> interpreter exit.

Well the atexit docs say "Functions thus registered are automatically
executed upon normal interpreter termination".

My reasoning is:
- atexit functions are already called at interpreter destruction (*),
not at process shutdown. If you call Py_Initialize and Py_Finalize
several times in a row, you will have several atexit calls.

- registering a function in an interpreter and calling it in another is
undefined and potentially dangerous. That function can for example refer
to global objects which are different from the calling interpreter's
global objects. These objects or their internal structures could have
become invalid when the sub-interpreter was shut down.

I expect uses of sub-interpreters to be quite rare apart from mod_wsgi,
so following mod_wsgi makes sense IMO.

(*) note atexit functions are even called *before* module globals are
garbage collected. It's quite early in the cleanup phase.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6531>
_______________________________________

From report at bugs.python.org  Wed Jan 18 11:57:59 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Wed, 18 Jan 2012 10:57:59 +0000
Subject: [issue13814] Generators as context managers.
In-Reply-To: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>
Message-ID: <1326884279.69.0.860432347652.issue13814@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

If you want to call .close() automatically on something you can use contextlib.closing(): http://docs.python.org/library/contextlib.html#contextlib.closing

----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Wed Jan 18 12:01:34 2012
From: report at bugs.python.org (Sumudu Fernando)
Date: Wed, 18 Jan 2012 11:01:34 +0000
Subject: [issue10109] itertools.product with infinite iterator cause
	MemoryError.
In-Reply-To: <1287103502.64.0.571191585998.issue10109@psf.upfronthosting.co.za>
Message-ID: <1326884494.62.0.671988482658.issue10109@psf.upfronthosting.co.za>


Sumudu Fernando <sumuduf at gmail.com> added the comment:

I don't agree with the response to this.

It is true that as implemented (at least in 2.7, I don't have 3.x handy to check) itertools.product requires finite iterables.  However this seems to be simply a consequence of the implementation and not part of the "spirit" of the function, which as falsetru pointed out is stated to be "equivalent to nested for-loops in a generator expression".

Indeed, implementing product in Python (in a recursive way) doesn't have this problem.

Perhaps a more convincing set of testcases to show why this could be considered a problem:

>>> import itertools
>>> itertools.product(xrange(100))
<itertools.product object at 0xb7ed334c>
>>> itertools.product(xrange(1000000))
<itertools.product object at 0xb7ed620c>
>>> itertools.product(xrange(1000000000))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

Note that I'm not even using an infinite iterable, just a really big one.  The issue is that creating the iterator fails with a MemoryError, before I've even asked for any values.  Consider the following:

for (i, v) in enumerate(itertools.product(a, b, c)):
    if i < 1000:
        print v
    else:
        break

When a, b, and c are relatively small, finite iterables, this code works fine.  However, if *any* of them are too large (or infinite), we see a MemoryError before the loop even starts, even though only 1000 elements are required.  I think it's conceivable that we might want something like "a = itertools.cycle(xrange(5))", and even that will break this loop.

That said, in all such cases I could think of, we can always either truncate big iterators before passing them to product, or use zip/comprehensions to add their values into the tuple (or some combination of those).  So maybe it isn't a huge deal.

I've attached my implementation of product which deals with infinite iterators by leveraging enumerate and itertools.cycle, and is pretty much a direct translation of the "odometer" idea.  This doesn't support the "repeat" parameter (but probably could using itertools.tee).  One thing that should be changed is itertools.cycle shouldn't be called / doesn't need to be called on infinite iterators, but I couldn't figure out how to do that.  Maybe there is some way to handle it in the C implementation?)

In summary: the attached implementation of product can accept any mix of infinite / finite iterators, returning a generator intended for partial consumption.  The existing itertools.product doesn't work in this case.

----------
nosy: +Sumudu.Fernando
Added file: http://bugs.python.org/file24270/product.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________

From report at bugs.python.org  Wed Jan 18 13:04:21 2012
From: report at bugs.python.org (Pedro Andres Aranda Gutierrez)
Date: Wed, 18 Jan 2012 12:04:21 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
In-Reply-To: <CAHRxO8irhc-vMHx5oXQo+w+sSo6J+Kao2O0YeTGJNv9nER0eOQ@mail.gmail.com>
Message-ID: <CAO48Bk8NtcxO151e1pCGwG3Fv+a3-S30aRP=K=NBw6vSC0WZXQ@mail.gmail.com>


Pedro Andres Aranda Gutierrez <paaguti at gmail.com> added the comment:

Thanks a lot again :-)
We have a saying here: you'll never go to sleep without having learnt
something new :-)

On Tue, Jan 17, 2012 at 4:11 PM, patrick vrijlandt
<report at bugs.python.org> wrote:
>
> patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:
>
> Hi,
>
> Did you look at lxml (http://lxml.de)?
>
> from lxml.builder import E
> from lxml import etree
>
> tree = etree.ElementTree(
> ? ?E.Hello(
> ? ? ? ?"Good morning!",
> ? ? ? ?E.World("How do you do", humour = "excellent"),
> ? ? ? ?"Fine",
> ? ? ? ?E.Goodbye(),
> ? ? ? ?),
> ? ?)
>
> print(etree.tostring(tree, pretty_print=True).decode())
>
> # output, even more prettified
>
> <Hello>
> ? ?Good morning!
> ? ?<World humour="excellent">
> ? ? ? ? ?How do you do
> ? ?</World>
> ? ? ? ? ?Fine
> ? ?<Goodbye/>
> </Hello>
>
> By the way, your Element enhancement is buggy, because all newly create
> elements will share the same attrib dictionary (if attrib is not given).
> Notice that Donald Duck will be sad; by the time we print even Hello is sad.
>
> import xml.etree.ElementTree as etree
>
> class Element(etree.Element):
> ? ?def __init__(self, tag, attrib={}, **extra):
> ? ? ? ?super().__init__(tag)
> ? ? ? ?self.tag = tag
> ? ? ? ?self.attrib = attrib
> ? ? ? ?self.attrib.update(extra)
> ? ? ? ?self.text = self.attrib.pop('text', None)
> ? ? ? ?self.tail = self.attrib.pop('tail', None)
> ? ? ? ?self._children = []
>
> if __name__ == '__main__':
>
> ? ?test = Element('Hello',)
> ? ?test2 = Element('World',{'humour':'excelent'},text = 'How do you do',
> tail="Fine")
> ? ?test3 = Element('Goodbye', humour='sad')
> ? ?test4 = Element('Donaldduck')
> ? ?test.append(test2)
> ? ?test.append(test3)
> ? ?test.append(test4)
> ? ?tree = etree.ElementTree(test)
> ? ?print(etree.tostring(test, encoding="utf-8", method="xml"))
>
> <Hello humour="sad">
> ? ?<World humour="excelent">How do you do</World>Fine
> ? ?<Goodbye humour="sad" />
> ? ?<Donaldduck humour="sad" />
> </Hello>'
>
> The correct idiom would be:
>
> ? ?def __init__(self, tag, attrib=None, **extra):
> ? ? ? ?if attrib is None:
> ? ? ? ? ? ?attrib = {}
> ? ? ? ?super().__init__(tag)
>
> Cheers,
>
> Patrick
>
> 2012/1/16 Pedro Andres Aranda Gutierrez <report at bugs.python.org>
>
>>
>> Pedro Andres Aranda Gutierrez <paaguti at gmail.com> added the comment:
>>
>> Touch? :-)
>> I was just frustrated because my XMLs never have tail or text as
>> attributes and I wanted to have more compact code...
>>
>> On Mon, Jan 16, 2012 at 12:14 PM, patrick vrijlandt
>> <report at bugs.python.org> wrote:
>> >
>> > patrick vrijlandt <patrick.vrijlandt at gmail.com> added the comment:
>> >
>> > I agree the Element syntax is sometimes awkward.
>> >
>> > But how would you represent text or tail attributes within this enhanced
>> element?
>> > <animal name="cat" tail="yes"> comes to mind ...
>> >
>> > ----------
>> > nosy: +patrick.vrijlandt
>> >
>> > _______________________________________
>> > Python tracker <report at bugs.python.org>
>> > <http://bugs.python.org/issue13796>
>> > _______________________________________
>>
>> ----------
>>
>> _______________________________________
>> Python tracker <report at bugs.python.org>
>> <http://bugs.python.org/issue13796>
>> _______________________________________
>>
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13796>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Wed Jan 18 14:04:28 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 13:04:28 +0000
Subject: [issue5689] Support xz compression in tarfile module
In-Reply-To: <1238861315.81.0.348941469726.issue5689@psf.upfronthosting.co.za>
Message-ID: <E1RnVBj-0003rH-DV@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b86b54fcb5c2 by Lars Gust?bel in branch 'default':
Issue #5689: Avoid excessive memory usage by using the default lzma preset.
http://hg.python.org/cpython/rev/b86b54fcb5c2

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5689>
_______________________________________

From report at bugs.python.org  Wed Jan 18 14:16:17 2012
From: report at bugs.python.org (Colin Watson)
Date: Wed, 18 Jan 2012 13:16:17 +0000
Subject: [issue13815] tarfile.ExFileObject can't be wrapped using
	io.TextIOWrapper
Message-ID: <1326892577.13.0.456815776311.issue13815@psf.upfronthosting.co.za>


New submission from Colin Watson <cjwatson at users.sourceforge.net>:

The file-like object returned by TarFile.extractfile can't be wrapped in an io.TextIOWrapper (which would be rather convenient in some cases to get something that reads str rather than bytes).

The attached patch demonstrates the problem by way of a test case, and fixes it.  It's just a matter of adding a no-op flush method so that TextIOWrapper.close is happy with it.

----------
components: Library (Lib)
files: tarfile-exfileobject-flush.patch
keywords: patch
messages: 151536
nosy: cjwatson
priority: normal
severity: normal
status: open
title: tarfile.ExFileObject can't be wrapped using io.TextIOWrapper
versions: Python 3.3
Added file: http://bugs.python.org/file24271/tarfile-exfileobject-flush.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13815>
_______________________________________

From report at bugs.python.org  Wed Jan 18 14:34:54 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 18 Jan 2012 13:34:54 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326893694.59.0.983687342873.issue10181@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Revisiting memoryview.size: I foresee problems for NumPy users, since array.size
has a different meaning there:

>>> x = array([[1,2,3], [4,5,6]], dtype='q')
>>> x.shape
(2, 3)
>>> x.itemsize
8
>>> len(x)
2
>>> x.size
6
>>> x.nbytes
48

So here we have:

x.nbytes == product(shape) * itemsize == Py_buffer.len  == (virtual!) byte length
x.size   == product(shape) == number of elements


My suggestion is to use memoryview.nbytes as well. memoryview.size would have
the additional problem that Py_buffer.len is always the byte size of the logical 
structure (e.g. after slicing) and not necessarily the byte size of the physical
memory area.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:40:42 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 18 Jan 2012 14:40:42 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326897642.44.0.34435018835.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

"nbytes" sounds reasonable to me, given the unfortunate ambiguity of both size and len.

As far as #12834 goes, I'm happy to go along with whatever you think is best. You've spent a lot more time down in the guts of the implementation than I have, and the approach you describe seems to make sense :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:41:24 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 18 Jan 2012 14:41:24 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326897684.68.0.708133663505.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Err, make that #12384 (oops)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:41:48 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 18 Jan 2012 14:41:48 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1326897708.5.0.333566690162.issue10181@psf.upfronthosting.co.za>


Changes by Nick Coghlan <ncoghlan at gmail.com>:


----------
Removed message: http://bugs.python.org/msg151539

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:46:10 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 14:46:10 +0000
Subject: [issue13803] Under Solaris,
	distutils doesn't include bitness in the directory name
In-Reply-To: <1326773319.51.0.412338546649.issue13803@psf.upfronthosting.co.za>
Message-ID: <1326897970.1.0.650629345224.issue13803@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Final code looks OK to me.

----------
type: enhancement -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13803>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:57:49 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 14:57:49 +0000
Subject: [issue13813] "sysconfig.py" and "distutils/util.py" redundancy
In-Reply-To: <1326859118.01.0.584725172013.issue13813@psf.upfronthosting.co.za>
Message-ID: <1326898669.1.0.348713571639.issue13813@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I am afraid the distutils feature freeze prevents us from doing this, even in 3.3.  Remember that Tarek initially moved sysconfig from distutils to the top level, and the removal was reverted alongside other improvements when there was outcry at the distutils changes and distutils2 was started.  We are aware of the discrepancies between distutils.sysconfig and syconfig; for example, the Mac framework stuff for --user (PEP 371) is different, and the docs for sysconfig misleadingly says that the paths are used by distutils.  These will get fixed when I get a chance.  distutils2/packaging exclusively uses sysconfig, no more duplication.

----------
assignee: tarek -> eric.araujo
resolution:  -> wont fix
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13813>
_______________________________________

From report at bugs.python.org  Wed Jan 18 15:58:38 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 14:58:38 +0000
Subject: [issue13813] "sysconfig.py" and "distutils/util.py" redundancy
In-Reply-To: <1326859118.01.0.584725172013.issue13813@psf.upfronthosting.co.za>
Message-ID: <1326898718.83.0.687402840693.issue13813@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

s/PEP 371/PEP 370/

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13813>
_______________________________________

From report at bugs.python.org  Wed Jan 18 16:03:19 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Wed, 18 Jan 2012 15:03:19 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326898999.64.0.352961027858.issue11805@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

I figured it would let people comment on the syntax I propose more easily if I extracted it from the patch.  Here?s the example:

  package_data =
      cheese = data/templates/* doc/*
        doc/images/*.png

We have a package name, equals sign, then specs/globs separated by whitespace, including newlines (without any indent rules for the continuation lines, the code just adds them to the last seen package name if there is no equals sign in the lign).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Wed Jan 18 16:47:32 2012
From: report at bugs.python.org (Vincent Pelletier)
Date: Wed, 18 Jan 2012 15:47:32 +0000
Subject: [issue1785] "inspect" gets broken by some descriptors
In-Reply-To: <1199986772.67.0.313630472197.issue1785@psf.upfronthosting.co.za>
Message-ID: <1326901652.23.0.0837709945921.issue1785@psf.upfronthosting.co.za>


Vincent Pelletier <plr.vincent at gmail.com> added the comment:

This change causes the following behaviour:

>>> import inspect
>>> class B(object):
...   def f(self):
...     pass
... 
>>> inspect.getmembers(B, inspect.ismethod)
[]

While I would expect the result to contain f:

>>> inspect.ismethod(B.f)
True

Isn't this a regression ?

Regards,
Vincent Pelletier

----------
nosy: +vpelletier

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1785>
_______________________________________

From report at bugs.python.org  Wed Jan 18 16:49:44 2012
From: report at bugs.python.org (Vincent Pelletier)
Date: Wed, 18 Jan 2012 15:49:44 +0000
Subject: [issue1785] "inspect" gets broken by some descriptors
In-Reply-To: <1199986772.67.0.313630472197.issue1785@psf.upfronthosting.co.za>
Message-ID: <1326901784.05.0.833836711947.issue1785@psf.upfronthosting.co.za>


Vincent Pelletier <plr.vincent at gmail.com> added the comment:

Sorry, I forgot to mention I'm using python2.7 .

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1785>
_______________________________________

From report at bugs.python.org  Wed Jan 18 16:49:48 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Wed, 18 Jan 2012 15:49:48 +0000
Subject: [issue13815] tarfile.ExFileObject can't be wrapped using
	io.TextIOWrapper
In-Reply-To: <1326892577.13.0.456815776311.issue13815@psf.upfronthosting.co.za>
Message-ID: <1326901788.36.0.186519067395.issue13815@psf.upfronthosting.co.za>


Changes by Barry A. Warsaw <barry at python.org>:


----------
nosy: +barry

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13815>
_______________________________________

From report at bugs.python.org  Wed Jan 18 16:55:01 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 15:55:01 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326902101.21.0.438528703903.issue13695@psf.upfronthosting.co.za>


Changes by Bo?tjan Mejak <bostjan.mejak at gmail.com>:


----------
nosy:  -georg.brandl
resolution: remind -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:05:19 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 16:05:19 +0000
Subject: [issue13695] "type specific" to "type-specific"
In-Reply-To: <1325483290.25.0.872020961031.issue13695@psf.upfronthosting.co.za>
Message-ID: <1326902719.95.0.547350436508.issue13695@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I am closing this issue report and opening another issue report with the two new doc typos that were not reported here before the commit of Ezio Melotti.

----------
nosy:  -docs at python, ezio.melotti, python-dev, rhettinger
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13695>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:23:15 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 16:23:15 +0000
Subject: [issue13816] Two typos in the docs
Message-ID: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>


New submission from Bo?tjan Mejak <bostjan.mejak at gmail.com>:

There's a typo in the docs for cmp_to_key() function. Fix the first sentence "Transform an old-style comparison function to a key-function." to "Transform an old-style comparison function to a key function." (delete the hyphen between words "key" and "function")

http://docs.python.org/library/functools.html#functools.cmp_to_key

------------------------------------------------------------------

There's another typo in the docs lurking in http://docs.python.org/library/stdtypes.html#sequence-types-str-unicode-list-tuple-bytearray-buffer-xrange

Look for the table with the columns "Operation", "Result", "Notes" (scroll down and you'll find it), and fix the text of the fifth cell under the "Result" column like this:
"i'th item of s, origin 0"  -->  "i-th item of s, origin 0".

Change that apostrophe into a hyphen, so "i'th" to "i-th". Thanks, Ezio.

----------
assignee: docs at python
components: Documentation
messages: 151547
nosy: Retro, docs at python, ezio.melotti
priority: normal
severity: normal
status: open
title: Two typos in the docs
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:29:59 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 16:29:59 +0000
Subject: [issue1785] "inspect" gets broken by some descriptors
In-Reply-To: <1199986772.67.0.313630472197.issue1785@psf.upfronthosting.co.za>
Message-ID: <1326904199.49.0.396418860061.issue1785@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks for noticing. The doc for ismethod() says:

  ?Return true if the object is a bound method written in Python.?

and the docstring agrees with that:

  ?Return true if the object is an instance method. [...]?

So the change isn't properly a regression when reading the docs. On the other hand, it's true that some code may rely on the previous behaviour, and the discrepancy between getmembers() and a manual test can be confusing.

By the way, Python 3 has ismethod() right:

>>> class B:
...   def f(self): pass
... 
>>> inspect.ismethod(B.f)
False
>>> inspect.ismethod(B().f)
True

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1785>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:41:04 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 16:41:04 +0000
Subject: [issue1785] "inspect" gets broken by some descriptors
In-Reply-To: <1199986772.67.0.313630472197.issue1785@psf.upfronthosting.co.za>
Message-ID: <E1RnYZK-0000XF-JA@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f824744557ba by Antoine Pitrou in branch '2.7':
Revert part of 13f56cd8dec1 (issue #1785) to avoid breaking getmembers() with unbound methods.
http://hg.python.org/cpython/rev/f824744557ba

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1785>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:46:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 16:46:10 +0000
Subject: [issue1785] "inspect" gets broken by some descriptors
In-Reply-To: <1199986772.67.0.313630472197.issue1785@psf.upfronthosting.co.za>
Message-ID: <1326905170.41.0.529387088374.issue1785@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I've backed out the part of the changeset that "fixed" getmembers(), so the old behaviour is restored. Other parts of the changeset (that e.g. fixed pydoc) have not been reverted.

----------
versions: +Python 2.7, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1785>
_______________________________________

From report at bugs.python.org  Wed Jan 18 17:46:58 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 18 Jan 2012 16:46:58 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1326905218.34.0.868958929432.issue13816@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Just as a note: It is not acceptable to be rude on the tracker or
to remove people from the nosy list as you did in #13695.

----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 18:14:01 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 18 Jan 2012 17:14:01 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1326906841.06.0.293153932321.issue13816@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

And not really working, as I get updates for all assignments to docs at python anyway.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 18:17:38 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 18 Jan 2012 17:17:38 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1326907058.27.0.910307887968.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I?am?deeply?and?truly?sorry.?Can?we?now?fix?this?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 18:33:17 2012
From: report at bugs.python.org (Erik Bray)
Date: Wed, 18 Jan 2012 17:33:17 +0000
Subject: [issue11805] package_data only allows one glob per-package
In-Reply-To: <1302301429.27.0.797210396018.issue11805@psf.upfronthosting.co.za>
Message-ID: <1326907997.14.0.186347566834.issue11805@psf.upfronthosting.co.za>


Erik Bray <erik.m.bray at gmail.com> added the comment:

This patch works for me, and I'm happy with the syntax.  Thanks!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11805>
_______________________________________

From report at bugs.python.org  Wed Jan 18 18:54:32 2012
From: report at bugs.python.org (Daniel Stutzbach)
Date: Wed, 18 Jan 2012 17:54:32 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326909272.96.0.00490943713126.issue13804@psf.upfronthosting.co.za>


Daniel Stutzbach <stutzbach at google.com> added the comment:

Ah - in your first example (the one with 3 lines) did you mean to use .extend instead of .append?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:00:36 2012
From: report at bugs.python.org (Christoph Glaubitz)
Date: Wed, 18 Jan 2012 18:00:36 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
Message-ID: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>


New submission from Christoph Glaubitz <chris+lp at chrigl.de>:

Starting several threads, each just starting a subprocess.Popen and use communicate cause a deadlock in subprocess. (see attached file)

I can only reproduce this with python 2.7.2, not with any other versions. 2.6.5, 2.6.7 and 3.2.2 are working fine. Also 2.4.6 is working, with a problem fixed by http://bugs.python.org/issue1731717.

The attached script does not dead lock all the time, but at least every second call.

I am on a linux x86_64. The python's are compiled by hand, except of 2.6.5.

----------
files: subprocess_deadlock.py
messages: 151556
nosy: glaubich
priority: normal
severity: normal
status: open
title: deadlock in subprocess while running several threads using Popen
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file24272/subprocess_deadlock.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:10:39 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Wed, 18 Jan 2012 18:10:39 +0000
Subject: [issue13795] CDATA Element missing
In-Reply-To: <1326698204.07.0.970512725442.issue13795@psf.upfronthosting.co.za>
Message-ID: <1326910239.25.0.0104570288756.issue13795@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Note that there is no "need" to emit CDATA section: it's just another method to write data, just like in Python "\x41" and "A" are not distinct.

The workaround there is a hack, since it redefines an "internal" method _write().  This function is an implementation detail, and changed in newer releases.
I posted another hack on the stackoverflow page above, that works for python 3.2.

----------
nosy: +amaury.forgeotdarc
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13795>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:28:54 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Wed, 18 Jan 2012 18:28:54 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1326911334.2.0.341579982776.issue13816@psf.upfronthosting.co.za>


Justin Wehnes <jwehnes at gmail.com> added the comment:

Removed the hyphen in function keys.
Didn't really see a problem with using an apostrophe in 'ith' instead of a hyphen because I have seen it done both ways but changed it anyways.
This is my first contribution so i needed the practice.  Hope I did everything correctly.

----------
keywords: +patch
nosy: +JWEH
Added file: http://bugs.python.org/file24273/hyphen_removed_key_function.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:30:00 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Wed, 18 Jan 2012 18:30:00 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1326911400.31.0.464874885274.issue13816@psf.upfronthosting.co.za>


Changes by Justin Wehnes <jwehnes at gmail.com>:


Added file: http://bugs.python.org/file24274/i_th_hyphen.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:34:33 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 18 Jan 2012 18:34:33 +0000
Subject: [issue13704] Random number generator in Python core
In-Reply-To: <1325622273.87.0.90076525746.issue13704@psf.upfronthosting.co.za>
Message-ID: <1326911673.02.0.704417082875.issue13704@psf.upfronthosting.co.za>


Changes by Dave Malcolm <dmalcolm at redhat.com>:


----------
nosy: +dmalcolm

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13704>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:43:35 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 18:43:35 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326912215.81.0.00347393111501.issue10278@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

There are some issue on the Windows buildbots:



======================================================================
FAIL: test_wallclock (test.test_time.TimeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_time.py", line 340, in test_wallclock
    self.assertAlmostEqual(t, 0.1, places=2)
AssertionError: 0.09138312271531701 != 0.1 within 2 places

----------------------------------------------------------------------

----------
assignee:  -> haypo
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Wed Jan 18 19:59:57 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Wed, 18 Jan 2012 18:59:57 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326765231.61.0.190596727505.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F1716A6.7070903@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> 
> Patch version 7:
>  - Make PyOS_URandom() private (renamed to _PyOS_URandom)
>  - os.urandom() releases the GIL for I/O operation for its implementation reading /dev/urandom
>  - move _Py_unicode_hash_secret_t documentation into unicode_hash()
> 
> I moved also fixes for tests in a separated patch: random_fix-tests.patch.

Don't you think that the number of corrections you have to apply in order
to get the tests working again shows how much impact such a change would
have in real-world applications ?

Perhaps we should start to think about a compromise: make both the
collision counting and the hash seeding optional and let the user
decide which option is best.

BTW: The patch still includes the unnecessary _Py_unicode_hash_secret.suffix
which needlessly complicates the code and doesn't any additional
protection against hash value collisions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 20:08:19 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Wed, 18 Jan 2012 19:08:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1716A6.7070903@egenix.com>
Message-ID: <CAP7+vJKEN1dqMbnt1wShnkNjE1e=87scESw+bBr4hp8rjOqJLQ@mail.gmail.com>


Guido van Rossum <guido at python.org> added the comment:

On Wed, Jan 18, 2012 at 10:59 AM, Marc-Andre Lemburg <report at bugs.python.org
> wrote:

>
> Marc-Andre Lemburg <mal at egenix.com> added the comment:
>
> STINNER Victor wrote:
> >
> > Patch version 7:
> >  - Make PyOS_URandom() private (renamed to _PyOS_URandom)
> >  - os.urandom() releases the GIL for I/O operation for its
> implementation reading /dev/urandom
> >  - move _Py_unicode_hash_secret_t documentation into unicode_hash()
> >
> > I moved also fixes for tests in a separated patch:
> random_fix-tests.patch.
>
> Don't you think that the number of corrections you have to apply in order
> to get the tests working again shows how much impact such a change would
> have in real-world applications ?
>
> Perhaps we should start to think about a compromise: make both the
> collision counting and the hash seeding optional and let the user
> decide which option is best.
>

I like this, esp. if for old releases the collision counting is on by
default and the hash seeding is off by default, while in 3.3 both should be
on by default. Different env vars or flags should be used to enable/disable
them.

> BTW: The patch still includes the unnecessary
> _Py_unicode_hash_secret.suffix
> which needlessly complicates the code and doesn't any additional
> protection against hash value collisions.
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 20:23:55 2012
From: report at bugs.python.org (Miguel Godinho)
Date: Wed, 18 Jan 2012 19:23:55 +0000
Subject: [issue13818] argparse: -h listening required options under optional
	arguments
Message-ID: <1326914635.35.0.671947933448.issue13818@psf.upfronthosting.co.za>


New submission from Miguel Godinho <m at miguelgodinho.com>:

Adding a 'required optional argument' as with:
```
app.add_argument('--dbsnp', required=True)
```

will still result on having that argument listed under the optional when the app is called with the help option (-h)

Please note that the usage line is rendered ok (no square brackets around the 'required optional argument').

----------
components: Library (Lib)
messages: 151562
nosy: mgodinho
priority: normal
severity: normal
status: open
title: argparse: -h listening required options under optional arguments
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13818>
_______________________________________

From report at bugs.python.org  Wed Jan 18 21:20:11 2012
From: report at bugs.python.org (=?utf-8?q?Lars_Gust=C3=A4bel?=)
Date: Wed, 18 Jan 2012 20:20:11 +0000
Subject: [issue13815] tarfile.ExFileObject can't be wrapped using
	io.TextIOWrapper
In-Reply-To: <1326892577.13.0.456815776311.issue13815@psf.upfronthosting.co.za>
Message-ID: <1326918011.56.0.193933459114.issue13815@psf.upfronthosting.co.za>


Changes by Lars Gust?bel <lars at gustaebel.de>:


----------
assignee:  -> lars.gustaebel
nosy: +lars.gustaebel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13815>
_______________________________________

From report at bugs.python.org  Wed Jan 18 21:50:17 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 20:50:17 +0000
Subject: [issue13819] _warnings settings are process-wide
Message-ID: <1326919817.29.0.252163037725.issue13819@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

The settings in the C _warnings module are system-wide instead of being interpreter-wide. It seems the latter would be more desireable (and probably more compatible with the Python warnings module).

----------
components: Extension Modules, Interpreter Core
messages: 151563
nosy: brett.cannon, pitrou
priority: low
severity: normal
status: open
title: _warnings settings are process-wide
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13819>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:01:05 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 18 Jan 2012 21:01:05 +0000
Subject: [issue13763] Potentially hard to understand wording in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326920465.82.0.0131326586619.issue13763@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I read 'program name' as referring to 'Mercurial', not 'hg'. Perhaps Tshepang did also. Read that way, it is not right. Reading it the intended way is not so obvious to one who has never typed 'hg' on a command line. It would be impossible for one who does not even know that hg is the executable name.

Given that the intended audience is not limited to experienced hg users, I suggest changing "(commonly abbreviated hg, after the program name)" to the much clearer and explicit "The Mercurial executable is named 'hg', and 'hg' is often used to refer to Mercurial itself." (This is just a slight rewording of what ?ric said is meant to be said.) 

I agree that we should not discuss the origin of 'hg'. The chemical symbol is 'Hg', whereas 'hg' (normally) abbreviates 'hectogram'. My suggestion simply states two facts that beginning cpython developers need to know.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:04:53 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Wed, 18 Jan 2012 21:04:53 +0000
Subject: [issue13819] _warnings settings are process-wide
In-Reply-To: <1326919817.29.0.252163037725.issue13819@psf.upfronthosting.co.za>
Message-ID: <1326920693.73.0.411916906931.issue13819@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13819>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:05:30 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 21:05:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAP7+vJKEN1dqMbnt1wShnkNjE1e=87scESw+bBr4hp8rjOqJLQ@mail.gmail.com>
Message-ID: <1326920621.3395.69.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I like this, esp. if for old releases the collision counting is on by
> default and the hash seeding is off by default, while in 3.3 both should be
> on by default. Different env vars or flags should be used to enable/disable
> them.

I would hope 3.3 only gets randomized hashing. Collision counting is a
hack to make bugfix releases 99.999%-compatible instead of 99.9% ;)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:10:50 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Wed, 18 Jan 2012 21:10:50 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326920621.3395.69.camel@localhost.localdomain>
Message-ID: <CAP7+vJKcyJDzMfHWd2Lz=nbfHQc8g6xVDmSBn83KZ1q-=_Wikw@mail.gmail.com>


Guido van Rossum <guido at python.org> added the comment:

On Wed, Jan 18, 2012 at 1:05 PM, Antoine Pitrou <report at bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou at free.fr> added the comment:
>
> > I like this, esp. if for old releases the collision counting is on by
> > default and the hash seeding is off by default, while in 3.3 both should
> be
> > on by default. Different env vars or flags should be used to
> enable/disable
> > them.
>
> I would hope 3.3 only gets randomized hashing. Collision counting is a
> hack to make bugfix releases 99.999%-compatible instead of 99.9% ;)
>

Really? I'd expect the difference to be more than 2 nines. The randomized
hashing has two problems: (a) change in dict order; (b) hash varies between
processes. I cannot imagine counterexamples to the collision counting that
weren't constructed specifically as counterexamples.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:14:02 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 21:14:02 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAP7+vJKcyJDzMfHWd2Lz=nbfHQc8g6xVDmSBn83KZ1q-=_Wikw@mail.gmail.com>
Message-ID: <1326921133.3395.72.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Really? I'd expect the difference to be more than 2 nines. The randomized
> hashing has two problems: (a) change in dict order; (b) hash varies between
> processes.

Personally I don't think the change in dict order is a problem (hashing
already changes between 32-bit and 64-bit builds, and we sometimes
change the calculation too: it might change *more* often with random
hashes, while it went unnoticed in some cases before). So only (b) is a
problem and I don't think it affects more than 0.01% of
applications/users :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:19:27 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Wed, 18 Jan 2012 21:19:27 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326921567.46.0.180043144258.issue13781@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

Looks like you've got commit privs (yay) so i'm assigning this to you to take care of that way for 2.7 as well.

I'd add a comment to the fdopen C code where the "<fdopen>" constant lives as well as to the gzip.py module around the special case for this mentioning that they should be kept in sync.  (not that either is _ever_ likely to be changed in 2.7)

----------
assignee: gregory.p.smith -> nadeem.vawda
versions:  -Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:35:48 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 21:35:48 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <E1RndAY-0003xK-Jz@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 46b245f03f54 by Antoine Pitrou in branch '3.2':
Issue #13722: Avoid silencing ImportErrors when initializing the codecs registry.
http://hg.python.org/cpython/rev/46b245f03f54

New changeset f55529aa023d by Antoine Pitrou in branch 'default':
Issue #13722: Avoid silencing ImportErrors when initializing the codecs registry.
http://hg.python.org/cpython/rev/f55529aa023d

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:37:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 21:37:47 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <E1RndCT-00047y-Q2@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 83e8c3a6a81c by Antoine Pitrou in branch 'default':
Be more lenient in test_wallclock (issue #10278).
http://hg.python.org/cpython/rev/83e8c3a6a81c

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:39:29 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 21:39:29 +0000
Subject: [issue13722] "distributions can disable the encodings package"
In-Reply-To: <1325881942.74.0.535921078744.issue13722@psf.upfronthosting.co.za>
Message-ID: <1326922769.79.0.792891966307.issue13722@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13722>
_______________________________________

From report at bugs.python.org  Wed Jan 18 22:39:43 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 21:39:43 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <1326922783.09.0.520220229013.issue10278@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Wed Jan 18 23:27:25 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 22:27:25 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326925645.14.0.646529351624.issue13811@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

Eric V. Smith wrote:
> I'm not sure what you're saying here. Is it that 'xx' should be ignored?

yes, the description says they are assumed absent

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 23:41:58 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 18 Jan 2012 22:41:58 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <E1RneCa-0001et-Fn@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a08e9e84f33f by Nadeem Vawda in branch '2.7':
Issue #13781: Fix GzipFile to work with os.fdopen()'d file objects.
http://hg.python.org/cpython/rev/a08e9e84f33f

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Wed Jan 18 23:52:12 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Wed, 18 Jan 2012 22:52:12 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326927132.98.0.302614053594.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

The only error is the text of the ValueError. I'll look into fixing that. These characters will not be ignored.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Wed Jan 18 23:52:46 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 18 Jan 2012 22:52:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1716A6.7070903@egenix.com>
Message-ID: <CAMpsgwav+2G-BZ4fw_7HU9zev-L2zk4Jwzsb_UBpjn-TNPDtCA@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Don't you think that the number of corrections you have to apply in order
> to get the tests working again shows how much impact such a change would
> have in real-world applications ?

Let see the diffstat:

 Doc/using/cmdline.rst                       |    7
 Include/pythonrun.h                         |    2
 Include/unicodeobject.h                     |    6
 Lib/json/__init__.py                        |    4
 Lib/os.py                                   |   17 -
 Lib/packaging/create.py                     |    7
 Lib/packaging/tests/test_create.py          |   18 -
 Lib/test/mapping_tests.py                   |    2
 Lib/test/regrtest.py                        |    5
 Lib/test/test_builtin.py                    |    1
 Lib/test/test_dis.py                        |   36 ++-
 Lib/test/test_gdb.py                        |   11 -
 Lib/test/test_inspect.py                    |    1
 Lib/test/test_os.py                         |   35 ++-
 Lib/test/test_set.py                        |   25 ++
 Lib/test/test_unicode.py                    |   39 ++++
 Lib/test/test_urllib.py                     |   16 -
 Lib/test/test_urlparse.py                   |    6
 Lib/tkinter/test/test_ttk/test_functions.py |    2
 Makefile.pre.in                             |    1
 Modules/posixmodule.c                       |  126 ++-----------
 Objects/unicodeobject.c                     |   20 +-
 PCbuild/pythoncore.vcproj                   |    4
 Python/pythonrun.c                          |    3
 Python/random.c                             |  268 ++++++++++++++++++++++++++++
 25 files changed, 488 insertions(+), 174 deletions(-)

Except Lib/packaging/create.py, all other changes are related to the
introduction of the randomized hash function, or fix tests... Even
Lib/packaging/create.py change is related to fixing tests. The test
can be changed differently, but I like the idea of having always the
same output in packaging (e.g. it is more readable for the user if
files are sorted).

I expected to have to do something on multiprocessing, but nope, it
doesn't care of the hash value.

So I expect something similar in applications: no change in the
applications, but a lot of hacks/tricks in tests.

> Perhaps we should start to think about a compromise: make both the
> collision counting and the hash seeding optional and let the user
> decide which option is best.

I don't think that we need two fixes for a single vulnerability (in
the same Python version), one is enough. If we decide to count
collisions, the randomized hash idea can be simply dropped. But we may
use a different fix for Python 3.3 and for stable versions (e.g. count
collisions for stable versions and use randomized hash for 3.3).

> BTW: The patch still includes the unnecessary _Py_unicode_hash_secret.suffix
> which needlessly complicates the code and doesn't any additional
> protection against hash value collisions

How does it complicate the code? It adds an extra XOR to hash(str) and
4 or 8 bytes in memory, that's all. It is more difficult to compute
the secret from hash(str) output if there is a prefix *and* a suffix.
If there is only a prefix, knowning a single hash(str) value is just
enough to retrieve directly the secret.
.
> I don't think it affects more than 0.01% of applications/users :)

It would help to try a patched Python on a real world application like
Django to realize how much code is broken (or not) by a randomized
hash function.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 18 23:55:13 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 18 Jan 2012 22:55:13 +0000
Subject: [issue10109] itertools.product with infinite iterator cause
	MemoryError.
In-Reply-To: <1287103502.64.0.571191585998.issue10109@psf.upfronthosting.co.za>
Message-ID: <1326927313.91.0.741022863307.issue10109@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Proposing an expansion of the definition of product() is a *completely* different issue from the validity of count() as an input. I answered correctly given the current definition of product(): it is not valid input. It is also not valid input to your proposed revision:
>>> tuple(itertools.cycle(enumerate(it)) for it in itertools.count())
...
TypeError: 'int' object is not iterable
-- just as I said.

If you want to propose an enhancement, either open an new, enhancement issue or post to python-ideas. Since new features can only go in 3.3+, post 3.x code, not 2.x. And please do not quibble about the difference between 'infinite' and 'too large to fit in memory'.

----------
stage:  -> committed/rejected
versions: +Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:00:20 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 23:00:20 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326927620.72.0.60985114601.issue13811@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

"If the second character of format_spec is not a valid alignment option, then it is assumed that both the fill character and the alignment option are absent."

what does it mean ?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:04:05 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 18 Jan 2012 23:04:05 +0000
Subject: [issue13811] In str.format an incorrect alignment option doesn't make
	fill char and onself absent
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326927845.3.0.520725844833.issue13811@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

py.user: The format string must always match the grammar, which is just
above the paragraph that you quoted: 

   [[fill]align][sign][#][0][width][,][.precision][type]


Thus, if fill and align are absent, it does not mean that you can add
arbitrary characters like "xx".



I think the docs are crystal clear; I also prefer Eric's suggestion
for a new error message.

----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:10:56 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Wed, 18 Jan 2012 23:10:56 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326928256.95.0.636856576956.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

Changing to 3.3: I don't think applying this to 3.2 would be appropriate.

----------
assignee:  -> eric.smith
keywords: +easy
priority: normal -> low
stage:  -> needs patch
title: In str.format an incorrect alignment option doesn't make fill char and onself absent -> In str.format, if invalid fill and alignment are specified, the text of the ValueError message is misleading.
versions: +Python 3.3 -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:13:38 2012
From: report at bugs.python.org (Sumudu Fernando)
Date: Wed, 18 Jan 2012 23:13:38 +0000
Subject: [issue10109] itertools.product with infinite iterator cause
	MemoryError.
In-Reply-To: <1287103502.64.0.571191585998.issue10109@psf.upfronthosting.co.za>
Message-ID: <1326928418.62.0.692296920767.issue10109@psf.upfronthosting.co.za>


Sumudu Fernando <sumuduf at gmail.com> added the comment:

>>> tuple(itertools.cycle(enumerate(it)) for it in itertools.count())
  ...
  TypeError: 'int' object is not iterable

That is not what happens in the function, though!  That would correspond to doing product(*itertools.count(2010)), but if you try that you won't even get past argument expansion (obviously).  Doing product(*xrange(10)) gives the error you're talking about, for example.

product(itertools.count(2010)) works perfectly well with the version I posted, though it is a bit silly to do it that way since it produces the same values as count itself (which is what "cartesian product" should do), while saving extra bookkeeping along the way.

Anyway, I'm pretty new to python and I don't think this is quite relevant enough to warrant opening a new ticket.  I'm happy to leave it here for the education of the next neophyte who stumbles across this idiosyncracy of itertools.product.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:15:34 2012
From: report at bugs.python.org (py.user)
Date: Wed, 18 Jan 2012 23:15:34 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326928534.83.0.870965496942.issue13811@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

Stefan Krah wrote:
> Thus, if fill and align are absent, it does not mean that you can add
arbitrary characters like "xx".

the descriptions says in other words:
"if you have used an incorrect alignment option, then the interpreter behaves like you didn't use fill and alignment"

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:16:41 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 18 Jan 2012 23:16:41 +0000
Subject: [issue13781] gzip module does the wrong thing with an os.fdopen()'ed
	fileobj
In-Reply-To: <1326493916.44.0.574522946696.issue13781@psf.upfronthosting.co.za>
Message-ID: <1326928601.53.0.482115048109.issue13781@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Done.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13781>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:23:13 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 18 Jan 2012 23:23:13 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326928993.39.0.703997645895.issue13703@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

A possible advantage of having the 3.3 fix available in earlier versions is that people will be able to turn it on and have that be the *only* change -- just as with __future__ imports done one at a time.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:25:38 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Wed, 18 Jan 2012 23:25:38 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAP7+vJKcyJDzMfHWd2Lz=nbfHQc8g6xVDmSBn83KZ1q-=_Wikw@mail.gmail.com>
Message-ID: <CAGE7PNKD0eh4MyKFcQ5Zd-OLt7i8-mgMJmxcWupS4Xs6Ncdsgw@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

On Wed, Jan 18, 2012 at 1:10 PM, Guido van Rossum
<report at bugs.python.org> wrote:
> On Wed, Jan 18, 2012 at 1:05 PM, Antoine Pitrou <report at bugs.python.org>wrote:
> >
> > I would hope 3.3 only gets randomized hashing. Collision counting is a
> > hack to make bugfix releases 99.999%-compatible instead of 99.9% ;)
> >
>
> Really? I'd expect the difference to be more than 2 nines. The randomized
> hashing has two problems: (a) change in dict order; (b) hash varies between
> processes. I cannot imagine counterexamples to the collision counting that
> weren't constructed specifically as counterexamples.

For the purposes of 3.3 I'd prefer to just have randomized hashing and
not the collision counting in order to keep things from getting too
complicated.  But I will not object if we opt to do both.

As much as the counting idea rubs me wrong, even if it were on by
default I agree that most non-contrived things will never encounter it
and it is easy to document how to work around it by disabling it
should anyone actually be impeded by it.

The concern I have with that approach from a web service point of view
is that it too can be gamed in the more rare server situation of
someone managing to fill a persistent data structure up with enough
colliding values to be _close_ to the limit such that the application
then dies while trying to process all future requests that _hit_ the
limit (a persisting 500 error DOS rather than an exception raised only
in one offending request that deserved that 500 error anyways). Not
nearly as likely a scenario but it is one I'd keep an eye open for
with an attacker hat on.

MvL's suggestion of using AVL trees for hash bucket slots instead of
our linear slot finding algorithm is a better way to fix the ultimate
problem by never devolving into linear behavior at all. It is
naturally more complicated but could likely even be done while
maintaining ABI compatibility. I haven't pondered designs and
performance costs for that. Possibly a memory hit and one or two extra
indirect lookups in the normal case and some additional complexity in
the iteration case.

-gps

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:30:12 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 23:30:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNKD0eh4MyKFcQ5Zd-OLt7i8-mgMJmxcWupS4Xs6Ncdsgw@mail.gmail.com>
Message-ID: <1326929302.3395.75.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> MvL's suggestion of using AVL trees for hash bucket slots instead of
> our linear slot finding algorithm is a better way to fix the ultimate
> problem by never devolving into linear behavior at all.

A dict can contain non-orderable keys, I don't know how an AVL tree can
fit into that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:31:26 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Wed, 18 Jan 2012 23:31:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326929302.3395.75.camel@localhost.localdomain>
Message-ID: <CAGE7PNLShv_oVZAXfPd-yT9_s6a9FDJJx6aqMYXN6okzuwAmrA@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

> A dict can contain non-orderable keys, I don't know how an AVL tree can
> fit into that.

good point!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:37:47 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Wed, 18 Jan 2012 23:37:47 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNKD0eh4MyKFcQ5Zd-OLt7i8-mgMJmxcWupS4Xs6Ncdsgw@mail.gmail.com>
Message-ID: <4F1757BA.8040306@udel.edu>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

> As much as the counting idea rubs me wrong,

FWIW, the original 2003 paper reported that the url-caching system that 
they tested used collision-counting to evade attacks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:40:41 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 18 Jan 2012 23:40:41 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326930041.96.0.244953106949.issue13811@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

The text speaks about the regular case of a second character that
is not a valid alignment character, e.g.:

    format(3.222, ".2f")

Clearly the '2' fulfills this criterion, so the parser knows that the
leading '.' is *not* a fill character. This is all that the text says.


But even in your irregular case the text is still correct: After
it has been established that [[fill]align] is not present you have
to match the *whole string* with the rest of the grammar:

    [sign][#][0][width][,][.precision][type]


There is no match for "xx10d", hence the error.



BTW, I think this is out of scope for the tracker now. If you
have further questions, you could ask on:

    http://mail.python.org/mailman/listinfo/python-list

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:42:36 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Wed, 18 Jan 2012 23:42:36 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326930156.77.0.87659655019.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

As I look at it a little closer, I think I'm going to change the message to: "Invalid format type specified". The code has determined that instead of a type that's a single character long, it's received "xx10d". That's because "xx" doesn't match any of "[[fill]align][sign][#][0][width][,][.precision]", so it must be the "[type]" field.

I'm open to a better message, though.

Due to the variable width chars in the format_spec string, include the "xx10d" along with the error text is a little complicated. But maybe including it would be an improvement: "Invalid format type 'xx10d' found, expected a single character".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:44:34 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Wed, 18 Jan 2012 23:44:34 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1757BA.8040306@udel.edu>
Message-ID: <CAP7+vJL4z9oOXEe0xHbqrJt1pOiU8bS2MH8AbCqT3BH39CeYOA@mail.gmail.com>


Guido van Rossum <guido at python.org> added the comment:

On Wed, Jan 18, 2012 at 3:37 PM, Terry J. Reedy <report at bugs.python.org>wrote:

>
> Terry J. Reedy <tjreedy at udel.edu> added the comment:
>
> > As much as the counting idea rubs me wrong,
>
> FWIW, the original 2003 paper reported that the url-caching system that
> they tested used collision-counting to evade attacks.

You mean as a fix or that they successfully attacked a collision-counting
system?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 00:46:12 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 18 Jan 2012 23:46:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1757BA.8040306@udel.edu>
Message-ID: <1326930263.3395.79.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > As much as the counting idea rubs me wrong,
> 
> FWIW, the original 2003 paper reported that the url-caching system that 
> they tested used collision-counting to evade attacks.

I think that was DJB's DNS server/cache actually.
But deciding to limit collisions in a specific application is not the
same as limiting them in the general case. Python dicts have a lot of
use cases that are not limited to storing URL parameters, domain names
or instance attributes: there is a greater risk of meeting pathological
cases with legitimate keys.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:00:02 2012
From: report at bugs.python.org (py.user)
Date: Thu, 19 Jan 2012 00:00:02 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326931202.67.0.189904568508.issue13811@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

Stefan Krah wrote:
> After it has been established that [[fill]align] is not present you have to match the *whole string* with the rest of the grammar

I think, there should be a text in the documentation: "if the alignment optiont is invalid, it will be raised a ValueError exception"

thanx for the mailing list

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:34:55 2012
From: report at bugs.python.org (Stefan Krah)
Date: Thu, 19 Jan 2012 00:34:55 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326930156.77.0.87659655019.issue13811@psf.upfronthosting.co.za>
Message-ID: <20120119003453.GB23755@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Eric V. Smith <report at bugs.python.org> wrote:
> As I look at it a little closer, I think I'm going to change the message to:
> "Invalid format type specified". The code has determined that instead of a
> type that's a single character long, it's received "xx10d". That's because
> "xx" doesn't match any of "[[fill]align][sign][#][0][width][,][.precision]",
> so it must be the "[type]" field.

I think this has the potential of being more confusing for people who are
not very familiar with the format specification mini-language. I didn't
look at the code now, but would the message also be raised for this spec?

    format(9, "xx10f")

> I'm open to a better message, though.

IMO "invalid format specifier" is fine. Even the existing error message
is not really terrible.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:38:05 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Thu, 19 Jan 2012 00:38:05 +0000
Subject: [issue13813] "sysconfig.py" and "distutils/util.py" redundancy
In-Reply-To: <1326859118.01.0.584725172013.issue13813@psf.upfronthosting.co.za>
Message-ID: <1326933485.14.0.630820595641.issue13813@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Could you possibly provide some background?. I am afraid I am not familiar with the situation. I want to learn :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13813>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:43:55 2012
From: report at bugs.python.org (Stefan Krah)
Date: Thu, 19 Jan 2012 00:43:55 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <20120119003453.GB23755@sleipnir.bytereef.org>
Message-ID: <20120119004354.GA23861@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Stefan Krah <report at bugs.python.org> wrote:
>> ["xx10d"]
> look at the code now, but would the message also be raised for this spec?
> 
>     format(9, "xx10f")

Argh, 'd' is of course also a valid type specifier.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:45:57 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 00:45:57 +0000
Subject: [issue13820] 2.6 is no longer in the future
Message-ID: <1326933957.4.0.830452813069.issue13820@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

http://docs.python.org/reference/lexical_analysis.html

Changed in version 2.5: Both as and with are only recognized when the with_statement future feature has been enabled. It will always be enabled in Python 2.6. See section The with statement for details. Note that using as and with as identifiers will always issue a warning, even when the with_statement future directive is not in effect.


That was reasonable wording for 2.5 itself, but at this point, I think it would be simpler to add a Changed in version 2.6 entry.  Perhaps:

Changed in version 2.5: Using as or with as identifiers triggers a warning.  Using them as statements requires the with_statement future feature.
Changed in Python 2.6: as and with became full keywords.

----------
assignee: docs at python
components: Documentation
messages: 151595
nosy: Jim.Jewett, docs at python
priority: normal
severity: normal
status: open
title: 2.6 is no longer in the future
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13820>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:46:44 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Thu, 19 Jan 2012 00:46:44 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326930263.3395.79.camel@localhost.localdomain>
Message-ID: <CAP7+vJ+qyQSvnTsG3a5b-83s+iO=E-uYPuz4yg7nCf+8YF4qQg@mail.gmail.com>


Guido van Rossum <guido at python.org> added the comment:

On Wed, Jan 18, 2012 at 3:46 PM, Antoine Pitrou <report at bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou at free.fr> added the comment:
>
> > > As much as the counting idea rubs me wrong,
> >
> > FWIW, the original 2003 paper reported that the url-caching system that
> > they tested used collision-counting to evade attacks.
>
> I think that was DJB's DNS server/cache actually.
> But deciding to limit collisions in a specific application is not the
> same as limiting them in the general case. Python dicts have a lot of
> use cases that are not limited to storing URL parameters, domain names
> or instance attributes: there is a greater risk of meeting pathological
> cases with legitimate keys.
>

Really? This sounds like FUD to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:54:19 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 00:54:19 +0000
Subject: [issue13821] misleading return from isidentifier
Message-ID: <1326934459.59.0.614334427327.issue13821@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

Python identifiers are in NFKC form; string method .isidentifier() returns true on strings that are not in that form.  In some contexts, these non-canonical strings will be replaced with their NFKC equivalent, but in other contexts (such as the builtins hasattr, getattr, delattr) they will not.


>>> cha=chr(170)
>>> cha
'?'

>>> cha.isidentifier()
True

>>> uc.normalize("NFKC", cha)
'a'

>>> obj.? = 5
>>> hasattr(obj, "?")
False
>>> obj.a
5

----------
components: Unicode
messages: 151597
nosy: Jim.Jewett, ezio.melotti
priority: normal
severity: normal
status: open
title: misleading return from isidentifier

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:54:45 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Thu, 19 Jan 2012 00:54:45 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <1326934485.72.0.678976232178.issue13811@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

The existing exceptions use the text "format code" for what the documentation calls "type":

>>> format(9, "h")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'h' for object of type 'int'

So to be consistent, it should say:

>>> format(9, "xx10f")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format code

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Thu Jan 19 01:56:33 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 00:56:33 +0000
Subject: [issue13821] misleading return from isidentifier
In-Reply-To: <1326934459.59.0.614334427327.issue13821@psf.upfronthosting.co.za>
Message-ID: <1326934593.74.0.932535839419.issue13821@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I don't see why that's invalid. "str.isidentifier()" returning True means Python will accept it as an identifier.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 02:05:13 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 01:05:13 +0000
Subject: [issue13821] misleading return from isidentifier
In-Reply-To: <1326934459.59.0.614334427327.issue13821@psf.upfronthosting.co.za>
Message-ID: <1326935113.08.0.846670743641.issue13821@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

My preference would be for non_NFKC.isidentifier() to return False, but that may be a problem for backwards compatibility.

It *may* be worth adding an asidentifier() method that returns either False or the canonicalized string that should be used instead.

At a minimum, the documentation (including docstring) should warn that the method doesn't check for NFKC form, and that if the input is not ASCII, the caller should first ensure this by calling str1=unicodedata.normalize("NFKC", str1)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 02:06:31 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 01:06:31 +0000
Subject: [issue13821] misleading return from isidentifier
In-Reply-To: <1326935113.08.0.846670743641.issue13821@psf.upfronthosting.co.za>
Message-ID: <CAPZV6o8RkKbfHkizf9o4qfvACLEDK-GbSPwaxeRK3WE0wR=C=g@mail.gmail.com>


Benjamin Peterson <benjamin at python.org> added the comment:

2012/1/18 Jim Jewett <report at bugs.python.org>:
>
> Jim Jewett <jimjjewett at gmail.com> added the comment:
>
> My preference would be for non_NFKC.isidentifier() to return False

It *is* an identifier, though. Python will happily accept it.

>
> It *may* be worth adding an asidentifier() method that returns either False or the canonicalized string that should be used instead.
>
> At a minimum, the documentation (including docstring) should warn that the method doesn't check for NFKC form, and that if the input is not ASCII, the caller should first ensure this by calling str1=unicodedata.normalize("NFKC", str1)

Sounds fine to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 02:07:54 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 01:07:54 +0000
Subject: [issue13821] misleading return from isidentifier
In-Reply-To: <1326934459.59.0.614334427327.issue13821@psf.upfronthosting.co.za>
Message-ID: <1326935274.7.0.498177239166.issue13821@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

@Benjamin -- the catch is, if it isn't already in NFKC form, then python won't really accept it as an identifier.  Sometimes it will silently canonicalize it for you so that it seems to work, but other times it won't.  And program calling isidentifier is likely to be a program that uses the strings directly for access, instead of always routing them through the parser.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 02:10:04 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 01:10:04 +0000
Subject: [issue13821] misleading return from isidentifier
In-Reply-To: <1326935274.7.0.498177239166.issue13821@psf.upfronthosting.co.za>
Message-ID: <CAPZV6o9f1_JVsspj+3uf0BWHS0wzMYjaRySuoqFk8b+uCw=a_w@mail.gmail.com>


Benjamin Peterson <benjamin at python.org> added the comment:

2012/1/18 Jim Jewett <report at bugs.python.org>:
>
> Jim Jewett <jimjjewett at gmail.com> added the comment:
>
> @Benjamin -- the catch is, if it isn't already in NFKC form, then python won't really accept it as an identifier. ?Sometimes it will silently canonicalize it for you so that it seems to work, but other times it won't. ?And program calling isidentifier is likely to be a program that uses the strings directly for access, instead of always routing them through the parser.

AFAIK, the only time it will "silently" canonicalize it for you is
parsing. Even if it wasn't, you can't say it's not an identifier, it's
just not normalized.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13821>
_______________________________________

From report at bugs.python.org  Thu Jan 19 02:15:25 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 19 Jan 2012 01:15:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAP7+vJL4z9oOXEe0xHbqrJt1pOiU8bS2MH8AbCqT3BH39CeYOA@mail.gmail.com>
Message-ID: <4F176E88.3090503@udel.edu>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

> You mean as a fix or that they successfully attacked a collision-counting
> system?

Successful anticipation and blocking of hash attack: after a chain of 
100 DNS 'treats the request as a cache miss'. What is somewhat special 
for this app is being able to bail at that point. Crosby & Wallach still 
think 'his fix could be improved', I presume by using one of their 
recommended hashes.
http://www.cs.rice.edu/~scrosby/hash/CrosbyWallach_UsenixSec2003.pdf
section 3.2, DJB DNS server; section 5, fixes

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 04:44:26 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 19 Jan 2012 03:44:26 +0000
Subject: [issue10109] itertools.product with infinite iterator cause
	MemoryError.
In-Reply-To: <1287103502.64.0.571191585998.issue10109@psf.upfronthosting.co.za>
Message-ID: <1326944666.43.0.932824448111.issue10109@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

A relatively simple change would be to allow the first iterable to be 'infinite', when repeat==1, by not calling tuple() on it. The reason for turning the iterables into concrete sequences is because they might not be reiterable. (cycle() does the same for the same reason.) But since the first iterable is only iterated once, this does not apply to it.

    if repeat == 1:
        pools = [args[0:1]].extend(tuple(pool) for pool in args[1:])
    else:
        pools = [tuple(pool) for pool in args] * repeat

The counter argument to this or any generalized proposal is that one can expand the product() into enough loops to avoid infinite (or very large) args. For example, the following produces '1AA', '1AB', ..., '1EE', '2AA', ... indefinitely.

naa=(''.join((str(n),)+s) for n in itertools.count(1)
     for s in itertools.product(string.ascii_uppercase[0:5], repeat=2))

RAYMOND: Do you think the doc should specify that each iterable must be finite, and that explicit loops are the alternative if not?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________

From report at bugs.python.org  Thu Jan 19 05:22:55 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 04:22:55 +0000
Subject: [issue13822] is(upper/lower/title) are not exactly correct
Message-ID: <1326946975.13.0.733756833132.issue13822@psf.upfronthosting.co.za>


New submission from Benjamin Peterson <benjamin at python.org>:

The isupper() and islower() methods currently use the Lowercase and Uppercase derived properties. Technically, they should use the Changes_When_(Lowercased/Uppercased/Titlecased) and Changes_When_Casemapped derived properties to compute their results.

----------
components: Unicode
messages: 151606
nosy: benjamin.peterson, ezio.melotti
priority: normal
severity: normal
status: open
title: is(upper/lower/title) are not exactly correct
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13822>
_______________________________________

From report at bugs.python.org  Thu Jan 19 07:46:56 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 06:46:56 +0000
Subject: [issue2134] Add new attribute to TokenInfo to report specific token
	IDs
In-Reply-To: <1203289230.78.0.790200923315.issue2134@psf.upfronthosting.co.za>
Message-ID: <E1Rnllu-0004r1-6H@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 75baef657770 by Meador Inge in branch '2.7':
Issue #2134: Clarify token.OP handling rationale in tokenize documentation.
http://hg.python.org/cpython/rev/75baef657770

New changeset dfd74d752b0e by Meador Inge in branch '3.2':
Issue #2134: Clarify token.OP handling rationale in tokenize documentation.
http://hg.python.org/cpython/rev/dfd74d752b0e

New changeset f4976fa6e830 by Meador Inge in branch 'default':
Issue #2134: Add support for tokenize.TokenInfo.exact_type.
http://hg.python.org/cpython/rev/f4976fa6e830

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
_______________________________________

From report at bugs.python.org  Thu Jan 19 07:49:14 2012
From: report at bugs.python.org (Meador Inge)
Date: Thu, 19 Jan 2012 06:49:14 +0000
Subject: [issue2134] Add new attribute to TokenInfo to report specific token
	IDs
In-Reply-To: <1203289230.78.0.790200923315.issue2134@psf.upfronthosting.co.za>
Message-ID: <1326955754.57.0.544644383467.issue2134@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

Fixed.  Thanks for the reviews everyone.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
_______________________________________

From report at bugs.python.org  Thu Jan 19 08:11:11 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 07:11:11 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <E1Rnm9O-0006XC-5q@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2bd7f40108b4 by Meador Inge in branch 'default':
Issue #12705: Raise SyntaxError when compiling multiple statements as single interactive statement
http://hg.python.org/cpython/rev/2bd7f40108b4

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Thu Jan 19 08:12:15 2012
From: report at bugs.python.org (Meador Inge)
Date: Thu, 19 Jan 2012 07:12:15 +0000
Subject: [issue12705] Make compile('1\n2\n', '',
	'single') raise an exception instead of silently truncating?
In-Reply-To: <1312681191.75.0.586407900237.issue12705@psf.upfronthosting.co.za>
Message-ID: <1326957135.36.0.634593479197.issue12705@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

Fixed in 3.3.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12705>
_______________________________________

From report at bugs.python.org  Thu Jan 19 08:23:09 2012
From: report at bugs.python.org (Florent Xicluna)
Date: Thu, 19 Jan 2012 07:23:09 +0000
Subject: [issue7980] time.strptime not thread safe
In-Reply-To: <1266835598.13.0.361031999331.issue7980@psf.upfronthosting.co.za>
Message-ID: <1326957789.74.0.732678498322.issue7980@psf.upfronthosting.co.za>


Changes by Florent Xicluna <florent.xicluna at gmail.com>:


----------
nosy: +flox

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7980>
_______________________________________

From report at bugs.python.org  Thu Jan 19 08:23:42 2012
From: report at bugs.python.org (Florent Xicluna)
Date: Thu, 19 Jan 2012 07:23:42 +0000
Subject: [issue11108] Intermittent AttributeError when using time.strptime in
	threads
In-Reply-To: <1296767289.19.0.293158150558.issue11108@psf.upfronthosting.co.za>
Message-ID: <1326957822.32.0.53454677873.issue11108@psf.upfronthosting.co.za>


Changes by Florent Xicluna <florent.xicluna at gmail.com>:


----------
nosy: +flox

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11108>
_______________________________________

From report at bugs.python.org  Thu Jan 19 09:45:08 2012
From: report at bugs.python.org (=?utf-8?q?Martin_H=C3=A4cker?=)
Date: Thu, 19 Jan 2012 08:45:08 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1326962708.52.0.33145001056.issue13804@psf.upfronthosting.co.za>


Martin H?cker <spamfaenger at gmx.de> added the comment:

Jup - oh the joys of writing code in a bugtracker :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Thu Jan 19 09:51:47 2012
From: report at bugs.python.org (patrick vrijlandt)
Date: Thu, 19 Jan 2012 08:51:47 +0000
Subject: [issue13823] xml.etree.ElementTree.ElementTree.write - argument
	checking
Message-ID: <1326963107.43.0.928827185581.issue13823@psf.upfronthosting.co.za>


New submission from patrick vrijlandt <patrick.vrijlandt at gmail.com>:

(1) The docs say:
xml_declaration controls if an XML declaration should be added to the file. Use False for never, True for always, None for only if not US-ASCII or UTF-8 or Unicode (default is None). 

The method also accepts other values, like xml_declaration = "yes". This behavior should be documented, or raise a ValueError (up to effbot, I think)

(2) The docs say (in a note):
The encoding string included in XML output should conform to the appropriate standards. For example, ?UTF-8? is valid, but ?UTF8? is not. See http://www.w3.org/

But the method accepts both values, (?UTF-8? and ?UTF8?). Since this will result in invalid xml, (but not invalid python) it should probably raise ValueError too.

(3) Open issue 9458 also refers to this method. It might be wise to raise ValueError if the encoding does not match the (mode of the) file target (binary or text).

----------
assignee: docs at python
components: Documentation, XML
messages: 151612
nosy: docs at python, patrick.vrijlandt
priority: normal
severity: normal
status: open
title: xml.etree.ElementTree.ElementTree.write - argument checking
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13823>
_______________________________________

From report at bugs.python.org  Thu Jan 19 11:32:52 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 10:32:52 +0000
Subject: [issue11948] Tutorial/Modules - small fix to better clarify the
	modules search path
In-Reply-To: <1304011111.24.0.834772686878.issue11948@psf.upfronthosting.co.za>
Message-ID: <E1RnpIZ-0007P2-0g@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset d01208ba482f by Sandro Tosi in branch '2.7':
Issue #11948: clarify modules search path
http://hg.python.org/cpython/rev/d01208ba482f

New changeset 6d663db63705 by Sandro Tosi in branch '3.2':
Issue #11948: clarify modules search path
http://hg.python.org/cpython/rev/6d663db63705

New changeset 93769b8ff40b by Sandro Tosi in branch 'default':
Issue #11948: merge with 3.2
http://hg.python.org/cpython/rev/93769b8ff40b

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11948>
_______________________________________

From report at bugs.python.org  Thu Jan 19 11:34:17 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 10:34:17 +0000
Subject: [issue11948] Tutorial/Modules - small fix to better clarify the
	modules search path
In-Reply-To: <1304011111.24.0.834772686878.issue11948@psf.upfronthosting.co.za>
Message-ID: <1326969257.13.0.120828457766.issue11948@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Thanks Terry for the ping, I've just committed it - thanks again to everyone for the help/suggestions.

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11948>
_______________________________________

From report at bugs.python.org  Thu Jan 19 13:25:39 2012
From: report at bugs.python.org (David Layton)
Date: Thu, 19 Jan 2012 12:25:39 +0000
Subject: [issue13824] argparse.FileType opens a file without excepting
	resposibility for closing it
Message-ID: <1326975939.94.0.524131708506.issue13824@psf.upfronthosting.co.za>


New submission from David Layton <dmlayton at gmail.com>:

argparse.FileType.__call__ opens the specified file and returns it. This is well documented as an anit-idiom in http://docs.python.org/howto/doanddont.html#exceptions. 

"...a serious problem ? due to implementation details in CPython, the file would not be closed when an exception is raised until the exception handler finishes; and, worse, in other implementations (e.g., Jython) it might not be closed at all regardless of whether or not an exception is raised."

Disregarding the above, handling a file which may or may not have been opened depending the users input requires a bit of boilerplate code compared to the usual with-open idiom.  

Additionally, there is no way to prevent FileType from clobbering an existing file when used with write mode. 

Given these issues and others, it seems to me that the usefulness of FileType is outweighed by propensity to encourage bad coding. Perhaps, it would be best if FileType (or some replacement) simply checked that the file exists (when such a check is appropriate), it can be opened in the specified mode, and, curry the call to open (i.e. return lambda: open(string, self._mode, self._bufsize))

----------
components: Library (Lib)
messages: 151615
nosy: David.Layton, Paolo.Elvati, Stefan.Pfeiffer, bethard, manveru
priority: normal
severity: normal
status: open
title: argparse.FileType opens a file without excepting resposibility for closing it
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13824>
_______________________________________

From report at bugs.python.org  Thu Jan 19 13:54:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 19 Jan 2012 12:54:46 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1326977686.53.0.70296380901.issue13817@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:01:32 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 19 Jan 2012 13:01:32 +0000
Subject: [issue13813] "sysconfig.py" and "distutils/util.py" redundancy
In-Reply-To: <1326859118.01.0.584725172013.issue13813@psf.upfronthosting.co.za>
Message-ID: <1326978092.7.0.0922774389299.issue13813@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Background on the distutils freeze?  Right now I don?t have the time and I?m going to be offline until the end of the month.  You can look for Tarek Ziad??s blog posts after PyCon 2010, or wait until I come back and put the links on a wiki page :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13813>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:03:20 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 19 Jan 2012 13:03:20 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326978200.16.0.354886914251.issue13703@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> Even Lib/packaging/create.py change is related to fixing tests. The test can be changed
> differently, but I like the idea of having always the same output in packaging (e.g. it is
> more readable for the user if files are sorted).

See #13712 for why this is a fake fix.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:03:20 2012
From: report at bugs.python.org (mike)
Date: Thu, 19 Jan 2012 13:03:20 +0000
Subject: [issue9631] Python 2.7 installation issue for Linux gcc-4.1.0-3
	(Fedora Core 5?)
In-Reply-To: <1282108953.92.0.943632360126.issue9631@psf.upfronthosting.co.za>
Message-ID: <1326978200.15.0.362186247534.issue9631@psf.upfronthosting.co.za>


mike <mikaelpetterson1 at gmail.com> added the comment:

Hi,

I downloaded source and did the following instructions.


We use Red Hat Enterprise Linux Server release 5.5.

./configure --prefix=/home/mike/python_rh_32
make
make install

I also changed the line in site.py

from:
    
s = os.path.join(os.path.dirname(sys.path.pop()), s)

to:

s = os.path.join(os.path.dirname(sys.path[-1]), s)

but I still get:

/usr/bin/install -c -m 644 ./LICENSE /home/mike/python_rh_32/lib/python2.7/LICENSE.txt
PYTHONPATH=/home/mike/python_rh_32/lib/python2.7   \
                ./python -Wi -tt /home/mike/python_rh_32/lib/python2.7/compileall.py \
                -d /home/mike/python_rh_32/lib/python2.7 -f \
                -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
                /home/mike/python_rh_32/lib/python2.7
Traceback (most recent call last):
  File "/home/mike/python_rh_32/lib/python2.7/compileall.py", line 16, in <module>
    import struct
  File "/home/mike/python_rh_32/lib/python2.7/struct.py", line 1, in <module>
    from _struct import *
ImportError: No module named _struct
make: *** [libinstall] Error 1


Did anyone slove this problem?

----------
nosy: +eraonel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9631>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:08:36 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Thu, 19 Jan 2012 13:08:36 +0000
Subject: [issue13763] Potentially hard to understand wording in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1326978516.95.0.485032361629.issue13763@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for commenting.  Please commit your wording, or this alternate version:

?``hg`` is the name of the Mercurial command-line program, and is often used as an abbreviation for "Mercurial".?

(Just to avoid repetition and to add ?command-line?.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:13:43 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 19 Jan 2012 13:13:43 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F176E88.3090503@udel.edu>
Message-ID: <CAMpsgwbsVX0g9o40EvjXbXt2FBugdd35AkH7zFOPVf99VNqB8A@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I tried the collision counting with a low number of collisions:

less than 15 collisions
-----------------------

Fail at startup.

5 collisions (32 buckets, 21 used=65.6%): hash=ceb3152f => f
10 collisions (32 buckets, 21 used=65.6%): hash=ceb3152f => f

dict((str(k), 0) for k in range(2000000))
-----------------------------------------

15 collisions (32,768 buckets, 18024 used=55.0%): hash=0e4631d2 => 31d2
20 collisions (131,072 buckets, 81568 used=62.2%): hash=12660719 => 719
25 collisions (1,048,576 buckets, 643992 used=61.4%): hash=6a1f6d21 => f6d21
30 collisions (1,048,576 buckets, 643992 used=61.4%): hash=6a1f6d21 => f6d21
35 collisions => ? (more than 10,000,000 integers)

random_dict('', 50000, charset, 1, 3)
--------------------------------------

charset = 'abcdefghijklmnopqrstuvwxyz0123456789'

15 collisions (8192 buckets, 5083 used=62.0%): hash=1526677a => 77a
20 collisions (32768 buckets, 19098 used=58.3%): hash=5d7760e6 => 60e6
25 collisions => <unable to generate a new key>

random_dict('', 50000, charset, 1, 3)
--------------------------------------

charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'

15 collisions (32768 buckets, 20572 used=62.8%): hash=789fe1e6 => 61e6
20 collisions (2048 buckets, 1297 used=63.3%): hash=2052533d => 33d
25 collisions => nope

random_dict('', 50000, charset, 1, 10)
--------------------------------------

charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'

15 collisions (32768 buckets, 18964 used=57.9%): hash=94d7c4f5 => 44f5
20 collisions (32768 buckets, 21548 used=65.8%): hash=acb5b39e => 339e
25 collisions (8192 buckets, 5395 used=65.9%): hash=04d367ae => 7ae
30 collisions => nope

random_dict() comes from the following script:
***
import random

def random_string(charset, minlen, maxlen):
    strlen = random.randint(minlen, maxlen)
    return ''.join(random.choice(charset) for index in xrange(strlen))

def random_dict(prefix, count, charset, minlen, maxlen):
    dico = {}
    keys = set()
    for index in xrange(count):
        for tries in xrange(10000):
            key = prefix + random_string(charset, minlen, maxlen)
            if key in keys:
                continue
            keys.add(key)
            break
        else:
            raise ValueError("unable to generate a new key")
        dico[key] = None

charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'
charset = 'abcdefghijklmnopqrstuvwxyz0123456789'
random_dict('', 50000, charset, 1, 3)
***

I ran the Django test suite. With a limit of 20 collisions, 60 tests
fail. With a limit of 50 collisions, there is no failure. But I don't
think that the test suite uses large data sets.

I also triend the Django test suite with a randomized hash function.
There are 46 failures. Many (all?) are related to the order of dict
keys: repr(dict) or indirectly in a HTML output. I didn't analyze all
failures. I suppose that Django can simply run the test suite using
PYTHONHASHSEED=0 (disable the randomized hash function), at least in a
first time.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 14:36:59 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 19 Jan 2012 13:36:59 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1326980219.67.0.369728887578.issue13817@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's the backtrace:

"""
#0  0x0000003bfb20c9b1 in sem_wait () from /lib64/libpthread.so.0
#1  0x000000000051a7c3 in PyThread_acquire_lock (lock=0x17db0750, waitflag=1)
    at Python/thread_pthread.h:321
#2  0x000000000051a9b4 in find_key (key=1, value=0x0) at Python/thread.c:268
#3  0x000000000051abdc in PyThread_get_key_value (key=1) at Python/thread.c:360
#4  0x00000000005025b1 in PyGILState_GetThisThreadState () at Python/pystate.c:598
#5  0x00000000005024f5 in _PyGILState_Reinit () at Python/pystate.c:547
#6  0x0000000000521fc7 in PyOS_AfterFork () at ./Modules/signalmodule.c:979
#7  0x000000000052461d in posix_fork (self=0x0, noargs=0x0) at ./Modules/posixmodule.c:3695
"""

It's stuck in _PyGILState_Reinit(), when calling PyGILState_GetThisThreadState().
That's because in 2.7, TLS is emulated (see Python/thread.c), and it uses a global mutex.
If this mutex is locked at the time of fork(), then the next call to TLS primitives (even PyGILState_GetThisThreadState()) will deadlock.

Now, this particular bug is fixed in 2.7 since #13156, which backed-out  _PyGILState_Reinit() because it was only relevant for native TLS implementations.

The code is still present in 3.2 and and default, but this problem doesn't affect native TLS implementations.

Just to be extra safe, we PyThread_ReInitTLS() - which resets this global mutex on emulated implementations, and is just a no-op on pthread and windows - should be moved earlier in PyOS_AfterFork(), to avoid this type of deadlock (I mean, PyGILState_GetThisThreadState() deadlock after fork() is bad).
Patch attached.

----------
keywords: +needs review, patch
stage:  -> patch review
versions: +Python 3.2, Python 3.3 -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:14:17 2012
From: report at bugs.python.org (Chris Gill)
Date: Thu, 19 Jan 2012 14:14:17 +0000
Subject: [issue13825] Datetime failing while reading active directory time
	attribute
Message-ID: <1326982457.08.0.645011585605.issue13825@psf.upfronthosting.co.za>


New submission from Chris Gill <chrisfgill at gmail.com>:

I believe I am having a similar issue to this: http://bugs.python.org/issue7150



I am in the middle of programming a quick script and now I cannot seem to get beyond this issue; as it is printing up the expiration times from the AD user listings (many of which print 1601 year) it finally fails after the same user account, I have compared accounts and the expiration is the same as other accounts in AD: which is set to (never)

any ideas of what's going on here and how I can bypass this error?


error report: 

Traceback (most recent call last):
... line 14, in <module>
    print user.name + ": " + str(user.accountExpires)
  File "C:\Python27\lib\site-packages\active_directory.py", line 425, in __getattr__
    self._delegate_map[name] = converter (attr)
  File "C:\Python27\lib\site-packages\active_directory.py", line 335, in convert_to_datetime
    return ad_time_to_datetime (item)
  File "C:\Python27\lib\site-packages\active_directory.py", line 319, in ad_time_to_datetime
    return BASE_TIME + delta
OverflowError: date value out of range



code:

import active_directory
from datetime import datetime,timedelta

##check AD for account expirations
users = active_directory.AD_object ("LDAP://ou=administration,dc=domain,dc=com")
for user in users.search (objectCategory='Person'):

    dn = user.distinguishedName
    dn = dn.encode("utf-8") #for the occasional apostrophe

    if "Adjuncts" in str(dn):
        print user.name + ": " + str(user.accountExpires)

print "done"


example output:

CN=John Hancock: 1601-01-01 00:00:00
CN=Jane Smith: 1601-01-01 00:00:00
...

----------
components: IDLE, Windows
messages: 151622
nosy: Chris.Gill
priority: normal
severity: normal
status: open
title: Datetime failing while reading active directory time attribute
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13825>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:16:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 19 Jan 2012 14:16:32 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1326982592.36.0.69058670699.issue13817@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think you forgot to attach the patch :)

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:25:15 2012
From: report at bugs.python.org (Julian Berman)
Date: Thu, 19 Jan 2012 14:25:15 +0000
Subject: [issue13826] Having a shlex example in the subprocess.Popen docs is
	confusing
Message-ID: <1326983115.48.0.551767400168.issue13826@psf.upfronthosting.co.za>


New submission from Julian Berman <Julian+Python.org at GrayVines.com>:

The example at http://docs.python.org/dev/library/subprocess.html#popen-constructor seems a bit misplaced, as it seems to suggest that one should use the shlex module. Most of the other examples in the module seem to use a list to provide the args, so if there was a need to just point out that shlex could be used for a corner case perhaps it'd be better suited as a footnote or another subsection somewhere.

----------
assignee: docs at python
components: Documentation
messages: 151624
nosy: Julian, docs at python
priority: normal
severity: normal
status: open
title: Having a shlex example in the subprocess.Popen docs is confusing
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13826>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:27:36 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 19 Jan 2012 14:27:36 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwav+2G-BZ4fw_7HU9zev-L2zk4Jwzsb_UBpjn-TNPDtCA@mail.gmail.com>
Message-ID: <4F182851.5090506@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> ...
> So I expect something similar in applications: no change in the
> applications, but a lot of hacks/tricks in tests.

Tests usually check output of an application given a certain
input. If those fail with the randomization, then it's likely
real-world application uses will show the same kinds of failures
due to the application changing from deterministic to
non-deterministic via the randomization.

>> BTW: The patch still includes the unnecessary _Py_unicode_hash_secret.suffix
>> which needlessly complicates the code and doesn't any additional
>> protection against hash value collisions
> 
> How does it complicate the code? It adds an extra XOR to hash(str) and
> 4 or 8 bytes in memory, that's all. It is more difficult to compute
> the secret from hash(str) output if there is a prefix *and* a suffix.
> If there is only a prefix, knowning a single hash(str) value is just
> enough to retrieve directly the secret.

The suffix only introduces a constant change in all hash values
output, so even if you don't know the suffix, you can still
generate data sets with collisions by just having the prefix.

>> I don't think it affects more than 0.01% of applications/users :)
> 
> It would help to try a patched Python on a real world application like
> Django to realize how much code is broken (or not) by a randomized
> hash function.

That would help for both approaches, indeed.

Please note, that you'd have to extend the randomization to
all other Python data types as well in order to reach the same level
of security as the collision counting approach.

As-is the randomization patch does not solve the integer key attack and
even though parsers such as JSON and XML-RPC aren't directly affected,
it is well possible that stringified integers such as IDs are converted
back to integers later during processing, thereby triggering the
attack.

Note that the integer attack also applies to other number types
in Python:

(3, 3, 3)

See Tim's post I referenced earlier on for the reasons. Here's
a quick summary ;-) ...

{3: 3}

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:31:44 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 19 Jan 2012 14:31:44 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F182851.5090506@egenix.com>
Message-ID: <1326983393.3395.14.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Please note, that you'd have to extend the randomization to
> all other Python data types as well in order to reach the same level
> of security as the collision counting approach.

You also have to extend the collision counting to sets, by the way.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:36:48 2012
From: report at bugs.python.org (R. David Murray)
Date: Thu, 19 Jan 2012 14:36:48 +0000
Subject: [issue13826] Having a shlex example in the subprocess.Popen docs is
	confusing
In-Reply-To: <1326983115.48.0.551767400168.issue13826@psf.upfronthosting.co.za>
Message-ID: <1326983808.87.0.793563902467.issue13826@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

It is not particularly intuitive what goes in to a Popen non-shell argument list, unless you are an experienced programmer.  The real purpose of the note is to convey a lot of information about how tokenization works in a short example, and it also demonstrates how to investigate other complex cases the user may have to deal with.  Because of the first part of that (showing tokenization quirks) I don't think it should be relegated to a footnote.

That said, the example could perhaps be reworded slightly to make its expositional purpose clearer.  Suggestions welcome.

----------
nosy: +r.david.murray

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13826>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:37:14 2012
From: report at bugs.python.org (R. David Murray)
Date: Thu, 19 Jan 2012 14:37:14 +0000
Subject: [issue13826] Having a shlex example in the subprocess.Popen docs is
	confusing
In-Reply-To: <1326983115.48.0.551767400168.issue13826@psf.upfronthosting.co.za>
Message-ID: <1326983834.66.0.654229482048.issue13826@psf.upfronthosting.co.za>


Changes by R. David Murray <rdmurray at bitdance.com>:


----------
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13826>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:37:53 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 19 Jan 2012 14:37:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326983393.3395.14.camel@localhost.localdomain>
Message-ID: <4F182ABA.4060703@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou at free.fr> added the comment:
> 
>> Please note, that you'd have to extend the randomization to
>> all other Python data types as well in order to reach the same level
>> of security as the collision counting approach.
> 
> You also have to extend the collision counting to sets, by the way.

Indeed, but that's easy, since the set implementation derives from
the dict implementation.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:43:53 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Thu, 19 Jan 2012 14:43:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326984233.37.0.357847234691.issue13703@psf.upfronthosting.co.za>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

Django's tests will *not* be run with HASHEED=0, if they're broken with hash randomization then they are likely broken on random.choice(["32-bit", "64-bit", "pypy", "jython", "ironpython"]) and we strive to run on all those platforms. If our tests are order dependent then they're broken, and we'll fix the tests.

Further, most of the failures I can think of would be failures in the tests that wouldn't actually be failures in a real application, such as the rendered HTML being different because a tag's attributes are in a different order.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:45:40 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Thu, 19 Jan 2012 14:45:40 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1326984340.54.0.361309903337.issue13817@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here's the patch.
It's probably possible to add a test for this, however I don't have access to my development machine, so I can't write it now.

----------
Added file: http://bugs.python.org/file24275/reinit_tls.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Thu Jan 19 15:45:47 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 14:45:47 +0000
Subject: [issue13826] Having a shlex example in the subprocess.Popen docs is
	confusing
In-Reply-To: <1326983115.48.0.551767400168.issue13826@psf.upfronthosting.co.za>
Message-ID: <1326984347.34.0.658167761064.issue13826@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Maybe we can add a very small example before the whole note to show just how to use Popen in simple situation, and so the shlex part below will add more details for more advanced cases.

----------
nosy: +sandro.tosi
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13826>
_______________________________________

From report at bugs.python.org  Thu Jan 19 16:10:16 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Thu, 19 Jan 2012 15:10:16 +0000
Subject: [issue13752] add a str.casefold() method
In-Reply-To: <1326159624.07.0.383633968301.issue13752@psf.upfronthosting.co.za>
Message-ID: <1326985816.78.0.356799085817.issue13752@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13752>
_______________________________________

From report at bugs.python.org  Thu Jan 19 16:11:55 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 19 Jan 2012 15:11:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwbsVX0g9o40EvjXbXt2FBugdd35AkH7zFOPVf99VNqB8A@mail.gmail.com>
Message-ID: <4F1832B4.4040603@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

STINNER Victor wrote:
> 
> I tried the collision counting with a low number of collisions:
> ... no false positives with a limit of 50 collisions ...

Thanks for running those tests. Looks like a limit lower than 1000
would already do just fine.

Some timings showing how long it would take to hit a limit:

# 100
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 100))"
100 loops, best of 3: 297 usec per loop

# 250
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 250))"
100 loops, best of 3: 1.46 msec per loop

# 500
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 500))"
100 loops, best of 3: 5.73 msec per loop

# 750
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 750))"
100 loops, best of 3: 12.7 msec per loop

# 1000
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 1000))"
100 loops, best of 3: 22.4 msec per loop

These timings have to matched against the size of the payload
needed to trigger those limits.

In any case, the limit needs to be configurable like the hash seed
in the randomization patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 16:13:20 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Thu, 19 Jan 2012 15:13:20 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F182851.5090506@egenix.com>
Message-ID: <4F183309.3020200@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

[Reposting, since roundup removed part of the Python output]

M.-A. Lemburg wrote:
> Note that the integer attack also applies to other number types
> in Python:
> 
> --> (hash(3), hash(3.0), hash(3+0j)
> (3, 3, 3)
> 
> See Tim's post I referenced earlier on for the reasons. Here's
> a quick summary ;-) ...
> 
> --> {3:1, 3.0:2, 3+0j:3}
> {3: 3}

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 16:32:57 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 15:32:57 +0000
Subject: [issue13478] No documentation for timeit.default_timer
In-Reply-To: <1322208815.17.0.598781679655.issue13478@psf.upfronthosting.co.za>
Message-ID: <1326987177.69.0.981004282108.issue13478@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

I propose these 2 patches: I'm not exactly proud of 

- 'in a platform specific manner' in the first hunk
- the second hunk

I know that there should be a better way to express it, but since I can't get to it I'd just ask for suggestions :)

----------
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24276/issue13478-py27.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13478>
_______________________________________

From report at bugs.python.org  Thu Jan 19 16:33:03 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 15:33:03 +0000
Subject: [issue13478] No documentation for timeit.default_timer
In-Reply-To: <1322208815.17.0.598781679655.issue13478@psf.upfronthosting.co.za>
Message-ID: <1326987183.97.0.268194083684.issue13478@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


Added file: http://bugs.python.org/file24277/issue13478-py32.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13478>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:02:19 2012
From: report at bugs.python.org (Michael Hipp)
Date: Thu, 19 Jan 2012 16:02:19 +0000
Subject: [issue13827] Unexecuted import changes namespace
Message-ID: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>


New submission from Michael Hipp <michael at redmule.com>:

A local *unexecuted* import appears to be changing the namespace. Attached files are ready to run.

# over.py
SOMETHING = "overridden"

# main.py
OVERRIDE = False
SOMETHING = "original"

def main():
    #global SOMETHING  # uncomment and it works
    if OVERRIDE:
        from over import SOMETHING  # comment out and it works
        pass
    print SOMETHING  # UnboundLocalError: local variable 'SOMETHING' referenced before assignment

The SOMETHING variable has a value from the module global namespace, but it gets lost due to an import that is never executed.

I would think an unexecuted statement shouldn't have any effect on anything.

The second file will have to be submitted in a follow-on, it appears

----------
components: None
files: main.py
messages: 151635
nosy: hippmr
priority: normal
severity: normal
status: open
title: Unexecuted import changes namespace
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file24278/main.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:03:11 2012
From: report at bugs.python.org (Michael Hipp)
Date: Thu, 19 Jan 2012 16:03:11 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326988991.64.0.416049311848.issue13827@psf.upfronthosting.co.za>


Michael Hipp <michael at redmule.com> added the comment:

Add'l over.py file

----------
Added file: http://bugs.python.org/file24279/over.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:03:51 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 16:03:51 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326989031.02.0.332404813928.issue13827@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Not a bug. Basically, import is an explicit assignment statement.

----------
nosy: +benjamin.peterson
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:05:31 2012
From: report at bugs.python.org (Michael Hipp)
Date: Thu, 19 Jan 2012 16:05:31 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326989131.3.0.631370746138.issue13827@psf.upfronthosting.co.za>


Michael Hipp <michael at redmule.com> added the comment:

Even an *unexecuted* import assignment statement?

----------
resolution: invalid -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:07:00 2012
From: report at bugs.python.org (Michael Foord)
Date: Thu, 19 Jan 2012 16:07:00 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326989220.86.0.607950876749.issue13827@psf.upfronthosting.co.za>


Michael Foord <michael at voidspace.org.uk> added the comment:

hippmr: the problem is that by importing SOMETHING inside that function you're creating a *local variable* called SOMETHING. If the override isn't executed, and SOMETHING isn't global, then that local variable doesn't exist - which is why you get that error.

So even if the import isn't executed, its existence in the function tells Python that name is local to the function.

----------
nosy: +michael.foord
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:08:35 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 19 Jan 2012 16:08:35 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326989315.33.0.318781960663.issue13827@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

>>> OVERRIDE = False
>>> SOMETHING = "original"
>>> 
>>> def main():
...     if OVERRIDE:
...         SOMETHING = None
...     print SOMETHING
... 
>>> main()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in main
UnboundLocalError: local variable 'SOMETHING' referenced before assignment

http://docs.python.org/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value

----------
nosy: +ezio.melotti
resolution: invalid -> 
stage: committed/rejected -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:09:00 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 19 Jan 2012 16:09:00 +0000
Subject: [issue13827] Unexecuted import changes namespace
In-Reply-To: <1326988939.6.0.324372233485.issue13827@psf.upfronthosting.co.za>
Message-ID: <1326989340.31.0.334709192907.issue13827@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
components:  -None
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13827>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:18:35 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 16:18:35 +0000
Subject: [issue11551] test_dummy_thread.py test coverage improvement
In-Reply-To: <1300167076.06.0.564640850905.issue11551@psf.upfronthosting.co.za>
Message-ID: <1326989915.6.0.261330919677.issue11551@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Hi Denver, do you have the time to address the review on rietveld and propose a new patch?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11551>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:33:52 2012
From: report at bugs.python.org (scape)
Date: Thu, 19 Jan 2012 16:33:52 +0000
Subject: [issue13825] Datetime failing while reading active directory time
	attribute
In-Reply-To: <1326982457.08.0.645011585605.issue13825@psf.upfronthosting.co.za>
Message-ID: <1326990832.64.0.582513558089.issue13825@psf.upfronthosting.co.za>


scape <chrisfgill at gmail.com> added the comment:

I dug a little deeper using an error trap and found some of the problematic accounts in AD have their attribute set to a wildly long number and not 0 (as are others when 'never' is specified.) i'll dig further, it also does not seem to be an issue with python but more of an issue with the module I am using (active_directory) and its datetime handling (likely not fixed as was Python)

I don't think the issue is necessarily solved, but I'll close it anyways as I think I have atleast my solution now

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13825>
_______________________________________

From report at bugs.python.org  Thu Jan 19 17:52:02 2012
From: report at bugs.python.org (Eric Snow)
Date: Thu, 19 Jan 2012 16:52:02 +0000
Subject: [issue11948] Tutorial/Modules - small fix to better clarify the
	modules search path
In-Reply-To: <1304011111.24.0.834772686878.issue11948@psf.upfronthosting.co.za>
Message-ID: <1326991922.23.0.177000049641.issue11948@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

FYI: unless importlib took undue liberties (unlikely), frozen modules also precede path-based modules.  See the implicit additions to sys.meta_path in Lib/importlib/_bootstrap.py.

Whether or not to include a mention of frozen modules in the tutorial...I'll leave that to you.  :)

----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11948>
_______________________________________

From report at bugs.python.org  Thu Jan 19 18:06:03 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 17:06:03 +0000
Subject: [issue13828] Further improve casefold documentation
Message-ID: <1326992763.46.0.576190503242.issue13828@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

> http://hg.python.org/cpython/rev/0b5ce36a7a24
> changeset: ? 74515:0b5ce36a7a24


> + ? Casefolding is similar to lowercasing but more aggressive because it is
> + ? intended to remove all case distinctions in a string. For example, the German
> + ? lowercase letter ``'?'`` is equivalent to ``"ss"``. Since it is already
> + ? lowercase, :meth:`lower` would do nothing to ``'?'``; :meth:`casefold`
> + ? converts it to ``"ss"``.

Perhaps add the recommendation to canonicalize as well.

A complete, but possibly too long, try is below:


Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string. For example, the German lowercase letter ``'?'`` is equivalent to ``"ss"``. Since it is already lowercase, :meth:`lower` would do nothing to ``'?'``; :meth:`casefold` converts it to ``"ss"``.  Note that most case-insensitive matches should also match compatibility equivalent characters.  

The casefolding algorithm is described in section 3.13 of the Unicode Standard.  Per D146, a compatibility caseless match can be achieved by

    from unicodedata import normalize
    def caseless_compat(string):
        nfd_string = normalize("NFD", string)
        nfkd1_string = normalize("NFKD", nfd_string.casefold())
        return normalize("NFKD", nfkd1_string.casefold())

----------
assignee: docs at python
components: Documentation
messages: 151644
nosy: Jim.Jewett, benjamin.peterson, docs at python
priority: normal
severity: normal
status: open
title: Further improve casefold documentation
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13828>
_______________________________________

From report at bugs.python.org  Thu Jan 19 18:09:52 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 17:09:52 +0000
Subject: [issue13828] Further improve casefold documentation
In-Reply-To: <1326992763.46.0.576190503242.issue13828@psf.upfronthosting.co.za>
Message-ID: <1326992992.82.0.889737652746.issue13828@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Frankly, I do think that sample code is too long, but correctness matters ... perhaps a better solution would be to add either a method or a unicodedata function that does the work, then the extra note could just say

Note that most case-insensitive matches should also match compatibility equivalent characters; see unicodedata.compatibity_casefold

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13828>
_______________________________________

From report at bugs.python.org  Thu Jan 19 18:54:21 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 19 Jan 2012 17:54:21 +0000
Subject: [issue13641] decoding functions in the base64 module could accept
	unicode strings
In-Reply-To: <1324386392.48.0.968435833624.issue13641@psf.upfronthosting.co.za>
Message-ID: <1326995661.94.0.707926287052.issue13641@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thanks for the updated patch!
Two comments:
- I see no tests for map01 and altchars being passed as an str, is this supported by the patch or am I reading it wrong?
- apparently b16decode is not tackled, is it deliberate?

Thanks again.

----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13641>
_______________________________________

From report at bugs.python.org  Thu Jan 19 19:05:53 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Thu, 19 Jan 2012 18:05:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1326996353.21.0.0703813111949.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

> The suffix only introduces a constant change in all hash values
> output, so even if you don't know the suffix, you can still
> generate data sets with collisions by just having the prefix.

That's true. But without the suffix, I can pretty easy and efficient guess the prefix by just seeing the result of a few well-chosen and short repr(dict(X)). I suppose that's harder with the suffix.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 19 19:42:03 2012
From: report at bugs.python.org (Dan kamp)
Date: Thu, 19 Jan 2012 18:42:03 +0000
Subject: [issue13829] exception error
Message-ID: <1326998522.6.0.790472668429.issue13829@psf.upfronthosting.co.za>


New submission from Dan kamp <roontoon at gmail.com>:

Get this error when trying to run Moviegrabber on a mac running v2.7. Crash report below.

Process:         Python [2444]
Path:            /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/C ontents/MacOS/Python
Identifier:      Python
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  Python [2431]
 
 
Date/Time:       2011-12-21 20:32:41.233 -0500
OS Version:      Mac OS X 10.7.2 (11C74)
Report Version:  9
 
 
Interval Since Last Report:          357638 sec
Crashes Since Last Report:           17
Per-App Crashes Since Last Report:   405
Anonymous UUID:                      3C387DB7-0AA3-4F34-A100-31D1736D2668
 
 
Crashed Thread:  0
 
 
Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000108
 
 
VM Regions Near 0x108:
-->
    __TEXT                 000000010c5c5000-000000010c5c6000 [    4K] r-x/rwx SM=COW  /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/C ontents/MacOS/Python
 
 
Application Specific Information:
objc[2429]: garbage collection is OFF
 
 
Thread 0 Crashed:
0   libdispatch.dylib                       0x00007fff8b14ace9 _dispatch_wakeup + 108
1   libdispatch.dylib                       0x00007fff8b14d876 _dispatch_resume_slow + 20
2   com.apple.CoreFoundation                0x00007fff8af8262c -[NSXPCConnection start] + 92
3   com.apple.CoreFoundation                0x00007fff8af84ca3 __CFXNotificationCenterSetupConnection + 387
4   com.apple.CoreFoundation                0x00007fff8af84b11 __CFXNotificationCenterCreate + 273
5   com.apple.CoreFoundation                0x00007fff8af849ea __CFNotificationCenterGetDistributedCenter_block_invoke_1 + 26
6   libdispatch.dylib                       0x00007fff8b14d224 dispatch_once_f + 53
7   com.apple.CoreFoundation                0x00007fff8af73e0a CFNotificationCenterGetDistributedCenter + 74
8   com.apple.CoreFoundation                0x00007fff8afb2eb8 ____CFXPreferencesGetSourceForTriplet_block_invoke_1 + 40
9   libdispatch.dylib                       0x00007fff8b14d224 dispatch_once_f + 53
10  com.apple.CoreFoundation                0x00007fff8af7fa4a __CFXPreferencesGetSourceForTriplet + 58
11  com.apple.CoreFoundation                0x00007fff8af8be67 __CFXPreferencesGetSearchListForBundleID + 215
12  com.apple.CoreFoundation                0x00007fff8af8bd48 ___CFXPreferencesCopyAppValue_block_invoke_1 + 24
13  com.apple.CoreFoundation                0x00007fff8af8bcea CFPreferencesCopyAppValue + 218
14  com.apple.SystemConfiguration           0x00007fff928f388c SCDynamicStoreCopyProxies + 43
15  _scproxy.so                             0x000000010ceae9ba 0x10ceae000 + 2490
16  org.python.python                       0x000000010c651b58 PyEval_EvalFrameEx + 13318
17  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
18  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
19  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
20  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
21  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
22  org.python.python                       0x000000010c5f2abf 0x10c5ca000 + 166591
23  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
24  org.python.python                       0x000000010c5e06e9 0x10c5ca000 + 91881
25  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
26  org.python.python                       0x000000010c64dc40 PyEval_CallObjectWithKeywords + 180
27  org.python.python                       0x000000010c5dd489 PyInstance_New + 273
28  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
29  org.python.python                       0x000000010c651f63 PyEval_EvalFrameEx + 14353
30  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
31  org.python.python                       0x000000010c654e6c 0x10c5ca000 + 568940
32  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
33  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
34  org.python.python                       0x000000010c654e6c 0x10c5ca000 + 568940
35  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
36  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
37  org.python.python                       0x000000010c5f2abf 0x10c5ca000 + 166591
38  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
39  org.python.python                       0x000000010c6525ec PyEval_EvalFrameEx + 16026
40  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
41  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
42  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
43  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
44  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
45  org.python.python                       0x000000010c5f2abf 0x10c5ca000 + 166591
46  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
47  org.python.python                       0x000000010c5e06e9 0x10c5ca000 + 91881
48  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
49  org.python.python                       0x000000010c61c484 0x10c5ca000 + 337028
50  org.python.python                       0x000000010c616b7a 0x10c5ca000 + 314234
51  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
52  org.python.python                       0x000000010c651f63 PyEval_EvalFrameEx + 14353
53  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
54  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
55  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
56  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
57  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
58  org.python.python                       0x000000010c5f2abf 0x10c5ca000 + 166591
59  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
60  org.python.python                       0x000000010c6525ec PyEval_EvalFrameEx + 16026
61  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
62  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
63  org.python.python                       0x000000010c654df7 0x10c5ca000 + 568823
64  org.python.python                       0x000000010c651e0a PyEval_EvalFrameEx + 14008
65  org.python.python                       0x000000010c654cd8 PyEval_EvalCodeEx + 1996
66  org.python.python                       0x000000010c5f2abf 0x10c5ca000 + 166591
67  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
68  org.python.python                       0x000000010c5e06e9 0x10c5ca000 + 91881
69  org.python.python                       0x000000010c5d1d32 PyObject_Call + 97
70  org.python.python                       0x000000010c64dc40 PyEval_CallObjectWithKeywords + 180
71  org.python.python                       0x000000010c67f40d 0x10c5ca000 + 742413
72  libsystem_c.dylib                       0x00007fff978098bf _pthread_start + 335
73  libsystem_c.dylib                       0x00007fff9780cb75 thread_start + 13
 
 
Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000100  rbx: 0x00007ffc6a6e8610  rcx: 0x00007ffc6a6e8628  rdx: 0x0000000000000001
  rdi: 0x00007ffc6a6e8620  rsi: 0x00007fff8c3d6c40  rbp: 0x000000010d471080  rsp: 0x000000010d471070
   r8: 0x00007ffc6a6dc398   r9: 0x0000000000000596  r10: 0x0000000000000081  r11: 0x00007fff8af825d0
  r12: 0x0000000000000000  r13: 0x0000000000000000  r14: 0x00007fff7ad93180  r15: 0x00007ffc6a6bde28
  rip: 0x00007fff8b14ace9  rfl: 0x0000000000010206  cr2: 0x0000000000000108
Logical CPU: 0
 
 
Binary Images:
       0x10c5c5000 -        0x10c5c5fff  org.python.python (2.7.1 - 2.7.1) <E2C46621-158F-349B-9131-263E8958B23A> /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/C ontents/MacOS/Python
       0x10c5ca000 -        0x10c6e5ff7  org.python.python (2.7.1 - 2.7.1) <A7E34EB5-24D2-3B8F-A26B-4A5B28CD2C60> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
       0x10c7eb000 -        0x10c7effff  _struct.so (??? - ???) <A871A47E-0BF6-3A21-BF66-A5438C54B237> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_struct.so
       0x10c7f5000 -        0x10c7f7ff7  binascii.so (??? - ???) <9B353DAC-B0E8-3B4B-91A1-50F6F86AA928> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/binascii.so
       0x10c7fb000 -        0x10c7fcfff  _hashlib.so (??? - ???) <C6196B8C-5451-3875-BCD9-2942C6A32FCF> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_hashlib.so
       0x10c940000 -        0x10c941fff  _locale.so (??? - ???) <FCAE62F7-D07F-3F5F-8057-18224582F344> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_locale.so
       0x10c945000 -        0x10c951fff  cPickle.so (??? - ???) <FE1D948F-998C-350E-8002-224D3DCDE8B5> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/cPickle.so
       0x10c957000 -        0x10c957fff  grp.so (??? - ???) <563EC9A6-F38F-3518-9BBB-912E9F01FDA1> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/grp.so
       0x10c95b000 -        0x10c95efff  _multiprocessing.so (??? - ???) <CFAE1A97-134D-3A6D-9646-D5109117D153> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_multiprocessing.so
       0x10c963000 -        0x10c965fff  select.so (??? - ???) <FEC6A324-8D3E-3439-9315-BC2AF15A0514> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/select.so
       0x10ce30000 -        0x10ce35fff  array.so (??? - ???) <177F1D09-ACEE-3E39-8F1E-3EE8BA8A7AC6> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/array.so
       0x10ce3b000 -        0x10ce41ff7  _socket.so (??? - ???) <CB2B2DF1-46C2-3491-A737-4522C807F5A2> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_socket.so
       0x10ce49000 -        0x10ce4afff  _functools.so (??? - ???) <D4679747-9993-34F6-B702-3946FA5BF171> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_functools.so
       0x10ce4e000 -        0x10ce52fff  _ssl.so (??? - ???) <50FC05D5-0434-3054-9C09-CA8923FED0C0> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_ssl.so
       0x10ce58000 -        0x10ce59fff  cStringIO.so (??? - ???) <AE5A2A12-8E57-3FE2-B998-C175083B27F7> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/cStringIO.so
       0x10ce5e000 -        0x10ce61fff  _collections.so (??? - ???) <62C1B5B7-654D-397A-8840-7EBB907DBCA1> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_collections.so
       0x10ce67000 -        0x10ce6afff  operator.so (??? - ???) <59A152D0-52ED-354C-9C2D-D7390E3EC216> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/operator.so
       0x10ce70000 -        0x10ce76fff  itertools.so (??? - ???) <7C8350B9-8DD3-377A-A5C1-1103A6C955A6> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/itertools.so
       0x10ce7f000 -        0x10ce7ffff  _bisect.so (??? - ???) <CD57DCA1-74B4-37F8-B338-ABDDB1750F22> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_bisect.so
       0x10ce83000 -        0x10ce84fff  _heapq.so (??? - ???) <A3448C7E-3268-32A5-B4C5-859372FFA7E6> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_heapq.so
       0x10ce89000 -        0x10ce8efff  math.so (??? - ???) <BC93F501-541B-3BA8-B1D7-2644AAD6D6CE> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/math.so
       0x10ce94000 -        0x10ce95ff7  _random.so (??? - ???) <3C7A7C17-1698-32D0-BF09-F50A19F75E4B> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_random.so
       0x10ce99000 -        0x10ce9afff  fcntl.so (??? - ???) <D2F4AAAE-3C27-3134-84AE-021647AEB1C4> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/fcntl.so
       0x10ce9e000 -        0x10ce9ffff  time.so (??? - ???) <DA274462-B33A-33EB-90F7-BB9FB8120A5D> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/time.so
       0x10cea6000 -        0x10cea9fff  strop.so (??? - ???) <A317730F-EDDF-3998-8A22-0E632B3E67A4> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/strop.so
       0x10ceae000 -        0x10ceaefff  _scproxy.so (??? - ???) <F392F576-344C-3CCB-B93C-E11205241E1D> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_scproxy.so
       0x10cf72000 -        0x10cf7dff7  datetime.so (??? - ???) <9EC1E3BD-3BD3-3B39-AE19-448CCEEA747A> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/datetime.so
       0x10cf86000 -        0x10cf8ffff  _sqlite3.so (??? - ???) <4F97C26B-1A2D-33D9-A7B4-C791FA3F645B> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_sqlite3.so
       0x10cf99000 -        0x10cf9dfff  _json.so (??? - ???) <A8E3EEA8-354B-35C6-A3C5-653928D7419D> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_json.so
       0x10cfa2000 -        0x10cfaafff  parser.so (??? - ???) <2E88032A-DFD4-38F0-BE83-03DA706702B1> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/parser.so
       0x10d1f0000 -        0x10d201fff  _io.so (??? - ???) <5451CA4B-98A0-3F70-9322-DBE6B7D93CF1> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_io.so
       0x10d210000 -        0x10d221fff  _ctypes.so (??? - ???) <2D2AE6AF-704A-3CBB-954B-33BA49B78254> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/_ctypes.so
       0x10da17000 -        0x10da19fff  zlib.so (??? - ???) <81E54FCE-EEAC-3E97-BB05-1143EB6AECA6> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynl oad/zlib.so
    0x7fff6c1c5000 -     0x7fff6c1f9ac7  dyld (195.5 - ???) <4A6E2B28-C7A2-3528-ADB7-4076B9836041> /usr/lib/dyld
    0x7fff8ace9000 -     0x7fff8ad22fe7  libssl.0.9.8.dylib (44.0.0 - compatibility 0.9.8) <79AAEC98-1258-3DA4-B1C0-4120049D390B> /usr/lib/libssl.0.9.8.dylib
    0x7fff8ae97000 -     0x7fff8aeb7fff  libsystem_kernel.dylib (1699.22.73 - compatibility 1.0.0) <69F2F501-72D8-3B3B-8357-F4418B3E1348> /usr/lib/system/libsystem_kernel.dylib
    0x7fff8af21000 -     0x7fff8af23fff  libquarantine.dylib (36.0.0 - compatibility 1.0.0) <4C3BFBC7-E592-3939-B376-1C2E2D7C5389> /usr/lib/system/libquarantine.dylib
    0x7fff8af37000 -     0x7fff8b10bfff  com.apple.CoreFoundation (6.7.1 - 635.15) <FE4A86C2-3599-3CF8-AD1A-822F1FEA820F> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff8b10c000 -     0x7fff8b147ff7  libsystem_info.dylib (??? - ???) <9C8C2DCB-96DB-3471-9DCE-ADCC26BE2DD4> /usr/lib/system/libsystem_info.dylib
    0x7fff8b148000 -     0x7fff8b156fff  libdispatch.dylib (187.7.0 - compatibility 1.0.0) <712AAEAC-AD90-37F7-B71F-293FF8AE8723> /usr/lib/system/libdispatch.dylib
    0x7fff8c79d000 -     0x7fff8c7a2fff  libcache.dylib (47.0.0 - compatibility 1.0.0) <B7757E2E-5A7D-362E-AB71-785FE79E1527> /usr/lib/system/libcache.dylib
    0x7fff8d64a000 -     0x7fff8d64bff7  libremovefile.dylib (21.0.0 - compatibility 1.0.0) <C6C49FB7-1892-32E4-86B5-25AD165131AA> /usr/lib/system/libremovefile.dylib
    0x7fff8edae000 -     0x7fff8edf0ff7  libcommonCrypto.dylib (55010.0.0 - compatibility 1.0.0) <A5B9778E-11C3-3F61-B740-1F2114E967FB> /usr/lib/system/libcommonCrypto.dylib
    0x7fff8f492000 -     0x7fff8f4afff7  libxpc.dylib (77.17.0 - compatibility 1.0.0) <72A16104-2F23-3C22-B474-1953F06F9376> /usr/lib/system/libxpc.dylib
    0x7fff8f778000 -     0x7fff8f779fff  libsystem_sandbox.dylib (??? - ???) <8D14139B-B671-35F4-9E5A-023B4C523C38> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff90a8c000 -     0x7fff90b70def  libobjc.A.dylib (228.0.0 - compatibility 1.0.0) <C5F2392D-B481-3A9D-91BE-3D039FFF4DEC> /usr/lib/libobjc.A.dylib
    0x7fff911f8000 -     0x7fff91201ff7  libsystem_notify.dylib (80.1.0 - compatibility 1.0.0) <A4D651E3-D1C6-3934-AD49-7A104FD14596> /usr/lib/system/libsystem_notify.dylib
    0x7fff912ac000 -     0x7fff91321ff7  libc++.1.dylib (19.0.0 - compatibility 1.0.0) <C0EFFF1B-0FEB-3F99-BE54-506B35B555A9> /usr/lib/libc++.1.dylib
    0x7fff9132e000 -     0x7fff9132ffff  libunc.dylib (24.0.0 - compatibility 1.0.0) <C67B3B14-866C-314F-87FF-8025BEC2CAAC> /usr/lib/system/libunc.dylib
    0x7fff923c8000 -     0x7fff92416fff  libauto.dylib (??? - ???) <D8AC8458-DDD0-3939-8B96-B6CED81613EF> /usr/lib/libauto.dylib
    0x7fff928da000 -     0x7fff928defff  libdyld.dylib (195.5.0 - compatibility 1.0.0) <F1903B7A-D3FF-3390-909A-B24E09BAD1A5> /usr/lib/system/libdyld.dylib
    0x7fff928df000 -     0x7fff9292aff7  com.apple.SystemConfiguration (1.11.1 - 1.11) <F832FE21-5509-37C6-B1F1-48928F31BE45> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfi guration
    0x7fff92a44000 -     0x7fff92ab7fff  libstdc++.6.dylib (52.0.0 - compatibility 7.0.0) <6BDD43E4-A4B1-379E-9ED5-8C713653DFF2> /usr/lib/libstdc++.6.dylib
    0x7fff92c68000 -     0x7fff92c73ff7  libc++abi.dylib (14.0.0 - compatibility 1.0.0) <8FF3D766-D678-36F6-84AC-423C878E6D14> /usr/lib/libc++abi.dylib
    0x7fff92c74000 -     0x7fff92c78fff  libmathCommon.A.dylib (2026.0.0 - compatibility 1.0.0) <FF83AFF7-42B2-306E-90AF-D539C51A4542> /usr/lib/system/libmathCommon.A.dylib
    0x7fff92c79000 -     0x7fff92c8bff7  libz.1.dylib (1.2.5 - compatibility 1.0.0) <30CBEF15-4978-3DED-8629-7109880A19D4> /usr/lib/libz.1.dylib
    0x7fff92c8e000 -     0x7fff92c8efff  libkeymgr.dylib (23.0.0 - compatibility 1.0.0) <61EFED6A-A407-301E-B454-CD18314F0075> /usr/lib/system/libkeymgr.dylib
    0x7fff92c8f000 -     0x7fff92c90ff7  libsystem_blocks.dylib (53.0.0 - compatibility 1.0.0) <8BCA214A-8992-34B2-A8B9-B74DEACA1869> /usr/lib/system/libsystem_blocks.dylib
    0x7fff92e7c000 -     0x7fff92e82ff7  libunwind.dylib (30.0.0 - compatibility 1.0.0) <1E9C6C8C-CBE8-3F4B-A5B5-E03E3AB53231> /usr/lib/system/libunwind.dylib
    0x7fff93681000 -     0x7fff93686ff7  libsystem_network.dylib (??? - ???) <5DE7024E-1D2D-34A2-80F4-08326331A75B> /usr/lib/system/libsystem_network.dylib
    0x7fff93687000 -     0x7fff9368cfff  libcompiler_rt.dylib (6.0.0 - compatibility 1.0.0) <98ECD5F6-E85C-32A5-98CD-8911230CB66A> /usr/lib/system/libcompiler_rt.dylib
    0x7fff936ce000 -     0x7fff938d0fff  libicucore.A.dylib (46.1.0 - compatibility 1.0.0) <38CD6ED3-C8E4-3CCD-89AC-9C3198803101> /usr/lib/libicucore.A.dylib
    0x7fff95141000 -     0x7fff95149fff  libsystem_dnssd.dylib (??? - ???) <7749128E-D0C5-3832-861C-BC9913F774FA> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff95396000 -     0x7fff95397fff  libffi.dylib (??? - ???) <DB96CC4B-0D38-3102-80AA-91DDE9AF3886> /usr/lib/libffi.dylib
    0x7fff95c12000 -     0x7fff95d1efff  libcrypto.0.9.8.dylib (44.0.0 - compatibility 0.9.8) <3A8E1F89-5E26-3C8B-B538-81F5D61DBF8A> /usr/lib/libcrypto.0.9.8.dylib
    0x7fff95e42000 -     0x7fff95e43fff  libdnsinfo.dylib (395.6.0 - compatibility 1.0.0) <718A135F-6349-354A-85D5-430B128EFD57> /usr/lib/system/libdnsinfo.dylib
    0x7fff9653c000 -     0x7fff96543fff  libcopyfile.dylib (85.1.0 - compatibility 1.0.0) <172B1985-F24A-34E9-8D8B-A2403C9A0399> /usr/lib/system/libcopyfile.dylib
    0x7fff96544000 -     0x7fff96546fff  com.apple.TrustEvaluationAgent (2.0 - 1) <1F31CAFF-C1C6-33D3-94E9-11B721761DDF> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/Tru stEvaluationAgent
    0x7fff96547000 -     0x7fff9654dfff  libmacho.dylib (800.0.0 - compatibility 1.0.0) <D86F63EC-D2BD-32E0-8955-08B5EAFAD2CC> /usr/lib/system/libmacho.dylib
    0x7fff96845000 -     0x7fff96948fff  libsqlite3.dylib (9.6.0 - compatibility 9.0.0) <7F60B0FF-4946-3639-89AB-B540D318B249> /usr/lib/libsqlite3.dylib
    0x7fff96949000 -     0x7fff96976fe7  libSystem.B.dylib (159.1.0 - compatibility 1.0.0) <095FDD3C-3961-3865-A59B-A5B0A4B8B923> /usr/lib/libSystem.B.dylib
    0x7fff977b9000 -     0x7fff977bafff  libDiagnosticMessagesClient.dylib (??? - ???) <3DCF577B-F126-302B-BCE2-4DB9A95B8598> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fff977bb000 -     0x7fff97898fef  libsystem_c.dylib (763.12.0 - compatibility 1.0.0) <FF69F06E-0904-3C08-A5EF-536FAFFFDC22> /usr/lib/system/libsystem_c.dylib
    0x7fff978b6000 -     0x7fff978c0ff7  liblaunch.dylib (392.18.0 - compatibility 1.0.0) <39EF04F2-7F0C-3435-B785-BF283727FFBD> /usr/lib/system/liblaunch.dylib
 
 
External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 2200
    thread_create: 0
    thread_set_state: 0
 
 
VM Region Summary:
ReadOnly portion of Libraries: Total=60.9M resident=29.0M(48%) swapped_out_or_unallocated=31.9M(52%)
Writable regions: Total=49.5M written=2296K(5%) resident=19.6M(40%) swapped_out=0K(0%) unallocated=29.9M(60%)
 
REGION TYPE                      VIRTUAL
===========                      =======
MALLOC                             34.4M
MALLOC guard page                    48K
SQLite page cache                    96K
STACK GUARD                           4K
Stack                              70.6M
VM_ALLOCATE                           8K
__DATA                             1780K
__LINKEDIT                         48.0M
__TEXT                             12.9M
__UNICODE                           544K
shared memory                        12K
===========                      =======
TOTAL                             168.4M
 
 
Model: Macmini3,1, BootROM MM31.0081.B06, 2 processors, Intel Core 2 Duo, 2 GHz, 4 GB, SMC 1.35f0
Graphics: NVIDIA GeForce 9400, NVIDIA GeForce 9400, PCI, 256 MB
Memory Module: BANK 0/DIMM0, 2 GB, DDR3, 1067 MHz, 0x802C, 0x31364A53463235363634485A2D3147314631
Memory Module: BANK 1/DIMM0, 2 GB, DDR3, 1067 MHz, 0x802C, 0x31364A53463235363634485A2D3147314631
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x90), Broadcom BCM43xx 1.0 (5.10.131.36.11)
Bluetooth: Version 4.0.1f4, 2 service, 18 devices, 1 incoming serial ports
Network Service: AirPort, AirPort, en1
Network Service: Ethernet, Ethernet, en0
Serial ATA Device: Hitachi HTS543212L9SA02, 120.03 GB
Serial ATA Device: PIONEER DVD-RW  DVRTS08
USB Device: Mac OS X Install Disk, apple_vendor_id, 0x1502, 0x26200000 / 2
USB Device: BRCM2046 Hub, 0x0a5c  (Broadcom Corp.), 0x4500, 0x06100000 / 2
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x8216, 0x06110000 / 3
USB Device: IR Receiver, apple_vendor_id, 0x8242, 0x04500000 / 2
FireWire Device: Rev2, Drobo, 400mbit_speed

----------
messages: 151648
nosy: Dan.kamp
priority: normal
severity: normal
status: open
title: exception error
type: crash
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Thu Jan 19 20:41:58 2012
From: report at bugs.python.org (Julian Berman)
Date: Thu, 19 Jan 2012 19:41:58 +0000
Subject: [issue13826] Having a shlex example in the subprocess.Popen docs is
	confusing
In-Reply-To: <1326983115.48.0.551767400168.issue13826@psf.upfronthosting.co.za>
Message-ID: <1327002118.25.0.675845405273.issue13826@psf.upfronthosting.co.za>


Julian Berman <Julian+Python.org at GrayVines.com> added the comment:

Sounds reasonable to me.

I'll take a look at adding one unless someone manages to beat me to it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13826>
_______________________________________

From report at bugs.python.org  Thu Jan 19 20:56:37 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Thu, 19 Jan 2012 19:56:37 +0000
Subject: [issue13830] codecs error handler is called with a UnicodeDecodeError
	with the same args
Message-ID: <1327002997.43.0.435824061039.issue13830@psf.upfronthosting.co.za>


New submission from Amaury Forgeot d'Arc <amauryfa at gmail.com>:

The script below shows that the error handler is always called with the same error object.  The 'start', 'end', and 'reason' properties are correctly updated, but the 'args' is always the same and holds the values used for the first call.

It's a bit weird that error.args[2] is not equal to error.start, for example. All versions are affected: 2.7, 3.2, 3.3.
And by the way, I could not find where these are attributes documented.



def custom_handler(error):
    print(error.args,
          (error.start, error.end, error.reason))
    return b'?'.decode(), error.end

import codecs
codecs.register_error('custom', custom_handler)
b'\x80\xd0'.decode('utf-8', 'custom')

----------
components: Unicode
messages: 151650
nosy: amaury.forgeotdarc, ezio.melotti
priority: normal
severity: normal
status: open
title: codecs error handler is called with a UnicodeDecodeError with the same args
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13830>
_______________________________________

From report at bugs.python.org  Thu Jan 19 21:46:40 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Thu, 19 Jan 2012 20:46:40 +0000
Subject: [issue13831] get method of multiprocessing.pool.Async should return
	full traceback
Message-ID: <1327006000.05.0.152231015377.issue13831@psf.upfronthosting.co.za>


New submission from Faheem Mitha <faheem at faheem.info>:

The documentation in 
http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool

says

"""class multiprocessing.pool.AsyncResult?
The class of the result returned by Pool.apply_async() and Pool.map_async().

get([timeout])
Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get()."""

Consider the example code

################################

from multiprocessing import Pool

def go():
    print 1
    raise Exception("foobar")
    print 2

p = Pool()
x = p.apply_async(go)
x.get()
p.close()
p.join()

###########################

The traceback from this is

Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get
    raise self._value
Exception: foobar
1

As is clear in this example, this is not a full traceback - it only shows the traceback to the line where get is located and gives no further information. This is the case in all the other places I have used get. It seems to me that it *should* return the full traceback, which may contain important information missing in such a partial one. I don't know whether one would call this a feature request or a bug report. Maybe there is some technical reason why this is not possible, but I can't think of one.

----------
components: Library (Lib)
messages: 151651
nosy: fmitha
priority: normal
severity: normal
status: open
title: get method of  multiprocessing.pool.Async should return full traceback
type: enhancement
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13831>
_______________________________________

From report at bugs.python.org  Thu Jan 19 21:52:37 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 19 Jan 2012 20:52:37 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
Message-ID: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

Parser/parsetok.c was recently changed (e.g. http://hg.python.org/cpython/rev/2bd7f40108b4 ) to raise an error if multiple statements were found in a single-statement compile call.  It sensibly ignores trailing whitespace and comments.  Unfortunately,

(1)  It looks only at (c == ' ' || c == '\t' || c == '\n' || c == '\014') as opposed to using Py_UNICODE_ISSPACE(ch)
(2)  It assumes that a "#" means the rest of the line is OK, instead of looking for additional linebreaks.

Not sure whether to mark this a bug or an enhancement, since it is already strictly better than the 3.2 behavior of never warning about extra text.

----------
components: Interpreter Core
messages: 151652
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: tokenization assuming ASCII whitespace; missing multiline case
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:03:06 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 19 Jan 2012 21:03:06 +0000
Subject: [issue11948] Tutorial/Modules - small fix to better clarify the
	modules search path
In-Reply-To: <1304011111.24.0.834772686878.issue11948@psf.upfronthosting.co.za>
Message-ID: <1327006986.29.0.238160529646.issue11948@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Definitely out-of-scope for the tutorial. I consider this akin to monkey patching imported modules.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11948>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:03:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 21:03:39 +0000
Subject: [issue13605] document argparse's nargs=REMAINDER
In-Reply-To: <1323955710.42.0.753468095058.issue13605@psf.upfronthosting.co.za>
Message-ID: <E1Rnz8t-0004vI-FT@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 35665f6f3674 by Sandro Tosi in branch '2.7':
Issue #13605: add documentation for nargs=argparse.REMAINDER
http://hg.python.org/cpython/rev/35665f6f3674

New changeset 6f3d55f5a31e by Sandro Tosi in branch '3.2':
Issue #13605: add documentation for nargs=argparse.REMAINDER
http://hg.python.org/cpython/rev/6f3d55f5a31e

New changeset 6b4cec0719a3 by Sandro Tosi in branch 'default':
Issue #13605: merge with 3.2
http://hg.python.org/cpython/rev/6b4cec0719a3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13605>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:09:23 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Thu, 19 Jan 2012 21:09:23 +0000
Subject: [issue13605] document argparse's nargs=REMAINDER
In-Reply-To: <1323955710.42.0.753468095058.issue13605@psf.upfronthosting.co.za>
Message-ID: <1327007363.77.0.140012269799.issue13605@psf.upfronthosting.co.za>


Changes by Sandro Tosi <sandro.tosi at gmail.com>:


----------
nosy: +sandro.tosi
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13605>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:26:27 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 21:26:27 +0000
Subject: [issue13605] document argparse's nargs=REMAINDER
In-Reply-To: <1323955710.42.0.753468095058.issue13605@psf.upfronthosting.co.za>
Message-ID: <E1RnzV3-0007fS-Oa@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1b481e76cd16 by Sandro Tosi in branch '2.7':
Issue #13605: more meaningful example + fixes
http://hg.python.org/cpython/rev/1b481e76cd16

New changeset d6e53d1f46eb by Sandro Tosi in branch '3.2':
Issue #13605: more meaningful example + fixes
http://hg.python.org/cpython/rev/d6e53d1f46eb

New changeset 4c3271527794 by Sandro Tosi in branch 'default':
Issue #13605: merge with 3.2
http://hg.python.org/cpython/rev/4c3271527794

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13605>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:43:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 19 Jan 2012 21:43:25 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
In-Reply-To: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>
Message-ID: <1327009405.93.0.456716081211.issue13832@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:52:33 2012
From: report at bugs.python.org (Torsten Landschoff)
Date: Thu, 19 Jan 2012 21:52:33 +0000
Subject: [issue13020] structseq.c: refleak
In-Reply-To: <1316530655.8.0.189777030615.issue13020@psf.upfronthosting.co.za>
Message-ID: <1327009953.64.0.0971332436855.issue13020@psf.upfronthosting.co.za>


Torsten Landschoff <t.landschoff at gmx.net> added the comment:

+1 for the patch.
All the error paths above the change do

  Py_DECREF(arg);
  return NULL;

arg is initialized with PySequence_Fast, which returns a new reference.
Hard to create a test case for this...

----------
nosy: +torsten

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13020>
_______________________________________

From report at bugs.python.org  Thu Jan 19 22:56:14 2012
From: report at bugs.python.org (Torsten Landschoff)
Date: Thu, 19 Jan 2012 21:56:14 +0000
Subject: [issue13833] No documentation for PyStructSequence
Message-ID: <1327010174.09.0.407534369688.issue13833@psf.upfronthosting.co.za>


New submission from Torsten Landschoff <t.landschoff at gmx.net>:

While writing a C extension I wanted to create a namedtuple like object as os.statvfs and friends do.

I was unable to find a simple way to do this from C and was wondering how the posixmodule does it. It turned out that there is a PyStructSequence type for this.

It would be nice to have it documented. First draft for the documentation update attached.

----------
assignee: docs at python
components: Documentation
files: structseq_doc.diff
keywords: patch
messages: 151657
nosy: docs at python, torsten
priority: normal
severity: normal
status: open
title: No documentation for PyStructSequence
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24280/structseq_doc.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13833>
_______________________________________

From report at bugs.python.org  Thu Jan 19 23:48:32 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 22:48:32 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
In-Reply-To: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>
Message-ID: <E1Ro0mV-0006Gc-3P@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 00c4efbf57c3 by Benjamin Peterson in branch 'default':
check after comments, too (#13832)
http://hg.python.org/cpython/rev/00c4efbf57c3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Thu Jan 19 23:49:13 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Thu, 19 Jan 2012 22:49:13 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
In-Reply-To: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>
Message-ID: <1327013353.35.0.540804160425.issue13832@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

The tokenizer doesn't consider unicode spaces, either.

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Fri Jan 20 00:13:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 19 Jan 2012 23:13:47 +0000
Subject: [issue13763] Potentially hard to understand wording in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <E1Ro1Av-0008Sk-8t@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 081106c142ec by Terry Jan Reedy in branch 'default':
#13763 Clarify 'hg' usage.
http://hg.python.org/devguide/rev/081106c142ec

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Fri Jan 20 00:19:04 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 19 Jan 2012 23:19:04 +0000
Subject: [issue13763] Potentially hard to understand wording in devguide
In-Reply-To: <1326278487.37.0.199929224509.issue13763@psf.upfronthosting.co.za>
Message-ID: <1327015144.4.0.857818996137.issue13763@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I do not much like sentences starting with lowercase, so I combined our sentences. The result is good enough, I think. Closing.

----------
assignee:  -> terry.reedy
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13763>
_______________________________________

From report at bugs.python.org  Fri Jan 20 01:38:01 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 20 Jan 2012 00:38:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1326996353.21.0.0703813111949.issue13703@psf.upfronthosting.co.za>
Message-ID: <4F18B762.1000608@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Frank Sievertsen wrote:
> 
> Frank Sievertsen <python at sievertsen.de> added the comment:
> 
>> The suffix only introduces a constant change in all hash values
>> output, so even if you don't know the suffix, you can still
>> generate data sets with collisions by just having the prefix.
> 
> That's true. But without the suffix, I can pretty easy and efficient guess the prefix by just seeing the result of a few well-chosen and short repr(dict(X)). I suppose that's harder with the suffix.

Since the hash function is known, it doesn't make things much
harder. Without suffix you just need hash('') to find out what
the prefix is. With suffix, two values are enough.

Say P is your prefix and S your suffix. Let's say you can get the
hash values of A = hash('') and B = hash('\x00').

With Victor's hash function you have (IIRC):

A = hash('')     = P ^ (0<<7) ^ 0 ^ S = P ^ S
B = hash('\x00') = ((P ^ (0<<7)) * 1000003) ^ 0 ^ 1 ^ S = (P * 1000003) ^ 1 ^ S

Let X = A ^ B, then

X = P ^ (P * 1000003) ^ 1

since S ^ S = 0 and 0 ^ Y = Y (for any Y), i.e. the suffix doesn't
make any difference.

For P < 500000, you can then easily calculate P from X
using:

P = X // 1000002

(things obviously get tricky once overflow kicks in)

Note that for number hashes the randomization doesn't work at all,
since there's no length or feedback loop involved.

With Victor's approach hash(0) would output the whole seed,
but even if the seed is not known, creating an attack data
set is trivial, since hash(x) = P ^ x ^ S.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 02:04:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 01:04:47 +0000
Subject: [issue13811] In str.format,
	if invalid fill and alignment are specified,
	the text of the ValueError message is misleading.
In-Reply-To: <1326848044.1.0.328293668269.issue13811@psf.upfronthosting.co.za>
Message-ID: <E1Ro2uK-0001DM-Rt@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5c33ebb50702 by Eric V. Smith in branch 'default':
Improve exception text. Closes issue 13811.
http://hg.python.org/cpython/rev/5c33ebb50702

----------
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13811>
_______________________________________

From report at bugs.python.org  Fri Jan 20 02:11:24 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 20 Jan 2012 01:11:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F18B762.1000608@egenix.com>
Message-ID: <CAMpsgwYbiLgUvVnm+Pgv0o_Y1JbLVgRDuC_KXb5PoA+6HATMKQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Since the hash function is known, it doesn't make things much
> harder. Without suffix you just need hash('') to find out what
> the prefix is. With suffix, two values are enough.

With my patch, hash('') always return zero. I don't remember who asked
me to do that, but it avoids to leak too easily the secret :-) I wrote
some info how to compute the secret:
http://bugs.python.org/issue13703#msg150706

I don't see how to compute the secret, but it doesn't mean that it is
impossible :-) I suppose that you have to brute force some bits, at
least if you only have repr(dict) which gives only (indirectly) the
lower bits of the hash.

> (things obviously get tricky once overflow kicks in)

hash() doesn't overflow: if you know the string, you can run the
algorithm backward. To divide, you can compute 1/1000003 mod 2^32 (or
mod 2^64): 2021759595 and 16109806864799210091. So x/1000003 mod 2^32
= x*2021759595 mod 2^32.

See my invert_mod() function of:
https://bitbucket.org/haypo/misc/src/tip/python/mathfunc.py

> With Victor's approach hash(0) would output the whole seed,
> but even if the seed is not known, creating an attack data
> set is trivial, since hash(x) = P ^ x ^ S.

I suppose that it would be too simple to compute the secret of a
randomized integer hash, so it is maybe better to leave them
unchanged. Using a different secret from strings and integer would not
protect Python against an attack only using integers, but integer keys
are less common than string keys (especially on web applications).

Anyway, I changed my mind about randomized hash: I now prefer counting
collisions :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 02:12:45 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 20 Jan 2012 01:12:45 +0000
Subject: [issue13828] Further improve casefold documentation
In-Reply-To: <1326992763.46.0.576190503242.issue13828@psf.upfronthosting.co.za>
Message-ID: <1327021965.02.0.64214478526.issue13828@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

It's a bit unfriendly to launch into discussion of "compatiblity caseless matching" when the new reader probably has no idea what "compatibility-equivalence" is.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13828>
_______________________________________

From report at bugs.python.org  Fri Jan 20 02:22:52 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 01:22:52 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <E1Ro3Bp-0002Yf-49@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8502a9236c2e by Victor Stinner in branch 'default':
Issue #10278: Be more explicit in tests than wallclock() is monotonic (cannot
http://hg.python.org/cpython/rev/8502a9236c2e

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:03:41 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:03:41 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327028621.53.0.982454171845.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


Added file: http://bugs.python.org/file24281/5458412752d5.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:04:29 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:04:29 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327028669.51.0.941648088081.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


Added file: http://bugs.python.org/file24282/f86bb02fd8f4.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:05:13 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:05:13 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327028713.2.0.563184446543.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


Added file: http://bugs.python.org/file24283/f86bb02fd8f4.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:07:03 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:07:03 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327028823.47.0.15954844456.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


Removed file: http://bugs.python.org/file23920/f73be85b9a7e.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:07:28 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Fri, 20 Jan 2012 03:07:28 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
In-Reply-To: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>
Message-ID: <1327028848.1.0.440827263613.issue13807@psf.upfronthosting.co.za>


Changes by Vinay Sajip <vinay_sajip at yahoo.co.uk>:


----------
assignee:  -> vinay.sajip
nosy: +vinay.sajip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:07:31 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:07:31 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327028851.36.0.0344641444671.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


Removed file: http://bugs.python.org/file23921/43d1a819a63d.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:16:44 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:16:44 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327029404.05.0.178044137069.issue13405@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Code ready for commit.

Please, review.

After the first commit, I will try to make it compatible with MacOS and FreeBSD. Current target is Solaris and derivatives (OpenIndiana, for instance)

Stan Cox, if you want systemtap compatibility at this moment, please provide a patch. The first patch doesn't need "ustack" compatibility. In fact, MacOS dtrace, for instance, seems not to have "ustack" support at all (according to Google), so the feature is very convenient but optional.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:31:20 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 20 Jan 2012 03:31:20 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327030280.55.0.114900630123.issue13405@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I'm -1 on this patch for essentially the same reasons as Charles-Fran?ois. It introduces a lot of code (and hacks!) in critical pathways of the interpreter. Someone would have to be constantly maintaining and testing it. In return, what do we get?

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:48:26 2012
From: report at bugs.python.org (Meador Inge)
Date: Fri, 20 Jan 2012 03:48:26 +0000
Subject: [issue12949] Documentation of PyCode_New() lacks kwonlyargcount
	argument
In-Reply-To: <1315592789.75.0.632556595518.issue12949@psf.upfronthosting.co.za>
Message-ID: <1327031306.8.0.898248612484.issue12949@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
keywords: +easy
stage:  -> needs patch
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12949>
_______________________________________

From report at bugs.python.org  Fri Jan 20 04:51:36 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 03:51:36 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327031496.47.0.0201973945798.issue13405@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

NOTE to myself: In OpenIndiana we could have this error:

"""
dtrace: failed to compile script ./Include/phelper.d: Preprocessor not found
"""

In that case, we must install the C preprocessor:

"""
pfexec pkg install cpp
"""

I confirm that current 3.3 patch works in the 32 bits OpenIndiana Python buildbots, after adding this patch (already available in my mercurial repository, but not in the patch published in this issue):

"""
diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py
--- a/Lib/test/test_dtrace.py
+++ b/Lib/test/test_dtrace.py
@@ -173,7 +173,7 @@
         # Verify that we are checking:
         opcodes = set(["CALL_FUNCTION", "CALL_FUNCTION_VAR",
             "CALL_FUNCTION_KW", "CALL_FUNCTION_VAR_KW"])
-        obj = compile(open(sample).read(), "sample", "exec")
+        obj = compile(open(sample, encoding="utf-8").read(), "sample", "exec")
         class dump() :
             def __init__(self) :
                 self.buf = []
"""

Also, of course, the user must have dtrace permissions, as documented in a previous message.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:28:48 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 04:28:48 +0000
Subject: [issue12949] Documentation of PyCode_New() lacks kwonlyargcount
	argument
In-Reply-To: <1315592789.75.0.632556595518.issue12949@psf.upfronthosting.co.za>
Message-ID: <E1Ro65m-0008Q4-G6@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b22a35c14a91 by Meador Inge in branch '3.2':
Issue #12949: Document the kwonlyargcount argument for the PyCode_New C API function.
http://hg.python.org/cpython/rev/b22a35c14a91

New changeset 218b167ff521 by Meador Inge in branch 'default':
Issue #12949: Document the kwonlyargcount argument for the PyCode_New C API function.
http://hg.python.org/cpython/rev/218b167ff521

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12949>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:30:22 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Fri, 20 Jan 2012 04:30:22 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327033822.49.0.0348549225218.issue13806@psf.upfronthosting.co.za>


Changes by Oleg Plakhotnyuk <olegus8 at gmail.com>:


Removed file: http://bugs.python.org/file24260/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:31:38 2012
From: report at bugs.python.org (Meador Inge)
Date: Fri, 20 Jan 2012 04:31:38 +0000
Subject: [issue12949] Documentation of PyCode_New() lacks kwonlyargcount
	argument
In-Reply-To: <1315592789.75.0.632556595518.issue12949@psf.upfronthosting.co.za>
Message-ID: <1327033898.54.0.519873177155.issue12949@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

Fixed.  Thanks for the report Stefan.

----------
nosy: +meador.inge
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12949>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:32:31 2012
From: report at bugs.python.org (Glyph Lefkowitz)
Date: Fri, 20 Jan 2012 04:32:31 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327033951.29.0.591725183373.issue13405@psf.upfronthosting.co.za>


Glyph Lefkowitz <glyph at twistedmatrix.com> added the comment:

Charles-Fran?ois:
> Also, I must admit I'm quite skeptical about the real benefit of
explicit probes for user-land, especially for CPython which isn't used
for performance-critical systems...

I beg to differ.  CPython is totally used on performance-critical systems, and I know I'm not the only user who thinks that.  "Performance-critical" doesn't necessarily mean "goes as fast as it ever possibly can", clearly PyPy is the place to go for that, but "can process at least X work in Y time".  Meeting performance goals with CPython is already challenging enough, please don't make it artificially hard by refusing to integrate tools which help users understand and improve performance.

Benjamin:
> I'm -1 on this patch for essentially the same reasons as Charles-Fran?ois. It introduces a lot of code (and hacks!) in critical pathways of the interpreter. Someone would have to be constantly maintaining and testing it. In return, what do we get?

You get support for a highly sophisticated and low-impact profiling and tracing technology which provides support for illuminating performance problems *as well as* complicated behavioral problems that only happen under load, without slowing down the interpreter as a whole.  Not to mention possible integration with a whole slew of tools that know how to deal with data from that system.

I'm not saying that this is necessarily worth the maintenance burden; your analysis of the tradeoff may ultimately be correct.  I can't presume to know that because I am not intimately familiar with all the code it touches.  But it's definitely not nothing.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:35:04 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Fri, 20 Jan 2012 04:35:04 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327034104.81.0.213790621655.issue13806@psf.upfronthosting.co.za>


Oleg Plakhotnyuk <olegus8 at gmail.com> added the comment:

Yep, you're right. Didn't noticed audioop_check_size() function at first.

The fact that audioop accepts unicode strings seems weird to me too. I've replaced strings with bytes in tests. However, I'm afraid to add restrictions to library itself because of backward compatibility.

----------
Added file: http://bugs.python.org/file24284/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:39:07 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Fri, 20 Jan 2012 04:39:07 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327034347.13.0.0061836487062.issue13806@psf.upfronthosting.co.za>


Changes by Oleg Plakhotnyuk <olegus8 at gmail.com>:


Removed file: http://bugs.python.org/file24284/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:39:34 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Fri, 20 Jan 2012 04:39:34 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327034374.07.0.614741152528.issue13806@psf.upfronthosting.co.za>


Changes by Oleg Plakhotnyuk <olegus8 at gmail.com>:


Added file: http://bugs.python.org/file24285/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:44:16 2012
From: report at bugs.python.org (John Levon)
Date: Fri, 20 Jan 2012 04:44:16 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327034656.49.0.699358239071.issue13405@psf.upfronthosting.co.za>


John Levon <movement at users.sourceforge.net> added the comment:

I would strongly suggest those of you who are not "getting it" to actually try Jesus's patch out in some real scenarios. You'll quickly see what the benefit is. I think some of you are missing that it's a *not* just about performance: it's a system comprehension tool. It's there to help answer questions, on a live system; sometimes those are performance questions, but more often they're *not*.

I can't speak as to the maintenance burden. I'm sure we can all agree that the  ceval.c changes are not pretty - it's why I never submitted my original changes. But speaking for the Solaris organization, the burden of maintaining our patches across the various Python releases has been minimal.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:46:11 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 20 Jan 2012 04:46:11 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327034771.72.0.795847886613.issue13405@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

As great as a tool it maybe, it's still only available on a minority platform. So I couldn't really try it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:52:57 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Fri, 20 Jan 2012 04:52:57 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1327034771.72.0.795847886613.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327035145.4992.109.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Fri, 2012-01-20 at 04:46 +0000, Benjamin Peterson wrote:
> Benjamin Peterson <benjamin at python.org> added the comment:
> 
> As great as a tool it maybe, it's still only available on a minority platform. So I couldn't really try it.

FWIW, the analogous systemtap patch works great on Linux, or, at least
Fedora 13+/RHEL 6 (though beware: not all Linux distributions have
systemtap working properly out of the box, alas).

See:
http://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2
and http://bugs.python.org/issue4111 though this stalled due to lack of
documentation (I started writing some, but got bogged down in the
differences between the tapsets and the static markers; sorry).

Dave

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 05:58:37 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Fri, 20 Jan 2012 04:58:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327035517.22.0.845370766756.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

>> That's true. But without the suffix, I can pretty easy and efficient
>> guess the prefix by just seeing the result of a few well-chosen and
>> short repr(dict(X)). I suppose that's harder with the suffix.

> Since the hash function is known, it doesn't make things much
> harder. Without suffix you just need hash('') to find out what
> the prefix is. With suffix, two values are enough

This is obvious and absolutely correct!

But it's not what I talked about. I didn't talk about the result of
hash(X), but about the result of repr(dict([(str: val), (str:
val)....])), which is more likely to happen and not so trivial
(if you want to know more than the last 8 bits)

IMHO this problem shows that we can't advice dict() or set() for
(potential dangerous) user-supplied keys at the moment.

I prefer randomization because it fixes this problem. The
collision-counting->exception prevents a software from becoming slow,
but it doesn't make it work as expected.

Sure, you can catch the exception. But when you get the exception,
probably you wanted to add the items for a reason: Because you want
them to be in the dict and that's how your software works.

Imagine an irc-server using a dict to store the connected users, using
the nicknames as keys. Even if the irc-server catches the unexpected
exception while connecting a new user (when adding his/her name to the
dict), an attacker could connect 999 special-named users to prevent a
specific user from connecting in future.

Collision-counting->exception can make it possible to inhibit a
specific future add to the dict. The outcome is highly application
dependent.

I think it fixes 95% of the attack-vectors, but not all and it adds a
few new risks. However, of course it's much better then doing nothing
to fix the problem.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 09:18:51 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Fri, 20 Jan 2012 08:18:51 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1327035145.4992.109.camel@surprise>
Message-ID: <CAPkN8xLQrk5T4KKnObLq-TGt1j-tHoCJN8F4KsfeKYUPLS4F1w@mail.gmail.com>


anatoly techtonik <techtonik at gmail.com> added the comment:

I am an almost complete 0 in C system programming and in DTrace matters,
but I feel like DTrace has a potential to help me understand internal
CPython processes better. If maintenance of the code with DTrace is hard,
there are several ways to make it easier:

1. PyCon talk and a reference to video (why DTrace is cool for Python and
how to use it)
2. Chapter in Development guide explaining DTrace magic, how it works
3. Annotated example (like in "dive into" series)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 10:03:20 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 20 Jan 2012 09:03:20 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327050200.08.0.337334307592.issue13703@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> A dict can contain non-orderable keys, I don't know how an AVL tree
> can fit into that.

They may be non-orderable, but since they are required to be hashable,
I guess one can build an comparison function with the following:

def cmp(x, y):
    if x == y:
        return 0
     elif hash(x) <= hash(y):
         return -1
     else:
         return 1

It doesn't yield a mathematical order because it lacks the
anti-symmetry property, but it should be enough for a binary search
tree.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 10:30:41 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Fri, 20 Jan 2012 09:30:41 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327051841.74.0.786264271225.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

> They may be non-orderable, but since they are required to be hashable,
> I guess one can build an comparison function with the following:

Since we are are trying to fix a problem where hash(X) == hash(Y), you
can't make them orderable by using the hash-values and build a binary
out of the (equal) hash-values.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 11:17:35 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Fri, 20 Jan 2012 10:17:35 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>
Message-ID: <1327054655.58.0.870866828169.issue13736@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
assignee:  -> orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Fri Jan 20 11:39:46 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 20 Jan 2012 10:39:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327051841.74.0.786264271225.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM0cdFYPJgbMgbR60o-9B_t_4prfxfMcayCO7ZUjqg1yHQ@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Since we are are trying to fix a problem where hash(X) == hash(Y), you
> can't make them orderable by using the hash-values and build a binary
> out of the (equal) hash-values.

That's not what I suggested.
Keys would be considered equal if they are indeed equal (__eq__). The
hash value is just used to know if the key belongs to the left or the
right child tree. With a self-balanced binary search tree, you'd still
get O(log(N)) complexity.

Anyway, I still think that the hash randomization is the right way to
go, simply because it does solve the problem, whereas the collision
counting doesn't: Martin made a very good point on python-dev with his
database example.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 11:43:10 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Fri, 20 Jan 2012 10:43:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327056190.08.0.454659206043.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

> The hash value is just used to know if the key belongs to the left
> or the right child tree.

Yes, that's what I don't understand: How can you do this, when ALL
hash-values are equal.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 11:51:04 2012
From: report at bugs.python.org (Martijn van Oosterhout)
Date: Fri, 20 Jan 2012 10:51:04 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1327056664.78.0.958799644377.issue13817@psf.upfronthosting.co.za>


Martijn van Oosterhout <kleptog at gmail.com> added the comment:

What a concidence, we had this problem as well and just today tracked it down, and turns out someone filed this bug yesterday.

I note the referenced bug (#13156) says that it's been fixed in the release branch but not yet released. I don't suppose there will be a release of 2.7 in the near future?

----------
nosy: +Martijn.van.Oosterhout

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Fri Jan 20 11:52:35 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 20 Jan 2012 10:52:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327056190.08.0.454659206043.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3uCe1oa+zM0V-e-JwVJaphLGRjeYYeEarhDbMU0ikzMw@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Yes, that's what I don't understand: How can you do this, when ALL
> hash-values are equal.

You're right, that's stupid.
Short night...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 12:17:32 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Fri, 20 Jan 2012 11:17:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAH_1eM0cdFYPJgbMgbR60o-9B_t_4prfxfMcayCO7ZUjqg1yHQ@mail.gmail.com>
Message-ID: <4F194D43.4060705@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Charles-Fran?ois Natali wrote:
> 
> Anyway, I still think that the hash randomization is the right way to
> go, simply because it does solve the problem, whereas the collision
> counting doesn't: Martin made a very good point on python-dev with his
> database example.

For completeness, I quote Martin here:

"""
The main issue with that approach is that it allows a new kind of attack.

An attacker now needs to find 1000 colliding keys, and submit them
one-by-one into a database. The limit will not trigger, as those are
just database insertions.

Now, if the applications also as a need to read the entire database
table into a dictionary, that will suddenly break, and not for the
attacker (which would be ok), but for the regular user of the
application or the site administrator.

So it may be that this approach actually simplifies the attack, making
the cure worse than the disease.
"""

Martin is correct in that it is possible to trick an application
into building some data pool which can then be used as indirect
input for an attack.

What I don't see is what's wrong with the application raising
an exception in case it finds such data in an untrusted source
(reading arbitrary amounts of user data from a database is just
as dangerous as reading such data from any other source).

The exception will tell the programmer to be more careful and
patch the application not to read untrusted data without
additional precautions.

It will also tell the maintainer of the application that there
was indeed an attack on the system which may need to be
tracked down.

Note that the collision counting demo patch is trivial - I just
wanted to demonstrate how it works. As already mentioned, there's
room for improvement:

If Python objects were to provide an additional
method for calculating a universal hash value (based on an
integer input parameter), the dictionary in question could
use this to rehash itself and avoid the attack. Think of this
as "randomization when needed". (*)

Since the dict would still detect the problem, it could also
raise a warning to inform the maintainer of the application.

So you get the best of both worlds and randomization would only
kick in when it's really needed to keep the application running.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 12:32:13 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 11:32:13 +0000
Subject: [issue13807] logging.Handler.handlerError() may raise AttributeError
	in traceback.print_exception()
In-Reply-To: <1326816014.41.0.202399739958.issue13807@psf.upfronthosting.co.za>
Message-ID: <E1RoChX-0002Ry-Tt@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset b60c789c4ccd by Vinay Sajip in branch '2.7':
Closes #13807: Now checks for sys.stderr being there before writing to it.
http://hg.python.org/cpython/rev/b60c789c4ccd

New changeset 73dad4940b88 by Vinay Sajip in branch '3.1':
Closes #13807: Now checks for sys.stderr being there before writing to it.
http://hg.python.org/cpython/rev/73dad4940b88

New changeset 6a1a33a1fe93 by Vinay Sajip in branch '3.2':
Closes #13807: Merged fix from 3.1.
http://hg.python.org/cpython/rev/6a1a33a1fe93

New changeset de85a9c7fe94 by Vinay Sajip in branch 'default':
Closes #13807: Merged fix from 3.2.
http://hg.python.org/cpython/rev/de85a9c7fe94

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13807>
_______________________________________

From report at bugs.python.org  Fri Jan 20 12:37:39 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Fri, 20 Jan 2012 11:37:39 +0000
Subject: [issue13516] Gzip old log files in rotating handlers
In-Reply-To: <1322762423.02.0.0799773449724.issue13516@psf.upfronthosting.co.za>
Message-ID: <1327059459.92.0.0161334573242.issue13516@psf.upfronthosting.co.za>


Vinay Sajip <vinay_sajip at yahoo.co.uk> added the comment:

Refactoring in 57295c4d81ac supports this use case.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13516>
_______________________________________

From report at bugs.python.org  Fri Jan 20 13:13:45 2012
From: report at bugs.python.org (Maxim Yegorushkin)
Date: Fri, 20 Jan 2012 12:13:45 +0000
Subject: [issue8108] test_ftplib fails with OpenSSL 0.9.8m
In-Reply-To: <1268228679.44.0.26304365082.issue8108@psf.upfronthosting.co.za>
Message-ID: <1327061625.67.0.580157366105.issue8108@psf.upfronthosting.co.za>


Maxim Yegorushkin <maxim.yegorushkin at gmail.com> added the comment:

I am still having this issue with Python-2.7.2 (which includes the patch) and openssl-devel-1.0.0e-1.fc14.x86_64:

  File "./download.py", line 195, in download_file
    ftp.retrbinary("retr " + filename, lambda data: file.write(data))
  File "/usr/local/ots/python/lib/python2.7/ftplib.py", line 691, in retrbinary
    conn.unwrap()
  File "/usr/local/ots/python/lib/python2.7/ssl.py", line 275, in unwrap
    s = self._sslobj.shutdown()
socket.error: [Errno 0] Error

----------
nosy: +max0x7ba

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8108>
_______________________________________

From report at bugs.python.org  Fri Jan 20 13:58:05 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 20 Jan 2012 12:58:05 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F194D43.4060705@egenix.com>
Message-ID: <CAMpsgwZT5u82v54CmehuJU9jgTYpxRwEUw2ezyt8=un_Wmms6g@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Note that the collision counting demo patch is trivial - I just
> wanted to demonstrate how it works. As already mentioned, there's
> room for improvement:
>
> If Python objects were to provide an additional
> method for calculating a universal hash value (based on an
> integer input parameter), the dictionary in question could
> use this to rehash itself and avoid the attack. Think of this
> as "randomization when needed".

Yes, the solution can be improved, but maybe not in stable versions
(the patch for stable versions should be short and simple).

If the hash output depends on an argument, the result cannot be
cached. So I suppose that dictionary lookups become slower than the
dictionary switches to the randomized mode. It would require to add an
optional argument to hash functions, or add a new function to some (or
all?) builtin types.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 15:04:42 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 20 Jan 2012 14:04:42 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>
Message-ID: <1327068282.43.0.402913125309.issue13736@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think the general pattern of wrapping exceptions in other exceptions is rather unfortunate. It makes it harder to examine the original problem (for example the ``errno`` attribute) for no actual gain.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Fri Jan 20 15:42:50 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 20 Jan 2012 14:42:50 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F194D43.4060705@egenix.com>
Message-ID: <CAH_1eM0RLx65QTE1ZnyT6ihgkckeC4-Jc3psQmFbY0G4Pe7VMA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> So you get the best of both worlds and randomization would only
> kick in when it's really needed to keep the application running.

Of course, but then the collision counting approach loses its main
advantage over randomized hashing: smaller patch, easier to backport.
If you need to handle a potential abnormal number of collisions
anyway, why not account for it upfront, instead of drastically
complexifying the algorithm? While larger, the randomization is
conceptually simpler.

The only argument in favor the collision counting is that it will not
break applications relying on dict order: it has been argued several
times that such applications are already broken, but that's of course
not an easy decision to make, especially for stable versions...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 15:56:20 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 14:56:20 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <E1RoFt4-0005xe-Hg@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset e63e2471f46f by ?ukasz Langa in branch 'default':
#13760: picklability tests for configparser exceptions
http://hg.python.org/cpython/rev/e63e2471f46f

New changeset 76077971ee1f by ?ukasz Langa in branch '3.2':
#13760: picklability tests for configparser exceptions
http://hg.python.org/cpython/rev/76077971ee1f

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Fri Jan 20 16:02:42 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Fri, 20 Jan 2012 15:02:42 +0000
Subject: [issue13822] is(upper/lower/title) are not exactly correct
In-Reply-To: <1326946975.13.0.733756833132.issue13822@psf.upfronthosting.co.za>
Message-ID: <1327071762.91.0.0645580595658.issue13822@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
stage:  -> needs patch
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13822>
_______________________________________

From report at bugs.python.org  Fri Jan 20 16:07:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 20 Jan 2012 15:07:39 +0000
Subject: [issue13814] Generators as context managers.
In-Reply-To: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>
Message-ID: <1327072059.33.0.624233080987.issue13814@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +ncoghlan
versions: +Python 3.3 -Python 2.7, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Fri Jan 20 16:58:20 2012
From: report at bugs.python.org (Jim Jewett)
Date: Fri, 20 Jan 2012 15:58:20 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
In-Reply-To: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>
Message-ID: <1327075100.16.0.140736499373.issue13832@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Ignoring non-ascii whitespace is defensible, and I agree that it should match the rest of the parser.  Ignoring 2nd lines is still a problem, and supposedly part of what got fixed.  Test case:

s="""x=5  # comment
x=6
"""
compile(s, "<testbadsingle>", 'single')

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Fri Jan 20 17:00:44 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 20 Jan 2012 16:00:44 +0000
Subject: [issue13832] tokenization assuming ASCII whitespace;
	missing multiline case
In-Reply-To: <1327006357.41.0.958836774052.issue13832@psf.upfronthosting.co.za>
Message-ID: <1327075244.22.0.0379446839937.issue13832@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

$ ./python 
Python 3.3.0a0 (default:50a4af2ca654+, Jan 20 2012, 10:59:48) 
[GCC 4.5.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s="""x=5  # comment
... x=6
... """
>>> compile(s, "<blah>", "single")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<blah>", line 1
    x=5  # comment
                 ^
SyntaxError: multiple statements found while compiling a single statement

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________

From report at bugs.python.org  Fri Jan 20 17:03:15 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 20 Jan 2012 16:03:15 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <E1RoGvp-0004MB-K9@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5ecf650ede7c by ?ukasz Langa in branch '2.7':
Fixes #13760: picklability of ConfigParser exceptions
http://hg.python.org/cpython/rev/5ecf650ede7c

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Fri Jan 20 17:05:14 2012
From: report at bugs.python.org (=?utf-8?q?=C5=81ukasz_Langa?=)
Date: Fri, 20 Jan 2012 16:05:14 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1327075514.81.0.0244516744955.issue13760@psf.upfronthosting.co.za>


?ukasz Langa <lukasz at langa.pl> added the comment:

3.2 and 3.3 already worked as expected. For 2.7 I did the __reduce__ workaround that's also used by SQLAlchemy.

----------
resolution:  -> fixed
stage: test needed -> committed/rejected
status: open -> closed
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:00:58 2012
From: report at bugs.python.org (Jyotirmoy Bhattacharya)
Date: Fri, 20 Jan 2012 17:00:58 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1326003740.31.0.0410778534365.issue13736@psf.upfronthosting.co.za>
Message-ID: <1327078858.65.0.535044695004.issue13736@psf.upfronthosting.co.za>


Jyotirmoy Bhattacharya <jyotirmoy at jyotirmoy.net> added the comment:

Antoine, could "raise ... from" be used here? Perhaps also using new subclasses of URLError to allow the exceptions to be caught with finer granularity. That way no information would be lost while at the same time not surprising clients who only catch URLError based on the documentation.

At least one exception which I feel must be wrapped is socket.timeout. Since we allow a timeout argument to urlopen, it doesn't make sense for the fact of the timeout to be announced by an exception from another library.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:02:19 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Fri, 20 Jan 2012 17:02:19 +0000
Subject: [issue13822] is(upper/lower/title) are not exactly correct
In-Reply-To: <1326946975.13.0.733756833132.issue13822@psf.upfronthosting.co.za>
Message-ID: <1327078939.04.0.293207554902.issue13822@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13822>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:13:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 20 Jan 2012 17:13:25 +0000
Subject: [issue13736] urllib.request.urlopen leaks exceptions from socket and
	httplib.client
In-Reply-To: <1327078858.65.0.535044695004.issue13736@psf.upfronthosting.co.za>
Message-ID: <1327079490.3333.9.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Antoine, could "raise ... from" be used here?

That's a possible compromise indeed. It's up to Senthil to decide
anyway.

>  Perhaps also using new subclasses of URLError to allow the exceptions
> to be caught with finer granularity. That way no information would be
> lost while at the same time not surprising clients who only catch
> URLError based on the documentation.

I agree there is a problem with the documentation announcing that all
exceptions will be wrapped. Honestly I would like it better if that
guarantee were dropped. In the meantime I'm not sure what the best
course of action is.

> At least one exception which I feel must be wrapped is socket.timeout.
> Since we allow a timeout argument to urlopen, it doesn't make sense
> for the fact of the timeout to be announced by an exception from
> another library.

You still may be interested to know that the request failed because of a
timeout rather than (say) an authentication failure, no?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13736>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:31:08 2012
From: report at bugs.python.org (Jim Jewett)
Date: Fri, 20 Jan 2012 17:31:08 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAH_1eM0RLx65QTE1ZnyT6ihgkckeC4-Jc3psQmFbY0G4Pe7VMA@mail.gmail.com>
Message-ID: <CA+OGgf6rZqBuxjL5ZRypT+mK8_AMRNg1Q2wsAxFqLVVx39GmqQ@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Marc-Andre Lemburg:
>> So you get the best of both worlds and randomization would only
>> kick in when it's really needed to keep the application running.

Charles-Fran?ois Natali
> The only argument in favor the collision counting is that it will not
> break applications relying on dict order:

There is also the "taxes suck" argument; if hashing is made complex,
then every object (or at least almost every string) pays a price, even
if it will never be stuck in a dict big enough to matter.

With collision counting, there are no additional operations unless and
until there is at least one collision -- in other words, after the
base hash algorithm has already started to fail for that particular
piece of data.

In fact, the base algorithm can be safely simplified further,
precisely because it does not need to be quite as adequate for
reprobes on data that does have at least one collision.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:39:25 2012
From: report at bugs.python.org (Guido van Rossum)
Date: Fri, 20 Jan 2012 17:39:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327035517.22.0.845370766756.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAP7+vJJjNfzfaibbXXrXDjkzy_HC20H4GNoGbLQmK4o8087WRg@mail.gmail.com>


Guido van Rossum <guido at python.org> added the comment:

On Thu, Jan 19, 2012 at 8:58 PM, Frank Sievertsen <report at bugs.python.org>wrote:

>
> Frank Sievertsen <python at sievertsen.de> added the comment:
>
> >> That's true. But without the suffix, I can pretty easy and efficient
> >> guess the prefix by just seeing the result of a few well-chosen and
> >> short repr(dict(X)). I suppose that's harder with the suffix.
>
> > Since the hash function is known, it doesn't make things much
> > harder. Without suffix you just need hash('') to find out what
> > the prefix is. With suffix, two values are enough
>
> This is obvious and absolutely correct!
>
> But it's not what I talked about. I didn't talk about the result of
> hash(X), but about the result of repr(dict([(str: val), (str:
> val)....])), which is more likely to happen and not so trivial
> (if you want to know more than the last 8 bits)
>
> IMHO this problem shows that we can't advice dict() or set() for
> (potential dangerous) user-supplied keys at the moment.
>
> I prefer randomization because it fixes this problem. The
> collision-counting->exception prevents a software from becoming slow,
> but it doesn't make it work as expected.
>

That depends. If collision counting prevents the DoS attack that may be
"work as expected", assuming you believe (as I do) that "real life" data
won't ever have that many collisions.

Note that every web service is vulnerable to some form of DoS where a
sufficient number of malicious requests will keep all available servers
occupied so legitimate requests suffer delays and timeouts. The defense is
to have sufficient capacity so that a potential attacker would need a large
amount of resources to do any real damage. The hash collision attack vastly
reduces the amount of resources needed to bring down a service; crashing
early moves the balance of power significantly back, and that's all we can
ask for.

Sure, you can catch the exception. But when you get the exception,
> probably you wanted to add the items for a reason: Because you want
> them to be in the dict and that's how your software works.
>

No, real data would never make this happen, so it's a "don't care" case (at
least for the vast majority of services). An attacker could also send you
such a large amount of data that your server runs out of memory, or starts
swapping (which is almost worse). But that requires for the attacker to
have enough bandwidth to send you that data. Or they could send you very
many requests. Same requirement.

All we need to guard for here is the unfortunate multiplication of the
attacker's effort due to the behavior of the collision-resolution code in
the dict implementation. Beyond that it's every app for itself.

> Imagine an irc-server using a dict to store the connected users, using
> the nicknames as keys. Even if the irc-server catches the unexpected
> exception while connecting a new user (when adding his/her name to the
> dict), an attacker could connect 999 special-named users to prevent a
> specific user from connecting in future.
>

Or they could use many other tactics. At this point the attack is specific
to this IRC implementation and it's no longer Python's responsibility.

> Collision-counting->exception can make it possible to inhibit a
> specific future add to the dict. The outcome is highly application
> dependent.
>
> I think it fixes 95% of the attack-vectors, but not all and it adds a
> few new risks. However, of course it's much better then doing nothing
> to fix the problem.
>

Right -- it vastly increases the effort needed to attack any particular
service, and does not affect any behavior of existing Python apps.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:42:07 2012
From: report at bugs.python.org (Jim Jewett)
Date: Fri, 20 Jan 2012 17:42:07 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAMpsgwZT5u82v54CmehuJU9jgTYpxRwEUw2ezyt8=un_Wmms6g@mail.gmail.com>
Message-ID: <CA+OGgf5h4vtYdeS3_ZxFea1_7Rj=DT0zZkfpHJQWH-hXTo1LWw@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Fri, Jan 20, 2012 at 7:58 AM, STINNER Victor
> If the hash output depends on an argument, the result cannot be
> cached.

They can still be cached in a separate dict based on id, rather than
string contents.

It may also be possible to cache them in the dict itself; for a
string-only dict, the hash of each entry is already cached on the
object, and the cache member of the entry is technically redundant.
Entering a key with the alternative hash can also switch the lookup
function to one that handles that possibility, just as entering a
non-string key currently does.

> It would require to add an
> optional argument to hash functions, or add a new function to some
> (or all?) builtin types.

For backports, the alternative hashing could be done privately within
dict and set, and would not require new slots on other types.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 18:54:57 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 20 Jan 2012 17:54:57 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327082097.67.0.361480921448.issue13806@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Do you think it would be possible to add a test for valid compressed data that used to trigger the check?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Fri Jan 20 19:11:35 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 20 Jan 2012 18:11:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf5h4vtYdeS3_ZxFea1_7Rj=DT0zZkfpHJQWH-hXTo1LWw@mail.gmail.com>
Message-ID: <CAMpsgwYnKXaeKBOevrGEjB_ecC774POuo0EHsg68Y=1krttAng@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I ran the test suite of Twisted 11.1 using a limit of 20 collisions:
there is no test failing because of hash collisions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 20 19:23:30 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Fri, 20 Jan 2012 18:23:30 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1327083810.49.0.159033265269.issue13760@psf.upfronthosting.co.za>


Faheem Mitha <faheem at faheem.info> added the comment:

Thanks for the quick attention to this, Lukasz. I'm just curious. Why do 3.2 and 3.3 already work? My understanding was that the basic exception issue in http://bugs.python.org/issue1692335 was still open.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Fri Jan 20 20:06:28 2012
From: report at bugs.python.org (Jean-Lou Dupont)
Date: Fri, 20 Jan 2012 19:06:28 +0000
Subject: [issue11076] Iterable argparse Namespace
In-Reply-To: <1296461103.49.0.397676179483.issue11076@psf.upfronthosting.co.za>
Message-ID: <1327086388.86.0.88567785237.issue11076@psf.upfronthosting.co.za>


Changes by Jean-Lou Dupont <jeanlou.dupont at gmail.com>:


----------
nosy: +Jean-Lou.Dupont

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11076>
_______________________________________

From report at bugs.python.org  Fri Jan 20 22:20:00 2012
From: report at bugs.python.org (Brett Cannon)
Date: Fri, 20 Jan 2012 21:20:00 +0000
Subject: [issue13829] exception error
In-Reply-To: <1326998522.6.0.790472668429.issue13829@psf.upfronthosting.co.za>
Message-ID: <1327094400.28.0.652845359723.issue13829@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

Can you isolate the cause? There is way too much in that core dump to try to debug the problem. Without knowing what code in this Moviegrabber app caused the bug we can't do anything to debug the issue.

----------
nosy: +brett.cannon
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Fri Jan 20 22:34:09 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 20 Jan 2012 21:34:09 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327095249.99.0.114263160056.issue13405@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Fri Jan 20 23:21:31 2012
From: report at bugs.python.org (Dan kamp)
Date: Fri, 20 Jan 2012 22:21:31 +0000
Subject: [issue13829] exception error
In-Reply-To: <1327094400.28.0.652845359723.issue13829@psf.upfronthosting.co.za>
Message-ID: <AA05FC64-D2E5-47F8-8432-3CC823AA2CE7@roontoon.com>


Dan kamp <bitbucket at roontoon.com> added the comment:

On Jan 20, 2012, at 4:20 PM, Brett Cannon wrote:

> Can you isolate the cause? There is way too much in that core dump to try to debug the problem. Without knowing what code in this Moviegrabber app caused the bug we can't do anything to debug the issue.

All I know is that this happens with Moviegrabber and no other app that I am running (5 of them). Attached is the code I am using. This is WAY above what I know how to do but I can tell you that I am running it on a mac with 2.7.1 installed and OSX lion. Attached is the source. Thanks

----------
status: pending -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Fri Jan 20 23:55:15 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Fri, 20 Jan 2012 22:55:15 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325854341.31.0.572411522028.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327100080.4992.149.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Fri, 2012-01-06 at 12:52 +0000, Marc-Andre Lemburg wrote:
> Marc-Andre Lemburg <mal at egenix.com> added the comment:
> 
> Demo patch implementing the collision limit idea for Python 2.7.
> 
> ----------
> Added file: http://bugs.python.org/file24151/hash-attack.patch
> 

Marc: is this the latest version of your patch?

Whether or not we go with collision counting and/or adding a random salt
to hashes and/or something else, I've had a go at updating your patch

Although debate on python-dev seems to have turned against the
collision-counting idea, based on flaws reported by Frank Sievertsen
http://mail.python.org/pipermail/python-dev/2012-January/115726.html
it seemed to me to be worth at least adding some test cases to flesh out
the approach.  Note that the test cases deliberately avoid containing
"hostile" data.

Am attaching an updated version which:
  * adds various FIXMEs (my patch isn't ready yet, but I wanted to get
more eyes on this)

  * introduces a new TooManyHashCollisions exception, and uses that
rather than KeyError (currently it extends BaseException; am not sure
where it should sit in the exception hierarchy).

  * adds debug text to the above exception, including the repr() and
hash of the key for which the issue was triggered:
  TooManyHashCollisions: 1001 hash collisions within dict at key
ChosenHash(999, 42) with hash 42

  * moves exception-setting to a helper function, to avoid duplicated
code

  * adds a sys.max_dict_collisions, though currently with just a
copy-and-paste of the 1000 value from dictobject.c

  * starts adding a test suite to test_dict.py, using a ChosenHash
helper class (to avoid having to publish hostile data), and a context
manager for ensuring the timings of various operations fall within sane
bounds, so I can do things like this:
        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
            for i in range(sys.max_dict_collisions -1 ):
                key = ChosenHash(i, 42)
                d[key] = 0

The test suite reproduces the TooManyHashCollisions response to a basic
DoS, and also "successfully" fails due to scenario 2 in Frank's email
above (assuming I understood his email correctly).

Presumably this could also incorporate a reproducer for scenario 1 in
this email, though I don't have one yet (but I don't want to make
hostile data public).

The patch doesn't yet do anything for sets.

Hope this is helpful
Dave

----------
Added file: http://bugs.python.org/file24286/hash-collision-counting-dmalcolm-2012-01-20-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 3be60a4c8c63 Include/pyerrors.h
--- a/Include/pyerrors.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/pyerrors.h	Fri Jan 20 17:51:33 2012 -0500
@@ -207,6 +207,8 @@
 PyAPI_DATA(PyObject *) PyExc_BytesWarning;
 PyAPI_DATA(PyObject *) PyExc_ResourceWarning;
 
+PyAPI_DATA(PyObject *) PyExc_TooManyHashCollisions;
+
 
 /* Convenience functions */
 
diff -r 3be60a4c8c63 Lib/test/test_dict.py
--- a/Lib/test/test_dict.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_dict.py	Fri Jan 20 17:51:33 2012 -0500
@@ -3,6 +3,8 @@
 
 import collections, random, string
 import gc, weakref
+import sys
+import time
 
 
 class DictTest(unittest.TestCase):
@@ -757,6 +759,192 @@
         self._tracked(MyDict())
 
 
+# Support classes for HashCollisionTests:
+class ChosenHash:
+    """
+    Use this to create arbitrary collections of keys that are non-equal
+    but have equal hashes, without needing to include hostile data
+    within the test suite.
+    """
+    def __init__(self, variability, hash):
+        self.variability = variability
+        self.hash = hash
+
+    def __eq__(self, other):
+        # The variability field is used to handle non-equalness:
+        return self.variability == other.variability
+
+    def __hash__(self):
+        return self.hash
+
+    def __repr__(self):
+        return 'ChosenHash(%r, %r)' % (self.variability,
+                                       self.hash)
+
+class Timer:
+    """
+    Simple way to measure time elapsed during a test case
+    """
+    def __init__(self):
+        self.starttime = time.time()
+
+    def get_elapsed_time(self):
+        """Get elapsed time in seconds as a float"""
+        curtime = time.time()
+        return curtime - self.starttime
+
+    def elapsed_time_as_str(self):
+        """Get elapsed time as a string (with units)"""
+        return '%0.3f seconds' % self.get_elapsed_time()
+
+class TookTooLong(RuntimeError):
+    def __init__(self, timelimit, elapsed, itercount=None):
+        self.timelimit = timelimit
+        self.elapsed = elapsed
+        self.itercount = itercount
+
+    def __str__(self):
+        result = 'took >= %s seconds' % self.timelimit
+        if self.itercount is not None:
+            result += (' (%0.3f seconds elapsed after %i iterations)'
+                       % (self.elapsed, self.itercount))
+        else:
+            result += ' (%0.3f seconds elapsed)' % self.elapsed
+        return result
+
+# Some of the tests involve constructing large dictionaries.  How big
+# should they be?
+ITEM_COUNT = 1000000
+
+# Arbitrary threshold (in seconds) for a "reasonable amount of time"
+# that it should take to work with ITEM_COUNT items:
+TIME_LIMIT = 5
+
+class _FasterThanContext(object):
+    """
+    A context manager for implementing assertFasterThan
+    """
+    def __init__(self, test_case, **kwargs):
+        self.test_case = test_case
+        if 'seconds' in kwargs:
+            self.timelimit = kwargs['seconds']
+        else:
+            raise ValueError()
+
+    def __enter__(self):
+        self.timer = Timer()
+        return self
+
+    def __exit__(self, exc_type, exc_value, tb):
+        if exc_type is not None:
+            # let unexpected exceptions pass through
+            return
+
+        if 1:
+            print('timer within %s took %s'
+                  % (self.test_case, self.timer.elapsed_time_as_str()))
+
+    def handle(self, callable_obj, args, kwargs):
+        """
+        If callable_obj is None, assertRaises/Warns is being used as a
+        context manager, so check for a 'msg' kwarg and return self.
+        If callable_obj is not None, call it passing args and kwargs.
+        """
+        if callable_obj is None:
+            self.msg = kwargs.pop('msg', None)
+            return self
+        with self:
+            callable_obj(*args, **kwargs)
+
+    def check_for_timeout(self, itercount):
+        """
+        Allow directly checking for timeouts from within a loop, supplying 
+        an iteration count.  If the timer has elapsed, this will raise a
+        TookTooLong exception, indicating how many iterations were completed
+        when the time limit was reached.  Otherwise, it does nothing.
+        """
+        elapsed_time = self.timer.get_elapsed_time()
+        if elapsed_time > self.timelimit:
+            raise TookTooLong(self.timelimit,
+                              elapsed_time,
+                              itercount)
+        
+ at support.cpython_only
+class HashCollisionTests(unittest.TestCase):
+    """
+    Issue 13703: tests about the behavior of dicts in the face of hostile data
+    """
+
+    def assertFasterThan(self, callableObj=None, *args, **kwargs):
+        context = _FasterThanContext(self, *args, **kwargs)
+        return context.handle(callableObj, args, kwargs)
+
+    def test_timings_with_benign_data(self):
+        # Verify that inserting many keys into a dict only takes a few seconds
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[i] = 0
+
+        # Verify that we can also retrieve the values quickly:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d[i]
+
+        # Verify that we can quickly insert the same item many times
+        # (overwriting each time):
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[0] = 0
+
+    def test_not_reaching_limit(self):
+        # Ensure that we can insert equal-hashed keys up to (but not reaching)
+        # the collision climit:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d = dict()
+            for i in range(sys.max_dict_collisions - 1):
+                key = ChosenHash(i, 42)
+                d[key] = 0
+            
+    def test_reaching_collision_limit(self):
+        """
+        Ensure that too many non-equal keys with the same hash lead to a
+        TooManyCollisions exception
+        """
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            with self.assertRaisesRegex(TooManyHashCollisions,
+                                        ('1001 hash collisions within dict at'
+                                         ' key ChosenHash\(999, 42\)'
+                                         ' with hash 42')):
+                d = dict()
+                for i in range(sys.max_dict_collisions):
+                    key = ChosenHash(i, 42)
+                    d[key] = 0
+
+    # Frank Sievertsen found scenarios in which the collision-counting
+    # scheme could be attacked:
+    #   http://mail.python.org/pipermail/python-dev/2012-January/115726.html
+
+    def test_scenario_b_from_Frank_Sievertsen(self):
+        d = dict()
+
+        # Insert hash collisions up to (but not reaching) the limit:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(sys.max_dict_collisions -1 ):
+                key = ChosenHash(i, 42)
+                d[key] = 0
+
+        # Now try to add many equal values that collide
+        # with the hash, and see how long it takes
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                key = ChosenHash(0, 42)
+                d[key] = 0
+                cm.check_for_timeout(i)
+        # FIXME: currently this reproduces the 2nd issue described in his
+        # post, by failing for me here ^^^ with a message like this:
+        #   test.test_dict.TookTooLong: took >= 5 seconds (5.000 seconds elapsed after 18838 iterations)
+
 from test import mapping_tests
 
 class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol):
@@ -771,6 +959,7 @@
 def test_main():
     support.run_unittest(
         DictTest,
+        HashCollisionTests,
         GeneralMappingTests,
         SubclassMappingTests,
     )
diff -r 3be60a4c8c63 Objects/dictobject.c
--- a/Objects/dictobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/dictobject.c	Fri Jan 20 17:51:33 2012 -0500
@@ -10,6 +10,8 @@
 #include "Python.h"
 #include "stringlib/eq.h"
 
+/* Maximum number of allowed hash collisions. */
+#define Py_MAX_DICT_COLLISIONS 1000
 
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
@@ -25,6 +27,21 @@
     Py_DECREF(tup);
 }
 
+/* Set a TooManyHashCollisions error */
+static void
+set_too_many_collisions_error(size_t collisions, PyObject *key, Py_hash_t hash)
+{
+  PyErr_Format(PyExc_TooManyHashCollisions,
+                 ("%i hash collisions within dict"
+                  " at key %R with hash %zd"),
+                 collisions, key, hash);
+    /* Backporting notes: (FIXME)
+       %R is a Python 3-ism
+       %zd is for Py_ssize_t, which in Python 3 is the same as Py_hash_t
+    */
+}
+
+
 /* Define this out if you don't want conversion statistics on exit. */
 #undef SHOW_CONVERSION_COUNTS
 
@@ -326,6 +343,7 @@
     register PyDictEntry *ep;
     register int cmp;
     PyObject *startkey;
+    size_t collisions;
 
     i = (size_t)hash & mask;
     ep = &ep0[i];
@@ -360,6 +378,7 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
@@ -386,6 +405,10 @@
                  */
                 return lookdict(mp, key, hash);
             }
+	    if (++collisions > Py_MAX_DICT_COLLISIONS) {
+                set_too_many_collisions_error(collisions, key, hash);
+                return NULL;
+	    }
         }
         else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
@@ -413,6 +436,7 @@
     register size_t mask = (size_t)mp->ma_mask;
     PyDictEntry *ep0 = mp->ma_table;
     register PyDictEntry *ep;
+    size_t collisions;
 
     /* Make sure this function doesn't have to handle non-unicode keys,
        including subclasses of str; e.g., one reason to subclass
@@ -439,6 +463,7 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
@@ -451,6 +476,10 @@
             return ep;
         if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+	if (++collisions > Py_MAX_DICT_COLLISIONS) {
+            set_too_many_collisions_error(collisions, key, hash);
+	    return NULL;
+	}
     }
     assert(0);          /* NOT REACHED */
     return 0;
diff -r 3be60a4c8c63 Objects/exceptions.c
--- a/Objects/exceptions.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/exceptions.c	Fri Jan 20 17:51:33 2012 -0500
@@ -2205,6 +2205,12 @@
 SimpleExtendsException(PyExc_Warning, ResourceWarning,
     "Base class for warnings about resource usage.");
 
+/*
+ *    TooManyHashCollisions extends BaseException
+ */
+SimpleExtendsException(PyExc_BaseException, TooManyHashCollisions,
+    "Base class for warnings about computationally-infeasible data.");
+
 
 
 /* Pre-computed RuntimeError instance for when recursion depth is reached.
@@ -2318,6 +2324,7 @@
     PRE_INIT(UnicodeWarning)
     PRE_INIT(BytesWarning)
     PRE_INIT(ResourceWarning)
+    PRE_INIT(TooManyHashCollisions)
 
     /* OSError subclasses */
     PRE_INIT(ConnectionError);
@@ -2399,6 +2406,7 @@
     POST_INIT(UnicodeWarning)
     POST_INIT(BytesWarning)
     POST_INIT(ResourceWarning)
+    POST_INIT(TooManyHashCollisions)
 
     if (!errnomap) {
         errnomap = PyDict_New();
diff -r 3be60a4c8c63 Python/sysmodule.c
--- a/Python/sysmodule.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Python/sysmodule.c	Fri Jan 20 17:51:33 2012 -0500
@@ -1619,6 +1619,10 @@
     SET_SYS_FROM_STRING("thread_info", PyThread_GetInfo());
 #endif
 
+    SET_SYS_FROM_STRING("max_dict_collisions",
+                        PyLong_FromLong(1000)); // FIXME
+
+
 #undef SET_SYS_FROM_STRING
     if (PyErr_Occurred())
         return NULL;

From report at bugs.python.org  Sat Jan 21 00:18:29 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 20 Jan 2012 23:18:29 +0000
Subject: [issue13782] xml.etree.ElementTree: Element.append doesn't type-check
	its argument
In-Reply-To: <1326502453.32.0.206779556991.issue13782@psf.upfronthosting.co.za>
Message-ID: <1327101509.93.0.722164403262.issue13782@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Confirmed with on 3.2.2 Win7 for all three methods.
Docs say .append should raise AssertionError (see below).
However, that is unusual and TypeError is normal practice.
I am not sure what to do.

Our docs say: '''
append(subelement) 
Adds the element subelement to the end of this elements internal list of subelements.

extend(subelements) 
Appends subelements from a sequence object with zero or more elements. Raises AssertionError if a subelement is not a valid object.

insert(index, element) 
Inserts a subelement at the given position in this element.
'''

This implies to me that append and insert should do the same.
In fact, the current code has the assertion checks but they are commented out in all methods.

 From
http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm
"append ... Raises AssertionError:
    If a sequence member is not a valid object."
Ditto for .insert

Florent: why are all the assertion checks commented out in
Changeset: 59511 (831e30c0fd73) Issue #6472:
The xml.etree package is updated to ElementTree 1.3.
(March 2010)

Is the change in Fredrik's code, in spite of his docs?

----------
nosy: +effbot, flox, terry.reedy
stage:  -> test needed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13782>
_______________________________________

From report at bugs.python.org  Sat Jan 21 00:44:42 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 20 Jan 2012 23:44:42 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327103082.72.0.839598267156.issue13790@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Doc patch attached to make sure correct. Should {} be quoted?

Eric, do you want to close off the idea of changing :d errors, or switch back after the doc fix?

----------
keywords: +patch
nosy: +terry.reedy
stage:  -> test needed
Added file: http://bugs.python.org/file24287/i13790.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 01:03:43 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 21 Jan 2012 00:03:43 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1321299726.66.0.343368151185.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327104223.53.0.217811935108.issue13405@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

My point about DTrace probes is not performance per-se but observability.

An OS with DTrace support can be completely inspected and shows you ALL the inner working. You can see a particular pattern in the operating system and locate the syscalls and the process involved. If it is a compiled program, you can actually look inside it and locate the particular function responsible and, even, its arguments.

You can not do that easily with an interpreted program, because when you inspect the stacktrace, you see the C interpreter functions, not the actual interpreted functions running. So, to be appropriately integrated with the impressive observability of the entire system, the interpreter must cooperate. That is the role of this patch. And that role is crucial.

Just an example I was checking today, just for fun: an email server written in Python.

I am interested in knowing which function is involved in database "misses", and so real READ harddisk activity. Since Berkeley DB is used as the backend, and Oracle has included dtrace probes on the library (http://docs.oracle.com/cd/E17076_02/html/programmer_reference/program_perfmon.html), we can do something like this:

"""
dtrace -n "bdb1066:::mpool-miss {@misses[jstack(20)]=count();}"
"""

This on-liner will trace process "1066" and with plug into the Berkeley DB cache miss event, recording the python stack (up to 20 levels) at each event fire. We let the monitor script go for a few seconds and ^c it, showing something like:

"""
dtrace: description 'bdb1066:::mpool-miss ' matched 1 probe
^C


              libdb-5.3.so`__memp_fget+0x22cb
              libdb-5.3.so`__ham_get_cpage+0x140
              libdb-5.3.so`__ham_lookup+0x8b
              libdb-5.3.so`__hamc_get+0x3d5
              libdb-5.3.so`__dbc_iget+0x40d
              libdb-5.3.so`__dbc_get+0x9d
              libdb-5.3.so`__db_get+0xb8
              libdb-5.3.so`__db_get_pp+0x2db
              _pybsddb.so`DB_get+0x1f5
              libpython2.7.so.1.0`PyCFunction_Call+0x148
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x55f2
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:317 (_prefetch_thread) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x18f
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x1d8e
                [ /usr/local/lib/python2.7/threading.py:505 (run) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ /usr/local/lib/python2.7/threading.py:552 (__bootstrap_inner) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ /usr/local/lib/python2.7/threading.py:525 (__bootstrap) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x9a
              libpython2.7.so.1.0`PyObject_Call+0x5c
                1

              libdb-5.3.so`__memp_fget+0x22cb
              libdb-5.3.so`__ham_get_cpage+0x140
              libdb-5.3.so`__ham_lookup+0x8b
              libdb-5.3.so`__hamc_get+0x3d5
              libdb-5.3.so`__dbc_iget+0x40d
              libdb-5.3.so`__dbc_get+0x9d
              libdb-5.3.so`__db_get+0xb8
              libdb-5.3.so`__db_get_pp+0x2db
              _pybsddb.so`DB_get+0x1f5
              libpython2.7.so.1.0`PyCFunction_Call+0x148
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x55f2
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:725 (load) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x18f
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x1d8e
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:430 (f) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x4f89
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:302 (_send_load_response) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:295 (handle_L) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:261 (handle) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:227 (serve) ]
                2

              libdb-5.3.so`__memp_fget+0x22cb
              libdb-5.3.so`__ham_get_cpage+0x140
              libdb-5.3.so`__ham_lookup+0x8b
              libdb-5.3.so`__hamc_get+0x3d5
              libdb-5.3.so`__dbc_iget+0x40d
              libdb-5.3.so`__dbc_get+0x9d
              libdb-5.3.so`__db_get+0xb8
              libdb-5.3.so`__db_get_pp+0x2db
              _pybsddb.so`DB_get+0x1f5
              libpython2.7.so.1.0`PyCFunction_Call+0x148
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x55f2
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:307 (_prefetch_thread) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x18f
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x1d8e
                [ /usr/local/lib/python2.7/threading.py:505 (run) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ /usr/local/lib/python2.7/threading.py:552 (__bootstrap_inner) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ /usr/local/lib/python2.7/threading.py:525 (__bootstrap) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x9a
              libpython2.7.so.1.0`PyObject_Call+0x5c
               25

              libdb-5.3.so`__memp_fget+0x22cb
              libdb-5.3.so`__ham_get_cpage+0x140
              libdb-5.3.so`__ham_lookup+0x8b
              libdb-5.3.so`__hamc_get+0x3d5
              libdb-5.3.so`__dbc_iget+0x40d
              libdb-5.3.so`__dbc_get+0x9d
              libdb-5.3.so`__db_get+0xb8
              libdb-5.3.so`__db_get_pp+0x2db
              _pybsddb.so`DB_get+0x1f5
              libpython2.7.so.1.0`PyCFunction_Call+0x148
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x53ee
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:758 (end) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`function_call+0x18f
              libpython2.7.so.1.0`PyObject_Call+0x5c
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x1d8e
                [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:444 (f) ]
              libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7a4
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x4f89
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:354 (handle_C) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:261 (handle) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ build/bdist.solaris-2.10-i86pc/egg/durus/storage_server.py:227 (serve) ]
              libpython2.7.so.1.0`PyEval_EvalFrameEx+0x63ed
                [ /export/home/correo/lmtp.py:1206 (storage_background) ]
               72
"""

We can see which function (even which thread) is causing the database miss. The number after each frame shows how many times we saw that particular stacktrace.

Let's suppose now I am interested in knowing how long SMTP transactions are. Let's do something like this:

"""
python$target:::function-entry
/copyinstr(arg1)=="lmtp2"/
{
  self->ts = timestamp;
}

python$target:::function-return
/(copyinstr(arg1)=="lmtp2") && (self->ts)/
{
  @time_lmtp = quantize(timestamp-self->ts);
  self->ts = 0;
}

"""

And the result after a few seconds: (time in nanoseconds)

"""
           value  ------------- Distribution ------------- count    
         8388608 |                                         0        
        16777216 |@@@@@@@@@@@@@@@@@                        8        
        33554432 |@@@@@@@@@@@@@@@@@@@@@@@                  11       
        67108864 |                                         0 
"""

(The values are pessimistic, because the activated probes have performance hit).

You can do things like monitor garbage collection events, count object allocation/deallocation to find memory leaks, relate network activity with disk activity, whatever you can imagine. And not only inside the program, but connect any activity in the system with the root cause.

Another example. In this particular program the database runs in a separate thread inside the same process, connected to the other threads via a socket. Sometimes the socket can timeout if the database is being slow. In that case, the thread with the timeout will close the connection to the database and will open a new connection, redoing the transaction (the system is ACID, so we can do this). How often is that happening?. No idea. I could change python code to log reconnections or I could run an one-liner dtrace script on my server hooking to the reconnection routine to get an inmediate feedback of how frequent is this event firing.

Note that during all this detailed monitoring, the email server keeps working. We don't stop it. We don't change a single line of production code. Any effect the probes could have (performance, for instance) will vanish as soon as the monitoring script is stopped.

And yes, you can do this, frequently, stopping the process and instrumentalizing it by hand. But that disturb the service, it is error prone and there are things you simply can't do because your are instrumentalizing your code, not the C libraries neither the entire operating system. For instance, you can't easily correlate events between two different processes. With DTrace, you can. 

Moreover, this kind of investigation is iterative. You make an hypothesis, write an small script to validate, run it and collect some data, and write another script to explore what you found. If you do that modifying the original sourcecode, you must patch and restart the process constantly. And your patch could crash the server, for instance.

DTrace is very useful too when a process has sporadic problems and you can't just stop it because the problem goes away if you do, and you can't easily find the reason.

So, yes, usually you are interested in performance, but that is a consequence of "visibility" into your program and the entire system. The real deal is the "observability".

About DTrace being a niche, current code works in Solaris and derivatives. It is a quite popular OS in servers. Beside, I plan to support FreeBSD and MacOS too. I think this could be easy (although MacOS lacks the stack dump feature, shame on you, Apple :). Moreover, there are interest in using this with Linux systemtap capabilities, via the stap/DTrace compatibility layer. That seems to be doable too.

You could say "OK, come back when you have ALL that support". Previous attempts failed because of this, to try to cover ALL bases. I am quite sure that if we commit current code and enable it in the current Solaris & OpenIndiana buildbots, other people is going to "kill" :-) to get support in FreeBSD and MacOS too, and Fedora folks -for instance- sure would provide systemtap.

About the maintenance burden, I am a core developer and quite committed to Solaris&family technology. I plan to be around for quite a while, actually. But this technology can be deleted COMPLETELY from current code simply deleting the conditional compilation (there is no 'convenient' code that is not conditionally compiled). You could drop the support anytime, since it would affect no python program at all. Python programs don't know anything about dtracing. DTrace is used by external tools.

There is a point about "let's keep this patch as an external patch that interested parties can apply by themselves". Beside considering that this work deserves to be in mainline python, a "patch" doesn't have buildbot testing and most users are stuck with whatever precompiled python their distributions choose to adopt. Even if linux distributions are interested, an external patch doesn't have the same credibility.

I have the python2.7 patch in production in my servers for the last month or so. 99.99% of the time I don't use it, and the performance hit I am paying for having this patch compiled in (it is completely optional via "configure") is about 1.4%. The other 0.01% of the time I really need it. REALLY. So badly that even decide what operating system I am using, or push me to work hard to create a patch useful to others.

So, yes. The code is intrusive. The code deals with a lot of internal machinery (PEP393 support in the ustack helper was quite difficult). It is going to break from time to time, sure. At the same time, I am committed to support it. And even if it is dropped in 3.4, no Python program will be affected.

Last but not least, other interpreted languages already have DTrace support:

* PHP:

 - http://techportal.ibuildings.com/2009/11/24/analysing-php-web-applications-with-dtrace/
 - https://wiki.php.net/rfc/dtrace

* PERL:

 - https://dgl.cx/2011/01/dtrace-and-perl

* RUBY:

 - http://ruby-dtrace.rubyforge.org/

 - https://rubygems.org/gems/ruby-dtrace

* HASKELL:

 - http://hackage.haskell.org/trac/ghc/wiki/DTrace

 - http://justtesting.org/profiling-garbage-collection-in-haskell-with

* JAVA:

 - http://docs.oracle.com/javase/6/docs/technotes/guides/vm/dtrace.html

* JAVASCRIPT:

 - https://wiki.mozilla.org/Performance/Optimizing_JavaScript_with_DTrace

Googling around you can see dtrace probes everywhere, from languages to libraries to, even, the X-Server: http://www.x.org/releases/X11R7.6/doc/xorg-server/Xserver-DTrace.html

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Sat Jan 21 01:16:25 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sat, 21 Jan 2012 00:16:25 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327104985.35.0.90065015845.issue13790@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

I don't think "{}" is the correct way to document this. These all have an empty format specifier:

"{}".format(foo)
"{:}".format(foo)
"{0}".format(foo)
"{0:}".format(foo)
"{name}".format(name=foo)
format(foo, "")
format(foo)

That is, they all call foo.__format__(""). If foo.__format__ (well, really type(foo).__format__) doesn't exist, then object.__format__(foo, "") gets called. It's object.__format__ that's checking for the empty format string, and if so it returns str(foo).

What would you suggest changing the ':d' error message to, for objects that don't support a format type of 'd'? This makes sense to me:

>>> format('', 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

The problem, if there is one, is:
>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

The problem is that the str that's producing this error doesn't know that it exists because object.__format__ returned str([]).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 03:29:14 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 02:29:14 +0000
Subject: [issue13796] use 'text=...' to define the text attribute of and
	xml.etree.ElementTree.Element
In-Reply-To: <1326698487.2.0.0622950382199.issue13796@psf.upfronthosting.co.za>
Message-ID: <1327112954.74.0.796654122079.issue13796@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Pedro and Patrick: if you are going to respond by email rather than by typing in the browser box, PLEASE delete the quoted text. Or on occasion, quote a line or two if really necessary, as when responding to a previous message other than the last one. Each message is assumed to be a response to the one immediately preceding, and that message is visible just above yours. This is not like a mail list where previous messages may have come and gone.

In any case, etree.ElementTree is documented as being a stable version F. Lundh's package and I am pretty sure that we are not going to change that by adding things on our own. So continue using your customized subclass.

"See http://effbot.org/zone/element-index.htm for tutorials and links to other docs. Fredrik Lundh?s page is also the location of the development version of the xml.etree.ElementTree.

Changed in version 3.2: The ElementTree API is updated to 1.3. For more information, see Introducing ElementTree 1.3."

----------
nosy: +terry.reedy
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13796>
_______________________________________

From report at bugs.python.org  Sat Jan 21 03:32:48 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 02:32:48 +0000
Subject: [issue13798] Pasting and then running code doesn't work in the IDLE
	Shell
In-Reply-To: <1326717884.33.0.870144380911.issue13798@psf.upfronthosting.co.za>
Message-ID: <1327113168.63.0.245570695556.issue13798@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
resolution:  -> duplicate
status: open -> closed
superseder:  -> Pasted \n not same as typed \n

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13798>
_______________________________________

From report at bugs.python.org  Sat Jan 21 03:34:10 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 02:34:10 +0000
Subject: [issue13799] Base 16 should be hexadecimal in Unicode HOWTO
In-Reply-To: <1326718246.49.0.980962494093.issue13799@psf.upfronthosting.co.za>
Message-ID: <1327113250.09.0.0466778333393.issue13799@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13799>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:10:55 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 03:10:55 +0000
Subject: [issue13804] Python library structure creates hard to read code when
	using higher order functions
In-Reply-To: <1326793717.4.0.393580852991.issue13804@psf.upfronthosting.co.za>
Message-ID: <1327115455.85.0.854154712781.issue13804@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I am closing this because map has even less chance of being made a collection method than join. Unlike join, map takes any positive number of iterables as args, not just one. 'Iterables' includes iterators, which intentionally need have no methods other than __iter__ and __next__.

If map were an attribute, it should be an attribute of 'function' (except that there is no one function class).

To abstract attributes, use get/setattr. Untested example: 

def atcat(self, src,  src_at, dst):
  res = []
  for col in getattr(self, src):
    res += getattr(col, src_at)
  setattr(self, dst, res)

----------
nosy: +terry.reedy
resolution:  -> rejected
status: open -> closed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13804>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:14:17 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 03:14:17 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327115657.29.0.612103645683.issue13812@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +jnoller
stage:  -> test needed
versions:  -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:16:25 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 21 Jan 2012 03:16:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327100080.4992.149.camel@surprise>
Message-ID: <1327115744.4992.180.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Fri, 2012-01-20 at 22:55 +0000, Dave Malcolm wrote:
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
> 
> On Fri, 2012-01-06 at 12:52 +0000, Marc-Andre Lemburg wrote:
> > Marc-Andre Lemburg <mal at egenix.com> added the comment:
> > 
> > Demo patch implementing the collision limit idea for Python 2.7.
> > 
> > ----------
> > Added file: http://bugs.python.org/file24151/hash-attack.patch
> > 
> 
> Marc: is this the latest version of your patch?
> 
> Whether or not we go with collision counting and/or adding a random salt
> to hashes and/or something else, I've had a go at updating your patch
> 
> Although debate on python-dev seems to have turned against the
> collision-counting idea, based on flaws reported by Frank Sievertsen
> http://mail.python.org/pipermail/python-dev/2012-January/115726.html
> it seemed to me to be worth at least adding some test cases to flesh out
> the approach.  Note that the test cases deliberately avoid containing
> "hostile" data.

I had a brainstorm, and I don't yet know if the following makes sense,
but here's a crude patch with another approach, which might get around
the issues Frank raises.

Rather than count the number of equal-hash collisions within each call
to lookdict, instead keep a per-dict count of the total number of
iterations through the probe sequence (regardless of the hashing),
amortized across all calls to lookdict, and if it looks like we're going
O(n^2) rather than O(n), raise an exception.  Actually, that's not quite
it, but see below...

We potentially have 24 words of per-dictionary storage hiding in the
ma_smalltable area within PyDictObject, which we can use when ma_mask >=
PyDict_MINSIZE (when mp->ma_table != mp->ma_smalltable), without
changing sizeof(PyDictObject) and thus breaking ABI.  I hope there isn't
any code out there that uses this space.  (Anyone know of any?)

This very crude patch uses that area to add per-dict tracking of the
total number of iterations spent probing for a free PyDictEntry whilst
constructing the dictionary.  It rules that if we've gone more than (32
* ma_used) iterations whilst constructing the dictionary (counted across
all ma_lookup calls), then we're degenerating into O(n^2) behavior, and
this triggers an exception.  Any other usage of ma_lookup resets the
count (e.g. when reading values back).  I picked the scaling factor of
32 from out of the air; I hope there's a smarter threshold.  

I'm assuming that an attack scenario tends to involve a dictionary that
goes through a construction phase (which the attacker is aiming to
change from O(N) to O(N^2)), and then a usage phase, whereas there are
other patterns of dictionary usage in which insertion and lookup are
intermingled for which this approach wouldn't raise an exception.

This leads to exceptions like this:

AlgorithmicComplexityError: dict construction used 4951 probes for 99
entries at key 99 with hash 42

(i.e. the act of constructing a dict with 99 entries required traversing
4951 PyDictEntry slots, suggesting someone is sending deliberately
awkward data).

Seems to successfully handle both the original DoS and the second
scenario in Frank's email.  I don't have a reproducer for the first of
Frank's scenarios, but in theory it ought to handle it.  (I hope!)

Have seen two failures within python test suite from this, which I hope
can be fixed by tuning the thresholds and the reset events (they seem to
happen when a large dict is emptied).

May have a performance impact, but I didn't make any attempt to optimize
it (beyond picking a power of two for the scaling factor).

(There may be random bits of the old patch thrown in; sorry)

Thoughts? (apart from "ugh! it's ugly!" yes I know - it's late here)
Dave

----------
Added file: http://bugs.python.org/file24288/amortized-probe-counting-dmalcolm-2012-01-20-002.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 3be60a4c8c63 Include/pyerrors.h
--- a/Include/pyerrors.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/pyerrors.h	Fri Jan 20 22:11:43 2012 -0500
@@ -207,6 +207,8 @@
 PyAPI_DATA(PyObject *) PyExc_BytesWarning;
 PyAPI_DATA(PyObject *) PyExc_ResourceWarning;
 
+PyAPI_DATA(PyObject *) PyExc_AlgorithmicComplexityError;
+
 
 /* Convenience functions */
 
diff -r 3be60a4c8c63 Lib/test/test_dict.py
--- a/Lib/test/test_dict.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_dict.py	Fri Jan 20 22:11:43 2012 -0500
@@ -3,6 +3,8 @@
 
 import collections, random, string
 import gc, weakref
+import sys
+import time
 
 
 class DictTest(unittest.TestCase):
@@ -757,6 +759,196 @@
         self._tracked(MyDict())
 
 
+# Support classes for HashCollisionTests:
+class ChosenHash:
+    """
+    Use this to create arbitrary collections of keys that are non-equal
+    but have equal hashes, without needing to include hostile data
+    within the test suite.
+    """
+    def __init__(self, variability, hash):
+        self.variability = variability
+        self.hash = hash
+
+    def __eq__(self, other):
+        # The variability field is used to handle non-equalness:
+        return self.variability == other.variability
+
+    def __hash__(self):
+        return self.hash
+
+    def __repr__(self):
+        return 'ChosenHash(%r, %r)' % (self.variability,
+                                       self.hash)
+
+class Timer:
+    """
+    Simple way to measure time elapsed during a test case
+    """
+    def __init__(self):
+        self.starttime = time.time()
+
+    def get_elapsed_time(self):
+        """Get elapsed time in seconds as a float"""
+        curtime = time.time()
+        return curtime - self.starttime
+
+    def elapsed_time_as_str(self):
+        """Get elapsed time as a string (with units)"""
+        return '%0.3f seconds' % self.get_elapsed_time()
+
+class TookTooLong(RuntimeError):
+    def __init__(self, timelimit, elapsed, itercount=None):
+        self.timelimit = timelimit
+        self.elapsed = elapsed
+        self.itercount = itercount
+
+    def __str__(self):
+        result = 'took >= %s seconds' % self.timelimit
+        if self.itercount is not None:
+            result += (' (%0.3f seconds elapsed after %i iterations)'
+                       % (self.elapsed, self.itercount))
+        else:
+            result += ' (%0.3f seconds elapsed)' % self.elapsed
+        return result
+
+# Some of the tests involve constructing large dictionaries.  How big
+# should they be?
+ITEM_COUNT = 1000000
+
+# Arbitrary threshold (in seconds) for a "reasonable amount of time"
+# that it should take to work with ITEM_COUNT items:
+TIME_LIMIT = 5
+
+class _FasterThanContext(object):
+    """
+    A context manager for implementing assertFasterThan
+    """
+    def __init__(self, test_case, **kwargs):
+        self.test_case = test_case
+        if 'seconds' in kwargs:
+            self.timelimit = kwargs['seconds']
+        else:
+            raise ValueError()
+
+    def __enter__(self):
+        self.timer = Timer()
+        return self
+
+    def __exit__(self, exc_type, exc_value, tb):
+        if exc_type is not None:
+            # let unexpected exceptions pass through
+            return
+
+        if 1:
+            print('timer within %s took %s'
+                  % (self.test_case, self.timer.elapsed_time_as_str()))
+
+    def handle(self, callable_obj, args, kwargs):
+        """
+        If callable_obj is None, assertRaises/Warns is being used as a
+        context manager, so check for a 'msg' kwarg and return self.
+        If callable_obj is not None, call it passing args and kwargs.
+        """
+        if callable_obj is None:
+            self.msg = kwargs.pop('msg', None)
+            return self
+        with self:
+            callable_obj(*args, **kwargs)
+
+    def check_for_timeout(self, itercount):
+        """
+        Allow directly checking for timeouts from within a loop, supplying 
+        an iteration count.  If the timer has elapsed, this will raise a
+        TookTooLong exception, indicating how many iterations were completed
+        when the time limit was reached.  Otherwise, it does nothing.
+        """
+        elapsed_time = self.timer.get_elapsed_time()
+        if elapsed_time > self.timelimit:
+            raise TookTooLong(self.timelimit,
+                              elapsed_time,
+                              itercount)
+        
+ at support.cpython_only
+class HashCollisionTests(unittest.TestCase):
+    """
+    Issue 13703: tests about the behavior of dicts in the face of hostile data
+    """
+
+    def assertFasterThan(self, callableObj=None, *args, **kwargs):
+        context = _FasterThanContext(self, *args, **kwargs)
+        return context.handle(callableObj, args, kwargs)
+
+    def test_timings_with_benign_data(self):
+        # Verify that inserting many keys into a dict only takes a few seconds
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[i] = 0
+
+        # Verify that we can also retrieve the values quickly:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d[i]
+
+        # Verify that we can quickly insert the same item many times
+        # (overwriting each time):
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[0] = 0
+
+    def test_not_reaching_limit(self):
+        # Ensure that we can insert equal-hashed keys up to (but not reaching)
+        # the collision climit:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d = dict()
+            for i in range(50):
+                key = ChosenHash(i, 42)
+                d[key] = 0
+            
+    def test_reaching_collision_limit(self):
+        """
+        Ensure that too many non-equal keys with the same hash lead to a
+        TooManyCollisions exception
+        """
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            with self.assertRaisesRegex(AlgorithmicComplexityError,
+                                        ('dict construction used 2049 probes'
+                                         ' for 64 entries'
+                                         ' at key ChosenHash\(64, 42\)'
+                                         ' with hash 42')):
+                d = dict()
+                for i in range(1000):
+                    key = ChosenHash(i, 42)
+                    d[key] = 0
+
+    # Frank Sievertsen found scenarios in which the collision-counting
+    # scheme could be attacked:
+    #   http://mail.python.org/pipermail/python-dev/2012-January/115726.html
+
+    def test_scenario_b_from_Frank_Sievertsen(self):
+        d = dict()
+
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            with self.assertRaisesRegex(AlgorithmicComplexityError,
+                                        ('dict construction used 2049 probes'
+                                         ' for 64 entries'
+                                         ' at key ChosenHash\(64, 42\)'
+                                         ' with hash 42')):
+                # Insert hash collisions up to (but not reaching) the proposed
+                # limit:
+                for i in range(1000):
+                    key = ChosenHash(i, 42)
+                    d[key] = 0
+                    cm.check_for_timeout(i)
+
+                # Now try to add many equal values that collide
+                # with the hash, and see how long it takes
+                for i in range(ITEM_COUNT):
+                    key = ChosenHash(0, 42)
+                    d[key] = 0
+                    cm.check_for_timeout(i)
+
 from test import mapping_tests
 
 class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol):
@@ -771,6 +963,7 @@
 def test_main():
     support.run_unittest(
         DictTest,
+        HashCollisionTests,
         GeneralMappingTests,
         SubclassMappingTests,
     )
diff -r 3be60a4c8c63 Objects/dictobject.c
--- a/Objects/dictobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/dictobject.c	Fri Jan 20 22:11:43 2012 -0500
@@ -10,6 +10,29 @@
 #include "Python.h"
 #include "stringlib/eq.h"
 
+#define Py_MAX_AVERAGE_PROBES_PER_INSERT 32
+/* power-of-two to make multiplication fast */
+
+/* For large dictionaries, reuse the space allocated for ma_smalltable */
+typedef struct _Py_LargeDictFields {
+    size_t iter_count;
+} _Py_LargeDictFields;
+
+#define PyDict_LARGEDICTFIELDS(mp) \
+  ((_Py_LargeDictFields*)&(mp)->ma_smalltable)
+/* FIXME: add assert(mp->ma_table != mp->ma_smalltable) */
+
+static int is_inserting = 0;
+
+static void reset_iter_count(PyDictObject *mp)
+{
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        assert(mp->ma_table != mp->ma_smalltable);
+        PyDict_LARGEDICTFIELDS(mp)->iter_count = 0;
+    }
+}
+
+
 
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
@@ -25,6 +48,22 @@
     Py_DECREF(tup);
 }
 
+/* Set a AlgorithmicComplexityError error */
+static void
+set_algorithmic_complexity_error(PyDictObject *mp, PyObject *key, Py_hash_t hash)
+{
+    PyErr_Format(PyExc_AlgorithmicComplexityError,
+                 ("dict construction used %i probes for %i entries"
+                  " at key %R with hash %zd"),
+                 PyDict_LARGEDICTFIELDS(mp)->iter_count, mp->ma_used,
+                 key, hash);
+    /* Backporting notes: (FIXME)
+       %R is a Python 3-ism
+       %zd is for Py_ssize_t, which in Python 3 is the same as Py_hash_t
+    */
+}
+
+
 /* Define this out if you don't want conversion statistics on exit. */
 #undef SHOW_CONVERSION_COUNTS
 
@@ -358,6 +397,11 @@
         freeslot = NULL;
     }
 
+    /* This is a non-insertion probe: reset the cost-of-insertion count: */
+    if (!is_inserting) {
+        reset_iter_count(mp);
+    }
+
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
@@ -389,6 +433,15 @@
         }
         else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+
+        if (is_inserting && mask >= PyDict_MINSIZE) {
+            assert(mp->ma_table != mp->ma_smalltable);
+            if (++PyDict_LARGEDICTFIELDS(mp)->iter_count
+                > (mp->ma_used * Py_MAX_AVERAGE_PROBES_PER_INSERT) ) {
+                set_algorithmic_complexity_error(mp, key, hash);
+                return NULL;
+            }
+        }
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -437,6 +490,11 @@
         freeslot = NULL;
     }
 
+    /* This is a non-insertion probe: reset the cost-of-insertion count: */
+    if (!is_inserting) {
+        reset_iter_count(mp);
+    }
+
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
@@ -451,6 +509,14 @@
             return ep;
         if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+        if (is_inserting && mask >= PyDict_MINSIZE) {
+            assert(mp->ma_table != mp->ma_smalltable);
+            if (++PyDict_LARGEDICTFIELDS(mp)->iter_count
+                  > (mp->ma_used * Py_MAX_AVERAGE_PROBES_PER_INSERT) ) {
+                set_algorithmic_complexity_error(mp, key, hash);
+                return NULL;
+            }
+	}
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -532,7 +598,9 @@
     typedef PyDictEntry *(*lookupfunc)(PyDictObject *, PyObject *, Py_hash_t);
 
     assert(mp->ma_lookup != NULL);
+    is_inserting = 1;
     ep = mp->ma_lookup(mp, key, hash);
+    is_inserting = 0;
     if (ep == NULL) {
         Py_DECREF(key);
         Py_DECREF(value);
@@ -675,8 +743,15 @@
         /* else key == value == NULL:  nothing to do */
     }
 
-    if (is_oldtable_malloced)
+    if (is_oldtable_malloced) {
         PyMem_DEL(oldtable);
+    } else {
+        if (mp->ma_table != mp->ma_smalltable) {
+            /* clean ma_smalltable for use as _Py_LargeDictFields: */
+            memset(mp->ma_smalltable, 0, sizeof(mp->ma_smalltable));
+        }
+    }
+
     return 0;
 }
 
diff -r 3be60a4c8c63 Objects/exceptions.c
--- a/Objects/exceptions.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/exceptions.c	Fri Jan 20 22:11:43 2012 -0500
@@ -2205,6 +2205,12 @@
 SimpleExtendsException(PyExc_Warning, ResourceWarning,
     "Base class for warnings about resource usage.");
 
+/*
+ *    AlgorithmicComplexityError extends BaseException
+ */
+SimpleExtendsException(PyExc_BaseException, AlgorithmicComplexityError,
+    "Base class for warnings about computationally-infeasible data.");
+
 
 
 /* Pre-computed RuntimeError instance for when recursion depth is reached.
@@ -2318,6 +2324,7 @@
     PRE_INIT(UnicodeWarning)
     PRE_INIT(BytesWarning)
     PRE_INIT(ResourceWarning)
+    PRE_INIT(AlgorithmicComplexityError)
 
     /* OSError subclasses */
     PRE_INIT(ConnectionError);
@@ -2399,6 +2406,7 @@
     POST_INIT(UnicodeWarning)
     POST_INIT(BytesWarning)
     POST_INIT(ResourceWarning)
+    POST_INIT(AlgorithmicComplexityError)
 
     if (!errnomap) {
         errnomap = PyDict_New();

From report at bugs.python.org  Sat Jan 21 04:42:49 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 21 Jan 2012 03:42:49 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1249306383.87.0.829314243328.issue6631@psf.upfronthosting.co.za>
Message-ID: <1327117369.68.0.422982477872.issue6631@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
title: urlparse.urlunsplit() can't handle relative files (for urllib*.open() -> Disallow relative files paths in urllib*.open()

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:43:26 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 03:43:26 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1249306383.87.0.829314243328.issue6631@psf.upfronthosting.co.za>
Message-ID: <E1RoRrR-0008R6-0f@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f6008e936fbc by Senthil Kumaran in branch '2.7':
Fix Issue6631 - Disallow relative files paths in urllib*.open()
http://hg.python.org/cpython/rev/f6008e936fbc

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:55:59 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 03:55:59 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1249306383.87.0.829314243328.issue6631@psf.upfronthosting.co.za>
Message-ID: <E1RoS3Z-0000sO-PC@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 4366c0df2c73 by Senthil Kumaran in branch '2.7':
NEWS entry for Issue6631
http://hg.python.org/cpython/rev/4366c0df2c73

New changeset 514994d7a9f2 by Senthil Kumaran in branch '3.2':
Fix  Issue6631 - Disallow relative file paths in urllib urlopen
http://hg.python.org/cpython/rev/514994d7a9f2

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 04:57:05 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 21 Jan 2012 03:57:05 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1249306383.87.0.829314243328.issue6631@psf.upfronthosting.co.za>
Message-ID: <1327118225.71.0.07609615432.issue6631@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type: performance -> behavior
versions: +Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:03:30 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 04:03:30 +0000
Subject: [issue13814] Generators as context managers.
In-Reply-To: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>
Message-ID: <1327118610.71.0.185139816508.issue13814@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Calling g.close() is pointless for a generator used in normal pull mode and run to completion, as in the example. The generator is already 'closed', so g.close() does not do anything useful. See
http://docs.python.org/py3k/reference/expressions.html#yield-expressions

The method was added to be used with .send() so that generators used in push mode could be told to finish up when there is nothing more to send. For such rare uses, contextlib.closing should usually be sufficient, I think.

So I think this issue should be closed.

----------
nosy: +terry.reedy
stage:  -> test needed
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:17:09 2012
From: report at bugs.python.org (py.user)
Date: Sat, 21 Jan 2012 04:17:09 +0000
Subject: [issue13834] In help(bytes.strip) there is no info about leading
	ASCII whitespace
Message-ID: <1327119429.45.0.897722607985.issue13834@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

help(bytes.strip):

strip(...)
    B.strip([bytes]) -> bytes
    
    Strip leading and trailing bytes contained in the argument.
    If the argument is omitted, strip trailing ASCII whitespace.

----------
assignee: docs at python
components: Documentation
messages: 151718
nosy: docs at python, py.user
priority: normal
severity: normal
status: open
title: In help(bytes.strip) there is no info about leading ASCII whitespace
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13834>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:20:34 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 04:20:34 +0000
Subject: [issue13815] tarfile.ExFileObject can't be wrapped using
	io.TextIOWrapper
In-Reply-To: <1326892577.13.0.456815776311.issue13815@psf.upfronthosting.co.za>
Message-ID: <1327119634.35.0.229086847148.issue13815@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Based on other examples in the doc, I think the note
"... and also supports iteration over its lines."
should be extended with 
" It also has a dummy `flush` method, so that it can be wrapped using :class:`io.TextIOWrapper`."

Then just add
".. versionchanged:: 3.3
      Added the `flush` method."

I leave the test to Lars.

----------
nosy: +terry.reedy
stage:  -> patch review
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13815>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:48:37 2012
From: report at bugs.python.org (R. David Murray)
Date: Sat, 21 Jan 2012 04:48:37 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327121317.63.0.157154868056.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Oh, I see.  Yes, that is a problem.

object.__format__ knows the type of the object it was called on, right?  Couldn't it catch the error and re-raise it with the correct type?  (If the type isn't str, of course, we don't want to get into an infinite recursion.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:50:14 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 04:50:14 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327121414.24.0.905508535086.issue13816@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The Chicago Manual of Style. which is perhaps the mostly widely used general style manual in the US, uses 'th' as a suffix without ' or -.
ie, 28th, etc except for 22d ('preferred' to 22nd) and 23d ('preferred' to 23rd).

https://en.wiktionary.org/wiki/nth
gives nth, which I use, with *n*th (italic n) as alternate form (when italic is available) and n'th and n-th as 'sometimes used'. I think both of the last two are ugly and that we should use either nth or, since we can, *n*th.  I see that we already have *i*.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:50:16 2012
From: report at bugs.python.org (R. David Murray)
Date: Sat, 21 Jan 2012 04:50:16 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327121416.09.0.67580699625.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Oh, never mind that comment about recursion, I wasn't thinking it through.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 05:52:28 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 04:52:28 +0000
Subject: [issue13818] argparse: -h listening required options under optional
	arguments
In-Reply-To: <1326914635.35.0.671947933448.issue13818@psf.upfronthosting.co.za>
Message-ID: <1327121548.06.0.655813090469.issue13818@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +bethard
stage:  -> test needed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13818>
_______________________________________

From report at bugs.python.org  Sat Jan 21 06:33:42 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 05:33:42 +0000
Subject: [issue13820] 2.6 is no longer in the future
In-Reply-To: <1326933957.4.0.830452813069.issue13820@psf.upfronthosting.co.za>
Message-ID: <E1RoTa9-0003Dj-4B@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset c84b39c18963 by Terry Jan Reedy in branch '2.7':
Issue13820 as and with really became keywords in 2.6.
http://hg.python.org/cpython/rev/c84b39c18963

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13820>
_______________________________________

From report at bugs.python.org  Sat Jan 21 06:35:57 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 05:35:57 +0000
Subject: [issue13820] 2.6 is no longer in the future
In-Reply-To: <1326933957.4.0.830452813069.issue13820@psf.upfronthosting.co.za>
Message-ID: <1327124157.85.0.503671828907.issue13820@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I verified that 'as' and 'with' are in the 2.6 keyword.kwlist.
http://svn.python.org/projects/python/branches/release26-maint/Lib/keyword.py. I verified that they work as keywords in 2.7.
http://svn.python.org/projects/python/branches/release26-maint/Doc/reference/compound_stmts.rst
says with is always enabled in 2.6.

I fixed whitespace error in 74551 (117d58fc6e8d)

----------
assignee: docs at python -> terry.reedy
nosy: +terry.reedy
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13820>
_______________________________________

From report at bugs.python.org  Sat Jan 21 06:56:25 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 21 Jan 2012 05:56:25 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327125385.02.0.628755184945.issue13790@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

OK, the example of an empty format spec should be dropped. Let people figure it out ;-).

>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

One possibility is to give (str of) the object instead of the type:

ValueError: Unknown format code 'd' for object '[]'

The downside is a long message for long strings. It would need to be limited (as is done in test error reports).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 06:59:05 2012
From: report at bugs.python.org (akira)
Date: Sat, 21 Jan 2012 05:59:05 +0000
Subject: [issue1602] windows console doesn't print or input Unicode
In-Reply-To: <1197453390.87.0.813702844893.issue1602@psf.upfronthosting.co.za>
Message-ID: <1327125545.64.0.735952527449.issue1602@psf.upfronthosting.co.za>


Changes by akira <4kir4.1i at gmail.com>:


----------
nosy: +akira

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1602>
_______________________________________

From report at bugs.python.org  Sat Jan 21 09:35:32 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Sat, 21 Jan 2012 08:35:32 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1249306383.87.0.829314243328.issue6631@psf.upfronthosting.co.za>
Message-ID: <1327134932.67.0.0307013405669.issue6631@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Sorry, why was this change backported?
Does this fix a specific issue in 2.7 or 3.2?
On the contrary, it seems to me that code which (incorrectly) used urllib.urlopen() to allow both urls and local files will suddenly break.

----------
nosy: +amaury.forgeotdarc
stage: committed/rejected -> commit review
status: closed -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 10:19:51 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sat, 21 Jan 2012 09:19:51 +0000
Subject: [issue6631] Disallow relative files paths in urllib*.open()
In-Reply-To: <1327134932.67.0.0307013405669.issue6631@psf.upfronthosting.co.za>
Message-ID: <20120121091944.GB1988@mathmagic>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Actually, I saw this as a bug with urllib.urlopen and urllib2 had
exhibited proper behaviour previously. Now, both behaviour will be
consistent now.

But, you are right that an *incorrect* usage of urllib.urlopen would
break in 2.7.2. 

If we need to be lenient on that incorrect usage, then this change can
be there in 3.x series, because of urllib.request.urlopen would be
interface which users will be using and it can be reverted from 2.7.

Personally, I am +/- 0 on reverting this in 2.7. Initially, I saw this
as a bug, but later when I added tests for ValueError and checkedin,
I realized that it can break some incorrect usages, as you say.

----------
status: pending -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6631>
_______________________________________

From report at bugs.python.org  Sat Jan 21 10:47:19 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sat, 21 Jan 2012 09:47:19 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327139239.07.0.359898408574.issue13790@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

While looking at object.__format__, I recall that we've already addressed this, sort of. For a different reason, this is already deprecated in 3.3 and will become an error in 3.4. See issues 9856 and 7994.

$ ./python -Wd
Python 3.3.0a0 (default:40e1be1e0707, Jan 15 2012, 00:58:51) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> format([], 'd')
__main__:1: DeprecationWarning: object.__format__ with a non-empty format string is deprecated
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
[67288 refs]
>>> 

We could still have object.__format__ catch and re-throw the ValueError with a better message. I'd have to think it through if we could catch all ValueErrors, or if it's possible for another ValueError to be thrown and we'd only catch and rethrow this specific ValueError.

But since this is deprecated, I'm not sure it's worth the hassle. I'd advocate closing this issue as "won't fix".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 14:57:37 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sat, 21 Jan 2012 13:57:37 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327154257.69.0.829737696328.issue13609@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Does this need need more discussion, code review, testing,
> or just more time?

As I already wrote, I would prefer a very simple os.get_terminal_size() function: don't read environment varaiables, use a simple tuple instead of a new type, and raise an error if the size cannot be read (so no need of default values). The os module is written as a thin wrapper between Python and the OS. A more high level function (read environment variables, handle the error, use a namedtuple) can be written in your application, or maybe in another module.

This is just my opinion, other core developers may prefer your version :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan 21 15:20:13 2012
From: report at bugs.python.org (R. David Murray)
Date: Sat, 21 Jan 2012 14:20:13 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327155613.9.0.462869428462.issue13790@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

So the error is going to be something about the source type not supporting '__format__'?

That change will also address the OP's concern about truncated reprs when a fixed string length is specified, so I agree that the title issue can be closed.  Terry's patch with the ("{}") removed should be committed, though.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 15:27:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 14:27:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327115744.4992.180.camel@surprise>
Message-ID: <1327155912.3382.4.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Thoughts? (apart from "ugh! it's ugly!" yes I know - it's late here)

Is it guaranteed that no usage pattern can render this protection
inefficient? What if a dict is constructed by intermingling lookups and
inserts?
Similarly, what happens with e.g. the common use case of
dictdefault(list), where you append() after the lookup/insert? Does some
key distribution allow the attack while circumventing the protection?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 15:49:56 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 14:49:56 +0000
Subject: [issue13706] non-ascii fill characters no longer work in formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <E1RocGQ-00023r-Vj@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 231c6042c40c by Victor Stinner in branch 'default':
Issue #13706: Support non-ASCII fill characters
http://hg.python.org/cpython/rev/231c6042c40c

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Sat Jan 21 15:50:05 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sat, 21 Jan 2012 14:50:05 +0000
Subject: [issue13706] non-ascii fill characters no longer work in formatting
In-Reply-To: <1325630185.42.0.0487894537415.issue13706@psf.upfronthosting.co.za>
Message-ID: <1327157405.96.0.578741989737.issue13706@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I fixed the original report, but there is still an issue with non-ASCII thousands separator.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13706>
_______________________________________

From report at bugs.python.org  Sat Jan 21 16:36:11 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Sat, 21 Jan 2012 15:36:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf6rZqBuxjL5ZRypT+mK8_AMRNg1Q2wsAxFqLVVx39GmqQ@mail.gmail.com>
Message-ID: <4F1ADB5B.20600@in.waw.pl>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

The hashing with random seed is only marginally slower or more 
complicated than current version.

The patch is big because it moves random number generator initialization 
code around. There's no "per object" tax, and the cost of the random 
number generator initialization is only significant on windows. 
Basically, there's no "tax".

Zbyszek

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 18:02:55 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 21 Jan 2012 17:02:55 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327155912.3382.4.camel@localhost.localdomain>
Message-ID: <1327165340.4992.235.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-21 at 14:27 +0000, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou at free.fr> added the comment:
> 
> > Thoughts? (apart from "ugh! it's ugly!" yes I know - it's late here)
> 
> Is it guaranteed that no usage pattern can render this protection
> inefficient? What if a dict is constructed by intermingling lookups and
> inserts?
> Similarly, what happens with e.g. the common use case of
> dictdefault(list), where you append() after the lookup/insert? Does some
> key distribution allow the attack while circumventing the protection?

Yes, I agree that I was making an unrealistic assumption about usage
patterns.  There was also some global state (the "is_inserting"
variable).

I've tweaked the approach somewhat, moved the global to be per-dict, and
am attaching a revised version of the patch:
   amortized-probe-counting-dmalcolm-2012-01-21-003.patch

In this patch, rather than reset the count each time, I keep track of
the total number of calls to insertdict() that have happened for each
"large dict" (i.e. for which ma_table != ma_smalltable), and the total
number of probe iterations that have been needed to service those
insertions/overwrites.  It raises the exception when the *number of
probe iterations per insertion* exceeds a threshold factor (rather than
merely comparing the number of iterations against the current ma_used of
the dict).  I believe this means that it's tracking and checking every
time the dict is modified, and (I hope) protects us against any data
that drives the dict implementation away from linear behavior (because
that's essentially what it's testing for).  [the per-dict stats are
reset each time that it shrinks down to using ma_smalltable again, but I
think at-risk usage patterns in which that occurs are uncommon relative
to those in which it doesn't].

When attacked, this leads to exceptions like this:
AlgorithmicComplexityError: dict construction used 1697 probes whilst
performing 53 insertions (len() == 58) at key 58 with hash 42

i.e we have a dictionary containing 58 keys, which has seen 53
insert/overwrite operations since transitioning to the non-ma_smalltable
representation (at size 6); presumably it has 128 PyDictEntries.
Servicing those 53 operations has required a total 1697 iterations
through the probing loop, or a little over 32 probes per insert.

I just did a full run of the test suite (using run_tests.py), and it
mostly passed the new tests I've added (included the test for scenario 2
from Frank's email).

There were two failures:
======================================================================
FAIL: test_inheritance (test.test_pep352.ExceptionClassTests)
----------------------------------------------------------------------
AssertionError: 1 != 0 : {'AlgorithmicComplexityError'} not accounted
for
----------------------------------------------------------------------
which is obviously fixable (given a decision on where the exception
lives in the hierarchy)

and this one:
test test_mutants crashed -- Traceback (most recent call last):
  File
"/home/david/coding/python-hg/cpython-count-collisions/Lib/test/regrtest.py", line 1214, in runtest_inner
    the_package = __import__(abstest, globals(), locals(), [])
  File
"/home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py", line 159, in <module>
    test(100)
  File
"/home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py", line 156, in test
    test_one(random.randrange(1, 100))
  File
"/home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py", line 132, in test_one
    dict2keys = fill_dict(dict2, range(n), n)
  File
"/home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py", line 118, in fill_dict
    Horrid(random.choice(candidates))
AlgorithmicComplexityError: dict construction used 2753 probes whilst
performing 86 insertions (len() == 64) at key Horrid(86) with hash 42
though that seems to be deliberately degenerate code.

Caveats:
* no overflow handling (what happens after 2**32 modifications to a
long-lived dict on a 32-bit build?) - though that's fixable.
* no idea what the scaling factor for the threshold should be (there may
also be a deep mathematical objection here, based on how big-O notation
is defined in terms of an arbitrary scaling factor and limit)
* not optimized; I haven't looked at performance yet
* doesn't cover set(), though that also has spare space (I hope) via its
own smalltable array.

BTW, note that although I've been working on this variant of the
collision counting approach, I'm not opposed to the hash randomization
approach, or to adding extra checks in strategic places within the
stdlib: I'm keen to get some kind of appropriate fix approved by the
upstream Python development community so I can backport it to the
various recent-to-ancient versions of CPython I support in RHEL (and
Fedora), before we start seeing real-world attacks.

Hope this is helpful
Dave

----------
Added file: http://bugs.python.org/file24289/amortized-probe-counting-dmalcolm-2012-01-21-003.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 3be60a4c8c63 Include/pyerrors.h
--- a/Include/pyerrors.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/pyerrors.h	Sat Jan 21 11:22:34 2012 -0500
@@ -207,6 +207,8 @@
 PyAPI_DATA(PyObject *) PyExc_BytesWarning;
 PyAPI_DATA(PyObject *) PyExc_ResourceWarning;
 
+PyAPI_DATA(PyObject *) PyExc_AlgorithmicComplexityError;
+
 
 /* Convenience functions */
 
diff -r 3be60a4c8c63 Lib/test/test_dict.py
--- a/Lib/test/test_dict.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_dict.py	Sat Jan 21 11:22:34 2012 -0500
@@ -3,6 +3,8 @@
 
 import collections, random, string
 import gc, weakref
+import sys
+import time
 
 
 class DictTest(unittest.TestCase):
@@ -757,6 +759,198 @@
         self._tracked(MyDict())
 
 
+# Support classes for HashCollisionTests:
+class ChosenHash:
+    """
+    Use this to create arbitrary collections of keys that are non-equal
+    but have equal hashes, without needing to include hostile data
+    within the test suite.
+    """
+    def __init__(self, variability, hash):
+        self.variability = variability
+        self.hash = hash
+
+    def __eq__(self, other):
+        # The variability field is used to handle non-equalness:
+        return self.variability == other.variability
+
+    def __hash__(self):
+        return self.hash
+
+    def __repr__(self):
+        return 'ChosenHash(%r, %r)' % (self.variability,
+                                       self.hash)
+
+class Timer:
+    """
+    Simple way to measure time elapsed during a test case
+    """
+    def __init__(self):
+        self.starttime = time.time()
+
+    def get_elapsed_time(self):
+        """Get elapsed time in seconds as a float"""
+        curtime = time.time()
+        return curtime - self.starttime
+
+    def elapsed_time_as_str(self):
+        """Get elapsed time as a string (with units)"""
+        return '%0.3f seconds' % self.get_elapsed_time()
+
+class TookTooLong(RuntimeError):
+    def __init__(self, timelimit, elapsed, itercount=None):
+        self.timelimit = timelimit
+        self.elapsed = elapsed
+        self.itercount = itercount
+
+    def __str__(self):
+        result = 'took >= %s seconds' % self.timelimit
+        if self.itercount is not None:
+            result += (' (%0.3f seconds elapsed after %i iterations)'
+                       % (self.elapsed, self.itercount))
+        else:
+            result += ' (%0.3f seconds elapsed)' % self.elapsed
+        return result
+
+# Some of the tests involve constructing large dictionaries.  How big
+# should they be?
+ITEM_COUNT = 1000000
+
+# Arbitrary threshold (in seconds) for a "reasonable amount of time"
+# that it should take to work with ITEM_COUNT items:
+TIME_LIMIT = 5
+
+class _FasterThanContext(object):
+    """
+    A context manager for implementing assertFasterThan
+    """
+    def __init__(self, test_case, **kwargs):
+        self.test_case = test_case
+        if 'seconds' in kwargs:
+            self.timelimit = kwargs['seconds']
+        else:
+            raise ValueError()
+
+    def __enter__(self):
+        self.timer = Timer()
+        return self
+
+    def __exit__(self, exc_type, exc_value, tb):
+        if exc_type is not None:
+            # let unexpected exceptions pass through
+            return
+
+        if 1:
+            print('timer within %s took %s'
+                  % (self.test_case, self.timer.elapsed_time_as_str()))
+
+    def handle(self, callable_obj, args, kwargs):
+        """
+        If callable_obj is None, assertRaises/Warns is being used as a
+        context manager, so check for a 'msg' kwarg and return self.
+        If callable_obj is not None, call it passing args and kwargs.
+        """
+        if callable_obj is None:
+            self.msg = kwargs.pop('msg', None)
+            return self
+        with self:
+            callable_obj(*args, **kwargs)
+
+    def check_for_timeout(self, itercount):
+        """
+        Allow directly checking for timeouts from within a loop, supplying 
+        an iteration count.  If the timer has elapsed, this will raise a
+        TookTooLong exception, indicating how many iterations were completed
+        when the time limit was reached.  Otherwise, it does nothing.
+        """
+        elapsed_time = self.timer.get_elapsed_time()
+        if elapsed_time > self.timelimit:
+            raise TookTooLong(self.timelimit,
+                              elapsed_time,
+                              itercount)
+        
+ at support.cpython_only
+class HashCollisionTests(unittest.TestCase):
+    """
+    Issue 13703: tests about the behavior of dicts in the face of hostile data
+    """
+
+    def assertFasterThan(self, callableObj=None, *args, **kwargs):
+        context = _FasterThanContext(self, *args, **kwargs)
+        return context.handle(callableObj, args, kwargs)
+
+    def test_timings_with_benign_data(self):
+        # Verify that inserting many keys into a dict only takes a few seconds
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[i] = 0
+
+        # Verify that we can also retrieve the values quickly:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d[i]
+
+        # Verify that we can quickly insert the same item many times
+        # (overwriting each time):
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[0] = 0
+
+    def test_not_reaching_limit(self):
+        # Ensure that we can insert equal-hashed keys up to (but not reaching)
+        # the collision climit:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d = dict()
+            for i in range(50):
+                key = ChosenHash(i, 42)
+                d[key] = 0
+            
+    def test_reaching_collision_limit(self):
+        """
+        Ensure that too many non-equal keys with the same hash lead to a
+        TooManyCollisions exception
+        """
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            with self.assertRaisesRegex(AlgorithmicComplexityError,
+                                        ('dict construction used 1697 probes'
+                                         ' whilst performing 53 insertions'
+                                         ' \(len\(\) == 58\)'
+                                         ' at key ChosenHash\(58, 42\)'
+                                         ' with hash 42')):
+                d = dict()
+                for i in range(1000):
+                    key = ChosenHash(i, 42)
+                    d[key] = 0
+
+    # Frank Sievertsen found scenarios in which the collision-counting
+    # scheme could be attacked:
+    #   http://mail.python.org/pipermail/python-dev/2012-January/115726.html
+
+    def test_scenario_b_from_Frank_Sievertsen(self):
+        d = dict()
+
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            with self.assertRaisesRegex(AlgorithmicComplexityError,
+                                        ('dict construction used 1697 probes'
+                                         ' whilst performing 53 insertions'
+                                         ' \(len\(\) == 58\)'
+                                         ' at key ChosenHash\(58, 42\)'
+                                         ' with hash 42')):
+                # Insert hash collisions up to (but not reaching) the proposed
+                # limit:
+                for i in range(1000):
+                    key = ChosenHash(i, 42)
+                    d[key] = 0
+                    cm.check_for_timeout(i)
+
+                # Now try to add many equal values that collide
+                # with the hash, and see how long it takes
+                for i in range(ITEM_COUNT):
+                    key = ChosenHash(0, 42)
+                    d[key] = 0
+                    cm.check_for_timeout(i)
+
 from test import mapping_tests
 
 class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol):
@@ -771,6 +965,7 @@
 def test_main():
     support.run_unittest(
         DictTest,
+        HashCollisionTests,
         GeneralMappingTests,
         SubclassMappingTests,
     )
diff -r 3be60a4c8c63 Objects/dictobject.c
--- a/Objects/dictobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/dictobject.c	Sat Jan 21 11:22:34 2012 -0500
@@ -10,6 +10,19 @@
 #include "Python.h"
 #include "stringlib/eq.h"
 
+#define Py_MAX_AVERAGE_PROBES_PER_INSERT 32
+/* power-of-two to make multiplication fast */
+
+/* For large dictionaries, reuse the space allocated for ma_smalltable */
+typedef struct _Py_LargeDictFields {
+    int is_inserting;
+    size_t num_insertions;
+    size_t insertion_iter_count;
+} _Py_LargeDictFields;
+
+#define PyDict_LARGEDICTFIELDS(mp) \
+  ((_Py_LargeDictFields*)&(mp)->ma_smalltable)
+/* FIXME: add assert(mp->ma_table != mp->ma_smalltable) */
 
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
@@ -25,6 +38,45 @@
     Py_DECREF(tup);
 }
 
+/* Set a AlgorithmicComplexityError error */
+static void
+set_algorithmic_complexity_error(PyDictObject *mp, PyObject *key, Py_hash_t hash)
+{
+    _Py_LargeDictFields *large_fields = PyDict_LARGEDICTFIELDS(mp);
+    assert(mp->ma_table != mp->ma_smalltable);
+
+    PyErr_Format(PyExc_AlgorithmicComplexityError,
+                 ("dict construction used %i probes"
+                  " whilst performing %i insertions"
+                  " (len() == %i)"
+                  " at key %R with hash %zd"),
+                 large_fields->insertion_iter_count,
+                 large_fields->num_insertions,
+                 mp->ma_used,
+                 key, hash);
+    /* Backporting notes: (FIXME)
+       %R is a Python 3-ism
+       %zd is for Py_ssize_t, which in Python 3 is the same as Py_hash_t
+    */
+}
+
+static int
+check_iter_count(PyDictObject *mp, PyObject *key, Py_hash_t hash)
+{
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        _Py_LargeDictFields *large_fields = PyDict_LARGEDICTFIELDS(mp);
+        assert(mp->ma_table != mp->ma_smalltable);
+        if (large_fields->is_inserting) {
+            if (++large_fields->insertion_iter_count
+                > (large_fields->num_insertions * Py_MAX_AVERAGE_PROBES_PER_INSERT) ) {
+                set_algorithmic_complexity_error(mp, key, hash);
+                return 1; /* error */
+            }
+        }
+    }
+    return 0; /* no error */
+}
+
 /* Define this out if you don't want conversion statistics on exit. */
 #undef SHOW_CONVERSION_COUNTS
 
@@ -389,6 +441,10 @@
         }
         else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+
+        if (check_iter_count(mp, key, hash)) {
+            return NULL;
+        }
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -451,6 +507,9 @@
             return ep;
         if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+        if (check_iter_count(mp, key, hash)) {
+            return NULL;
+        }
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -532,7 +591,14 @@
     typedef PyDictEntry *(*lookupfunc)(PyDictObject *, PyObject *, Py_hash_t);
 
     assert(mp->ma_lookup != NULL);
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        PyDict_LARGEDICTFIELDS(mp)->is_inserting = 1;
+        PyDict_LARGEDICTFIELDS(mp)->num_insertions += 1;
+    }
     ep = mp->ma_lookup(mp, key, hash);
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        PyDict_LARGEDICTFIELDS(mp)->is_inserting = 0;
+    }
     if (ep == NULL) {
         Py_DECREF(key);
         Py_DECREF(value);
@@ -675,8 +741,15 @@
         /* else key == value == NULL:  nothing to do */
     }
 
-    if (is_oldtable_malloced)
+    if (is_oldtable_malloced) {
         PyMem_DEL(oldtable);
+    } else {
+        if (mp->ma_table != mp->ma_smalltable) {
+            /* clean ma_smalltable for use as _Py_LargeDictFields: */
+            memset(mp->ma_smalltable, 0, sizeof(mp->ma_smalltable));
+        }
+    }
+
     return 0;
 }
 
diff -r 3be60a4c8c63 Objects/exceptions.c
--- a/Objects/exceptions.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/exceptions.c	Sat Jan 21 11:22:34 2012 -0500
@@ -2205,6 +2205,12 @@
 SimpleExtendsException(PyExc_Warning, ResourceWarning,
     "Base class for warnings about resource usage.");
 
+/*
+ *    AlgorithmicComplexityError extends BaseException
+ */
+SimpleExtendsException(PyExc_BaseException, AlgorithmicComplexityError,
+    "Base class for warnings about computationally-infeasible data.");
+
 
 
 /* Pre-computed RuntimeError instance for when recursion depth is reached.
@@ -2318,6 +2324,7 @@
     PRE_INIT(UnicodeWarning)
     PRE_INIT(BytesWarning)
     PRE_INIT(ResourceWarning)
+    PRE_INIT(AlgorithmicComplexityError)
 
     /* OSError subclasses */
     PRE_INIT(ConnectionError);
@@ -2399,6 +2406,7 @@
     POST_INIT(UnicodeWarning)
     POST_INIT(BytesWarning)
     POST_INIT(ResourceWarning)
+    POST_INIT(AlgorithmicComplexityError)
 
     if (!errnomap) {
         errnomap = PyDict_New();

From report at bugs.python.org  Sat Jan 21 18:03:22 2012
From: report at bugs.python.org (Brett Cannon)
Date: Sat, 21 Jan 2012 17:03:22 +0000
Subject: [issue13829] exception error
In-Reply-To: <1326998522.6.0.790472668429.issue13829@psf.upfronthosting.co.za>
Message-ID: <1327165402.06.0.913738022612.issue13829@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

Then I'm going to assume the bug lies with Moviegrabber doing something wrong and it isn't Python's direct fault.

----------
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Sat Jan 21 18:07:57 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 21 Jan 2012 17:07:57 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327165340.4992.235.camel@surprise>
Message-ID: <1327165644.4992.237.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

(or combination of fixes, of course)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 18:35:19 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sat, 21 Jan 2012 17:35:19 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327167319.61.0.511540551219.issue13790@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

The error message will be: "non-empty format string passed to object.__format__".

I agree with your comment about Terry's patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sat Jan 21 19:57:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 18:57:39 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327165340.4992.235.camel@surprise>
Message-ID: <1327172141.3382.12.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> In this patch, rather than reset the count each time, I keep track of
> the total number of calls to insertdict() that have happened for each
> "large dict" (i.e. for which ma_table != ma_smalltable), and the total
> number of probe iterations that have been needed to service those
> insertions/overwrites.  It raises the exception when the *number of
> probe iterations per insertion* exceeds a threshold factor (rather than
> merely comparing the number of iterations against the current ma_used of
> the dict).

This sounds much more robust than the previous attempt.

> When attacked, this leads to exceptions like this:
> AlgorithmicComplexityError: dict construction used 1697 probes whilst
> performing 53 insertions (len() == 58) at key 58 with hash 42

We'll have to discuss the name of the exception and the error message :)

> Caveats:
> * no overflow handling (what happens after 2**32 modifications to a
> long-lived dict on a 32-bit build?) - though that's fixable.

How do you suggest to fix it?

> * no idea what the scaling factor for the threshold should be (there may
> also be a deep mathematical objection here, based on how big-O notation
> is defined in terms of an arbitrary scaling factor and limit)

I'd make the threshold factor a constant, e.g. 64 or 128 (it should not
be too small, to avoid false positives).
We're interested in the actual slowdown factor, which a constant factor
models adequately. It's the slowdown factor which makes a DOS attack
using this technique efficient. Whether or not dict construction truely
degenerates into a O(n**2) operation is less relevant.

There needs to be a way to disable it: an environment variable would be
the minimum IMO.
Also, in 3.3 there should probably be a sys function to enable or
disable it at runtime. Not sure it should be backported since it's a new
API.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 20:04:57 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 19:04:57 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1327154257.69.0.829737696328.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327172559.3382.15.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > Does this need need more discussion, code review, testing,
> > or just more time?
> 
> As I already wrote, I would prefer a very simple
> os.get_terminal_size() function: don't read environment varaiables,
> use a simple tuple instead of a new type, and raise an error if the
> size cannot be read (so no need of default values). The os module is
> written as a thin wrapper between Python and the OS. A more high level
> function (read environment variables, handle the error, use a
> namedtuple) can be written in your application, or maybe in another
> module.

I think we have reached the point where we won't be in total agreement
over the API, so let's choose whatever is submitted as a patch.

I only have two remaining issues with the patch:
- the tests needn't be in a separate file, they can go in test_os
- there should be a test for get_terminal_size_raw as well (and of
course it should be skipped if the function doesn't exist)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan 21 20:30:16 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 19:30:16 +0000
Subject: [issue12922] StringIO and seek()
In-Reply-To: <1315342130.67.0.227650351498.issue12922@psf.upfronthosting.co.za>
Message-ID: <E1Rogdi-0004mt-I9@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 03e61104f7a2 by Antoine Pitrou in branch '3.2':
Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods.
http://hg.python.org/cpython/rev/03e61104f7a2

New changeset f7e5abfb31ea by Antoine Pitrou in branch 'default':
Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods.
http://hg.python.org/cpython/rev/f7e5abfb31ea

New changeset fcf4d547bed8 by Antoine Pitrou in branch '2.7':
Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods.
http://hg.python.org/cpython/rev/fcf4d547bed8

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12922>
_______________________________________

From report at bugs.python.org  Sat Jan 21 20:30:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 19:30:32 +0000
Subject: [issue12922] StringIO and seek()
In-Reply-To: <1315342130.67.0.227650351498.issue12922@psf.upfronthosting.co.za>
Message-ID: <1327174232.3.0.12263572179.issue12922@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12922>
_______________________________________

From report at bugs.python.org  Sat Jan 21 20:40:19 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Sat, 21 Jan 2012 19:40:19 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327174819.2.0.625826601057.issue13609@psf.upfronthosting.co.za>


Giampaolo Rodola' <g.rodola at gmail.com> added the comment:

> read environment varaiables [...] and raise an error if the size cannot be
> read (so no need of default values). The os module is written as a thin
> wrapper between Python and the OS. A more high level function (read 
> environment variables, handle the error, use a namedtuple) can be written in 
> your application, or maybe in another module.

+1. I also find weird that a function, especially one living in the os module, has such a high level of abstraction (basically this is why I was originally proposing shutil module for this to go in).

Given the different opinions about the API, I think it's best to expose the lowest level functionality as-is, and let the user decide what to do (read env vars first, suppress the exception, use a fallback, etc.). 


> I think we have reached the point where we won't be in total 
> agreement over the API, so let's choose whatever is submitted 
> as a patch.

I'd be more careful. Once this gets in it will be too late for a change.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan 21 21:35:34 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 21 Jan 2012 20:35:34 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327178134.95.0.907115165097.issue13816@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

> ... with *n*th (italic n) as alternate form

Knuth uses that in TAOCP, too. I think with or without italics it's
the most frequently used form overall.

Also the Lisp function is called nth and not n-th, even though in Lisp
it is possible to use hyphens in function names.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sat Jan 21 22:07:41 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 21 Jan 2012 21:07:41 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327172141.3382.12.camel@localhost.localdomain>
Message-ID: <1327176822.4992.257.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

Well, the old attempt was hardly robust :)

Can anyone see any vulnerabilities in this approach?

Yeah; I was mostly trying to add raw data (to help me debug the
implementation).

I wonder if the dict statistics should be exposed with extra attributes
or a method on the dict; e.g. a __stats__ attribute, something like
this:

LargeDictStats(keys=58, mask=127, insertions=53, iterations=1697)

SmallDictStats(keys=3, mask=7)

or somesuch. Though that's a detail, I think.

> > Caveats:
> > * no overflow handling (what happens after 2**32 modifications to a
> > long-lived dict on a 32-bit build?) - though that's fixable.
> 
> How do you suggest to fix it?

If the dict is heading towards overflow of these counters, it's either
long-lived, or *huge*.

Possible approaches:
(a) use 64-bit counters rather than 32-bit, though that's simply
delaying the inevitable
(b) when one of the counters gets large, divide both of them by a
constant (e.g. 2).  We're interested in their ratio, so dividing both by
a constant preserves this.

By "a constant" do you mean from the perspective of big-O notation, or
do you mean that it should be hardcoded (I was wondering if it should be
a sys variable/environment variable etc?).

> We're interested in the actual slowdown factor, which a constant factor
> models adequately. It's the slowdown factor which makes a DOS attack
> using this technique efficient. Whether or not dict construction truely
> degenerates into a O(n**2) operation is less relevant.

OK.

> There needs to be a way to disable it: an environment variable would be
> the minimum IMO.

e.g. set it to 0 to enable it, set it to nonzero to set the scale
factor.
Any idea what to call it? 

PYTHONALGORITHMICCOMPLEXITYTHRESHOLD=0 would be quite a mouthful.

OK

BTW, presumably if we do it, we should do it for sets as well?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:20:48 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 22:20:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327176822.4992.257.camel@surprise>
Message-ID: <1327184329.3382.23.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I wonder if the dict statistics should be exposed with extra attributes
> or a method on the dict; e.g. a __stats__ attribute, something like
> this:
> 
> LargeDictStats(keys=58, mask=127, insertions=53, iterations=1697)
> 
> SmallDictStats(keys=3, mask=7)

Sounds a bit overkill, and it shouldn't be a public API (which
__methods__ are). Even a private API on dicts would quickly become
visible, since dicts are so pervasive.

> > > Caveats:
> > > * no overflow handling (what happens after 2**32 modifications to a
> > > long-lived dict on a 32-bit build?) - though that's fixable.
> > 
> > How do you suggest to fix it?
> 
> If the dict is heading towards overflow of these counters, it's either
> long-lived, or *huge*.
> 
> Possible approaches:
> (a) use 64-bit counters rather than 32-bit, though that's simply
> delaying the inevitable

Well, even assuming one billion lookup probes per second on a single
dictionary, the inevitable will happen in 584 years with a 64-bit
counter (but only 4 seconds with a 32-bit counter).

A real issue, though, may be the cost of 64-bit arithmetic on 32-bit
CPUs.

> (b) when one of the counters gets large, divide both of them by a
> constant (e.g. 2).  We're interested in their ratio, so dividing both by
> a constant preserves this.

Sounds good, although we may want to pull this outside of the critical
loop.

> By "a constant" do you mean from the perspective of big-O notation, or
> do you mean that it should be hardcoded (I was wondering if it should be
> a sys variable/environment variable etc?).

Hardcoded, as in your patch.

> > There needs to be a way to disable it: an environment variable would be
> > the minimum IMO.
> 
> e.g. set it to 0 to enable it, set it to nonzero to set the scale
> factor.

0 to enable it sounds misleading. I'd say:
- 0 to disable it
- 1 to enable it and use the default scaling factor
- >= 2 to enable it and set the scaling factor

> Any idea what to call it? 

PYTHONDICTPROTECTION ?
Most people should either enable or disable it, not change the scaling
factor.

> BTW, presumably if we do it, we should do it for sets as well?

Yeah, and use the same env var / sys function.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:39:43 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 22:39:43 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <E1Rojb3-0003Z5-Ip@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 61aa484a3e54 by Gregory P. Smith in branch '3.2':
Fixes issue #8052: The posix subprocess module's close_fds behavior was
http://hg.python.org/cpython/rev/61aa484a3e54

New changeset 8879874d66a2 by Gregory P. Smith in branch 'default':
Fixes issue #8052: The posix subprocess module's close_fds behavior was
http://hg.python.org/cpython/rev/8879874d66a2

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:41:58 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 21 Jan 2012 22:41:58 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327184329.3382.23.camel@localhost.localdomain>
Message-ID: <1327185685.4992.269.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-21 at 22:20 +0000, Antoine Pitrou wrote:

> Sounds a bit overkill, and it shouldn't be a public API (which
> __methods__ are). Even a private API on dicts would quickly become
> visible, since dicts are so pervasive.

Fair enough.

> > > > Caveats:
> > > > * no overflow handling (what happens after 2**32 modifications to a
> > > > long-lived dict on a 32-bit build?) - though that's fixable.
> > > 
> > > How do you suggest to fix it?
> > 
> > If the dict is heading towards overflow of these counters, it's either
> > long-lived, or *huge*.
> > 
> > Possible approaches:
> > (a) use 64-bit counters rather than 32-bit, though that's simply
> > delaying the inevitable
> 
> Well, even assuming one billion lookup probes per second on a single
> dictionary, the inevitable will happen in 584 years with a 64-bit
> counter (but only 4 seconds with a 32-bit counter).
> 
> A real issue, though, may be the cost of 64-bit arithmetic on 32-bit
> CPUs.
> 
> > (b) when one of the counters gets large, divide both of them by a
> > constant (e.g. 2).  We're interested in their ratio, so dividing both by
> > a constant preserves this.
> 
> Sounds good, although we may want to pull this outside of the critical
> loop.

OK; I'll look at implementing (b).

Oops, yeah, that was a typo; I meant 0 to disable.

> - 0 to disable it
> - 1 to enable it and use the default scaling factor
> - >= 2 to enable it and set the scaling factor

You said above that it should be hardcoded; if so, how can it be changed
at run-time from an environment variable?  Or am I misunderstanding.

Works for me.

> > BTW, presumably if we do it, we should do it for sets as well?
> 
> Yeah, and use the same env var / sys function.

Despite the "DICT" in the title?  OK.

Thanks for the feedback.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:45:29 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 22:45:29 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327185685.4992.269.camel@surprise>
Message-ID: <1327185811.3382.28.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> You said above that it should be hardcoded; if so, how can it be changed
> at run-time from an environment variable?  Or am I misunderstanding.

You're right, I used the wrong word. I meant it should be a constant
independently of the dict size. But, indeed, not hard-coded in the
source.

> > > BTW, presumably if we do it, we should do it for sets as well?
> > 
> > Yeah, and use the same env var / sys function.
> 
> Despite the "DICT" in the title?  OK.

Well, dict is the most likely target for these attacks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:47:27 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 22:47:27 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1327174819.2.0.625826601057.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327185929.3382.30.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> +1. I also find weird that a function, especially one living in the os
> module, has such a high level of abstraction (basically this is why I
> was originally proposing shutil module for this to go in).
> 
> Given the different opinions about the API, I think it's best to
> expose the lowest level functionality as-is, and let the user decide
> what to do (read env vars first, suppress the exception, use a
> fallback, etc.). 

Fair enough, but other people expressed sympathy for the two-function
approach :) I'm personally indifferent, although I find
"get_terminal_size_raw" a bit ugly and liked "query_terminal_size"
better.

(and looking up ROWS and COLUMNS make sense, since they are de facto
standards)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sat Jan 21 23:51:05 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 22:51:05 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <E1Rojm4-0004RA-IO@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 780992c9afea by Gregory P. Smith in branch '3.2':
Add a Misc/NEWS entry for issue 8052.
http://hg.python.org/cpython/rev/780992c9afea

New changeset 1f0a01dc723c by Gregory P. Smith in branch 'default':
A Misc/NEWS entry for issue 8052.
http://hg.python.org/cpython/rev/1f0a01dc723c

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 22 00:16:16 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 21 Jan 2012 23:16:16 +0000
Subject: [issue13405] Add DTrace probes
In-Reply-To: <1327104223.53.0.217811935108.issue13405@psf.upfronthosting.co.za>
Message-ID: <1327187656.3382.40.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> So, yes. The code is intrusive. The code deals with a lot of internal
> machinery (PEP393 support in the ustack helper was quite difficult).
> It is going to break from time to time, sure. At the same time, I am
> committed to support it. And even if it is dropped in 3.4, no Python
> program will be affected.

To ease the concerns, I think you should make it so that dtrace-specific
code gets out of the way as much as possible.
I suggest you create a Python/ceval-dtrace.h header and put most
dtrace-specific code from ceval.c there. Its inclusion should be
conditional on WITH_DTRACE so that other core devs can ignore its
presence.

A couple other comments:
- in the makefile, DTRACEOBJS is inconsistent with DTRACE_STATIC and
LIBRARY_OBJS. You should make it DTRACE_OBJS.
- please add comments at the top of whatever header files you add, to
make it clear that they are dtrace-specific. Mentions of "ustack helper"
are a bit too specific to be helpful.
- some code lacks error checking, e.g. when calling PyUnicode_AsUTF8.
- is co_linenos ever freed or is it a memory leak?
- your indices and offsets should be Py_ssize_t, not int
- is an empty dtrace module really needed? a flag variable in the sys
module should be enough
- as you can see the Makefile uses "-rm -f", you should probably do the
same instead of "rm -f"
- you have a rather strange "if true" in your configure.in additions

Thanks in advance.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13405>
_______________________________________

From report at bugs.python.org  Sun Jan 22 00:21:28 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 21 Jan 2012 23:21:28 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <E1RokFL-0007cA-IW@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset d0acd8169c2a by Gregory P. Smith in branch '3.2':
Bugfix for issue #8052 fix on *BSD variants.
http://hg.python.org/cpython/rev/d0acd8169c2a

New changeset 5be3dadd2eef by Gregory P. Smith in branch '3.2':
Another issue #8052 bugfix (related to previous commit).
http://hg.python.org/cpython/rev/5be3dadd2eef

New changeset e52d81e0c750 by Gregory P. Smith in branch 'default':
bugfix for issue 8052 fixes on *BSD platforms.
http://hg.python.org/cpython/rev/e52d81e0c750

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 22 00:42:30 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Sat, 21 Jan 2012 23:42:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327185811.3382.28.camel@localhost.localdomain>
Message-ID: <CAGE7PNJX1GmJpf9A=x-qS+LdBZMbp0=YRfn1KZhtKjNT2W9j8g@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

On Sat, Jan 21, 2012 at 2:45 PM, Antoine Pitrou <report at bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou at free.fr> added the comment:
>
>> You said above that it should be hardcoded; if so, how can it be changed
>> at run-time from an environment variable? ?Or am I misunderstanding.
>
> You're right, I used the wrong word. I meant it should be a constant
> independently of the dict size. But, indeed, not hard-coded in the
> source.
>
>> > > BTW, presumably if we do it, we should do it for sets as well?
>> >
>> > Yeah, and use the same env var / sys function.
>>
>> Despite the "DICT" in the title? ?OK.
>
> Well, dict is the most likely target for these attacks.
>

While true I wouldn't make that claim as there will be applications
using a set in a vulnerable manner. I'd prefer to see any such
environment variable name used to configure this behavior not mention
DICT or SET but just say HASHTABLE.  That is a much better bikeshed
color. ;)

I'm still in the hash seed randomization camp but I'm finding it
interesting all of the creative ways others are trying to "solve" this
problem in a way that could be enabled by default in stable versions
regardless. :)

-gps

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 00:47:58 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Sat, 21 Jan 2012 23:47:58 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNJX1GmJpf9A=x-qS+LdBZMbp0=YRfn1KZhtKjNT2W9j8g@mail.gmail.com>
Message-ID: <CAFRnB2Webjv+mEHt9UrQOmVq82+NHsDW3zuyLM-rnqVxhD0LJQ@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

On Sat, Jan 21, 2012 at 5:42 PM, Gregory P. Smith <report at bugs.python.org>wrote:

>
> Gregory P. Smith <greg at krypto.org> added the comment:
>
> On Sat, Jan 21, 2012 at 2:45 PM, Antoine Pitrou <report at bugs.python.org>
> wrote:
> >
> > Antoine Pitrou <pitrou at free.fr> added the comment:
> >
> >> You said above that it should be hardcoded; if so, how can it be changed
> >> at run-time from an environment variable?  Or am I misunderstanding.
> >
> > You're right, I used the wrong word. I meant it should be a constant
> > independently of the dict size. But, indeed, not hard-coded in the
> > source.
> >
> >> > > BTW, presumably if we do it, we should do it for sets as well?
> >> >
> >> > Yeah, and use the same env var / sys function.
> >>
> >> Despite the "DICT" in the title?  OK.
> >
> > Well, dict is the most likely target for these attacks.
> >
>
> While true I wouldn't make that claim as there will be applications
> using a set in a vulnerable manner. I'd prefer to see any such
> environment variable name used to configure this behavior not mention
> DICT or SET but just say HASHTABLE.  That is a much better bikeshed
> color. ;)
>
> I'm still in the hash seed randomization camp but I'm finding it
> interesting all of the creative ways others are trying to "solve" this
> problem in a way that could be enabled by default in stable versions
> regardless. :)
>
> -gps
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

I'm a little slow, so bear with me, but David, does this counting scheme in
any way address the issue of:

I'm able to put N pieces of data into the database on successive requests,
but then *rendering* that data puts it in a dictionary, which renders that
page unviewable by anyone.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 01:20:02 2012
From: report at bugs.python.org (=?utf-8?q?Denilson_Figueiredo_de_S=C3=A1?=)
Date: Sun, 22 Jan 2012 00:20:02 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1327174819.2.0.625826601057.issue13609@psf.upfronthosting.co.za>
Message-ID: <CACGt9ykO9q9yHVCnB8-MdC4K79mzn63G38im-N5MVdzYjvsmsQ@mail.gmail.com>


Denilson Figueiredo de S? <denilsonsa at gmail.com> added the comment:

On Sat, Jan 21, 2012 at 17:40, Giampaolo Rodola' <report at bugs.python.org> wrote:
>
> Given the different opinions about the API, I think it's best to expose the lowest
> level functionality as-is, and let the user decide what to do (read env vars first,
> suppress the exception, use a fallback, etc.).

As a Python user (and not a committer), I disagree.

As an user, I don't care too much where the function should be placed
(although I believe os or sys are sensible choices). What I do care is
that I want a extremely simple function that will "just work". Don't
make me add code for handling all the extra cases, such code should be
inside the function.

All this discussion about the API made me remember this presentation:
http://python-for-humans.heroku.com/

Also, I see no downside of using a Named Tuple. Issue 4285 actually
added a named tuple to the sys.version_info.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sun Jan 22 03:13:48 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sun, 22 Jan 2012 02:13:48 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNJX1GmJpf9A=x-qS+LdBZMbp0=YRfn1KZhtKjNT2W9j8g@mail.gmail.com>
Message-ID: <1327198390.4992.284.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

5 more characters:
PYTHONHASHTABLEPROTECTION
or
PYHASHTABLEPROTECTION
maybe?

I'm in *both* camps: I like hash seed randomization fwiw.  I'm nervous
about enabling either of the approaches by default, but I can see myself
backporting both approaches into RHEL's ancient Python versions,
compiled in, disabled by default, but available at runtime via env vars
(assuming that no major flaws are discovered in my patch e.g.
performance).

I'm sorry if I'm muddying the waters by working on this approach.

Is the hash randomization approach ready to go, or is more work needed?
If the latter, is there a clear TODO list?
(for backporting to 2.*, presumably we'd want PyStringObject to be
randomized; I think this means that PyBytesObject needs to be randomized
also in 3.*; don't we need hash(b'foo') == hash('foo') ?).  Does the
patch needs to also randomize the hashes of the numeric types? (I think
not; that may break too much 3rd-party code (NumPy?)).

[If we're bikeshedding,  I prefer the term "salt" to "seed" in the hash
randomization approach: there's a per-process "hash salt", which is
either randomly generated, or comes from the environment, set to 0 to
disable]

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 03:16:17 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 22 Jan 2012 02:16:17 +0000
Subject: [issue13790] In str.format an incorrect error message for list, tuple,
	dict, set
In-Reply-To: <1326605478.95.0.601079964751.issue13790@psf.upfronthosting.co.za>
Message-ID: <1327198577.87.0.724414001571.issue13790@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Looking further, I noticed that 'string' needed to be changed to 'specification' in the following sentence also. Then I decided that the preceding sentence
 
"Most built-in types implement the following options for format specifications, although some of the formatting options are only supported by the numeric types."

should really follow the one about non-empty format specs. This positioning should make it more obvious that most of the options affect the string representation of the object after, not before, the string is produced, and are therefore applicable to all objects and not just string and number objects. I also propose to modify it so it is shorter and no longer contradictory, to read

"Most built-in types implement various options for such modifications, although some are only supported by the numeric types."

Further on, under "The available string presentation types are:"
I think "``'s'`` String format. This is the default type for strings and may be omitted." should have 'and other non-numeric types ' inserted after strings. New patch i13790b.diff attached

The point of these additional changes is to make it clearer that the default formatting of non-number, non-string objects is to call str() and then apply the options to the resulting string. That makes something like
>>> format(range(5), '-^20s') # same with object.__format__(), 3.3.0a0
'----range(0, 5)-----'
predictable and comprehensible.

I agree with not making a temporary change (but see below ;-).

But it seems that the 3.4 message should at least be
"numeric format string passed to object.__format__" or
"format string with number-only options passed to object.__format__" or
"object.__format__ cannot handle number-only options"
as string formats work fine and, I presume, are not deprecated (?).

However, if the new ValueError message did not specify object.__format__ (which could still be confusing, even if more accurate), the change could be make now. For instance
'Numeric option 'd' for non-number object'.
It would not really matter if it is later raised in object.__format__ instead of str.__format__. I believe *all* of the format codes 'unknown' to str (and by extension, by default, to all other non-number types) *are* number codes.

----------
assignee: docs at python -> terry.reedy
Added file: http://bugs.python.org/file24290/i13790b.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13790>
_______________________________________

From report at bugs.python.org  Sun Jan 22 04:43:38 2012
From: report at bugs.python.org (Paul McMillan)
Date: Sun, 22 Jan 2012 03:43:38 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2Webjv+mEHt9UrQOmVq82+NHsDW3zuyLM-rnqVxhD0LJQ@mail.gmail.com>
Message-ID: <CAO_YWRUTA2zpFFX8ER8fnPORYjgVNAc7ZEy5B-fz5ZHzAxY=uQ@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

On Sat, Jan 21, 2012 at 3:47 PM, Alex Gaynor <report at bugs.python.org> wrote:
> I'm able to put N pieces of data into the database on successive requests,
> but then *rendering* that data puts it in a dictionary, which renders that
> page unviewable by anyone.

This and the problems Frank mentions are my primary concerns about the
counting approach. Without the original suggestion of modifying the
hash and continuing without an exception (which has its own set of
problems), the "valid data python can't process" problem is a pretty
big one. Allowing attackers to poison interactions for other users is
unacceptable.

The other thing I haven't seen mentioned yet is that while it is true
that most web applications do have robust error handling to produce
proper 500s, an unexpected error will usually result in restarting the
server process - something that can carry significant weight by
itself. I would consider it a serious problem if every attack request
required a complete application restart, a la original cgi.

I'm strongly in favor of randomization. While there are many broken
applications in the wild that depend on dictionary ordering, if we
ship with this feature disabled by default for security and bugfix
branches, and enable it for 3.3, users can opt-in to protection as
they need it and as they fix their applications. Users who have broken
applications can still safely apply the security fix (without even
reading the release notes) because it won't change the default
behavior. Distro managers can make an appropriate choice for their
user base. Most importantly, it negates the entire "compute once,
attack everywhere" class of collision problems, even if we haven't
explicitly discovered them.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 05:09:32 2012
From: report at bugs.python.org (Meador Inge)
Date: Sun, 22 Jan 2012 04:09:32 +0000
Subject: [issue13783] Clean up PEP 380 C API additions
In-Reply-To: <1326521987.14.0.4759346281.issue13783@psf.upfronthosting.co.za>
Message-ID: <1327205372.09.0.568705477007.issue13783@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

'PyStopIteration_Create' is just a trivial wrapper:

PyObject *
PyStopIteration_Create(PyObject *value)
{
    return PyObject_CallFunctionObjArgs(PyExc_StopIteration, value, NULL);
}

It is not needed.

As for 'PyGen_FetchStopIterationValue', does it really need to be public?  It is trivial to make it private because all calls to it are in 'genobject.c'.  However, I am not sure if there is a strong use case for having it public.

----------
nosy: +meador.inge
stage:  -> needs patch
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13783>
_______________________________________

From report at bugs.python.org  Sun Jan 22 05:29:29 2012
From: report at bugs.python.org (Denver Coneybeare)
Date: Sun, 22 Jan 2012 04:29:29 +0000
Subject: [issue11551] test_dummy_thread.py test coverage improvement
In-Reply-To: <1300167076.06.0.564640850905.issue11551@psf.upfronthosting.co.za>
Message-ID: <1327206569.54.0.0936875712238.issue11551@psf.upfronthosting.co.za>


Denver Coneybeare <denver.coneybeare at gmail.com> added the comment:

I've looked at the review (thanks for the review) and can submit an updated patch.  I don't have the Python source code pulled down to my PC anymore so it might take a week or two before I'm able to update the patch and test it out.  I imagine that's not too much of a problem though :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11551>
_______________________________________

From report at bugs.python.org  Sun Jan 22 06:06:34 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 22 Jan 2012 05:06:34 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <E1RopdR-0004IB-5y@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 754c2eb0a92c by Gregory P. Smith in branch '3.2':
Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix.
http://hg.python.org/cpython/rev/754c2eb0a92c

New changeset 7d4658a8de96 by Gregory P. Smith in branch 'default':
Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix.
http://hg.python.org/cpython/rev/7d4658a8de96

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 22 07:27:54 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Sun, 22 Jan 2012 06:27:54 +0000
Subject: [issue8052] subprocess close_fds behavior should only close open fds
In-Reply-To: <1267672679.06.0.725064778002.issue8052@psf.upfronthosting.co.za>
Message-ID: <1327213674.35.0.972805853091.issue8052@psf.upfronthosting.co.za>


Gregory P. Smith <greg at krypto.org> added the comment:

For FreeBSD, Python 3.2 and 3.3 now check to see if /dev/fd is valid.  Be sure and "mount -t fdescfs none /dev/fd" on FreeBSD if you want faster subprocess launching.  Run a FreeBSD buildbot?  Please do it!

For Python 3.1 the fix for #13788 would fix this, but I believe 3.1 is in security fix only mode at this point so we're not going to backport that os.closerange change that far.

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8052>
_______________________________________

From report at bugs.python.org  Sun Jan 22 07:58:17 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 22 Jan 2012 06:58:17 +0000
Subject: [issue13814] Document why generators don't support the context
	management protocol
In-Reply-To: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>
Message-ID: <1327215497.25.0.0529876633689.issue13814@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Generators deliberately don't support the context management protocol. This is so that they raise an explicit TypeError or AttributeError (pointing out that __exit__ is missing) if you leave out the @contextmanager decorator when you're using a generator to write an actual context manager.

Generators supporting the context management protocol natively would turn that into a far more subtle (and confusing) error: your code would silently fail to invoke the generator body.

Ensuring this common error remains easy to detect is far more important than making it easier to invoke close() on a generator object (particularly when contextlib.closing() already makes that very easy).

----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python
stage: test needed -> needs patch
status: pending -> open
title: Generators as context managers. -> Document why generators don't support the context management protocol

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Sun Jan 22 10:24:15 2012
From: report at bugs.python.org (Thomas Dybdahl Ahle)
Date: Sun, 22 Jan 2012 09:24:15 +0000
Subject: [issue457066] pow(a,b,c) should accept b<0
Message-ID: <1327224255.45.0.632221812206.issue457066@psf.upfronthosting.co.za>


Thomas Dybdahl Ahle <lobais at gmail.com> added the comment:

For anyone who finds this through google,
if you are finding the inverse mod a prime, you can use fermats little theorem: pow(a, -1, mod) = pow(a, a-2, mod).
(You also need that mod doesn't divide a).

----------
nosy: +Thomas.Dybdahl.Ahle

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue457066>
_______________________________________

From report at bugs.python.org  Sun Jan 22 11:03:47 2012
From: report at bugs.python.org (July Tikhonov)
Date: Sun, 22 Jan 2012 10:03:47 +0000
Subject: [issue13835] whatsnew/3.3 misspelling/mislink
Message-ID: <1327226626.95.0.153685106815.issue13835@psf.upfronthosting.co.za>


New submission from July Tikhonov <july.tikh at gmail.com>:

1) Paragraph describing range() comparison links to issue13021.
This issue seems unrelated. It should be issue13201.

2) Paragraph describing of unicode_internal codec, mentions
"(utf-16-le or utf-16-le)"
and
"(utf-32-le or utf-32-le)".
It should be
"(utf-16-le or utf-16-be)"
and
"(utf-32-le or utf-32-be)"
respectively.

----------
assignee: docs at python
components: Documentation
files: whatsnew-3.3-misspelling.diff
keywords: patch
messages: 151765
nosy: docs at python, july
priority: normal
severity: normal
status: open
title: whatsnew/3.3 misspelling/mislink
versions: Python 3.3
Added file: http://bugs.python.org/file24291/whatsnew-3.3-misspelling.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13835>
_______________________________________

From report at bugs.python.org  Sun Jan 22 11:11:42 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Sun, 22 Jan 2012 10:11:42 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327227102.74.0.949739739162.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Terry, I agree with you on having *i*th instead of *i*-th. The fact that "i" is written in italics eliminates the need of a hyphen.

Justin, can I ask you to make a new patch which fixes "key-function" to "key function" and *i*'th to *i*th

This is the last decision. Also, if anyone of you can, please then just incorporate that final patch that Justin will make. Thanks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 11:22:01 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sun, 22 Jan 2012 10:22:01 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327227721.84.0.231542669654.issue13816@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

> This is the last decision. Also, if anyone of you can, please then just > incorporate that final patch that Justin will make. Thanks.

Stop acting like a manager. A while ago a person had his account
disabled for constantly bumping up issues and telling other people
what to do.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 11:39:21 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Sun, 22 Jan 2012 10:39:21 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327228761.25.0.0102515940117.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Yeah,?I?guess?I?was?kind?of?rude.?Sorry?about?that.?I?think?my?proposal?is?acceptable.?What?do?you?think?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:19:23 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 22 Jan 2012 11:19:23 +0000
Subject: [issue13835] whatsnew/3.3 misspelling/mislink
In-Reply-To: <1327226626.95.0.153685106815.issue13835@psf.upfronthosting.co.za>
Message-ID: <E1RovSD-0000AB-FS@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8a38bbf92048 by Sandro Tosi in branch 'default':
Issue #13835: fixes to What's new 3.3; patch by July Tikhonov
http://hg.python.org/cpython/rev/8a38bbf92048

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13835>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:19:53 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 22 Jan 2012 11:19:53 +0000
Subject: [issue13835] whatsnew/3.3 misspelling/mislink
In-Reply-To: <1327226626.95.0.153685106815.issue13835@psf.upfronthosting.co.za>
Message-ID: <1327231193.39.0.204446472017.issue13835@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Thanks July, I've just committed your patches!

----------
nosy: +sandro.tosi
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13835>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:40:12 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 22 Jan 2012 11:40:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327232413.0.0.62695134111.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24222/random-6.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:40:16 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 22 Jan 2012 11:40:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327232416.75.0.870979409398.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24254/random-fix_tests.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:40:30 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 22 Jan 2012 11:40:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327232430.09.0.51838396251.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24253/random-7.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:40:31 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 22 Jan 2012 11:40:31 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327232431.24.0.260794158039.issue13703@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24198/random-5.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 22 12:45:37 2012
From: report at bugs.python.org (olivier)
Date: Sun, 22 Jan 2012 11:45:37 +0000
Subject: [issue13836] Define key failed
Message-ID: <1327232737.47.0.872325277438.issue13836@psf.upfronthosting.co.za>


New submission from olivier <python at noetika.com>:

Hi,

I tried to define new key in python idle and then python 2.5.1 failed to launch. 

What I did : I defined a new key, applied and changed my mind, removed my key set named 'ole'.

I launched cmd
C:\Python25\Lib\idlelib>..\..\python idle.py

got dozens of messages :

Warning: configHandler.py - IdleConf.GetOption -
problem retrieving configration option 'redo'
from section 'ole'.
returning default value: ''

Finally, this opened python idle so I went again to configure idle/keys and saw that my key set was still there. I selected again "Delete custom Key set" and now it's ok.

----------
messages: 151771
nosy: olivier57
priority: normal
severity: normal
status: open
title: Define key failed
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13836>
_______________________________________

From report at bugs.python.org  Sun Jan 22 14:09:47 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 22 Jan 2012 13:09:47 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <CACGt9ykO9q9yHVCnB8-MdC4K79mzn63G38im-N5MVdzYjvsmsQ@mail.gmail.com>
Message-ID: <CAH_1eM2JBA5EvptU93gwhs_FNr_iE09=hgMKzRfeUxYGXZN2VA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> As a Python user (and not a committer), I disagree.
>
> As an user, I don't care too much where the function should be placed
> (although I believe os or sys are sensible choices). What I do care is
> that I want a extremely simple function that will "just work". Don't
> make me add code for handling all the extra cases, such code should be
> inside the function.

For what it's worth, as a Python committer, I agree with you.
Python is a very high level language, and I think the standard library
should be as natural and offer the same productivity gain as the core
language does.
Exposing to the user a mere wrapper around a syscall/library just
doesn't make sense to me. Sure, it should be made available for those
who want/need to do low-level system programming (and I'm one of
those), but the vast majority of users want something higher level
than the POSIX/Windows library.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Sun Jan 22 14:11:20 2012
From: report at bugs.python.org (Dan kamp)
Date: Sun, 22 Jan 2012 13:11:20 +0000
Subject: [issue13829] exception error
In-Reply-To: <1327165402.06.0.913738022612.issue13829@psf.upfronthosting.co.za>
Message-ID: <A6817460-A9F9-4E4E-A9F4-6178C193149C@roontoon.com>


Dan kamp <bitbucket at roontoon.com> added the comment:

I have received this from the macpython listserv it that helps. Would really like to find this issue.

>From the traceback, it appears that there is a problem with Python's
_scproxy module; that's an internal helper C module that provides an
interface to the OS X System Configuration framework to access Internet
proxy configurations for the urllib module. You should open an issue
for this at bugs.python.org. Please include the original crash report
traceback.

On Jan 21, 2012, at 12:03 PM, Brett Cannon wrote:

> 
> Brett Cannon <brett at python.org> added the comment:
> 
> Then I'm going to assume the bug lies with Moviegrabber doing something wrong and it isn't Python's direct fault.
> 
> ----------
> resolution:  -> invalid
> status: open -> closed
> 
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13829>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Sun Jan 22 14:17:18 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 22 Jan 2012 13:17:18 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327238238.8.0.0314095003203.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

?ric, are you still planning to work on this? Otherwise I'll make a first pass at doing the split into 3 sections (as per my earlier comment) and implementing some of Terry's suggestions.

Linked Hg repo is a 2.7 based feature branch where I'll be publishing my changes as I make them.

----------
hgrepos: +106

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sun Jan 22 15:25:39 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 22 Jan 2012 14:25:39 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327242339.2.0.469844130566.issue13816@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Justin, if you do a new patch, put both changes in one .diff.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 15:27:22 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Sun, 22 Jan 2012 14:27:22 +0000
Subject: [issue6727] ImportError when package is symlinked on Windows
In-Reply-To: <1250608383.93.0.471187637916.issue6727@psf.upfronthosting.co.za>
Message-ID: <1327242442.52.0.596927141216.issue6727@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

After hearing back from Microsoft support (and by proxy, the Visual Studio development team), it is clear that this issue is very low priority for them (they see it as having trivial business impact), so we cannot expect it to be fixed in the upstream libraries anytime soon.

This response essentially means that if Python wants to support these symbolically-linked directories, it cannot use stat/wstat on Windows and must use instead the Windows APIs.

As seen in issue13412, this bug not only affects import.c, but also affects posixmodule (such as with os.listdir). After addressing this issue for import.c, it will probably be necessary to also address it for other parts of Python that use stat/wstat (or other calls that depend on those calls).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6727>
_______________________________________

From report at bugs.python.org  Sun Jan 22 16:33:10 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sun, 22 Jan 2012 15:33:10 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327246390.06.0.421151799401.issue4966@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

?ric is without Internet till the end of the month, so I think it's OK if you go ahead and start working on this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sun Jan 22 16:51:35 2012
From: report at bugs.python.org (Meador Inge)
Date: Sun, 22 Jan 2012 15:51:35 +0000
Subject: [issue13814] Document why generators don't support the context
	management protocol
In-Reply-To: <1326883396.25.0.884733470267.issue13814@psf.upfronthosting.co.za>
Message-ID: <1327247495.56.0.777253196716.issue13814@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13814>
_______________________________________

From report at bugs.python.org  Sun Jan 22 18:05:55 2012
From: report at bugs.python.org (vr gamer)
Date: Sun, 22 Jan 2012 17:05:55 +0000
Subject: [issue6792] Distutils-based installer does not detect 64bit versions
	of Python
In-Reply-To: <1251449540.11.0.0670330819334.issue6792@psf.upfronthosting.co.za>
Message-ID: <1327251955.78.0.547432775167.issue6792@psf.upfronthosting.co.za>


vr gamer <vrgamerz at gmail.com> added the comment:

I'm not certain that I agree with the argument used to justify keeping this as a 'normal' priority issue. Apparently, since it doesn't effect the entire python community and being as there is no readily available solution, the decision is to treat it as a minor problem. Were one to apply the same logic to, what say, epidemiology, then I suppose the lack of a vaccine for HIV would not be a particularly pressing issue either. 

If priority escalation is out of the question, can we at least have an update? After three years of workarounds, I'm beginning to suspect that the idea is to wait until the effected packages become deprecated and then declare 'solved!'.

----------
nosy: +vr.gamer

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6792>
_______________________________________

From report at bugs.python.org  Sun Jan 22 18:32:31 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 22 Jan 2012 17:32:31 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327253551.19.0.0924446459025.issue13812@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> But I observe that unless I explicitly flush stdout and stderr before 
> terminating, the output is lost entirely, even if the exit is not 
> abnormal. This isn't the desired behavior, is it?

Indeed that's a bit surprising. Which Python version are you using? Under Unix or Windows?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Sun Jan 22 18:55:47 2012
From: report at bugs.python.org (=?utf-8?q?Micha=C5=82_Michalski?=)
Date: Sun, 22 Jan 2012 17:55:47 +0000
Subject: [issue9625] argparse: Problem with defaults for variable nargs
In-Reply-To: <1282036744.37.0.964764969496.issue9625@psf.upfronthosting.co.za>
Message-ID: <1327254947.06.0.862779926342.issue9625@psf.upfronthosting.co.za>


Micha? Michalski <python at michalski.im> added the comment:

Maybe it will sound strange, but what is this task REALLY about? I mean - I can see two problems here, but no clear information about which problem is a real problem and - if it is - what is the expected behavior.

Problems I can see are:
1) Type of returned value changes depending on the value source (list for user provided or list for default)
2) It's impossible to set list as 'default'

I understand that this task concentrates on 1st one, however changing behavior described in 2nd would - in my oppinion - fix 1st one too - am I right? User would "define" the returned type by defining the 'default' as a list or string. 

But - if we assume that we only discuss 1st problem here - what is the expected behavior? Provided workaround is suggesting that we expect string, but - basing on current nargs behavior and "intuition" - I'd rather expect to get a list, when I define nargs with "*" or "+". This sounds "natural" for me.

Could someone explain me the problem and the expected behavior in a clear way?

Sorry if I'm not clear, but it's my first post here and maybe I've missed something in the description or I assume something that is not true (especially that the task is quite old... ;) ).

----------
nosy: +regis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9625>
_______________________________________

From report at bugs.python.org  Sun Jan 22 18:58:04 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Sun, 22 Jan 2012 17:58:04 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327255084.11.0.555630044333.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

On Windows, the problem appears under Python 3.2.2 and 3.1.3, but not under 2.7.1. On Linux, I have not reproduced the problem on versions 2.6.3, 2.7.2, 3.1.1, or 3.2.2.

So to summarize:

  - It seems there should be a stderr flush call on the line I indicated, for symmetry with the surrounding code.
  - Even without this line, it should still be flushed automatically upon child process exit, but this doesn't happen under Windows and Python 3.x.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Sun Jan 22 19:05:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 22 Jan 2012 18:05:46 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1327255084.11.0.555630044333.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327255424.3382.42.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

Le dimanche 22 janvier 2012 ? 17:58 +0000, Jon Brandvein a ?crit :
> Jon Brandvein <jon.brandvein at gmail.com> added the comment:
> 
> On Windows, the problem appears under Python 3.2.2 and 3.1.3, but not
> under 2.7.1. On Linux, I have not reproduced the problem on versions
> 2.6.3, 2.7.2, 3.1.1, or 3.2.2.

Thanks.

>   - Even without this line, it should still be flushed automatically
> upon child process exit, but this doesn't happen under Windows and
> Python 3.x.

Yes, that's what surprises me. There's no reason for stderr not to be
flushed implicitly at process end, since that's part of sys.stderr's
destructor, which should be called at shutdown.

That said, it certainly doesn't harm to add a flush() call there.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Sun Jan 22 19:10:07 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 22 Jan 2012 18:10:07 +0000
Subject: [issue13837] test_shutil fails with symlinks enabled under Windows
Message-ID: <1327255807.39.0.63998442373.issue13837@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

This happens when symlinks are enabled under Windows. This doesn't affect any buildbots since they don't run the tests as administrator (or they are not recent enough to have symlink support):


======================================================================
FAIL: test_copymode_follow_symlinks (test.test_shutil.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\t\cpython\lib\test\test_shutil.py", line 193, in test_copymode_follow
_symlinks
    self.assertEqual(os.stat(src).st_mode, os.stat(dst).st_mode)
AssertionError: 33206 != 33060

======================================================================
FAIL: test_move_dangling_symlink (test.test_shutil.TestMove)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\t\cpython\lib\test\test_shutil.py", line 58, in wrap
    return func(*args, **kwargs)
  File "C:\t\cpython\lib\test\test_shutil.py", line 1136, in test_move_dangling_
symlink
    self.assertEqual(os.path.realpath(src), os.path.realpath(dst_link))
AssertionError: 'c:\\users\\antoine\\appdata\\local\\temp\\tmp8bl4eu\\baz' != 'c
:\\users\\antoine\\appdata\\local\\temp\\tmplc6h2h\\quux'
- c:\users\antoine\appdata\local\temp\tmp8bl4eu\baz
?                                        -- ^^ ^^^^
+ c:\users\antoine\appdata\local\temp\tmplc6h2h\quux
?                                         ^^^^^^^ ^^

----------
components: Library (Lib), Tests
messages: 151783
nosy: brian.curtin, hynek, pitrou, tim.golden
priority: normal
severity: normal
stage: needs patch
status: open
title: test_shutil fails with symlinks enabled under Windows
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13837>
_______________________________________

From report at bugs.python.org  Sun Jan 22 19:10:39 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 22 Jan 2012 18:10:39 +0000
Subject: [issue13837] test_shutil fails with symlinks enabled under Windows
In-Reply-To: <1327255807.39.0.63998442373.issue13837@psf.upfronthosting.co.za>
Message-ID: <1327255839.06.0.771520699659.issue13837@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

(tests were added in #12715 and #9993)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13837>
_______________________________________

From report at bugs.python.org  Sun Jan 22 19:14:42 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 22 Jan 2012 18:14:42 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1327256082.83.0.102389504228.issue13772@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Another issue with the detection scheme is that it's not atomic: there can be a race condition between the GetFileAttributesExW() and CreateSymbolicLinkW() calls.

I propose applying the following patch to 3.2 and 3.3 (+ doc fix, not included in the patch).

----------
keywords: +patch
stage:  -> patch review
Added file: http://bugs.python.org/file24292/windirlink.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Sun Jan 22 20:48:15 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sun, 22 Jan 2012 19:48:15 +0000
Subject: [issue13520] Patch to make pickle aware of __qualname__
In-Reply-To: <1322843353.56.0.297081736881.issue13520@psf.upfronthosting.co.za>
Message-ID: <1327261695.62.0.810221703039.issue13520@psf.upfronthosting.co.za>


Changes by Hynek Schlawack <hs at ox.cx>:


----------
nosy: +hynek

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13520>
_______________________________________

From report at bugs.python.org  Sun Jan 22 20:49:39 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Sun, 22 Jan 2012 19:49:39 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327261779.79.0.573736404464.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I fixed Justin's patch. Anyone cares to incorporate it?

----------
Added file: http://bugs.python.org/file24293/fixed patch final.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 21:24:50 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 22 Jan 2012 20:24:50 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327263890.28.0.862699539561.issue13816@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Sorry, but the patch introduces two markup errors:

- *i*th is invalid reST, it needs to be *i*\ th
- you broke the table markup (the vertical lines must be aligned)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Sun Jan 22 21:31:48 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 22 Jan 2012 20:31:48 +0000
Subject: [issue13834] In help(bytes.strip) there is no info about leading
	ASCII whitespace
In-Reply-To: <1327119429.45.0.897722607985.issue13834@psf.upfronthosting.co.za>
Message-ID: <E1Rp44n-0005Xc-Tz@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 960d93deb8c2 by Georg Brandl in branch '3.2':
Fix #13834: strip() strips leading and trailing whitespace.
http://hg.python.org/cpython/rev/960d93deb8c2

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13834>
_______________________________________

From report at bugs.python.org  Sun Jan 22 21:32:03 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 22 Jan 2012 20:32:03 +0000
Subject: [issue13834] In help(bytes.strip) there is no info about leading
	ASCII whitespace
In-Reply-To: <1327119429.45.0.897722607985.issue13834@psf.upfronthosting.co.za>
Message-ID: <1327264323.32.0.547252098153.issue13834@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Fixed, thanks!

----------
nosy: +georg.brandl
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13834>
_______________________________________

From report at bugs.python.org  Sun Jan 22 23:25:30 2012
From: report at bugs.python.org (py.user)
Date: Sun, 22 Jan 2012 22:25:30 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
Message-ID: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

http://docs.python.org/py3k/library/string.html#format-specification-mini-language

The '#' option:
"For floats, complex and Decimal the alternate form causes the result of the conversion to always contain a decimal-point character, even if no digits follow it. Normally, a decimal-point character appears in the result of these conversions only if a digit follows it. In addition, for 'g' and 'G' conversions, trailing zeros are not removed from the result."

1)
>>> import decimal
>>> '{0:#.5g}'.format(1.5)
'1.5000'
>>> '{0:.5f}'.format(decimal.Decimal(1.5))
'1.50000'
>>> '{0:.5g}'.format(decimal.Decimal(1.5))
'1.5'
>>> '{0:#.5g}'.format(decimal.Decimal(1.5))
'1.5'
>>>

no zeros with "#"

2)
>>> import decimal
>>> '{0:#.5g}'.format(decimal.Decimal('1.500000000000'))
'1.5000'
>>> '{0:.5g}'.format(decimal.Decimal('1.500000000000'))
'1.5000'
>>>

zeros without "#"

----------
components: Interpreter Core, Library (Lib)
messages: 151790
nosy: py.user
priority: normal
severity: normal
status: open
title: In str.format "{0:#.5g}" for decimal.Decimal doesn't print trailing zeros
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Sun Jan 22 23:53:43 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sun, 22 Jan 2012 22:53:43 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327272823.72.0.800045282102.issue13838@psf.upfronthosting.co.za>


Eric V. Smith <eric at trueblade.com> added the comment:

See issue #7098 for a discussion.

I propose to close this issue.

----------
nosy: +eric.smith, mark, skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 00:12:57 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Sun, 22 Jan 2012 23:12:57 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327273977.15.0.790508794189.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

I've been looking over this package some more, and in particular, /Lib/multiprocessing/forking.py. There's plenty I don't understand, and I do have questions, if you would be willing to indulge me.

I see that both the unix and windows codepaths define an "exit" alias for terminating the child process without performing any cleanup. On unix, this is os._exit (though it calls it directly in Popen.__init__() instead of using the alias). On Windows, it is the win32 ExitProcess() function. A quick check confirms that replacing this windows alias with "exit = sys.exit" causes flushing to occur.

So my main question is: What's wrong with using sys.exit()? I would speculate it's either because there's shared state between child and parent, or to avoid propagating SystemExit through user code in the case freeze_support() was used.

If forking.py is to terminate directly via the OS, I think it's forking.py's responsibility to flush stdout and stderr in main() on the Windows side, the way it does in Popen.__init__() on the unix side.

I also want to point out that the sys.exit() in freeze_support() is unreachable due to the exit() in main().

So it no longer surprises me that the output is not being flushed in Python 3 under windows. What surprises me is that it *is* flushed in Python 2. I remember hearing something about differences between 2 and 3 in how they handle buffering, so I'm investigating this for my own curiosity.

Incidentally, Python 2 still flushes when I remove newlines from my output text, so line buffering doesn't seem to be impacting my observations.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 00:32:34 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Sun, 22 Jan 2012 23:32:34 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327275154.18.0.12832357498.issue13838@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
nosy: +mark.dickinson -mark

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 00:37:31 2012
From: report at bugs.python.org (py.user)
Date: Sun, 22 Jan 2012 23:37:31 +0000
Subject: [issue7098] g formatting for decimal types should always strip
	trailing zeros.
In-Reply-To: <1255198932.03.0.979291900289.issue7098@psf.upfronthosting.co.za>
Message-ID: <1327275451.11.0.451742410695.issue7098@psf.upfronthosting.co.za>


Changes by py.user <port139 at yandex.ru>:


----------
nosy: +py.user

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7098>
_______________________________________

From report at bugs.python.org  Mon Jan 23 00:48:28 2012
From: report at bugs.python.org (py.user)
Date: Sun, 22 Jan 2012 23:48:28 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327276108.84.0.909539802799.issue13838@psf.upfronthosting.co.za>


py.user <port139 at yandex.ru> added the comment:

my question is about the "#" option
it is described as working with Decimal but it doesn't work with Decimal

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 01:22:37 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 23 Jan 2012 00:22:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327278157.52.0.944871798107.issue13703@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

@dmalcolm: How did you chose Py_MAX_AVERAGE_PROBES_PER_INSERT=32? Did you try your patch on applications like the test suite of Django or Twisted?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 04:38:17 2012
From: report at bugs.python.org (Matt Joiner)
Date: Mon, 23 Jan 2012 03:38:17 +0000
Subject: [issue13839] -m pstats should combine all the profiles given as
	arguments
Message-ID: <1327289897.02.0.607924329906.issue13839@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

Frequently when profiling multiple threads, I need to combine several dump stat files. Currently -m pstats reads the profiling data at only the first path given. It should merge all the profiling data from all the paths given.

$ python3.3 -m pstats prof/5506-7f00f*

Minimalist patch attached.

----------
components: Library (Lib)
files: m-pstats-merge-profiles.patch
keywords: patch
messages: 151795
nosy: anacrolix
priority: normal
severity: normal
status: open
title: -m pstats should combine all the profiles given as arguments
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file24294/m-pstats-merge-profiles.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13839>
_______________________________________

From report at bugs.python.org  Mon Jan 23 04:42:42 2012
From: report at bugs.python.org (Matt Joiner)
Date: Mon, 23 Jan 2012 03:42:42 +0000
Subject: [issue12684] profile does not dump stats on exception like cProfile
	does
In-Reply-To: <1312335798.44.0.743453347572.issue12684@psf.upfronthosting.co.za>
Message-ID: <1327290162.0.0.334313911128.issue12684@psf.upfronthosting.co.za>


Changes by Matt Joiner <anacrolix at gmail.com>:


----------
resolution: works for me -> 
status: languishing -> open
versions: +Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12684>
_______________________________________

From report at bugs.python.org  Mon Jan 23 04:48:51 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 23 Jan 2012 03:48:51 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2Webjv+mEHt9UrQOmVq82+NHsDW3zuyLM-rnqVxhD0LJQ@mail.gmail.com>
Message-ID: <1327290494.4992.298.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-21 at 23:47 +0000, Alex Gaynor wrote:
> Alex Gaynor <alex.gaynor at gmail.com> added the comment:
> 
> On Sat, Jan 21, 2012 at 5:42 PM, Gregory P. Smith <report at bugs.python.org>wrote:
> 
> >
> > Gregory P. Smith <greg at krypto.org> added the comment:
> >
> > On Sat, Jan 21, 2012 at 2:45 PM, Antoine Pitrou <report at bugs.python.org>
> > wrote:
> > >
> > > Antoine Pitrou <pitrou at free.fr> added the comment:
> > >
> > >> You said above that it should be hardcoded; if so, how can it be changed
> > >> at run-time from an environment variable?  Or am I misunderstanding.
> > >
> > > You're right, I used the wrong word. I meant it should be a constant
> > > independently of the dict size. But, indeed, not hard-coded in the
> > > source.
> > >
> > >> > > BTW, presumably if we do it, we should do it for sets as well?
> > >> >
> > >> > Yeah, and use the same env var / sys function.
> > >>
> > >> Despite the "DICT" in the title?  OK.
> > >
> > > Well, dict is the most likely target for these attacks.
> > >
> >
> > While true I wouldn't make that claim as there will be applications
> > using a set in a vulnerable manner. I'd prefer to see any such
> > environment variable name used to configure this behavior not mention
> > DICT or SET but just say HASHTABLE.  That is a much better bikeshed
> > color. ;)
> >
> > I'm still in the hash seed randomization camp but I'm finding it
> > interesting all of the creative ways others are trying to "solve" this
> > problem in a way that could be enabled by default in stable versions
> > regardless. :)
> >
> > -gps
> >
> > ----------
> >
> > _______________________________________
> > Python tracker <report at bugs.python.org>
> > <http://bugs.python.org/issue13703>
> > _______________________________________
> >
> 
> I'm a little slow, so bear with me, but David, does this counting scheme in
> any way address the issue of:
> 
> I'm able to put N pieces of data into the database on successive requests,
> but then *rendering* that data puts it in a dictionary, which renders that
> page unviewable by anyone.

It doesn't address this issue - though if the page is taking many hours
to render, is that in practice less unviewable that everyone getting an
immediate exception with (perhaps) a useful error message?

Unfortunately, given the current scale factor, my patch may make it
worse: in my tests, this approach rejected malicious data much more
quickly than the old collision-counting one, which I thought was a good
thing - but then I realized that this means that an attacker adopting
the strategy you describe would have to do less work to trigger the
exception than to trigger the slowdown.  So I'm not convinced my
approach flies, and I'm leaning towards working on the hash
randomization patch rather than pursuing this.

I need sleep though, so I'm not sure the above is coherent
Dave

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 04:58:15 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Mon, 23 Jan 2012 03:58:15 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327291095.44.0.201690922715.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Some more information: When I write to a new file created by open(), all versions flush correctly. However, if I reassign sys.stdout to that file, Python 3.x does not (again, under Windows). I wonder what it is that causes these other files to flush.

(Note: I am testing by calling time.sleep() and inspecting the output file during and after the pause.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 05:04:43 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 23 Jan 2012 04:04:43 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327278157.52.0.944871798107.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327291448.4992.312.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I arbitrarily started with 50, and then decided a power of two would be
quicker when multiplying.  There wasn't any rigorous analysis behind the
choice of factor.

Though, as noted in msg151796, I've gone off this idea, since I think
the "protection" creates additional avenues of attack.

I think getting some kind of hash randomization patch into the hands of
users ASAP is the way forward here (even if disabled by default).

If we're going to support shipping backported versions of the hash
randomization patch with the randomization disabled, did we decide on a
way of enabling it?  If not, then I propose that those who want to ship
with it disabled by default standardize on (say):

  PYTHONHASHRANDOMIZATION

as an environment variable: if set to nonzero, it enables hash
randomization (reading the random seed as per the 3.3. patch, and
respecting the PYTHONHASHSEED variable if that's also set).  If set to
zero or not present, hash randomization is disabled.

Does that sound sane?

(we can't use PYTHONHASHSEED for this, since if a number is given, that
means "use this number", right?)

FWIW, I favor hash randomization in 2.* for PyStringObject,
PyUnicodeObject, PyBufferObject, and the 3 datetime classes in
Modules/_datetimemodule.c (see the implementation of generic_hash in
that file), but to not do it for the numeric types.

Sorry; I only tried it on the python test suite (and on a set of
reproducers for the DoS that I've written for RH's in-house test suite).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 07:38:41 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Mon, 23 Jan 2012 06:38:41 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327300721.85.0.421846539348.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

It turns out the file output was flushing due to garbage collection. When I created and held a global reference to it, it ceased to flush. Clearly, reassigning sys.stdout also held a reference to it. So it wasn't any kind of special sys.stdout-specific logic.

I tried using os.fsync to determine whether data was being saved in an OS-level buffer. But since the OS may be free to ignore fsync, it's kind of worthless for this purpose.

I also found that under Python 2.x, even a low-level exit like os._exit or multiprocessing.win32.ExitProcess, called from within a user-level function in the child, caused flushing.

My best guess is that
  1. Python 2.x is not buffering data at the Python level. I can't see how it could be and still flush it out when calling _exit().
  2. Python 3.x is buffering at the Python level, and the Python File object needs to be destroyed or explicitly flushed before the hard low-level exit() in forking.py.

The solutions I can think of for Python 3.x are:
  1. Replace "exit = win32.ExitProcess" with "exit = sys.exit". All outstanding file objects will be destroyed and flushed naturally as the interpreter is torn down.
  2. Add an explicit stdout/stderr flush where appropriate in forking.py and process.py, to ensure tracebacks get written and to match the unix behavior. Leave it to the user to worry about flushing their own streams.
  3. Continue to use win32.ExitProcess, but add some kind of mechanism for walking through all existing Python File objects and flushing/destroying them. This was a fleeting thought; it really amounts to reimplementing the behavior of destructing the Python interpreter.

I'd really like to hear if there are good reasons for why (1) isn't how it's done currently. I'd also like to hear an explanation of Python 2.x's buffering.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 08:53:08 2012
From: report at bugs.python.org (Vincent Pelletier)
Date: Mon, 23 Jan 2012 07:53:08 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
Message-ID: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>


New submission from Vincent Pelletier <plr.vincent at gmail.com>:

ctypes.create_string_buffer documentation[1] says init_or_size parameter should accept a string. As of 3.2, it raises:
>>> import ctypes
>>> ctypes.create_string_buffer('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/ctypes/__init__.py", line 59, in create_string_buffer
    buf.value = init
TypeError: str/bytes expected instead of str instance

It works fine as of 2.7 (and very probably any 2.x up to ctypes introduction):
>>> import ctypes
>>> ctypes.create_string_buffer('foo')
<ctypes.c_char_Array_4 object at 0x7fbdcb8b95f0>

[1] http://docs.python.org/py3k/library/ctypes.html#ctypes.create_string_buffer

Regards,
Vincent Pelletier

----------
components: ctypes
messages: 151800
nosy: vpelletier
priority: normal
severity: normal
status: open
title: create_string_buffer rejects str init_or_size parameter
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Mon Jan 23 08:58:50 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 23 Jan 2012 07:58:50 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <1327305530.67.0.110373507683.issue13840@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

It should only take bytes; "str" is Unicode in 3.x.

So the docs and the error message are wrong, the behavior is correct.  Reclassifying as a docs issue.

----------
assignee:  -> docs at python
components: +Documentation -ctypes
nosy: +docs at python, georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Mon Jan 23 09:27:50 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 23 Jan 2012 08:27:50 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327307270.36.0.256054191742.issue4966@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

+1 for splitting.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 23 09:40:45 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Mon, 23 Jan 2012 08:40:45 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327308045.49.0.810858379407.issue13838@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

IMO, the behaviour is fine;  it's the docs that are unclear.  The rules for Decimal are different mainly because trailing zeros have meaning for the Decimal type.  (Decimal('1.250') and Decimal('1.25') are two distinct Decimal objects, unlike float('1.250') and float('1.25').)

See also issue #7094.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 09:52:38 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Mon, 23 Jan 2012 08:52:38 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327308758.15.0.0336121333848.issue13838@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Ah no, I take it back.  I think (2) is fine---this is the usual preservation of trailing zeros where possible.  (1) needs to be fixed (and issue #7094 was left open waiting for this fix).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 09:58:54 2012
From: report at bugs.python.org (Catalin Iacob)
Date: Mon, 23 Jan 2012 08:58:54 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327309134.17.0.83038421233.issue6210@psf.upfronthosting.co.za>


Changes by Catalin Iacob <iacobcatalin at gmail.com>:


----------
nosy: +catalin.iacob

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Mon Jan 23 10:28:59 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 23 Jan 2012 09:28:59 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327310939.36.0.175388989146.issue4966@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

+1 for Nick's suggested breakout:

4.6 Sequence Types - list, tuple, range
4.7 Text Sequence Type - str
4.8 Binary Data Sequence Types - bytes, bytearray, memoryview

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 23 11:23:05 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 23 Jan 2012 10:23:05 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1327314185.64.0.854913166381.issue13793@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

I concur with Benjamin on all counts.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 23 11:40:57 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 23 Jan 2012 10:40:57 +0000
Subject: [issue13797] Allow objects implemented in pure Python to export PEP
	3118 buffers
In-Reply-To: <1326709751.32.0.965094376059.issue13797@psf.upfronthosting.co.za>
Message-ID: <1327315257.22.0.681871476268.issue13797@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13797>
_______________________________________

From report at bugs.python.org  Mon Jan 23 11:50:52 2012
From: report at bugs.python.org (=?utf-8?q?Micha=C5=82_M=2E?=)
Date: Mon, 23 Jan 2012 10:50:52 +0000
Subject: [issue9625] argparse: Problem with defaults for variable nargs
In-Reply-To: <1282036744.37.0.964764969496.issue9625@psf.upfronthosting.co.za>
Message-ID: <1327315852.08.0.793032363449.issue9625@psf.upfronthosting.co.za>


Micha? M. <python at michalski.im> added the comment:

Of course I've made a mistake:

"list for user provided or list for default"

should be:

"list for user provided or STRING for default"

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9625>
_______________________________________

From report at bugs.python.org  Mon Jan 23 12:10:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 11:10:53 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1327300721.85.0.421846539348.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327316927.3395.10.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I also found that under Python 2.x, even a low-level exit like
> os._exit or multiprocessing.win32.ExitProcess, called from within a
> user-level function in the child, caused flushing.

The difference is the following:
- Python 2.x uses C stdio (write() calls C fwrite(), flush() calls C
fflush(), etc.); buffering is managed by the libc and what you see at
shutdown is the behaviour of your platform's libc
- Python 3.x uses its own buffering mechanism; it flushes automatically
when the object destructor is called, but os._exit() bypasses all
destructors

Now why os._exit() is used. The problem with sys.exit() is that it's too
high-level: it merely raises a SystemExit exception. That exception can
be caught by upper levels in the code stack. When you use fork() and you
are in the child process, you don't want to give back control to the
calling function, especially if that function isn't fork()-aware (think
about what happens if that function writes to a file, both in the child
and the parent).

This happens for example when running multiprocessing's own test suite:
if forking.py used sys.exit(), the child's SystemExit exception would be
caught by the unittest framework, be interpreted as a test failure, and
the rest of the test suite would proceed... in both processes!

It is less obvious, however, why ExitProcess is used under Windows.
Windows doesn't use fork(), it launches a separate process from scratch.
Perhaps for "consistency" with the Unix behaviour.

>   2. Add an explicit stdout/stderr flush where appropriate in
> forking.py and process.py, to ensure tracebacks get written and to
> match the unix behavior. Leave it to the user to worry about flushing
> their own streams.

That's the simplest solution, and the least likely to break
compatibility with user code.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 12:20:09 2012
From: report at bugs.python.org (Vincent Pelletier)
Date: Mon, 23 Jan 2012 11:20:09 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <1327317609.57.0.95906574741.issue13840@psf.upfronthosting.co.za>


Vincent Pelletier <plr.vincent at gmail.com> added the comment:

Thanks for the quick reply.

FWIW, in 2.7 doc ctype.create_string_buffer is said to accept unicode objects as parameter. I don't use this personally, so I don't mind 3.x only working on bytes - and already fixed my code accordingly. It's just that I noticed this after your answer. Also, I didn't try to confirm if it actually works.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Mon Jan 23 12:38:03 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 11:38:03 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327318683.03.0.00960515143404.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Georg,?thanks?for?the?tip.?Is?there?any?difference?in?reST?between?*i*\ th?and?*i*\th??

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 12:48:35 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 23 Jan 2012 11:48:35 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327319315.65.0.567715535191.issue13816@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

You can check on the devguide the section about building the doc and see it yourself on the generated HTML.  While building you will also see all the warnings caused by invalid markup.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 14:07:26 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 23 Jan 2012 13:07:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327100080.4992.149.camel@surprise>
Message-ID: <4F1D5B87.8050907@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Dave Malcolm wrote:
> 
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
> 
> On Fri, 2012-01-06 at 12:52 +0000, Marc-Andre Lemburg wrote:
>> Marc-Andre Lemburg <mal at egenix.com> added the comment:
>>
>> Demo patch implementing the collision limit idea for Python 2.7.
>>
>> ----------
>> Added file: http://bugs.python.org/file24151/hash-attack.patch
>>
> 
> Marc: is this the latest version of your patch?

Yes. As mentioned in the above message, it's just a demo of how
the collision limit idea can be implemented.

> Whether or not we go with collision counting and/or adding a random salt
> to hashes and/or something else, I've had a go at updating your patch
> 
> Although debate on python-dev seems to have turned against the
> collision-counting idea, based on flaws reported by Frank Sievertsen
> http://mail.python.org/pipermail/python-dev/2012-January/115726.html
> it seemed to me to be worth at least adding some test cases to flesh out
> the approach.  Note that the test cases deliberately avoid containing
> "hostile" data.

Martin's example is really just a red herring: it doesn't matter
where the hostile data originates or how it gets into the application.
There are many ways an attacker can get the O(n^2) worst case
timing triggered.

Frank's example is an attack on the second possible way to
trigger the O(n^2) behavior. See msg150724 further above where I
listed the two possibilities:

"""
An attack can be based on trying to find many objects with the same
hash value, or trying to find many objects that, as they get inserted
into a dictionary, very often cause collisions due to the collision
resolution algorithm not finding a free slot.
"""

My demo patch only addresses the first variant. In order to cover
the second variant as well, you'd have to count and limit the
number of iterations in the perturb for-loop of the lookdict()
functions where the hash value of the slot does not match the
key's hash value.

Note that the second variant is both a lot less likely to trigger
(due to the dict getting resized on a regular basis) and the
code involved a lot faster than the code for the first
variant (which requires a costly object comparison), so the
limit for the second variant would have to be somewhat higher
than for the first.

BTW: The collision counting patch chunk for the string dicts in my
demo patch is wrong. I've attached a corrected version. In the
original patch it was counting both collision variants with the
same counter and limit.

----------
Added file: http://bugs.python.org/file24295/hash-attack-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
Index: Objects/dictobject.c
===================================================================
--- Objects/dictobject.c	(revision 88933)
+++ Objects/dictobject.c	(working copy)
@@ -9,6 +9,8 @@
 
 #include "Python.h"
 
+/* Maximum number of allowed hash collisions. */
+#define Py_MAX_DICT_COLLISIONS 1000
 
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
@@ -327,6 +329,7 @@
     register PyDictEntry *ep;
     register int cmp;
     PyObject *startkey;
+    size_t collisions;
 
     i = (size_t)hash & mask;
     ep = &ep0[i];
@@ -361,6 +364,7 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
@@ -387,6 +391,11 @@
                  */
                 return lookdict(mp, key, hash);
             }
+	    if (++collisions > Py_MAX_DICT_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many hash collisions");
+		return NULL;
+	    }
         }
         else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
@@ -413,6 +422,7 @@
     register size_t mask = (size_t)mp->ma_mask;
     PyDictEntry *ep0 = mp->ma_table;
     register PyDictEntry *ep;
+    size_t collisions;
 
     /* Make sure this function doesn't have to handle non-string keys,
        including subclasses of str; e.g., one reason to subclass
@@ -439,17 +449,24 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
         if (ep->me_key == NULL)
             return freeslot == NULL ? ep : freeslot;
-        if (ep->me_key == key
-            || (ep->me_hash == hash
-            && ep->me_key != dummy
-            && _PyString_Eq(ep->me_key, key)))
+        if (ep->me_key == key)
             return ep;
-        if (ep->me_key == dummy && freeslot == NULL)
+        if (ep->me_hash == hash && ep->me_key != dummy) {
+	    if (_PyString_Eq(ep->me_key, key))
+		return ep;
+	    if (++collisions > Py_MAX_DICT_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many hash collisions");
+		return NULL;
+	    }
+	}
+        else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
     }
     assert(0);          /* NOT REACHED */

From report at bugs.python.org  Mon Jan 23 14:38:26 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 23 Jan 2012 13:38:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2Webjv+mEHt9UrQOmVq82+NHsDW3zuyLM-rnqVxhD0LJQ@mail.gmail.com>
Message-ID: <4F1D62CD.4000408@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Alex Gaynor wrote:
> I'm able to put N pieces of data into the database on successive requests,
> but then *rendering* that data puts it in a dictionary, which renders that
> page unviewable by anyone.

I think you're asking a bit much here :-) A broken app is a broken
app, no matter how nice Python tries to work around it. If an
app puts too much trust into user data, it will be vulnerable
one way or another and regardless of how the user data enters
the app.

These are the collision counting possibilities we've discussed
so far:

With an collision counting exception you'd get a clear notice that
something in your data and your application is wrong and needs
fixing. The rest of your web app will continue to work fine and
you won't run into a DoS problem taking down all of your web
server.

With the proposed enhancement of collision counting + universal hash
function for Python 3.3, you'd get a warning printed to the logs, the
dict implementation would self-heal and your page is viewable nonetheless.
The admin would then see the log entry and get a chance to fix the
problem.

Note: Even if Python works around the problem successfully, there's no
guarantee that the data doesn't end up being processed by some other
tool in the chain with similar problems. All this is a work-around
for an application bug, nothing more. Silencing the problem
by e.g. using randomization in the string hash algorithm
doesn't really help in identifying the bug.

Overall, I don't think we should make Python's hash function
non-deterministic. Even with the universal hash function idea,
the dict implementation should use a predefined way of determining
the next hash parameter to use, so that running the application
twice against attack data will still result in the same data
output.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 14:40:28 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 13:40:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D5B87.8050907@egenix.com>
Message-ID: <1327325904.3395.13.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Frank's example is an attack on the second possible way to
> trigger the O(n^2) behavior. See msg150724 further above where I
> listed the two possibilities:
> 
> """
> An attack can be based on trying to find many objects with the same
> hash value, or trying to find many objects that, as they get inserted
> into a dictionary, very often cause collisions due to the collision
> resolution algorithm not finding a free slot.
> """

No, Frank's examples attack both possible ways.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 14:56:34 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 13:56:34 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D62CD.4000408@egenix.com>
Message-ID: <1327326870.3395.20.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> With an collision counting exception you'd get a clear notice that
> something in your data and your application is wrong and needs
> fixing. The rest of your web app will continue to work fine

Except when it doesn't, because you've also broken batch processing
functions and the like.

> Note: Even if Python works around the problem successfully, there's no
> guarantee that the data doesn't end up being processed by some other
> tool in the chain with similar problems.

Non-Python tools don't use Python's hash functions, they are therefore
not vulnerable to the same data.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:02:12 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 14:02:12 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327327332.28.0.333190653531.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I?hope?the?patch?is?now?good.?What?do?you?think?

----------
Added file: http://bugs.python.org/file24296/done?deal.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:03:22 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Mon, 23 Jan 2012 14:03:22 +0000
Subject: [issue13793] hasattr, delattr, getattr fail with unnormalized names
In-Reply-To: <1326690325.64.0.50515484047.issue13793@psf.upfronthosting.co.za>
Message-ID: <1327327402.55.0.69971746047.issue13793@psf.upfronthosting.co.za>


Changes by Benjamin Peterson <benjamin at python.org>:


----------
resolution:  -> wont fix
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13793>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:26:53 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 14:26:53 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327328813.16.0.598727660248.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Ah, darn it! My last patch is a total garbage. But then an interesting thing happened: I wanted to delete my last patch set, but I got a

"ProgrammingError at /review/13816/patchset/4039/delete
schema "datetime" does not exist"

and got all the Django nitty-gritty info about your server. This is not good because someone can misuse that information. Please fix this. You have "DEBUG = True" set and so all the details of your server is leaked. Better change that to "DEBUG = False" and let the user get a standard 500 error page instead.

It says at the bottom: "You're seeing this error because you have DEBUG = True in your Django settings file. Change that to False, and Django will display a standard 500 page."

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:36:07 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 14:36:07 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327329367.83.0.435382671723.issue13816@psf.upfronthosting.co.za>


Changes by Bo?tjan Mejak <bostjan.mejak at gmail.com>:


Removed file: http://bugs.python.org/file24296/done?deal.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:36:08 2012
From: report at bugs.python.org (sbt)
Date: Mon, 23 Jan 2012 14:36:08 +0000
Subject: [issue8713] multiprocessing needs option to eschew fork() under Linux
In-Reply-To: <1273853591.12.0.245920632829.issue8713@psf.upfronthosting.co.za>
Message-ID: <1327329368.48.0.917371893215.issue8713@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Attached is an updated version of the mp_fork_exec.patch.  This one is able to reliably clean up any unlinked semaphores if the program exits abnormally.

----------
Added file: http://bugs.python.org/file24297/mp_fork_exec.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8713>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:36:31 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 14:36:31 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327329391.21.0.510914963031.issue13816@psf.upfronthosting.co.za>


Changes by Bo?tjan Mejak <bostjan.mejak at gmail.com>:


Removed file: http://bugs.python.org/file24293/fixed patch final.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:42:58 2012
From: report at bugs.python.org (sbt)
Date: Mon, 23 Jan 2012 14:42:58 +0000
Subject: [issue8713] multiprocessing needs option to eschew fork() under Linux
In-Reply-To: <1273853591.12.0.245920632829.issue8713@psf.upfronthosting.co.za>
Message-ID: <1327329778.01.0.554231069658.issue8713@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

mp_split_tests.patch splits up the test_multiprocessing.py:

test_multiprocessing_misc.py
  miscellaneous tests which need not be run with multiple configurations

mp_common.py
  testcases which should be run with multiple configurations

test_multiprocessing_fork.py
test_multiprocessing_nofork.py
test_multiprocessing_manager_fork.py
test_multiprocessing_manager_nofork.py
test_multiprocessing_threads.py
  run the testcases in mp_common.py with various configurations

----------
Added file: http://bugs.python.org/file24298/mp_split_tests.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8713>
_______________________________________

From report at bugs.python.org  Mon Jan 23 15:46:52 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 14:46:52 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327330012.01.0.382549785739.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I wasn't able to remove the patch via "Delete Patch Set" but I did it by clicking on the Unlink button. So the "Delete Patch Set" option has a bug.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 16:02:14 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 15:02:14 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327330934.07.0.927447504257.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Georg, I object to your comment about *i*th needing to be *i*\ th

because currently in the source code we have it written as *i*'th which, by your logic, would need to be written as *i*\ 'th

which is not so and is compiled/displayed/whatever perfectly as is, so *i*th would be compiled/displayed/whatever, too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 16:18:19 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Mon, 23 Jan 2012 15:18:19 +0000
Subject: [issue13838] In str.format "{0:#.5g}" for decimal.Decimal doesn't
	print trailing zeros
In-Reply-To: <1327271130.3.0.873784301126.issue13838@psf.upfronthosting.co.za>
Message-ID: <1327331899.9.0.634869188902.issue13838@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
resolution:  -> duplicate
status: open -> closed
superseder:  -> Add alternate float formatting styles to new-style formatting.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13838>
_______________________________________

From report at bugs.python.org  Mon Jan 23 16:25:16 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 15:25:16 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327332316.9.0.786321425002.issue13816@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

This stupid HG is so cryptic. SVN was so easy, now I can't even create a patch. Can you just fix those two typos as "key function" and "*i*th" please? Thank you.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 16:27:56 2012
From: report at bugs.python.org (Stefan Krah)
Date: Mon, 23 Jan 2012 15:27:56 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327332476.92.0.998897652112.issue13816@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
nosy:  -skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 17:31:49 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 23 Jan 2012 16:31:49 +0000
Subject: [issue4686] Exceptions in ConfigParser don't set .args
In-Reply-To: <1229545362.85.0.86911392572.issue4686@psf.upfronthosting.co.za>
Message-ID: <E1RpMo7-0001BL-HX@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 8e091e36fc80 by ?ukasz Langa in branch '2.7':
Fixes #4686. Reverts redundant picklability code from r74544.
http://hg.python.org/cpython/rev/8e091e36fc80

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4686>
_______________________________________

From report at bugs.python.org  Mon Jan 23 17:33:45 2012
From: report at bugs.python.org (=?utf-8?q?=C5=81ukasz_Langa?=)
Date: Mon, 23 Jan 2012 16:33:45 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1327336425.93.0.588234388142.issue13760@psf.upfronthosting.co.za>


?ukasz Langa <lukasz at langa.pl> added the comment:

The reason why 3.2 and 3.3 work is that some time ago I fixed #4686 by explicitly settings .args in configparser exceptions. Unfortunately that patch was not backported at the time. I did that just now.

You're right, that is specific to configparser and doesn't affect #1692335 which is still unsolved.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Mon Jan 23 17:43:24 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 23 Jan 2012 16:43:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D5B87.8050907@egenix.com>
Message-ID: <4F1D8E27.3000707@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

Here's a version of the collision counting patch that takes both hash
and slot collisions into account.

I've also added a test script which demonstrates both types of
collisions using integer objects (since it's trivial to calculate
their hashes).

To see the collision counting, enable the DEBUG_DICT_COLLISIONS
macro variable.

----------
Added file: http://bugs.python.org/file24299/hash-attack-3.patch
Added file: http://bugs.python.org/file24300/integercollision.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
Index: Objects/dictobject.c
===================================================================
--- Objects/dictobject.c	(revision 88933)
+++ Objects/dictobject.c	(working copy)
@@ -9,7 +9,13 @@
 
 #include "Python.h"
 
+/* Maximum number of allowed collisions. */
+#define Py_MAX_DICT_HASH_COLLISIONS 1000
+#define Py_MAX_DICT_SLOT_COLLISIONS 1000
 
+/* Debug collision detection */
+#define DEBUG_DICT_COLLISIONS 0
+
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
  * exception arguments. */
@@ -327,6 +333,7 @@
     register PyDictEntry *ep;
     register int cmp;
     PyObject *startkey;
+    size_t hash_collisions, slot_collisions;
 
     i = (size_t)hash & mask;
     ep = &ep0[i];
@@ -361,6 +368,8 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    hash_collisions = 1;
+    slot_collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
@@ -387,9 +396,27 @@
                  */
                 return lookdict(mp, key, hash);
             }
+	    #if DEBUG_DICT_COLLISIONS
+	    printf("hash collisions = %zu (i=%zu)\n", hash_collisions, i);
+	    #endif
+	    if (++hash_collisions > Py_MAX_DICT_HASH_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many hash collisions");
+		return NULL;
+	    }
         }
-        else if (ep->me_key == dummy && freeslot == NULL)
-            freeslot = ep;
+        else {
+	    if (ep->me_key == dummy && freeslot == NULL)
+		freeslot = ep;
+	    #if DEBUG_DICT_COLLISIONS
+	    printf("slot collisions = %zu (i=%zu)\n", slot_collisions, i);
+	    #endif
+	    if (++slot_collisions > Py_MAX_DICT_SLOT_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many slot collisions");
+		return NULL;
+	    }
+	}
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -413,6 +440,7 @@
     register size_t mask = (size_t)mp->ma_mask;
     PyDictEntry *ep0 = mp->ma_table;
     register PyDictEntry *ep;
+    size_t hash_collisions, slot_collisions;
 
     /* Make sure this function doesn't have to handle non-string keys,
        including subclasses of str; e.g., one reason to subclass
@@ -439,18 +467,39 @@
 
     /* In the loop, me_key == dummy is by far (factor of 100s) the
        least likely outcome, so test for that last. */
+    hash_collisions = 1;
+    slot_collisions = 1;
     for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
         i = (i << 2) + i + perturb + 1;
         ep = &ep0[i & mask];
         if (ep->me_key == NULL)
             return freeslot == NULL ? ep : freeslot;
-        if (ep->me_key == key
-            || (ep->me_hash == hash
-            && ep->me_key != dummy
-            && _PyString_Eq(ep->me_key, key)))
+        if (ep->me_key == key)
             return ep;
-        if (ep->me_key == dummy && freeslot == NULL)
-            freeslot = ep;
+        if (ep->me_hash == hash && ep->me_key != dummy) {
+	    if (_PyString_Eq(ep->me_key, key))
+		return ep;
+	    #if DEBUG_DICT_COLLISIONS
+	    printf("hash collisions = %zu (i=%zu)\n", hash_collisions, i);
+	    #endif
+	    if (++hash_collisions > Py_MAX_DICT_HASH_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many hash collisions");
+		return NULL;
+	    }
+	}
+        else {
+	    if (ep->me_key == dummy && freeslot == NULL)
+		freeslot = ep;
+	    #if DEBUG_DICT_COLLISIONS
+	    printf("slot collisions = %zu (i=%zu)\n", slot_collisions, i);
+	    #endif
+	    if (++slot_collisions > Py_MAX_DICT_SLOT_COLLISIONS) {
+		PyErr_SetString(PyExc_KeyError,
+				"too many slot collisions");
+		return NULL;
+	    }
+	}
     }
     assert(0);          /* NOT REACHED */
     return 0;
-------------- next part --------------
def integer_hash_collisions(max=1000):

    print 'Hash collision attack:'

    d = {}
    for x in xrange(max):
        i = x*(2**64 - 1) + 1
        #print 'adding %i with hash %i' % (i, hash(i))
        d[i] = 1
    print 'generated dict with %i items' % len(d)
    return d

def integer_slot_collisions(max=1000):

    print 'Slot collision attack:'

    # Fill the dict slots starting at (hash) position 1
    d = {1:1}
    i = 1
    perturb = i
    print 'seeding the dict:'
    for x in xrange(max):
        i = ((i << 2) + i + perturb + 1) & 0xffffffffffffffff
        #print 'adding %i with hash %i' % (i, hash(i))
        d[i] = 1
        perturb >>= 5

    # Add new keys
    print 'generating slot collisions:'
    # cause a hash collision on the first try to enter the
    # prepared slot collision path
    i = 1*(2**64 - 1) + 1
    for x in xrange(max):
        #print 'adding %i with hash %i' % (i, hash(i))
        d[i] = 1
    print 'generated dict with %i items' % len(d)
    return d

integer_hash_collisions(1000)
integer_slot_collisions(1000)

From report at bugs.python.org  Mon Jan 23 17:45:03 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 23 Jan 2012 16:45:03 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D8E27.3000707@egenix.com>
Message-ID: <4F1D8E8A.7000407@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

> I've also added a test script which demonstrates both types of
> collisions using integer objects (since it's trivial to calculate
> their hashes).

I forgot to mention: the test script is for 64-bit platforms. It's
easy to adapt it to 32-bit if needed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:00:20 2012
From: report at bugs.python.org (Brian Curtin)
Date: Mon, 23 Jan 2012 17:00:20 +0000
Subject: [issue6792] Distutils-based installer does not detect 64bit versions
	of Python
In-Reply-To: <1251449540.11.0.0670330819334.issue6792@psf.upfronthosting.co.za>
Message-ID: <1327338020.22.0.533581640671.issue6792@psf.upfronthosting.co.za>


Brian Curtin <brian at python.org> added the comment:

Without a patch or a solution, the priority doesn't really matter (like Tarek said in msg127630). If anyone is actively working on this feel free to say otherwise, but I see no status to update.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6792>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:10:03 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 17:10:03 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327338603.79.0.307755692134.issue13812@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I propose applying the following patch.
We could open a separate issue to suggest "exit = sys.exit" under Windows (it doesn't seem to break any tests).

----------
keywords: +patch
nosy: +neologix
stage: test needed -> patch review
Added file: http://bugs.python.org/file24301/mpstderrflush.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:17:37 2012
From: report at bugs.python.org (=?utf-8?q?=C5=81ukasz_Langa?=)
Date: Mon, 23 Jan 2012 17:17:37 +0000
Subject: [issue13190] ConfigParser uses wrong newline on Windows
In-Reply-To: <1318787272.4.0.221500033494.issue13190@psf.upfronthosting.co.za>
Message-ID: <1327339057.22.0.631866264436.issue13190@psf.upfronthosting.co.za>


?ukasz Langa <lukasz at langa.pl> added the comment:

That is indeed the behaviour. Citing the tutorial: "Python on Windows makes a distinction between text and binary files; the end-of-line characters in text files are automatically altered slightly when data is read or written."

When you're opening the file in binary mode, \n characters are not altered. Opening a file that way specifies programmer intent and I don't think we should force os.linesep in that case.

----------
resolution:  -> works for me
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13190>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:23:03 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Mon, 23 Jan 2012 17:23:03 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327339383.51.0.965308244556.issue13816@psf.upfronthosting.co.za>


Justin Wehnes <jwehnes at gmail.com> added the comment:

Fixed.

----------
Added file: http://bugs.python.org/file24302/issue13816.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:35:34 2012
From: report at bugs.python.org (Santoso Wijaya)
Date: Mon, 23 Jan 2012 17:35:34 +0000
Subject: [issue8713] multiprocessing needs option to eschew fork() under Linux
In-Reply-To: <1273853591.12.0.245920632829.issue8713@psf.upfronthosting.co.za>
Message-ID: <1327340134.07.0.343726399254.issue8713@psf.upfronthosting.co.za>


Changes by Santoso Wijaya <santoso.wijaya at gmail.com>:


----------
nosy:  -santa4nt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8713>
_______________________________________

From report at bugs.python.org  Mon Jan 23 18:50:42 2012
From: report at bugs.python.org (Ned Deily)
Date: Mon, 23 Jan 2012 17:50:42 +0000
Subject: [issue13829] exception error in _scproxy.so
In-Reply-To: <1326998522.6.0.790472668429.issue13829@psf.upfronthosting.co.za>
Message-ID: <1327341042.47.0.313645249002.issue13829@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

The problem was originally discussed in these threads:

http://mail.python.org/pipermail/pythonmac-sig/2011-December/023428.html
http://mail.python.org/pipermail/pythonmac-sig/2011-December/023430.html

Because _scproxy is used to get information about Internet proxies, a test case should be constructed on OS X 10.7.x with a proxy configured.

----------
assignee:  -> ronaldoussoren
components: +Macintosh
nosy: +ned.deily, ronaldoussoren
resolution: invalid -> 
status: closed -> open
title: exception error -> exception error in _scproxy.so

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13829>
_______________________________________

From report at bugs.python.org  Mon Jan 23 19:03:37 2012
From: report at bugs.python.org (Faheem Mitha)
Date: Mon, 23 Jan 2012 18:03:37 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1327341817.76.0.571542121059.issue13760@psf.upfronthosting.co.za>


Faheem Mitha <faheem at faheem.info> added the comment:

I see. Thanks for the pointer to the earlier (2008) bug report. I notice you fixed the bug differently for 2.7 (define __reduce__) and 3.2 (set args). Is there some reason for that?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Mon Jan 23 19:08:59 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Mon, 23 Jan 2012 18:08:59 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327342139.37.0.976627950141.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Regarding the patch: I'd also like to see sys.stdout.flush() and sys.stderr.flush() between "exitcode = self._boostrap()" and "exit(exitcode)" in /Lib/multiprocessing/forking.py :: main(). (The extra stderr flush would be for symmetry with Popen.__init__() for unix.)

The remainder of this post is what I wrote before seeing your patch.


Thank you for the explanation concerning libc and buffering.

As for os._exit, I can see the need under unix, where the child process has a stack inherited from the parent. Also, allowing the child to cleanup shared resources would be disastrous for the parent. (I was going to propose the idea of making a sys.exit()-like function that did not rely on the exception mechanism to unwind the stack. But it wouldn't be usable here for this reason.)

Under Windows, the child process's multiprocessing.forking.main() is launched either by directly calling main() on the command line, or from within freeze_support() in the user program. In the latter case, the user can be advised not to catch SystemExit around freeze_support(), just as they are already advised to make calling freeze_support() the first statement within "if __name__ == '__main__':". So I don't see any harm in switching to sys.exit() there.


I think that in general, the differences between multiprocessing's behavior under unix and windows are a weakness, as is the lack of a precise specification for this behavior.

At the same time, multiprocessing is a lot more convenient to use than the subprocess module when the child process is a Python program. In particular, I use multiprocessing for its support of passing pickled objects between parent and child. With subprocess, I don't think it's even possible to share a file descriptor between parent and child under Windows.

I'm wondering whether it would be desirable to make an option for the unix side of multiprocessing to behave more like the windows one, and invoke the Python interpreter from the beginning with no trace of the parent's stack. That is, it'd follow a more traditional fork()-then-exec*() pattern. Then you'd be able to have the interpreter destruct naturally under both platforms, close resources, etc.

Are there use cases that require just forking under unix? Is the performance significantly better without an exec*()?

Perhaps it would be better If I took this discussion to python-ideas.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 19:21:39 2012
From: report at bugs.python.org (Jesse Noller)
Date: Mon, 23 Jan 2012 18:21:39 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1327342139.37.0.976627950141.issue13812@psf.upfronthosting.co.za>
Message-ID: <499E4B39867B40F5A8DB91A00B0B7503@gmail.com>


Jesse Noller <jnoller at gmail.com> added the comment:

There's already a bug / pending patch for this behavior here:

http://bugs.python.org/issue8713

No need to take it to -ideas.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Mon Jan 23 19:58:52 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Mon, 23 Jan 2012 18:58:52 +0000
Subject: [issue13841] multiprocessing should use sys.exit() where possible
Message-ID: <1327345132.43.0.775594933755.issue13841@psf.upfronthosting.co.za>


New submission from Jon Brandvein <jon.brandvein at gmail.com>:

Currently the multiprocessing library calls a hard exit function (os._exit under unix, ExitProcess under Windows) to terminate the child process.

Under unix, this is necessary because the child is forked without exec-ing. Calling sys.exit() would make it possible for the child to run code on the part of the stack inherited from the parent, via the exception handling mechanism. It might also allow the child to accidentally destroy shared state through an object destructor, even when that object was not explicitly relied on by the child.

Under Windows, I do not see any benefit besides symmetry. Processes are not forked, and the only way control can pass to user code after executing the target function, is if the process is frozen and the user puts the call to freeze_support() inside a try block. This special case can be taken care of by advising the user not to do that. (We already tell the user where freeze_support() should be located.)

Changing the multiprocessing exit routine from ExitProcess to sys.exit on Windows would ensure that all objects holding resources get properly destroyed. In particular, it would ensure that all file streams (including standard output and standard error) are flushed. This is especially important under Python 3, since the new IO system uses its own buffering which cannot be flushed by ExitProcess -- from the user's point of view, a potential regression from Python 2.x.

Related issues:
  - #13812 would not have been a problem under windows.
  - If #8713 gets adopted, unix can use sys.exit() as well.

----------
components: Library (Lib)
messages: 151835
nosy: brandj, jnoller, neologix, pitrou
priority: normal
severity: normal
status: open
title: multiprocessing should use sys.exit() where possible
type: enhancement
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13841>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:21:49 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 23 Jan 2012 19:21:49 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327346509.32.0.0536309002105.issue13816@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

1. Please report issues with the tracker to the meta-tracker at http://psf.upfronthosting.co.za/roundup/meta/.

2. You may object all you want, *i*th is invalid reST and a very quick tryout with "rst2html" would have shown you that.

3. Lastly, it is usual to test patches before submitting. Had you tried to build your changed docs instead of complaining here, you might have been enlightened.

Justin: thanks for the patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:21:54 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 23 Jan 2012 19:21:54 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <E1RpPSi-0003PV-Q6@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a72ca8b23cdf by Georg Brandl in branch '3.2':
#13816: fix two minor style issues. Thanks to Justin Wehnes for the patch.
http://hg.python.org/cpython/rev/a72ca8b23cdf

New changeset f4f9ab2fd51b by Georg Brandl in branch '2.7':
#13816: fix two minor style issues. Thanks to Justin Wehnes for the patch.
http://hg.python.org/cpython/rev/f4f9ab2fd51b

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:24:19 2012
From: report at bugs.python.org (James Sanders)
Date: Mon, 23 Jan 2012 19:24:19 +0000
Subject: [issue13842] Cannot pickle Ellipsis or NotImplemented
Message-ID: <1327346659.12.0.712807876778.issue13842@psf.upfronthosting.co.za>


New submission from James Sanders <bistromathic1 at gmail.com>:

At present, the built-in constants Ellipsis (...) and NotImplemented cannot be pickled.  Perhaps there is a good reason for this, but the only discussion I can find is at msg<108957>, where it is stated that these values (along with their types, and type(None)) cannot be pickled.

I ran across this in a class that keeps track of numpy-style slicing operations, and so sometimes stores references to Ellipsis.  While this is easy to work around, it does seem a little surprising that ... cannot be pickled, when slice objects can be.  I don't know if there is a likely use for pickling NotImplemented.

If this is not changed, perhaps it could be explicitly stated in the documentation that these objects cannot be pickled?

----------
messages: 151838
nosy: James.Sanders, alexandre.vassalotti, pitrou
priority: normal
severity: normal
status: open
title: Cannot pickle Ellipsis or NotImplemented
type: enhancement
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13842>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:32:58 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 19:32:58 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327347178.46.0.770227807761.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Is?this?fixed?or?what?

----------
nosy: +jwehnes
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:33:53 2012
From: report at bugs.python.org (Brian Curtin)
Date: Mon, 23 Jan 2012 19:33:53 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327347233.31.0.15765871444.issue10580@psf.upfronthosting.co.za>


Brian Curtin <brian at python.org> added the comment:

No.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 23 20:49:34 2012
From: report at bugs.python.org (Brian Curtin)
Date: Mon, 23 Jan 2012 19:49:34 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1327348174.67.0.336157258095.issue13772@psf.upfronthosting.co.za>


Brian Curtin <brian at python.org> added the comment:

Looks good to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:05:05 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 23 Jan 2012 21:05:05 +0000
Subject: [issue13737] bugs.python.org/review's Django settings file DEBUG=True
In-Reply-To: <1326009127.55.0.187877288169.issue13737@psf.upfronthosting.co.za>
Message-ID: <1327352705.19.0.941585139818.issue13737@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

I now set DEBUG = False in gae2django/settings.py and rietveld/settings.py.

----------
nosy: +ezio.melotti
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13737>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:05:59 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 21:05:59 +0000
Subject: [issue13842] Cannot pickle Ellipsis or NotImplemented
In-Reply-To: <1327346659.12.0.712807876778.issue13842@psf.upfronthosting.co.za>
Message-ID: <1327352759.24.0.32692076606.issue13842@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think it's a reasonable feature request. Now someone has to write a patch for it.

----------
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13842>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:12:48 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 23 Jan 2012 21:12:48 +0000
Subject: [issue13816] Two typos in the docs
In-Reply-To: <1326903795.6.0.0593092042228.issue13816@psf.upfronthosting.co.za>
Message-ID: <1327353168.95.0.686325991967.issue13816@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13816>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:18:18 2012
From: report at bugs.python.org (Alexandre Vassalotti)
Date: Mon, 23 Jan 2012 21:18:18 +0000
Subject: [issue13842] Cannot pickle Ellipsis or NotImplemented
In-Reply-To: <1327346659.12.0.712807876778.issue13842@psf.upfronthosting.co.za>
Message-ID: <1327353498.14.0.890256937704.issue13842@psf.upfronthosting.co.za>


Alexandre Vassalotti <alexandre at peadrop.com> added the comment:

We will need to bump the protocol number to add support for None, Ellipsis, and NotImplemented. Antoine, can you add this to PEP 3154?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13842>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:23:24 2012
From: report at bugs.python.org (sbt)
Date: Mon, 23 Jan 2012 21:23:24 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1250550378.97.0.072881968798.issue6721@psf.upfronthosting.co.za>
Message-ID: <1327353804.31.0.0419255276351.issue6721@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Attached is a patch (without documentation) which creates an atfork module for Unix.

Apart from the atfork() function modelled on pthread_atfork() there is also a get_fork_lock() function.  This returns a recursive lock which is held whenever a child process is created using os.fork(), subprocess.Popen() or multiprocessing.Process().  It can be used like

    with atfork.get_fork_lock():
        r, w = os.pipe()
        pid = os.fork()
        if pid == 0:
            try:
                os.close(r)
                # do something with w
            finally:
                os._exit(0)
        else:
            os.close(w)

    # do something with r

This prevents processes forked by other threads from accidentally inheriting the writable end (which would potentially cause EOF to be delayed when reading from the pipe).  It can also be used to eliminate the potential race where you create an fd and then set the CLOEXEC flag on it.

The patch modifies Popen() and Process.start() to acquire the lock when they create their pipes.  (A race condition previously made Process.sentinel and Process.join() potentially unreliable in a multithreaded program.)

Note that using the deprecated os.popen?() and os.spawn?() functions can still cause accidental inheritance of fds.

(I have also done a hopefully complete patch to multiprocessing to optionally allow fork+exec on Unix -- see Issue 8713.)

----------
Added file: http://bugs.python.org/file24303/atfork.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:29:11 2012
From: report at bugs.python.org (sbt)
Date: Mon, 23 Jan 2012 21:29:11 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1250550378.97.0.072881968798.issue6721@psf.upfronthosting.co.za>
Message-ID: <1327354151.46.0.144996455673.issue6721@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Is there any particular reason not to merge Charles-Fran?ois's reinit_locks.diff?

Reinitialising all locks to unlocked after a fork seems the only sane option.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:32:00 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 23 Jan 2012 21:32:00 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327354320.17.0.230581644784.issue13703@psf.upfronthosting.co.za>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I'm attaching an attempt at backporting haypo's random-8.patch to 2.7

Changes relative to random-8.patch:

   * The randomization is off by default, and must be enabled by setting
     a new environment variable PYTHONHASHRANDOMIZATION to a non-empty string.
     (if so then, PYTHONHASHSEED also still works, if provided, in the same
     way as in haypo's patch)

   * All of the various "Py_hash_t" become "long" again (Py_hash_t was
     added in 3.2: issue9778)

   * I expanded the randomization from just PyUnicodeObject to also cover
     these types:

     * PyStringObject

     * PyBufferObject

     The randomization does not cover numeric types: if we change the hash of
     int so that hash(i) no longer equals i, we also have to change it
     consistently for long, float, complex, decimal.Decimal and
     fractions.Fraction; however, there are 3rd-party numeric types that
     have their own __hash__ implementation that mimics int.__hash__ (see
     e.g. numpy)

     As noted in http://bugs.python.org/issue13703#msg151063 and
     http://bugs.python.org/issue13703#msg151064, it's not possible
     to directly create a dict with integer keys via JSON or XML-RPC.

     This seems like a tradeoff between the risk of attack via other means
     vs breakage induced by not having hash() == hash() for the various
     equivalent numerical representations in pre-existing code.

   * To support my expanded usage of the random secret, I moved:
       
       PyAPI_DATA(_Py_unicode_hash_secret_t) _Py_unicode_hash_secret

     from unicodeobject.h to object.h and renamed it to:

       PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;

     This also exposes it for usage by C extension modules, just in case
     they need it (Murphy's Law suggests we will need if we don't expose
     it).   This is an extension of the API, but warranted, I feel.  My
     plan for downstream RHEL is to add this explicitly to the RPM metadata
     as a "Provides" of the RPM providing libpython.so so that if something
     needs to use it, it can express a "Requires" on it; I assume that
     something similar is possible with .deb)

   * generalized test_unicode.HashTest to support the new env var and the
     additional types.  In my version, get_hash takes a _repr string rather
     than an object, so that I can test it with a buffer().  Arguably the
     tests should thus be moved from test_unicode to somewhere else, but this
     location keeps things consistent with haypo's patch.

     haypo: in random-8.patch, within test_unicode.HashTest.test_null_hash,
     "hash_empty" seems to be misnamed

   * dropped various selftest fixes where the corresponding selftests don't
     exist in 2.7

   * adds a description of the new environment variables to the manpage;
     arguably this should be done for the patch for the default branch also

Caveats:

   * only tested on Linux (Fedora 15 x86_64); not tested on Windows.  Tested
     via "make test" both with and without PYTHONHASHRANDOMIZATION=1

   * not yet benchmarked

 Doc/using/cmdline.rst                      |   28 ++
 Include/object.h                           |    7 
 Include/pythonrun.h                        |    2 
 Lib/lib-tk/test/test_ttk/test_functions.py |    2 
 Lib/os.py                                  |   19 -
 Lib/test/mapping_tests.py                  |    2 
 Lib/test/regrtest.py                       |    5 
 Lib/test/test_gdb.py                       |   15 +
 Lib/test/test_inspect.py                   |    1 
 Lib/test/test_os.py                        |   47 +++-
 Lib/test/test_unicode.py                   |   55 +++++
 Makefile.pre.in                            |    1 
 Misc/python.man                            |   22 ++
 Modules/posixmodule.c                      |  126 ++----------
 Objects/bufferobject.c                     |    8 
 Objects/object.c                           |    2 
 Objects/stringobject.c                     |    8 
 Objects/unicodeobject.c                    |   17 +
 PCbuild/pythoncore.vcproj                  |    4 
 Python/pythonrun.c                         |    2 
 b/Python/random.c                          |  284 +++++++++++++++++++++++++++++
 21 files changed, 510 insertions(+), 147 deletions(-)

----------
Added file: http://bugs.python.org/file24304/backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:32:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 23 Jan 2012 21:32:15 +0000
Subject: [issue13843] Python doesn't compile anymore on our Solaris buildbot:
	undefined libintl_* symbols
Message-ID: <1327354335.3.0.0960870172041.issue13843@psf.upfronthosting.co.za>


New submission from STINNER Victor <victor.stinner at haypocalc.com>:

gcc    -o python Modules/python.o libpython3.3dm.a -lsocket -lnsl -lintl -lrt -ldl -lsendfile   -lm  
Undefined                       first referenced
 symbol                             in file
libintl_bind_textdomain_codeset     libpython3.3dm.a(_localemodule.o)
libintl_gettext                     libpython3.3dm.a(_localemodule.o)
libintl_textdomain                  libpython3.3dm.a(_localemodule.o)
libintl_dcgettext                   libpython3.3dm.a(_localemodule.o)
libintl_bindtextdomain              libpython3.3dm.a(_localemodule.o)
libintl_dgettext                    libpython3.3dm.a(_localemodule.o)
ld: fatal: Symbol referencing errors. No output written to python
collect2: ld returned 1 exit status
*** Error code 1
make: Fatal error: Command failed for target `python'
program finished with exit code 1

http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203.x/builds/4131/steps/compile/logs/stdio

----------
components: Build
messages: 151848
nosy: haypo, pitrou
priority: normal
severity: normal
status: open
title: Python doesn't compile anymore on our Solaris buildbot: undefined libintl_* symbols
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13843>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:34:28 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 23 Jan 2012 21:34:28 +0000
Subject: [issue13843] Python doesn't compile anymore on our Solaris buildbot:
	undefined libintl_* symbols
In-Reply-To: <1327354335.3.0.0960870172041.issue13843@psf.upfronthosting.co.za>
Message-ID: <1327354468.19.0.325141968803.issue13843@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

The issue doesn't occur on Python 2.7.

----------
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13843>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:39:32 2012
From: report at bugs.python.org (Marc-Andre Lemburg)
Date: Mon, 23 Jan 2012 21:39:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D8E27.3000707@egenix.com>
Message-ID: <4F1DD38E.7000605@egenix.com>


Marc-Andre Lemburg <mal at egenix.com> added the comment:

> To see the collision counting, enable the DEBUG_DICT_COLLISIONS
> macro variable.

Running (part of (*)) the test suite with debugging enabled on a 64-bit
machine shows that slot collisions are much more frequent than
hash collisions, which only account for less than 0.01% of all
collisions.

It also shows that slot collisions in the low 1-10 range are
most frequent, with very few instances of a dict lookup
reaching 20 slot collisions (less than 0.0002% of all
collisions).

The great number of cases with 1 or 2 slot collisions surprised
me. It seems that there's potential for improvement of
the perturbation formula left.

Due to the large number of 1 or 2 slot collisions, the patch
is going to cause a minor hit to dict lookup performance.
It may make sense to unroll the slot search loop and only
start counting after the third round of misses.

(*) I stopped the run after several hours run-time, producing
some 148GB log data.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:55:17 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 23 Jan 2012 21:55:17 +0000
Subject: [issue6792] Distutils-based installer does not detect 64bit versions
	of Python
In-Reply-To: <1327251955.78.0.547432775167.issue6792@psf.upfronthosting.co.za>
Message-ID: <4F1DD744.5070703@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> If priority escalation is out of the question

It's not out of the question - it's just that setting the priority on
the issue is not a proper way to escalate.

Instead, there are two forms of escalation available:
1. submit a patch that fixes the issue
2. pay somebody to fix the issue if you can't fix it yourself

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6792>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:56:31 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 21:56:31 +0000
Subject: [issue13842] Cannot pickle Ellipsis or NotImplemented
In-Reply-To: <1327353498.14.0.890256937704.issue13842@psf.upfronthosting.co.za>
Message-ID: <1327355665.3370.1.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> We will need to bump the protocol number to add support for None,
> Ellipsis, and NotImplemented. Antoine, can you add this to PEP 3154?

I don't think this needs a protocol bump. These are global singletons,
they can be pickled as such, and they will be unpickleable on any
Python.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13842>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:58:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 21:58:32 +0000
Subject: [issue6721] Locks in python standard library should be sanitized on
	fork
In-Reply-To: <1327354151.46.0.144996455673.issue6721@psf.upfronthosting.co.za>
Message-ID: <1327355786.3370.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Is there any particular reason not to merge Charles-Fran?ois's reinit_locks.diff?
> 
> Reinitialising all locks to unlocked after a fork seems the only sane option.

I agree with this. 
I haven't looked at the patch very closely. I think perhaps each lock
could have an optional callback for specific code to be run after
forking, but that may come in another patch.
(this would allow to make e.g. the C RLock fork-safe)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________

From report at bugs.python.org  Mon Jan 23 22:58:58 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 21:58:58 +0000
Subject: [issue13843] Python doesn't compile anymore on our Solaris buildbot:
	undefined libintl_* symbols
In-Reply-To: <1327354335.3.0.0960870172041.issue13843@psf.upfronthosting.co.za>
Message-ID: <1327355938.28.0.0254307158645.issue13843@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The buildbot is Martin's.

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13843>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:06:43 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 23 Jan 2012 22:06:43 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327356403.62.0.714633275039.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Can you please fix this?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:07:11 2012
From: report at bugs.python.org (Brian Curtin)
Date: Mon, 23 Jan 2012 22:07:11 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327356431.05.0.278027934962.issue10580@psf.upfronthosting.co.za>


Brian Curtin <brian at python.org> added the comment:

Yes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:23:15 2012
From: report at bugs.python.org (Martin Pengelly-Phillips)
Date: Mon, 23 Jan 2012 22:23:15 +0000
Subject: [issue9625] argparse: Problem with defaults for variable nargs when
	using choices
In-Reply-To: <1282036744.37.0.964764969496.issue9625@psf.upfronthosting.co.za>
Message-ID: <1327357395.1.0.444380733928.issue9625@psf.upfronthosting.co.za>


Martin Pengelly-Phillips <dev at thesociable.net> added the comment:

The real issue is that the choices flag does not work with a default flag and * nargs.

The following works as expected:
>>> parser.add_argument('chosen', nargs='*', default=['a'])
>>> print(parser.parse_args())
Namespace(chosen=['a'])
>>> print(parser.parse_args(['a', 'b']))
Namespace(chosen=['a', 'b'])

Introducing a choices constraint breaks down when using the defaults:
>>> parser.add_argument('chosen', nargs='*', default=['a'], choices=['a', 'b'])
>>> print(parser.parse_args(['a']))
Namespace(chosen=['a'])
>>> print(parser.parse_args())
error: argument chosen: invalid choice: ['a'] (choose from 'a', 'b')

I would expect instead to have Namespace.chosen populated with the default list as before, but the choices constraint check does not validate correctly.

I think that changing the choices constraint logic to iterate over the default values if nargs results in a list would be a possible solution.

----------
title: argparse: Problem with defaults for variable nargs -> argparse: Problem with defaults for variable nargs when using choices

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9625>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:47:07 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 23 Jan 2012 22:47:07 +0000
Subject: [issue10278] add time.wallclock() method
In-Reply-To: <1288624210.23.0.0497533122542.issue10278@psf.upfronthosting.co.za>
Message-ID: <E1RpSfJ-0007hf-JS@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset fb0f4fe8123e by Victor Stinner in branch 'default':
Issue #10278: wallclock() cannot go backward, but two consecutive calls
http://hg.python.org/cpython/rev/fb0f4fe8123e

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10278>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:50:14 2012
From: report at bugs.python.org (C Fraire)
Date: Mon, 23 Jan 2012 22:50:14 +0000
Subject: [issue5305] imaplib should support international mailbox names
In-Reply-To: <1234935371.11.0.35650041724.issue5305@psf.upfronthosting.co.za>
Message-ID: <1327359014.38.0.795088613413.issue5305@psf.upfronthosting.co.za>


C Fraire <cfraire at yahoo.com> added the comment:

I've used the PloneMailList implementation in another project. It works well to add 'imap4-utf-7' as codec.

The twisted imap implementation seems to have been updated to properly support non-printable ASCII, but the twisted imap API is problematic for imaplib because twisted seems to expect its arguments to already be Python unicode.

So can we be specific about what kind of API change would satisfy this issue:

1) a number of API methods take one or more mailbox arguments. Of course, imaplib currently expects these to be ASCII, but what kind of argument should the methods take? UTF? Unicode? So would the library need a class property to describe an optional specified input encoding? Would it be expected to take Python unicode?

2) some methods, such as list and lsub, return mailbox names UTF-7 encoded and embedded in larger ASCII strings. Would imaplib be expected to alter the contents of these large strings and transform them into another other encoding (when a switch as described in 1) is active)?

----------
nosy: +cfraire

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5305>
_______________________________________

From report at bugs.python.org  Mon Jan 23 23:56:25 2012
From: report at bugs.python.org (Robert Xiao)
Date: Mon, 23 Jan 2012 22:56:25 +0000
Subject: [issue13844] hg.python.org doesn't escape title attributes in
	annotate view
Message-ID: <1327359385.01.0.222520184458.issue13844@psf.upfronthosting.co.za>


New submission from Robert Xiao <nneonneo at gmail.com>:

On hg.python.org, the "annotate" view doesn't properly escape the title attribute of the <a> elements, resulting in breakage on the left column:

http://hg.python.org/cpython/annotate/728cfc671d15/Modules/Setup.config.in

----------
components: None
messages: 151860
nosy: nneonneo
priority: normal
severity: normal
status: open
title: hg.python.org doesn't escape title attributes in annotate view
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13844>
_______________________________________

From report at bugs.python.org  Tue Jan 24 00:03:21 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 23 Jan 2012 23:03:21 +0000
Subject: [issue13844] hg.python.org doesn't escape title attributes in
	annotate view
In-Reply-To: <1327359385.01.0.222520184458.issue13844@psf.upfronthosting.co.za>
Message-ID: <1327359801.18.0.538536193225.issue13844@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

This is more of a Mercurial issue than a Python issue, so I suggest you report it at http://mercurial.selenic.com/bts/ instead. Thanks!

----------
nosy: +georg.brandl, pitrou
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13844>
_______________________________________

From report at bugs.python.org  Tue Jan 24 00:16:31 2012
From: report at bugs.python.org (=?utf-8?q?=C5=81ukasz_Langa?=)
Date: Mon, 23 Jan 2012 23:16:31 +0000
Subject: [issue13760] ConfigParser exceptions are not pickleable
In-Reply-To: <1326268212.81.0.448148296094.issue13760@psf.upfronthosting.co.za>
Message-ID: <1327360591.21.0.410510519764.issue13760@psf.upfronthosting.co.za>


?ukasz Langa <lukasz at langa.pl> added the comment:

Currently both branches use the same solution to the problem (e.g. setting `args` in `__init__()`). I simply forgot about the old ticket when I prepared a fix for this one.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13760>
_______________________________________

From report at bugs.python.org  Tue Jan 24 00:17:24 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 23 Jan 2012 23:17:24 +0000
Subject: [issue13844] hg.python.org doesn't escape title attributes in
	annotate view
In-Reply-To: <1327359385.01.0.222520184458.issue13844@psf.upfronthosting.co.za>
Message-ID: <1327360644.98.0.515934038736.issue13844@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Quickfixed it locally now.  Let's hope hg itself fixes it more comprehensively.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13844>
_______________________________________

From report at bugs.python.org  Tue Jan 24 00:36:43 2012
From: report at bugs.python.org (Robert Xiao)
Date: Mon, 23 Jan 2012 23:36:43 +0000
Subject: [issue13844] hg.python.org doesn't escape title attributes in
	annotate view
In-Reply-To: <1327359385.01.0.222520184458.issue13844@psf.upfronthosting.co.za>
Message-ID: <1327361803.43.0.499182670184.issue13844@psf.upfronthosting.co.za>


Robert Xiao <nneonneo at gmail.com> added the comment:

My testing suggests that this issue is already fixed in Mercurial itself, since using "hg serve" on a local copy gives the expected result. Thus, the problem is probably with hg.python.org's local installation.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13844>
_______________________________________

From report at bugs.python.org  Tue Jan 24 00:41:48 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 23 Jan 2012 23:41:48 +0000
Subject: [issue13844] hg.python.org doesn't escape title attributes in
	annotate view
In-Reply-To: <1327359385.01.0.222520184458.issue13844@psf.upfronthosting.co.za>
Message-ID: <1327362108.27.0.208651640877.issue13844@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
stage:  -> committed/rejected

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13844>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:02:45 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 00:02:45 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
Message-ID: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>


New submission from STINNER Victor <victor.stinner at haypocalc.com>:

Python implements time.time() using gettimeofday() which has only a resolution of 1 microsecond because it uses the timeval structure which is only able to store microseconds.

Attached patch changes _PyTime_gettimeofday() to make it uses the timespec structure (which has a resolution has 1 nanosecond) and use GetSystemTimeAsFileTime() on Windows. So time.time() has a theorical resolution 10 times better than currently.

----------
files: timespec.patch
keywords: patch
messages: 151865
nosy: belopolsky, haypo
priority: normal
severity: normal
status: open
title: Use GetSystemTimeAsFileTime() to get a resolution of 100 ns on Windows
versions: Python 3.3
Added file: http://bugs.python.org/file24305/timespec.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:11:37 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 00:11:37 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327363897.23.0.971762704078.issue13845@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Oops, my first patch contains an unrelated change for Windows.

New patch fixes this bug, and change time_clock() to reuse time_time() if time_clock() fails to get the CPU frequency (unlikely) because it has a better resolution than clock().

----------
Added file: http://bugs.python.org/file24306/timespec-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:11:42 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 00:11:42 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327363902.39.0.0936913048856.issue13845@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24305/timespec.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:14:31 2012
From: report at bugs.python.org (Paul McMillan)
Date: Tue, 24 Jan 2012 00:14:31 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1D62CD.4000408@egenix.com>
Message-ID: <CAO_YWRWeU-ZH5+3r4f7NhzeUDUOph1UCQntMSE7O3VS9C01G=w@mail.gmail.com>


Paul McMillan <paul at mcmillan.ws> added the comment:

> I think you're asking a bit much here :-) A broken app is a broken
> app, no matter how nice Python tries to work around it. If an
> app puts too much trust into user data, it will be vulnerable
> one way or another and regardless of how the user data enters
> the app.

I notice your patch doesn't include fixes for the entire standard
library to work around this problem. Were you planning on writing
those, or leaving that for others?

As a developer, I honestly don't know how I can state with certainty
that input data is clean or not, until I actually see the error you
propose. I can't check validity before the fact, the way I can check
for invalid unicode before storing it in my database. Once I see the
error (probably only after my application is attacked, certainly not
during development), it's too late. My application can't know which
particular data triggered the error, so it can't delete it. I'm
reduced to trial and error to remove the offending data, or to writing
code that never stores more than 1000 things in a dictionary. And I
have to accept that the standard library may not work on any
particular data I want to process, and must write code that detects
the error state and somehow magically removes the offending data.

The alternative, randomization, simply means that my dictionary
ordering is not stable, something that is already the case.

While I appreciate that the counting approach feels cleaner;
randomization is the only solution that makes practical sense.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:22:30 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 00:22:30 +0000
Subject: [issue13846] Add time.monotonic() function
Message-ID: <1327364549.64.0.960699645726.issue13846@psf.upfronthosting.co.za>


New submission from STINNER Victor <victor.stinner at haypocalc.com>:

After time.wallclock() (issue #10278), let add a time.monotomic() function. It is similar to time.wallclock() (try to get the most accurate clock) but is not available if the system doesn't provide a monotonic clock. It may also fail at runtime if Python cannot find a monotonic clock, whereas time.clock() and time.wallclock() fallback on a wallclock which may go backward on NTP adjust.

The documentation of the patch should be improved :-)

wallclock() tests may be simplified or dropped because they may fail on NTP adjust.

----------
files: monotonic.patch
keywords: patch
messages: 151868
nosy: belopolsky, haypo
priority: normal
severity: normal
status: open
title: Add time.monotonic() function
versions: Python 3.3
Added file: http://bugs.python.org/file24307/monotonic.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13846>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:42:46 2012
From: report at bugs.python.org (Jim Jewett)
Date: Tue, 24 Jan 2012 00:42:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F1DD38E.7000605@egenix.com>
Message-ID: <CA+OGgf42Di+mL_GaH6HxVEX4xpOTAQTp0mVEVtLDjEv4sS4-Wg@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Mon, Jan 23, 2012 at 4:39 PM, Marc-Andre Lemburg
<report at bugs.python.org> wrote:

> Running (part of (*)) the test suite with debugging enabled on a 64-bit
> machine shows that slot collisions are much more frequent than
> hash collisions, which only account for less than 0.01% of all
> collisions.

Even 1 in 10,000 seems pretty high, though I suppose it is a result of
non-random input.  (For a smalldict with 8 == 2^3 slots, on a 64-bit
machine, true hash collisions "should" only account for 1 in 2^61 slot
collisions.)

> It also shows that slot collisions in the low 1-10 range are
> most frequent, with very few instances of a dict lookup
> reaching 20 slot collisions (less than 0.0002% of all
> collisions).

Thus the argument that collisions > N implies (possibly malicious)
data that really needs a different hash -- and that this dict instance
in particular should take the hit to use an alternative hash.  (Do
note that this alternative hash could be stored in the hash member of
the PyDictEntry; if anything actually *equal* to the key comes along,
it will have gone through just as many collisions, and therefore also
have been rehashed.)

> The great number of cases with 1 or 2 slot collisions surprised
> me. It seems that there's potential for improvement of
> the perturbation formula left.

In retrospect, this makes sense.

    for (perturb = hash; ; perturb >>= PERTURB_SHIFT) {
        i = (i << 2) + i + perturb + 1;

If two objects collided then they have the same last few last few bits
in their hashes -- which means they also have the same last few bits
in their initial perturb.  And since the first probe is to slot 6i+1,
it funnels down to only even consider half the slots until the second
probe.

Also note that this explains why Randomization could make the Django
tests fail, even though 64-bit users haven't complained.  The initial
hash(&mask) is the same, and the first probe is the same, and (for a
small enough dict) so are the next several.  In a dict with 2^12
slots, the first 6 tries will be the same ... so I doubt the test
cases have sufficiently large amounts of sufficiently unlucky data to
notice very often -- unless the hash itself is changed, as in the
patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 24 01:44:44 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Tue, 24 Jan 2012 00:44:44 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327354320.17.0.230581644784.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAGE7PNKhR_J+=r0EhVch3WiiY706M9VXDT1aYgE-HovH5A=AHA@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

On Mon, Jan 23, 2012 at 1:32 PM, Dave Malcolm <report at bugs.python.org> wrote:
>
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
>
> I'm attaching an attempt at backporting haypo's random-8.patch to 2.7
>
> Changes relative to random-8.patch:
>
> ? * The randomization is off by default, and must be enabled by setting
> ? ? a new environment variable PYTHONHASHRANDOMIZATION to a non-empty string.
> ? ? (if so then, PYTHONHASHSEED also still works, if provided, in the same
> ? ? way as in haypo's patch)
>
> ? * All of the various "Py_hash_t" become "long" again (Py_hash_t was
> ? ? added in 3.2: issue9778)
>
> ? * I expanded the randomization from just PyUnicodeObject to also cover
> ? ? these types:
>
> ? ? * PyStringObject
>
> ? ? * PyBufferObject
>
> ? ? The randomization does not cover numeric types: if we change the hash of
> ? ? int so that hash(i) no longer equals i, we also have to change it
> ? ? consistently for long, float, complex, decimal.Decimal and
> ? ? fractions.Fraction; however, there are 3rd-party numeric types that
> ? ? have their own __hash__ implementation that mimics int.__hash__ (see
> ? ? e.g. numpy)
>
> ? ? As noted in http://bugs.python.org/issue13703#msg151063 and
> ? ? http://bugs.python.org/issue13703#msg151064, it's not possible
> ? ? to directly create a dict with integer keys via JSON or XML-RPC.
>
> ? ? This seems like a tradeoff between the risk of attack via other means
> ? ? vs breakage induced by not having hash() == hash() for the various
> ? ? equivalent numerical representations in pre-existing code.

Exactly.  I would NOT worry about hash repeatability for integers and
complex data structures.  It is not at the core of the common problem
(maybe a couple application specific problems but not a general "all
python web apps" severity problem).

Doing it for base byte string and unicode string like objects is
sufficient.  Good catch on doing it for buffer objects, I'd forgotten
about those. ;)  A big flaw with haypo's patch is that it only
considers unicode instead of all byte-string-ish stuff.  (the code in
issue13704 does that better).

>
> ? * To support my expanded usage of the random secret, I moved:
>
> ? ? ? PyAPI_DATA(_Py_unicode_hash_secret_t) _Py_unicode_hash_secret
>
> ? ? from unicodeobject.h to object.h and renamed it to:
>
> ? ? ? PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
>
> ? ? This also exposes it for usage by C extension modules, just in case
> ? ? they need it (Murphy's Law suggests we will need if we don't expose
> ? ? it). ? This is an extension of the API, but warranted, I feel. ?My
> ? ? plan for downstream RHEL is to add this explicitly to the RPM metadata
> ? ? as a "Provides" of the RPM providing libpython.so so that if something
> ? ? needs to use it, it can express a "Requires" on it; I assume that
> ? ? something similar is possible with .deb)

Exposing this is good.  There is a hash table implementation within
modules/expat/xmlparse.c that should probably use it as well.

> ? * generalized test_unicode.HashTest to support the new env var and the
> ? ? additional types. ?In my version, get_hash takes a _repr string rather
> ? ? than an object, so that I can test it with a buffer(). ?Arguably the
> ? ? tests should thus be moved from test_unicode to somewhere else, but this
> ? ? location keeps things consistent with haypo's patch.
>
> ? ? haypo: in random-8.patch, within test_unicode.HashTest.test_null_hash,
> ? ? "hash_empty" seems to be misnamed

Lets move this to a better location in all patches.  At this point
haypo's patch is not done yet so relevant bits of what you are doing
here is likely to be fed back into the eventual 3.3 tip patch.

-gps

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 24 02:06:30 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 01:06:30 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
Message-ID: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>


New submission from STINNER Victor <victor.stinner at haypocalc.com>:

Attach patch catchs errors on time(), ftime(), localtime() and clock(). It changes floattime() and _Py_gettimeofday() signature: result now indicates if an error occurred or not, and these function now directly raise an exception.

Catching time() error is maybe overflow: modern version of its manual page don't mention errors.

----------
components: Library (Lib)
files: time_error.patch
keywords: patch
messages: 151871
nosy: belopolsky, haypo
priority: normal
severity: normal
status: open
title: Catch time(), ftime(), localtime() and clock() errors
versions: Python 3.3
Added file: http://bugs.python.org/file24308/time_error.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Tue Jan 24 02:20:16 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 01:20:16 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327368016.27.0.513208455745.issue11457@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Attached patch prepares time.wallclock() to be able to return the result as an integer (seconds, nanoseconds).

----------
keywords: +patch
Added file: http://bugs.python.org/file24309/time_integer.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Tue Jan 24 02:25:38 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 01:25:38 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327368338.75.0.94048788967.issue11457@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

With the new function time.wallclock() and time.clock_gettime() (issue #10278), and maybe time.monotonic() will maybe be also added (issue #13846), I now agree that it is important to support t2-t1 to compute a difference. Using a tuple, it's not easy to compute a difference.

time.wallclock(), time.clock_gettime() and time.monotonic() have a nanosecond resolution on Linux. Using issue #13845, time.time() will have a resolution of 100 ns on Windows.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Tue Jan 24 02:27:55 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 01:27:55 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327368475.79.0.856469170743.issue13845@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

See also #11457 for discussion on nanosecond resolution and a potential new type to avoid loose of resolution of the Python float type (IEEE binary64).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 02:35:54 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 01:35:54 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <1327368954.11.0.614125862466.issue11235@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Fixing this is much easier than Victor's suggestion, we just have to ignore the higher bits of the timestamp (that is, store it modulo 2**32). This is enough for the purposes of testing the freshness of a pyc file.

And by avoiding modifying the pyc format, we can also ship the fix in bugfix releases.

----------
components: +Interpreter Core -None
nosy: +brett.cannon, pitrou
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 03:03:20 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 02:03:20 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
Message-ID: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

>>> open("\x00abc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: ''

Contrast with 2.x open():

>>> open("\x00abc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: file() argument 1 must be encoded string without NULL bytes, not str

----------
components: IO, Library (Lib)
keywords: easy
messages: 151876
nosy: hynek, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: io.open() doesn't check for embedded NUL characters
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 03:05:54 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Tue, 24 Jan 2012 02:05:54 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
Message-ID: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>


New submission from Alex Gaynor <alex.gaynor at gmail.com>:

ATM there's no tests (at least in 2.x, I haven't checked 3.x yet) for this behavior:

>>> os.path.exists("/tmp\x00abcds")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/genericpath.py", line 18, in exists
    os.stat(path)
TypeError: embedded NUL character

----------
components: Tests
messages: 151877
nosy: alex, amaury.forgeotdarc
priority: normal
severity: normal
status: open
title: Add tests for NUL checking in certain strs
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 03:06:27 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Tue, 24 Jan 2012 02:06:27 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327370787.26.0.101549020025.issue13848@psf.upfronthosting.co.za>


Changes by Alex Gaynor <alex.gaynor at gmail.com>:


----------
nosy: +alex

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 03:23:36 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 24 Jan 2012 02:23:36 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327371816.83.0.339750356042.issue13849@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Why is that a bug? There is no feature in Python saying that the test suite covers the code fully (by some kind of measurement).

New tests should only be added to 3.3, unless they test for a newly-fixed bug (and even then the test may not be backported to the maintenance release).

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 03:42:07 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 24 Jan 2012 02:42:07 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327372927.31.0.72028226279.issue13849@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Adding tests helps the other VMs, which generally are trailing behind the CPython releases.

----------
nosy: +r.david.murray

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 05:11:51 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Tue, 24 Jan 2012 04:11:51 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327378311.82.0.533921060695.issue13845@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 05:11:58 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Tue, 24 Jan 2012 04:11:58 +0000
Subject: [issue13846] Add time.monotonic() function
In-Reply-To: <1327364549.64.0.960699645726.issue13846@psf.upfronthosting.co.za>
Message-ID: <1327378318.95.0.915434226454.issue13846@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13846>
_______________________________________

From report at bugs.python.org  Tue Jan 24 06:54:58 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 05:54:58 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
Message-ID: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>


New submission from Nick Coghlan <ncoghlan at gmail.com>:

With the current argparse docs, it's very hard to get a quick reminder of how to spell the various parameters for add_argument, and just what they do. This issue suggests adding a "Quick Reference" section for add_argument, with the following elements:

1. Summary table with a one line description of each parameter
2. Summary table with a one line description of each alternative for the "action" parameter (including noting which other parameters are potentially relevant, such as 'choices' and 'metavar' for 'store' and 'const' for 'store_const')
3. Summary table with a one line description of each alternative for the "nargs" parameter

----------
assignee: docs at python
components: Documentation
messages: 151880
nosy: bethard, docs at python, ncoghlan
priority: normal
severity: normal
status: open
title: Summary tables for argparse add_argument options
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 07:10:00 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 06:10:00 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327385400.21.0.657988483274.issue13850@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Looking at the docs, a 4th table in the quick reference section would be useful: the parameters for ArgumentParser itself.

Note that the ArgumentParser and add_arguments() parameters are already summarised in their respective entries, but there are currently no summaries at all for the possible "action" and "nargs" values

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 08:58:55 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 24 Jan 2012 07:58:55 +0000
Subject: [issue8713] multiprocessing needs option to eschew fork() under Linux
In-Reply-To: <1327329778.01.0.554231069658.issue8713@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3aaHOQQy9HijX84rMRn1Vossn31uzzdfVk0YRffypupA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I don't know what the others think, but I'm still -1 on this patch.
Not that I don't like the principle - it's actually the contrary: in a
perfect world, I think this should be made the default -and only -
behavior on POSIX. But since it may break existing code, we'd have to
keep both implementations for POSIX systems, with - at least to me -
little benefit.
Having three different implementations, with different codepaths, will
increase the cognitive burden, make the code less readable, and
debugging harder:
- user: I'm getting this error with multiprocessing
- dev: On Windows or on Unix?
- user: On Unix
- dev: Do you use the fork()+exec() version or the bare fork() version?
- user: what's fork() and exec()?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8713>
_______________________________________

From report at bugs.python.org  Tue Jan 24 09:12:11 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 24 Jan 2012 08:12:11 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327392731.52.0.156744111397.issue13849@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

If other VMs need this test for some reason, they can easily add a test case themselves. I'm -1 on adding test cases to bug fix releases just for completeness. A lacking test is not a bug, and hence must not be added to a bug fix release.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 09:13:49 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 24 Jan 2012 08:13:49 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <E1RpbVj-00014v-Oo@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 839fa289e226 by Antoine Pitrou in branch '3.2':
Issue #13772: In os.symlink() under Windows, do not try to guess the link
http://hg.python.org/cpython/rev/839fa289e226

New changeset a7406565ef1c by Antoine Pitrou in branch 'default':
Issue #13772: In os.symlink() under Windows, do not try to guess the link
http://hg.python.org/cpython/rev/a7406565ef1c

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Tue Jan 24 09:14:38 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 08:14:38 +0000
Subject: [issue13772] listdir() doesn't work with non-trivial symlinks
In-Reply-To: <1326312074.88.0.322514160628.issue13772@psf.upfronthosting.co.za>
Message-ID: <1327392878.18.0.275601209228.issue13772@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Should be fixed now!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13772>
_______________________________________

From report at bugs.python.org  Tue Jan 24 10:07:40 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 24 Jan 2012 09:07:40 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1327338603.79.0.307755692134.issue13812@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2GDAgDa_R7wOn2fpBC-YtW_6qdY1hbudVB0TJ9SRCanA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> I propose applying the following patch.

The test looks good to me (except we don't remove TESTFN explicitely,
but I'm not sure it's really necessary).

As for the patch, couldn't we put all the file stream flushing in one place?

stdout and stderr are already flushed on Unix (which explains why this
only happens on Windows: the libc puts non-tty streams in _IOFBF, so
this wouldn't work without explicit flushing neither).

Lib/multiprocessing/forking.py:Popen.__init__:
           code = process_obj._bootstrap()
           sys.stdout.flush()
           sys.stderr.flush()
           os._exit(code)

Also, there's already a flush() a couple lines above (in _bootstrap):
           sys.stderr.write(e.args[0] + '\n')
           sys.stderr.flush()

So I'd suggest to just put:

           sys.stdout.flush()
           sys.stderr.flush()

At the end of _bootstrap, and remove the flushing done by
Popen.__init__ on Unix.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Tue Jan 24 10:19:25 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 09:19:25 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <1327396765.95.0.300927643554.issue11235@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Here is a patch for 3.2. importlib doesn't have the issue, so I just added a test.

----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 10:19:41 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 09:19:41 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <1327396781.41.0.941959019876.issue11235@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24310/imptimestampoverflow.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 10:36:23 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 09:36:23 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327397783.18.0.98297886192.issue13812@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Updated patch sanitizing the various flushes done on exit, as per Charles-Fran?ois's recommendation. Also removes TESTFN explicitly.

----------
Added file: http://bugs.python.org/file24311/mpstderrflush2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:00:24 2012
From: report at bugs.python.org (Xavier Morel)
Date: Tue, 24 Jan 2012 10:00:24 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327399224.62.0.973812508967.issue13850@psf.upfronthosting.co.za>


Xavier Morel <xavier.morel at masklinn.net> added the comment:

Creating the tables should not be too hard, especially using e.g. org-mode, but:

1. Where should those tables live? The argparse documentation is pretty big and there's no completely obvious place. I would guess table 1. could just replace the list of arguments in http://docs.python.org/py3k/library/argparse.html#the-add-argument-method but things are harder for `action` (as many actions have examples as well, so the listing can't just be replaced) and for `nargs`

2. If the current lists of `argument: role` (in ArgumentParser and add_argument) are not sufficient, why would a table somehow be considering it would *add* visual clutter (table borders, for instance)?

Maybe a good alternative would be to build a table style for info fields lists and to use :param: wherever that belongs?

Or the doc could be split into a "quickstart" with just a listing of arguments and a *very simple* example or two, and then the exhaustive documentation, which could even be a separate document?

----------
nosy: +xmorel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:32:01 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 24 Jan 2012 10:32:01 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1327397783.18.0.98297886192.issue13812@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM1-z_80Ay8nvpTaUin920k+ZSsyVZ4QQ+g3MRU-ONAYaA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

LGTM.

(I just noticed a bug in Rietveld: when one selects "expand 10 after",
the line right after that marker appears duplicated in the new view).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:40:13 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 10:40:13 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327401613.65.0.84994526555.issue13850@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

My specific suggestion is to have a dedicated "Quick Reference" section before the first example.

This section would be aimed at two groups of people:
- those wanting a quick overview of the features argparse offers them ("This looks complicated, what can it do for me?")
- those wanting a reminder of the exact spelling of various items ("I know what I want to do, and I know argparse can do it because I've done it before, but how do I tell argparse exactly what I want?")

Since the heart of argparse is the ability to map arguments to actions, I'd suggest the quick reference section actually lead with a table of "actions" that argparse natively supports, along with a final entry pointing to the information on custom actions (i.e. subclasses of argparse.Action).

Likely columns for this first table: Action Name, Description, Parameters

The "Parameters" column would span multiple lines, with one parameter and a brief description of the parameter on each line.

The second table could then just be a short summary of the various 'nargs' values.

Repeating the list of parameters to ArgumentParser in the quick reference section probably isn't necessary, and the short parameter descriptions in the actions table should suffice for add_argument().

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:40:28 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 24 Jan 2012 10:40:28 +0000
Subject: [issue13686] Some notes on the docs of multiprocessing
In-Reply-To: <1325307613.69.0.259805833126.issue13686@psf.upfronthosting.co.za>
Message-ID: <1327401628.89.0.846261201388.issue13686@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13686>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:40:44 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 24 Jan 2012 10:40:44 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327401644.27.0.687407309746.issue13850@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:48:57 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 24 Jan 2012 10:48:57 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327402137.35.0.336450658447.issue13845@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

GetSystemTimeAsFileTime() represent durations as multiple of 100ns, unfortunately its value is only updated every 15ms or so.  Precision is not accuracy...

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 11:56:28 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 10:56:28 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327402588.29.0.0202685255231.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I realised that the lack of a clear binary/text distinction would make it messy to do the split docs in 2.7, so I made a new branch based on 3.2 instead (link to repo updated accordingly).

----------
assignee: eric.araujo -> ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Tue Jan 24 12:22:54 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 11:22:54 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327402137.35.0.336450658447.issue13845@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwY2ds7cdT1GKoDM4ADNYpBwTGGyr88J0bLOWh_GTfaSdw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> GetSystemTimeAsFileTime() represent durations as multiple of 100ns, unfortunately its value is only updated every 15ms or so. ?Precision is not accuracy...

It is possible to improve the accuracy of this clock using the
undocumented NtSetTimerResolution() function:
http://undocumented.ntinternals.net/UserMode/Undocumented%20Functions/Time/NtSetTimerResolution.html

There are applications using this undocumented function. For example:
http://www.lucashale.com/timer-resolution/

See also the timeBeginPeriod() function:
http://msdn.microsoft.com/en-us/library/ms713413%28VS.85%29.aspx

The user may have a special hardware (and its driver) or a special
softwared (ntpd?) with a better accuracy.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 12:42:55 2012
From: report at bugs.python.org (Xavier Morel)
Date: Tue, 24 Jan 2012 11:42:55 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327405375.82.0.629987577583.issue13850@psf.upfronthosting.co.za>


Xavier Morel <xavier.morel at masklinn.net> added the comment:

> My specific suggestion is to have a dedicated "Quick Reference" section before the first example.

OK, that looks like a good plan.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 12:49:38 2012
From: report at bugs.python.org (sbt)
Date: Tue, 24 Jan 2012 11:49:38 +0000
Subject: [issue13841] multiprocessing should use sys.exit() where possible
In-Reply-To: <1327345132.43.0.775594933755.issue13841@psf.upfronthosting.co.za>
Message-ID: <1327405778.02.0.881326717424.issue13841@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Currently, on both Windows and Unix, when the main thread of a child process exits:

* atexit callbacks are NOT run (although multiprocessing.util._exit_function IS run),

* the main thread does NOT wait for non-daemonic background threads.

A simple replacement of ExitProcess() by sys.exit() would change this.  Whether that would be a good thing, I don't know.

----------
nosy: +sbt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13841>
_______________________________________

From report at bugs.python.org  Tue Jan 24 12:49:48 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 24 Jan 2012 11:49:48 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327405788.06.0.199136694727.issue13845@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

NtSetTimerResolution is a system-wide change, and may have impact on other running applications.  It may be an option to set it during the execution of profile.run() for example, but I would not enable it just to call time.clock().

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 13:47:39 2012
From: report at bugs.python.org (Vikash Agrawal)
Date: Tue, 24 Jan 2012 12:47:39 +0000
Subject: [issue13851] Packaging distutils2 for Fedora
Message-ID: <1327409259.69.0.346795065424.issue13851@psf.upfronthosting.co.za>


New submission from Vikash Agrawal <vikashagrawal1990 at gmail.com>:

Packaging distutils2 for Fedora

The spec file, is attached and lack many things. I also need to add files like README, COPYING etc.

The problem is, after installtion of the generated rpm, "python -distutils2-1.0-0.1.0.a3.fc16.noarch.rpm" 
, distutils2 functionalities (such as pysetup) are still not available in the system, though I could find the distutils2 module in the interpretor

Please guide, for the necessary changes/modifications, and I hope to see this package soon in Fedora repository

this is my first packaging initiative

Regards
Vikash Agrawal

----------
assignee: tarek
components: Distutils2
files: python-distutils2.spec
messages: 151898
nosy: alexis, eric.araujo, tarek, vikash
priority: normal
severity: normal
status: open
title: Packaging distutils2 for Fedora
versions: Python 2.7
Added file: http://bugs.python.org/file24312/python-distutils2.spec

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13851>
_______________________________________

From report at bugs.python.org  Tue Jan 24 13:55:32 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Tue, 24 Jan 2012 12:55:32 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327409732.07.0.807798280175.issue13848@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

I took a deep dive into parts of CPython that were unknown to me :) and dug up the following:

Methods like os.stat or even os.open convert the file name using "et" in PyArg_ParseTuple[AndKeywords].

OTOH, open() and io.open() just hand through the object as "O" to the respective low-level io module.


The result in 2.7 is that file() tries to convert it for it's own usage eventually ? which fails as seen. While a more explicit error message wouldn't hurt, this seems safe to me insofar.


In 3.3, file() aka Modules/_io/fileio.c , io_open does no such thing because it seems to handle fds as "nameobj" as well and does a wide range of checks on the argument.

After io_open is certain that nameobj is a file name, it uses PyObject_AsCharBuffer()on bytes and PyUnicode_FromObject() + encoding magic on unicode to get an encoded string as a file name.

Neither does a check for NUL bytes so the (w)open(er) that follows reacts as demonstrated by Antoine.

I presume fixing/breaking PyObject_AsCharBuffer()/PyUnicode_FromObject() is out of question so the most obvious part to fix would be the conversion block inside io_open.

Should I have a stab at it or do you disagree with my approach?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:14:45 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 24 Jan 2012 13:14:45 +0000
Subject: [issue13852] Doc fixes with patch
Message-ID: <1327410885.19.0.883089570823.issue13852@psf.upfronthosting.co.za>


New submission from Bo?tjan Mejak <bostjan.mejak at gmail.com>:

I have collected a small amount of documentation fixes in my patch. Please review it and apply it. No pressure. ;)

----------
assignee: docs at python
components: Documentation
files: changes.diff
keywords: patch
messages: 151900
nosy: Retro, docs at python
priority: normal
severity: normal
status: open
title: Doc fixes with patch
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24313/changes.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13852>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:17:00 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 24 Jan 2012 13:17:00 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327405788.06.0.199136694727.issue13845@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwZ84TiU4pPcmMdpX+Jhf=UPFapUvkC4BOXk4ahVtH3tUQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> NtSetTimerResolution is a system-wide change, and may have impact on other running applications. ?It may be an option to set it during the execution of profile.run() for example, but I would not enable it just to call time.clock().

I was not proposing to call, but it was trying to say that under
certain circumstances, you may have a better resolution than 15 ms.
Python should not limit the resolution if the OS provides a better
resolution.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:20:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 13:20:50 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327411250.75.0.464495984273.issue13848@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Yes, fixing the conversion block is probably the right approach.
Apparently posixmodule.c uses PyUnicode_FSConverter, perhaps that would work?
(also make sure that the case where a bytes string is given is fixed too:

>>> open(b"\x00")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: ''
)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:21:51 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 24 Jan 2012 13:21:51 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327411311.34.0.477333529154.issue13849@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

My understanding (and recollection, but I don't have notes I can point at to hand) is that one goal that arose from recent VM and language summits was for the CPython test suite to be used as the validating test suite, with CPython-specific tests marked as such so that the other VMs can easily "not conform" to those tests.  Thus my comment that it is helpful to the other VMs to add tests.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:25:17 2012
From: report at bugs.python.org (R. David Murray)
Date: Tue, 24 Jan 2012 13:25:17 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327411517.1.0.0907731912615.issue13849@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Oh, and in case it isn't clear, this request is *coming* from one of the other VMs (pypy), so if my summit recollection is correct, they are in fact "adding a test that they need" by submitting this issue :)

(Or at least they will have once Alex posts the patch.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:30:22 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 13:30:22 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327411822.65.0.388159491842.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Pushed an initial cut to my sandbox branch. Built HTML is attached so you can get a general idea of how it looks (links, etc, obviously won't work).

So far, I have made the split into 3 sections and updated the new (shorter) Sequence Types section.

That section now has 6 subsections:
- Common Sequence Operations
- Immutable Sequence Operations (very short, just mentions hash support)
- Mutable Sequence Operations
- Lists
- Tuples
- Ranges

I haven't really touched the Text and Binary sections as yet - the only changes there are things that I copied down before removing them from the updated Sequence Types section.

----------
Added file: http://bugs.python.org/file24314/stdtypes.html

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:31:45 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Tue, 24 Jan 2012 13:31:45 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327411905.88.0.792996624073.issue13849@psf.upfronthosting.co.za>


Changes by Hynek Schlawack <hs at ox.cx>:


----------
nosy: +hynek

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:42:02 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Tue, 24 Jan 2012 13:42:02 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327412522.06.0.289721878301.issue13848@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

JFTR, file()'s C equivalent is fileio_init and not io_open, I lost track of all the opens. ;)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:46:32 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 24 Jan 2012 13:46:32 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327412792.69.0.881348157826.issue13849@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

If so, I think this change should not checked into the 2.7 branch. Instead, a separate branch should be made for changes not intended for CPython, but for Python implementations in general. Making the Python test suite usable for other implementations is certainly not a feature of Python 2.7.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:49:58 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Tue, 24 Jan 2012 13:49:58 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327412998.16.0.670413631654.issue13849@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

Reconsidering: I think it shouldn't be checked into the cpython *repository*. Instead, if PyPy developers want to contribute changes to the test suite and standard library to improve the standard library, there should be a separate Mercurial repository for that, and PyPy developers might get a blanket write permission to add changes unless they explicitly feel a need for review.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Tue Jan 24 14:51:07 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Tue, 24 Jan 2012 13:51:07 +0000
Subject: [issue13851] Packaging distutils2 for Fedora
In-Reply-To: <1327409259.69.0.346795065424.issue13851@psf.upfronthosting.co.za>
Message-ID: <1327413067.76.0.600616807017.issue13851@psf.upfronthosting.co.za>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

Hi Vikash - thanks for working on this.

It's normal when packaging code "downstream" for Fedora to file a package review request at bugzilla.redhat.com, following the process here:

http://fedoraproject.org/wiki/Package_Review_Process

(sorry if the process seems byzantine; I can help if need be).

I just a did quick search and didn't see one for distutils2 there.

Some immediate things struck me about the specfile:
* it's normal to linewrap the %description clause (the specfile should be readable in a 80-column display)
* you don't need the CFLAGS= stuff for a noarch package

As for the question about things like "pysetup" not being available - are these meant to end up in /usr/bin ?  If so, this may indeed be an issue for upstream distutils2 (and hence for this bug tracker).

Hope this is helpful
Dave

----------
nosy: +dmalcolm

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13851>
_______________________________________

From report at bugs.python.org  Tue Jan 24 15:09:08 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 24 Jan 2012 14:09:08 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327414148.15.0.471726090459.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Note: without the Python docs CSS to create the sidebar, the internal table of contents appears at the *bottom* of the rendered page.

Really, reviewing this sensibly is probably going to require building the docs locally after using hg pull to retrieve the changes from my sandbox.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Tue Jan 24 15:33:52 2012
From: report at bugs.python.org (James Sanders)
Date: Tue, 24 Jan 2012 14:33:52 +0000
Subject: [issue13842] Cannot pickle Ellipsis or NotImplemented
In-Reply-To: <1327346659.12.0.712807876778.issue13842@psf.upfronthosting.co.za>
Message-ID: <1327415632.34.0.139831531678.issue13842@psf.upfronthosting.co.za>


James Sanders <bistromathic1 at gmail.com> added the comment:

I've submitted a patch that just uses save_global to pickle Ellipsis and NotImplemented, so the resulting pickle should be unpicklable anywhere.  I'm completely new to the C API so not sure if the way I am building python strings (to pass to save_global) is correct, but it seems to work.

----------
keywords: +patch
Added file: http://bugs.python.org/file24315/pickling_ellipsis_notimplemented.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13842>
_______________________________________

From report at bugs.python.org  Tue Jan 24 15:56:26 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Tue, 24 Jan 2012 14:56:26 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327416986.5.0.999379525719.issue11457@psf.upfronthosting.co.za>


Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com> added the comment:

st_atim, st_ctim and st_mtim attributes could be instances of a class (implemented in posixmodule.c) similar to:

class timespec(tuple):
    def __init__(self, arg):
        if not isinstance(arg, tuple):
            raise TypeError
        if len(arg) != 2:
            raise TypeError
        if not isinstance(arg[0], int):
            raise TypeError
        if not isinstance(arg[1], int):
            raise TypeError
        self.sec = arg[0]
        self.nsec = arg[1]
        tuple.__init__(self)
    def __add__(self, other):
        if not isinstance(other, timespec):
            raise TypeError
        ns_sum = (self.sec * 10 ** 9 + self.nsec) + (other.sec * 10 ** 9 + other.nsec)
        return timespec(divmod(ns_sum, 10 ** 9))
    def __sub__(self, other):
        if not isinstance(other, timespec):
            raise TypeError
        ns_diff = (self.sec * 10 ** 9 + self.nsec) - (other.sec * 10 ** 9 + other.nsec)
        return timespec(divmod(ns_diff, 10 ** 9))

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Tue Jan 24 16:52:14 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Tue, 24 Jan 2012 15:52:14 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327420334.39.0.229704124266.issue13848@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

So I have good news and bad news. The good is: I fixed it for non-Win platforms and the patch is truly beautiful:

 Lib/test/test_builtin.py |   6 ++++++
 Modules/_io/fileio.c     |  25 ++++---------------------
 2 files changed, 10 insertions(+), 21 deletions(-)

;)

Two problems:

  1. I'm not sure if it's okay for me to put the test where I put it? 
  2. I'm not sure how to fix it for Win32 (and I also can't test it :().

It's just about the case when it's called with a Unicode path name. The current code looks like as following:

    if (PyUnicode_Check(nameobj)) {
        widename = PyUnicode_AsUnicode(nameobj);
        if (widename == NULL)
            return -1;
    }

We can't use the nifty PyUnicode_FSConverter because we want to keep Unicode. So I assume the way to go would be the C equivalent of somthing like:

if '\0' in widename:
  raise TypeError()

Right?

I hope someone would be so kind to implement it, otherwise the patch attached passes all test on Linux and Mac (except for test_recursion_limit but that fails currently for me on the Mac even without my patch).

----------
keywords: +patch
Added file: http://bugs.python.org/file24316/open-nul.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Tue Jan 24 17:10:24 2012
From: report at bugs.python.org (Brett Cannon)
Date: Tue, 24 Jan 2012 16:10:24 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <1327421424.76.0.219056108126.issue11235@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

LGTM

----------
assignee:  -> pitrou
stage: patch review -> commit review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 17:54:36 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 24 Jan 2012 16:54:36 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <E1Rpjdh-0007Lo-Fc@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a2f3d6986bfa by Antoine Pitrou in branch '3.2':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/a2f3d6986bfa

New changeset cb13d8cff050 by Antoine Pitrou in branch 'default':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/cb13d8cff050

New changeset 0499eed74126 by Antoine Pitrou in branch '2.7':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/0499eed74126

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 17:57:32 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 24 Jan 2012 16:57:32 +0000
Subject: [issue11235] Source files with date modifed in 2106 cause
	OverflowError
In-Reply-To: <1297974058.91.0.622548974342.issue11235@psf.upfronthosting.co.za>
Message-ID: <1327424252.42.0.825867729336.issue11235@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Committed, thanks.

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11235>
_______________________________________

From report at bugs.python.org  Tue Jan 24 18:09:57 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Tue, 24 Jan 2012 17:09:57 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327424997.23.0.0130950433212.issue13812@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Patch looks fine. I like the use of "finally" for the flush.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Tue Jan 24 18:12:09 2012
From: report at bugs.python.org (Steven Bethard)
Date: Tue, 24 Jan 2012 17:12:09 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327425129.11.0.665927340177.issue13850@psf.upfronthosting.co.za>


Steven Bethard <steven.bethard at gmail.com> added the comment:

Sounds like an excellent plan to me too.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 18:15:43 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Tue, 24 Jan 2012 17:15:43 +0000
Subject: [issue13841] multiprocessing should use sys.exit() where possible
In-Reply-To: <1327345132.43.0.775594933755.issue13841@psf.upfronthosting.co.za>
Message-ID: <1327425343.13.0.0359375327342.issue13841@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Good point. I don't think those particular behaviors are documented, so I'm not sure whether we need to worry about breaking them.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13841>
_______________________________________

From report at bugs.python.org  Tue Jan 24 18:42:32 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Tue, 24 Jan 2012 17:42:32 +0000
Subject: [issue13853] SystemExit/sys.exit() doesn't print boolean argument
Message-ID: <1327426952.31.0.648876756339.issue13853@psf.upfronthosting.co.za>


New submission from Jon Brandvein <jon.brandvein at gmail.com>:

Raising SystemExit manually, or calling sys.exit, with an argument of "True" or "False" results in no output to the screen. According to Doc/library/exceptions.rst and Doc/library/sys.rst, any object that is not an integer or None should be printed to stderr.

Also, I'm not sure whether this is a bug, but "raise SystemExit(None)" differs from "sys.exit(None)", in that the former produces an exception with an args tuple of "(None,)", and the latter produces one with an empty args tuple.

----------
components: Interpreter Core
messages: 151920
nosy: brandj
priority: normal
severity: normal
status: open
title: SystemExit/sys.exit() doesn't print boolean argument
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13853>
_______________________________________

From report at bugs.python.org  Tue Jan 24 19:08:46 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Tue, 24 Jan 2012 18:08:46 +0000
Subject: [issue13854] multiprocessing: SystemExit from child with non-int,
	non-str arg causes TypeError
Message-ID: <1327428526.26.0.964378708084.issue13854@psf.upfronthosting.co.za>


New submission from Jon Brandvein <jon.brandvein at gmail.com>:

In a child process, raising SystemExit or calling sys.exit with a non-integer, non-string argument value causes a TypeError at Lib/multiprocessing/process.py :: _bootstrap. This is from concatenating the argument with '\n' and writing it to stderr.

Suggested fix: replace
    sys.stderr.write(e.args[0] + '\n')
with
    sys.stderr.write(str(e.args[0]) + '\n')

This problem also occurs when the value is None, but only for raising SystemExit (not calling sys.exit()).

----------
components: Library (Lib)
messages: 151921
nosy: brandj, jnoller
priority: normal
severity: normal
status: open
title: multiprocessing: SystemExit from child with non-int, non-str arg causes TypeError
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13854>
_______________________________________

From report at bugs.python.org  Tue Jan 24 19:44:49 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 24 Jan 2012 18:44:49 +0000
Subject: [issue13852] Doc fixes with patch
In-Reply-To: <1327410885.19.0.883089570823.issue13852@psf.upfronthosting.co.za>
Message-ID: <1327430689.19.0.491836671152.issue13852@psf.upfronthosting.co.za>


Changes by Bo?tjan Mejak <bostjan.mejak at gmail.com>:


----------
nosy: +ezio.melotti

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13852>
_______________________________________

From report at bugs.python.org  Tue Jan 24 20:23:15 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 24 Jan 2012 19:23:15 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <1327432995.45.0.994734828507.issue13840@psf.upfronthosting.co.za>


Changes by Meador Inge <meadori at gmail.com>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Tue Jan 24 20:41:25 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 24 Jan 2012 19:41:25 +0000
Subject: [issue13672] Add co_qualname attribute in code objects
In-Reply-To: <1325106685.0.0.411832471872.issue13672@psf.upfronthosting.co.za>
Message-ID: <CAK1Qooo_Ct4OJRJz=PswVF5ObbaR11arKUc4uiOKgZRCqWeVcA@mail.gmail.com>


Meador Inge <meadori at gmail.com> added the comment:

On Wed, Dec 28, 2011 at 3:11 PM, Eric Snow <report at bugs.python.org> wrote:

> One sticky point is that there isn't a guarantee of one-to-one between function object and code object. ?A code object could be bound to several
> different functions as happens with function definitions (particularly lambdas) inside comprehensions.
>
> Also, if a code object is not associated with a function, i.e. one generated by exec, what should the qualname for the code object be? ?How
> about, in CPython, the code objects created for classes and modules?

We already these issues with 'co_name', though.  These cases can be
treated the same as they are for 'co_name':

'<listcomp>'
>>> compile('[i for i in [1, 2]]', '<foo>', 'exec').co_consts[0].co_qualname
'<listcomp>'
>>> compile('class T: pass', '<foo>', 'exec').co_consts[0].co_qualname
'T'
>>> compile('class T: pass', '<foo>', 'exec').co_consts[0].co_name
'T'
>>> compile('a = 12', '<foo>', 'exec').co_name
'<module>'
>>> compile('a = 12', '<foo>', 'exec').co_qualname
'<module>'
>>> compile('lambda x: x', '<foo>', 'exec').co_consts[0].co_qualname
'<lambda>'
>>> compile('lambda x: x', '<foo>', 'exec').co_consts[0].co_name
'<lambda>'

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13672>
_______________________________________

From report at bugs.python.org  Tue Jan 24 21:05:55 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 24 Jan 2012 20:05:55 +0000
Subject: [issue12501] callable(): remove/amend the deprecation warning in
	Python 2.7
In-Reply-To: <1309865750.34.0.541248470896.issue12501@psf.upfronthosting.co.za>
Message-ID: <E1Rpmcs-0006E6-1K@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 62bd0553693f by Florent Xicluna in branch '2.7':
Issue #12501: merge the discordant NEWS entries.
http://hg.python.org/cpython/rev/62bd0553693f

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12501>
_______________________________________

From report at bugs.python.org  Tue Jan 24 21:09:55 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 24 Jan 2012 20:09:55 +0000
Subject: [issue13672] Add co_qualname attribute in code objects
In-Reply-To: <1325100000.16.0.854542819355.issue13672@psf.upfronthosting.co.za>
Message-ID: <1327435795.91.0.886840574793.issue13672@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

This seems to be a useful feature to me.  Another area where it can help
is with fixing function types.  Currently the qualname can be lost:

>>> def f():
...    def g():
...       pass
...    return g
... 
>>> g = f()
>>> g
<function f.<locals>.g at 0x7f1dac4d8ba0>
>>> types.FunctionType(f.__code__, {})
<function f at 0x7f1dac4dfae0>
>>> types.FunctionType(g.__code__, {})

There is no way to specify a qualname when constructing a
'FunctionType'.  The name is derived from the code object.  The
qualname could be too if we fix this issue.  I will open another issue
for the 'FunctionType' problem.

I have attached a WIP patch.  It passes the full test suite.  I am 
somewhat unsure about the CAPI changes, though.  They are a breaking API
change.  Do we need a *_QualName version instead like we did for
'PyFunction_NewWithQualName'?  I think probably so.  I will add doc
updates to the patch once the API stuff gets worked out.

----------
keywords: +patch
Added file: http://bugs.python.org/file24317/issue13672-v0.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13672>
_______________________________________

From report at bugs.python.org  Tue Jan 24 21:14:47 2012
From: report at bugs.python.org (Meador Inge)
Date: Tue, 24 Jan 2012 20:14:47 +0000
Subject: [issue13855] Add qualname support to types.FunctionType
Message-ID: <1327436087.83.0.0965982153281.issue13855@psf.upfronthosting.co.za>


New submission from Meador Inge <meadori at gmail.com>:

As mentioned in issue13672 currently there is no way to specify a 
qualname on types.FunctionType:

>>> def f():
...    def g():
...       pass
...    return g
... 
>>> g = f()
>>> g
<function f.<locals>.g at 0x7f1dac4d8ba0>
>>> types.FunctionType(f.__code__, {})
<function f at 0x7f1dac4dfae0>
>>> types.FunctionType(g.__code__, {})

If issue13672 is fixed, then the qualname can be derived from the passed
in code object like name is.  We can also add an optional parameter to
the FunctionType constructor for qualname.

I am working on a patch relative to the one I just posted for 
issue13672.

----------
components: Library (Lib)
messages: 151925
nosy: meador.inge, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: Add qualname support to types.FunctionType
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13855>
_______________________________________

From report at bugs.python.org  Tue Jan 24 21:20:56 2012
From: report at bugs.python.org (Xavier Morel)
Date: Tue, 24 Jan 2012 20:20:56 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327436456.97.0.540077296119.issue13850@psf.upfronthosting.co.za>


Xavier Morel <xavier.morel at masklinn.net> added the comment:

> The "Parameters" column would span multiple lines, with one parameter and a brief description of the parameter on each line.

I started looking into that, and it turns out that's more annoying than expected: a bunch of parameters are shared by many (to all) actions, leading to lots of duplications in the table. And the full matrix of actions to parameters is not really explained in the doc.

In fact, I'm coming around to thinking a matrix of the interaction between actions and arguments could be better and clearer than a table of actions with parameters bunched together at the end.

In any case, it would certainly be more maintainable... except for rST not really having support for attributes on data tables, and (as far as I can tell) can't handle horizontal headers.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Tue Jan 24 21:36:24 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Tue, 24 Jan 2012 20:36:24 +0000
Subject: [issue13855] Add qualname support to types.FunctionType
In-Reply-To: <1327436087.83.0.0965982153281.issue13855@psf.upfronthosting.co.za>
Message-ID: <1327437384.26.0.34223118617.issue13855@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13855>
_______________________________________

From report at bugs.python.org  Tue Jan 24 22:39:03 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Tue, 24 Jan 2012 21:39:03 +0000
Subject: [issue13846] Add time.monotonic() function
In-Reply-To: <1327364549.64.0.960699645726.issue13846@psf.upfronthosting.co.za>
Message-ID: <1327441143.28.0.114089193638.issue13846@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13846>
_______________________________________

From report at bugs.python.org  Wed Jan 25 00:02:55 2012
From: report at bugs.python.org (Nathanael Noblet)
Date: Tue, 24 Jan 2012 23:02:55 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
Message-ID: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>


New submission from Nathanael Noblet <nathanaelnoblet at gmail.com>:

If I wanted to create a HTTPS connection to a xmlrpc server *and* validate its certificate, I need to override the HTTPSConnection and SafeTransport objects. However it seems like they could easily support both methods. At least it seems to work both ways for me

----------
files: xmlrpc-http-withcerts.patch
keywords: patch
messages: 151927
nosy: Nathanael.Noblet, loewis
priority: normal
severity: normal
status: open
title: xmlrpc / httplib changes to allow for certificate verification
type: enhancement
versions: Python 2.7
Added file: http://bugs.python.org/file24318/xmlrpc-http-withcerts.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Wed Jan 25 02:18:34 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 25 Jan 2012 01:18:34 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327454314.95.0.637234082115.issue13850@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Having a second table of "Applicable Parameters" could definitely work. I don't think the "no horizontal headers" limitation should be a big problem - the matrix should be readable even if the action names are just listed in an ordinary column.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Wed Jan 25 02:41:19 2012
From: report at bugs.python.org (Mark Nottingham)
Date: Wed, 25 Jan 2012 01:41:19 +0000
Subject: [issue7097] urlparse.urljoin of simple "http://" and "somedomain.com"
	produces incorrect result
In-Reply-To: <1255196414.88.0.948496525016.issue7097@psf.upfronthosting.co.za>
Message-ID: <1327455679.53.0.722400588928.issue7097@psf.upfronthosting.co.za>


Mark Nottingham <mnot at mnot.net> added the comment:

http: and http:// are both valid base URIs; see RFC3986. 

More to the point, it's a useful thing to use a scheme as a base URI; many users omit the HTTP:// from their URIs.

----------
nosy: +mnot

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7097>
_______________________________________

From report at bugs.python.org  Wed Jan 25 03:20:04 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Wed, 25 Jan 2012 02:20:04 +0000
Subject: [issue13850] Summary tables for argparse add_argument options
In-Reply-To: <1327384498.23.0.981832544617.issue13850@psf.upfronthosting.co.za>
Message-ID: <1327458004.63.0.0268207277581.issue13850@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

+1 from me.  The docs in their present form are a great tutorial but are a total failure as a quick reference.

Besides having table of parameters, it may also be worthwhile to move some of the examples to a HOWTO document (much as was done with the logging package).

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13850>
_______________________________________

From report at bugs.python.org  Wed Jan 25 04:14:26 2012
From: report at bugs.python.org (Meador Inge)
Date: Wed, 25 Jan 2012 03:14:26 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <1327461266.32.0.943331805695.issue13840@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

The 'create_unicode_buffer' docs are currently wrong too:

"""
If the first parameter is a bytes object, it is converted into an 
unicode string according to ctypes conversion rules.
"""

>>> ctypes.create_unicode_buffer(b'foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meadori/src/python/cpython/Lib/ctypes/__init__.py", line 294, in create_unicode_buffer
    buf.value = init
TypeError: unicode string expected instead of bytes instance

The attached patch fixes the documentation and exception messages.

Although, it might be more friendly to implement things according to
the current docs (i.e. do the conversions).

----------
components: +ctypes
keywords: +patch
stage:  -> patch review
versions: +Python 3.3
Added file: http://bugs.python.org/file24319/issue13840.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Wed Jan 25 04:37:15 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 25 Jan 2012 03:37:15 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Message-ID: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>


New submission from Nick Coghlan <ncoghlan at gmail.com>:

As far I am aware, the simplest way to indent a multi-line string is with the following snippet:

    '\n'.join((4 * ' ') + x for x in s.splitlines())

It would be a lot simpler and clearer if I could just write that as "textwrap.indent(s, 4 * ' ')".

(i.e. indent would accept a prefix string to be inserted before each line in the supplied string, as in the original comprehension)

----------
components: Library (Lib)
messages: 151932
nosy: ncoghlan
priority: normal
severity: normal
stage: needs patch
status: open
title: Add textwrap.indent() as counterpart to textwrap.dedent()
type: enhancement
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Wed Jan 25 06:13:48 2012
From: report at bugs.python.org (Matt Joiner)
Date: Wed, 25 Jan 2012 05:13:48 +0000
Subject: [issue13858] readline fails on nonblocking,
	unbuffered io.FileIO objects
Message-ID: <1327468428.38.0.416513768906.issue13858@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

_io._IOBase.readline doesn't seem to like _io.FileIO.read returning None, which occurs when it's unbuffered and nonblocking. (Modules/_io/fileio.c:745 in trunk). Can this be handled some other way?

$ python3.3
Python 3.3.0a0 (default:fb0f4fe8123e+, Jan 24 2012, 11:21:36) 
[GCC 4.6.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, io
>>> r, w = os.pipe2(os.O_NONBLOCK)
>>> f = io.open(r, 'rb', 0)
>>> f.readline()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: read() should have returned a bytes object, not 'NoneType'

----------
components: IO, Library (Lib)
messages: 151933
nosy: anacrolix
priority: normal
severity: normal
status: open
title: readline fails on nonblocking, unbuffered io.FileIO objects
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13858>
_______________________________________

From report at bugs.python.org  Wed Jan 25 07:41:57 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 25 Jan 2012 06:41:57 +0000
Subject: [issue12500] Skip test_ssl.test_connect_ex() on connection error
In-Reply-To: <1309854315.46.0.604055413236.issue12500@psf.upfronthosting.co.za>
Message-ID: <1327473717.07.0.0611273969861.issue12500@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12500>
_______________________________________

From report at bugs.python.org  Wed Jan 25 07:44:46 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 25 Jan 2012 06:44:46 +0000
Subject: [issue12804] "make test" fails on systems without internet access
In-Reply-To: <1313935702.14.0.838210022398.issue12804@psf.upfronthosting.co.za>
Message-ID: <E1Rpwb6-0008FT-Ri@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 585d3664da89 by Nadeem Vawda in branch 'default':
Issue #12804: Fix test failures on systems without internet access.
http://hg.python.org/cpython/rev/585d3664da89

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12804>
_______________________________________

From report at bugs.python.org  Wed Jan 25 08:32:07 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 25 Jan 2012 07:32:07 +0000
Subject: [issue12804] "make test" fails on systems without internet access
In-Reply-To: <1313935702.14.0.838210022398.issue12804@psf.upfronthosting.co.za>
Message-ID: <1327476727.34.0.713833955905.issue12804@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12804>
_______________________________________

From report at bugs.python.org  Wed Jan 25 09:22:32 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Wed, 25 Jan 2012 08:22:32 +0000
Subject: [issue13858] readline fails on nonblocking,
	unbuffered io.FileIO objects
In-Reply-To: <1327468428.38.0.416513768906.issue13858@psf.upfronthosting.co.za>
Message-ID: <1327479752.63.0.800226260613.issue13858@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Can this be handled some other way?

Yeah, that's an hairy issue.
See #13322 for the details.

----------
nosy: +neologix
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> buffered read() and write() does not raise BlockingIOError

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13858>
_______________________________________

From report at bugs.python.org  Wed Jan 25 09:30:37 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Wed, 25 Jan 2012 08:30:37 +0000
Subject: [issue13841] multiprocessing should use sys.exit() where possible
In-Reply-To: <1327405778.02.0.881326717424.issue13841@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3KVyjYawzOgJaM7q1abA=oXshWLcaHQU2O00TVbuBkZA@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> * atexit callbacks are NOT run (although multiprocessing.util._exit_function IS run),

It may be a good thing after a fork() (e.g. you don't want to remove
the same file twice), but it most definitely looks wrong for a new
interpreter process (? la Windows / fork() + exec()).

> * the main thread does NOT wait for non-daemonic background threads.

Same thing here.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13841>
_______________________________________

From report at bugs.python.org  Wed Jan 25 09:40:07 2012
From: report at bugs.python.org (Matt Joiner)
Date: Wed, 25 Jan 2012 08:40:07 +0000
Subject: [issue13322] buffered read() and write() does not raise
	BlockingIOError
In-Reply-To: <1320246535.71.0.465783773129.issue13322@psf.upfronthosting.co.za>
Message-ID: <1327480807.13.0.855324044595.issue13322@psf.upfronthosting.co.za>


Matt Joiner <anacrolix at gmail.com> added the comment:

The patches only fix write? What about read?

http://bugs.python.org/issue13858

----------
nosy: +anacrolix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13322>
_______________________________________

From report at bugs.python.org  Wed Jan 25 10:27:07 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 25 Jan 2012 09:27:07 +0000
Subject: [issue13852] Doc fixes with patch
In-Reply-To: <1327410885.19.0.883089570823.issue13852@psf.upfronthosting.co.za>
Message-ID: <1327483627.91.0.100906911288.issue13852@psf.upfronthosting.co.za>


Changes by Bo?tjan Mejak <bostjan.mejak at gmail.com>:


----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13852>
_______________________________________

From report at bugs.python.org  Wed Jan 25 10:45:03 2012
From: report at bugs.python.org (Yuval Greenfield)
Date: Wed, 25 Jan 2012 09:45:03 +0000
Subject: [issue9285] Add a profile decorator to profile and cProfile
In-Reply-To: <1279375274.49.0.102313390752.issue9285@psf.upfronthosting.co.za>
Message-ID: <1327484703.63.0.520865803932.issue9285@psf.upfronthosting.co.za>


Changes by Yuval Greenfield <ubershmekel at gmail.com>:


----------
nosy: +ubershmekel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9285>
_______________________________________

From report at bugs.python.org  Wed Jan 25 11:17:34 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 25 Jan 2012 10:17:34 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327486654.89.0.598093688886.issue13857@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

David Miller pointed out a shorter spelling:

    s.replace('\n', '\n' + (4 * ' '))

Still not particularly obvious to the reader (or writer), though.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Wed Jan 25 12:06:01 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 25 Jan 2012 11:06:01 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327489561.69.0.280670857261.issue13703@psf.upfronthosting.co.za>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I'm attaching a patch which implements a hybrid approach:
  hybrid-approach-dmalcolm-2012-01-25-001.patch

This is a blend of various approaches from the discussion, taking aspects of both hash randomization *and* collision-counting.

It incorporates code from
  amortized-probe-counting-dmalcolm-2012-01-21-003.patch
  backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch
  random-8.patch
along with ideas from:
  http://mail.python.org/pipermail/python-dev/2012-January/115812.html

The patch is against the default branch (although my primary goal here is eventual backporting).

As per haypo's random-8.patch, a randomization seed is read at startup.

By default, the existing hash() values are preserved, and no randomization is performed until a dict comes under attack.  This preserves existing behaviors (such as dict ordering) under non-attack conditions.

For large dictionaries, it reuses the ma_smalltable area to track the amortized cost of all modifications to this dictionary.

When the cost exceeds a set threshold, we convert the dictionary's ma_lookup function from lookdict/lookdict_unicode to a "paranoid" variant.  These variants ignore the hash passed in, and instead uses a new function:
   PyObject_RandomizedHash(obj)
to give a second hash value, which is fixed value for a given object within the process, but not predictable to an attacker for the most high-risk types (PyUnicodeObject and PyBytesObject).

This patch is intended as a base for backporting, and takes it as given that we can't expand PyTypeObject or hide something in one of the Py*Methods tables; iirc we've run out of tp_flags in 2.*, hence we're forced to implement PyObject_RandomizedHash via direct ob_type comparison, for the most high-risk types.  

As noted in http://bugs.python.org/issue13703#msg151870:

> I would NOT worry about hash repeatability for integers and
> complex data structures.  It is not at the core of the common problem
> (maybe a couple application specific problems but not a general "all
> python web apps" severity problem).

> Doing it for base byte string and unicode string like objects is
> sufficient.

[We can of course implement hash randomization by default in 3.3, but I care more about getting a fix into the released branches ASAP]

Upon transition of a dict to paranoid mode, the hash values become unpredictable to an attacker, and all PyDictEntries are rebuilt based on the new hash values.

Handling the awkward case within custom ma_lookup functions allows us to move most of the patch from out of the fast path, and lookdict/lookdict_unicode only need minimal changes (stat gathering for the above cost analysis tracking).

Once a dict has transitioned to paranoid mode, it isn't using PyObject_Hash anymore, and thus isn't using cached object values, performing a more expensive calculation, but I believe this calculation is essentially constant-time.

This preserves hash() and dict order for the cases where you're not under attack, and gracefully handles the attack without having to raise an exception: it doesn't introduce any new exception types.

It preserves ABI, assuming no-one else is reusing ma_smalltable.

It is suitable for backporting to 3.2, 2.7, and earlier (I'm investigating fixing this going all the way back to Python 2.2)

Under the old implementation, there were 4 types of PyDictObject, given these two booleans:
  * "small vs large" i.e ma_table == ma_smalltable vs ma_table != ma_smalltable
  * "all keys are str" vs arbitary keys i.e ma_lookdict == lookdict_unicode vs lookdict

Under this implementation, this doubles to 8 kinds, adding the boolean:
  * normal hash vs randomized hash (normal vs "paranoid").

This is expressed via the ma_lookdict callback, adding two new variants, lookdict_unicode_paranoid and lookdict_paranoid

Note that if a paranoid dict goes small again (ma_table == ma_smalltable), it stays paranoid.  This is for simplicity: it avoids having to rebuild all of the non-randomized me_hash values again (which could fail).

Naturally the patch adds selftests.  I had to add some diagnostic methods to support them; dict gains _stats() and _make_paranoid() methods, and sys gains a _getrandomizedhash() method.  These could be hidden more thoroughly if need be (see DICT_PROTECTION_TRACKING in dictobject.c).  Amongst other things, the selftests measure wallclock time taken for various dict operations (and so might introduce failures on a heavily-loaded machine, I guess).

Hopefully this approach is a viable way forward.

Caveats and TODO items:

TODO: I haven't yet tuned the safety threshold.  According to http://bugs.python.org/issue13703#msg151850:
> slot collisions are much more frequent than
> hash collisions, which only account for less than 0.01% of all
> collisions.
>
> It also shows that slot collisions in the low 1-10 range are
> most frequent, with very few instances of a dict lookup
> reaching 20 slot collisions (less than 0.0002% of all
> collisions).

This suggests that the threshold of 32 slot/hash collisions per lookup may already be high enough.

TODO: in a review of an earlier version of the complexity detection idea, Antoine Pitrou suggested that make the protection scale factor be a run-time configurable value, rather than a #define.  This isn't done yet.

TODO: run more extensive tests (e.g. Django and Twisted), monitoring the worst-case complexity that's encountered

TODO: not yet benchmarked and optimized.  I want to get feedback on the approach before I go in and hand-optimize things (e.g. by hand-inlining check_iter_count, and moving the calculations out of the loop etc).  I believe any performance issues ought to be fixable, in that the we can get the cost of this for the "we're not under attack" case to be negligible, and the "under attack" case should transition from O(N^2) to O(N), albeit it with a larger constant factor.

TODO: this doesn't cover sets, but assuming this approach works, the patch can be extended to cover it in an analogous way.

TODO: should it cover PyMemoryViewObject, buffer object, etc?

TODO: should it cover the hashing in Modules/expat/xmlparse.c?  FWIW I rip this code out when doing my downstream builds in RHEL and Fedora, and instead dynamically link against a system copy of expat

TODO: only tested on Linux so far (which is all I've got).  Fedora 15 x86_64 fwiw

 Doc/using/cmdline.rst     |    6 
 Include/bytesobject.h     |    2 
 Include/object.h          |    8 
 Include/pythonrun.h       |    2 
 Include/unicodeobject.h   |    2 
 Lib/os.py                 |   17 --
 Lib/test/regrtest.py      |    5 
 Lib/test/test_dict.py     |  298 +++++++++++++++++++++++++++++++++++++
 Lib/test/test_hash.py     |   53 ++++++
 Lib/test/test_os.py       |   35 +++-
 Makefile.pre.in           |    1 
 Modules/posixmodule.c     |  126 ++-------------
 Objects/bytesobject.c     |    7 
 Objects/dictobject.c      |  369 +++++++++++++++++++++++++++++++++++++++++++++-
 Objects/object.c          |   37 ++++
 Objects/unicodeobject.c   |   51 ++++++
 PCbuild/pythoncore.vcproj |    4 
 Python/pythonrun.c        |    3 
 Python/sysmodule.c        |   16 +
 b/Python/random.c         |  268 +++++++++++++++++++++++++++++++++
 20 files changed, 1173 insertions(+), 137 deletions(-)

----------
Added file: http://bugs.python.org/file24320/hybrid-approach-dmalcolm-2012-01-25-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 12:16:10 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Wed, 25 Jan 2012 11:16:10 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327490170.17.0.109982937612.issue13857@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

If such a function is added, I'd like the option to not indent empty lines: trailing spaces are often not a good idea.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Wed Jan 25 13:45:34 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 25 Jan 2012 12:45:34 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327489561.69.0.280670857261.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327495499.2388.12.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I've found a bug in my patch; insertdict writes the old non-randomized
hash value into me_hash at:
        ep->me_hash = hash;
rather than using the randomized hash, leading to issues when tested
against a real attack.

I'm looking into fixing it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 13:47:38 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Wed, 25 Jan 2012 12:47:38 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327495499.2388.12.camel@surprise>
Message-ID: <CAFRnB2WBdTNQPN-QCUsMCWK4Swmf=qety=fcHo7rRq0x+8eaYw@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

On Wed, Jan 25, 2012 at 7:45 AM, Dave Malcolm <report at bugs.python.org>wrote:

>
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
>
> I've found a bug in my patch; insertdict writes the old non-randomized
> hash value into me_hash at:
>        ep->me_hash = hash;
> rather than using the randomized hash, leading to issues when tested
> against a real attack.
>
> I'm looking into fixing it.
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

What happens if I have a dict with str keys that goes into paranoid mode,
and I then do:

class A(object):
   def __init__(self, s):
       self.s = s
   def __eq__(self, other):
       return self.s == other
   def __hash__(self):
       return hash(self.s)

d[A("some str that's a key in d")]

Is it still able to find the value?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 13:53:35 2012
From: report at bugs.python.org (STINNER Victor)
Date: Wed, 25 Jan 2012 12:53:35 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1327416986.5.0.999379525719.issue11457@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwbUhybdhoqSsmwWHvDPfftCZg31ZhagNOGjBWxAE8njtQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Attached patch adds an optional format argument to time.time():
time.time("float") is the same than time.time(), but
time.time("decimal") returns a Decimal object. The Decimal object
stores the resolution of the clock and doesn't loose lower bits for
big numbers. I configured the Decimal context to be able to store
10,000 years in seconds with a resolution of 1 nanosecond and
ROUND_HALF_EVEN rounding method.

Example: time.time("decimal") returns Decimal('1327495951.346024').

It is really cool for have directly the resolution in the result,
because the resolution may change at each call: time.time() has 3
different implementations (with fallbacks), each has a different
resolution. time.clock() has also 2 implementations (one is used as a
fallback) with different resolution.

The internal time_to_format() takes integer arguments: the floating
part is written as (floatpart, divisor).

If you like the idea, I will also write a patch for time.clock(),
time.wallclock() and time.clock_gettime(), and also maybe for
time.clock_getres().

We may use a registry to allow to add user defined formats, but I
prefer to keep the patch simple (only allow "float" and "decimal"
right now).

----------
Added file: http://bugs.python.org/file24321/time_decimal.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________
-------------- next part --------------
changeset:   74495:6fcdb1d46abb
tag:         tip
user:        Victor Stinner <vstinner at wyplay.com>
date:        Tue Jan 24 14:27:20 2012 +0100
files:       Doc/library/time.rst Modules/timemodule.c
description:
Add a format optional argument to time.time() to return the current time
as a decimal.Decimal object, instead of a float.


diff --git a/Doc/library/time.rst b/Doc/library/time.rst
--- a/Doc/library/time.rst
+++ b/Doc/library/time.rst
@@ -444,9 +444,10 @@ The module defines the following functio
    :exc:`TypeError` is raised.
 
 
-.. function:: time()
+.. function:: time(format="float")
 
-   Return the time as a floating point number expressed in seconds since the epoch,
+   Return the time as a floating point number, or a :class:`decimal.Decimal` if
+   format is ``"decimal"``, expressed in seconds since the epoch,
    in UTC.  Note that even though the time is always returned as a floating point
    number, not all systems provide time with a better precision than 1 second.
    While this function normally returns non-decreasing values, it can return a
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -44,16 +44,246 @@
 static int floatsleep(double);
 static double floattime(void);
 
+typedef unsigned long floatpart_t;
+
+static PyObject*
+PyLong_FromTimeT(time_t value)
+{
+#ifdef HAVE_LONG_LONG
+    return PyLong_FromUnsignedLongLong(value);
+#else
+    assert(sizeof(time_t) <= sizeof(unsigned long));
+    return PyLong_FromUnsignedLong(value);
+#endif
+}
+
+static PyObject*
+PyLong_FromFloatpartT(floatpart_t value)
+{
+#ifdef HAVE_LONG_LONG
+    return PyLong_FromUnsignedLongLong(value);
+#else
+    assert(sizeof(floatpart_t) <= sizeof(unsigned long));
+    return PyLong_FromUnsignedLong(value);
+#endif
+}
+
+/* Convert a timestamp to a decimal.Decimal object of the requested
+   resolution. */
+static PyObject*
+time_to_decimal(time_t value,
+                floatpart_t floatpart, floatpart_t divisor,
+                int resolution)
+{
+    static PyObject* module = NULL;
+    static PyObject* decimal = NULL;
+    static PyObject* context = NULL;
+    /* 10^resolution cache, dictionary of int=>Decimal */
+    static PyObject* exponents = NULL;
+    PyObject *t = NULL;
+
+    if (!module) {
+        module = PyImport_ImportModuleNoBlock("decimal");
+        if (module == NULL)
+            return NULL;
+    }
+
+    if (!decimal) {
+        decimal = PyObject_GetAttrString(module, "Decimal");
+        if (decimal == NULL)
+            return NULL;
+    }
+
+    if (context == NULL)
+    {
+        /* context = decimal.Context(22, rounding=decimal.ROUND_HALF_EVEN) */
+        PyObject *context_class, *rounding;
+        /* Use 12 decimal digits to store 10,000 years in seconds + 9
+           decimal digits for the floating part in nanoseconds + 1 decimal
+           digit to round correctly */
+        context_class = PyObject_GetAttrString(module, "Context");
+        if (context_class == NULL)
+            return NULL;
+        rounding = PyObject_GetAttrString(module, "ROUND_HALF_EVEN");
+        if (rounding == NULL)
+        {
+            Py_DECREF(context_class);
+            return NULL;
+        }
+        context = PyObject_CallFunction(context_class, "iO", 22, rounding);
+        Py_DECREF(context_class);
+        Py_DECREF(rounding);
+        if (context == NULL)
+            return NULL;
+    }
+
+    /* t = decimal.Decimal(value, context) */
+    if (value) {
+        PyObject *f = PyLong_FromTimeT(value);
+        t = PyObject_CallFunction(decimal, "OO", f, context);
+        Py_CLEAR(f);
+    }
+    else {
+        t = PyObject_CallFunction(decimal, "iO", 0, context);
+    }
+    if (t == NULL)
+        return NULL;
+
+    if (floatpart)
+    {
+        /* t += decimal.Decimal(floatpart, ctx) / decimal.Decimal(divisor, ctx) */
+        PyObject *a, *b, *c, *d, *x;
+
+        x = PyLong_FromFloatpartT(floatpart);
+        if (x == NULL)
+            goto error;
+        a = PyObject_CallFunction(decimal, "OO", x, context);
+        Py_CLEAR(x);
+        if (a == NULL)
+            goto error;
+
+        x = PyLong_FromFloatpartT(divisor);
+        if (x == NULL)
+        {
+            Py_DECREF(a);
+            goto error;
+        }
+        b = PyObject_CallFunction(decimal, "OO", x, context);
+        Py_CLEAR(x);
+        if (b == NULL)
+        {
+            Py_DECREF(a);
+            goto error;
+        }
+
+        c = PyNumber_TrueDivide(a, b);
+        Py_DECREF(a);
+        Py_DECREF(b);
+        if (c == NULL)
+            goto error;
+
+        d = PyNumber_Add(t, c);
+        Py_DECREF(c);
+        if (d == NULL)
+            goto error;
+        Py_DECREF(t);
+        t = d;
+    }
+
+    if (resolution != 0) {
+        PyObject *key, *exponent, *quantized;
+
+        if (exponents == NULL) {
+            exponents = PyDict_New();
+            if (exponents == NULL)
+                goto error;
+        }
+
+        key = PyLong_FromLong(resolution);
+        if (key == NULL)
+            goto error;
+        exponent = PyDict_GetItem(exponents, key);
+        if (exponent == NULL) {
+            /* exponent = decimal.Decimal(10) ** decimal.Decimal(resolution) */
+            PyObject *ten, *pow;
+
+            ten = PyObject_CallFunction(decimal, "i", 10);
+            if (ten == NULL) {
+                Py_DECREF(key);
+                goto error;
+            }
+
+            pow = PyObject_CallFunction(decimal, "O", key);
+            if (pow == NULL) {
+                Py_DECREF(key);
+                Py_DECREF(ten);
+                goto error;
+            }
+
+            exponent = PyNumber_Power(ten, pow, Py_None);
+            Py_DECREF(ten);
+            Py_DECREF(pow);
+            if (exponent == NULL) {
+                Py_DECREF(key);
+                goto error;
+            }
+
+            if (PyDict_SetItem(exponents, key, exponent) < 0) {
+                Py_DECREF(key);
+                Py_DECREF(exponent);
+                goto error;
+            }
+            Py_DECREF(key);
+        }
+
+        /* t = t.quantize(exponent, None, context) */
+        quantized = PyObject_CallMethod(t, "quantize", "OOO", exponent, Py_None, context);
+        if (quantized == NULL)
+            goto error;
+        Py_DECREF(t);
+        t = quantized;
+    }
+    return t;
+
+error:
+    Py_XDECREF(t);
+    return NULL;
+}
+
+static PyObject*
+time_to_float(time_t value,
+              floatpart_t floatpart, floatpart_t divisor,
+              int resolution)
+{
+    double t;
+    t = (double)value;
+    t += (double)floatpart / (double)divisor;
+    return PyFloat_FromDouble(t);
+}
+
+/* Convert an integer timestamp to the requested format. Arguments:
+    - value: int or float
+    - floatpart: floating part, int or float, can be zero
+    - divisor: divisor of the floating part, int or float, cannot be zero
+    - format: "float", "decimal", "tuple"
+
+   The result type depends on the requested format.
+
+   Raise a ValueError if the format is unknown. */
+static PyObject*
+time_to_format(time_t value,
+               floatpart_t floatpart, floatpart_t divisor,
+               int resolution,
+               const char *format)
+{
+    assert(divisor != 0);
+    if (!format || strcmp(format, "float") == 0) {
+        return time_to_float(value, floatpart, divisor, resolution);
+    }
+    else if (strcmp(format, "decimal") == 0) {
+        return time_to_decimal(value, floatpart, divisor, resolution);
+    }
+    else {
+        PyErr_Format(PyExc_ValueError, "Unknown format: %s", format);
+        return NULL;
+    }
+}
+
 static PyObject *
-time_time(PyObject *self, PyObject *unused)
+time_time(PyObject *self, PyObject *args)
 {
-    double secs;
-    secs = floattime();
-    if (secs == 0.0) {
+    const char *format = NULL;
+    _PyTime_timeval t;
+
+    if (!PyArg_ParseTuple(args, "|s:time", &format))
+        return NULL;
+
+    _PyTime_gettimeofday(&t);
+    if (t.tv_sec == 0 && t.tv_usec == 0) {
         PyErr_SetFromErrno(PyExc_IOError);
         return NULL;
     }
-    return PyFloat_FromDouble(secs);
+    return time_to_format(t.tv_sec, t.tv_usec, 1000000, -6, format);
 }
 
 PyDoc_STRVAR(time_doc,
@@ -893,7 +1123,7 @@ PyInit_timezone(PyObject *m) {
 
 
 static PyMethodDef time_methods[] = {
-    {"time",            time_time, METH_NOARGS, time_doc},
+    {"time",            time_time, METH_VARARGS, time_doc},
 #if (defined(MS_WINDOWS) && !defined(__BORLANDC__)) || defined(HAVE_CLOCK)
     {"clock",           time_clock, METH_NOARGS, clock_doc},
 #endif


From report at bugs.python.org  Wed Jan 25 14:12:24 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Wed, 25 Jan 2012 13:12:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327497144.77.0.14827612434.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

> Is it still able to find the value?

Probably not. :( 

That's exactly why I stopped thinking about all two-hash-functions or rehashing ideas.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 14:59:03 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Wed, 25 Jan 2012 13:59:03 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327499943.96.0.468988827529.issue13857@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I'd actually suggest that as the default behaviour (and is a good argument in favour of a dedicated function in textwrap - both suggested alternatives will blithely add whitespace to otherwise empty lines).

To handle the empty line requires either switching to an re.sub() based solution or adding a conditional expression:
    '\n'.join(((4 * ' ') + x if x else x) for x in s.splitlines())

I should probably also explicitly address the "why not textwrap.fill()?" alternative: because fill() does a lot more than simple indenting.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Wed Jan 25 15:26:05 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Wed, 25 Jan 2012 14:26:05 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327501565.97.0.23191226018.issue13857@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

> If such a function is added, I'd like the option to not indent empty lines: trailing spaces are often not a good idea.

>From dedent's documentation, it wasn't immediately clear to me that it ignores blank lines when determining common whitespace. (In fact the comment in the example suggests otherwise.) Perhaps a note could be added to the documentation when this change is made?

----------
nosy: +brandj

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Wed Jan 25 15:28:03 2012
From: report at bugs.python.org (Tycho Andersen)
Date: Wed, 25 Jan 2012 14:28:03 +0000
Subject: [issue13051] Infinite recursion in curses.textpad.Textbox
In-Reply-To: <1317170831.29.0.145546521568.issue13051@psf.upfronthosting.co.za>
Message-ID: <1327501683.19.0.427352978958.issue13051@psf.upfronthosting.co.za>


Tycho Andersen <tycho at tycho.ws> added the comment:

Hi, any movement on this?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13051>
_______________________________________

From report at bugs.python.org  Wed Jan 25 15:32:58 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Wed, 25 Jan 2012 14:32:58 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>
Message-ID: <1327501978.73.0.947200982818.issue13856@psf.upfronthosting.co.za>


Changes by Senthil Kumaran <senthil at uthcode.com>:


----------
nosy: +orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Wed Jan 25 16:20:50 2012
From: report at bugs.python.org (Matt Joiner)
Date: Wed, 25 Jan 2012 15:20:50 +0000
Subject: [issue13859] Lingering StandardError in logging module
Message-ID: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

There's a lingering StandardError referenced in the logging module.

StandardError was removed in Python 3, and execution across this code path generates a NameError:

  File "/home/matt/src/cpython/Lib/logging/__init__.py", line 291, in __init__
    except StandardError: #pragma: no cover
NameError: global name 'StandardError' is not defined

Patch attached.

----------
components: 2to3 (2.x to 3.x conversion tool), Library (Lib)
files: logging-uncovered-standarderror.patch
keywords: patch
messages: 151948
nosy: anacrolix
priority: normal
severity: normal
status: open
title: Lingering StandardError in logging module
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24322/logging-uncovered-standarderror.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Wed Jan 25 16:27:44 2012
From: report at bugs.python.org (Brian Curtin)
Date: Wed, 25 Jan 2012 15:27:44 +0000
Subject: [issue13859] Lingering StandardError in logging module
In-Reply-To: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>
Message-ID: <1327505264.24.0.271860502831.issue13859@psf.upfronthosting.co.za>


Changes by Brian Curtin <brian at python.org>:


----------
components:  -2to3 (2.x to 3.x conversion tool)
nosy: +vinay.sajip
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Wed Jan 25 17:07:47 2012
From: report at bugs.python.org (Catalin Iacob)
Date: Wed, 25 Jan 2012 16:07:47 +0000
Subject: [issue10042] total_ordering
In-Reply-To: <1286440012.74.0.624864175042.issue10042@psf.upfronthosting.co.za>
Message-ID: <1327507667.63.0.824011194876.issue10042@psf.upfronthosting.co.za>


Changes by Catalin Iacob <iacobcatalin at gmail.com>:


----------
nosy: +catalin.iacob

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10042>
_______________________________________

From report at bugs.python.org  Wed Jan 25 17:15:11 2012
From: report at bugs.python.org (Matt Joiner)
Date: Wed, 25 Jan 2012 16:15:11 +0000
Subject: [issue13859] Lingering StandardError in logging module
In-Reply-To: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>
Message-ID: <1327508111.94.0.221444750943.issue13859@psf.upfronthosting.co.za>


Matt Joiner <anacrolix at gmail.com> added the comment:

Interesting this also occurs in 3.2 and 2.7, but not 2.6 or 3.1. It's probably not an error in 2.x tho.

----------
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Wed Jan 25 17:48:47 2012
From: report at bugs.python.org (Brett Cannon)
Date: Wed, 25 Jan 2012 16:48:47 +0000
Subject: [issue13853] SystemExit/sys.exit() doesn't print boolean argument
In-Reply-To: <1327426952.31.0.648876756339.issue13853@psf.upfronthosting.co.za>
Message-ID: <1327510127.78.0.28673385188.issue13853@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

Thanks for going to the trouble to report this, but this is working as intended since bool is a subclass of int.

----------
nosy: +brett.cannon
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13853>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:25:48 2012
From: report at bugs.python.org (Jon Brandvein)
Date: Wed, 25 Jan 2012 17:25:48 +0000
Subject: [issue13854] multiprocessing: SystemExit from child with non-int,
	non-str arg causes TypeError
In-Reply-To: <1327428526.26.0.964378708084.issue13854@psf.upfronthosting.co.za>
Message-ID: <1327512348.28.0.497913714186.issue13854@psf.upfronthosting.co.za>


Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Also, as Brett pointed out to me in #13853, bool is a subclass of int, so they should follow the same code path. I suggest replacing

    elif type(e.args[0]) is int:
        exitcode = e.args[0]

with something like

    elif isinstance(e.args[0], int):
        exitcode = e.args[0]

which assumes that a subtype of int is convertible to int.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13854>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:26:06 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Wed, 25 Jan 2012 17:26:06 +0000
Subject: [issue13843] Python doesn't compile anymore on our Solaris buildbot:
	undefined libintl_* symbols
In-Reply-To: <1327354335.3.0.0960870172041.issue13843@psf.upfronthosting.co.za>
Message-ID: <1327512366.45.0.924739787365.issue13843@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13843>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:37:35 2012
From: report at bugs.python.org (Vinay Sajip)
Date: Wed, 25 Jan 2012 17:37:35 +0000
Subject: [issue13859] Lingering StandardError in logging module
In-Reply-To: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>
Message-ID: <1327513055.55.0.425888312631.issue13859@psf.upfronthosting.co.za>


Vinay Sajip <vinay_sajip at yahoo.co.uk> added the comment:

> Interesting this also occurs in 3.2 and 2.7, but not 2.6 or 3.1

Most likely it happened because a fix was backported and this got missed. I'll take care of it.

----------
assignee:  -> vinay.sajip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:41:19 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 17:41:19 +0000
Subject: [issue13860] PyBuffer_FillInfo() return value
Message-ID: <1327513279.1.0.452976234811.issue13860@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

This came up in #10419, which is already a little crowded. PyBuffer_FillInfo() returns success if the value for 'view' is NULL.

I can't really see a reason for that. Any thoughts?

----------
components: Interpreter Core
messages: 151953
nosy: ncoghlan, pitrou, skrah
priority: normal
severity: normal
status: open
title: PyBuffer_FillInfo() return value
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13860>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:43:55 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 17:43:55 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327513435.19.0.611475891082.issue10181@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


Added file: http://bugs.python.org/file24323/8dd9f0ea4876.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:44:17 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 17:44:17 +0000
Subject: [issue13411] Hashable memoryviews
In-Reply-To: <1321395643.62.0.259606214561.issue13411@psf.upfronthosting.co.za>
Message-ID: <1327513457.46.0.0262531301684.issue13411@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Done in: http://hg.python.org/features/pep-3118/rev/508d5e3c579c

----------
dependencies:  -Problems with Py_buffer management in memoryobject.c (and elsewhere?)

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13411>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:44:39 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 17:44:39 +0000
Subject: [issue13411] Hashable memoryviews
In-Reply-To: <1321395643.62.0.259606214561.issue13411@psf.upfronthosting.co.za>
Message-ID: <1327513479.67.0.930087037593.issue13411@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
resolution: remind -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13411>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:50:26 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 25 Jan 2012 17:50:26 +0000
Subject: [issue13859] Lingering StandardError in logging module
In-Reply-To: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>
Message-ID: <E1Rq6zI-0007YX-Ap@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset e506848d6381 by Vinay Sajip in branch '3.2':
Closes #13859: Replaced reference to StandardError with reference to Exception. Thanks to Matt Joiner for spotting this and submitting a patch.
http://hg.python.org/cpython/rev/e506848d6381

New changeset fec45282dc28 by Vinay Sajip in branch 'default':
Closes #13859: Merged fix from 3.2 - thanks to Matt Joiner for spotting this and the patch.
http://hg.python.org/cpython/rev/fec45282dc28

----------
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:00:19 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 18:00:19 +0000
Subject: [issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous
	arrays
In-Reply-To: <1314211505.01.0.756597830367.issue12834@psf.upfronthosting.co.za>
Message-ID: <1327514419.3.0.0520049565666.issue12834@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

I removed the dependency since PyBuffer_ToContiguous() still needs to
be fixed in abstract.c while memoryview.tobytes() now works in the
PEP-3118 repo.

----------
dependencies:  -Problems with Py_buffer management in memoryobject.c (and elsewhere?)
title: memoryview.tobytes() incorrect for non-contiguous arrays -> PyBuffer_ToContiguous() incorrect for non-contiguous arrays

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12834>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:05:27 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 25 Jan 2012 18:05:27 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327513722.2388.20.camel@surprise>
Message-ID: <1327514598.3428.10.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I'm attaching a revised version of the patch that should fix the above
> issue:
>   hybrid-approach-dmalcolm-2012-01-25-002.patch

It looks like that approach will break any non-builtin type (in either C
or Python) which can compare equal to bytes or str objects. If that's
the case, then I think the likelihood of acceptance is close to zero.

Also, the level of complication is far higher than in any other of the
proposed approaches so far (I mean those with patches), which isn't
really a good thing.

So I'm rather -1 myself on this approach, and would much prefer to
randomize hashes in all conditions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:14:07 2012
From: report at bugs.python.org (Jim Jewett)
Date: Wed, 25 Jan 2012 18:14:07 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327489561.69.0.280670857261.issue13703@psf.upfronthosting.co.za>
Message-ID: <CA+OGgf5wR7YyvcUA3F1GjVGxw6oiaiw3Q=WTKcCKu-DxxBFhEg@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Wed, Jan 25, 2012 at 6:06 AM, Dave Malcolm <dmalcolm at redhat.com>
added the comment:

> ?hybrid-approach-dmalcolm-2012-01-25-001.patch

> As per haypo's random-8.patch, a randomization seed is read at startup.

Why not wait until it is needed?  I suspect a lot of scripts will
never need it for any dict, so why add the overhead to startup?

> Once a dict has transitioned to paranoid mode, it isn't using
> PyObject_Hash anymore, and thus isn't using cached object values

The alternative hashes could be stored in an id-keyed dict

 performing a more expensive calculation, but I believe this
calculation is essentially constant-time.
>
> This preserves hash() and dict order for the cases where you're not under attack, and gracefully handles the attack without having to raise an exception: it doesn't introduce any new exception types.
>
> It preserves ABI, assuming no-one else is reusing ma_smalltable.
>
> It is suitable for backporting to 3.2, 2.7, and earlier (I'm investigating fixing this going all the way back to Python 2.2)
>
> Under the old implementation, there were 4 types of PyDictObject, given these two booleans:
> ?* "small vs large" i.e ma_table == ma_smalltable vs ma_table != ma_smalltable
> ?* "all keys are str" vs arbitary keys i.e ma_lookdict == lookdict_unicode vs lookdict
>
> Under this implementation, this doubles to 8 kinds, adding the boolean:
> ?* normal hash vs randomized hash (normal vs "paranoid").
>
> This is expressed via the ma_lookdict callback, adding two new variants, lookdict_unicode_paranoid and lookdict_paranoid
>
> Note that if a paranoid dict goes small again (ma_table == ma_smalltable), it stays paranoid. ?This is for simplicity: it avoids having to rebuild all of the non-randomized me_hash values again (which could fail).
>
> Naturally the patch adds selftests. ?I had to add some diagnostic methods to support them; dict gains _stats() and _make_paranoid() methods, and sys gains a _getrandomizedhash() method. ?These could be hidden more thoroughly if need be (see DICT_PROTECTION_TRACKING in dictobject.c). ?Amongst other things, the selftests measure wallclock time taken for various dict operations (and so might introduce failures on a heavily-loaded machine, I guess).
>
> Hopefully this approach is a viable way forward.
>
> Caveats and TODO items:
>
> TODO: I haven't yet tuned the safety threshold. ?According to http://bugs.python.org/issue13703#msg151850:
>> slot collisions are much more frequent than
>> hash collisions, which only account for less than 0.01% of all
>> collisions.
>>
>> It also shows that slot collisions in the low 1-10 range are
>> most frequent, with very few instances of a dict lookup
>> reaching 20 slot collisions (less than 0.0002% of all
>> collisions).
>
> This suggests that the threshold of 32 slot/hash collisions per lookup may already be high enough.
>
> TODO: in a review of an earlier version of the complexity detection idea, Antoine Pitrou suggested that make the protection scale factor be a run-time configurable value, rather than a #define. ?This isn't done yet.
>
> TODO: run more extensive tests (e.g. Django and Twisted), monitoring the worst-case complexity that's encountered
>
> TODO: not yet benchmarked and optimized. ?I want to get feedback on the approach before I go in and hand-optimize things (e.g. by hand-inlining check_iter_count, and moving the calculations out of the loop etc). ?I believe any performance issues ought to be fixable, in that the we can get the cost of this for the "we're not under attack" case to be negligible, and the "under attack" case should transition from O(N^2) to O(N), albeit it with a larger constant factor.
>
> TODO: this doesn't cover sets, but assuming this approach works, the patch can be extended to cover it in an analogous way.
>
> TODO: should it cover PyMemoryViewObject, buffer object, etc?
>
> TODO: should it cover the hashing in Modules/expat/xmlparse.c? ?FWIW I rip this code out when doing my downstream builds in RHEL and Fedora, and instead dynamically link against a system copy of expat
>
> TODO: only tested on Linux so far (which is all I've got). ?Fedora 15 x86_64 fwiw
>
> ?Doc/using/cmdline.rst ? ? | ? ?6
> ?Include/bytesobject.h ? ? | ? ?2
> ?Include/object.h ? ? ? ? ?| ? ?8
> ?Include/pythonrun.h ? ? ? | ? ?2
> ?Include/unicodeobject.h ? | ? ?2
> ?Lib/os.py ? ? ? ? ? ? ? ? | ? 17 --
> ?Lib/test/regrtest.py ? ? ?| ? ?5
> ?Lib/test/test_dict.py ? ? | ?298 +++++++++++++++++++++++++++++++++++++
> ?Lib/test/test_hash.py ? ? | ? 53 ++++++
> ?Lib/test/test_os.py ? ? ? | ? 35 +++-
> ?Makefile.pre.in ? ? ? ? ? | ? ?1
> ?Modules/posixmodule.c ? ? | ?126 ++-------------
> ?Objects/bytesobject.c ? ? | ? ?7
> ?Objects/dictobject.c ? ? ?| ?369 +++++++++++++++++++++++++++++++++++++++++++++-
> ?Objects/object.c ? ? ? ? ?| ? 37 ++++
> ?Objects/unicodeobject.c ? | ? 51 ++++++
> ?PCbuild/pythoncore.vcproj | ? ?4
> ?Python/pythonrun.c ? ? ? ?| ? ?3
> ?Python/sysmodule.c ? ? ? ?| ? 16 +
> ?b/Python/random.c ? ? ? ? | ?268 +++++++++++++++++++++++++++++++++
> ?20 files changed, 1173 insertions(+), 137 deletions(-)
>
> ----------
> Added file: http://bugs.python.org/file24320/hybrid-approach-dmalcolm-2012-01-25-001.patch
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:29:42 2012
From: report at bugs.python.org (Jim Jewett)
Date: Wed, 25 Jan 2012 18:29:42 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf5wR7YyvcUA3F1GjVGxw6oiaiw3Q=WTKcCKu-DxxBFhEg@mail.gmail.com>
Message-ID: <CA+OGgf7AbEiVP-XuULVP21bp0w2JvXSdC0AsgLKRcyVzdfW+jQ@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Sorry; hit the wrong key... intended message below:

On Wed, Jan 25, 2012 at 6:06 AM, Dave Malcolm <dmalcolm at redhat.com>
added the comment:

[lots of good stuff]

> ?hybrid-approach-dmalcolm-2012-01-25-001.patch

> As per haypo's random-8.patch, a randomization seed is read at
> startup.

Why not wait until it is needed? ?I suspect a lot of scripts will
never need it for any dict, so why add the overhead to startup?

> Once a dict has transitioned to paranoid mode, it isn't using
> PyObject_Hash anymore, and thus isn't using cached object values

The alternative hashes could be stored in an id-keyed
WeakKeyDictionary; that would handle at least the normal case of using
exactly the same string for the lookup.

> Note that if a paranoid dict goes small again
> (ma_table == ma_smalltable), it stays paranoid.

As I read it, that couldn't happen, because paranoid dicts couldn't
shrink at all.  (Not letting them shrink beneath 2*PyDict_MINSIZE does
seem like a reasonable solution.)

Additional TODOs...

The checks for Unicode and Dict should not be exact; it is OK to do on
a subclass so long as they are using the same lookdict (and, for
unicode, the same eq).

Additional small strings should be excluded from the new hash, to
avoid giving away the secret.  At a minimum, single-char strings
should be excluded, and I would prefer to exclude all strings of
length <= N (where N defaults to 4).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:43:37 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 18:43:37 +0000
Subject: [issue13861] test_pydoc failure
Message-ID: <1327517017.59.0.678002352905.issue13861@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

test_pydoc fails on Ubuntu Lucid:

======================================================================
FAIL: test_apropos_with_bad_package (test.test_pydoc.PydocImportTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefan/pydev/cpython/Lib/test/test_pydoc.py", line 409, in test_apropos_with_bad_package
    self.assertEqual(b'', result)
AssertionError: b'' != b'Crypto.Protocol.AllOrNothing - This file implements all-or-nothing package transformations.\nCrypto.SelfTest.Protocol.test_AllOrNothing'

======================================================================
FAIL: test_apropos_with_unreadable_dir (test.test_pydoc.PydocImportTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefan/pydev/cpython/Lib/test/test_pydoc.py", line 419, in test_apropos_with_unreadable_dir
    self.assertEqual(b'', result)
AssertionError: b'' != b'Crypto.Protocol.AllOrNothing - This file implements all-or-nothing package transformations.\nCrypto.SelfTest.Protocol.test_AllOrNothing'

----------------------------------------------------------------------

----------
components: Tests
messages: 151962
nosy: skrah
priority: normal
severity: normal
stage: needs patch
status: open
title: test_pydoc failure
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13861>
_______________________________________

From report at bugs.python.org  Wed Jan 25 19:46:38 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 18:46:38 +0000
Subject: [issue13862] test_zlib failure
Message-ID: <1327517198.75.0.31759393536.issue13862@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

test_zlib currently fails on Ubuntu Lucid:

======================================================================
FAIL: test_library_version (test.test_zlib.VersionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefan/pydev/cpython/Lib/test/test_zlib.py", line 24, in test_library_version
    self.assertEqual(zlib.ZLIB_RUNTIME_VERSION, zlib.ZLIB_VERSION)
AssertionError: '1.2.3.3' != '1.2.5'
- 1.2.3.3
+ 1.2.5

----------
components: Tests
messages: 151963
nosy: nadeem.vawda, skrah
priority: normal
severity: normal
stage: needs patch
status: open
title: test_zlib failure
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13862>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:02:37 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 25 Jan 2012 19:02:37 +0000
Subject: [issue13862] test_zlib failure
In-Reply-To: <1327517198.75.0.31759393536.issue13862@psf.upfronthosting.co.za>
Message-ID: <1327518157.49.0.0175501886868.issue13862@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Do you have a self-compiled version of zlib (1.2.5) installed on this
system? It looks like this is due to a (benign) version mismatch between
zlib.h and the actual shared lib.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13862>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:04:24 2012
From: report at bugs.python.org (Jim Jewett)
Date: Wed, 25 Jan 2012 19:04:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327514598.3428.10.camel@localhost.localdomain>
Message-ID: <CA+OGgf5U+yo2X+oE9QEbyoqf7s5hczdsPmY0bgSGrco_TfyeYQ@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Wed, Jan 25, 2012 at 1:05 PM,  Antoine Pitrou <pitrou at free.fr>
added the comment:

> It looks like that approach will break any non-builtin type (in either C
> or Python) which can compare equal to bytes or str objects. If that's
> the case, then I think the likelihood of acceptance is close to zero.

(1)  Isn't that true of *any* patch that changes hashing?  (Thus the
PYTHONHASHSEED=0 escape hatch.)

(2)  I think it would still work for the lookdict_string (or
lookdict_unicode) case ... which is the normal case, and also where
most vulnerabilities should appear.

(3)  If the alternate hash is needed for non-string keys, there is no
perfect resolution, but I suppose you could get closer with

    if obj == str(obj):
        newhash=hash(str(obj))

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:13:09 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 25 Jan 2012 19:13:09 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf5U+yo2X+oE9QEbyoqf7s5hczdsPmY0bgSGrco_TfyeYQ@mail.gmail.com>
Message-ID: <1327518658.3428.21.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Jim Jewett <jimjjewett at gmail.com> added the comment:
> 
> On Wed, Jan 25, 2012 at 1:05 PM,  Antoine Pitrou <pitrou at free.fr>
> added the comment:
> 
> > It looks like that approach will break any non-builtin type (in either C
> > or Python) which can compare equal to bytes or str objects. If that's
> > the case, then I think the likelihood of acceptance is close to zero.
> 
> (1)  Isn't that true of *any* patch that changes hashing?  (Thus the
> PYTHONHASHSEED=0 escape hatch.)

If a third-party type wants to compare equal to bytes or str objects,
its __hash__ method will usually end up calling hash() on the equivalent
bytes/str object. That's especially true for Python types (I don't think
anyone wants to re-implement a slow str-alike hash in pure Python).

> (2)  I think it would still work for the lookdict_string (or
> lookdict_unicode) case ... which is the normal case, and also where
> most vulnerabilities should appear.

It would probably still work indeed.

> (3)  If the alternate hash is needed for non-string keys, there is no
> perfect resolution, but I suppose you could get closer with
> 
>     if obj == str(obj):
>         newhash=hash(str(obj))

That may be slowing down things quite a bit. It looks correct though.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:19:32 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 25 Jan 2012 19:19:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327514598.3428.10.camel@localhost.localdomain>
Message-ID: <1327519137.2388.28.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Wed, 2012-01-25 at 18:05 +0000, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou at free.fr> added the comment:
> 
> > I'm attaching a revised version of the patch that should fix the above
> > issue:
> >   hybrid-approach-dmalcolm-2012-01-25-002.patch
> 
> It looks like that approach will break any non-builtin type (in either C
> or Python) which can compare equal to bytes or str objects. If that's
> the case, then I think the likelihood of acceptance is close to zero.

How?

> Also, the level of complication is far higher than in any other of the
> proposed approaches so far (I mean those with patches), which isn't
> really a good thing.

So would I.  I want something I can backport, though.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:24:11 2012
From: report at bugs.python.org (Derek Wilson)
Date: Wed, 25 Jan 2012 19:24:11 +0000
Subject: [issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
In-Reply-To: <1242668944.28.0.292370665479.issue6056@psf.upfronthosting.co.za>
Message-ID: <1327519451.11.0.635247676558.issue6056@psf.upfronthosting.co.za>


Derek Wilson <jderekwilson at gmail.com> added the comment:

Any chance this patch will be accepted (or at least reviewed) soon?

----------
type: behavior -> crash
versions: +Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6056>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:28:01 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Wed, 25 Jan 2012 19:28:01 +0000
Subject: [issue13863] Using py_compile does not prevent recompilation due to
	'bad mtime'.
Message-ID: <1327519681.79.0.252690630069.issue13863@psf.upfronthosting.co.za>


New submission from Mark Dickinson <dickinsm at gmail.com>:

(Marking this as 'Interpreter Core' because the issue really seems to come from Python/import.c rather than from the py_compile module.)

On a Windows 7 VM (64-bit, NTFS) running Python 2.7 (also reproduced on a non-VM Windows 7 installation), I'm seeing the following surprising behaviour:


C:\Python27\Lib>dir decimal.py*
 Volume in drive C is Local
 Volume Serial Number is 5083-D43D

 Directory of C:\Python27\Lib

06/10/2011  08:46 PM           219,383 decimal.py
01/25/2012  07:05 PM           165,322 decimal.pyc
01/25/2012  04:54 PM           169,386 decimal.pyo
               3 File(s)        554,091 bytes
               0 Dir(s)     966,266,880 bytes free

C:\Python27\Lib>python
Enthought Python Distribution -- www.enthought.com
Version: 7.2-2 (64-bit)

Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "packages", "demo" or "enthought" for more information.
>>> import py_compile; py_compile.compile('decimal.py')
>>> import os; os.stat('decimal.pyc').st_mtime
1327518430.306176
>>> import decimal
>>> os.stat('decimal.pyc').st_mtime
1327518443.9094632

Notice that in spite of calling py_compile.compile on decimal.py, the .pyc file is still regenerated on import.
Relevant details: note that the decimal.py timestamp is in the summer, and that it's currently winter time.  Also, my Windows 7 timezone setting is UTC, with 'automatically adjust for DST' set to true.

This bit me today when working with a 'post-install' script for an MSI installer for a large Python application, where the post-install script did a compileall.compile_path to make sure that all the .pyc files were up to date, and thereby avoid a *long* (> 90 second) application first startup time.  It turned out that on application startup some of the .pyc files got regenerated while others didn't, and it took me far too long to notice that it was the .py files with summer timestamps that lead to .pyc regeneration, and that those with winter timestamps were okay.

When I set my timezone to plain UTC with no DST adjustments, the above issue disappears.  Also, when I reset my timezone to that of Saudi Arabia (no DST), the issue also disappears.


The cause of the issue seems to be that:

(1) import.c uses an 'fstat' call to get mtime for a .py file.
(2) on my Windows installation, the result of fstat on a file varies (a) with time of year, and (b) with computer DST and timezone settings.  (No such problems on OS X.)
(3) in contrast, py_compile uses os.stat(...).st_mtime, which appears to be invariant under time changes.


The difference between the 'fstat' result and the os.stat result can be seen directly:

Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "packages", "demo" or "enthought" for more information.
>>> import decimal  # make sure .pyc file is up to date
>>> import os; os.stat('decimal.py').st_mtime  # actual mtime of .py file
1307738784.0
>>> import struct; struct.unpack('LL', open('decimal.pyc', 'rb').read(8))[1]  # stored mtime from .pyc; uses fstat
1307735184



This presumably also means that in this part of the world, .pyc files will be regenerated on import after any DST change, since the reported 'fstat' result used by import.c will no longer match the stored mtime timestamp in the .pyc file.

----------
components: Interpreter Core
messages: 151969
nosy: mark.dickinson
priority: normal
severity: normal
status: open
title: Using py_compile does not prevent recompilation due to 'bad mtime'.
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:28:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 25 Jan 2012 19:28:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327519137.2388.28.camel@surprise>
Message-ID: <1327519559.3428.33.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

Le mercredi 25 janvier 2012 ? 19:19 +0000, Dave Malcolm a ?crit :
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
> 
> On Wed, 2012-01-25 at 18:05 +0000, Antoine Pitrou wrote:
> > Antoine Pitrou <pitrou at free.fr> added the comment:
> > 
> > > I'm attaching a revised version of the patch that should fix the above
> > > issue:
> > >   hybrid-approach-dmalcolm-2012-01-25-002.patch
> > 
> > It looks like that approach will break any non-builtin type (in either C
> > or Python) which can compare equal to bytes or str objects. If that's
> > the case, then I think the likelihood of acceptance is close to zero.
> 
> How?

This kind of type, for example:

class C:
    def __hash__(self):
        return hash(self._real_str)

    def __eq__(self, other):
        if isinstance(other, C):
           other = other._real_str
        return self._real_str == other

If I'm not mistaken, looking up C("abc") will stop matching "abc" when
there are too many collisions in one of your dicts.

> > Also, the level of complication is far higher than in any other of the
> > proposed approaches so far (I mean those with patches), which isn't
> > really a good thing.
> 
> So would I.  I want something I can backport, though.

Well, your approach sounds like it subtly and unpredictably changes the
behaviour of dicts when there are too many collisions, so I'm not sure
it's a good idea to backport it, either.

If we don't want to backport full hash randomization, I think I much
prefer raising a BaseException when there are too many collisions,
rather than this kind of (excessively) sophisticated workaround. You
*are* changing a fundamental datatype in a rather complicated way.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 20:52:13 2012
From: report at bugs.python.org (stephen Andel)
Date: Wed, 25 Jan 2012 19:52:13 +0000
Subject: [issue13864] Python 2.7.2 refuses to open
Message-ID: <1327521133.43.0.177079679673.issue13864@psf.upfronthosting.co.za>


New submission from stephen Andel <elden.andel at gmail.com>:

I recently changed one of the keybindings in python to just control. Python did not like this and, when I tried to fix this by swapping back to the default settings it closed itself and now will not open. Th program will attempt to open then stop, and the process with cancel. I have reinstalled and deleted all apparent files associated with python but this was not enough and continues to fail, badly. Sorry for the informal dialogue I'm just a bit peeved right now.

----------
components: IDLE
messages: 151971
nosy: stephen.Andel
priority: normal
severity: normal
status: open
title: Python 2.7.2 refuses to open
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13864>
_______________________________________

From report at bugs.python.org  Wed Jan 25 21:07:58 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Wed, 25 Jan 2012 20:07:58 +0000
Subject: [issue13863] Using py_compile does not prevent recompilation due to
	'bad mtime'.
In-Reply-To: <1327519681.79.0.252690630069.issue13863@psf.upfronthosting.co.za>
Message-ID: <1327522078.27.0.989850159661.issue13863@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Issue reproduced on stock Python 2.7 and Python 3.2 from www.python.org.  I don't have a working 3.3 install on Windows, but looking at the source, it seems likely that it's an issue there, too.

----------
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________

From report at bugs.python.org  Wed Jan 25 21:23:40 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 25 Jan 2012 20:23:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327519559.3428.33.camel@localhost.localdomain>
Message-ID: <1327522985.2388.57.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I think you're right: it will stop matching it during lookup within such
a dict, since the dict will be using the secondary hash for "abc", but
hash() for the C instance.

It will still match outside of the dict, and within other dicts.

So yes, this would be a subtle semantic change when under attack.
Bother.

Having said that, note that within the typical attack scenarios (HTTP
headers, HTTP POST, XML-RPC, JSON), we have a pure-str dict (or
sometimes a pure-bytes dict).  Potentially I could come up with a patch
that only performs this change for such a case (pure-str is easier,
given that we already track this), which would avoid the semantic change
you identify, whilst still providing protection against a wide range of
attacks.

Is it worth me working on this?

> > > Also, the level of complication is far higher than in any other of the
> > > proposed approaches so far (I mean those with patches), which isn't
> > > really a good thing.
> > 
> > So would I.  I want something I can backport, though.
> 
> Well, your approach sounds like it subtly and unpredictably changes the
> behaviour of dicts when there are too many collisions, so I'm not sure
> it's a good idea to backport it, either.
> 
> If we don't want to backport full hash randomization, I think I much
> prefer raising a BaseException when there are too many collisions,
> rather than this kind of (excessively) sophisticated workaround. You
> *are* changing a fundamental datatype in a rather complicated way.

Well, each approach has pros and cons, and we've circled around between
hash randomization vs collision counting vs other approaches for several
weeks.  I've supplied patches for 3 different approaches.

Is this discussion likely to reach a conclusion soon?  Would it be
regarded as rude if I unilaterally ship something close to:
  backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch
in RHEL/Fedora, so that my users have some protection they can enable if
they get attacked? (see http://bugs.python.org/msg151847).  If I do
this, I can post the patches here in case other distributors want to
apply them.

As for python.org, who is empowered to make a decision here?  How can we
move this forward?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:04:53 2012
From: report at bugs.python.org (Miki Tebeka)
Date: Wed, 25 Jan 2012 21:04:53 +0000
Subject: [issue13865] distutils documentation says Extension has "optional"
	argument
Message-ID: <1327525493.46.0.373823292504.issue13865@psf.upfronthosting.co.za>


New submission from Miki Tebeka <miki.tebeka at gmail.com>:

The Extension documentation says:

--------
2.3.5. Other options
There are still some other options which can be used to handle special cases.

The optional option is a boolean; if it is true, a build failure in the extension will not abort the build process, but instead simply not install the failing extension.
---------

However there not such option, and distutils will complain:

/usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'optional'

----------
messages: 151974
nosy: tebeka
priority: normal
severity: normal
status: open
title: distutils documentation says Extension has "optional" argument

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13865>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:05:12 2012
From: report at bugs.python.org (Miki Tebeka)
Date: Wed, 25 Jan 2012 21:05:12 +0000
Subject: [issue13865] distutils documentation says Extension has "optional"
	argument
In-Reply-To: <1327525493.46.0.373823292504.issue13865@psf.upfronthosting.co.za>
Message-ID: <1327525512.8.0.477621209893.issue13865@psf.upfronthosting.co.za>


Changes by Miki Tebeka <miki.tebeka at gmail.com>:


----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python
versions: +Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13865>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:25:25 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 25 Jan 2012 21:25:25 +0000
Subject: [issue13862] test_zlib failure
In-Reply-To: <1327517198.75.0.31759393536.issue13862@psf.upfronthosting.co.za>
Message-ID: <E1RqALM-0004rb-GD@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset a5f8611ce81d by Nadeem Vawda in branch 'default':
Issue #13862: Relax zlib version test to avoid spurious failures.
http://hg.python.org/cpython/rev/a5f8611ce81d

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13862>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:26:43 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Wed, 25 Jan 2012 21:26:43 +0000
Subject: [issue13862] test_zlib failure
In-Reply-To: <1327517198.75.0.31759393536.issue13862@psf.upfronthosting.co.za>
Message-ID: <1327526803.61.0.626033555736.issue13862@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Either way, the failure should be fixed now.

----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13862>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:34:39 2012
From: report at bugs.python.org (Frank Sievertsen)
Date: Wed, 25 Jan 2012 21:34:39 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327527279.92.0.607241526075.issue13703@psf.upfronthosting.co.za>


Frank Sievertsen <python at sievertsen.de> added the comment:

For the sake of completeness:
Collision-counting (with Exception) has interesting effects, too.

>>> d={((1<<(65+i))-2**(i+4)): 9 for i in range(1001)}
>>> for i in list(d): 
...  del d[i]

>>> d
{}
>>> 9 in d
False
>>> 0 in d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'too many slot collisions'
>>> d[9] = 1
>>> d
{9: 1}
>>> d == {0: 1}
False
>>> {0: 1} == d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'too many slot collisions'

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:36:54 2012
From: report at bugs.python.org (Roundup Robot)
Date: Wed, 25 Jan 2012 21:36:54 +0000
Subject: [issue13852] Doc fixes with patch
In-Reply-To: <1327410885.19.0.883089570823.issue13852@psf.upfronthosting.co.za>
Message-ID: <E1RqAWS-00063O-Jb@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset e2b98a332070 by Georg Brandl in branch '2.7':
#13852: some small doc fixes.
http://hg.python.org/cpython/rev/e2b98a332070

New changeset 5b8800012e88 by Georg Brandl in branch '3.2':
#13852: some small doc fixes.
http://hg.python.org/cpython/rev/5b8800012e88

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13852>
_______________________________________

From report at bugs.python.org  Wed Jan 25 22:37:07 2012
From: report at bugs.python.org (Georg Brandl)
Date: Wed, 25 Jan 2012 21:37:07 +0000
Subject: [issue13852] Doc fixes with patch
In-Reply-To: <1327410885.19.0.883089570823.issue13852@psf.upfronthosting.co.za>
Message-ID: <1327527427.97.0.474186960246.issue13852@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Done. Thanks!

----------
nosy: +georg.brandl
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13852>
_______________________________________

From report at bugs.python.org  Wed Jan 25 23:12:02 2012
From: report at bugs.python.org (Stephen Day)
Date: Wed, 25 Jan 2012 22:12:02 +0000
Subject: [issue13866] {urllib,
	urllib.parse}.urlencode should not use quote_plus
Message-ID: <1327529522.13.0.464287316871.issue13866@psf.upfronthosting.co.za>


New submission from Stephen Day <stevvooe at gmail.com>:

The current behavior of the urlencode function (2.7: urllib, 3.x: urllib.parse) encodes spaces as pluses:

>>> from urllib import urlencode
>>> urlencode({'a': 'some param'})
'a=some+param'

However, in most instances, it would be desirable to merely encode spaces using percent encoding:

>>> urlencode({'a': 'some param'})
'a=some%20param'

But there is no way to get this behavior in the standard library. 

It would probably best to change this so it defaults to use the regular quote function, but allows callers who need the legacy quote_plus behavior to pass that in as a function parameter.

An acceptable fix would be to have the quote function taken as a keyword parameter, so legacy behavior remains:

>>> urlencode({'a': 'some param'})
'a=some+param'

Then the behavior could be adjusted where needed:

>>> from urllib import quote
>>> urlencode({'a': 'some param'}, quote=quote)
'a=some%20param'

----------
components: Library (Lib)
messages: 151980
nosy: Stephen.Day
priority: normal
severity: normal
status: open
title: {urllib,urllib.parse}.urlencode should not use quote_plus
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13866>
_______________________________________

From report at bugs.python.org  Wed Jan 25 23:41:32 2012
From: report at bugs.python.org (Stefan Krah)
Date: Wed, 25 Jan 2012 22:41:32 +0000
Subject: [issue13862] test_zlib failure
In-Reply-To: <1327517198.75.0.31759393536.issue13862@psf.upfronthosting.co.za>
Message-ID: <1327531292.52.0.677920938743.issue13862@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Thanks! There was a 32-bit 1.2.5 version in /usr/local/lib and
a 64-bit 1.2.3.3 version in /lib.

gcc picks up the header from /usr/local/include, while ld is smart
enough to choose the 64-bit version. So it was a multilib issue.


While this is not the most robust setup, it's very convenient on
a dev machine, so I'm glad that this is fixed.

----------
status: pending -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13862>
_______________________________________

From report at bugs.python.org  Wed Jan 25 23:55:28 2012
From: report at bugs.python.org (Jim Jewett)
Date: Wed, 25 Jan 2012 22:55:28 +0000
Subject: [issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
In-Reply-To: <1242668944.28.0.292370665479.issue6056@psf.upfronthosting.co.za>
Message-ID: <1327532128.61.0.520747079214.issue6056@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

The wording in 138415 suggested this patch was changing socket to not support timeouts -- which would be unacceptable.  

But the actual patch only seems to touch multiprocessing/connection.py -- a far more reasonable change.

Unfortunately, the patch no longer applies to the development tip.  I *think* the places you wanted to change are still there, and just moved.

(1)  Is it sufficiently clear that this is not-a-feature to justify a backport?

(2)  Are the problems already fixed by some of the other changes?  (It doesn't look like it, but I'm not sure.)

(3)  Can you produce an updated patch?  (The current tip is http://hg.python.org/cpython/file/fec45282dc28/Lib/multiprocessing/connection.py  )

(4)  If I understand the intent, then s.setblocking(True) would be slightly more clear than s.settimeout(None), though that change obviously isn't essential.

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6056>
_______________________________________

From report at bugs.python.org  Thu Jan 26 00:01:16 2012
From: report at bugs.python.org (Jim Jewett)
Date: Wed, 25 Jan 2012 23:01:16 +0000
Subject: [issue13867] misleading comment in weakrefobject.h
Message-ID: <1327532476.25.0.872465463087.issue13867@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

http://hg.python.org/cpython/file/fec45282dc28/Include/weakrefobject.h#l54

The comment makes sense -- but doesn't appear to be true, so perhaps it is the macro that should change.


 
/* This macro calls PyWeakref_CheckRef() last since that can involve a
   function call; this makes it more likely that the function call
   will be avoided. */
#define PyWeakref_Check(op) \
        (PyWeakref_CheckRef(op) || PyWeakref_CheckProxy(op))

----------
assignee: docs at python
components: Documentation, Extension Modules
messages: 151983
nosy: Jim.Jewett, docs at python
priority: normal
severity: normal
status: open
title: misleading comment in weakrefobject.h

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13867>
_______________________________________

From report at bugs.python.org  Thu Jan 26 00:14:04 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Wed, 25 Jan 2012 23:14:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327522985.2388.57.camel@surprise>
Message-ID: <1327533114.3428.62.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I think you're right: it will stop matching it during lookup within such
> a dict, since the dict will be using the secondary hash for "abc", but
> hash() for the C instance.
> 
> It will still match outside of the dict, and within other dicts.
> 
> So yes, this would be a subtle semantic change when under attack.
> Bother.

Hmm, you're right, perhaps it's not as important as I thought.

By the way, have you run benchmarks on some of your patches?

> Is this discussion likely to reach a conclusion soon?  Would it be
> regarded as rude if I unilaterally ship something close to:
>   backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch
> in RHEL/Fedora, so that my users have some protection they can enable if
> they get attacked?

I don't think Fedora shipping its own patches can be considered "rude"
by anyone else than its users. And deciding what is best for your users
is indeed your job as a distro maintainer, not python-dev's.

> As for python.org, who is empowered to make a decision here?  How can we
> move this forward?

I don't know. Guido is empowered if he wants to make a pronouncement.
Otherwise, we have the following data points:

- hash randomization is generally considered the cleanest solution
- hash randomization cannot be enabled by default in bugfix, let alone
security releases
- collision counting can mitigate some of the attacks, although it can
have weaknesses (see Frank's emails) and it comes with its own problems
(breaking the program "later on")

So I'd suggest the following course of action:
- ship and enable some form of collision counting on bugfix and security
releases
- ship and enable hash randomization in 3.3

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 00:58:11 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Wed, 25 Jan 2012 23:58:11 +0000
Subject: [issue13868] Add hyphen doc fix
Message-ID: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>


New submission from Bo?tjan Mejak <bostjan.mejak at gmail.com>:

When you have time, incorporate this patch. Thanks.

----------
assignee: docs at python
components: Documentation
files: smallfix.diff
keywords: patch
messages: 151985
nosy: Retro, docs at python
priority: normal
severity: normal
status: open
title: Add hyphen doc fix
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24325/smallfix.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Thu Jan 26 01:11:57 2012
From: report at bugs.python.org (Stefan Krah)
Date: Thu, 26 Jan 2012 00:11:57 +0000
Subject: [issue13869] CFLAGS="-UNDEBUG" build failure
Message-ID: <1327536717.01.0.741263593585.issue13869@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

Passing -UNDEBUG as CFLAGS currently doesn't work:

./configure CFLAGS="-UNDEBUG" && make

[...]

libpython3.3m.a(object.o): In function `PyObject_Str':
/home/stefan/pydev/cpython/Objects/object.c:441: undefined reference to `_PyUnicode_CheckConsistency'
libpython3.3m.a(unicodeobject.o): In function `unicode_result_ready':
/home/stefan/pydev/cpython/Objects/unicodeobject.c:476: undefined reference to `_PyUnicode_CheckConsistency'
/home/stefan/pydev/cpython/Objects/unicodeobject.c:468: undefined reference to `_PyUnicode_CheckConsistency'
libpython3.3m.a(unicodeobject.o): In function `PyUnicode_New':
/home/stefan/pydev/cpython/Objects/unicodeobject.c:1066: undefined reference to `_PyUnicode_CheckConsistency'
libpython3.3m.a(unicodeobject.o): In function `get_latin1_char':
/home/stefan/pydev/cpython/Objects/unicodeobject.c:1647: undefined reference to `_PyUnicode_CheckConsistency'
libpython3.3m.a(unicodeobject.o):/home/stefan/pydev/cpython/Objects/unicodeobject.c:1787: more undefined references to `_PyUnicode_CheckConsistency' follow

----------
components: Build
messages: 151986
nosy: skrah
priority: normal
severity: normal
stage: needs patch
status: open
title: CFLAGS="-UNDEBUG" build failure
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13869>
_______________________________________

From report at bugs.python.org  Thu Jan 26 02:05:59 2012
From: report at bugs.python.org (Larry Hastings)
Date: Thu, 26 Jan 2012 01:05:59 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327539959.66.0.0825523451548.issue11457@psf.upfronthosting.co.za>


Larry Hastings <larry at hastings.org> added the comment:

Victor: I think your patch merits its own tracker issue; it's only tangentially related to the proposed changes to os.stat.

That said, please do add me to the nosy list if you create one.

One more thing: I haven't given it a lot of thought, so there might be an even better API out there.  But given your proposed API, wouldn't it be slightly better if it took the type object rather than the string?  time.time(float) or time.time(Decimal) as examples.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Thu Jan 26 03:36:57 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 26 Jan 2012 02:36:57 +0000
Subject: [issue13864] Python 2.7.2 refuses to open
In-Reply-To: <1327521133.43.0.177079679673.issue13864@psf.upfronthosting.co.za>
Message-ID: <1327545417.78.0.74205878788.issue13864@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13864>
_______________________________________

From report at bugs.python.org  Thu Jan 26 03:55:53 2012
From: report at bugs.python.org (Arnaud Fontaine)
Date: Thu, 26 Jan 2012 02:55:53 +0000
Subject: [issue12776] argparse: type conversion function should be called only
	once
In-Reply-To: <1313657878.1.0.958946060112.issue12776@psf.upfronthosting.co.za>
Message-ID: <1327546553.26.0.357056948234.issue12776@psf.upfronthosting.co.za>


Arnaud Fontaine <arnau at debian.org> added the comment:

ping?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12776>
_______________________________________

From report at bugs.python.org  Thu Jan 26 04:42:18 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 26 Jan 2012 03:42:18 +0000
Subject: [issue10042] total_ordering
In-Reply-To: <1286440012.74.0.624864175042.issue10042@psf.upfronthosting.co.za>
Message-ID: <1327549338.33.0.506951268974.issue10042@psf.upfronthosting.co.za>


Jim Jewett <jimjjewett at gmail.com> added the comment:

I like Nick Coghlan's suggestion in msg140493, but I think he was giving up too soon in the "or" cases, and I think the confusion could be slightly reduced by some re-spellings around return values and comments about short-circuiting.
   
def not_op(op, other):
    # "not a < b" handles "a >= b"
    # "not a <= b" handles "a > b"
    # "not a >= b" handles "a < b"
    # "not a > b" handles "a <= b"
    op_result = op(other)
    if op_result is NotImplemented:
        return NotImplemented
    return not op_result

def op_or_eq(op, self, other):
    # "a < b or a == b" handles "a <= b"
    # "a > b or a == b" handles "a >= b"
    op_result = op(other)
    if op_result is NotImplemented
        return self.__eq__(other) or NotImplemented
    if op_result:
        return True
    return self.__eq__(other)

def not_op_and_not_eq(op, self, other):
    # "not (a < b or a == b)" handles "a > b"
    # "not a < b and a != b" is equivalent
    # "not (a > b or a == b)" handles "a < b"
    # "not a > b and a != b" is equivalent
    op_result = op(other)
    if op_result is NotImplemented:
        return NotImplemented
    if op_result:
        return False
    return self.__ne__(other)

def not_op_or_eq(op, self, other):
    # "not a <= b or a == b" handles "a >= b"
    # "not a >= b or a == b" handles "a <= b"
    op_result = op(other)
    if op_result is NotImplemented:
        return self.__eq__(other) or NotImplemented
    if op_result:
        return self.__eq__(other)
    return True

def op_and_not_eq(op, self, other):
    # "a <= b and not a == b" handles "a < b"
    # "a >= b and not a == b" handles "a > b"
    op_result = op(other)
    if op_result is NotImplemented:
        return NotImplemented
    if op_result:
        return self.__ne__(other)
    return False

----------
nosy: +Jim.Jewett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10042>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:10:44 2012
From: report at bugs.python.org (Matt Joiner)
Date: Thu, 26 Jan 2012 04:10:44 +0000
Subject: [issue13859] Lingering StandardError in logging module
In-Reply-To: <1327504850.36.0.736318881933.issue13859@psf.upfronthosting.co.za>
Message-ID: <1327551044.37.0.692453713774.issue13859@psf.upfronthosting.co.za>


Matt Joiner <anacrolix at gmail.com> added the comment:

Cheers, thanks for the fast turn around.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13859>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:26:35 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:26:35 +0000
Subject: [issue5707] IDLE will not load
In-Reply-To: <1239003620.62.0.108807854341.issue5707@psf.upfronthosting.co.za>
Message-ID: <1327551995.18.0.598045768819.issue5707@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

The patch fixed a real issue, the difference return of filter in 3.x versus 2.x. Bad key bindings came up in #11437 and #13864 also.

----------
nosy: +serwy, terry.reedy
versions: +Python 3.2 -Python 3.0, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5707>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:27:05 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 04:27:05 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327552025.93.0.750450422819.issue11457@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Have you researched how other languages plan to expose sub-millisecond times?  The isn't an API that will get points for originality.  Also, it needs to be an API that is time efficient (many scripts use os.stat() frequently to scan files for changes and that check needs to be fast).

Please do not "just check it in".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:27:24 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:27:24 +0000
Subject: [issue11437] IDLE crash on startup with typo in config-keys.cfg
In-Reply-To: <1299543243.52.0.970222997696.issue11437@psf.upfronthosting.co.za>
Message-ID: <1327552044.9.0.0523353068793.issue11437@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

#13864 is a duplicate, where I mentioned the remedy of deleting the bad file.

This is similar to #5707, but I am not sure if exactly the same.
The patch there fixed one problem but not the bad key binding problem.

----------
nosy: +serwy
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11437>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:29:40 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:29:40 +0000
Subject: [issue4765] IDLE fails to "Delete Custom Key Set" properly
In-Reply-To: <1230525520.13.0.417394561266.issue4765@psf.upfronthosting.co.za>
Message-ID: <1327552180.73.0.892780175764.issue4765@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +serwy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4765>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:33:23 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:33:23 +0000
Subject: [issue11437] IDLE crash on startup with typo in config-keys.cfg
In-Reply-To: <1299543243.52.0.970222997696.issue11437@psf.upfronthosting.co.za>
Message-ID: <1327552403.21.0.190102173833.issue11437@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

#13071 is another duplicate closed in favor of this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11437>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:35:35 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:35:35 +0000
Subject: [issue13071] IDLE accepts, then crashes, on invalid key bindings.
In-Reply-To: <1317341122.37.0.866117748717.issue13071@psf.upfronthosting.co.za>
Message-ID: <1327552535.36.0.498629500051.issue13071@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +serwy
resolution:  -> duplicate
status: open -> closed
superseder:  -> IDLE crash on startup with typo in config-keys.cfg

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13071>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:39:54 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 04:39:54 +0000
Subject: [issue13864] IDLE: Python 2.7.2 refuses to open
In-Reply-To: <1327521133.43.0.177079679673.issue13864@psf.upfronthosting.co.za>
Message-ID: <1327552794.76.0.233549599235.issue13864@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

What system are you running on?
I presume by 'keybindings in Python', you mean 'keybindings in IDLE'. Correct?
By 'change to control', do you mean you made a line in the IDLE Preferences dialog, Keys tab, binding box, look like
"keyname - <Control-Key>" instead of "keyname - <Control-Key-X>" ?

The default configuration files are stored in and loaded from pythonxp/Lib/idlelib/. The user configuration files are in <userhome>/.idlerc/. They are not deleted when you delete or install Python. The reason is so that users do not lose their custom configuration when they upgrade or re-install for non-idle reasons. The location of <userhome> depends on the system. On my Win7 system, <userhome> appears as C;/user/Terry. On XP, I believe it was C:/Documents and Settings/Users/Terry/, possibly with 'Application Settings/idle' (or something) added. In any case, delete the bad file, perhaps after uploading config-keys.cfg here with the name 'poisonous_config_keys'.

PS. Usage questions are generally best directed to python-list, where you generally get faster answers. In this case, though, there seems to be a nasty bug. A bad key binding should give a nice error message when one first tries to store it.

Also, try searching the tracker before opening a new issue. Searching for 'key binding' in #11437 Component: IDLE turns up "IDLE crash on startup with typo in config-keys.cfg". So I am closing this as a duplicate. Please post any answers there.

----------
nosy: +serwy
resolution:  -> duplicate
stage:  -> test needed
status: open -> closed
superseder:  -> IDLE crash on startup with typo in config-keys.cfg
title: Python 2.7.2 refuses to open -> IDLE: Python 2.7.2 refuses to open
type:  -> behavior
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13864>
_______________________________________

From report at bugs.python.org  Thu Jan 26 05:59:34 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 26 Jan 2012 04:59:34 +0000
Subject: [issue13870] false comment in collections/__init__.py ordered dict
Message-ID: <1327553973.99.0.0526575154681.issue13870@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

http://hg.python.org/cpython/file/tip/Lib/collections/__init__.py#l37 states that the prev/next links are weakref proxies; as of http://hg.python.org/cpython/diff/3977dc349ae7/Lib/collections.py this is no longer true of the next links.  

It could be fixed by changing

# The prev/next links are weakref proxies (to prevent circular references).

to 

# The prev links are weakref proxies (to prevent circular references).

----------
components: Library (Lib)
files: collections_init.patch
keywords: patch
messages: 151996
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: false comment in collections/__init__.py ordered dict
Added file: http://bugs.python.org/file24326/collections_init.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13870>
_______________________________________

From report at bugs.python.org  Thu Jan 26 06:47:42 2012
From: report at bugs.python.org (Jim Jewett)
Date: Thu, 26 Jan 2012 05:47:42 +0000
Subject: [issue13871] namedtuple does not normalize field names when sanitizing
Message-ID: <1327556862.47.0.690638642414.issue13871@psf.upfronthosting.co.za>


New submission from Jim Jewett <jimjjewett at gmail.com>:

collections.namedtuple raises a ValueError if any of the field names are not valid identifiers, or are duplicates.

It does not normalize the identifiers when checking for duplicates.

(Similar issue with the typename)

>>> namedtuple("dup_fields", ["a", "a"])
Traceback (most recent call last):
  File "<pyshell#23>", line 1, in <module>
    namedtuple("dup_fields", ["a", "a"])
  File "C:\python32\lib\collections.py", line 345, in namedtuple
    raise ValueError('Encountered duplicate field name: %r' % name)
ValueError: Encountered duplicate field name: 'a'



>>> namedtuple("nfk_tester", ["a", "?"])
Traceback (most recent call last):
  File "<pyshell#22>", line 1, in <module>
    namedtuple("nfk_tester", ["a", "?"])
  File "C:\python32\lib\collections.py", line 365, in namedtuple
    raise SyntaxError(e.msg + ':\n\n' + class_definition)
  File "<string>", line None
SyntaxError: duplicate argument 'a' in function definition:
...



and 


>>> namedtuple("just?", "?")
Traceback (most recent call last):
  File "<pyshell#24>", line 1, in <module>
    namedtuple("just?", "?")
  File "C:\python32\lib\collections.py", line 366, in namedtuple
    result = namespace[typename]
KeyError: 'just?'

----------
messages: 151997
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: namedtuple does not normalize field names when sanitizing

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13871>
_______________________________________

From report at bugs.python.org  Thu Jan 26 08:32:18 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 07:32:18 +0000
Subject: [issue13871] namedtuple does not normalize field names when sanitizing
In-Reply-To: <1327556862.47.0.690638642414.issue13871@psf.upfronthosting.co.za>
Message-ID: <1327563138.91.0.859529645186.issue13871@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee:  -> rhettinger
nosy: +rhettinger
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13871>
_______________________________________

From report at bugs.python.org  Thu Jan 26 08:48:56 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 07:48:56 +0000
Subject: [issue13871] namedtuple does not normalize field names when sanitizing
In-Reply-To: <1327556862.47.0.690638642414.issue13871@psf.upfronthosting.co.za>
Message-ID: <1327564136.45.0.19080831073.issue13871@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

The SyntaxError is fine.  The dupcheck isn't about sanitization anyway; rather, it was part of a suite of tests designed to add a more helpful error messages than the usual ones you get if you had rolled a class by hand.  I would close as "invalid" but want to think it over for a while -- we'll see how all your other normalization "bugs" resolve.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13871>
_______________________________________

From report at bugs.python.org  Thu Jan 26 08:49:45 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 07:49:45 +0000
Subject: [issue13870] false comment in collections/__init__.py ordered dict
In-Reply-To: <1327553973.99.0.0526575154681.issue13870@psf.upfronthosting.co.za>
Message-ID: <1327564185.28.0.624049062477.issue13870@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee:  -> rhettinger
nosy: +rhettinger
priority: normal -> low

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13870>
_______________________________________

From report at bugs.python.org  Thu Jan 26 08:52:56 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 07:52:56 +0000
Subject: [issue13870] Out-of-date comment in collections/__init__.py ordered
	dict
In-Reply-To: <1327553973.99.0.0526575154681.issue13870@psf.upfronthosting.co.za>
Message-ID: <1327564376.23.0.354257980477.issue13870@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

This patch is fine.  Anyone can feel free to apply it to Py3.2 and Py3.3 if I don't get to it first.

----------
title: false comment in collections/__init__.py ordered dict -> Out-of-date comment in collections/__init__.py ordered dict
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13870>
_______________________________________

From report at bugs.python.org  Thu Jan 26 08:54:29 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 07:54:29 +0000
Subject: [issue13871] namedtuple does not normalize field names when checking
	for duplicates
In-Reply-To: <1327556862.47.0.690638642414.issue13871@psf.upfronthosting.co.za>
Message-ID: <1327564469.59.0.647996615535.issue13871@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
title: namedtuple does not normalize field names when sanitizing -> namedtuple does not normalize field names when checking for duplicates

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13871>
_______________________________________

From report at bugs.python.org  Thu Jan 26 09:15:47 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 08:15:47 +0000
Subject: [issue13870] Out-of-date comment in collections/__init__.py ordered
	dict
In-Reply-To: <1327553973.99.0.0526575154681.issue13870@psf.upfronthosting.co.za>
Message-ID: <E1RqKUk-0008O5-1D@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f7283825effa by Raymond Hettinger in branch '3.2':
Issue 13870: Fix out of date comment.
http://hg.python.org/cpython/rev/f7283825effa

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13870>
_______________________________________

From report at bugs.python.org  Thu Jan 26 09:19:22 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Thu, 26 Jan 2012 08:19:22 +0000
Subject: [issue13870] Out-of-date comment in collections/__init__.py ordered
	dict
In-Reply-To: <1327553973.99.0.0526575154681.issue13870@psf.upfronthosting.co.za>
Message-ID: <1327565962.51.0.826954135137.issue13870@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13870>
_______________________________________

From report at bugs.python.org  Thu Jan 26 11:01:00 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Thu, 26 Jan 2012 10:01:00 +0000
Subject: [issue13863] import.c sometimes generates incorrect timestamps on
	Windows + NTFS
In-Reply-To: <1327519681.79.0.252690630069.issue13863@psf.upfronthosting.co.za>
Message-ID: <1327572060.36.0.289669435351.issue13863@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Also seen on Windows Vista;  seems to be a general Windows + NTFS problem.

Changing title to make it clearer that this is a core language issue.

It seems as though the correct fix would be to use something like GetFileInformationByHandle in place of the fstat calls in import.c.

----------
title: Using py_compile does not prevent recompilation due to 'bad mtime'. -> import.c sometimes generates incorrect timestamps on Windows + NTFS

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________

From report at bugs.python.org  Wed Jan 25 18:49:19 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Wed, 25 Jan 2012 17:49:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327495499.2388.12.camel@surprise>
Message-ID: <1327513722.2388.20.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Wed, 2012-01-25 at 12:45 +0000, Dave Malcolm wrote:
> Dave Malcolm <dmalcolm at redhat.com> added the comment:
> 
> I've found a bug in my patch; insertdict writes the old non-randomized
> hash value into me_hash at:
>         ep->me_hash = hash;
> rather than using the randomized hash, leading to issues when tested
> against a real attack.

I'm attaching a revised version of the patch that should fix the above
issue:
  hybrid-approach-dmalcolm-2012-01-25-002.patch

Changes relative to -001.patch:
* updated insertdict() so that when it write ep->me_hash, it uses the
correct hash value.  Unfortunately there doesn't seem to be a good way
of reusing the value we calculated in the "paranoid" ma_lookup
callbacks, without breaking ABI (suggestions welcome),  so we call
PyObject_RandomizedHash again.
* slightly reworked the two _paranoid ma_lookup callbacks to capture the
randomized hash as a local variable, in case there's a way of reusing it
in insertdict()
* when lookdict() calls into itself, it now calls mp->ma_lookup instead
* don't generate a fatal error with an unknown ma_lookup callback.

With this, I'm able to insert 200,000 non-equal PyUnicodeObject with
hash()==0 into a dict on a 32-bit build --with-pydebug in 2.2 seconds;
it can retrieve all the values correctly in about 4 seconds [compare
with ~1.5 hours of CPU churn for inserting the same data on an optimized
build without the patch on the same guest].

The amortized ratio of total work done per modification increases
linearly when under an O(N^2) attack, and the dict switches itself to
paranoid mode 56 insertions after ma_table stops using ma_smalltable
(that's when we start tracking stats).  After the transition to paranoid
mode, it drops to an average of a little under 2 probes per insertion
(the amortized ratio seems to be converging to about 1.9 probes per key
insertion at the point where my hostile test data runs out).

----------
Added file: http://bugs.python.org/file24324/hybrid-approach-dmalcolm-2012-01-25-002.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 3be60a4c8c63 Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 20 11:01:06 2012 -0500
+++ b/Doc/using/cmdline.rst	Wed Jan 25 12:28:05 2012 -0500
@@ -460,6 +460,12 @@
    option.
 
 
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed when randomizing hashes.
+   It should be a number in the range [0; 4294967295].  The value 0 disables
+   the hash randomization.
+
 .. envvar:: PYTHONIOENCODING
 
    If this is set before running the interpreter, it overrides the encoding used
diff -r 3be60a4c8c63 Include/bytesobject.h
--- a/Include/bytesobject.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/bytesobject.h	Wed Jan 25 12:28:05 2012 -0500
@@ -124,6 +124,8 @@
 #define F_ALT	(1<<3)
 #define F_ZERO	(1<<4)
 
+PyAPI_FUNC(Py_hash_t) _PyBytes_RandomizedHash(PyObject *bytes);
+
 #ifdef __cplusplus
 }
 #endif
diff -r 3be60a4c8c63 Include/object.h
--- a/Include/object.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/object.h	Wed Jan 25 12:28:05 2012 -0500
@@ -488,6 +488,8 @@
 PyAPI_FUNC(int) PyObject_GenericSetAttr(PyObject *,
                                               PyObject *, PyObject *);
 PyAPI_FUNC(Py_hash_t) PyObject_Hash(PyObject *);
+PyAPI_FUNC(Py_hash_t) PyObject_RandomizedHash(PyObject *);
+
 PyAPI_FUNC(Py_hash_t) PyObject_HashNotImplemented(PyObject *);
 PyAPI_FUNC(int) PyObject_IsTrue(PyObject *);
 PyAPI_FUNC(int) PyObject_Not(PyObject *);
@@ -521,6 +523,12 @@
 PyAPI_FUNC(Py_hash_t) _Py_HashDouble(double);
 PyAPI_FUNC(Py_hash_t) _Py_HashPointer(void*);
 PyAPI_FUNC(Py_hash_t) _Py_HashBytes(unsigned char*, Py_ssize_t);
+PyAPI_FUNC(Py_hash_t) _Py_RandomizedHashBytes(unsigned char*, Py_ssize_t);
+typedef struct {
+    Py_hash_t prefix;
+    Py_hash_t suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
 #endif
 
 /* Helper for passing objects to printf and the like */
diff -r 3be60a4c8c63 Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/pythonrun.h	Wed Jan 25 12:28:05 2012 -0500
@@ -246,6 +246,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r 3be60a4c8c63 Include/unicodeobject.h
--- a/Include/unicodeobject.h	Fri Jan 20 11:01:06 2012 -0500
+++ b/Include/unicodeobject.h	Wed Jan 25 12:28:05 2012 -0500
@@ -2154,6 +2154,8 @@
 /* Clear all static strings. */
 PyAPI_FUNC(void) _PyUnicode_ClearStaticStrings(void);
 
+PyAPI_FUNC(Py_hash_t) _PyUnicode_RandomizedHash(PyObject *unicode);
+
 #ifdef __cplusplus
 }
 #endif
diff -r 3be60a4c8c63 Lib/os.py
--- a/Lib/os.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/os.py	Wed Jan 25 12:28:05 2012 -0500
@@ -761,23 +761,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r 3be60a4c8c63 Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/regrtest.py	Wed Jan 25 12:28:05 2012 -0500
@@ -559,6 +559,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(selected)
diff -r 3be60a4c8c63 Lib/test/test_dict.py
--- a/Lib/test/test_dict.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_dict.py	Wed Jan 25 12:28:05 2012 -0500
@@ -3,6 +3,8 @@
 
 import collections, random, string
 import gc, weakref
+import sys
+import time
 
 
 class DictTest(unittest.TestCase):
@@ -757,6 +759,301 @@
         self._tracked(MyDict())
 
 
+# Support classes for HashCollisionTests:
+class ChosenHash:
+    """
+    Use this to create arbitrary collections of keys that are non-equal
+    but have equal hashes, without needing to include hostile data
+    within the test suite.
+    """
+    def __init__(self, variability, hash):
+        self.variability = variability
+        self.hash = hash
+
+    def __eq__(self, other):
+        # The variability field is used to handle non-equalness:
+        return self.variability == other.variability
+
+    def __hash__(self):
+        return self.hash
+
+    def __repr__(self):
+        return 'ChosenHash(%r, %r)' % (self.variability,
+                                       self.hash)
+
+class Timer:
+    """
+    Simple way to measure time elapsed during a test case
+    """
+    def __init__(self):
+        self.starttime = time.time()
+
+    def get_elapsed_time(self):
+        """Get elapsed time in seconds as a float"""
+        curtime = time.time()
+        return curtime - self.starttime
+
+    def elapsed_time_as_str(self):
+        """Get elapsed time as a string (with units)"""
+        return '%0.3f seconds' % self.get_elapsed_time()
+
+class TookTooLong(RuntimeError):
+    def __init__(self, timelimit, elapsed, itercount=None):
+        self.timelimit = timelimit
+        self.elapsed = elapsed
+        self.itercount = itercount
+
+    def __str__(self):
+        result = 'took >= %s seconds' % self.timelimit
+        if self.itercount is not None:
+            result += (' (%0.3f seconds elapsed after %i iterations)'
+                       % (self.elapsed, self.itercount))
+        else:
+            result += ' (%0.3f seconds elapsed)' % self.elapsed
+        return result
+
+# Some of the tests involve constructing large dictionaries.  How big
+# should they be?
+ITEM_COUNT = 100000
+
+# Arbitrary threshold (in seconds) for a "reasonable amount of time"
+# that it should take to work with ITEM_COUNT items:
+TIME_LIMIT = 5
+
+class _FasterThanContext(object):
+    """
+    A context manager for implementing assertFasterThan
+    """
+    def __init__(self, test_case, **kwargs):
+        self.test_case = test_case
+        if 'seconds' in kwargs:
+            self.timelimit = kwargs['seconds']
+        else:
+            raise ValueError()
+
+    def __enter__(self):
+        self.timer = Timer()
+        return self
+
+    def __exit__(self, exc_type, exc_value, tb):
+        if exc_type is not None:
+            # let unexpected exceptions pass through
+            return
+
+        self.check_for_timeout()
+        if 1:
+            print('timer within %s took %s'
+                  % (self.test_case, self.timer.elapsed_time_as_str()))
+
+    def handle(self, callable_obj, args, kwargs):
+        """
+        If callable_obj is None, assertRaises/Warns is being used as a
+        context manager, so check for a 'msg' kwarg and return self.
+        If callable_obj is not None, call it passing args and kwargs.
+        """
+        if callable_obj is None:
+            self.msg = kwargs.pop('msg', None)
+            return self
+        with self:
+            callable_obj(*args, **kwargs)
+
+    def check_for_timeout(self, itercount=None):
+        """
+        Allow directly checking for timeouts,  potentially supplying an
+        iteration count if within a loop.  If the timer has elapsed, this will
+        raise a TookTooLong exception (potentially indicating how many
+        iterations were completed when the time limit was reached).
+
+        Otherwise, it does nothing.
+        """
+        elapsed_time = self.timer.get_elapsed_time()
+        if elapsed_time > self.timelimit:
+            raise TookTooLong(self.timelimit,
+                              elapsed_time,
+                              itercount)
+        
+ at support.cpython_only
+class HashCollisionTests(unittest.TestCase):
+    """
+    Issue 13703: tests about the behavior of dicts in the face of hostile data
+    """
+
+    def assertFasterThan(self, callableObj=None, *args, **kwargs):
+        context = _FasterThanContext(self, *args, **kwargs)
+        return context.handle(callableObj, args, kwargs)
+
+    def test_timings_with_benign_data(self):
+        # Verify that inserting many keys into a dict only takes a few seconds
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[i] = 0
+
+        # Verify that we can also retrieve the values quickly:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d[i]
+
+        # Verify that we can quickly insert the same item many times
+        # (overwriting each time):
+        d = dict()
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            for i in range(ITEM_COUNT):
+                d[0] = 0
+
+    def assert_ma_lookup(self, d, name):
+        # verify that a dict's ma_lookup hook has a specific value
+        if hasattr(d, '_stats'):
+            self.assertEqual(d._stats()['ma_lookup'],
+                             name)
+
+    def test_dict_stats(self):
+        # test for the diagnostic _stats() method, if it exists:
+        if hasattr(dict, '_stats'):
+            # Simple cases in which no probing should happen at all:
+            for d in [dict(zip(range(100), range(100))),
+                      dict(zip(range(1000), range(1000))),
+                      dict(zip(range(10000), range(10000)))]:
+                stats = d._stats()
+                self.assertIn('ma_mask', stats)
+                # ma_mask will be large for these dicts
+
+                # We don't expect any probing happened during construction:
+                self.assertEqual(stats['insertion_iter_count'], 0)
+                # They transition from ma_smalltable at size 6:
+                self.assertEqual(stats['num_insertions'], len(d) - 6)
+                self.assertEqual(stats['iter_count_per_insertion'], 0.0)
+
+                # We shouldn't have transitioned to paranoid mode:
+                self.assert_ma_lookup(d, 'lookdict')
+
+            # Cases in which we expect some hash collisions:
+            from itertools import permutations
+
+            for name, d, exp_length in [
+                ('All strings of length 4',
+                 dict([(''.join(chars), 1)
+                       for chars in permutations('abcdefghijklmnopqrstuvwxyz', 4)]),
+                 358800),
+                ]:
+
+                self.assertEqual(len(d), exp_length)
+                stats = d._stats()
+
+                # ma_mask will be large for this dict:
+                self.assertGreater(stats['ma_mask'], 7)
+
+                self.assertEqual(stats['num_insertions'], len(d) - 6)
+
+                # We expect significant probing happened during construction:
+                self.assertGreater(stats['insertion_iter_count'], 100000)
+                # ...but it should be sane relative to the amount of work we were
+                # asked to do:
+                self.assertLessEqual(stats['iter_count_per_insertion'], 2.0)
+                # and so we didn't transition:
+                self.assert_ma_lookup(d, 'lookdict_unicode')
+            
+    def test_not_reaching_limit(self):
+        # Ensure that we can insert equal-hashed keys up to (but not reaching)
+        # the collision limit:
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d = dict()
+            for i in range(50):
+                key = ChosenHash(variability=i, hash=42)
+                d[key] = 0
+            self.assert_ma_lookup(d, 'lookdict')
+
+    def test_many_hash_collisions(self):
+        """
+        Ensure we gracefully handle inserting a large number of non-equal keys
+        with the same hash that transition the dict to "paranoid" mode:
+        """
+        with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+            d = dict()
+            for i in range(60):
+                key = ChosenHash(variability=i, hash=42)
+                d[key] = 0
+
+            # Verify that we exceeded the safety threshold:
+            if hasattr(d, '_stats'):
+                self.assertGreaterEqual(d._stats()['iter_count_per_insertion'],
+                                        32.0)
+
+            # Verify that the dict transitioned to paranoid mode:
+            self.assert_ma_lookup(d, 'lookdict_paranoid')
+
+            # Verify key lookup:
+            for i in range(60):
+                key = ChosenHash(variability=i, hash=42)
+                self.assertEqual(d[key], 0)
+
+    # Frank Sievertsen found scenarios in which the collision-counting
+    # scheme could be attacked:
+    #   http://mail.python.org/pipermail/python-dev/2012-January/115726.html
+
+    def test_scenario_b_from_Frank_Sievertsen(self):
+        # This typically takes about 3 seconds on my box (with a debug build):
+        with self.assertFasterThan(seconds=TIME_LIMIT * 2) as cm:
+            d = dict()
+
+            # Insert non-equal hash collisions up to but not reaching the
+            # limit where we take effect:
+            for i in range(50):
+                key = ChosenHash(variability=i, hash=42)
+                d[key] = 0
+                cm.check_for_timeout(i)
+
+            # We shouldn't have transitioned yet:
+            self.assert_ma_lookup(d, 'lookdict')
+
+            # Now try to add many equal values that collide
+            # with the hash, and see how long it takes
+            for i in range(ITEM_COUNT):
+                key = ChosenHash(variability=0, hash=42)
+                d[key] = 0
+                cm.check_for_timeout(i)
+
+            # We still shouldn't have transitioned:
+            self.assert_ma_lookup(d, 'lookdict')
+
+    def test_paranoid_str_dict(self):
+        # Exercise the handling of pure-str paranoid dict:
+        if hasattr(dict, '_make_paranoid'):
+            from itertools import permutations
+
+            d = dict([(''.join(chars), 1)
+                       for chars in permutations('abcdefghijklmnopqrstuvwxyz', 4)])
+
+            # It shouldn't be in paranoid mode:
+            self.assert_ma_lookup(d, 'lookdict_unicode')
+
+            with self.assertFasterThan(seconds=TIME_LIMIT) as cm:
+                # Force it into paranoid mode:
+                d._make_paranoid()
+                self.assert_ma_lookup(d, 'lookdict_unicode_paranoid')
+                self.assertIn('abcd', d)
+                self.assertEqual(d['zyxw'], 1)
+
+                # Now empty the dict:
+                d.clear()
+
+                # We should now have a smalltable dict, but it still uses the
+                # paranoid ma_lookup:
+                if hasattr(d, '_stats'):
+                    self.assertEqual(d._stats()['ma_mask'], 7)
+                    self.assert_ma_lookup(d, 'lookdict_unicode_paranoid')
+                
+                # Verify that the now-empty dict is still usable:
+                self.assertNotIn('abcd', d)
+                d['foo'] = 'bar'
+                self.assertIn('foo', d)
+                self.assertEqual(d['foo'], 'bar')
+
+                # Transition it from pure-str to mixed mode:
+                d[1] = 'not a str'
+                self.assert_ma_lookup(d, 'lookdict_paranoid')
+                self.assertIn(1, d)
+                self.assertEqual(d[1], 'not a str')
+
 from test import mapping_tests
 
 class GeneralMappingTests(mapping_tests.BasicTestMappingProtocol):
@@ -771,6 +1068,7 @@
 def test_main():
     support.run_unittest(
         DictTest,
+        HashCollisionTests,
         GeneralMappingTests,
         SubclassMappingTests,
     )
diff -r 3be60a4c8c63 Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_hash.py	Wed Jan 25 12:28:05 2012 -0500
@@ -4,9 +4,12 @@
 # Also test that hash implementations are inherited as expected
 
 import unittest
+import struct
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -117,10 +120,56 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+ at support.cpython_only
+class HashRandomizationTest(unittest.TestCase):
+    types_to_check = [str,
+                      lambda s: bytes(s, encoding='ascii')]
+
+    def get_randomized_hash(self, text, seed=None):
+        env = {}
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'import sys; print(sys._getrandomizedhash(%r))' % text,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        for t in self.types_to_check:
+            self.assertEqual(hash(t("")), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            h = 1453079729188098211
+        else:
+            h = -1600925533
+        for t in self.types_to_check:
+            self.assertEqual(self.get_randomized_hash(t("abc"), seed=0), h)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        for t in self.types_to_check:
+            self.assertEqual(self.get_randomized_hash(t("abc"), seed=42), h)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        for t in self.types_to_check:
+            run1 = self.get_randomized_hash(t("abc"))
+            run2 = self.get_randomized_hash(t("abc"))
+            self.assertNotEqual(run1, run2)
+
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
-                              HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                         HashInheritanceTestCase,
+                         HashBuiltinsTestCase,
+                         HashRandomizationTest)
 
 
 if __name__ == "__main__":
diff -r 3be60a4c8c63 Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 20 11:01:06 2012 -0500
+++ b/Lib/test/test_os.py	Wed Jan 25 12:28:05 2012 -0500
@@ -12,6 +12,7 @@
 import time
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 import contextlib
 import mmap
 import platform
@@ -630,14 +631,32 @@
             self.assertEqual(f.read(), b'')
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 @contextlib.contextmanager
 def _execvpe_mockup(defpath=None):
diff -r 3be60a4c8c63 Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 20 11:01:06 2012 -0500
+++ b/Makefile.pre.in	Wed Jan 25 12:28:05 2012 -0500
@@ -322,6 +322,7 @@
 		Python/pystate.o \
 		Python/pythonrun.o \
 		Python/pytime.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r 3be60a4c8c63 Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Modules/posixmodule.c	Wed Jan 25 12:28:05 2012 -0500
@@ -9329,81 +9329,36 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
+Return n pseudo-random bytes.");
 
 static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
 
     /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
 
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandleW(L"advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
@@ -9445,42 +9400,6 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
-    }
-    return result;
-}
-#endif
-
 #ifdef HAVE_SETRESUID
 PyDoc_STRVAR(posix_setresuid__doc__,
 "setresuid(ruid, euid, suid)\n\n\
@@ -10887,12 +10806,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
 #ifdef HAVE_SETRESUID
     {"setresuid",       posix_setresuid, METH_VARARGS, posix_setresuid__doc__},
 #endif
diff -r 3be60a4c8c63 Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/bytesobject.c	Wed Jan 25 12:28:05 2012 -0500
@@ -867,6 +867,13 @@
     return a->ob_shash;
 }
 
+Py_hash_t
+_PyBytes_RandomizedHash(PyObject *a)
+{
+    return _Py_RandomizedHashBytes((unsigned char *)((PyBytesObject *)a)->ob_sval,
+                                   Py_SIZE(a));
+}
+
 static PyObject*
 bytes_subscript(PyBytesObject* self, PyObject* item)
 {
diff -r 3be60a4c8c63 Objects/dictobject.c
--- a/Objects/dictobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/dictobject.c	Wed Jan 25 12:28:05 2012 -0500
@@ -10,6 +10,32 @@
 #include "Python.h"
 #include "stringlib/eq.h"
 
+/* undefine this: */
+#define DICT_PROTECTION_TRACKING 1
+
+#define Py_MAX_AVERAGE_PROBES_PER_INSERT 32
+/* power-of-two to make multiplication fast */
+
+/* Avoid overflow by dividing stats when they exceed this: */
+#define Py_DICT_STATS_ROUNDING_THRESHOLD 0x40000000
+
+/* For large dictionaries, reuse the space allocated for ma_smalltable */
+typedef struct _Py_LargeDictFields {
+    int is_inserting;
+
+    /* Check the ratio between these, to track amortized cost per operation */
+    size_t num_insertions;
+    size_t insertion_iter_count;
+
+} _Py_LargeDictFields;
+
+#define DICT_IS_PARANOID(mp) \
+  ((mp)->ma_lookup == lookdict_paranoid \
+   || ((mp)->ma_lookup == lookdict_unicode_paranoid))
+
+#define PyDict_LARGEDICTFIELDS(mp) \
+  ((_Py_LargeDictFields*)&(mp)->ma_smalltable)
+/* FIXME: add assert(mp->ma_table != mp->ma_smalltable) */
 
 /* Set a key error with the specified argument, wrapping it in a
  * tuple automatically so that tuple keys are not unpacked as the
@@ -139,11 +165,28 @@
 
 /* forward declarations */
 static PyDictEntry *
+lookdict(PyDictObject *mp, PyObject *key, Py_hash_t hash);
+
+static PyDictEntry *
 lookdict_unicode(PyDictObject *mp, PyObject *key, Py_hash_t hash);
 
+static PyDictEntry *
+lookdict_paranoid(PyDictObject *mp, PyObject *key, Py_hash_t hash);
+
+static PyDictEntry *
+lookdict_unicode_paranoid(PyDictObject *mp, PyObject *key, Py_hash_t hash);
+
+static int
+transition_to_paranoid_dict(PyDictObject *mp);
+
+static int
+check_iter_count(PyDictObject *mp, PyObject *key, Py_hash_t hash);
+
 #ifdef SHOW_CONVERSION_COUNTS
 static long created = 0L;
-static long converted = 0L;
+static long converted_unicode_to_general = 0L;
+static long converted_unicode_to_paranoid = 0L;
+static long converted_general_to_paranoid = 0L;
 
 static void
 show_counts(void)
@@ -352,7 +395,7 @@
                  * XXX A clever adversary could prevent this
                  * XXX from terminating.
                  */
-                return lookdict(mp, key, hash);
+                return mp->ma_lookup(mp, key, hash);
             }
         }
         freeslot = NULL;
@@ -384,11 +427,21 @@
                  * XXX A clever adversary could prevent this
                  * XXX from terminating.
                  */
-                return lookdict(mp, key, hash);
+                return mp->ma_lookup(mp, key, hash);
             }
         }
         else if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+
+        if (check_iter_count(mp, key, hash)) {
+            if (-1 == transition_to_paranoid_dict(mp)) {
+                return NULL;
+            }
+            
+            /* Everything has changed.
+               Restart the lookup using the new function: */
+            return mp->ma_lookup(mp, key, hash);
+        }
     }
     assert(0);          /* NOT REACHED */
     return 0;
@@ -422,7 +475,14 @@
 #ifdef SHOW_CONVERSION_COUNTS
         ++converted;
 #endif
-        mp->ma_lookup = lookdict;
+        if (mp->ma_lookup == lookdict_unicode) {
+            /* transition from lookdict_unicode -> lookdict */
+            mp->ma_lookup = lookdict;
+        } else {
+            /* transition from lookdict_unicode_paranoid -> lookdict_paranoid */
+            assert(mp->ma_lookup == lookdict_unicode_paranoid);
+            mp->ma_lookup = lookdict_paranoid;
+        }
         return lookdict(mp, key, hash);
     }
     i = (size_t)hash & mask;
@@ -451,11 +511,66 @@
             return ep;
         if (ep->me_key == dummy && freeslot == NULL)
             freeslot = ep;
+        if (check_iter_count(mp, key, hash)) {
+            if (-1 == transition_to_paranoid_dict(mp)) {
+                return NULL;
+            }
+            
+            /* Everything has changed.
+               Restart the lookup using the new function: */
+            return mp->ma_lookup(mp, key, hash);
+        }
     }
     assert(0);          /* NOT REACHED */
     return 0;
 }
 
+/* Version of lookdict which ignores the hash value passed in, and instead
+   generates an alternate hash for each object using a secret.
+   Note that if the object can't do this, it simply reuses its cached hash
+*/
+static PyDictEntry *
+lookdict_paranoid(PyDictObject *mp, PyObject *key, Py_hash_t hash)
+{
+    /*
+      Ignore the object's cached hash value, and instead compute and use a
+      "secret" one.  This won't be cached, meaning that the per-object cost
+      of the call gets a lot larger.  However, it ought to still be O(1)
+    */
+    Py_hash_t randomized_hash = PyObject_RandomizedHash(key);
+    PyDictEntry *ep = lookdict(mp, key, randomized_hash);
+    return ep;
+}
+
+/*
+  Similar to lookdict_paranoid and lookdict_unicode: used when a dict has
+  received hostile data, where all keys are PyUnicodeObject
+*/
+static PyDictEntry *
+lookdict_unicode_paranoid(PyDictObject *mp, PyObject *key,
+                          Py_hash_t hash)
+{
+    /*
+      As per lookdict_paranoid, we ignore the object's cached hash value
+    */
+    Py_hash_t randomized_hash;
+    PyDictEntry *ep;
+
+    if (!PyUnicode_CheckExact(key)) {
+        randomized_hash = PyObject_RandomizedHash(key);
+        /* this call will transition internally to lookdict_paranoid: */
+        ep = lookdict_unicode(mp, key, randomized_hash); 
+    } else {
+        /* use optimized implementation when obtaining randomized hash, since
+           we know it's a PyUnicodeObject: */
+        randomized_hash = _PyUnicode_RandomizedHash(key);
+        ep = lookdict_unicode(mp, key, randomized_hash);
+    }
+
+    return ep;
+}
+
+
 int
 _PyDict_HasOnlyStringKeys(PyObject *dict)
 {
@@ -463,7 +578,8 @@
     PyObject *key, *value;
     assert(PyDict_Check(dict));
     /* Shortcut */
-    if (((PyDictObject *)dict)->ma_lookup == lookdict_unicode)
+    if (((PyDictObject *)dict)->ma_lookup == lookdict_unicode || 
+        ((PyDictObject *)dict)->ma_lookup == lookdict_unicode_paranoid)
         return 1;
     while (PyDict_Next(dict, &pos, &key, &value))
         if (!PyUnicode_Check(key))
@@ -530,9 +646,23 @@
     PyObject *old_value;
     register PyDictEntry *ep;
     typedef PyDictEntry *(*lookupfunc)(PyDictObject *, PyObject *, Py_hash_t);
+    _Py_LargeDictFields *large_fields = PyDict_LARGEDICTFIELDS(mp);
 
     assert(mp->ma_lookup != NULL);
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        large_fields->is_inserting = 1;
+
+        /* Increment num_insertions, protecting against stats overflow: */
+        if (++large_fields->num_insertions > Py_DICT_STATS_ROUNDING_THRESHOLD) {
+            /* Preserve the ratio between these two: */
+            large_fields->insertion_iter_count >>= 4;
+            large_fields->num_insertions >>= 4;
+        }
+    }
     ep = mp->ma_lookup(mp, key, hash);
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        large_fields->is_inserting = 0;
+    }
     if (ep == NULL) {
         Py_DECREF(key);
         Py_DECREF(value);
@@ -553,7 +683,15 @@
             Py_DECREF(dummy);
         }
         ep->me_key = key;
-        ep->me_hash = hash;
+        if (!DICT_IS_PARANOID(mp)) {
+            ep->me_hash = hash;
+        } else {
+            /* Ideally we need some way of getting back the randomized hash
+	       from the ma_lookup function, but there doesn't seem to be a
+	       way of doing that without breaking ABI.
+               For now, recalculate the randomized hash: */
+            ep->me_hash = PyObject_RandomizedHash(key);
+        }
         ep->me_value = value;
         mp->ma_used++;
     }
@@ -675,8 +813,15 @@
         /* else key == value == NULL:  nothing to do */
     }
 
-    if (is_oldtable_malloced)
+    if (is_oldtable_malloced) {
         PyMem_DEL(oldtable);
+    } else {
+        if (mp->ma_table != mp->ma_smalltable) {
+            /* clean ma_smalltable for use as _Py_LargeDictFields: */
+            memset(mp->ma_smalltable, 0, sizeof(mp->ma_smalltable));
+        }
+    }
+
     return 0;
 }
 
@@ -1996,6 +2141,91 @@
 PyDoc_STRVAR(copy__doc__,
 "D.copy() -> a shallow copy of D");
 
+#ifdef DICT_PROTECTION_TRACKING
+/* Debug code for use by selftests: */
+static const char *
+get_name_of_ma_lookup(PyDictObject *mp)
+{
+    /* can't do this as a switch statement */
+    if (mp->ma_lookup == lookdict) {
+        return "lookdict";
+    } else if (mp->ma_lookup == lookdict_unicode) {
+        return "lookdict_unicode";
+    } else if (mp->ma_lookup == lookdict_paranoid) {
+        return "lookdict_paranoid";
+    } else if (mp->ma_lookup == lookdict_unicode_paranoid) {
+        return "lookdict_unicode_paranoid";
+    } else {
+        return "unknown ma_lookup";
+    }
+}
+
+static PyObject *
+dict__stats(PyDictObject *mp)
+{
+    PyObject *result = PyDict_New();
+    if (!result) goto error;
+
+#define ADD_STAT(name, expr) \
+    {                                                             \
+        PyObject *value = (expr);                                 \
+        if (!value)                                               \
+            goto error;                                           \
+        if (-1 == PyDict_SetItemString(result, (name), value)) {  \
+            Py_DECREF(value);                                     \
+            goto error;                                           \
+        }                                                         \
+        Py_DECREF(value);                                         \
+    }
+
+    ADD_STAT("ma_mask", PyLong_FromLong(mp->ma_mask));
+    ADD_STAT("ma_lookup", PyUnicode_FromString(get_name_of_ma_lookup(mp)));
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        _Py_LargeDictFields *large_fields = PyDict_LARGEDICTFIELDS(mp);
+        ADD_STAT("num_insertions", PyLong_FromLong(large_fields->num_insertions));
+        ADD_STAT("insertion_iter_count", PyLong_FromLong(large_fields->insertion_iter_count));
+        if (large_fields->num_insertions > 0) {
+          ADD_STAT("iter_count_per_insertion",
+                   PyFloat_FromDouble((double)large_fields->insertion_iter_count
+                                      / (double)large_fields->num_insertions));
+        }
+    }
+
+    return result;
+
+ error:
+    Py_XDECREF(result);
+    return NULL;
+}
+
+static PyObject *
+dict__make_paranoid(register PyDictObject *mp)
+{
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        if (DICT_IS_PARANOID(mp)) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "dict is already in paranoid mode");
+            return NULL;
+        } else {
+            transition_to_paranoid_dict(mp);
+            Py_RETURN_NONE;
+        }
+    } else {
+        PyErr_SetString(PyExc_RuntimeError,
+                        ("dict is not large enough to be forced"
+                         " into paranoid mode"));
+        return NULL;        
+    }
+}
+
+
+PyDoc_STRVAR(_stats__doc__,
+"D._stats() -> for CPython selftests: obtain debugging stats on D");
+
+PyDoc_STRVAR(_make_paranoid__doc__,
+"D._make_paranoid() -> for CPython selftests: forcibly transition this dict into paranoid mode");
+#endif
+
 /* Forward */
 static PyObject *dictkeys_new(PyObject *);
 static PyObject *dictitems_new(PyObject *);
@@ -2037,6 +2267,12 @@
      clear__doc__},
     {"copy",            (PyCFunction)dict_copy,         METH_NOARGS,
      copy__doc__},
+#ifdef DICT_PROTECTION_TRACKING
+    {"_stats",          (PyCFunction)dict__stats,       METH_NOARGS,
+     _stats__doc__},
+    {"_make_paranoid",  (PyCFunction)dict__make_paranoid, METH_NOARGS,
+     _make_paranoid__doc__},
+#endif
     {NULL,              NULL}   /* sentinel */
 };
 
@@ -3067,3 +3303,144 @@
 {
     return dictview_new(dict, &PyDictValues_Type);
 }
+
+/*
+  Swap the hash values for the active entries with the values
+  in the corresponding indices in the given array (of size
+  (mp->ma_mask + 1) values)
+ */
+static void
+swap_hash_values(PyDictObject *mp, Py_hash_t *hashes)
+{
+    Py_ssize_t i;
+
+    assert(mp);
+    assert(hashes);
+
+    for (i=0; i <= mp->ma_mask;  i++) {
+        PyDictEntry *ep = &mp->ma_table[i];
+        if (ep->me_value != NULL) { /* active entry */
+            Py_hash_t old_hash = ep->me_hash;
+            ep->me_hash = hashes[i];
+            hashes[i] = old_hash;
+        }
+    }
+}
+
+/*
+  Switch to new lookup implementation, using secret hashes, reordering
+  all of the me_hash slots.
+
+  Returns -1 for failure, 0 for success
+*/
+static int
+transition_to_paranoid_dict(PyDictObject *mp)
+{
+    Py_hash_t *hashes;
+    Py_ssize_t i;
+
+    assert(mp->ma_table != mp->ma_smalltable);
+
+    assert(!DICT_IS_PARANOID(mp));
+
+    /*
+      Recalculate all of the me_hash values within ma_table
+      using the randomized new algorithm.
+
+      An error could occur during this process, so allocate
+      a temporary buffer:
+    */
+    hashes = PyMem_NEW(Py_hash_t, mp->ma_mask + 1);
+    if (hashes == NULL) {
+        PyErr_NoMemory();
+        return -1;
+    }
+
+    /* Populate "hashes" with new values for the active entries,
+       leaving the other indices untouched: */
+    for (i=0; i <= mp->ma_mask;  i++) {
+        PyDictEntry *ep = &mp->ma_table[i];
+        if (ep->me_value != NULL) { /* active entry */
+            Py_hash_t new_hash = PyObject_RandomizedHash(ep->me_key);
+            if (new_hash != -1) {
+                hashes[i] = new_hash;
+            } else {
+                /* Error: */
+                PyMem_DEL(hashes);
+                return -1;
+            }
+        }
+    }
+
+    /* OK, we have new hashes for all active entries, and no errors
+       occurred.  Copy them up into the ma_table, swapping the old
+       value into the buffer in case we need to reconstruct things
+       later: */
+    swap_hash_values(mp, hashes);
+
+    /* 
+       "hashes" now holds the old hash values ; mp->ma_table holds
+       the new hash values
+
+       Now use dictresize to reorganize the order of the entries.
+       We use ma_mask here as the minimum size to ensure that the dict
+       doesn't start using ma_smalltable again
+    */
+    assert(mp->ma_mask >= PyDict_MINSIZE);
+    if (-1 == dictresize(mp, mp->ma_mask)) {
+        /* This can fail under low memory conditions, before changing
+           anything.  If so, swap the old hash values back in: */
+        swap_hash_values(mp, hashes);
+        PyMem_DEL(hashes);
+        return -1;
+    }
+
+    /* Success.  We're done with the "hashes" buffer: */
+    PyMem_DEL(hashes);
+
+    /* Change ma_lookup so that we use randomized hashes: */
+    if (mp->ma_lookup == lookdict_unicode) {
+        /* transition from lookdict_unicode -> lookdict_unicode_paranoid */
+        mp->ma_lookup = lookdict_unicode_paranoid;
+    } else {
+        /* transition from lookdict -> lookdict_paranoid */
+        assert(mp->ma_lookup == lookdict);
+        mp->ma_lookup = lookdict_paranoid;
+    }
+
+    /* FIXME: track conversions? */
+    return 0;
+}
+
+/* may want to hand-inline this one
+
+  Returns 1 when the transition-to-paranoid should occur, 0 otherwise
+ */
+static int
+check_iter_count(PyDictObject *mp, PyObject *key, Py_hash_t hash)
+{
+    if (mp->ma_mask >= PyDict_MINSIZE) {
+        _Py_LargeDictFields *large_fields = PyDict_LARGEDICTFIELDS(mp);
+        assert(mp->ma_table != mp->ma_smalltable);
+        if (large_fields->is_inserting) {
+
+            /* Increment insertion_iter_count,
+               protecting against stats overflow: */
+            if (++large_fields->insertion_iter_count
+                >= Py_DICT_STATS_ROUNDING_THRESHOLD) {
+               /* Preserve the ratio between these two: */
+               large_fields->insertion_iter_count >>= 4;
+               large_fields->num_insertions >>= 4;
+            }
+
+            /* Transition into paranoid mode if the ratio is too bad: */
+            if (large_fields->insertion_iter_count
+                > (large_fields->num_insertions * Py_MAX_AVERAGE_PROBES_PER_INSERT) ) {
+                if (!DICT_IS_PARANOID(mp)) {
+                    return 1;
+                }
+            }
+        }
+    }
+    return 0;
+}
diff -r 3be60a4c8c63 Objects/object.c
--- a/Objects/object.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/object.c	Wed Jan 25 12:28:05 2012 -0500
@@ -769,6 +769,22 @@
 }
 
 Py_hash_t
+_Py_RandomizedHashBytes(unsigned char *p, Py_ssize_t len)
+{
+    Py_uhash_t x;
+    Py_ssize_t i;
+
+    x = _Py_HashSecret.prefix ^ (Py_uhash_t) *p << 7;
+    for (i = 0; i < len; i++)
+        x = (_PyHASH_MULTIPLIER * x) ^ (Py_uhash_t) *p++;
+    x ^= (Py_uhash_t) len;
+    x ^= _Py_HashSecret.suffix;
+    if (x == -1)
+        x = -2;
+    return x;
+}
+
+Py_hash_t
 PyObject_HashNotImplemented(PyObject *v)
 {
     PyErr_Format(PyExc_TypeError, "unhashable type: '%.200s'",
@@ -797,6 +813,27 @@
     return PyObject_HashNotImplemented(v);
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
+Py_hash_t
+PyObject_RandomizedHash(PyObject *v)
+{
+    /*
+      We only randomize the hash for a few specific types for now, to ease
+      backporting.  Ultimately this could be a new entry in PyTypeObject
+    */
+    if (Py_TYPE(v) == &PyUnicode_Type) {
+        return _PyUnicode_RandomizedHash(v);
+    } else if (Py_TYPE(v) == &PyBytes_Type) {
+        return _PyBytes_RandomizedHash(v);
+    }
+    /* FIXME: for Python 2 backport extend to cover PyBuffer_Type */
+
+    /* Otherwise, just use the regular hash value: */
+    return PyObject_Hash(v);
+}
+
+
 PyObject *
 PyObject_GetAttrString(PyObject *v, const char *name)
 {
diff -r 3be60a4c8c63 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Objects/unicodeobject.c	Wed Jan 25 12:28:05 2012 -0500
@@ -11242,6 +11242,57 @@
 }
 #undef HASH
 
+Py_hash_t
+_PyUnicode_RandomizedHash(PyObject *self)
+{
+    Py_ssize_t len;
+    Py_uhash_t x;
+
+    if (PyUnicode_READY(self) == -1)
+        return -1;
+    len = PyUnicode_GET_LENGTH(self);
+    if (len == 0) {
+        _PyUnicode_HASH(self) = 0;
+        return 0;
+    }
+
+    /* Alternate definition of the hash function as a macro to above */
+#define HASH(P) \
+    x = _Py_HashSecret.prefix ^ (Py_uhash_t)*P << 7; \
+    while (--len >= 0) \
+        x = (_PyHASH_MULTIPLIER*x) ^ (Py_uhash_t)*P++;
+
+    switch (PyUnicode_KIND(self)) {
+    case PyUnicode_1BYTE_KIND: {
+        const unsigned char *c = PyUnicode_1BYTE_DATA(self);
+        HASH(c);
+        break;
+    }
+    case PyUnicode_2BYTE_KIND: {
+        const Py_UCS2 *s = PyUnicode_2BYTE_DATA(self);
+        HASH(s);
+        break;
+    }
+    default: {
+        Py_UCS4 *l;
+        assert(PyUnicode_KIND(self) == PyUnicode_4BYTE_KIND &&
+               "Impossible switch case in unicode_hash");
+        l = PyUnicode_4BYTE_DATA(self);
+        HASH(l);
+        break;
+    }
+    }
+    x ^= (Py_uhash_t)PyUnicode_GET_LENGTH(self);
+    x ^= _Py_HashSecret.suffix;
+
+    if (x == -1)
+        x = -2;
+    /* we do *not* update _PyUnicode_HASH(self) here */
+    return x;
+}
+#undef HASH
+
+
 PyDoc_STRVAR(index__doc__,
              "S.index(sub[, start[, end]]) -> int\n\
 \n\
diff -r 3be60a4c8c63 PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 20 11:01:06 2012 -0500
+++ b/PCbuild/pythoncore.vcproj	Wed Jan 25 12:28:05 2012 -0500
@@ -1891,6 +1891,10 @@
 				>
 			</File>
 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
+			<File
 				RelativePath="..\Python\structmember.c"
 				>
 			</File>
diff -r 3be60a4c8c63 Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Python/pythonrun.c	Wed Jan 25 12:28:05 2012 -0500
@@ -71,6 +71,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 extern int _PyFaulthandler_Init(void);
 extern void _PyFaulthandler_Fini(void);
 
@@ -219,6 +220,8 @@
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
 
+    _PyRandom_Init();
+
     interp = PyInterpreterState_New();
     if (interp == NULL)
         Py_FatalError("Py_Initialize: can't make first interpreter");
diff -r 3be60a4c8c63 Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Wed Jan 25 12:28:05 2012 -0500
@@ -0,0 +1,268 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Fail to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Fail to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Fail to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Fail to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Fail to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized Unicode hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+
diff -r 3be60a4c8c63 Python/sysmodule.c
--- a/Python/sysmodule.c	Fri Jan 20 11:01:06 2012 -0500
+++ b/Python/sysmodule.c	Wed Jan 25 12:28:05 2012 -0500
@@ -993,6 +993,20 @@
 10. Number of stack pops performed by call_function()"
 );
 
+PyDoc_STRVAR(_getrandomizedhash_doc,
+"_getrandomizedhash(object) -> int\n\
+\n\
+For selftests only: return PyObject_RandomizedHash for the given object.");
+
+static PyObject *
+sys__getrandomizedhash(PyObject *self, PyObject *arg)
+{
+    return PyLong_FromSsize_t(PyObject_RandomizedHash(arg));
+    /* FIXME: when backporting, note the Py_hash_t to long change,
+      and make it just PyLong_FromLong on Python 2.* */
+}
+
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -1093,6 +1107,8 @@
     {"settrace",        sys_settrace, METH_O, settrace_doc},
     {"gettrace",        sys_gettrace, METH_NOARGS, gettrace_doc},
     {"call_tracing", sys_call_tracing, METH_VARARGS, call_tracing_doc},
+    {"_getrandomizedhash",   (PyCFunction)sys__getrandomizedhash,
+     METH_O, _getrandomizedhash_doc},
     {NULL,              NULL}           /* sentinel */
 };
 

From report at bugs.python.org  Thu Jan 26 13:41:46 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 26 Jan 2012 12:41:46 +0000
Subject: [issue13435] Copybutton does not hide tracebacks
In-Reply-To: <1321724543.66.0.854276029391.issue13435@psf.upfronthosting.co.za>
Message-ID: <1327581706.89.0.546150256088.issue13435@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

This should be fixed now.

----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13435>
_______________________________________

From report at bugs.python.org  Thu Jan 26 13:43:11 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 26 Jan 2012 12:43:11 +0000
Subject: [issue13455] Reorganize tracker docs in the devguide
In-Reply-To: <1321984554.54.0.834071786299.issue13455@psf.upfronthosting.co.za>
Message-ID: <1327581791.37.0.231801839356.issue13455@psf.upfronthosting.co.za>


Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
stage: needs patch -> patch review
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13455>
_______________________________________

From report at bugs.python.org  Thu Jan 26 13:50:32 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 12:50:32 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1327552025.93.0.750450422819.issue11457@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwbfqJMD1gWqVJYyb4Mk9s3M1Cg0YXrCxEisvXqNXwneWw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Have you researched how other languages plan to expose sub-millisecond times? ?The isn't an API that will get points for originality. ?Also, it needs to be an API that is time efficient (many scripts use os.stat() frequently to scan files for changes and that check needs to be fast).

Using decimal timestamps should be an option, float timestamps must
remain the default.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Thu Jan 26 13:51:31 2012
From: report at bugs.python.org (Larry Hastings)
Date: Thu, 26 Jan 2012 12:51:31 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327582291.64.0.107989505574.issue11457@psf.upfronthosting.co.za>


Larry Hastings <larry at hastings.org> added the comment:

Victor: I *think* Raymond's comments were directed at my patch, not yours.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Thu Jan 26 13:57:32 2012
From: report at bugs.python.org (Matt Joiner)
Date: Thu, 26 Jan 2012 12:57:32 +0000
Subject: [issue13872] socket.detach doesn't mark socket._closed
Message-ID: <1327582652.34.0.856044764189.issue13872@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

socket.socket.detach doesn't mark the socket._closed flag. The flag is specific to the Python wrapper, so the fix is put there. Test included.

----------
components: Library (Lib)
files: socket-detach-mark-closed.patch
keywords: patch
messages: 152005
nosy: anacrolix
priority: normal
severity: normal
status: open
title: socket.detach doesn't mark socket._closed
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24327/socket-detach-mark-closed.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13872>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:03:45 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 13:03:45 +0000
Subject: [issue5210] zlib does not indicate end of compressed stream properly
In-Reply-To: <1234295180.57.0.273384677161.issue5210@psf.upfronthosting.co.za>
Message-ID: <1327583026.0.0.557831953213.issue5210@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5210>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:04:16 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 13:04:16 +0000
Subject: [issue8536] Support new features of ZLIB 1.2.4
In-Reply-To: <1272302591.08.0.11918516326.issue8536@psf.upfronthosting.co.za>
Message-ID: <1327583056.7.0.508695140582.issue8536@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8536>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:05:18 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 13:05:18 +0000
Subject: [issue5784] raw deflate format and zlib module
In-Reply-To: <1240006221.86.0.263428051867.issue5784@psf.upfronthosting.co.za>
Message-ID: <1327583118.06.0.492844754992.issue5784@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5784>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:07:45 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 13:07:45 +0000
Subject: [issue5804] Add an 'offset' argument to zlib.decompress
In-Reply-To: <1240307503.01.0.128578916914.issue5804@psf.upfronthosting.co.za>
Message-ID: <1327583265.46.0.362864662338.issue5804@psf.upfronthosting.co.za>


Changes by Nadeem Vawda <nadeem.vawda at gmail.com>:


----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5804>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:16:44 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 13:16:44 +0000
Subject: [issue13873] SIGBUS in test_zlib on Debian bigmem buildbot
Message-ID: <1327583804.29.0.71599909936.issue13873@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

http://www.python.org/dev/buildbot/all/builders/AMD64%20debian%20bigmem%203.x/builds/58/steps/test/logs/stdio

----------
assignee: nadeem.vawda
messages: 152006
nosy: nadeem.vawda
priority: normal
severity: normal
stage: needs patch
status: open
title: SIGBUS in test_zlib on Debian bigmem buildbot
type: crash

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13873>
_______________________________________

From report at bugs.python.org  Thu Jan 26 14:34:47 2012
From: report at bugs.python.org (Stefan Krah)
Date: Thu, 26 Jan 2012 13:34:47 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
Message-ID: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>


New submission from Stefan Krah <stefan-usenet at bytereef.org>:

In non-debug mode the read_null test fails with clang-3.0:

======================================================================
FAIL: test_disable (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/stefan/hg/cpython/Lib/test/test_faulthandler.py", line 235, in test_disable
    self.assertNotEqual(exitcode, 0)
AssertionError: 0 == 0



clang "optimizes" the undefined behavior into a simple assignment:


$ ~/usr/bin/clang --version
clang version 3.0 (tags/RELEASE_30/final)
Target: x86_64-unknown-freebsd9.0
Thread model: posix
$ 
$ cat read_null.c
#include <stdio.h>

int
main(void)
{
    int *x = NULL, y;

    y = *x;
    printf("%d\n", y);

    return 0;
}
$ 
$ ~/usr/bin/clang -Wall -O0 -g -o read_null read_null.c 
$ ./read_null 
Segmentation fault: 11 (core dumped)
$ ~/usr/bin/clang -Wall -O3 -g -o read_null read_null.c 
$ ./read_null 
0

----------
components: Tests
messages: 152007
nosy: haypo, skrah
priority: normal
severity: normal
stage: needs patch
status: open
title: test_faulthandler: read_null test fails with current clang
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:02:54 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 14:02:54 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwYzuX3UGM=yb7Zt9GWD0PCZZVyhq+mvAHEH954ydJPz_w@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Can you try x = (int *)1; instead of x = NULL; ?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:15:43 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 26 Jan 2012 14:15:43 +0000
Subject: [issue13863] import.c sometimes generates incorrect timestamps on
	Windows + NTFS
In-Reply-To: <1327519681.79.0.252690630069.issue13863@psf.upfronthosting.co.za>
Message-ID: <1327587343.96.0.114366582914.issue13863@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Hmm, interesting. This is exactly what happened recently when debugging pyc timestamp issues under Windows:
http://www.python.org/dev/buildbot/all/builders/x86%20Windows7%202.7/builds/1204/steps/test/logs/stdio

Some decoding of the above crash:
- the test would set the .py file's timestamps to 2**33
- this is truncated (module 2**32) and therefore should become 0 in the .pyc file's embedded timestamp
- in reality, the .pyc file's embedded timestamps is equal to 4294963696. Which is 2**32 - 3600.

As a sidenote, we don't have any tests that the pyc file is re-used when it is fresh enough. Perhaps by running an interpreter in a subprocess with "-v" we could examine the verbose messages printed out in import.c.

> It seems as though the correct fix would be to use something like GetFileInformationByHandle in place of the fstat calls in import.c.

We must probably also replace the stat() call (through _Py_stat) with GetFileAttributesEx, or make _Py_stat re-use GetFileAttributesEx.

----------
components: +Windows
nosy: +brian.curtin, pitrou, tim.golden
priority: normal -> high
stage:  -> needs patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:17:03 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 26 Jan 2012 14:17:03 +0000
Subject: [issue13863] import.c sometimes generates incorrect timestamps on
	Windows + NTFS
In-Reply-To: <1327519681.79.0.252690630069.issue13863@psf.upfronthosting.co.za>
Message-ID: <1327587423.51.0.0750053238363.issue13863@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:27:57 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Thu, 26 Jan 2012 14:27:57 +0000
Subject: [issue13875] cmd: no user documentation
Message-ID: <1327588077.77.0.0954132482021.issue13875@psf.upfronthosting.co.za>


New submission from anatoly techtonik <techtonik at gmail.com>:

http://docs.python.org/library/cmd.html#

Documentation for cmd module is poor to explain the value of this module to users. Intro is too abstract - phrase "simple framework for writing line-oriented command interpreters" doesn't mean much. Perhaps word "interactive" is missing?

So, there is no part explaining the what cmd does exactly (intro fails) and no part explaining the main principle - How exactly does this framework allows to do this in a simple way? (I guess reference part under 'Cmd objects -> Cmd.cmdloop([intro]) -> p[4]` does that, but it is not the place you'd usually expect this info.

At the very least what could be done is a link to Doug's tutorial http://www.doughellmann.com/PyMOTW/cmd/

----------
assignee: docs at python
components: Documentation
messages: 152010
nosy: docs at python, techtonik
priority: normal
severity: normal
status: open
title: cmd: no user documentation

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13875>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:32:20 2012
From: report at bugs.python.org (Stefan Krah)
Date: Thu, 26 Jan 2012 14:32:20 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <CAMpsgwYzuX3UGM=yb7Zt9GWD0PCZZVyhq+mvAHEH954ydJPz_w@mail.gmail.com>
Message-ID: <20120126143219.GA4927@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

STINNER Victor <report at bugs.python.org> wrote:
> Can you try x = (int *)1; instead of x = NULL; ?

This works. - I must say that I find this new behavior of clang slightly
dangerous...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:36:03 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 14:36:03 +0000
Subject: [issue13876] Sporadic failure in test_socket
Message-ID: <1327588563.2.0.447366914719.issue13876@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

When running the test suite, I occasionally get the following failure:

    ERROR: testRecvmsgEOF (test.test_socket.RecvmsgSCTPStreamTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/code/src/cpython/def/Lib/test/test_socket.py", line 2187, in testRecvmsgEOF
        msg, ancdata, flags, addr = self.doRecvmsg(self.serv_sock, 1024)
      File "/home/nadeem/code/src/cpython/def/Lib/test/test_socket.py", line 1678, in doRecvmsg
        result = sock.recvmsg(bufsize, *args)
    OSError: [Errno 107] Transport endpoint is not connected

The machine in question is running Ubuntu 11.10 64-bit.

I haven't seen anything similar on any of the buildbots.

----------
messages: 152012
nosy: nadeem.vawda
priority: normal
severity: normal
stage: needs patch
status: open
title: Sporadic failure in test_socket
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13876>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:37:52 2012
From: report at bugs.python.org (R. David Murray)
Date: Thu, 26 Jan 2012 14:37:52 +0000
Subject: [issue13875] cmd: no user documentation
In-Reply-To: <1327588077.77.0.0954132482021.issue13875@psf.upfronthosting.co.za>
Message-ID: <1327588672.51.0.23541383654.issue13875@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

Well, since it isn't limited to interactive use, I don't think 'interactive' is missing.

MOTW links are a more global issue that was discussion on python-dev (I forget the outcome, but I think it was "no").

I don't see anything that needs done here, but if you want to suggest a patch for consideration, feel free.

----------
nosy: +r.david.murray
priority: normal -> low
resolution:  -> works for me
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13875>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:40:44 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Thu, 26 Jan 2012 14:40:44 +0000
Subject: [issue13875] cmd: no user documentation
In-Reply-To: <1327588077.77.0.0954132482021.issue13875@psf.upfronthosting.co.za>
Message-ID: <1327588844.11.0.975778504111.issue13875@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

What do you mean by saying it is not limited for interactive use? I thought it is used to provide command prompt for typing commands. What other use cases does it support?

----------
status: pending -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13875>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:49:36 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 14:49:36 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <E1RqQdr-0002Ne-4r@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset be9d02536a81 by Meador Inge in branch '3.2':
- Issue #13840: Fix ctypes.create_string_buffer exception message and docs.
http://hg.python.org/cpython/rev/be9d02536a81

New changeset 52f68c95e025 by Meador Inge in branch 'default':
- Issue #13840: Fix ctypes.create_string_buffer exception message and docs.
http://hg.python.org/cpython/rev/52f68c95e025

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Thu Jan 26 15:50:55 2012
From: report at bugs.python.org (Meador Inge)
Date: Thu, 26 Jan 2012 14:50:55 +0000
Subject: [issue13840] create_string_buffer rejects str init_or_size parameter
In-Reply-To: <1327305188.04.0.956774664708.issue13840@psf.upfronthosting.co.za>
Message-ID: <1327589455.45.0.54283163018.issue13840@psf.upfronthosting.co.za>


Meador Inge <meadori at gmail.com> added the comment:

I just fixed the docs and error message for now.  I might revisit the ASCII decoding later.  Thanks for the bug report Vincent.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13840>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:15:39 2012
From: report at bugs.python.org (R. David Murray)
Date: Thu, 26 Jan 2012 15:15:39 +0000
Subject: [issue13875] cmd: no user documentation
In-Reply-To: <1327588077.77.0.0954132482021.issue13875@psf.upfronthosting.co.za>
Message-ID: <1327590939.66.0.234948217955.issue13875@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

You can feed a cmd driven interface from stdin or via cmd.onecmd.

However, I agree that the intended and primary use case is interactive.  There wouldn't be much point in using cmd if the primary intent of your program wasn't interactive.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13875>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:41:04 2012
From: report at bugs.python.org (Jon Bringhurst)
Date: Thu, 26 Jan 2012 15:41:04 +0000
Subject: [issue1003195] segfault when running smtplib example
Message-ID: <1327592464.73.0.17598601608.issue1003195@psf.upfronthosting.co.za>


Jon Bringhurst <jon at bringhurst.org> added the comment:

I just ran into this while using the smtplib example on:

2.6 (r26:66714, Jan 17 2012, 11:02:11)
GCC 4.1.2 20080704 (Red Hat 4.1.2-44)

Running the program simply gives a "Segmentation Fault (core dumped)"

Running it under gdb...

[Thread debugging using libthread_db enabled]
[New Thread 0x2b0257007b80 (LWP 2567)]

Program received signal SIGSEGV, Segmentation fault.
PyEval_EvalFrameEx (f=0x17701d00, throwflag=<value optimized out>) at Python/ceval.c:4342
4342        if (!PyDict_Check(gobals)) {

----------
nosy: +Jon.Bringhurst

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1003195>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:43:40 2012
From: report at bugs.python.org (Jon Bringhurst)
Date: Thu, 26 Jan 2012 15:43:40 +0000
Subject: [issue1003195] segfault when running smtplib example
Message-ID: <1327592620.56.0.724366371417.issue1003195@psf.upfronthosting.co.za>


Changes by Jon Bringhurst <jon at bringhurst.org>:


----------
type:  -> crash
versions: +Python 2.6 -Python 2.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1003195>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:46:45 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Thu, 26 Jan 2012 15:46:45 +0000
Subject: [issue8828] Atomic function to rename a file
In-Reply-To: <1274917301.93.0.503310941815.issue8828@psf.upfronthosting.co.za>
Message-ID: <1327592805.02.0.237444259005.issue8828@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Here is a patch adding os.replace().

----------
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24328/osreplace.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8828>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:50:10 2012
From: report at bugs.python.org (Jon Bringhurst)
Date: Thu, 26 Jan 2012 15:50:10 +0000
Subject: [issue13877] segfault when running smtplib example
Message-ID: <1327593010.34.0.461726513407.issue13877@psf.upfronthosting.co.za>


New submission from Jon Bringhurst <jon at bringhurst.org>:

I just ran into this while using the smtplib example on:

2.6 (r26:66714, Jan 17 2012, 11:02:11)
GCC 4.1.2 20080704 (Red Hat 4.1.2-44)

Running the program simply gives a "Segmentation Fault (core dumped)"

Running it under gdb...

[Thread debugging using libthread_db enabled]
[New Thread 0x2b0257007b80 (LWP 2567)]

Program received signal SIGSEGV, Segmentation fault.
PyEval_EvalFrameEx (f=0x17701d00, throwflag=<value optimized out>) at Python/ceval.c:4342
4342        if (!PyDict_Check(gobals)) {


See (permanently closed?) similar bug at:
http://bugs.python.org/issue1003195

----------
components: Interpreter Core
messages: 152020
nosy: Jon.Bringhurst
priority: normal
severity: normal
status: open
title: segfault when running smtplib example
type: crash
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13877>
_______________________________________

From report at bugs.python.org  Thu Jan 26 16:58:08 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Thu, 26 Jan 2012 15:58:08 +0000
Subject: [issue13877] segfault when running smtplib example
In-Reply-To: <1327593010.34.0.461726513407.issue13877@psf.upfronthosting.co.za>
Message-ID: <1327593488.27.0.734134497872.issue13877@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Do you have a more complete traceback by any chance?
Also, does the "New Thread..." message indicate that a new thread is created?  Why?  This is not what I see here.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13877>
_______________________________________

From report at bugs.python.org  Thu Jan 26 17:57:18 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 16:57:18 +0000
Subject: [issue13878] test_sched failures on Windows buildbot
Message-ID: <1327597038.78.0.0899603043053.issue13878@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/4072/steps/test/logs/stdio:

    FAIL: test_enter (test.test_sched.TestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_sched.py", line 18, in test_enter
        self.assertEqual(l, [0.01, 0.02, 0.03, 0.04, 0.05])
    AssertionError: Lists differ: [0.01, 0.02, 0.03, 0.05, 0.04] != [0.01, 0.02, 0.03, 0.04, 0.05]

    First differing element 3:
    0.05
    0.04

    - [0.01, 0.02, 0.03, 0.05, 0.04]
    ?                        ------

    + [0.01, 0.02, 0.03, 0.04, 0.05]
    ?                    ++++++


http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/4062/steps/test/logs/stdio:

    FAIL: test_queue (test.test_sched.TestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_sched.py", line 74, in test_queue
        self.assertEqual(list(scheduler.queue), [e1, e2, e3, e4, e5])
    AssertionError: Lists differ: [Event(time=1327366698.525, pr... != [Event(time=1327366698.525, pr...

    First differing element 3:
    Event(time=1327366698.565, priority=1, action=<function TestCase.test_queue.<locals>.<lambda> at 0x03419158>, argument=[], kwargs={})
    Event(time=1327366698.5709999, priority=1, action=<function TestCase.test_queue.<locals>.<lambda> at 0x03419158>, argument=[], kwargs={})

    Diff is 1268 characters long. Set self.maxDiff to None to see it.

----------
messages: 152022
nosy: giampaolo.rodola, nadeem.vawda
priority: normal
severity: normal
stage: needs patch
status: open
title: test_sched failures on Windows buildbot
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13878>
_______________________________________

From report at bugs.python.org  Thu Jan 26 17:58:20 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Thu, 26 Jan 2012 16:58:20 +0000
Subject: [issue13878] test_sched failures on Windows buildbot
In-Reply-To: <1327597038.78.0.0899603043053.issue13878@psf.upfronthosting.co.za>
Message-ID: <1327597100.77.0.72260994174.issue13878@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Oops, those links should be:
  http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/4072/steps/test/logs/stdio
and:
  http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/4062/steps/test/logs/stdio

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13878>
_______________________________________

From report at bugs.python.org  Thu Jan 26 17:59:28 2012
From: report at bugs.python.org (Tim Willis)
Date: Thu, 26 Jan 2012 16:59:28 +0000
Subject: [issue13879] Argparse does not support subparser aliases in 2.7
Message-ID: <1327597168.42.0.828007379667.issue13879@psf.upfronthosting.co.za>


New submission from Tim Willis <schadenfreude328 at gmail.com>:

Argparse documentation in 2.7 indicates support for an 'aliases' kwarg.  (Fourth example down from http://docs.python.org/dev/library/argparse.html#sub-commands)

While aliases work as expected in 3.2, use in 2.7 results in TypeError: __init__() got an unexpected keyword argument 'aliases'

----------
components: Library (Lib)
messages: 152024
nosy: Tim.Willis
priority: normal
severity: normal
status: open
title: Argparse does not support subparser aliases in 2.7
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13879>
_______________________________________

From report at bugs.python.org  Thu Jan 26 18:09:27 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Thu, 26 Jan 2012 17:09:27 +0000
Subject: [issue13876] Sporadic failure in test_socket
In-Reply-To: <1327588563.2.0.447366914719.issue13876@psf.upfronthosting.co.za>
Message-ID: <1327597767.78.0.390590641472.issue13876@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13876>
_______________________________________

From report at bugs.python.org  Thu Jan 26 18:10:49 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 17:10:49 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <20120126143219.GA4927@sleipnir.bytereef.org>
Message-ID: <CAMpsgwbaQBFjkLGvDrYEzO=cZj2xC3uoup=XApptzLqzmY6t=Q@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

There was a similar bug which was declared as a vulnerability in the
Linux kernel:
http://isc.sans.edu/diary.html?storyid=6820

GCC has an option to disable this optimization: -fno-delete-null-pointer-checks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Thu Jan 26 19:25:43 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Thu, 26 Jan 2012 18:25:43 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327602343.28.0.0909085969522.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

After more than a year the patch is finally made. Can someone please applies this patch and closes this issue report? Thanks.

----------
keywords: +patch
versions: +Python 3.3
Added file: http://bugs.python.org/file24329/issue10580.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Thu Jan 26 20:34:28 2012
From: report at bugs.python.org (__KFL__)
Date: Thu, 26 Jan 2012 19:34:28 +0000
Subject: [issue13880] pydoc -k throws "AssertionError: distutils has already
	been patched by <class py2exe.Distribution at 0x0000000005987408>"
Message-ID: <1327606468.69.0.0708132604727.issue13880@psf.upfronthosting.co.za>


New submission from __KFL__ <lukefei at gmail.com>:

Pydoc fails on the following exception. There is a mail discussing it: http://mail.python.org/pipermail/python-list/2009-December/1230790.html

C:\Python27\Lib>pydoc -k file
...
...
Traceback (most recent call last):
  File "C:\Python27\Lib\pydoc.py", line 2338, in <module>
    if __name__ == '__main__': cli()
  File "C:\Python27\Lib\pydoc.py", line 2277, in cli
    apropos(val)
  File "C:\Python27\Lib\pydoc.py", line 1974, in apropos
    ModuleScanner().run(callback, key)
  File "C:\Python27\Lib\pydoc.py", line 1939, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "C:\Python27\Lib\pkgutil.py", line 110, in walk_packages
    __import__(name)
  File "C:\Python27\lib\site-packages\pypm\__init__.py", line 22, in <module>
    import setuptools
  File "C:\Python27\lib\site-packages\setuptools\__init__.py", line 2, in <module>
    from setuptools.extension import Extension, Library
  File "C:\Python27\lib\site-packages\setuptools\extension.py", line 2, in <module>
    from setuptools.dist import _get_unpatched
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 28, in <module>
    _Distribution = _get_unpatched(_Distribution)
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 24, in _get_unpatched
    "distutils has already been patched by %r" % cls
AssertionError: distutils has already been patched by <class py2exe.Distribution at 0x0000000005987408>

----------
messages: 152027
nosy: __KFL__
priority: normal
severity: normal
status: open
title: pydoc -k throws "AssertionError: distutils has already been patched by <class py2exe.Distribution at 0x0000000005987408>"
type: crash
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13880>
_______________________________________

From report at bugs.python.org  Thu Jan 26 20:37:12 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Thu, 26 Jan 2012 19:37:12 +0000
Subject: [issue13455] Reorganize tracker docs in the devguide
In-Reply-To: <1321984554.54.0.834071786299.issue13455@psf.upfronthosting.co.za>
Message-ID: <1327606632.47.0.0409666774317.issue13455@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Antoine, what reference other than the devguide are you referring to?
If you keep info I need out of the devguide, where am I supposed to find it?  python.org/dev now redirects to python.org/devguide.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13455>
_______________________________________

From report at bugs.python.org  Thu Jan 26 20:40:49 2012
From: report at bugs.python.org (Andrew McNabb)
Date: Thu, 26 Jan 2012 19:40:49 +0000
Subject: [issue13881] Stream encoder for zlib_codec doesn't use the
	incremental encoder
Message-ID: <1327606849.1.0.885842099772.issue13881@psf.upfronthosting.co.za>


New submission from Andrew McNabb <amcnabb at mcnabbs.org>:

The stream encoder for the zlib_codec doesn't use the incremental encoder, so it has limited usefulness in practice. This is easiest to show with an example.

Here is the behavior with the stream encoder:

>>> filelike = io.BytesIO()
>>> wrapped = codecs.getwriter('zlib_codec')(filelike)
>>> wrapped.write(b'hello')
>>> filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00y'
>>> wrapped.write(b'x')
>>> filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00yx\x9c\xab\x00\x00\x00y\x00y'
>>>

However, this is the behavior of the incremental encoder:

>>> ienc = codecs.getincrementalencoder('zlib_codec')()
>>> ienc.encode(b'x')
b'x\x9c'
>>> ienc.encode(b'x', final=True)
b'\xab\xa8\x00\x00\x01j\x00\xf1'
>>>

The stream encoder is apparently encoding each write as an individual block, but the incremental encoder buffers until it gets a block that's large enough to be meaningfully compressed.

Fixing this may require addressing a separate issue with stream encoders. Unlike with the GzipFile module, closing a stream encoder closes the underlying file. If this underlying file is a BytesIO, then closing makes it free its buffer, making it impossible to get at the completed file.

----------
components: IO
messages: 152029
nosy: amcnabb
priority: normal
severity: normal
status: open
title: Stream encoder for zlib_codec doesn't use the incremental encoder
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13881>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:00:16 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 26 Jan 2012 21:00:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327611616.77.0.49715270308.issue13703@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

I'd like to propose an entirely different approach: use AVL trees for colliding strings, for dictionaries containing only unicode or byte strings.

A prototype for this is in http://hg.python.org/sandbox/loewis/branches#avl
It is not fully working yet, but I'm now confident that this is a feasible approach.

It has the following advantages over the alternatives:
- performance in case of collisions is O(log(N)), where N is the number of colliding keys
- no new exceptions are raised, except for MemoryError if it runs out of memory for allocating nodes in the tree
- the hash values do not change
- the dictionary order does not change as long as no keys collide on hash values (which for all practical purposes should mean that the dictionary order does not change in all places where it matters)

----------
nosy: +loewis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:00:40 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 21:00:40 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
Message-ID: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>


New submission from STINNER Victor <victor.stinner at haypocalc.com>:

Attached patch adds an optional format argument to time.time(), time.clock(), time.wallclock(), time.clock_gettime() and time.clock_getres() to get the timestamp as a different format. By default, the float type is still used, but it will be possible to pass "format" to get the timestamp as a decimal.Decimal object.

Some advantages of using decimal.Decimal instead of float:
 - no loss of precision during conversion from base 2 to base 10 (converting the float to a string)
 - the resolution of the clock is stored in the Decimal object
 - for big number, Decimal doesn't loose precision

Using Decimal is also motivated by the fact than Python 3.3 has now access to clock having a resolution of 1 nanosecond: clock_gettime(CLOCK_REALTIME). Well, it doesn't mean that the clock is accurate, but Python should not loose precision just because it uses floatting point instead of integers.

About the API: I chose to add a string argument to allow maybe later to support user defined formats, or at least add new builtin formats. For example, someone proposed 128 bits float in the issue #11457.

--

The internal format is:

typedef struct {
    time_t seconds;
    /* floatpart can be zero */
    size_t floatpart;
    /* divisor cannot be zero */
    size_t divisor;
    /* log10 of the clock resoltion, the real resolution is 10^resolution,
       resolution is in range [-9; 0] */
    int resolution;
} _PyTime_t;

I don't know if size_t is big enough to store any "floatpart" value: time.clock() uses a LARGE_INTEGER internally. I tested my patch on Linux 32 and 64 bits, but not yet on Windows.

The internal function encoding a timestamp to Decimal caches the resolution objects (10^resolution, common clock resolutions: 10^-6, 10^-9) in a dict, and a Context object.

I computed that 22 decimal digits should be enough to compute a timestamp of 10,000 years. Extract of my patch:

"Use 12 decimal digits to store 10,000 years in seconds + 9 decimal digits for the floating part in nanoseconds + 1 decimal digit to round correctly"

>>> str(int(3600*24*365.25*10000))
'315576000000'
>>> len(str(int(3600*24*365.25*10000)))
12

--

See also the issue #11457 which is linked to this topic, but not exactly the same because it concerns a low level function (os.stat()).

----------
components: Library (Lib)
files: time_decimal-2.patch
keywords: patch
messages: 152031
nosy: belopolsky, haypo, loewis
priority: normal
severity: normal
status: open
title: Add format argument for time.time(), time.clock(), ... to get a timestamp as a Decimal object
versions: Python 3.3
Added file: http://bugs.python.org/file24330/time_decimal-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:04:27 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 21:04:27 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327611867.42.0.650618717793.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Some examples of the API:

$ ./python 
Python 3.3.0a0 (default:52f68c95e025+, Jan 26 2012, 21:54:31) 
>>> import time
>>> time.time()
1327611705.948446
>>> time.time('decimal')
Decimal('1327611708.988419')
>>> t1=time.time('decimal'); t2=time.time('decimal'); t2-t1
Decimal('0.000550')
>>> t1=time.time('float'); t2=time.time('float'); t2-t1
5.9604644775390625e-06
>>> time.clock_gettime(time.CLOCK_MONOTONIC, 'decimal')
Decimal('1211833.389740312')
>>> time.clock_getres(time.CLOCK_MONOTONIC, 'decimal')
Decimal('1E-9')
>>> time.clock()
0.12
>>> time.clock('decimal')
Decimal('0.120000')

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:04:28 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Thu, 26 Jan 2012 21:04:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327611616.77.0.49715270308.issue13703@psf.upfronthosting.co.za>
Message-ID: <CAFRnB2UAcLjHzZ7NJDnzHAU-Gw9urb5Oc5CF==WPKfQEd8q=Lw@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

On Thu, Jan 26, 2012 at 4:00 PM, Martin v. L?wis <report at bugs.python.org>wrote:

>
> Martin v. L?wis <martin at v.loewis.de> added the comment:
>
> I'd like to propose an entirely different approach: use AVL trees for
> colliding strings, for dictionaries containing only unicode or byte strings.
>
> A prototype for this is in
> http://hg.python.org/sandbox/loewis/branches#avl
> It is not fully working yet, but I'm now confident that this is a feasible
> approach.
>
> It has the following advantages over the alternatives:
> - performance in case of collisions is O(log(N)), where N is the number of
> colliding keys
> - no new exceptions are raised, except for MemoryError if it runs out of
> memory for allocating nodes in the tree
> - the hash values do not change
> - the dictionary order does not change as long as no keys collide on hash
> values (which for all practical purposes should mean that the dictionary
> order does not change in all places where it matters)
>
> ----------
> nosy: +loewis
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

Martin,

What happens if, instead of putting strings in a dictionary directly, I
have them wrapped in something.  For example, the classes Antoine and I
pasted early.  These define hash and equal as being strings, but don't have
an ordering.

Alex

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:18:20 2012
From: report at bugs.python.org (Brett Cannon)
Date: Thu, 26 Jan 2012 21:18:20 +0000
Subject: [issue13883] PYTHONCASEOK docs mistakenly says it is limited to
	Windows
Message-ID: <1327612700.31.0.384231885295.issue13883@psf.upfronthosting.co.za>


New submission from Brett Cannon <brett at python.org>:

The docs for PYTHONCASEOK say it is limited to Windows, but in actuality in Python 3.x it applies to both Windows (including cygwin) and OS X. On Python 2.7 it applies to those plus RISCOS OS/2.

----------
assignee: docs at python
components: Documentation
messages: 152034
nosy: brett.cannon, docs at python
priority: normal
severity: normal
stage: needs patch
status: open
title: PYTHONCASEOK docs mistakenly says it is limited to Windows
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13883>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:50:14 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 21:50:14 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327614614.37.0.0767277731871.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Windows code (win32_clock) was wrong in time_decimal-2.patch: it is fixed in patch version 3.

Some tests on Windows made me realize that time.time() has a resolution of 1 millisecond (10^-3) and not of a microsecond (10^-6) on Windows! It is time to use GetSystemTimeAsFileTime! => see issue #13845.

----------
Added file: http://bugs.python.org/file24331/time_decimal-3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Thu Jan 26 22:51:21 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 21:51:21 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327614681.59.0.710595649518.issue13845@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Using the patch of #13882, I realize that time.time() has a resolution of 1 millisecond (10^-3) and not of a microsecond (10^-6) on Windows! Windows doesn't provide gettimeofday(). Using GetSystemTimeAsFileTime() would provide a much better resolution!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:13:20 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Thu, 26 Jan 2012 22:13:20 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2UAcLjHzZ7NJDnzHAU-Gw9urb5Oc5CF==WPKfQEd8q=Lw@mail.gmail.com>
Message-ID: <1327615964.2219.35.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Thu, 2012-01-26 at 21:04 +0000, Alex Gaynor wrote:
> Alex Gaynor <alex.gaynor at gmail.com> added the comment:
> 
> On Thu, Jan 26, 2012 at 4:00 PM, Martin v. L?wis <report at bugs.python.org>wrote:
> 
> >
> > Martin v. L?wis <martin at v.loewis.de> added the comment:
> >
> > I'd like to propose an entirely different approach: use AVL trees for
> > colliding strings, for dictionaries containing only unicode or byte strings.
> >
> > A prototype for this is in
> > http://hg.python.org/sandbox/loewis/branches#avl
> > It is not fully working yet, but I'm now confident that this is a feasible
> > approach.
> >
> > It has the following advantages over the alternatives:
> > - performance in case of collisions is O(log(N)), where N is the number of
> > colliding keys
> > - no new exceptions are raised, except for MemoryError if it runs out of
> > memory for allocating nodes in the tree
> > - the hash values do not change
> > - the dictionary order does not change as long as no keys collide on hash
> > values (which for all practical purposes should mean that the dictionary
> > order does not change in all places where it matters)
> >
> > ----------
> > nosy: +loewis
> >
> > _______________________________________
> > Python tracker <report at bugs.python.org>
> > <http://bugs.python.org/issue13703>
> > _______________________________________
> >
> 
> Martin,
> 
> What happens if, instead of putting strings in a dictionary directly, I
> have them wrapped in something.  For example, the classes Antoine and I
> pasted early.  These define hash and equal as being strings, but don't have
> an ordering.

[Obviously I'm not Martin, but his idea really interests me]

Looking at:
http://hg.python.org/sandbox/loewis/file/58be269aa0b1/Objects/dictobject.c#l517
as soon as any key insertion or lookup occurs where the key isn't
exactly one of the correct types, the dict flattens any AVL trees back
into the regular flat representation (and switches to lookdict for
ma_lookup), analogous to the existing ma_lookup transition on dicts.

>From my reading of the code, if you have a dict purely of bytes/str,
collisions on a hash value lead to the PyDictEntry's me_key being set to
an AVL tree.  All users of the ma_lookup callback within dictobject.c
check to see if they're getting such PyDictEntry back.  If they are,
they call into the tree, which leads to TREE_FIND(), TREE_INSERT() and
TREE_DELETE() invocations as appropriate; ultimately, the AVL macros
call back to within node_cmp():
   PyObject_Compare(left->key, right->key)

[Martin, I'm sorry if I got this wrong]

So *if* I'm reading the code correctly, it might be possible to
generalize it from {str, bytes} to any set of types within which
ordering and equality checking of instances from any type is "sane",
which loosely, would seem to be: that we can reliably compare all
objects from any type within the set, so that we can use the comparisons
to perform a search to hone in on a pair of keys that compare as
"equal", without any chance of raising exceptions, or missing a valid
chance for two objects to be equal etc.

I suspect that you can't plug arbitrary user-defined types into it,
since there's no way to guarantee that ordering and comparison work in
the ways that the AVL lookup requires.

But I could be misreading Martin's code.  [thinking aloud, if a pair of
objects don't implement comparison at the PyObject_Compare level, is it
possible to instead simply compare the addresses of the objects?  I
don't think so, since you have a custom equality implementation in your
UDT, but maybe I've missed something?]

Going higher-level, I feel that there are plenty of attacks against
pure-str/bytes dicts, and having protection against them is worthwhile -
even if there's no direct way to use it to protect the use-case you
describe.

Hope this is helpful
Dave

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:17:29 2012
From: report at bugs.python.org (Stephen Kelly)
Date: Thu, 26 Jan 2012 22:17:29 +0000
Subject: [issue13666] datetime documentation typos
In-Reply-To: <1324936913.94.0.649454454071.issue13666@psf.upfronthosting.co.za>
Message-ID: <1327616249.19.0.408812843319.issue13666@psf.upfronthosting.co.za>


Stephen Kelly <steveire at gmail.com> added the comment:

There are actually other bugs in the same code example:

...     def __init__(self):         # DST starts last Sunday in March
...         d = datetime(dt.year, 4, 1)   # ends last Sunday in October
...         self.dston = d - timedelta(days=d.weekday() + 1)
...         d = datetime(dt.year, 11, 1)
...         self.dstoff = d - timedelta(days=d.weekday() + 1)

where does dt come from? this fragment should be in the implementation of dst (in both the GMT1 and GMT2 classes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13666>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:34:52 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 26 Jan 2012 22:34:52 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327615964.2219.35.camel@surprise>
Message-ID: <20120126233450.Horde.fi7KWVNNcXdPIdUKkOc26PA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> as soon as any key insertion or lookup occurs where the key isn't
> exactly one of the correct types, the dict flattens any AVL trees back
> into the regular flat representation (and switches to lookdict for
> ma_lookup), analogous to the existing ma_lookup transition on dicts.

Correct.

> TREE_DELETE() invocations as appropriate; ultimately, the AVL macros
> call back to within node_cmp():
>    PyObject_Compare(left->key, right->key)

Correct.

> I suspect that you can't plug arbitrary user-defined types into it,
> since there's no way to guarantee that ordering and comparison work in
> the ways that the AVL lookup requires.

That's all true. It would be desirable to automatically determine which
types also support total order in addition to hashing, alas, there is
currently no protocol for it. On the contrary, Python has moved away
of assuming that all objects form a total order.

> [thinking aloud, if a pair of
> objects don't implement comparison at the PyObject_Compare level, is it
> possible to instead simply compare the addresses of the objects?

2.x has an elaborate logic to provide a total order on objects. It
took the entire 1.x and 2.x series to fix issues with that order, only
to recognize that it is not feasible to provide one - hence the introduction
of rich comparisons and the omission of cmp in 3.x.

For the dictionary, using addresses does not work: the order of objects
needs to be consistent with equality (i.e. x < y and x == y must not
simultaneously hold, as must x == y and y < z imply that also x < z,
else the tree lookup won't find the equal keys).

> Going higher-level, I feel that there are plenty of attacks against
> pure-str/bytes dicts, and having protection against them is worthwhile -
> even if there's no direct way to use it to protect the use-case you
> describe.

Indeed. This issue doesn't need to fix *all* possible attacks using
hash collisions. Instead, it needs to cover the common case, and it
needs to allow users to rewrite their code so that they can protect
it against this family of attacks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:42:25 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 26 Jan 2012 22:42:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2UAcLjHzZ7NJDnzHAU-Gw9urb5Oc5CF==WPKfQEd8q=Lw@mail.gmail.com>
Message-ID: <20120126234223.Horde.hFEABFNNcXdPIdbP4pFm62A@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> What happens if, instead of putting strings in a dictionary directly, I
> have them wrapped in something.  For example, the classes Antoine and I
> pasted early.  These define hash and equal as being strings, but don't have
> an ordering.

As Dave has analysed: the dictionary falls back to the current implementation.
So wrt. your question "Is it still able to find the value?", the answer is

Yes, certainly. It's fully backwackwards compatible, with the limitation
in msg152030 (i.e. the dictionary order may change for dictionaries with
string keys colliding in their hash() values).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:43:58 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Thu, 26 Jan 2012 22:43:58 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120126234223.Horde.hFEABFNNcXdPIdbP4pFm62A@webmail.df.eu>
Message-ID: <CAFRnB2V+MkumT1i4+KpC-T=Zyt5ABZRUdKhnwoCLR5q8njRh5w@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

On Thu, Jan 26, 2012 at 5:42 PM, Martin v. L?wis <report at bugs.python.org>wrote:

>
> Martin v. L?wis <martin at v.loewis.de> added the comment:
>
> > What happens if, instead of putting strings in a dictionary directly, I
> > have them wrapped in something.  For example, the classes Antoine and I
> > pasted early.  These define hash and equal as being strings, but don't
> have
> > an ordering.
>
> As Dave has analysed: the dictionary falls back to the current
> implementation.
> So wrt. your question "Is it still able to find the value?", the answer is
>
> Yes, certainly. It's fully backwackwards compatible, with the limitation
> in msg152030 (i.e. the dictionary order may change for dictionaries with
> string keys colliding in their hash() values).
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

But using non-__builtin__.str objects (such as UserString) would expose the
user to an attack?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Thu Jan 26 23:49:54 2012
From: report at bugs.python.org (Ethan Furman)
Date: Thu, 26 Jan 2012 22:49:54 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327618194.01.0.140632280457.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Okay, here is a patch implementing the 'raise ... from None' method.

All critiques appreciated (especially if they include links to learn from!).

----------
keywords: +patch
Added file: http://bugs.python.org/file24332/raise_from_none.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:03:35 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 26 Jan 2012 23:03:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2V+MkumT1i4+KpC-T=Zyt5ABZRUdKhnwoCLR5q8njRh5w@mail.gmail.com>
Message-ID: <20120127000334.Horde.aM5EaVNNcXdPIdvGl__HGMA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> But using non-__builtin__.str objects (such as UserString) would expose the
> user to an attack?

Not necessarily: only if they use these strings as dictionary keys, and only
if they do so in contexts where arbitrary user input is consumed. In these
cases, users need to rewrite their code to replace the keys. Using dictionary
wrappers (such as UserDict), this is possible using only local changes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:08:51 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 23:08:51 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <E1RqYR0-00072M-BF@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9ee4a104e33d by Victor Stinner in branch 'default':
Issue #13847: time.localtime() and time.gmtime() now raise an OSError instead
http://hg.python.org/cpython/rev/9ee4a104e33d

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:12:14 2012
From: report at bugs.python.org (Ethan Furman)
Date: Thu, 26 Jan 2012 23:12:14 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327619534.66.0.808045331801.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

I went with

    raise ... from None

instead of 

    raise as ...

because there seemed to me more support for 'from None' on python-dev, possible confusion with 'raise as', and 'from None' follows the existing systax of

    raise SomeError() from SomeOtherError()

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:22:32 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Thu, 26 Jan 2012 23:22:32 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120127000334.Horde.aM5EaVNNcXdPIdvGl__HGMA@webmail.df.eu>
Message-ID: <CAFRnB2VH2m6h3C9q-1k_0A4gG8KYfKc=vCoL+ygv4ZJ_UYJkmQ@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

I'm sorry then, but I'm a little confused.  I think we pretty clearly
established earlier that requiring users to make changes anywhere they
stored user data would be dangerous, because these locations are often in
libraries or other places where the code creating and modifying the
dictionary has no idea it's user data in it.

The proposed AVL solution fails if it requires users to fundamentally
restructure their data depending on it's origin.

We have solution that is known to work in all cases: hash randomization.
 There were three discussed issues with it:

a) Code assuming a stable ordering to dictionaries
b) Code assuming hashes were stable across runs.
c) Code reimplementing the hashing algorithm of a core datatype that is now
randomized.

I don't think any of these are realistic issues the way "doesn't protect
all cases" is.  (a) was never a documented, or intended property, indeed it
breaks all the time, if you insert keys in the wrong order, use a different
platform, or anything else can change.  (b) For the same reasons code
relying on (b) only worked if you didn't change anything, and in practice
I'm convinced neither of these were common (if ever existed).  Finally (c),
while it's a concern, I've reviewed Django, SQLAlchemy, PyPy, and the
stdlib: there is only one place where compatibility with a core-hash is
attempted, decimal.Decimal.

In summary, I think the case against hash-randomization has been seriously
overstated, and in no way is more dangerous than having a solution that
fails to solve the problem comprehensively.  Further, I think it is
imperative that we reach a consensus on this quickly, as the only reason
this hasn't been widely exploited yet is the lack of availability of the
data, when it becomes available I firmly expect just about every high
profile Python site on the internet (of which there are many) to be
attacked.

On Thu, Jan 26, 2012 at 6:03 PM, Martin v. L?wis <report at bugs.python.org>wrote:

>
> Martin v. L?wis <martin at v.loewis.de> added the comment:
>
> > But using non-__builtin__.str objects (such as UserString) would expose
> the
> > user to an attack?
>
> Not necessarily: only if they use these strings as dictionary keys, and
> only
> if they do so in contexts where arbitrary user input is consumed. In these
> cases, users need to rewrite their code to replace the keys. Using
> dictionary
> wrappers (such as UserDict), this is possible using only local changes.
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13703>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:30:48 2012
From: report at bugs.python.org (Torsten Landschoff)
Date: Thu, 26 Jan 2012 23:30:48 +0000
Subject: [issue13873] SIGBUS in test_zlib on Debian bigmem buildbot
In-Reply-To: <1327583804.29.0.71599909936.issue13873@psf.upfronthosting.co.za>
Message-ID: <1327620648.49.0.0408522885586.issue13873@psf.upfronthosting.co.za>


Torsten Landschoff <t.landschoff at gmx.net> added the comment:

I tried to reproduce this crash on my desktop system.
AMD64, 8 GB RAM (only) and on Debian unstable from today.
Testing the exact same Python version (hg update d2cf8a34ddf90fb1bc8938de0f736694e61f73fa) the test passes just fine here...

----------
nosy: +torsten

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13873>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:37:21 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 23:37:21 +0000
Subject: [issue13883] PYTHONCASEOK docs mistakenly says it is limited to
	Windows
In-Reply-To: <1327612700.31.0.384231885295.issue13883@psf.upfronthosting.co.za>
Message-ID: <E1RqYsT-0001Lz-80@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 524795e8abe1 by Brett Cannon in branch '3.2':
Issue #13883: Document all platforms PYTHONCASEOK works on.
http://hg.python.org/cpython/rev/524795e8abe1

New changeset 0f00010c88f0 by Brett Cannon in branch 'default':
Issue #13883: PYTHONCASEOK also works with OS X.
http://hg.python.org/cpython/rev/0f00010c88f0

New changeset f18662285248 by Brett Cannon in branch '2.7':
Issue #13883: Document all platforms PYTHONCASEOK works on.
http://hg.python.org/cpython/rev/f18662285248

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13883>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:37:55 2012
From: report at bugs.python.org (Brett Cannon)
Date: Thu, 26 Jan 2012 23:37:55 +0000
Subject: [issue13883] PYTHONCASEOK docs mistakenly says it is limited to
	Windows
In-Reply-To: <1327612700.31.0.384231885295.issue13883@psf.upfronthosting.co.za>
Message-ID: <1327621075.27.0.507127819578.issue13883@psf.upfronthosting.co.za>


Changes by Brett Cannon <brett at python.org>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13883>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:39:23 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 23:39:23 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <E1RqYuY-0001YL-EG@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 94b7eb09d0b3 by Victor Stinner in branch 'default':
Issue #13847: time.clock() now raises a RuntimeError if the processor time used
http://hg.python.org/cpython/rev/94b7eb09d0b3

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:41:04 2012
From: report at bugs.python.org (Tim McGreevy)
Date: Thu, 26 Jan 2012 23:41:04 +0000
Subject: [issue13884] IDLE 2.6.5 Recent Files undocks
Message-ID: <1327621264.5.0.587291207239.issue13884@psf.upfronthosting.co.za>


New submission from Tim McGreevy <mcgrete.drs at gmail.com>:

When selecting from menu:
File --> Recent Files
the 'Recent Files' dropdown list undocks from the IDLE gui / File dropdown list.  Even after selecting a past file, it remains open until closed manually.

Ubuntu LUCID
amd64
IDLE 2.6.5
TK version 8.5
All installed using Ubuntu supported PPA (Synaptic)

After closing 'Recent Files' subwindow, problem no longer persists until IDLE is terminated and restarted.

----------
components: IDLE
files: IDLE2.6.5_RecentFilesUndocked.png
messages: 152050
nosy: mcgrete
priority: normal
severity: normal
status: open
title: IDLE 2.6.5 Recent Files undocks
versions: Python 2.6
Added file: http://bugs.python.org/file24333/IDLE2.6.5_RecentFilesUndocked.png

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13884>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:43:51 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Thu, 26 Jan 2012 23:43:51 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAFRnB2VH2m6h3C9q-1k_0A4gG8KYfKc=vCoL+ygv4ZJ_UYJkmQ@mail.gmail.com>
Message-ID: <20120127004349.Horde.PVOKDFNNcXdPIeU1WyDHUCA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> I'm sorry then, but I'm a little confused.  I think we pretty clearly
> established earlier that requiring users to make changes anywhere they
> stored user data would be dangerous, because these locations are often in
> libraries or other places where the code creating and modifying the
> dictionary has no idea it's user data in it.

I don't consider that established for the specific case of string-like
objects. Users can easily determine whether they use string-like objects,
and if so, in what places, and what data gets put into them.

> The proposed AVL solution fails if it requires users to fundamentally
> restructure their data depending on it's origin.

It doesn't fail at all. User don't *have* to restructure their code,
let alone fundamentally. Their code may currently be vulnerable, yet
not use string-like objects at all. With the proposed solution, such
code will be fixed for good.

It's true that the solution does not fix all cases of the vulnerability,
but neither does any other proposed solution.

> We have solution that is known to work in all cases: hash randomization.

Well, you *believe* that it fixes the problem, even though it actually
may not, assuming an attacker can somehow reproduce the hash function.

>  There were three discussed issues with it:
>
> a) Code assuming a stable ordering to dictionaries
> b) Code assuming hashes were stable across runs.
> c) Code reimplementing the hashing algorithm of a core datatype that is now
> randomized.
>
> I don't think any of these are realistic issues

I'm fairly certain that code will break in massive ways, despite any
argumentation that it should not. The question really is

Do we break code in a massive way, or do we fix the vulnerability
for most users with no code breakage?

I clearly value compatibility much higher than 100% protection against
a DoS-style attack (which has many other forms of protecting against
available also).

> (a) was never a documented, or intended property, indeed it
> breaks all the time, if you insert keys in the wrong order, use a different
> platform, or anything else can change.

Still, a lot of code relies on dictionary order, and successfully so,
in practice. Practicality beats purity.

> (b) For the same reasons code
> relying on (b) only worked if you didn't change anything

That's not true. You cannot practically change the way string hashing works
other than by changing the interpreter source. Hashes *are* currently stable
across runs.

> and in practice I'm convinced neither of these were common (if ever existed).

Are you willing to bet the trust people have in Python's bug fix policies
on that? I'm not.

> In summary, I think the case against hash-randomization has been seriously
> overstated, and in no way is more dangerous than having a solution that
> fails to solve the problem comprehensively.  Further, I think it is
> imperative that we reach a consensus on this quickly

Well, I cannot be part of a consensus that involves massive code breakage
in a bug fix release. Lacking consensus, either the release managers or
the BDFL will have to pronounce.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:44:17 2012
From: report at bugs.python.org (STINNER Victor)
Date: Thu, 26 Jan 2012 23:44:17 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <1327621457.05.0.242576778502.issue13847@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I added tests on localtime() and clock().

I read more carefully time(), ftime() and gettimeofday() manpage: it is not possible that they fail if the argument is an invalid pointer, the current code is correct.

I don't want to backport changes because they are incompatible.

--

There is a failure on FreeBSD 8.2:


======================================================================
FAIL: test_localtime_failure (test.test_time.TimeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_time.py", line 358, in test_localtime_failure
    self.assertRaises(OSError, time.localtime, invalid_time_t)
AssertionError: OSError not raised by localtime

http://www.python.org/dev/buildbot/all/builders/AMD64%20FreeBSD%208.2%203.x/builds/1746/steps/test/logs/stdio

test_localtime_failure() should be removed or at least skipped on FreeBSD, except if someone knows another invalid time_t value on this platform.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:50:34 2012
From: report at bugs.python.org (Roundup Robot)
Date: Thu, 26 Jan 2012 23:50:34 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <E1RqZ5M-0002uw-OM@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 516d42a6e518 by Victor Stinner in branch 'default':
Issue #13847: Fix test_mktime(), time.localtime() now raises OSError
http://hg.python.org/cpython/rev/516d42a6e518

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 00:58:21 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Thu, 26 Jan 2012 23:58:21 +0000
Subject: [issue13884] IDLE 2.6.5 Recent Files undocks
In-Reply-To: <1327621264.5.0.587291207239.issue13884@psf.upfronthosting.co.za>
Message-ID: <1327622301.0.0.832711321398.issue13884@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Could you try with IDLE 2.7/3.2?

----------
nosy: +ezio.melotti, terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13884>
_______________________________________

From report at bugs.python.org  Fri Jan 27 01:03:32 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 00:03:32 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <E1RqZHs-00048a-O7@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 856f0864437a by Victor Stinner in branch 'default':
Issue #13847: Make test_localtime_failure() more robust
http://hg.python.org/cpython/rev/856f0864437a

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 01:04:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 27 Jan 2012 00:04:15 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <1327622655.67.0.366451376708.issue13847@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

The issue should be done with the last commit.

----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 02:19:15 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 01:19:15 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120127004349.Horde.PVOKDFNNcXdPIeU1WyDHUCA@webmail.df.eu>
Message-ID: <1327627021.3454.11.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> >  There were three discussed issues with it:
> >
> > a) Code assuming a stable ordering to dictionaries
> > b) Code assuming hashes were stable across runs.
> > c) Code reimplementing the hashing algorithm of a core datatype that is now
> > randomized.
> >
> > I don't think any of these are realistic issues
> 
> I'm fairly certain that code will break in massive ways, despite any
> argumentation that it should not. The question really is
> 
> Do we break code in a massive way, or do we fix the vulnerability
> for most users with no code breakage?
> 
> I clearly value compatibility much higher than 100% protection against
> a DoS-style attack (which has many other forms of protecting against
> available also).

If I your read patch correctly, collisions will produce additional
allocations of one distinct PyObject (i.e. AVL node) per colliding key.
That's a pretty massive change in memory consumption for string dicts
(and also in memory fragmentation and cache friendliness, probably). The
performance effect in most situations is likely to be negative too,
despite the better worst-case complexity.

IMO that would be a rather controversial change for a feature release,
let alone a bugfix or security release.

It would be nice to have the release managers' opinions on this issue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 02:55:03 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 01:55:03 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327629303.27.0.324384792427.issue6210@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

1. Any syntax change requires a PEP (and, IMO, any such PEP for this issue should get rejected: I don't consider this an important enough feature to deserve dedicated syntax. Others disagree, which is one of the reasons why a PEP is needed. The other, more important, reason is to ensure the new syntax is spec'ed out clearly and incorporated into the language reference for the benefit of other implementations in the event that it *does* get approved)

2. A change that *doesn't* need a PEP is to just adjust the implicit exception chaining such that __context__ doesn't get set automatically if it has already been set explicitly (it turns out handling this situation was an open question in PEP 3134, not a specificied behaviour). That way, this task can be handled using a utility function:

    def no_context(new_exc):
        new_exc.__context__ = None
        return new_exc

    def doXY ():
        # ...
        try:
            page = urlopen( someRequest )
        except urllib.error.URLError as e:
            raise no_context(MyError( 'while doing XY', e ))
        # ...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 03:10:40 2012
From: report at bugs.python.org (tom kel)
Date: Fri, 27 Jan 2012 02:10:40 +0000
Subject: [issue13651] Improve redirection in urllib
In-Reply-To: <1324577379.51.0.5468672041.issue13651@psf.upfronthosting.co.za>
Message-ID: <1327630240.0.0.547627076459.issue13651@psf.upfronthosting.co.za>


tom kel <tomkel01 at gmail.com> added the comment:

I changed the patch and made it a little bit better.

----------
versions: +Python 3.1, Python 3.4 -Python 2.7
Added file: http://bugs.python.org/file24334/2012-1-26.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13651>
_______________________________________

From report at bugs.python.org  Fri Jan 27 03:26:28 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 27 Jan 2012 02:26:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327627021.3454.11.camel@localhost.localdomain>
Message-ID: <20120127032626.Horde.2FBmBcL8999PIgtSz1zBK2A@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> If I your read patch correctly, collisions will produce additional
> allocations of one distinct PyObject (i.e. AVL node) per colliding key.

That's correct.

> That's a pretty massive change in memory consumption for string dicts
> (and also in memory fragmentation and cache friendliness, probably).

That's not correct. It's not a massive change, as colliding hash values
never happen in practice, unless you are being attacked, in which case it
will be one additional PyObject for the set of all colliding keys (i.e.
one object per possible hundreds of string objects). Even including the
nodes of the tree (one per colliding node) is IMO a moderate increase
in memory usage, in order to solve the vulnerability.

It also doesn't impact memory fragmentation badly, as these objects
are allocated using the Python small objects allocator.

> The
> performance effect in most situations is likely to be negative too,
> despite the better worst-case complexity.

Compared to the status quo? Hardly. In all practical applications,
collision never happens, so none of the extra code is ever exexcuted -
except for AVL_Check invocations, which are a plain pointer
comparison.

> IMO that would be a rather controversial change for a feature release,
> let alone a bugfix or security release.

Apparently so, but it's not clear to me why that is. That change meets
all criteria of a security fix release nicely, as opposed to the proposed
changes to the hash function, which break existing code.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 04:27:34 2012
From: report at bugs.python.org (Ethan Furman)
Date: Fri, 27 Jan 2012 03:27:34 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1327629303.27.0.324384792427.issue6210@psf.upfronthosting.co.za>
Message-ID: <4F2219B4.50406@stoneleaf.us>


Ethan Furman <ethan at stoneleaf.us> added the comment:

> 1. Any syntax change requires a PEP (and, IMO, any such PEP for this issue should get rejected: I don't consider this an important enough feature to deserve dedicated syntax. Others disagree, which is one of the reasons why a PEP is needed. The other, more important, reason is to ensure the new syntax is spec'ed out clearly and incorporated into the language reference for the benefit of other implementations in the event that it *does* get approved)

I'm not sure why this would be a syntax change.  We can already do

try:
     1/0
except ZeroDivisionError:
     raise ValueError() from MathError()

to explicitly set the __cause__ and ignore the previous context.  The 
only difference would allowing 'None' to mean 'no cause, discard 
previous context'.

> 2. A change that *doesn't* need a PEP is to just adjust the implicit exception chaining such that __context__ doesn't get set automatically if it has already been set explicitly (it turns out handling this situation was an open question in PEP 3134, not a specificied behaviour). That way, this task can be handled using a utility function:
> 
>     def no_context(new_exc):
>         new_exc.__context__ = None
>         return new_exc
> 
>     def doXY ():
>         # ...
>         try:
>             page = urlopen( someRequest )
>         except urllib.error.URLError as e:
>             raise no_context(MyError( 'while doing XY', e ))

This seems like a lot more work than just allowing None to mean none.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 04:29:17 2012
From: report at bugs.python.org (Ethan Furman)
Date: Fri, 27 Jan 2012 03:29:17 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1327629303.27.0.324384792427.issue6210@psf.upfronthosting.co.za>
Message-ID: <4F221A1B.2050602@stoneleaf.us>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Nick Coghlan wrote:
> 1. Any syntax change requires a PEP

PEP is on python-dev.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 06:26:37 2012
From: report at bugs.python.org (Steven D'Aprano)
Date: Fri, 27 Jan 2012 05:26:37 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1327629303.27.0.324384792427.issue6210@psf.upfronthosting.co.za>
Message-ID: <4F223581.9090804@pearwood.info>


Steven D'Aprano <steve+python at pearwood.info> added the comment:

Nick Coghlan wrote:
> Nick Coghlan <ncoghlan at gmail.com> added the comment:
> 
> 1. Any syntax change requires a PEP (and, IMO, any such PEP for this issue
> should get rejected: I don't consider this an important enough feature to
> deserve dedicated syntax. Others disagree, which is one of the reasons why
> a PEP is needed. The other, more important, reason is to ensure the new
> syntax is spec'ed out clearly and incorporated into the language reference
> for the benefit of other implementations in the event that it *does* get
> approved)

This already has a PEP. This is an *explicitly* unresolved issue from the
original PEP that introduced exception chaining in the first place.

http://www.python.org/dev/peps/pep-3134/

I quote:

Open Issue: Suppressing Context

     As written, this PEP makes it impossible to suppress '__context__',
     since setting exc.__context__ to None in an 'except' or 'finally'
     clause will only result in it being set again when exc is raised.

With Ethan's patch, no new syntax is required. Since you can already say:

raise exception from another_exception

the syntax remains unchanged. There is an API change: currently 
another_exception must inherit from BaseException, with the patch it may also 
be None, but that doesn't change the syntax.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 06:31:49 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 27 Jan 2012 05:31:49 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <1327642309.4.0.883304606433.issue13847@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

There is still an error on Windows:

======================================================================
FAIL: test_localtime_failure (test.test_time.TimeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_time.py", line 364, in test_localtime_failure
    self.assertRaises(OSError, time.gmtime, invalid_time_t)
AssertionError: OSError not raised by gmtime

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/4074/steps/test/logs/stdio

time.ctime() uses localtime() internally, whereas time.gmtime() doesn't. Another time should maybe be written (to test a different time_t value). Or remove completly the whole test because it is not reliable :-)

----------
resolution: fixed -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Fri Jan 27 06:31:49 2012
From: report at bugs.python.org (Steven D'Aprano)
Date: Fri, 27 Jan 2012 05:31:49 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <4F223581.9090804@pearwood.info>
Message-ID: <4F2236BB.6090003@pearwood.info>


Steven D'Aprano <steve+python at pearwood.info> added the comment:

[...]

My comment has been overtaken by additional comments by Nick on the Python-Dev 
list.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Fri Jan 27 07:25:19 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Fri, 27 Jan 2012 06:25:19 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120127000334.Horde.aM5EaVNNcXdPIdvGl__HGMA@webmail.df.eu>
Message-ID: <CAGE7PNJRaASr_bLHDSbcMmJcjJ3CmS5s1Vo_MOkY9=UokkWYGA@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

>> But using non-__builtin__.str objects (such as UserString) would expose the
>> user to an attack?
>
> Not necessarily: only if they use these strings as dictionary keys, and only
> if they do so in contexts where arbitrary user input is consumed. In these
> cases, users need to rewrite their code to replace the keys. Using dictionary
> wrappers (such as UserDict), this is possible using only local changes.

Could the AVL tree approach be extended to apply to dictionaries
containing keys of any single type that supports comparison?  That
approach would autodetect UserString or similar and support it
properly.

I expect that dictionaries with keys of more than one type to be very
rare and highly unlikely when it comes to values generated directly
via user input.

(and on top of all of this I believe we're all settled on having per
interpreter hash randomization _as well_ in 3.3; but this AVL tree
approach is one nice option for a backport to fix the major
vulnerability)

-gps

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 07:45:40 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 06:45:40 +0000
Subject: [issue13884] IDLE 2.6.5 Recent Files undocks
In-Reply-To: <1327621264.5.0.587291207239.issue13884@psf.upfronthosting.co.za>
Message-ID: <1327646740.38.0.581140026.issue13884@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

IDLE has tear-off menus. From Help/IDLE Help:
"Click on the dotted line at the top of a menu to "tear it off":
a separate window containing the menu is created."
This is a feature, not a bug.

On 3.2.2, Win7, the Recent Files sub-menu cannot be torn off unless and until the File menu is torn off.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13884>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:25:54 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 08:25:54 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
Message-ID: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

Original e-mail from Apple security team:

> Follow-up:  187806281
> 
> SSL 3.0 and TLS 1.0 are vulnerable to an attack described at
> 
> http://www.openssl.org/~bodo/tls-cbc.txt
> 
> OpenSSL includes a countermeasure which prevents the attack, but python
> 2.7 has, around line 372 of Modules/_ssl.c:
> 
> SSL_CTX_set_options(self->ctx, SSL_OP_ALL);
> 
> SSL_OP_ALL includes SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS which disables the 
> countermeasure.
> 
> 2.6 is similar.

----------
components: Extension Modules
messages: 152068
nosy: barry, benjamin.peterson, loewis, pitrou
priority: critical
severity: normal
stage: commit review
status: open
title: CVE-2011-3389: _ssl module always disables the CBC IV attack countermeasure
type: security
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:34:03 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 08:34:03 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <1327653243.89.0.137778190029.issue13885@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24335/CVE-2011-3389-2.7.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:34:22 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 08:34:22 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <1327653262.15.0.677241763375.issue13885@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Attaching patches.

----------
Added file: http://bugs.python.org/file24336/CVE-2011-3389-3.2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:42:53 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 27 Jan 2012 08:42:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNJRaASr_bLHDSbcMmJcjJ3CmS5s1Vo_MOkY9=UokkWYGA@mail.gmail.com>
Message-ID: <20120127094251.Horde.Fx0gWElCcOxPImOLWnb3G0A@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> Could the AVL tree approach be extended to apply to dictionaries
> containing keys of any single type that supports comparison?  That
> approach would autodetect UserString or similar and support it
> properly.

I think we would need a place to store the single key type, which,
from an ABI point of view, might be difficult to find (but we could
overload ma_smalltable for that, or reserve ma_table[0]).

In addition, I think it is difficult to determine whether a type
supports comparison, at least in 2.x. For example,

class X:
  def __eq__(self, o):
    return self.a == o.a

allows to create objects x and y so that x<y<z, yet x==z.

For 3.x, we could assume that a failure to support comparison
raises an exception, in which case we could just wait for the
exception to happen, and then flatten the dictionary and start
over with the lookup. This would extend even to mixed key
types.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:49:23 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 08:49:23 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <E1RqhUn-0001py-N5@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9a4131ada792 by Antoine Pitrou in branch '2.6':
Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC IV attack countermeasure.
http://hg.python.org/cpython/rev/9a4131ada792

New changeset 8dec547c23d3 by Antoine Pitrou in branch '2.7':
Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC IV attack countermeasure.
http://hg.python.org/cpython/rev/8dec547c23d3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:57:50 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 08:57:50 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <E1Rqhcy-0002aV-S0@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset e7706bdaaa0d by Antoine Pitrou in branch '3.1':
Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC IV attack countermeasure.
http://hg.python.org/cpython/rev/e7706bdaaa0d

New changeset 4386686a035d by Antoine Pitrou in branch '3.2':
Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC IV attack countermeasure.
http://hg.python.org/cpython/rev/4386686a035d

New changeset d1390175fdc6 by Antoine Pitrou in branch 'default':
Issue #13885: CVE-2011-3389: the _ssl module would always disable the CBC IV attack countermeasure.
http://hg.python.org/cpython/rev/d1390175fdc6

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 09:58:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 08:58:46 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <1327654726.59.0.890421256237.issue13885@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

This hopefully fixes the issue.

----------
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:19:03 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Fri, 27 Jan 2012 09:19:03 +0000
Subject: [issue13873] SIGBUS in test_zlib on Debian bigmem buildbot
In-Reply-To: <1327583804.29.0.71599909936.issue13873@psf.upfronthosting.co.za>
Message-ID: <1327655943.03.0.495083444784.issue13873@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

I've also been unable to reproduce it on my own machine (AMD64; 8GB RAM).

I guess I'll have to do some trial-and-error debugging using the custom
builder to figure this out.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13873>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:28:30 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 09:28:30 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327656510.59.0.15052161836.issue13848@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Indeed, there seems to be no mechanism available to forbid NUL chars under Windows (for Python 3.x):

>>> open("LICENSE\x00foobar")
<_io.TextIOWrapper name='LICENSE\x00foobar' mode='r' encoding='cp1252'>
>>> os.stat("LICENSE\x00foobar")
nt.stat_result(st_mode=33206, st_ino=2251799813779714, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=15132, st_atime=8589934592, st_mtime=8589934592, st_ctime=1301169903)

I think PyUnicode_AsUnicode should grow a NUL char check in Python 3.3, since it doesn't return the size anyway. I don't think we can do that in previous versions, though, so we need an alternate strategy. Scanning the unicode string for NUL characters is enough. That should be easy by using PyUnicode_AsUnicodeAndSize.

As for the patch:
- the test should be in test_io; you may also add a separate in test_fileio
- conv_name is never decref'ed, and so there will be a memory leak

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:30:59 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 27 Jan 2012 09:30:59 +0000
Subject: [issue1003195] segfault when running smtplib example
Message-ID: <1327656659.03.0.0671688073886.issue1003195@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
resolution: wont fix -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1003195>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:32:13 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 27 Jan 2012 09:32:13 +0000
Subject: [issue13877] segfault when running smtplib example
In-Reply-To: <1327593010.34.0.461726513407.issue13877@psf.upfronthosting.co.za>
Message-ID: <1327656733.18.0.714208953597.issue13877@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> See (permanently closed?) similar bug at:

I reopened it.

----------
nosy: +neologix
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> segfault when running smtplib example

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13877>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:32:33 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 09:32:33 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327656753.35.0.296051814708.issue13848@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Since the NUL-scanning will be useful for Modules/posixmodule.c as well, perhaps it should be done as a private _PyUnicode_HasNULChars() function.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:58:05 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 09:58:05 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <E1RqiZG-0008Ha-Ai@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2863d9273abd by Antoine Pitrou in branch '3.2':
Issue #13812: When a multiprocessing Process child raises an exception, flush stderr after printing the exception traceback.
http://hg.python.org/cpython/rev/2863d9273abd

New changeset 96c1de5acbd3 by Antoine Pitrou in branch 'default':
Issue #13812: When a multiprocessing Process child raises an exception, flush stderr after printing the exception traceback.
http://hg.python.org/cpython/rev/96c1de5acbd3

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Fri Jan 27 10:58:33 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 09:58:33 +0000
Subject: [issue13812] multiprocessing package doesn't flush stderr on child
	exception
In-Reply-To: <1326849345.92.0.83488691035.issue13812@psf.upfronthosting.co.za>
Message-ID: <1327658313.11.0.330288190775.issue13812@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Patch now committed, thanks.

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________

From report at bugs.python.org  Fri Jan 27 11:23:39 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Fri, 27 Jan 2012 10:23:39 +0000
Subject: [issue13886] readline-related test_builtin failure
Message-ID: <1327659819.71.0.475160045664.issue13886@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

I've recently come across a strange failure in the tests for the input()
built-in function:

    $ ./python -E -m test -v test_readline test_builtin

    [... snip ...]

    ======================================================================
    FAIL: test_input_tty_non_ascii (test.test_builtin.BuiltinTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1079, in test_input_tty_non_ascii
        self.check_input_tty("prompt?", b"quux\xe9", "utf-8")
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1070, in check_input_tty
        self.assertEqual(input_result, expected)
    AssertionError: 'quux' != 'quux\udce9'
    - quux
    + quux\udce9
    ?     +


    ======================================================================
    FAIL: test_input_tty_non_ascii_unicode_errors (test.test_builtin.BuiltinTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1083, in test_input_tty_non_ascii_unicode_errors
        self.check_input_tty("prompt?", b"quux\xe9", "ascii")
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1070, in check_input_tty
        self.assertEqual(input_result, expected)
    AssertionError: 'quux' != 'quux\udce9'
    - quux
    + quux\udce9
    ?     +

The failure only manifests itself if the readline module is loaded before
test_builtin runs (hence the presence of test_readline above). It will
not occur if regrtest is run with either of the -j or -W flags (which is
why it hasn't been seen on the buildbots).

The problem seems to be that readline assumes that its input should use
the locale encoding, and silently strips out any undecodable chars. This
breaks the tests mentioned above, since they set up sys.stdin to use the
surrogateescape error handler, expecting invalid characters to be escaped
rather than discarded.

This problem doesn't crop up if readline is *not* loaded, because in that
case PyOS_Readline() falls back to a stdio-based implementation
(PyOS_StdioReadline()) that preserves invalid characters, allowing them
to be handled properly by sys.stdin's encoding and error handler.

I have been able to fix the test failures with the attached patch, which
stops readline from eating invalid characters, making it consistent with
the stdio-based fallback. Can someone with more knowledge of readline
and/or locale issues advise whether the change is a good idea?

----------
components: Extension Modules
files: rl-locale.diff
keywords: patch
messages: 152080
nosy: nadeem.vawda
priority: normal
severity: normal
stage: patch review
status: open
title: readline-related test_builtin failure
type: behavior
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24337/rl-locale.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13886>
_______________________________________

From report at bugs.python.org  Fri Jan 27 11:24:37 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 10:24:37 +0000
Subject: [issue13880] pydoc -k throws "AssertionError: distutils has already
	been patched by <class py2exe.Distribution at 0x0000000005987408>"
In-Reply-To: <1327606468.69.0.0708132604727.issue13880@psf.upfronthosting.co.za>
Message-ID: <1327659877.62.0.485668262907.issue13880@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

For the record, the proposed solution in that e-mail is the following:

"B) Patch the run() method of pydoc module at line 1862:

for importer, modname, ispkg in 
	pkgutil.walk_packages(onerror=lambda s:None):"

----------
components: +Library (Lib)
nosy: +pitrou
type: crash -> behavior
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13880>
_______________________________________

From report at bugs.python.org  Fri Jan 27 11:34:46 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Fri, 27 Jan 2012 10:34:46 +0000
Subject: [issue13886] readline-related test_builtin failure
In-Reply-To: <1327659819.71.0.475160045664.issue13886@psf.upfronthosting.co.za>
Message-ID: <1327660486.54.0.16097409848.issue13886@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Here's another patch that ensures the test always exercises the GNU
readline code path (rather than the stdio fallback). This will cause the
failure to occur when running just test_builtin (no need to also run
test_readline before it).

Ideally we'd want to test both code paths, but I'm not sure how to
accomplish that reliably, short of running the test in a subprocess.

----------
Added file: http://bugs.python.org/file24338/rl-test.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13886>
_______________________________________

From report at bugs.python.org  Fri Jan 27 11:50:03 2012
From: report at bugs.python.org (=?utf-8?q?J=C3=B6rn_Hees?=)
Date: Fri, 27 Jan 2012 10:50:03 +0000
Subject: [issue13887] defaultdict.get does not default to initial default but
	None
Message-ID: <1327661403.58.0.135329216168.issue13887@psf.upfronthosting.co.za>


New submission from J?rn Hees <nrej9TyO at joernhees.de>:

I wanted to create a "function registrar" d using a defaultdict. The library that this registrar is passed to expects it to return functions taking 3 args. Now if the first call is d.get(x) it seems that in contrast to d[x] the default arg of get is returned (None) instead of the defaultdicts default. If i call d[x] first and then d.get(x) i get what i expected.

Example:

In [1]: def foo(a,b,c):
   ...:     return (a,b,c)
   ...: 

In [2]: from collections import defaultdict

In [3]: d = defaultdict(lambda:foo)

In [4]: d.get(1)

In [5]: d[1]
Out[5]: <function foo at 0x1015a2ed8>

In [6]: d.get(1)
Out[6]: <function foo at 0x1015a2ed8>

In [7]: d.get(2)(1,2,3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/Users/joern/<ipython console> in <module>()

TypeError: 'NoneType' object is not callable

In [8]: d[2](1,2,3)
Out[8]: (1, 2, 3)

In [9]: d.get(2)(1,2,3)
Out[9]: (1, 2, 3)


I'm not sure this is the desired behavior, but it wasn't quite what i expected from a dictionary with a default.
If it is the desired behavior the documentation of defaultdict should include an explanation what happens.

----------
components: Extension Modules, Library (Lib)
messages: 152083
nosy: joern
priority: normal
severity: normal
status: open
title: defaultdict.get does not default to initial default but None
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13887>
_______________________________________

From report at bugs.python.org  Fri Jan 27 11:54:16 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Fri, 27 Jan 2012 10:54:16 +0000
Subject: [issue13888] test_builtin failure when run after test_tk
Message-ID: <1327661656.8.0.445863136974.issue13888@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

While investigating issue 13886, I found that test_builtin will fail when run
after test_tk:

    $ ./python -Wd -E -bb -m test -vuall test_tk test_builtin
    == CPython 3.3.0a0 (default:52f68c95e025, Jan 26 2012, 19:05:09) [GCC 4.6.1]
    ==   Linux-3.0.0-15-generic-x86_64-with-debian-wheezy-sid little-endian
    ==   /home/nadeem/src/cpython/def/build/test_python_4559
    Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0)

    [... snip ...]

    ======================================================================
    FAIL: test_input_tty (test.test_builtin.BuiltinTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1075, in test_input_tty
        self.check_input_tty("prompt", b"quux")
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1059, in check_input_tty
        % (len(lines), child_output.read()))
    AssertionError: got 0 lines in pipe but expected 2, child output was:
    quux

    promptCurrent thread 0x00007fb3bbed5720:
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1036 in check_input_tty
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1075 in test_input_tty
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 385 in _executeTestPart
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 440 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 492 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/runner.py", line 168 in run
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1369 in _run_suite
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1403 in run_unittest
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1469 in test_main
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 1221 in runtest_inner
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 918 in runtest
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 710 in main
      File "/home/nadeem/src/cpython/def/Lib/test/__main__.py", line 13 in <module>
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 73 in _run_code
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 160 in _run_module_as_main


    ======================================================================
    FAIL: test_input_tty_non_ascii (test.test_builtin.BuiltinTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1079, in test_input_tty_non_ascii
        self.check_input_tty("prompt?", b"quux\xe9", "utf-8")
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1059, in check_input_tty
        % (len(lines), child_output.read()))
    AssertionError: got 0 lines in pipe but expected 2, child output was:
    quux

    promptCurrent thread 0x00007fb3bbed5720:
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1036 in check_input_tty
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1079 in test_input_tty_non_ascii
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 385 in _executeTestPart
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 440 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 492 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/runner.py", line 168 in run
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1369 in _run_suite
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1403 in run_unittest
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1469 in test_main
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 1221 in runtest_inner
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 918 in runtest
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 710 in main
      File "/home/nadeem/src/cpython/def/Lib/test/__main__.py", line 13 in <module>
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 73 in _run_code
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 160 in _run_module_as_main


    ======================================================================
    FAIL: test_input_tty_non_ascii_unicode_errors (test.test_builtin.BuiltinTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1083, in test_input_tty_non_ascii_unicode_errors
        self.check_input_tty("prompt?", b"quux\xe9", "ascii")
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1059, in check_input_tty
        % (len(lines), child_output.read()))
    AssertionError: got 0 lines in pipe but expected 2, child output was:
    quux

    prompt?[xcb] Unknown sequence number while processing queue
    [xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
    [xcb] Aborting, sorry about that.
    python: ../../src/xcb_io.c:273: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
    Fatal Python error: Aborted

    Current thread 0x00007fb3bbed5720:
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1036 in check_input_tty
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1083 in test_input_tty_non_ascii_unicode_errors
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 385 in _executeTestPart
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 440 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/case.py", line 492 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 105 in run
      File "/home/nadeem/src/cpython/def/Lib/unittest/suite.py", line 67 in __call__
      File "/home/nadeem/src/cpython/def/Lib/unittest/runner.py", line 168 in run
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1369 in _run_suite
      File "/home/nadeem/src/cpython/def/Lib/test/support.py", line 1403 in run_unittest
      File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1469 in test_main
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 1221 in runtest_inner
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 918 in runtest
      File "/home/nadeem/src/cpython/def/Lib/test/regrtest.py", line 710 in main
      File "/home/nadeem/src/cpython/def/Lib/test/__main__.py", line 13 in <module>
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 73 in _run_code
      File "/home/nadeem/src/cpython/def/Lib/runpy.py", line 160 in _run_module_as_main

The failures will not occur if the test suite is run with the -j or -W flags,
as the failing tests are skipped if stdout is not a tty.

I've encountered the failures on both machines I've tested - one Ubuntu 11.10
64-bit (running Unity), and the other Xubuntu 11.10 32-bit (running XFCE).

----------
components: Tests
messages: 152084
nosy: gpolo, nadeem.vawda
priority: normal
severity: normal
stage: needs patch
status: open
title: test_builtin failure when run after test_tk
type: crash
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13888>
_______________________________________

From report at bugs.python.org  Fri Jan 27 12:17:39 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 11:17:39 +0000
Subject: [issue11682] PEP 380 reference implementation for 3.3
In-Reply-To: <1301110605.68.0.392943983038.issue11682@psf.upfronthosting.co.za>
Message-ID: <1327663059.42.0.471810474826.issue11682@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Just noting that the pep380 integration branch is also available in the hg.python.org clone of my sandbox repo.

----------
hgrepos: +107

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11682>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:06:48 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 12:06:48 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327666008.7.0.376048298515.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Aside from some minor comments that I included in my review, the latest patch gets a +1 from me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:18:29 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 12:18:29 +0000
Subject: [issue13887] defaultdict.get does not default to initial default but
	None
In-Reply-To: <1327661403.58.0.135329216168.issue13887@psf.upfronthosting.co.za>
Message-ID: <1327666709.51.0.257074064544.issue13887@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

It's certainly intentional behaviour:  all the defaultdict does is provide a __missing__ method.  And as explained in

http://docs.python.org/library/stdtypes.html#mapping-types-dict

"No other operations or methods invoke __missing__()."

So it looks to me as though this issue should be either a doc issue, or a feature request for 3.3.  Reclassifying as a doc issue.

----------
assignee:  -> docs at python
components: +Documentation -Extension Modules, Library (Lib)
nosy: +docs at python, mark.dickinson
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13887>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:21:39 2012
From: report at bugs.python.org (abhishek)
Date: Fri, 27 Jan 2012 12:21:39 +0000
Subject: [issue8536] Support new features of ZLIB 1.2.4
In-Reply-To: <1272302591.08.0.11918516326.issue8536@psf.upfronthosting.co.za>
Message-ID: <1327666899.93.0.0985214877381.issue8536@psf.upfronthosting.co.za>


abhishek <creativeabhishekgoel at gmail.com> added the comment:

I think New features like "inflateReset2()", "inflateMark()", or "Z_TREES" flags are included in python 3.2.2.

----------
nosy: +abhishek_bits

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8536>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:37:56 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Fri, 27 Jan 2012 12:37:56 +0000
Subject: [issue8536] Support new features of ZLIB 1.2.4
In-Reply-To: <1272302591.08.0.11918516326.issue8536@psf.upfronthosting.co.za>
Message-ID: <1327667876.17.0.400822893686.issue8536@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

No, the latest revision of Modules/zlibmodule.c doesn't use any of these
new features.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8536>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:41:03 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 12:41:03 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327668063.97.0.232435609355.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Antoine's review picked up on several issues I missed or glossed over - I actually agree with his point about making most of the new APIs private rather than public.

With regards to exposing _testbuffer in the documentation of memoryview's hash support, perhaps it would be better to use a 1D bytes object + memoryview.cast() to get an officially supported multi-dimensional view of a region of memory?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Fri Jan 27 13:42:40 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 12:42:40 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327668063.97.0.232435609355.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327668028.3385.0.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> With regards to exposing _testbuffer in the documentation of
> memoryview's hash support, perhaps it would be better to use a 1D
> bytes object + memoryview.cast() to get an officially supported
> multi-dimensional view of a region of memory?

By the way, I didn't point them, but there are other places in the
memoryview doc where _testbuffer is mentioned.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Fri Jan 27 14:05:28 2012
From: report at bugs.python.org (Tim Willis)
Date: Fri, 27 Jan 2012 13:05:28 +0000
Subject: [issue13879] Argparse does not support subparser aliases in 2.7
In-Reply-To: <1327597168.42.0.828007379667.issue13879@psf.upfronthosting.co.za>
Message-ID: <1327669528.92.0.484378022394.issue13879@psf.upfronthosting.co.za>


Tim Willis <schadenfreude328 at gmail.com> added the comment:

Attaching a patch which merges aliases code from 3.2 back into 2.7.

----------
keywords: +patch
Added file: http://bugs.python.org/file24339/argparse_aliases.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13879>
_______________________________________

From report at bugs.python.org  Fri Jan 27 14:06:14 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Fri, 27 Jan 2012 13:06:14 +0000
Subject: [issue13885] CVE-2011-3389: _ssl module always disables the CBC IV
	attack countermeasure
In-Reply-To: <1327652753.95.0.174471030951.issue13885@psf.upfronthosting.co.za>
Message-ID: <1327669574.97.0.321428220053.issue13885@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Thank you for handling this, Antoine!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13885>
_______________________________________

From report at bugs.python.org  Fri Jan 27 14:13:11 2012
From: report at bugs.python.org (Tim Willis)
Date: Fri, 27 Jan 2012 13:13:11 +0000
Subject: [issue13879] Argparse does not support subparser aliases in 2.7
In-Reply-To: <1327597168.42.0.828007379667.issue13879@psf.upfronthosting.co.za>
Message-ID: <1327669991.18.0.95969282422.issue13879@psf.upfronthosting.co.za>


Tim Willis <schadenfreude328 at gmail.com> added the comment:

adding package author to nosy list

----------
nosy: +bethard

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13879>
_______________________________________

From report at bugs.python.org  Fri Jan 27 14:58:50 2012
From: report at bugs.python.org (sbt)
Date: Fri, 27 Jan 2012 13:58:50 +0000
Subject: [issue12328] multiprocessing's overlapped PipeConnection on Windows
In-Reply-To: <1307991196.59.0.849860111943.issue12328@psf.upfronthosting.co.za>
Message-ID: <1327672730.99.0.239982977543.issue12328@psf.upfronthosting.co.za>


sbt <shibturn at gmail.com> added the comment:

Quite honestly I don't like the way that polling a pipe reads a partial message from the pipe.  If at all possible, polling should not modify the pipe.

I think the cleanest thing would be to switch to byte oriented pipes on Windows and create PipeIO which subclasses RawIOBase.  (Since socket handles are really just overlapped file handles, PipeIO works for them too as long as closesocket() is used instead of CloseHandle().)  On Unix FileIO would be used instead.  Then Connection can just be a thin wrapper around a file object.

Polling on Windows can then be done by creating a wrapper for an overlapped object which represents a zero length read, and can be used with WaitForMultipleObjects().  This lets us implement a select-like wait() function in Python which works with both sockets and Connection objects.

Attached is an extension implementing PipeIO (and the overlapped wrapper), a Python module implementing Connection and wait(), and a test.

----------
Added file: http://bugs.python.org/file24340/PipeIO.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12328>
_______________________________________

From report at bugs.python.org  Fri Jan 27 15:14:44 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 14:14:44 +0000
Subject: [issue13887] defaultdict.get does not default to initial default but
	None
In-Reply-To: <1327661403.58.0.135329216168.issue13887@psf.upfronthosting.co.za>
Message-ID: <E1RqmZf-0008OV-5e@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 089a086252fc by Benjamin Peterson in branch '3.2':
note that get() is not affected by default_factory (closes #13887)
http://hg.python.org/cpython/rev/089a086252fc

New changeset 26612ad451ad by Benjamin Peterson in branch 'default':
merge heads (#13887)
http://hg.python.org/cpython/rev/26612ad451ad

New changeset b2db66bc8e7f by Benjamin Peterson in branch '2.7':
note that get() is not affected by default_factory (closes #13887)
http://hg.python.org/cpython/rev/b2db66bc8e7f

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13887>
_______________________________________

From report at bugs.python.org  Fri Jan 27 15:20:20 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 14:20:20 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327674020.3.0.0306924515888.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Branch status update:
- Text Sequence Types section updated to reflect the new structure
- changed the prose that describes the relationship between printf-style formatting and the str.format method (deliberately removing the implication that the former is any real danger of disappearing - it's simply not practical for us to seriously contemplate killing it off)
- in the top level index, I split the old "String Services" section into "Text Processing Services" and "Binary Data Services". The latter contains 'struct' and 'codecs', the former contains everthing else that used to be in String Services. The index pages for the two sections do cross-reference modules in the other section a bit (Text Processing includes a pointer directly to the codecs module, Binary Data includes pointers to both re and difflib). The real driver for this change was that "struct" has no place in a "String Services" section in Py3k. Since "codecs" could really have gone in either section, I mainly moved it to the binary section so that 'struct' wasn't the only module in there.

Major remaining update is to the Binary Sequence Types section (since I haven't really reviewed that at all after rearranging things.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Fri Jan 27 15:33:47 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Fri, 27 Jan 2012 14:33:47 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327674827.58.0.530581465007.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

One other things the branch doesn't currently sort out is the official signature of count() and index().

In 3.2, for *all* of str, bytes, bytearray, tuple, list, range, the index() method takes the optional start:stop parameters.

collections.Sequence.index(), OTOH, does not.

count() splits the field more evenly: str, bytes, bytearray accept the extra parameters, but list, tuple, range and collections.Sequence only support counting values in the whole sequence.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Fri Jan 27 15:58:24 2012
From: report at bugs.python.org (Samuel Iseli)
Date: Fri, 27 Jan 2012 14:58:24 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
Message-ID: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>


New submission from Samuel Iseli <samuel.iseli at gmail.com>:

We are using python as an embedded scripting environment in our ERP-product.
Since upgrading to python2.7 we have serious issues with floats:

>>> 28710.0
'2870:.0'
>>> round(28710.0)
2870.0

We are embedding Python in a Delphi-application.

The problem was already discussed in issue9980 and has to do with Delphi setting the FPU precision to 64bit (and relying on this setting) while the standard with Microsoft Tools is 53bits.
The routines _Py_dg_dtoa and _Py_dg_strtod in dtoa.c rely on the FPU precision set to 53bits.

Issue9980 was closed as "won't fix" but I propose to reconsider this decision for the following reasons:

- Delphi is still an important development environment for native win32 applications and has excellent Python embedding support through PythonForDelphi (http://code.google.com/p/python4delphi).

- Ensuring 53bit before calling python code is not practical in an embedded python environment with extensions in delphi (python may also call code that is implemented in delphi).

- The changes needed in pythoncore are minimal. Tests documented in issue9980 found no measurable performance impact.

- FPU precision switching is needed and already (partially) implemented for linx, where 64bit prec is standard.

Fixing this issues is absolutely vital for us, so we needed to compile a custom version of pythoncore.

I appended a .diff file detailling the patch.

Changes are needed in 2 places:
- pyport.h, defining the _PY_SET_53_BIT_PRECISION macros for MS visual c compiler
- floatobject.c, insert precision switching macros in _Py_double_round function.

In pystrtod.c the precision switching macros are already used.
pystrtod.c and floatobject.c are the only places in CPython where the dtoa.c functions are called.

The macros for visual-c are activated by defining HAVE_VC_FUNC_FOR_X87 preprocessor-symbol.

Hoping for inclusion of this patch.

Cheers
Samuel

----------
components: Interpreter Core
files: 120127_float_dtoa_fix_py2.7.2.diff
keywords: patch
messages: 152099
nosy: samuel.iseli
priority: normal
severity: normal
status: open
title: str(float) and round(float) issues with FPU precision
versions: Python 2.7, Python 3.1
Added file: http://bugs.python.org/file24341/120127_float_dtoa_fix_py2.7.2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 16:05:46 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Fri, 27 Jan 2012 15:05:46 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327676746.12.0.752757044824.issue13874@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Fri Jan 27 16:12:22 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Fri, 27 Jan 2012 15:12:22 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327677142.77.0.0961228400291.issue4966@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Have you considered/planned to rework a bit the beginning of the page too?
(Technically the issue is about the Sequence types section, but the whole page could be improved.)
IMHO the sections about Truth value testing, Boolean operations, and Comparison are out of place there, and True/False/None should be described instead.
The idea is that a developer new to Python should be able to come to this page, take a look at the headers and figure out what the main types are (what you did so far is already a good step in the right direction).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Fri Jan 27 16:13:04 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Fri, 27 Jan 2012 15:13:04 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327677184.75.0.497559987602.issue13889@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
nosy: +eric.smith, mark.dickinson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 17:19:49 2012
From: report at bugs.python.org (STINNER Victor)
Date: Fri, 27 Jan 2012 16:19:49 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327676746.17.0.883842983352.issue13874@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwZ_zTHoBzQpVYfvBxxhP274y7TXXhqF2La1Jd6eenmkVQ@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Oh, you can also try something else: add the volatile keyword.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Fri Jan 27 17:50:57 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 16:50:57 +0000
Subject: [issue13890] test_importlib failures under Windows
Message-ID: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>


New submission from Antoine Pitrou <pitrou at free.fr>:

I don't know if these are related to the latest changes, but they only appear on the 3.3 buildbots:

======================================================================
FAIL: test_case_insensitivity (importlib.test.extension.test_case_sensitivity.ExtensionModuleCaseSensitivityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\importlib\test\extension\test_case_sensitivity.py", line 30, in test_case_insensitivity
    self.assertTrue(hasattr(loader, 'load_module'))
AssertionError: False is not true

======================================================================
FAIL: test_insensitive (importlib.test.source.test_case_sensitivity.CaseSensitivityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\importlib\test\source\test_case_sensitivity.py", line 51, in test_insensitive
    self.assertTrue(hasattr(insensitive, 'load_module'))
AssertionError: False is not true

----------
components: Library (Lib), Tests
messages: 152102
nosy: brett.cannon, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: test_importlib failures under Windows
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Fri Jan 27 18:33:53 2012
From: report at bugs.python.org (John Zimmerman)
Date: Fri, 27 Jan 2012 17:33:53 +0000
Subject: [issue13891] CPU DoS With Python's socket module
Message-ID: <1327685633.34.0.235701506916.issue13891@psf.upfronthosting.co.za>


New submission from John Zimmerman <johzimme at cisco.com>:

Python's socket module as included in Ubuntu Lucid (python version 2.6.5) does not correctly handle and exclude malformed UDP packets. This means that UDP listening programs written in python on this version are susceptible to malformed-UDP-packet based DoS attacks which cause severe CPU spikes in the python process. The spikes to recover once the attacks cease. If malformed UDP packets are properly identified in the library and excluded this will protect all UDP listening software written in python and using the standard sockets module from similar attacks. Currently all such software is vulnerable to such attacks.

----------
components: Library (Lib)
messages: 152103
nosy: johzimme
priority: normal
severity: normal
status: open
title: CPU DoS With Python's socket module
type: security
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 18:45:11 2012
From: report at bugs.python.org (Jim Jewett)
Date: Fri, 27 Jan 2012 17:45:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327627021.3454.11.camel@localhost.localdomain>
Message-ID: <CA+OGgf7euPmodnjKXc_j7HvyZFwV6cs-Ud-TiO++9qRp-yMHtg@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Thu, Jan 26, 2012 at 8:19 PM, Antoine Pitrou <report at bugs.python.org> wrote:

> If I read your [Martin v. L?wis' ] patch correctly, collisions will
> produce additional allocations ... That's a pretty massive
> change in memory consumption for string dicts

Not in practice.

The point I first missed is that this triggers only when the hash is
*fully* equal; if the hashes are merely equal after masking, then
today's try-another-slot approach will still be used, even for
strings.

Per ( http://bugs.python.org/issue13703#msg151850 ) Marc-Andre
Lemburg's measurements, full-hash equality explains only 1 in 10,000
collisions.  From a performance standpoint, we can almost ignore a
case that rare; it is almost certainly dwarfed by resizing.

I *am* a bit concerned that the possible contents of a dictentry
change; this could cause easily-missed-in-testing breakage for
anything that treats table as an array.  That said, it doesn't seem
much worse than the search finger, and there seemed to be recent
consensus that even promising an exact dict -- subclasses not allowed
-- didn't mean that direct access was sanctioned.  So it still seems
safer than changing the de-facto iteration order.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 18:46:46 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 17:46:46 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <1327685633.34.0.235701506916.issue13891@psf.upfronthosting.co.za>
Message-ID: <1327686406.54.0.709266031712.issue13891@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

What do you call "malformed UDP packets" and how should they be detected by the library?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 18:59:27 2012
From: report at bugs.python.org (John Zimmerman)
Date: Fri, 27 Jan 2012 17:59:27 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <1327686406.54.0.709266031712.issue13891@psf.upfronthosting.co.za>
Message-ID: <C0B8D844FF273B49AA6F0BDA44B9EBD70643F869@XMB-RCD-102.cisco.com>


John Zimmerman <johzimme at cisco.com> added the comment:

Thanks for your quick response, I downloaded the ISIC tool and used the following command to identify the problem:

udpsic -s rand -d server-ip-address,port

where port is 514 (syslogd) which uses a python script to process the incoming messages.

The command 'netstat -lnup | grep 514' shows python as the bound application.

Thanks,
John
-----Original Message-----
From: Antoine Pitrou [mailto:report at bugs.python.org] 
Sent: Friday, January 27, 2012 12:47 PM
To: John Zimmerman (johzimme)
Subject: [issue13891] CPU DoS With Python's socket module

Antoine Pitrou <pitrou at free.fr> added the comment:

What do you call "malformed UDP packets" and how should they be detected by the library?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 19:03:34 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 18:03:34 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <1327685633.34.0.235701506916.issue13891@psf.upfronthosting.co.za>
Message-ID: <1327687414.18.0.034310923234.issue13891@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

But what does the ISIC tool report and why do you think it is a problem in Python's socket module?

As far as I can read on its website, "ISIC is a suite of utilities to exercise the stability of an IP Stack and its component stacks (TCP, UDP, ICMP et. al.)". The IP stack, as well as the UDP and TCP implementations, is in the system's kernel, not in Python.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 19:15:21 2012
From: report at bugs.python.org (John Zimmerman)
Date: Fri, 27 Jan 2012 18:15:21 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <1327687414.18.0.034310923234.issue13891@psf.upfronthosting.co.za>
Message-ID: <C0B8D844FF273B49AA6F0BDA44B9EBD70643F86A@XMB-RCD-102.cisco.com>


John Zimmerman <johzimme at cisco.com> added the comment:

Hi Antoine, 

The issue is that the CPU spikes to ~90% utilization for the server during the attack, for as long as the attack lasts.  So the theory is that Python isn't throttling or processing the malformed packets properly.  Copying Renier for any additional info.

Thanks,
John

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 19:24:48 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 18:24:48 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <C0B8D844FF273B49AA6F0BDA44B9EBD70643F86A@XMB-RCD-102.cisco.com>
Message-ID: <1327688555.3358.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> The issue is that the CPU spikes to ~90% utilization for the server
> during the attack, for as long as the attack lasts.  So the theory is
> that Python isn't throttling or processing the malformed packets
> properly.  Copying Renier for any additional info.

I don't know who Renier is, but Python is a programming language and
doesn't integrate a "throttling" facility or ad-hoc protection against
network attacks. Other programming languages will show exactly the same
behaviour. The socket module gives access to the system's low-level
socket operations, it is not a high-level network programming framework.

Besides, truly malformed packets will never get processed by Python,
they will be blocked by the kernel (e.g. because of a checksum failure).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 19:43:29 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Fri, 27 Jan 2012 18:43:29 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327689809.09.0.624789490127.issue13868@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

There are a lot more occurrences of 'floating point' in our doc: are you going to fix them too?

----------
nosy: +sandro.tosi

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Fri Jan 27 20:06:34 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 19:06:34 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327691194.0.0.902809240168.issue13889@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Samuel, can you regenerate your patch?  The contents don't look right---I see changes to pyport.h, but none to floatobject.c  (and some other spurious-looking changes to the project file).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 20:32:11 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 27 Jan 2012 19:32:11 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf7euPmodnjKXc_j7HvyZFwV6cs-Ud-TiO++9qRp-yMHtg@mail.gmail.com>
Message-ID: <20120127203209.Horde.qxR-AUlCcOxPIvu5yQKlRBA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> I *am* a bit concerned that the possible contents of a dictentry
> change; this could cause easily-missed-in-testing breakage for
> anything that treats table as an array.

This is indeed a concern: the new code needs to be exercised.
I came up with a Py_REDUCE_HASH #define; if set, the dict will only
use the lowest 3 bits of the hash, producing plenty collisions.
In that mode, the branch currently doesn't work at all due to the
remaining bugs.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 20:34:08 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 27 Jan 2012 19:34:08 +0000
Subject: [issue13891] CPU DoS With Python's socket module
In-Reply-To: <1327685633.34.0.235701506916.issue13891@psf.upfronthosting.co.za>
Message-ID: <1327692848.3.0.386737129126.issue13891@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I don't want to be harsh, but this whole report just doesn't make sense
You're getting 90% CPU usage simply because you're flooding your server.
Closing.

----------
nosy: +neologix
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13891>
_______________________________________

From report at bugs.python.org  Fri Jan 27 20:34:16 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 19:34:16 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327692856.23.0.284472725784.issue13868@psf.upfronthosting.co.za>


Changes by Mark Dickinson <dickinsm at gmail.com>:


----------
nosy: +mark.dickinson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Fri Jan 27 20:36:35 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Fri, 27 Jan 2012 19:36:35 +0000
Subject: [issue1003195] segfault when running smtplib example
Message-ID: <1327692995.78.0.729918849317.issue1003195@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Could you please indicate exactly the command you're running, and provide the full backtrace?

----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1003195>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:14:38 2012
From: report at bugs.python.org (abhishek)
Date: Fri, 27 Jan 2012 20:14:38 +0000
Subject: [issue8536] Support new features of ZLIB 1.2.4
In-Reply-To: <1272302591.08.0.11918516326.issue8536@psf.upfronthosting.co.za>
Message-ID: <1327695278.99.0.516059251501.issue8536@psf.upfronthosting.co.za>


abhishek <creativeabhishekgoel at gmail.com> added the comment:

Check This out
http://hg.python.org/cpython/file/b99c54acb22d/Modules/zlib/inflate.c
http://hg.python.org/cpython/file/b99c54acb22d/Modules/zlib/zconf.h
http://hg.python.org/cpython/file/b99c54acb22d/Modules/zlib/zconf.in.h

All three files contain these functions

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8536>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:22:38 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 20:22:38 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327695758.79.0.4049267219.issue13889@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

I'm a bit disturbed that we're not using the _Py_SET_53BIT_PRECISION_* macros in _Py_double_round;  unless I'm missing something, that seems like a major omission.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:25:14 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 20:25:14 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327695914.19.0.0339252748661.issue13703@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

[Martin's approach]
> The point I first missed is that this triggers only when the hash is
> *fully* equal; if the hashes are merely equal after masking, then
> today's try-another-slot approach will still be used, even for
> strings.

But then isn't it vulnerable to Frank's first attack as exposed in
http://mail.python.org/pipermail/python-dev/2012-January/115726.html ?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:28:59 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 20:28:59 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327696139.26.0.922219737679.issue13889@psf.upfronthosting.co.za>


Changes by Mark Dickinson <dickinsm at gmail.com>:


----------
assignee:  -> mark.dickinson
priority: normal -> high

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:29:23 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 20:29:23 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327696163.65.0.242131858604.issue13889@psf.upfronthosting.co.za>


Changes by Mark Dickinson <dickinsm at gmail.com>:


----------
type:  -> behavior
versions: +Python 3.2, Python 3.3 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 21:59:39 2012
From: report at bugs.python.org (Patrick Hahn)
Date: Fri, 27 Jan 2012 20:59:39 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327697979.64.0.260472460079.issue13703@psf.upfronthosting.co.za>


Changes by Patrick Hahn <skorgu at gmail.com>:


----------
nosy: +skorgu

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:02:35 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Fri, 27 Jan 2012 21:02:35 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327695914.19.0.0339252748661.issue13703@psf.upfronthosting.co.za>
Message-ID: <20120127220233.Horde.-kL5FML8999PIxDp7fqjHtA@webmail.df.eu>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> But then isn't it vulnerable to Frank's first attack as exposed in
> http://mail.python.org/pipermail/python-dev/2012-January/115726.html ?

It would be, yes. That's sad.

That could be fixed by indeed creating trees in all cases (i.e. moving
away from open addressing altogether). The memory consumption does not worry
me here; however, dictionary order will change in more cases.

Compatibility could be restored by introducing a threshold for
tree creation: if insertion visits more than N slots, go back to the
original slot and put a tree there. I'd expect that N could be small,
e.g. N==4. Lookup would then have to consider all AVL trees along the
chain of visited slots, but ISTM it could also stop after visiting N
slots.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:09:25 2012
From: report at bugs.python.org (Jack Jansen)
Date: Fri, 27 Jan 2012 21:09:25 +0000
Subject: [issue13892] distutils handling of windows manifest isn't optimal
Message-ID: <1327698565.84.0.667656042788.issue13892@psf.upfronthosting.co.za>


New submission from Jack Jansen <jackjansen at users.sourceforge.net>:

I found a problem with the handling of manifest files in the msvc9compiler. Distutils removes the reference to the MSVC runtime from the manifest resource, to enable installing the runtime system locally (i.e. to allow non-admins to install Python in their own directory, without needing admin access to the winSxS directory). Embedders of Python are suggested to the manifest for the MSCV runtime to their main program (i.e. the program embedding Python).

This is all a good idea, except for one use case: if you are including Python into a product that is itself a plugin to another product. And, of course, if you need to extend this Python with an extension module. In this case, you have no control over the manifest of the main program (which is the outer embedder). You have control over your own plugin (the one incorporating Python), but this intermediate manifest file is not taken into account when loading the distutils-generated .pyd file.

(imagining the amount of time it took me before I got to this analysis is left as an exercise to the reader:-)

I would suggest an option to distutils to skip the _remove_visual_c_ref() step.

----------
assignee: tarek
components: Distutils
messages: 152119
nosy: eric.araujo, jackjansen, tarek
priority: normal
severity: normal
status: open
title: distutils handling of windows manifest isn't optimal
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13892>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:20:22 2012
From: report at bugs.python.org (Roundup Robot)
Date: Fri, 27 Jan 2012 21:20:22 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <E1RqtDV-00036L-1o@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5b8800004955 by Mark Dickinson in branch '3.2':
Issue #13889: Add missing _Py_SET_53BIT_PRECISION_* calls around uses of dtoa.c functions in float round.
http://hg.python.org/cpython/rev/5b8800004955

New changeset 265d35e8fe82 by Mark Dickinson in branch 'default':
Merge 3.2 -> default (issue 13889)
http://hg.python.org/cpython/rev/265d35e8fe82

New changeset eaf553b063a7 by Mark Dickinson in branch '2.7':
Issue #13889: Add missing _Py_SET_53BIT_PRECISION_* calls around uses of dtoa.c functions in float round.
http://hg.python.org/cpython/rev/eaf553b063a7

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:28:48 2012
From: report at bugs.python.org (Brett Cannon)
Date: Fri, 27 Jan 2012 21:28:48 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <1327699728.56.0.0548366361616.issue13890@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

I was getting that error the other day on my OS X laptop, but then I thought I tweaked the code well enough to solve it (since I am running on a case-insensitive filesystem as well). It seems to stem from what nt.environ contains when the test sets PYTHONCASEOK and importlib._bootstrap._case_ok() tries to see if b'PYTHONCASEOK' is in nt.environ.

You (or anyone) have a Windows box to quickly test what nt.environ contains after os.environ.set('PYTHONCASEOK', '1') is called?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:29:48 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 21:29:48 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327699788.21.0.0100433243523.issue13889@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

> but none to floatobject.c 

Bah, reading comprehension fail.  Now that I look at your patch again, the floatobject.c changes are there (and pretty much match what I just committed).

I assume that the changes to pythoncore.vcproj can be disregarded, though?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:32:17 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Fri, 27 Jan 2012 21:32:17 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327699937.48.0.704178005174.issue13889@psf.upfronthosting.co.za>


Changes by Mark Dickinson <dickinsm at gmail.com>:


----------
priority: high -> normal

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:36:23 2012
From: report at bugs.python.org (Brett Cannon)
Date: Fri, 27 Jan 2012 21:36:23 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <1327700183.21.0.984219756119.issue13890@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

Otherwise it's the nt.listdir() comparison that's failing. Probably should check that is returning reasonable stuff as well.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:42:26 2012
From: report at bugs.python.org (Stefan Krah)
Date: Fri, 27 Jan 2012 21:42:26 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327700546.53.0.650824694552.issue10181@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

I've added some comments on Rietveld. This time, I pasted the email
addresses manually into the CC field, apparently without success
(I didn't receive mail).

Regarding the use of _testbuffer in the docs: I agree that it's strange,
on the other hand it illustrates several examples much better (length of
a multi-dimensional view, result of cast to ND, etc).


So I'm not sure what to do. Of course I'll take out the examples if you
really don't like it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:42:37 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Fri, 27 Jan 2012 21:42:37 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120127220233.Horde.-kL5FML8999PIxDp7fqjHtA@webmail.df.eu>
Message-ID: <1327700522.2219.86.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Fri, 2012-01-27 at 21:02 +0000, Martin v. L?wis wrote:
> Martin v. L?wis <martin at v.loewis.de> added the comment:
> 
> > But then isn't it vulnerable to Frank's first attack as exposed in
> > http://mail.python.org/pipermail/python-dev/2012-January/115726.html ?
> 
> It would be, yes. That's sad.
> 
> That could be fixed by indeed creating trees in all cases (i.e. moving
> away from open addressing altogether). The memory consumption does not worry
> me here; however, dictionary order will change in more cases.
> 
> Compatibility could be restored by introducing a threshold for
> tree creation: if insertion visits more than N slots, go back to the
> original slot and put a tree there. I'd expect that N could be small,
> e.g. N==4. Lookup would then have to consider all AVL trees along the
> chain of visited slots, but ISTM it could also stop after visiting N
> slots.

Perhaps we could combine my attack-detection code from 
  http://bugs.python.org/issue13703#msg151714
with Martin's AVL approach?  Use the ma_smalltable to track stats, and
when a dict detects that it's under attack,  *if* all the keys are
AVL-compatible, it could transition to full-AVL mode.  [I believe that
my patch successfully handles both of Frank's attacks, but I don't have
the test data - I'd be very grateful to receive a copy (securely)].

[See hybrid-approach-dmalcolm-2012-01-25-002.patch for the latest
version of attack-detection; I'm working on a rewrite in which I
restrict it to working just on pure-str dicts.  With that idea, when a
dict detects that it's under attack, *if* all the keys satisfy this
condition
  (new_hash(keyA) == new_hash(keyB)) iff (hash(keyA) == hash(keyB))
then all hash values get recalculated using new_hash (which is
randomized), which should offer protection in many common attack
scenarios, without the semantic change Alex and Antoine indicated]

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:42:48 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 21:42:48 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <1327700568.22.0.963154831136.issue13890@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well apparently nt.environ doesn't reflect os.environ:

>>> os.environ['PYTHONCASEOK'] = '1'
>>> nt.environ['PYTHONCASEOK']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'PYTHONCASEOK'
>>> nt.environ[b'PYTHONCASEOK']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: b'PYTHONCASEOK'
>>> os.environ['PYTHONCASEOK']
'1'
>>> os.environ[b'PYTHONCASEOK']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\t\cpython\lib\os.py", line 450, in __getitem__
    value = self._data[self.encodekey(key)]
  File "C:\t\cpython\lib\os.py", line 508, in encodekey
    return encode(key).upper()
  File "C:\t\cpython\lib\os.py", line 503, in check_str
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes

This is silly and is because of how the environ mapping is implemented in Lib/os.py: under POXIX, os.environ reflects posix.environ (it uses the same underlying dict), while under Windows, os.environ uses a distinct dict from nt.environ.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:46:08 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Fri, 27 Jan 2012 21:46:08 +0000
Subject: [issue8536] Support new features of ZLIB 1.2.4
In-Reply-To: <1272302591.08.0.11918516326.issue8536@psf.upfronthosting.co.za>
Message-ID: <1327700768.49.0.989999574715.issue8536@psf.upfronthosting.co.za>


Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com> added the comment:

Modules/zlib directory contains only bundled copy of zlib-1.2.5.
zlib Python module uses Modules/zlibmodule.c file.

----------
nosy: +Arfrever
versions: +Python 3.3 -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8536>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:53:49 2012
From: report at bugs.python.org (Steven Bethard)
Date: Fri, 27 Jan 2012 21:53:49 +0000
Subject: [issue13879] Argparse does not support subparser aliases in 2.7
In-Reply-To: <1327597168.42.0.828007379667.issue13879@psf.upfronthosting.co.za>
Message-ID: <1327701229.91.0.963470495832.issue13879@psf.upfronthosting.co.za>


Steven Bethard <steven.bethard at gmail.com> added the comment:

This is a new feature, not a bug, so I think the correct fix is to change the 2.7 documentation, since at this point 2.7 can only get bugfixes, not new features.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13879>
_______________________________________

From report at bugs.python.org  Fri Jan 27 22:56:16 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Fri, 27 Jan 2012 21:56:16 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327701376.51.0.552330230363.issue13882@psf.upfronthosting.co.za>


Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:07:20 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Fri, 27 Jan 2012 22:07:20 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327702040.39.0.245264029025.issue13882@psf.upfronthosting.co.za>


Alexander Belopolsky <alexander.belopolsky at gmail.com> added the comment:

Can we pick an API for this functionality that does not follow the worst of design anti-patterns?  Constant arguments, varying return type, hidden import, and the list can go on.

What is wrong with simply creating a new module, say "hirestime" with functions called decimal_time(), float_time(), datetime_time() and whatever else you would like.  Let's keep the good old 'time' module simple.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:15:58 2012
From: report at bugs.python.org (Stefan Krah)
Date: Fri, 27 Jan 2012 22:15:58 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327702558.13.0.0809938805317.issue13874@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Yes, volatile works, too. That's probably the best solution.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:17:06 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 22:17:06 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327702626.58.0.87491170958.issue13882@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well, creating a separate module is an anti-pattern in itself. calendar vs. time vs. datetime, anyone?
I would instead propose separate functions: decimal_time, decimal_clock... or, if you prefer, time_decimal and so on.

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:17:32 2012
From: report at bugs.python.org (Stefan Krah)
Date: Fri, 27 Jan 2012 22:17:32 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327702652.78.0.778235628842.issue13874@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Well, to be completely unambiguous. This works:

diff -r d2cf8a34ddf9 Modules/faulthandler.c
--- a/Modules/faulthandler.c    Thu Jan 26 00:15:07 2012 -0800
+++ b/Modules/faulthandler.c    Fri Jan 27 23:16:27 2012 +0100
@@ -943,7 +943,7 @@
 static PyObject *
 faulthandler_read_null(PyObject *self, PyObject *args)
 {
-    int *x = NULL, y;
+    volatile int *x = NULL, y;
     int release_gil = 0;
     if (!PyArg_ParseTuple(args, "|i:_read_null", &release_gil))

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:25:23 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Fri, 27 Jan 2012 22:25:23 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327702626.58.0.87491170958.issue13882@psf.upfronthosting.co.za>
Message-ID: <CAP7h-xb-CTui-cWVj=BUdugeBZpj97L1DcdkRsyPEduE2qnR9g@mail.gmail.com>


Alexander Belopolsky <alexander.belopolsky at gmail.com> added the comment:

On Fri, Jan 27, 2012 at 5:17 PM, Antoine Pitrou <report at bugs.python.org> wrote:
> Well, creating a separate module is an anti-pattern in itself. calendar vs. time vs. datetime, anyone?

Are you serious? Since the invention of structural programming,
creating a separate module for distinct functionality has been one of
the most powerful design techniques.  If I recall correctly, most of
the original GoF patterns were about separating functionality into a
separate module or a separate class.  The calendar module is indeed a
historical odd-ball, but what is wrong with time and datetime?

----------
nosy: +Alexander.Belopolsky

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:31:58 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 22:31:58 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <CAP7h-xb-CTui-cWVj=BUdugeBZpj97L1DcdkRsyPEduE2qnR9g@mail.gmail.com>
Message-ID: <1327703384.3358.7.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Are you serious? Since the invention of structural programming,
> creating a separate module for distinct functionality has been one of
> the most powerful design techniques.

Yes, I'm serious, and I don't see what structural programming or design
patterns have to do with it.

And we're not even talking about "distinct functionality", we're talking
about the exact same functionality, except that the return type has
slightly different implementation characteristics. Doing a module split
is foolish.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:33:31 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Fri, 27 Jan 2012 22:33:31 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327703611.73.0.195879279586.issue13882@psf.upfronthosting.co.za>


Changes by Alexander Belopolsky <alexander.belopolsky at gmail.com>:


----------
nosy: +mark.dickinson

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:58:43 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 22:58:43 +0000
Subject: [issue13836] Define key failed
In-Reply-To: <1327232737.47.0.872325277438.issue13836@psf.upfronthosting.co.za>
Message-ID: <1327705123.7.0.468891729537.issue13836@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

We are aware that key-binding deletion is not working correctly.
I am pretty sure this report has the same underlying issue as #4765.

----------
nosy: +serwy, terry.reedy
resolution:  -> duplicate
status: open -> closed
superseder:  -> IDLE fails to "Delete Custom Key Set" properly

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13836>
_______________________________________

From report at bugs.python.org  Fri Jan 27 23:58:51 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 22:58:51 +0000
Subject: [issue4765] IDLE fails to "Delete Custom Key Set" properly
In-Reply-To: <1230525520.13.0.417394561266.issue4765@psf.upfronthosting.co.za>
Message-ID: <1327705131.07.0.672186259421.issue4765@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

#13836 is another report of key-binding deletion not working correctly, although IDLE did eventually open and allow re-deletion.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4765>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:06:08 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:06:08 +0000
Subject: [issue13839] -m pstats should combine all the profiles given as
	arguments
In-Reply-To: <1327289897.02.0.607924329906.issue13839@psf.upfronthosting.co.za>
Message-ID: <1327705568.23.0.7924695471.issue13839@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +georg.brandl
stage:  -> test needed
versions:  -Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13839>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:08:44 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:08:44 +0000
Subject: [issue13841] multiprocessing should use sys.exit() where possible
In-Reply-To: <1327345132.43.0.775594933755.issue13841@psf.upfronthosting.co.za>
Message-ID: <1327705724.11.0.248610569765.issue13841@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
stage:  -> test needed
versions:  -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13841>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:12:16 2012
From: report at bugs.python.org (Giovanni Funchal)
Date: Fri, 27 Jan 2012 23:12:16 +0000
Subject: [issue13893] Make CGIHTTPServer capable of redirects (and status
	other than 200)
Message-ID: <1327705936.4.0.817960222769.issue13893@psf.upfronthosting.co.za>


New submission from Giovanni Funchal <gafunchal at gmail.com>:

GIHTTPServer.py is a very handy module for quickly setting up a full HTTP server with CGI support. However, I noticed that it doesn't support responses other than "200 OK". So, for instance if the page wants to do a redirect (response 303), it just isn't supported. It's documented as so, but still I think this is a major drawback that can be easily overcome.

I have attached a patch against 2.7. I'd be happy to port it to dev branch if help provided as I'm new to Python. Reviews/suggestions more than welcome.

----------
components: Library (Lib)
files: CGIHTTPServer.patch
keywords: patch
messages: 152137
nosy: Giovanni.Funchal
priority: normal
severity: normal
status: open
title: Make CGIHTTPServer capable of redirects (and status other than 200)
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file24342/CGIHTTPServer.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13893>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:15:10 2012
From: report at bugs.python.org (Eric V. Smith)
Date: Fri, 27 Jan 2012 23:15:10 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327706110.49.0.309069094282.issue13882@psf.upfronthosting.co.za>


Changes by Eric V. Smith <eric at trueblade.com>:


----------
nosy: +eric.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:16:35 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:16:35 +0000
Subject: [issue13846] Add time.monotonic() function
In-Reply-To: <1327364549.64.0.960699645726.issue13846@psf.upfronthosting.co.za>
Message-ID: <1327706195.24.0.939528205838.issue13846@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Do we actually yet another function, or could this be covered by adding a parameter such as monotonic=False, perhaps to wallclock().

----------
nosy: +terry.reedy
stage:  -> patch review
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13846>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:18:02 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:18:02 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <1327706282.76.0.533862963929.issue13847@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
stage:  -> needs patch
type:  -> enhancement

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:26:33 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:26:33 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327706793.19.0.77237201114.issue13848@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

See also #13849

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:26:36 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:26:36 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327706796.33.0.280459319949.issue13849@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

This is a bit related to #13848 files 2 seconds earlier.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:34:02 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:34:02 +0000
Subject: [issue13866] {urllib,
	urllib.parse}.urlencode should not use quote_plus
In-Reply-To: <1327529522.13.0.464287316871.issue13866@psf.upfronthosting.co.za>
Message-ID: <1327707242.16.0.698267248598.issue13866@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
nosy: +orsenthil
stage:  -> test needed
type:  -> enhancement
versions:  -Python 2.7, Python 3.1, Python 3.2, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13866>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:35:44 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 23:35:44 +0000
Subject: [issue13849] Add tests for NUL checking in certain strs
In-Reply-To: <1327370754.29.0.66468771639.issue13849@psf.upfronthosting.co.za>
Message-ID: <1327707344.4.0.210568134715.issue13849@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Well, adding tests is certainly useful to avoid regressions in CPython (not only PyPy or Jython). It seems to me that the only reasonable discussion is whether they should be committed to 2.7 and 3.2, or only the default branch.

(I personally like adding tests to stable branches as well, since that avoids potential regressions in further bugfix releases)

If we had had such tests from the beginning, 3.x wouldn't have regressed in that matter (see #13848).

----------
nosy: +pitrou
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13849>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:36:27 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Fri, 27 Jan 2012 23:36:27 +0000
Subject: [issue13867] misleading comment in weakrefobject.h
In-Reply-To: <1327532476.25.0.872465463087.issue13867@psf.upfronthosting.co.za>
Message-ID: <1327707387.85.0.0213902941429.issue13867@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


----------
components: +Library (Lib) -Documentation, Extension Modules
nosy: +fdrake, pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13867>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:39:53 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 23:39:53 +0000
Subject: [issue13867] misleading comment in weakrefobject.h
In-Reply-To: <1327532476.25.0.872465463087.issue13867@psf.upfronthosting.co.za>
Message-ID: <1327707593.12.0.542879376588.issue13867@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Ha, good catch. The macro could be fixed, or the comment simply be dropped.

----------
versions: +Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13867>
_______________________________________

From report at bugs.python.org  Sat Jan 28 00:44:04 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Fri, 27 Jan 2012 23:44:04 +0000
Subject: [issue13893] Make CGIHTTPServer capable of redirects (and status
	other than 200)
In-Reply-To: <1327705936.4.0.817960222769.issue13893@psf.upfronthosting.co.za>
Message-ID: <1327707844.55.0.986217618258.issue13893@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13893>
_______________________________________

From report at bugs.python.org  Sat Jan 28 01:22:42 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 28 Jan 2012 00:22:42 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327710162.55.0.177560790713.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Yeah, the basic layout of this entire section has been in place for a *long* time (http://docs.python.org/release/1.4/lib/node4.html#SECTION00310000000000000000)

Some aspects haven't really aged all that well, as people have made minimalist changes to document new features without necessarily stepping back to see if the overall structure still makes sense.

However, rather than dumping one massive patch on python-checkins, I think it makes sense to try to tackle it by section (i.e. sequences + related changes for now, then look at mappings, sets, truth values, comparisons and numbers separately).

One thing I do plan to do is a quick scan for places that reference into the sequence types section to see if they should be adjusted (e.g. see if there's some duplication in the language reference that could be reduced, or cross-references in the glossary to add or update)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sat Jan 28 01:49:10 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 28 Jan 2012 00:49:10 +0000
Subject: [issue1625] bz2.BZ2File doesn't support multiple streams
In-Reply-To: <1197624030.23.0.503522059328.issue1625@psf.upfronthosting.co.za>
Message-ID: <1327711750.47.0.00335338719974.issue1625@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

I think this support should be backported to Python 2.7 and 3.2. Current code can't decompress files generated by "pbzip2", fairly popular. I would consider that a bug, not a feature request.

I am just recompressing a 77GB file because of this :-(.

----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1625>
_______________________________________

From report at bugs.python.org  Sat Jan 28 01:56:32 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 28 Jan 2012 00:56:32 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327712192.54.0.217732940271.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Radical suggestion: make it public as collections.simple_ndarray?

(prefixing with simple_ to be explicit that this is not even *close* to being the all-singing, all-dancing NumPy.ndarray)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 02:30:57 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 28 Jan 2012 01:30:57 +0000
Subject: [issue13877] segfault when running smtplib example
In-Reply-To: <1327593010.34.0.461726513407.issue13877@psf.upfronthosting.co.za>
Message-ID: <1327714257.6.0.992103455963.issue13877@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13877>
_______________________________________

From report at bugs.python.org  Sat Jan 28 02:31:36 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 28 Jan 2012 01:31:36 +0000
Subject: [issue1003195] segfault when running smtplib example
Message-ID: <1327714296.62.0.0295220817993.issue1003195@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1003195>
_______________________________________

From report at bugs.python.org  Sat Jan 28 04:03:12 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sat, 28 Jan 2012 03:03:12 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327719792.36.0.734414179886.issue13703@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

For the record, Barry and I agreed on what we'll be doing for stable releases [1]. David says he should have a patch soon.

[1] http://mail.python.org/pipermail/python-dev/2012-January/115892.html

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 28 04:12:39 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sat, 28 Jan 2012 03:12:39 +0000
Subject: [issue4765] IDLE fails to "Delete Custom Key Set" properly
In-Reply-To: <1230525520.13.0.417394561266.issue4765@psf.upfronthosting.co.za>
Message-ID: <1327720359.6.0.616394382184.issue4765@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

I am unable to reproduce this bug with the latest development versions in the 2.7 and 3.3 branches. 

Can someone verify this as well?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4765>
_______________________________________

From report at bugs.python.org  Sat Jan 28 04:48:11 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 28 Jan 2012 03:48:11 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327722491.7.0.946865440596.issue13874@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

That makes "x" and "y" volatile.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Sat Jan 28 05:10:59 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sat, 28 Jan 2012 04:10:59 +0000
Subject: [issue13893] Make CGIHTTPServer capable of redirects (and status
	other than 200)
In-Reply-To: <1327705936.4.0.817960222769.issue13893@psf.upfronthosting.co.za>
Message-ID: <1327723859.96.0.81640762385.issue13893@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13893>
_______________________________________

From report at bugs.python.org  Sat Jan 28 06:13:40 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 28 Jan 2012 05:13:40 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327719792.36.0.734414179886.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327727572.2219.95.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-28 at 03:03 +0000, Benjamin Peterson wrote:
> Benjamin Peterson <benjamin at python.org> added the comment:
> 
> For the record, Barry and I agreed on what we'll be doing for stable releases [1]. David says he should have a patch soon.
> 
> [1] http://mail.python.org/pipermail/python-dev/2012-January/115892.html

I'm attaching what I've got so far (need sleep).

Attached patch is for 3.1 and adds opt-in hash randomization.

It's based on haypo's work: random-8.patch (thanks haypo!), with
additional changes as seen in my backport of that to 2.7:
http://bugs.python.org/issue13703#msg151847

* The randomization is off by default, and must be enabled by setting
a new environment variable PYTHONHASHRANDOMIZATION to a non-empty
string. (if so then, PYTHONHASHSEED also still works, if provided, in
the same way as in haypo's patch)

* All of the various "Py_hash_t" become "long" again (Py_hash_t was
added in 3.2: issue9778)

* I expanded the randomization from just PyUnicodeObject to also cover
PyBytesObject, and the types within datetime.

* It doesn't cover numeric types; see my explanation in msg151847; also
see http://bugs.python.org/issue13703#msg151870

* It doesn't yet cover the embedded copy of expat.

* I moved the hash tests from test_unicode.py to test_hash.py

* I tweaked the wording of the descriptions of the envvars in
cmdline.rst and the manpage

* I've tested it on a 32-bit box, and it successfully protects against
one set of test data (four cases: assembling then reading back items by
key for a dict vs set, bytes vs str, with 200000 distinct items of data
which all have hash() == 0 in unmodified build; each takes about 1.5
seconds on this --with-pydebug build, vs of the order of hours).

* I haven't yet benchmarked it

* Only tested on Linux (Fedora x86_64 and i686).  I don't know the
impact on windows (e.g. startup time without the envvar vs with the env
vars).

I'm seeing one failing test:
======================================================================
FAIL: test_clear_dict_in_ref_cycle (__main__.ModuleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/david/coding/python-hg/cpython-3.1-hash-randomization/Lib/test/test_module.py", line 79, in test_clear_dict_in_ref_cycle
    self.assertEqual(destroyed, [1])
AssertionError: Lists differ: [] != [1]

----------
Added file: http://bugs.python.org/file24343/optin-hash-randomization-for-3.1-dmalcolm-2012-01-27-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 73dad4940b88 Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/using/cmdline.rst	Sat Jan 28 00:03:46 2012 -0500
@@ -435,6 +435,34 @@
    import of source modules.
 
 
+.. envvar:: PYTHONHASHRANDOMIZATION
+
+   If this is set to a non-empty string, the hash() values of str, bytes
+   and datetime objects are "salted" with an unpredictable random value.
+   Although they remain constant within an individual Python process, they
+   are not predictable between repeated invocations of Python.
+
+   This is intended to provide protection against a denial-of-service
+   caused by carefully-chosen inputs that exploit the worst case performance
+   of a dict lookup, O(n^2) complexity.  See:
+   
+       http://www.ocert.org/advisories/ocert-2011-003.html
+
+   for details.
+
+   Changing hash values affects the order in which keys are retrieved from
+   a dict.  Although Python has never made guarantees about this ordering
+   (and it typically varies between 32-bit and 64-bit builds), enough
+   real-world code implicitly relies on this non-guaranteed behavior that
+   the randomization is disabled by default.
+
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for generating the hash() of
+   the types covererd by PYTHONHASHRANDOMIZATION.  It should be a number in
+   the range [0; 4294967295].  The value 0 overrides the other variable and
+   disables the hash random salt.
+
 .. envvar:: PYTHONIOENCODING
 
    Overrides the encoding used for stdin/stdout/stderr, in the syntax
diff -r 73dad4940b88 Include/object.h
--- a/Include/object.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/object.h	Sat Jan 28 00:03:46 2012 -0500
@@ -473,6 +473,12 @@
 PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
+typedef struct {
+    long prefix;
+    long suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
+
 /* Helper for passing objects to printf and the like */
 #define PyObject_REPR(obj) _PyUnicode_AsString(PyObject_Repr(obj))
 
diff -r 73dad4940b88 Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pythonrun.h	Sat Jan 28 00:03:46 2012 -0500
@@ -174,6 +174,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r 73dad4940b88 Lib/os.py
--- a/Lib/os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/os.py	Sat Jan 28 00:03:46 2012 -0500
@@ -611,23 +611,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r 73dad4940b88 Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/regrtest.py	Sat Jan 28 00:03:46 2012 -0500
@@ -428,6 +428,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(tests)
diff -r 73dad4940b88 Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_hash.py	Sat Jan 28 00:03:46 2012 -0500
@@ -3,10 +3,14 @@
 #
 # Also test that hash implementations are inherited as expected
 
+import struct
 import unittest
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -118,10 +122,65 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+class RandomizationTestCase(unittest.TestCase):
+
+    # Examples of the various types having randomized hash:
+    test_reprs = [repr('abc'), repr(b'abc')]
+
+    def get_hash(self, repr_, randomization=None, seed=None):
+        env = {}
+        if randomization is not None:
+            env['PYTHONHASHRANDOMIZATION'] = str(randomization)
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'print(hash(%s))' % repr_,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+        self.assertEqual(hash(b""), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            known_hash_of_obj = 1453079729188098211
+        else:
+            known_hash_of_obj = -1600925533
+        for repr_ in self.test_reprs:
+            # Randomization is disabled by default:
+            self.assertEqual(self.get_hash(repr_), known_hash_of_obj)
+            
+            # If enabled, it can still be disabled by setting the seed to 0:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=0),
+                             known_hash_of_obj)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        # Note that all types share the same values:
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        for repr_ in self.test_reprs:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=42),
+                             h)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        for repr_ in self.test_reprs:
+            run1 = self.get_hash(repr_, randomization=1)
+            run2 = self.get_hash(repr_, randomization=1)
+            self.assertNotEqual(run1, run2)
+
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
                               HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                              HashBuiltinsTestCase,
+                         RandomizationTestCase)
 
 
 if __name__ == "__main__":
diff -r 73dad4940b88 Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_os.py	Sat Jan 28 00:03:46 2012 -0500
@@ -9,6 +9,7 @@
 import sys
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 
 # Detect whether we're on a Linux system that uses the (now outdated
 # and unmaintained) linuxthreads threading library.  There's an issue
@@ -574,14 +575,33 @@
         f.close()
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(0)), 0)
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 class ExecTests(unittest.TestCase):
     @unittest.skipIf(USING_LINUXTHREADS,
diff -r 73dad4940b88 Lib/test/test_set.py
--- a/Lib/test/test_set.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_set.py	Sat Jan 28 00:03:46 2012 -0500
@@ -734,6 +734,17 @@
         if self.repr is not None:
             self.assertEqual(repr(self.set), self.repr)
 
+    def check_repr_against_values(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
     def test_print(self):
         try:
             fo = open(support.TESTFN, "w")
@@ -892,7 +903,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -903,7 +916,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{b'a', b'c', b'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +931,13 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
-
+ 
     def tearDown(self):
         warnings.filters = self.warning_filters
 
+    def test_repr(self):
+        self.check_repr_against_values()
+
 #==============================================================================
 
 def baditer():
diff -r 73dad4940b88 Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 20 11:23:02 2012 +0000
+++ b/Makefile.pre.in	Sat Jan 28 00:03:46 2012 -0500
@@ -305,6 +305,7 @@
 		Python/pymath.o \
 		Python/pystate.o \
 		Python/pythonrun.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r 73dad4940b88 Misc/python.man
--- a/Misc/python.man	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/python.man	Sat Jan 28 00:03:46 2012 -0500
@@ -403,6 +403,28 @@
 If this is set to a non-empty string it is equivalent to specifying
 the \fB\-v\fP option. If set to an integer, it is equivalent to
 specifying \fB\-v\fP multiple times. 
+.IP PYTHONHASHRANDOMIZATION
+If this is set to a non-empty string, the hash() values of str, bytes and
+datetime objects are "salted" with an unpredictable random value.  Although they
+remain constant within an individual Python process, they are not predictable
+between repeated invocations of Python.
+.IP
+This is intended to provide protection against a denial of service
+caused by carefully-chosen inputs that exploit the worst case performance
+of a dict lookup, O(n^2) complexity.  See
+http://www.ocert.org/advisories/ocert-2011-003.html
+for details.
+.IP
+Changing hash values affects the order in which keys are retrieved from
+a dict.  Although Python has never made guarantees about this ordering
+(and it typically varies between 32-bit and 64-bit builds), enough
+real-world code implicitly relies on this non-guaranteed behavior that
+the randomization is disabled by default.
+.IP PYTHONHASHSEED
+If this is set, it is used as a fixed seed for generating the hash() of
+the types covererd by PYTHONHASHRANDOMIZATION.  It should be a number in
+the range [0; 4294967295].  The value 0 overrides the other variable and
+disables the hash random salt.
 .SH AUTHOR
 The Python Software Foundation: http://www.python.org/psf
 .SH INTERNET RESOURCES
diff -r 73dad4940b88 Modules/datetimemodule.c
--- a/Modules/datetimemodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/datetimemodule.c	Sat Jan 28 00:03:46 2012 -0500
@@ -2566,10 +2566,12 @@
     register long x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= len;
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
 
diff -r 73dad4940b88 Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/posixmodule.c	Sat Jan 28 00:03:46 2012 -0500
@@ -6942,82 +6942,6 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
-static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
-    }
-    return result;
-}
-#endif
-
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
 Return a string describing the encoding of the device\n\
@@ -7055,41 +6979,35 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+Return n pseudo-random bytes.");
+
+static PyObject *
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
+ 
+     /* Read arguments */
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -7374,12 +7292,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
     {NULL,              NULL}            /* Sentinel */
 };
 
diff -r 73dad4940b88 Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/bytesobject.c	Sat Jan 28 00:03:46 2012 -0500
@@ -899,11 +899,17 @@
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
+    if (len == 0) {
+        a->ob_shash = 0;
+        return 0;
+    }
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(a);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff -r 73dad4940b88 Objects/object.c
--- a/Objects/object.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/object.c	Sat Jan 28 00:03:46 2012 -0500
@@ -712,6 +712,8 @@
     return -1;
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
 long
 PyObject_Hash(PyObject *v)
 {
diff -r 73dad4940b88 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/unicodeobject.c	Sat Jan 28 00:03:46 2012 -0500
@@ -7344,11 +7344,17 @@
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
+    if (len == 0) {
+        self->hash = 0;
+        return 0;
+    }
     p = self->str;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(self);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     self->hash = x;
diff -r 73dad4940b88 PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 20 11:23:02 2012 +0000
+++ b/PCbuild/pythoncore.vcproj	Sat Jan 28 00:03:46 2012 -0500
@@ -1778,6 +1778,10 @@
 				RelativePath="..\Python\pythonrun.c"
 				>
 			</File>
+ 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
 			<File
 				RelativePath="..\Python\structmember.c"
 				>
diff -r 73dad4940b88 Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/pythonrun.c	Sat Jan 28 00:03:46 2012 -0500
@@ -73,6 +73,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 
 #ifdef WITH_THREAD
 extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@@ -196,6 +197,8 @@
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
 
+    _PyRandom_Init();
+
     interp = PyInterpreterState_New();
     if (interp == NULL)
         Py_FatalError("Py_Initialize: can't make first interpreter");
diff -r 73dad4940b88 Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Sat Jan 28 00:03:46 2012 -0500
@@ -0,0 +1,284 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Failed to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Failed to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Failed to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Failed to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Failed to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    /*
+      By default, hash randomization is disabled, and only
+      enabled if PYTHONHASHRANDOMIZATION is set to non-empty
+    */
+    env = Py_GETENV("PYTHONHASHRANDOMIZATION");
+    if (!env || *env == '\0') {
+        /* Not found: disable the randomized hash: */
+        memset(secret, 0, secret_size);
+        return;
+    }
+
+    /*
+      PYTHONHASHRANDOMIZATION was found; generate a per-process secret,
+      using PYTHONHASHSEED if provided.
+    */
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else /* #ifdef MS_WINDOWS */
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+

From report at bugs.python.org  Sat Jan 28 07:30:37 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Sat, 28 Jan 2012 06:30:37 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327732237.86.0.765971811418.issue13806@psf.upfronthosting.co.za>


Changes by Oleg Plakhotnyuk <olegus8 at gmail.com>:


Removed file: http://bugs.python.org/file24285/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Sat Jan 28 07:32:41 2012
From: report at bugs.python.org (Oleg Plakhotnyuk)
Date: Sat, 28 Jan 2012 06:32:41 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327732361.66.0.628322762103.issue13806@psf.upfronthosting.co.za>


Oleg Plakhotnyuk <olegus8 at gmail.com> added the comment:

Yes, that tests really should be there too. Added them to patch.

----------
Added file: http://bugs.python.org/file24344/audioop_size_check.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Sat Jan 28 09:44:02 2012
From: report at bugs.python.org (Matt Joiner)
Date: Sat, 28 Jan 2012 08:44:02 +0000
Subject: [issue13894] threading._CRLock should not be tested if _thread.RLock
	isn't implemented
Message-ID: <1327740242.45.0.88821278342.issue13894@psf.upfronthosting.co.za>


New submission from Matt Joiner <anacrolix at gmail.com>:

_threading.RLock is optional, and threading._CRLock is set to None if it isn't available. If this happens, the test_threading unittests crash.

Some implementations don't provide _thread.RLock.

Patch attached.

----------
components: Library (Lib)
files: skip-crlock-tests-if-not-implemented.patch
keywords: patch
messages: 152151
nosy: anacrolix
priority: normal
severity: normal
status: open
title: threading._CRLock should not be tested if _thread.RLock isn't implemented
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24345/skip-crlock-tests-if-not-implemented.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13894>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:03:49 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Sat, 28 Jan 2012 10:03:49 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1327745029.83.0.0176412735636.issue13496@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

Here's a patch.

----------
keywords: +patch
Added file: http://bugs.python.org/file24346/issue13496.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:05:08 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 28 Jan 2012 10:05:08 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327745108.83.0.060515498675.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

One other point... the branch is actually now relative to default, not 3.2. While that was due to a merging mistake on my part, it also means I can legitimately ignore the narrow/wide build distinction in the section on strings.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:35:04 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 28 Jan 2012 10:35:04 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327722491.7.0.946865440596.issue13874@psf.upfronthosting.co.za>
Message-ID: <20120128103502.GA1395@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Jes??s Cea Avi??n <report at bugs.python.org> wrote:
> That makes "x" and "y" volatile.

Well yes, but is that a problem?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:38:10 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 28 Jan 2012 10:38:10 +0000
Subject: [issue13894] threading._CRLock should not be tested if _thread.RLock
	isn't implemented
In-Reply-To: <1327740242.45.0.88821278342.issue13894@psf.upfronthosting.co.za>
Message-ID: <E1Rr5fc-0001As-Rl@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5cf181df7bea by Charles-Fran?ois Natali in branch '3.2':
Issue #13894: Skip test_threading.CRLockTests if _CRLock isn't available. Patch
http://hg.python.org/cpython/rev/5cf181df7bea

New changeset 279b6d778663 by Charles-Fran?ois Natali in branch 'default':
Issue #13894: Skip test_threading.CRLockTests if _CRLock isn't available. Patch
http://hg.python.org/cpython/rev/279b6d778663

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13894>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:38:16 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 10:38:16 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
Message-ID: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

Since changeset b99c54acb22d, the ARM Ubuntu buildbot has been hanging in test_ssl:

    http://www.python.org/dev/buildbot/all/builders/ARM%20Ubuntu%203.x/builds/250/steps/test/logs/stdio

I've been able to reproduce this behaviour on my AMD64 Ubuntu 11.10 machine.
The last output before it hangs is:

    test_protocol_sslv3 (test.test_ssl.ThreadedTests)
    Connecting to an SSLv3 server with various client options ... 
     SSLv3->SSLv3 CERT_NONE
     SSLv3->SSLv3 CERT_OPTIONAL
     SSLv3->SSLv3 CERT_REQUIRED
     {SSLv23->SSLv3} CERT_NONE

No such problem has shown up on the x86 Ubuntu Shared builder. This is probably
because it runs Ubuntu 8.04, whereas the ARM bot runs 11.10.

----------
messages: 152156
nosy: barry, giampaolo.rodola, janssen, nadeem.vawda, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: test_ssl hangs on Ubuntu
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 11:39:15 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 28 Jan 2012 10:39:15 +0000
Subject: [issue13894] threading._CRLock should not be tested if _thread.RLock
	isn't implemented
In-Reply-To: <1327740242.45.0.88821278342.issue13894@psf.upfronthosting.co.za>
Message-ID: <1327747155.44.0.591004130806.issue13894@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Committed, thanks!

----------
components: +Tests -Library (Lib)
nosy: +neologix
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13894>
_______________________________________

From report at bugs.python.org  Sat Jan 28 12:02:32 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 11:02:32 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>
Message-ID: <1327748552.77.0.376299742598.issue13895@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

The ARM Ubuntu 3.2 buildbot is having the same problem:

    http://www.python.org/dev/buildbot/all/builders/ARM%20Ubuntu%203.2/builds/158/steps/test/logs/stdio

The 2.7 bot is fine, though.

----------
versions: +Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 12:12:30 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sat, 28 Jan 2012 11:12:30 +0000
Subject: [issue13896] shelf doesn't work with 'with'
Message-ID: <1327749150.13.0.147725255938.issue13896@psf.upfronthosting.co.za>


New submission from Filip Gruszczy?ski <gruszczy at gmail.com>:

It would be cool if use could use with to manage shelf object and forget about close.

----------
messages: 152159
nosy: gruszczy
priority: normal
severity: normal
status: open
title: shelf doesn't work with 'with'

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13896>
_______________________________________

From report at bugs.python.org  Sat Jan 28 12:12:57 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sat, 28 Jan 2012 11:12:57 +0000
Subject: [issue13896] shelf doesn't work with 'with'
In-Reply-To: <1327749150.13.0.147725255938.issue13896@psf.upfronthosting.co.za>
Message-ID: <1327749177.01.0.667343190693.issue13896@psf.upfronthosting.co.za>


Changes by Filip Gruszczy?ski <gruszczy at gmail.com>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24347/13896.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13896>
_______________________________________

From report at bugs.python.org  Sat Jan 28 13:05:02 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sat, 28 Jan 2012 12:05:02 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327752302.86.0.140716337992.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I finished off the binary data section, so the first draft of the update is now complete in the bitbucket repo.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sat Jan 28 15:13:16 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 28 Jan 2012 14:13:16 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327759996.33.0.0190288037357.issue10181@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

I've been busy fixing the memoryview.h issues first. Could you take a
look at:

http://hg.python.org/features/pep-3118/file/f020716704d4/Include/memoryobject.h

Changes:

   a) Make all functions and the two buffer access macros part
      of the limited API again.

   b) Make the managed buffer API definitely private.

   c) Make PyBUF_MAX_NDIM=64 part of the official buffer API.


I think it might be OK to defer the decision about Py_MEMORYVIEW_C etc.,
since the comment already says "... Don't access their fields directly.".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 15:40:29 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sat, 28 Jan 2012 14:40:29 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1327761629.58.0.276254762892.issue13817@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

And here's the test.
To sum up:
- reinit_tls.diff makes sure to call PyThread_ReInitTLS() at the beginning of PyOS_AfterFork(), so that the TLS API is usable within PyOS_AfterFork() (e.g. in _after_fork()). It would be applied to 3.2 and default, even though they are not affected (at least on platform withs a native TLS implementation)
- reinit_tls_test.diff would be applied to 2.7, 3.2 and default: it should not trigger a failure on any of the active branches (2.7, 3.2 and default), it's just to prevent potential future regressions.
Notes regarding the test:
- I replaced subprocess with bare fork(), since it's way cheaper than spawning an entire subprocess
- I reduced the number of threads compared to Christoph's reproducer, because I feel uncomfortable forking up to 1024 processes in parallel (even though in practise they won't be that much, because child processes will terminate before the other threads call fork())
- even with 2.7.2, it won't deadlock 100%, since the race window is really short (but having a failure on one of the buildbots once would be enough to detect the problem, should it ever resurface)

----------
versions: +Python 2.7
Added file: http://bugs.python.org/file24348/reinit_tls_test.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Sat Jan 28 15:42:05 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 14:42:05 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>
Message-ID: <1327761725.65.0.23963068663.issue13895@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Can you try the following patch?

----------
keywords: +patch
Added file: http://bugs.python.org/file24349/test_ssl.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 16:09:09 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 15:09:09 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>
Message-ID: <1327763349.08.0.80105447072.issue13895@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

With the patch applied on 3.2 (it doesn't apply on default), the test passes,
but it also prints out this exception traceback:

    test_protocol_sslv3 (test.test_ssl.ThreadedTests)
    Connecting to an SSLv3 server with various client options ... 
     SSLv3->SSLv3 CERT_NONE
     SSLv3->SSLv3 CERT_OPTIONAL
     SSLv3->SSLv3 CERT_REQUIRED
     {SSLv23->SSLv3} CERT_NONE
    Exception in thread Thread-8:
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/3.2/Lib/threading.py", line 741, in _bootstrap_inner
        self.run()
      File "/home/nadeem/src/cpython/3.2/Lib/test/test_ssl.py", line 810, in run
        if not self.wrap_conn():
      File "/home/nadeem/src/cpython/3.2/Lib/test/test_ssl.py", line 764, in wrap_conn
        self.sock, server_side=True)
      File "/home/nadeem/src/cpython/3.2/Lib/ssl.py", line 189, in wrap_socket
        _context=self)
      File "/home/nadeem/src/cpython/3.2/Lib/ssl.py", line 276, in __init__
        raise x
      File "/home/nadeem/src/cpython/3.2/Lib/ssl.py", line 272, in __init__
        self.do_handshake()
      File "/home/nadeem/src/cpython/3.2/Lib/ssl.py", line 451, in do_handshake
        self._sslobj.do_handshake()
    socket.error: [Errno 104] Connection reset by peer

     {TLSv1->SSLv3} CERT_NONE
     SSLv23->SSLv3 CERT_NONE
    ok

I'm assuming it's unrelated, though - I get the same traceback on earlier
revisions (before the test started hanging). Should I file a separate bug?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 16:37:51 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Sat, 28 Jan 2012 15:37:51 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327765071.65.0.162653263941.issue13874@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

Nothing. I just read the sourcecode :-). I would add an "y=0" in the declaration too, just for aestetics.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:00:35 2012
From: report at bugs.python.org (Chen Huang)
Date: Sat, 28 Jan 2012 16:00:35 +0000
Subject: [issue10910] pyport.h FreeBSD/Mac OS X "fix" causes errors in C++
	compilation
In-Reply-To: <1295040100.28.0.0907494738582.issue10910@psf.upfronthosting.co.za>
Message-ID: <1327766435.69.0.752531239877.issue10910@psf.upfronthosting.co.za>


Chen Huang <chinux at gmail.com> added the comment:

I am experiencing the same issue. Googled for a bit but couldn't find much with the error, then changed the search context, come across to this thread. 


In file included from /opt/local/include/boost/date_time/gregorian/parsers.hpp:13,
                 from /opt/local/include/boost/date_time/gregorian/gregorian.hpp:34,
                 from /opt/local/include/boost/date_time/posix_time/time_formatters.hpp:12,
                 from /opt/local/include/boost/date_time/posix_time/posix_time.hpp:24,
                 from /opt/local/include/boost/date_time/local_time/local_time.hpp:11,
                 from /opt/local/include/boost/date_time.hpp:15,
                 from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.h:11,
                 from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.cpp:12:
/opt/local/include/boost/date_time/date_parsing.hpp:50:33: error: macro "tolower" passed 2 arguments, but takes just 1
In file included from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.cpp:12:
/Users/chen/GitRepo/Cpp_EmbeddingPython/Device.h:13:21: error: Devices.h: No such file or directory
In file included from /opt/local/include/boost/date_time/gregorian/parsers.hpp:13,
                 from /opt/local/include/boost/date_time/gregorian/gregorian.hpp:34,
                 from /opt/local/include/boost/date_time/posix_time/time_formatters.hpp:12,
                 from /opt/local/include/boost/date_time/posix_time/posix_time.hpp:24,
                 from /opt/local/include/boost/date_time/local_time/local_time.hpp:11,
                 from /opt/local/include/boost/date_time.hpp:15,
                 from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.h:11,
                 from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.cpp:12:
/opt/local/include/boost/date_time/date_parsing.hpp: In function ?std::string boost::date_time::convert_to_lower(std::string)?:
/opt/local/include/boost/date_time/date_parsing.hpp:50: error: cannot resolve overloaded function ?tolower? based on conversion to type ?char?
In file included from /Users/chen/GitRepo/Cpp_EmbeddingPython/Device.cpp:12:

----------
nosy: +Chen.Huang

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10910>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:24:48 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sat, 28 Jan 2012 16:24:48 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out of
	frame into generator/threadstate
Message-ID: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

The frame object has a number of fields which belong to the generator object.
By creating a CoState struct, these fields can be moved to generator/threadstate where they belong.

The interpreter no longer has to swap around exception state whenever generators are entered/exited.
The frame object is made more compact, reducing allocation/deallocation overhead (although benchmarking showed no significant difference) 

The attached patch reduces the code base by about 50 LOC overall.

----------
components: Interpreter Core
files: coro.patch
keywords: patch
messages: 152167
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Move fields relevant to coroutine/generators out of frame into generator/threadstate
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24350/coro.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:40:05 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 16:40:05 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327763349.08.0.80105447072.issue13895@psf.upfronthosting.co.za>
Message-ID: <1327768670.8904.0.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I'm assuming it's unrelated, though - I get the same traceback on earlier
> revisions (before the test started hanging). Should I file a separate bug?

Yes, please.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:40:25 2012
From: report at bugs.python.org (Chen Huang)
Date: Sat, 28 Jan 2012 16:40:25 +0000
Subject: [issue10910] pyport.h FreeBSD/Mac OS X "fix" causes errors in C++
	compilation
In-Reply-To: <1295040100.28.0.0907494738582.issue10910@psf.upfronthosting.co.za>
Message-ID: <1327768825.62.0.787819777375.issue10910@psf.upfronthosting.co.za>


Chen Huang <chinux at gmail.com> added the comment:

if I put "#include <cctype>" before boost/date_time, it solves my problem.

And by the way, i can't seem to get to the point where boost/week_ptr.hpp causes the error. I tried to compile Nasos's upload test-boost-python. That codes works on my machine.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10910>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:42:41 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 28 Jan 2012 16:42:41 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>
Message-ID: <E1RrBMO-0001yY-6a@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 10c79f29e417 by Antoine Pitrou in branch '3.2':
Issue #13895: fix test_ssl hanging under Ubuntu
http://hg.python.org/cpython/rev/10c79f29e417

New changeset db0c717cbc01 by Antoine Pitrou in branch 'default':
Issue #13895: fix test_ssl hanging under Ubuntu
http://hg.python.org/cpython/rev/db0c717cbc01

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:43:16 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 16:43:16 +0000
Subject: [issue13895] test_ssl hangs on Ubuntu
In-Reply-To: <1327747096.81.0.709589342449.issue13895@psf.upfronthosting.co.za>
Message-ID: <1327768996.97.0.197125027658.issue13895@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Hopefully it's fixed now, thank you.

----------
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13895>
_______________________________________

From report at bugs.python.org  Sat Jan 28 17:52:44 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 16:52:44 +0000
Subject: [issue13898] Ignored exception in test_ssl
Message-ID: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

While running test_ssl in verbose mode, I noticed the following exception:

    [...]
    test_protocol_sslv3 (test.test_ssl.ThreadedTests)
    Connecting to an SSLv3 server with various client options ... 
     SSLv3->SSLv3 CERT_NONE
     SSLv3->SSLv3 CERT_OPTIONAL
     SSLv3->SSLv3 CERT_REQUIRED
     {SSLv23->SSLv3} CERT_NONE
    Exception in thread Thread-54:
    Traceback (most recent call last):
      File "/home/nadeem/src/cpython/def/Lib/threading.py", line 730, in _bootstrap_inner
        self.run()
      File "/home/nadeem/src/cpython/def/Lib/test/test_ssl.py", line 926, in run
        if not self.wrap_conn():
      File "/home/nadeem/src/cpython/def/Lib/test/test_ssl.py", line 880, in wrap_conn
        self.sock, server_side=True)
      File "/home/nadeem/src/cpython/def/Lib/ssl.py", line 206, in wrap_socket
        _context=self)
      File "/home/nadeem/src/cpython/def/Lib/ssl.py", line 293, in __init__
        raise x
      File "/home/nadeem/src/cpython/def/Lib/ssl.py", line 289, in __init__
        self.do_handshake()
      File "/home/nadeem/src/cpython/def/Lib/ssl.py", line 489, in do_handshake
        self._sslobj.do_handshake()
    ConnectionResetError: [Errno 104] Connection reset by peer

     {TLSv1->SSLv3} CERT_NONE
     SSLv23->SSLv3 CERT_NONE
    ok
    [...]

Note that the exception is ignored and the test still passes.

----------
components: Tests
messages: 152172
nosy: giampaolo.rodola, janssen, nadeem.vawda, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: Ignored exception in test_ssl
type: behavior
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 18:19:18 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 28 Jan 2012 17:19:18 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327771158.38.0.886998726481.issue10181@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Nick Coghlan <report at bugs.python.org> wrote:
> Radical suggestion: make it public as collections.simple_ndarray?

Heh, that's a nice one. :)


While I think that the code in _testbuffer.c is sound and very well tested,
the API is low-level and optimized for testing all sorts of quirks.

Examples: _testbuffer.ndarray has the questionable capability of changing
views while buffers are exported. This is to test the pathological case
that came up in the discussion.

Then, similar to NumPy's ndarray, _testbuffer.ndarray constructs arrays
from a flat list. This is a bit clumsy but quite suitable for testing.

NumPy's ndarray is also the low level API. I think most people use
the high level array:

>>> a = array([[1,2], [3,4]], dtype="L")
>>> a
array([[1, 2],
       [3, 4]], dtype=uint64)

So it would take some effort to polish the API.


Meanwhile, to eliminate the use of _testbuffer in the documentation, I think
the following might work:

I've just added complete support for displaying multi-dimensional lists and
multi-dimensional comparisons to memoryobject.c in my private repo.

The code is quite succinct and follows exactly the same pattern as
copy_base/copy_rec.


Then, in the documentation we can use cast() and memoryview.tolist(),
but add a comment that cast() is just used for demonstration purposes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 18:29:12 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 17:29:12 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327771158.38.0.886998726481.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327771617.8904.3.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Nick Coghlan <report at bugs.python.org> wrote:
> > Radical suggestion: make it public as collections.simple_ndarray?
> 
> Heh, that's a nice one. :)

Well, the question is: does it have a point? Is this ndarray useful
outside of any third-party infrastructure such as the APIs offered by
numpy?

You might answer that we already have array.array but ISTM that
array.array is quite useless in general :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 18:51:37 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 28 Jan 2012 17:51:37 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327771617.8904.3.camel@localhost.localdomain>
Message-ID: <20120128175136.GA3074@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Antoine Pitrou <report at bugs.python.org> wrote:
> > Nick Coghlan <report at bugs.python.org> wrote:
> > > Radical suggestion: make it public as collections.simple_ndarray?
> > 
> > Heh, that's a nice one. :)
> 
> Well, the question is: does it have a point? Is this ndarray useful
> outside of any third-party infrastructure such as the APIs offered by
> numpy?

I think it would be mainly educational. It's easier to understand
the whole (multi-dimensional) point of PEP-3118 when there's a
concrete object that you can play with. Of course serious users
would go straight to NumPy.

The other issue is that's it's slightly strange to have a fully
featured memoryview with no object in the stdlib that can utilize
all features (at least the testing side is now complete).

Anyway, right now I don't know myself if I'm for or against it. :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 19:31:19 2012
From: report at bugs.python.org (Ralph Corderoy)
Date: Sat, 28 Jan 2012 18:31:19 +0000
Subject: [issue1633941] for line in sys.stdin: doesn't notice EOF the first
	time
Message-ID: <1327775479.85.0.714378132553.issue1633941@psf.upfronthosting.co.za>


Ralph Corderoy <ralph-pythonbugs at inputplus.co.uk> added the comment:

This most certainly is a bug under Unix and an annoying one.  "Since the
issue is with platform's stdio library" is wrong;  stdio is being used
incorrectly.  It would be nice to see it fixed in the 2.x line.

I've two test programs.

    $ head -42 stdin2.6 stdin3.1
    ==> stdin2.6 <==
    #! /usr/bin/python2.6

    import sys

    for l in sys.stdin:
        print repr(l)
    print 'end'

    ==> stdin3.1 <==
    #! /usr/bin/python3.1

    import sys

    for l in sys.stdin:
        print(repr(l))
    print('end')

    $

For both of them I will type "1 Enter 2 Enter 3 Enter Ctrl-D" without
the spaces, Ctrl-D being my tty's EOF, stty -a.

    $ ./stdin2.6
    1
    2
    3
    '1\n'
    '2\n'
    '3\n'

On the EOF the first iteration of sys.stdin returns and then so do the
others with the buffered lines.  The loop doesn't terminate, a second
Ctrl-D is required, giving.

    end
    $

Next,

    $ ./stdin3.1
    1
    '1\n'
    2
    '2\n'
    3
    '3\n'
    end
    $

perfect output.  Only one Ctrl-D required and better still each line is
returned as it's entered.

ltrace shows python2.6 uses fread(3).  I'm assuming it treats only a
zero return as EOF whereas whenever the return value is less than the
number of requested elements, EOF could have been reached;  feof(3) must
be called afterwards to decide.  Really, ferror(3) should also be called
to see if, as well as returning some elements, an error was detected.

It's this lack of feof() that means the second fread() is required to
trigger the flawed `only 0 return is EOF' logic.

Here's some C that shows stdio works fine if feof() and ferror() are
combined with fread().

    #include <stdio.h>

    int main(void)
    {
        unsigned char s[8192], *p;
        size_t n;

        while ((n = fread(s, 1, 8192, stdin))) {
            printf("%zd", n);
            p = s;
            while (n--)
                printf(" %02hhx", *p++);
            putchar('\n');

            if (feof(stdin)) {
                puts("end");
                break;
            }
            if (ferror(stdin)) {
                fputs("error", stderr);
                return 1;
            }
        }

        return 0;
    }

----------
nosy: +ralph.corderoy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1633941>
_______________________________________

From report at bugs.python.org  Sat Jan 28 19:46:04 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 18:46:04 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327776364.96.0.771954108755.issue13898@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

What is the OpenSSL version? (printed at the beginning of test_ssl)

Having an exception here is quite normal: the test checks that a certain (client protocol, server protocol) combination fails. However, a SSLError is expected rather than an OSError.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 19:50:44 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 18:50:44 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327776644.98.0.730811220812.issue13898@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

> What is the OpenSSL version?

    test_ssl: testing with 'OpenSSL 1.0.0e 6 Sep 2011' (1, 0, 0, 5, 15)
              under Linux ('debian', 'wheezy/sid', '')
              HAS_SNI = True

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 19:52:17 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 18:52:17 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327776644.98.0.730811220812.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327776602.8904.6.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> > What is the OpenSSL version?
> 
>     test_ssl: testing with 'OpenSSL 1.0.0e 6 Sep 2011' (1, 0, 0, 5, 15)
>               under Linux ('debian', 'wheezy/sid', '')
>               HAS_SNI = True

For the record, it works here under 'OpenSSL 1.0.0d 8 Feb 2011'.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 19:55:11 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 18:55:11 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327776911.98.0.548900592748.issue13898@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The changelog between 1.0.0d and 1.0.0e doesn't seem to list anything which could affect this test:

  *) Fix bug where CRLs with nextUpdate in the past are sometimes accepted
     by initialising X509_STORE_CTX properly. (CVE-2011-3207)
     [Kaspar Brand <ossl at velox.ch>]

  *) Fix SSL memory handling for (EC)DH ciphersuites, in particular
     for multi-threaded use of ECDH. (CVE-2011-3210)
     [Adam Langley (Google)]

  *) Fix x509_name_ex_d2i memory leak on bad inputs.
     [Bodo Moeller]

  *) Remove hard coded ecdsaWithSHA1 signature tests in ssl code and check
     signature public key algorithm by using OID xref utilities instead.
     Before this you could only use some ECC ciphersuites with SHA1 only.
     [Steve Henson]

  *) Add protection against ECDSA timing attacks as mentioned in the paper
     by Billy Bob Brumley and Nicola Tuveri, see:

	http://eprint.iacr.org/2011/232.pdf

(from http://www.openssl.org/news/changelog.html)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 20:02:16 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 19:02:16 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327777336.51.0.299603618401.issue13898@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

Might it be a distribution-specific issue, then? I'm running Ubuntu and
IIRC you're using Mageia?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 20:05:14 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 19:05:14 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327777336.51.0.299603618401.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327777378.8904.7.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Might it be a distribution-specific issue, then? I'm running Ubuntu and
> IIRC you're using Mageia?

Yes, I'm using Mageia. It would be nice if you could take a look at
Ubuntu's or Debian's OpenSSL patches (if any).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 20:26:04 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 28 Jan 2012 19:26:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327727572.2219.95.camel@surprise>
Message-ID: <1327778728.2219.115.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

This turns out to pass without PYTHONHASHRANDOMIZATION in the
environment, and fail intermittently with it.

Note that "make test" invokes the built python with "-E", so that it
ignores the setting of PYTHONHASHRANDOMIZATION in the environment.

Barry, Benjamin: does fixing this bug require getting the full test
suite to pass with randomization enabled (and fixing the intermittent
failures due to ordering issues), or is it acceptable to "merely" have
full passes without randomizing the hashes?

What do the buildbots do?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 28 20:50:07 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sat, 28 Jan 2012 19:50:07 +0000
Subject: [issue13898] Ignored exception in test_ssl
In-Reply-To: <1327769564.73.0.643604928649.issue13898@psf.upfronthosting.co.za>
Message-ID: <1327780207.72.0.0549839655353.issue13898@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

I can't see anything in the Ubuntu patches (available from
<https://launchpad.net/ubuntu/+source/openssl/1.0.0e-2ubuntu4>)
that relate to networking or handshakes, so maybe it's not that.
I could be wrong, though - I've never used OpenSSL myself.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13898>
_______________________________________

From report at bugs.python.org  Sat Jan 28 21:03:27 2012
From: report at bugs.python.org (Zachary Richey)
Date: Sat, 28 Jan 2012 20:03:27 +0000
Subject: [issue13896] shelf doesn't work with 'with'
In-Reply-To: <1327749150.13.0.147725255938.issue13896@psf.upfronthosting.co.za>
Message-ID: <1327781007.13.0.582137864108.issue13896@psf.upfronthosting.co.za>


Zachary Richey <mach1723 at gmail.com> added the comment:

The docs for shelve(see: http://docs.python.org/dev/library/shelve.html#shelve.open) recommend using contextlib.closing for this purpose.

----------
nosy: +zachary r.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13896>
_______________________________________

From report at bugs.python.org  Sat Jan 28 21:05:10 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sat, 28 Jan 2012 20:05:10 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327781110.64.0.777796899569.issue13703@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

I think we don't need to mess with tests in 2.6/3.1, but everything should pass under 2.7 and 3.2.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sat Jan 28 21:26:50 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Sat, 28 Jan 2012 20:26:50 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1327782410.72.0.868802638304.issue13496@psf.upfronthosting.co.za>


Changes by Mark Dickinson <dickinsm at gmail.com>:


----------
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Sat Jan 28 21:48:34 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sat, 28 Jan 2012 20:48:34 +0000
Subject: [issue11437] IDLE crash on startup with typo in config-keys.cfg
In-Reply-To: <1299543243.52.0.970222997696.issue11437@psf.upfronthosting.co.za>
Message-ID: <1327783714.67.0.272951418194.issue11437@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

Attached is a patch against 3.3a0 to validate the active key set in the configuration handler. It validates each key binding and then presents a dialog with a list of incorrect keys if there are any. 

In developing this patch I found that IDLE calls GetCurrentKeySet multiple times during initialization of an EditorWindow or a PyShell, once for each extension loaded. This inefficiency leads to long start times for IDLE.

----------
keywords: +patch
Added file: http://bugs.python.org/file24351/issue11437.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11437>
_______________________________________

From report at bugs.python.org  Sat Jan 28 21:59:27 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 20:59:27 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327759996.33.0.0190288037357.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327784231.8904.10.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

>    a) Make all functions and the two buffer access macros part
>       of the limited API again.

Hmm, I don't think buffer access macros should be part of the limited
API. If they are truely important (which I doubt), we should have
equivalent functions for them.

> I think it might be OK to defer the decision about Py_MEMORYVIEW_C etc.,
> since the comment already says "... Don't access their fields directly.".

My question is whether there is any point in making these flags. Does
3rd-party code want to manipulate memoryview internals, instead of
querying the Py_buffer?

(and of course the memoryview object is becoming more and more like
another Py_buffer :-))

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sat Jan 28 22:07:06 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sat, 28 Jan 2012 21:07:06 +0000
Subject: [issue5707] IDLE will not load
In-Reply-To: <1239003620.62.0.108807854341.issue5707@psf.upfronthosting.co.za>
Message-ID: <1327784826.11.0.422956203381.issue5707@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

Amaury's patch has already been applied in (d31be8c9c8a2).

GUO's description does cause an error. TclTk does not allow "<Control>" by itself to be bounded. This underlying cause for that error is addressed in #6739

I think this issue should be closed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5707>
_______________________________________

From report at bugs.python.org  Sat Jan 28 22:07:35 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sat, 28 Jan 2012 21:07:35 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <E1RrFUj-0003Au-SL@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 77188bc37c74 by Antoine Pitrou in branch '3.2':
Issue #13806: The size check in audioop decompression functions was too strict and could reject valid compressed data.
http://hg.python.org/cpython/rev/77188bc37c74

New changeset ce89dcba008f by Antoine Pitrou in branch 'default':
Issue #13806: The size check in audioop decompression functions was too strict and could reject valid compressed data.
http://hg.python.org/cpython/rev/ce89dcba008f

New changeset f1ee3bb6ba64 by Antoine Pitrou in branch '2.7':
Issue #13806: The size check in audioop decompression functions was too strict and could reject valid compressed data.
http://hg.python.org/cpython/rev/f1ee3bb6ba64

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Sat Jan 28 22:08:11 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 21:08:11 +0000
Subject: [issue13806] Audioop decompression frames size check fix
In-Reply-To: <1326813825.38.0.418557281447.issue13806@psf.upfronthosting.co.za>
Message-ID: <1327784891.58.0.541442039224.issue13806@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Thank you for your work Oleg! The patch is now committed.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13806>
_______________________________________

From report at bugs.python.org  Sat Jan 28 22:54:04 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sat, 28 Jan 2012 21:54:04 +0000
Subject: [issue13884] IDLE 2.6.5 Recent Files undocks
In-Reply-To: <1327621264.5.0.587291207239.issue13884@psf.upfronthosting.co.za>
Message-ID: <1327787644.0.0.181788331129.issue13884@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

Should tear-off menus still be a feature?  This relates to the #13504 meta-issue, in particular entry "4) ANNOYANCE: Get rid of the detachable menus feature.."

Attached is patch against 3.3a0 to disable tear-off menus. (It also makes modifications to Mac-specific code, but I can't test it since I don't have a Mac.)

----------
keywords: +patch
nosy: +serwy
Added file: http://bugs.python.org/file24352/issue13884.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13884>
_______________________________________

From report at bugs.python.org  Sat Jan 28 22:55:07 2012
From: report at bugs.python.org (Roger Serwy)
Date: Sat, 28 Jan 2012 21:55:07 +0000
Subject: [issue13504] Meta-issue for "Invent with Python" IDLE feedback
In-Reply-To: <1322613083.36.0.200533745275.issue13504@psf.upfronthosting.co.za>
Message-ID: <1327787707.21.0.484334843416.issue13504@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

#13884 has a patch to address point 4) ANNOYANCE: Get rid of the detachable menus feature.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13504>
_______________________________________

From report at bugs.python.org  Sat Jan 28 23:28:54 2012
From: report at bugs.python.org (John Machin)
Date: Sat, 28 Jan 2012 22:28:54 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
Message-ID: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>


New submission from John Machin <sjmachin at lexicon.net>:

Expected behaviour illustrated using "C":

>>> import re
>>> re.findall(r'[\C]', 'CCC')
['C', 'C', 'C']
>>> re.compile(r'[\C]', 128)
literal 67
<_sre.SRE_Pattern object at 0x01FC6E78>
>>> re.compile(r'C', 128)
literal 67
<_sre.SRE_Pattern object at 0x01FC6F08>

Incorrect behaviour exhibited by "A" (and by "B" and 
Z"):

>>> re.findall(r'[\A]', 'AAA')
[]
>>> re.compile(r'A', 128)
literal 65
<_sre.SRE_Pattern object at 0x01FC6F98>
>>> re.compile(r'[\A]', 128)
in
  at at_beginning_string #### FAIL ####
<_sre.SRE_Pattern object at 0x01FDF0B0>
>>>

Also there is no self-checking at runtime; the switch default has a comment to the effect that nothing can be done, so pretend that the unknown opcode matched nothing. Zen?

----------
messages: 152194
nosy: sjmachin
priority: normal
severity: normal
status: open
title: re pattern r"[\A]" should work like "A" but matches nothing. Ditto B and Z.
type: behavior
versions: Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sat Jan 28 23:42:34 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sat, 28 Jan 2012 22:42:34 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327790554.32.0.914804527022.issue13899@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

This happens because \A, \B and \Z are valid escape sequences[0].
If what you mean is that they shouldn't be recognized as such inside a character class, then I can agree with that.
^ and $ are similar to \A and \Z but they are considered as literals inside [].  I think the same could also be applied to \b and \B, unless you expect r'[\b]' to match the same as r'\b'.

(On an unrelated note, it's preferable to avoid using ints as flag -- using re.DEBUG is better)

[0]: http://docs.python.org/library/re.html#regular-expression-syntax

----------
nosy: +ezio.melotti, mrabarnett

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sat Jan 28 23:44:50 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sat, 28 Jan 2012 22:44:50 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327784231.8904.10.camel@localhost.localdomain>
Message-ID: <20120128224449.GA4160@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Antoine Pitrou <report at bugs.python.org> wrote:
> >    a) Make all functions and the two buffer access macros part
> >       of the limited API again.
> 
> Hmm, I don't think buffer access macros should be part of the limited
> API. If they are truely important (which I doubt), we should have
> equivalent functions for them.

I thought the whole Py_buffer API was only temporarily removed from the
limited API until the issues were sorted out:

http://bugs.python.org/issue10181#msg125462

For instance, here the macros are not excluded:

http://hg.python.org/cpython/file/3292cc222d5c/Include/memoryobject.h

Since the issues seem resolved in general, I thought it was time to
restore the previous state. [I just noticed that I didn't revert all
of Martin's changes, so xxlimited currently does not build in the
pep-3118 repo.]

As for the two macros specifically, I know Pauli was using them:

http://bugs.python.org/issue10181#msg139775

> > I think it might be OK to defer the decision about Py_MEMORYVIEW_C etc.,
> > since the comment already says "... Don't access their fields directly.".
> 
> My question is whether there is any point in making these flags. Does
> 3rd-party code want to manipulate memoryview internals, instead of
> querying the Py_buffer?

The flags are primarily for the memoryview itself to avoid repeated calls
to PyBuffer_IsContiguous(). So when 3rd-party uses PyMemoryView_GET_BUFFER
to get the view and also needs to determine the contiguity type, that
code could also use the flags.

Pauli: If you are still following the issue, do you think having access
to contiguity flags is useful for 3rd-party code or not?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:09:37 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 23:09:37 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <20120128224449.GA4160@sleipnir.bytereef.org>
Message-ID: <1327792039.4376.7.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Antoine Pitrou <report at bugs.python.org> wrote:
> > >    a) Make all functions and the two buffer access macros part
> > >       of the limited API again.
> > 
> > Hmm, I don't think buffer access macros should be part of the limited
> > API. If they are truely important (which I doubt), we should have
> > equivalent functions for them.
> 
> I thought the whole Py_buffer API was only temporarily removed from the
> limited API until the issues were sorted out:
> 
> http://bugs.python.org/issue10181#msg125462

I'm talking about the memoryview access macros. It's like
PyList_GET_SIZE, it isn't part of the limited API and you have to use
PyList_GetSize instead.

The limited API promises binary (ABI) compatibility accross releases.
It's a very strong promise to make and we must be careful not to put too
much stuff in the limited API. I don't think the rule for inclusion is
"we should put everything useful in the limited API".

> For instance, here the macros are not excluded:
> 
> http://hg.python.org/cpython/file/3292cc222d5c/Include/memoryobject.h

It might be a mistake, then.

> As for the two macros specifically, I know Pauli was using them:
> 
> http://bugs.python.org/issue10181#msg139775

Well, Pauli might use them, but it just means his code isn't compatible
with the limited API, then.

> The flags are primarily for the memoryview itself to avoid repeated calls
> to PyBuffer_IsContiguous(). So when 3rd-party uses PyMemoryView_GET_BUFFER
> to get the view and also needs to determine the contiguity type, that
> code could also use the flags.

But why would 3rd-party code use PyMemoryView_GET_BUFFER instead of, for
example, PyObject_GetBuffer? You seldom have APIs which *expect* a
memoryview, I think. Instead, they would expect a buffer-compatible
object.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:11:12 2012
From: report at bugs.python.org (John Machin)
Date: Sat, 28 Jan 2012 23:11:12 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327792272.74.0.646706361787.issue13899@psf.upfronthosting.co.za>


John Machin <sjmachin at lexicon.net> added the comment:

@ezio: Of course the context is "inside a character class".

I expect r'[\b]' to act like r'\b' aka r'\x08' aka backspace because (1) that is the treatment applied to all other C-like control char escapes (2) the docs say so explicitly: "Inside a character range, \b represents the backspace character, for compatibility with Python?s string literals."

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:14:28 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sat, 28 Jan 2012 23:14:28 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327781110.64.0.777796899569.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327792429.2219.126.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-28 at 20:05 +0000, Benjamin Peterson wrote:
> Benjamin Peterson <benjamin at python.org> added the comment:
> 
> I think we don't need to mess with tests in 2.6/3.1, but everything should pass under 2.7 and 3.2.

New version of the patch for 3.1
  optin-hash-randomization-for-3.1-dmalcolm-2012-01-28-001.patch

This version adds a command-line flag to enable hash-randomization: -R
(given that the -E flag disables env vars and thus disabled
PYTHONHASHRANDOMIZATION). See [1] below

[Is there a convenient way to check the length of the usage messages in
Modules/main.c?  I see this comment:
   /* Long usage message, split into parts < 512 bytes */ ]

I reworded the documentation somewhat based on input from Barry and
Antoine.

Also adds a NEWS item.

Passes "make test" on this x86_64 Fedora 15 box, --with-pydebug, though
that's without randomization enabled (it just does it within individual
test cases that explicitly enable it).

No performance testing done yet (though hopefully similar to that of
Victor's patch; see msg151078)

No idea of the impact on Windows users (I don't have a windows dev box).
It still has the stuff from Victor's patch described in msg151158.

How is this looking?
Dave

[1] IRC transcript concerning "-R" follows:
<__ap__> dmalcolm: IMO it would be simpler if there was only one env var
(preferably not too clumsy to type)
<__ap__> also, despite being neither barry nor gutworth, I think the
test suite *should* pass with randomized hashes
<__ap__> :)
<dmalcolm> :)
<__ap__> also the failure you're having is a bit worrying, since
apparently it's not about dict ordering
<dmalcolm> PYTHONHASHSEED exists mostly for selftesting (also for
compat, if you absolutely need to reproduce a specific random dict
ordering)
<__ap__> ok
<__ap__> if -E suppresses hash randomization, I think we should also add
a command-line flag
<__ap__> -R seems untaken
<__ap__> also it'll make things easier for Windows users, I think

----------
Added file: http://bugs.python.org/file24353/optin-hash-randomization-for-3.1-dmalcolm-2012-01-28-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 73dad4940b88 Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/reference/datamodel.rst	Sat Jan 28 18:05:49 2012 -0500
@@ -1265,6 +1265,8 @@
    inheritance of :meth:`__hash__` will be blocked, just as if :attr:`__hash__`
    had been explicitly set to :const:`None`.
 
+   See also the :option:`-R` command-line option.
+
 
 .. method:: object.__bool__(self)
 
diff -r 73dad4940b88 Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/using/cmdline.rst	Sat Jan 28 18:05:49 2012 -0500
@@ -21,7 +21,7 @@
 
 When invoking Python, you may specify any of these options::
 
-    python [-bBdEhiOsSuvVWx?] [-c command | -m module-name | script | - ] [args]
+    python [-bBdEhiORsSuvVWx?] [-c command | -m module-name | script | - ] [args]
 
 The most common use case is, of course, a simple invocation of a script::
 
@@ -215,6 +215,30 @@
    Discard docstrings in addition to the :option:`-O` optimizations.
 
 
+.. cmdoption:: -R
+
+   Turn on "hash randomization, so that the :meth:`__hash__` values of str,
+   bytes and datetime objects are "salted" with an unpredictable random value.
+   Although they remain constant within an individual Python process, they
+   are not predictable between repeated invocations of Python.
+
+   This is intended to provide protection against a denial-of-service
+   caused by carefully-chosen inputs that exploit the worst case performance
+   of a dict lookup, O(n^2) complexity.  See:
+
+       http://www.ocert.org/advisories/ocert-2011-003.html
+
+   for details.
+
+   Changing hash values affects the order in which keys are retrieved from
+   a dict.  Although Python has never made guarantees about this ordering
+   (and it typically varies between 32-bit and 64-bit builds), enough
+   real-world code implicitly relies on this non-guaranteed behavior that
+   the randomization is disabled by default.
+
+   See also :envvar:`PYTHONHASHRANDOMIZATION`.
+
+
 .. cmdoption:: -s
 
    Don't add user site directory to sys.path
@@ -435,6 +459,25 @@
    import of source modules.
 
 
+.. envvar:: PYTHONHASHRANDOMIZATION
+
+   If this is set to a non-empty string it is equivalent to specifying the
+   :option:`-R` option.
+
+
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for generating the hash() of
+   the types covered by the :option:`-R` option (or its equivalent,
+   :envvar:`PYTHONHASHRANDOMIZATION`.
+
+   It is primarily intended for use in selftests for the interpreter, but
+   may perhaps be of use for reproducing a specific dict ordering.
+
+   It should be a decimal number in the range [0; 4294967295].  Specifying
+   the value 0 overrides the other setting, disabling the hash random salt.
+
+
 .. envvar:: PYTHONIOENCODING
 
    Overrides the encoding used for stdin/stdout/stderr, in the syntax
diff -r 73dad4940b88 Include/object.h
--- a/Include/object.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/object.h	Sat Jan 28 18:05:49 2012 -0500
@@ -473,6 +473,12 @@
 PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
+typedef struct {
+    long prefix;
+    long suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
+
 /* Helper for passing objects to printf and the like */
 #define PyObject_REPR(obj) _PyUnicode_AsString(PyObject_Repr(obj))
 
diff -r 73dad4940b88 Include/pydebug.h
--- a/Include/pydebug.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pydebug.h	Sat Jan 28 18:05:49 2012 -0500
@@ -19,6 +19,7 @@
 PyAPI_DATA(int) Py_DontWriteBytecodeFlag;
 PyAPI_DATA(int) Py_NoUserSiteDirectory;
 PyAPI_DATA(int) Py_UnbufferedStdioFlag;
+PyAPI_DATA(int) Py_HashRandomizationFlag;
 
 /* this is a wrapper around getenv() that pays attention to
    Py_IgnoreEnvironmentFlag.  It should be used for getting variables like
diff -r 73dad4940b88 Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pythonrun.h	Sat Jan 28 18:05:49 2012 -0500
@@ -174,6 +174,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r 73dad4940b88 Lib/os.py
--- a/Lib/os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/os.py	Sat Jan 28 18:05:49 2012 -0500
@@ -611,23 +611,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r 73dad4940b88 Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/regrtest.py	Sat Jan 28 18:05:49 2012 -0500
@@ -428,6 +428,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(tests)
diff -r 73dad4940b88 Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_cmd_line.py	Sat Jan 28 18:05:49 2012 -0500
@@ -190,6 +190,16 @@
             self.assertTrue(path1.encode('ascii') in stdout)
             self.assertTrue(path2.encode('ascii') in stdout)
 
+    def test_hash_randomization(self):
+        # Verify that -R enables hash randomization:
+        self.verify_valid_flag('-R')
+        hashes = []
+        for i in range(2):
+            code = 'print(hash("spam"))'
+            data, rc = self.start_python_and_exit_code('-R', '-c', code)
+            self.assertEqual(rc, 0)
+            hashes.append(data)
+        self.assertNotEqual(hashes[0], hashes[1])
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff -r 73dad4940b88 Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_hash.py	Sat Jan 28 18:05:49 2012 -0500
@@ -3,10 +3,14 @@
 #
 # Also test that hash implementations are inherited as expected
 
+import struct
 import unittest
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -118,10 +122,65 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+class RandomizationTestCase(unittest.TestCase):
+
+    # Examples of the various types having randomized hash:
+    test_reprs = [repr('abc'), repr(b'abc')]
+
+    def get_hash(self, repr_, randomization=None, seed=None):
+        env = {}
+        if randomization is not None:
+            env['PYTHONHASHRANDOMIZATION'] = str(randomization)
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'print(hash(%s))' % repr_,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+        self.assertEqual(hash(b""), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            known_hash_of_obj = 1453079729188098211
+        else:
+            known_hash_of_obj = -1600925533
+        for repr_ in self.test_reprs:
+            # Randomization is disabled by default:
+            self.assertEqual(self.get_hash(repr_), known_hash_of_obj)
+
+            # If enabled, it can still be disabled by setting the seed to 0:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=0),
+                             known_hash_of_obj)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        # Note that all types share the same values:
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        for repr_ in self.test_reprs:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=42),
+                             h)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        for repr_ in self.test_reprs:
+            run1 = self.get_hash(repr_, randomization=1)
+            run2 = self.get_hash(repr_, randomization=1)
+            self.assertNotEqual(run1, run2)
+
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
                               HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                              HashBuiltinsTestCase,
+                         RandomizationTestCase)
 
 
 if __name__ == "__main__":
diff -r 73dad4940b88 Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_os.py	Sat Jan 28 18:05:49 2012 -0500
@@ -9,6 +9,7 @@
 import sys
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 
 # Detect whether we're on a Linux system that uses the (now outdated
 # and unmaintained) linuxthreads threading library.  There's an issue
@@ -574,14 +575,33 @@
         f.close()
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(0)), 0)
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 class ExecTests(unittest.TestCase):
     @unittest.skipIf(USING_LINUXTHREADS,
diff -r 73dad4940b88 Lib/test/test_set.py
--- a/Lib/test/test_set.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_set.py	Sat Jan 28 18:05:49 2012 -0500
@@ -734,6 +734,17 @@
         if self.repr is not None:
             self.assertEqual(repr(self.set), self.repr)
 
+    def check_repr_against_values(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
     def test_print(self):
         try:
             fo = open(support.TESTFN, "w")
@@ -892,7 +903,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -903,7 +916,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{b'a', b'c', b'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +931,13 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
 
     def tearDown(self):
         warnings.filters = self.warning_filters
 
+    def test_repr(self):
+        self.check_repr_against_values()
+
 #==============================================================================
 
 def baditer():
diff -r 73dad4940b88 Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 20 11:23:02 2012 +0000
+++ b/Makefile.pre.in	Sat Jan 28 18:05:49 2012 -0500
@@ -305,6 +305,7 @@
 		Python/pymath.o \
 		Python/pystate.o \
 		Python/pythonrun.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r 73dad4940b88 Misc/NEWS
--- a/Misc/NEWS	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/NEWS	Sat Jan 28 18:05:49 2012 -0500
@@ -10,6 +10,12 @@
 Core and Builtins
 -----------------
 
+- Issue #13703: add -R command-line option and PYTHONHASHRANDOMIZATION (and
+  PYTHONHASHSEED) environment variables, to provide an opt-in way to protect
+  against denial of service attacks due to hash collisions within the dict
+  and set types. See http://www.ocert.org/advisories/ocert-2011-003.html
+  Patch by David Malcolm, based on work by Victor Stinner.
+
 Library
 -------
 
diff -r 73dad4940b88 Misc/python.man
--- a/Misc/python.man	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/python.man	Sat Jan 28 18:05:49 2012 -0500
@@ -34,6 +34,9 @@
 .B \-OO
 ]
 [
+.B \-R
+]
+[
 .B -Q
 .I argument
 ]
@@ -145,6 +148,18 @@
 .B \-OO
 Discard docstrings in addition to the \fB-O\fP optimizations.
 .TP
+.B \-R
+Turn on "hash randomization", so that the hash() values of str, bytes and
+datetime objects are "salted" with an unpredictable pseudo-random value.
+Although they remain constant within an individual Python process, they are
+not predictable between repeated invocations of Python.
+.IP
+This is intended to provide protection against a denial of service
+caused by carefully-chosen inputs that exploit the worst case performance
+of a dict lookup, O(n^2) complexity.  See
+http://www.ocert.org/advisories/ocert-2011-003.html
+for details.
+.TP
 .BI "\-Q " argument
 Division control; see PEP 238.  The argument must be one of "old" (the
 default, int/int and long/long return an int or long), "new" (new
@@ -403,6 +418,17 @@
 If this is set to a non-empty string it is equivalent to specifying
 the \fB\-v\fP option. If set to an integer, it is equivalent to
 specifying \fB\-v\fP multiple times. 
+.IP PYTHONHASHRANDOMIZATION
+If this is set to a non-empty string it is equivalent to specifying the
+\fB\-R\fP option.
+.IP PYTHONHASHSEED
+If this is set, it is used as a fixed seed for generating the hash() of
+the types covered by the \fB\-R\fP  (or its equivalent environment variable,
+\fB\PYTHONHASHRANDOMIZATION\fP).
+It is primarily intended for use in selftests for the interpreter, but
+may perhaps be of use for reproducing a specific dict ordering.
+It should be a decimal number in the range [0; 4294967295].  Specifying
+the value 0 overrides the other setting, disabling the hash random salt.
 .SH AUTHOR
 The Python Software Foundation: http://www.python.org/psf
 .SH INTERNET RESOURCES
diff -r 73dad4940b88 Modules/datetimemodule.c
--- a/Modules/datetimemodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/datetimemodule.c	Sat Jan 28 18:05:49 2012 -0500
@@ -2566,10 +2566,12 @@
     register long x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= len;
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
 
diff -r 73dad4940b88 Modules/main.c
--- a/Modules/main.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/main.c	Sat Jan 28 18:05:49 2012 -0500
@@ -47,7 +47,7 @@
 static int  orig_argc;
 
 /* command line options */
-#define BASE_OPTS L"bBc:dEhiJm:OsStuvVW:xX?"
+#define BASE_OPTS L"bBc:dEhiJm:ORsStuvVW:xX?"
 
 #define PROGRAM_OPTS BASE_OPTS
 
@@ -72,6 +72,9 @@
 -m mod : run library module as a script (terminates option list)\n\
 -O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x\n\
 -OO    : remove doc-strings in addition to the -O optimizations\n\
+-R     : use a pseudo-random salt to make hash() values of various types be\n\
+         unpredictable between separate invocations of the interpreter, as\n\
+         a defence against denial-of-service attacks\n\
 -s     : don't add user site directory to sys.path; also PYTHONNOUSERSITE\n\
 -S     : don't imply 'import site' on initialization\n\
 ";
@@ -99,6 +102,11 @@
 PYTHONCASEOK : ignore case in 'import' statements (Windows).\n\
 PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n\
 ";
+static char *usage_6 = "\
+PYTHONHASHRANDOMIZATION : make hash() values of various types be unpredictable\n\
+                          between separate invocations of the interpreter, as\n\
+                          a defence against denial-of-service attacks\n\
+";
 
 #ifndef MS_WINDOWS
 static FILE*
@@ -136,6 +144,7 @@
         fputs(usage_3, f);
         fprintf(f, usage_4, DELIM);
         fprintf(f, usage_5, DELIM, PYTHONHOMEHELP);
+        fputs(usage_6, f);
     }
 #if defined(__VMS)
     if (exitcode == 0) {
@@ -373,6 +382,10 @@
             PySys_AddWarnOption(_PyOS_optarg);
             break;
 
+        case 'R':
+            Py_HashRandomizationFlag++;
+            break;
+
         /* This space reserved for other options */
 
         default:
diff -r 73dad4940b88 Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/posixmodule.c	Sat Jan 28 18:05:49 2012 -0500
@@ -6942,82 +6942,6 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
-static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
-    }
-    return result;
-}
-#endif
-
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
 Return a string describing the encoding of the device\n\
@@ -7055,41 +6979,35 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+Return n pseudo-random bytes.");
+
+static PyObject *
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
+ 
+     /* Read arguments */
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -7374,12 +7292,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
     {NULL,              NULL}            /* Sentinel */
 };
 
diff -r 73dad4940b88 Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/bytesobject.c	Sat Jan 28 18:05:49 2012 -0500
@@ -899,11 +899,17 @@
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
+    if (len == 0) {
+        a->ob_shash = 0;
+        return 0;
+    }
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(a);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff -r 73dad4940b88 Objects/object.c
--- a/Objects/object.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/object.c	Sat Jan 28 18:05:49 2012 -0500
@@ -712,6 +712,8 @@
     return -1;
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
 long
 PyObject_Hash(PyObject *v)
 {
diff -r 73dad4940b88 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/unicodeobject.c	Sat Jan 28 18:05:49 2012 -0500
@@ -7344,11 +7344,17 @@
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
+    if (len == 0) {
+        self->hash = 0;
+        return 0;
+    }
     p = self->str;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(self);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     self->hash = x;
diff -r 73dad4940b88 PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 20 11:23:02 2012 +0000
+++ b/PCbuild/pythoncore.vcproj	Sat Jan 28 18:05:49 2012 -0500
@@ -1778,6 +1778,10 @@
 				RelativePath="..\Python\pythonrun.c"
 				>
 			</File>
+ 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
 			<File
 				RelativePath="..\Python\structmember.c"
 				>
diff -r 73dad4940b88 Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/pythonrun.c	Sat Jan 28 18:05:49 2012 -0500
@@ -73,6 +73,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 
 #ifdef WITH_THREAD
 extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@@ -91,6 +92,7 @@
 int Py_IgnoreEnvironmentFlag; /* e.g. PYTHONPATH, PYTHONHOME */
 int Py_NoUserSiteDirectory = 0; /* for -s and site.py */
 int Py_UnbufferedStdioFlag = 0; /* Unbuffered binary std{in,out,err} */
+int Py_HashRandomizationFlag = 0; /* for -R and PYTHONHASHRANDOMIZATION */
 
 /* PyModule_GetWarningsModule is no longer necessary as of 2.6
 since _warnings is builtin.  This API should not be used. */
@@ -195,6 +197,10 @@
         Py_OptimizeFlag = add_flag(Py_OptimizeFlag, p);
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
+    if ((p = Py_GETENV("PYTHONHASHRANDOMIZATION")) && *p != '\0')
+        Py_HashRandomizationFlag = add_flag(Py_HashRandomizationFlag, p);
+    
+    _PyRandom_Init();
 
     interp = PyInterpreterState_New();
     if (interp == NULL)
diff -r 73dad4940b88 Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Sat Jan 28 18:05:49 2012 -0500
@@ -0,0 +1,284 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Failed to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Failed to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Failed to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Failed to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Failed to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    /*
+      By default, hash randomization is disabled, and only
+      enabled if PYTHONHASHRANDOMIZATION is set to non-empty
+      or if "-R" is provided at the command line:
+    */
+    if (!Py_HashRandomizationFlag) {
+        /* Disable the randomized hash: */
+        memset(secret, 0, secret_size);
+        return;
+    }
+
+    /*
+      Hash randomization is enabled.  Generate a per-process secret,
+      using PYTHONHASHSEED if provided.
+    */
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else /* #ifdef MS_WINDOWS */
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+

From report at bugs.python.org  Sun Jan 29 00:24:42 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 23:24:42 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327792429.2219.126.camel@surprise>
Message-ID: <1327792945.4376.9.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Passes "make test" on this x86_64 Fedora 15 box, --with-pydebug, though
> that's without randomization enabled (it just does it within individual
> test cases that explicitly enable it).

I think you should check with randomization enabled, if only to see the
nature of the failures and if they are expected.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:34:32 2012
From: report at bugs.python.org (Chen Huang)
Date: Sat, 28 Jan 2012 23:34:32 +0000
Subject: [issue10910] pyport.h FreeBSD/Mac OS X "fix" causes errors in C++
	compilation
In-Reply-To: <1295040100.28.0.0907494738582.issue10910@psf.upfronthosting.co.za>
Message-ID: <1327793672.47.0.40318153623.issue10910@psf.upfronthosting.co.za>


Chen Huang <chinux at gmail.com> added the comment:

Nevermind, putting #include <cctype> doesn't help. I have to put <python.h> on top to get away this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10910>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:53:56 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Sat, 28 Jan 2012 23:53:56 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out of
	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1327794836.94.0.29143192128.issue13897@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

- Why is it called CoState?  is it related to coroutines?
- I'm a bit worried about the removal of tstate->exc_value, and the changes in sys.exc_info().  "tstate->exc_value" is used by many extension modules, do you really want to replace this?

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:56:25 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sat, 28 Jan 2012 23:56:25 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327792945.4376.9.camel@localhost.localdomain>
Message-ID: <4F248B01.4050702@udel.edu>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

> I think you should check with randomization enabled, if only to see the
> nature of the failures and if they are expected.

Including the list of when-enabled expected failures in the release 
notes would help those who compile and test.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 00:57:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sat, 28 Jan 2012 23:57:50 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out of
	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1327795070.7.0.145092622853.issue13897@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +benjamin.peterson, ncoghlan
stage:  -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 01:06:30 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Sun, 29 Jan 2012 00:06:30 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F248B01.4050702@udel.edu>
Message-ID: <1327795553.2219.128.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-28 at 23:56 +0000, Terry J. Reedy wrote:
> Terry J. Reedy <tjreedy at udel.edu> added the comment:
> 
> > I think you should check with randomization enabled, if only to see the
> > nature of the failures and if they are expected.
> 
> Including the list of when-enabled expected failures in the release 
> notes would help those who compile and test.

OK, though note that because it's random, I'll have to run it a few
times, and we'll see what shakes out.

Am running with:
$  make test TESTPYTHONOPTS=-R
leading to:
   ./python -E -bb -R ./Lib/test/regrtest.py -l 

BTW, I see:
  Testing with flags: sys.flags(debug=0, division_warning=0, inspect=0,
interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0,
no_site=0, ignore_environment=1, verbose=0, bytes_warning=2)

which doesn't list the new flag.  Should I add it to sys.flags?  (or
does anyone ever do tuple-unpacking of that PyStructSequence and thus
rely on the number of elements?)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 01:41:26 2012
From: report at bugs.python.org (Benjamin Peterson)
Date: Sun, 29 Jan 2012 00:41:26 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out of
	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1327797686.48.0.760738117994.issue13897@psf.upfronthosting.co.za>


Benjamin Peterson <benjamin at python.org> added the comment:

Extensions modules should really be using PyErr_Occurred/Fetch etc.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 02:09:54 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 01:09:54 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327799394.42.0.967372448776.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

I'm with Antoine here - we want to be *very* conservative with what we expose through the limited API. Due to the ABI compatibility promise, anything exposed that way is very hard to change.

Keeping things out of the limited API isn't really an issue - it just means that anyone that wants to use them needs to rebuild their extensions for each new version of the C API (just as they do now).

Addings things to the stable ABI later is easy though - that's why we had Martin take all the Py_buffer related stuff out of the initial version of the limited API.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 03:00:25 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 02:00:25 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out of
	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1327802425.89.0.202313759093.issue13897@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

The division of responsibilities between generator objects and the eval loop is currently a little messy. The eval loop deals almost entirely with frame objects and also handles swapping exception states around on behalf of generators, which is why the generator specific exception state currently lives on frame objects (i.e. to avoid the eval loop needing to know too much about the internal structure of generator objects).

Before generator related state can reasonably be moved out of the frame objects, those responsibilities should be divided more cleanly. Ron Adam has made an initially attempt at tackling that problem in issue 13607.

Once the responsibilities are divided appropriately, *then* we can look at the fields that ceval no longer touches and see if they should be moved somewhere else (whether that's directly into the generator struct or into a new struct referenced from the generator struct).

----------
nosy: +ron_adam

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 03:24:36 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 02:24:36 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1327803876.78.0.146738607935.issue10181@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Also, +1 to replacing _testbuffer with .cast() to demonstrate the multi-dimensional support. Without an actual multi-dimensional array object in the standard library, demonstrating those aspects is always going to be a little tricky. However, it's worth doing so that the likes of NumPy and PIL don't need to explain those interactions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 06:29:06 2012
From: report at bugs.python.org (Parker Phinney)
Date: Sun, 29 Jan 2012 05:29:06 +0000
Subject: [issue13900]
	http://docs.python.org/library/email.parser.html#email.parser.Parser.parse
Message-ID: <1327814946.33.0.356095199195.issue13900@psf.upfronthosting.co.za>


Changes by Parker Phinney <gameguy43 at gmail.com>:


----------
assignee: docs at python
components: Documentation
nosy: Parker.Phinney, docs at python
priority: normal
severity: normal
status: open
title: http://docs.python.org/library/email.parser.html#email.parser.Parser.parse
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 06:31:29 2012
From: report at bugs.python.org (Parker Phinney)
Date: Sun, 29 Jan 2012 05:31:29 +0000
Subject: [issue13900] documentation page on email.parser contains
	self-referential non-definition of headersonly parameter
Message-ID: <1327815089.31.0.591392873917.issue13900@psf.upfronthosting.co.za>


New submission from Parker Phinney <gameguy43 at gmail.com>:

to recreate:
go here:
http://docs.python.org/library/email.parser.html#email.parser.Parser.parse
notice this line:
"Optional headersonly is as with the parse() method."

But that doesn't make sense, because this /is/ the parse() method!

----------
title: http://docs.python.org/library/email.parser.html#email.parser.Parser.parse -> documentation page on email.parser contains self-referential non-definition of headersonly parameter
type: behavior -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 07:26:32 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 06:26:32 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327818392.2.0.900717970931.issue2636@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

As part of the PEP 408 discussions, Guido approved the addition of 'regex' in 3.3 (using that name, rather than as a drop-in replacement for re) [1,2]

That should greatly ease the backwards compatibility concerns, even if it isn't as transparent an upgrade path.

[1] http://mail.python.org/pipermail/python-dev/2012-January/115961.html
[2] http://mail.python.org/pipermail/python-dev/2012-January/115962.html

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 07:28:44 2012
From: report at bugs.python.org (Alex Gaynor)
Date: Sun, 29 Jan 2012 06:28:44 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1327818392.2.0.900717970931.issue2636@psf.upfronthosting.co.za>
Message-ID: <CAFRnB2VrFUxV55mxcH9acStvr80H2obB03KtVqTzdoXUWjGX7Q@mail.gmail.com>


Alex Gaynor <alex.gaynor at gmail.com> added the comment:

So, to my reading of teh compatibility PEP this cannot be added wholesale,
unless there is a pure Python version as well.  However, if it replaced re
(read: patched) it would be valid.

On Sun, Jan 29, 2012 at 1:26 AM, Nick Coghlan <report at bugs.python.org>wrote:

>
> Nick Coghlan <ncoghlan at gmail.com> added the comment:
>
> As part of the PEP 408 discussions, Guido approved the addition of 'regex'
> in 3.3 (using that name, rather than as a drop-in replacement for re) [1,2]
>
> That should greatly ease the backwards compatibility concerns, even if it
> isn't as transparent an upgrade path.
>
> [1] http://mail.python.org/pipermail/python-dev/2012-January/115961.html
> [2] http://mail.python.org/pipermail/python-dev/2012-January/115962.html
>
> ----------
> nosy: +ncoghlan
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue2636>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:24:38 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:24:38 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821878.03.0.385101856415.issue2636@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

I created a new sandbox branch to integrate regex into CPython, see "remote repo" field.

I mainly had to adapt the test suite to use unittest.

----------
hgrepos: +108

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:25:23 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:25:23 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821923.58.0.31422606141.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file10645/issue2636-patches.tar.bz2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:25:38 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:25:38 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821938.78.0.211178170184.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file10647/issue2636-02.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:25:49 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:25:49 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821949.07.0.358229272818.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file11657/issue2636-01+09-02+17_backport.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:26:05 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:26:05 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821965.87.0.212710455745.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file11669/issue2636+01+09-02+17+18+19+20+21+24+26_speedup.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:26:14 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:26:14 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821974.57.0.816483317775.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file12932/issue2636-features.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:26:24 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:26:24 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821984.25.0.180601142216.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file12974/issue2636-features-2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:26:37 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:26:37 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327821997.31.0.219371443869.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13165/issue2636-features-3.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:26:46 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:26:46 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822006.18.0.221347661797.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13185/issue2636-features-4.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:27:02 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:27:02 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822022.49.0.97229622258.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13216/issue2636-features-5.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:27:11 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:27:11 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822031.88.0.253792846543.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13257/issue2636-features-6.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:27:24 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:27:24 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822044.57.0.246882620761.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13449/issue2636-patch-1.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:27:38 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:27:38 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822058.12.0.417120493589.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file13707/issue2636-patch-2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:27:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:27:57 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822077.27.0.611835342737.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14570/issue2636-20090726.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:28:21 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:28:21 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822101.87.0.42592624088.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14576/issue2636-20090727.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:28:30 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:28:30 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822110.77.0.629619809473.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14594/issue2636-20090729.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:28:39 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:28:39 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822119.56.0.259882822591.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14642/issue2636-20090804.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:28:48 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:28:48 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822128.13.0.568418298183.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14682/issue2636-20090810.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:28:58 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:28:58 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822138.84.0.853993559896.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14683/issue2636-20090810#2.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:29:10 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:29:10 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822150.6.0.598924986165.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14687/issue2636-20090810#3.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:29:20 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:29:20 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822160.89.0.126267747716.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file14733/issue2636-20090815.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:29:30 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:29:30 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822170.13.0.808775653175.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file15901/issue2636-20100116.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:30:29 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:30:29 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822229.63.0.0936960936823.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16122/issue2636-20100204.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:30:42 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:30:42 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822242.49.0.0348007902742.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16195/issue2636-20100210.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:30:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:30:57 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822257.87.0.313481426495.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16205/issue2636-20100211.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:31:06 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:31:06 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822266.5.0.0500778277955.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16243/issue2636-20100217.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:31:15 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:31:15 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822275.79.0.979653297736.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16247/issue2636-20100218.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:31:28 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:31:28 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822288.98.0.305709999056.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16255/issue2636-20100219.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:31:40 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:31:40 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822300.96.0.514316242333.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16278/Features-backslashes.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:31:58 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:31:58 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822318.59.0.569915720647.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16327/issue2636-20100222.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:32:18 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:32:18 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822338.47.0.177456266241.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
nosy:  -akitada, akoumjian, alex, amaury.forgeotdarc, belopolsky, davide.rizzo, eric.snow, ezio.melotti, georg.brandl, giampaolo.rodola, gregory.p.smith, jacques, jaylogan, jhalcrow, jimjjewett, loewis, mark, mattchaput, moreati, mrabarnett, ncoghlan, nneonneo, pitrou, r.david.murray, ronnix, rsc, sjmachin, stevenjd, stiv, timehorse, vbr, zdwiel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:32:27 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:32:27 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822347.51.0.443407111157.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16331/issue2636-20100223.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:32:36 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:32:36 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822356.0.0.0404212683046.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16362/issue2636-20100224.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:32:44 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:32:44 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822364.46.0.819382544232.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16365/issue2636-20100225.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:32:56 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:32:56 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822376.34.0.104719476506.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16375/issue2636-20100226.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:33:05 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:33:05 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822385.39.0.529608762824.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16424/issue2636-20100304.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:33:14 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:33:14 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822394.21.0.232011946401.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16444/issue2636-20100305.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:33:22 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:33:22 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822402.67.0.285018657684.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16626/issue2636-20100323.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:33:31 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:33:31 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822411.25.0.991860472227.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16709/issue2636-20100331.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:33:43 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:33:43 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822423.09.0.504540570812.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16905/issue2636-20100413.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:34:28 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:34:28 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822468.39.0.540784697901.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16910/test_regex_20100413

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:34:37 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:34:37 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822477.62.0.120764663104.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16913/setup.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:34:45 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:34:45 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822485.88.0.0403115681256.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16914/build.log

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:34:56 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:34:56 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822496.45.0.853969184278.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file16916/issue2636-20100414.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:05 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:05 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822505.14.0.103083935906.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file17877/issue2636-20100706.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:16 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:16 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822516.08.0.139774207072.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file17912/issue2636-20100709.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:24 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:24 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822524.88.0.111533744651.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18054/issue2636-20100719.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:34 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:34 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822534.04.0.175156631523.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18192/issue2636-20100725.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:42 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:42 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822542.71.0.0138326422617.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18532/issue2636-20100814.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:35:51 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:35:51 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822551.07.0.44093970408.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18541/issue2636-20100816.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:02 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:02 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822562.75.0.844833621066.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file21144/unnamed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:17 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:17 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822577.1.0.0905653236975.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18621/issue2636-20100824.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:28 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:28 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822588.69.0.0134464428842.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18854/issue2636-20100912.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:38 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:38 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822598.08.0.668954136387.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18865/issue2636-20100913.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:46 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:46 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822606.84.0.223836837189.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file18913/issue2636-20100918.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:36:55 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:36:55 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822615.27.0.415781707178.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19168/issue2636-20101009.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:03 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:37:03 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822623.51.0.0570719339892.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19419/issue2636-20101029.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:12 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:37:12 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822632.19.0.582705959588.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19422/issue2636-20101030.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:27 2012
From: report at bugs.python.org (Ethan Furman)
Date: Sun, 29 Jan 2012 07:37:27 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327822647.05.0.686082736493.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

It looks like agreement is forming around the

    raise ... from None

method.  It has been mentioned more than once that having the context saved on the exception would be a Good Thing, and for further debugging (or logging or what-have-you) I must agree.

The patch attached now sets __cause__ to True, leaving __context__ unclobbered.  The exception printing routine checks to see if __cause__ is True, and if so simply skips the display of either cause or __context__, but __context__ can still be queried by later code.

One concern raised was that since it is possible to write (even before this patch)

    raise KeyError from NameError

outside of a try block that some would get into the habit of writing

    raise KeyError from None

as a way of preemptively suppressing implicit context chaining;  I am happy to report that this is not an issue, since when that exception is caught and a new exception raised, it is the new exception that controls the display.

In other words:

>>> try:
...   raise ValueError from None
... except:
...   raise NameError
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
NameError

----------
Added file: http://bugs.python.org/file24354/raise_from_none_v2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:36 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:37:36 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822656.04.0.0494369693078.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19435/issue2636-20101030a.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:48 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:37:48 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822668.61.0.214766097278.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19456/issue2636-20101101.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:37:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:37:57 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822677.43.0.898920082833.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19460/issue2636-20101102.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:06 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:06 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822686.55.0.0483419461504.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19469/issue2636-20101102a.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:14 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:14 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822694.92.0.70711796928.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19514/issue2636-20101106.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:26 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:26 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822706.22.0.94469472951.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19597/issue2636-20101113.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:35 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:35 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822715.24.0.5581040707.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19651/issue2636-20101120.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:43 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:43 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822723.98.0.898023999078.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19723/issue2636-20101121.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:38:52 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:38:52 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822732.58.0.307113201527.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19786/issue2636-20101123.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:00 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:00 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822740.59.0.148673518438.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19881/issue2636-20101130.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:09 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:09 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822749.35.0.224679002594.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file19965/issue2636-20101207.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:20 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:20 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822760.04.0.80461845893.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file20001/issue2636-20101210.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:28 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:28 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822768.74.0.847175094037.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file20154/issue2636-20101224.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:37 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:37 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822777.54.0.225511079716.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file20176/issue2636-20101228.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:45 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:45 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822785.92.0.473815472204.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file20182/issue2636-20101228a.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:39:54 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:39:54 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822794.0.0.628044892013.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file20185/issue2636-20101229.zip

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:40:06 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 07:40:06 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822806.14.0.756341722124.issue2636@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
nosy: +akitada, akoumjian, alex, amaury.forgeotdarc, belopolsky, davide.rizzo, eric.snow, ezio.melotti, georg.brandl, giampaolo.rodola, gregory.p.smith, jacques, jaylogan, jhalcrow, jimjjewett, loewis, mark, mattchaput, moreati, mrabarnett, ncoghlan, nneonneo, pitrou, r.david.murray, ronnix, rsc, sjmachin, stevenjd, stiv, timehorse, vbr, zdwiel

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 08:42:23 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 07:42:23 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327822943.42.0.870185036171.issue2636@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Alex has a valid point in relation to PEP 399, since, like lzma, regex will be coming in under the "special permission" clause that allows the addition of C extension modules without pure Python equivalents. Unlike lzma, though, the new regex engine isn't a relatively simple wrapper around an existing library - supporting the new API features on other implementations is going to mean a substantial amount of work.

In practice, I expect that a pure Python implementation of a regular expression engine would only be fast enough to be usable on PyPy. So while we'd almost certainly accept a patch that added a parallel Python implementation, I doubt it would actually help Jython or IronPython all that much - they're probably going to need versions written in Java and C# to be effective (as I believe they already have for the re module).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 09:01:44 2012
From: report at bugs.python.org (Devin Jeanpierre)
Date: Sun, 29 Jan 2012 08:01:44 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327824104.2.0.639258920542.issue2636@psf.upfronthosting.co.za>


Devin Jeanpierre <jeanpierreda at gmail.com> added the comment:

> In practice, I expect that a pure Python implementation of a regular expression engine would only be fast enough to be usable on PyPy.

Not sure why this is necessarily true. I'd expect a pure-Python implementation to be maybe 200 times as slow. Many queries (those on relatively short strings that backtrack little) finish within microseconds. On this scale, a couple of orders of magnitudes is not noticeable by humans (unless it adds up), and even where it gets noticeable, it's better than having nothing at all or a non-working program (up until a point).

python -m timeit -n 1000000 -s "import re; x = re.compile(r'.*<\s*help\s*>([^<]*)<\s*/\s*help.*>'); data = ' '*1000 + '< help >' + 'abc'*100 + '</help>'" "x.match(data)"
1000000 loops, best of 3: 3.27 usec per loop

----------
nosy: +Devin Jeanpierre

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 09:04:33 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 08:04:33 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327824273.13.0.407851383654.issue6210@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Ah, nice idea of bringing the boolean constants into the mix so we don't need to invent a new sentinel value.

However, to preserve the current behaviour that "raise X from Y" is essentially just syntactic sugar for: "_var = X; _var.__cause__ = Y; raise Y", I suggest setting the default value for "__cause__" to "False" (indicating "not set"). Then:

  "__cause__ is False" means "no cause set, display context"
  "__cause__ is None" means "cause explicitly set to None, suppress context"
  Any other value means "display cause"

The default value for cause is controlled by get_cause and set_cause in exceptions.c [1]. The basic idea would be to replace the current usage of Py_None and Py_RETURN_NONE in that code with Py_False and Py_RETURN_FALSE, and then change the criteria for valid causes to "arg != Py_None && !PyExceptionInstance_Check(arg)".

In addition to the files already touched by the patch, Lib/traceback.py [2] and its tests will also require updating.

[1] http://hg.python.org/cpython/file/default/Objects/exceptions.c#l293
[2] http://hg.python.org/cpython/file/default/Lib/traceback.py#l117

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 09:31:34 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 08:31:34 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327825894.16.0.781617775837.issue2636@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Well, REs are very often used to process large chunks of text by repeated application.  So if the whole operation takes 0.1 or 20 seconds you're going to notice :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 09:37:53 2012
From: report at bugs.python.org (Devin Jeanpierre)
Date: Sun, 29 Jan 2012 08:37:53 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1208260672.14.0.711874677361.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327826273.02.0.709366491224.issue2636@psf.upfronthosting.co.za>


Devin Jeanpierre <jeanpierreda at gmail.com> added the comment:

It'd be nice if we had some sort of representative benchmark for real-world uses of Python regexps. The JS guys have all pitched in to create such a thing for uses of regexps on thew web. I don't know of any such thing for Python.

I agree that a Python implementation wouldn't be useful for some cases. On the other hand, I believe it would be fine (or at least tolerable) for some others. I don't know the ratio between the two.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 09:46:45 2012
From: report at bugs.python.org (toggtc)
Date: Sun, 29 Jan 2012 08:46:45 +0000
Subject: [issue13901] test_get_outputs (test_distutils) failure with
	--enable-shared on Mac OS X
Message-ID: <1327826805.0.0.304314009246.issue13901@psf.upfronthosting.co.za>


New submission from toggtc <toggtc at gmail.com>:

Current 2.7.2 + building on OS X 10.7.2 and gcc 4.2.1 (Apple build 5666.3):

i686-apple-darwin11-gcc-4.2.1: /private/var/folders/jy/dhptnvj90b34s0135sb_g6w80000gn/T/tmpAfN6sj/foo.so: No such file or directory

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/tests/test_build_ext.py", line 291, in test_get_outputs
    cmd.run()
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 531, in build_extension
    target_lang=language)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/ccompiler.py", line 741, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/unixccompiler.py", line 258, in link
    raise LinkError, msg
LinkError: command 'gcc' failed with exit status 1
----

"DYLD_LIBRARY_PATH" is not defined in default environment of OS X.
but, ./configure and ./configure.in are:
    Darwin*)
    	LDLIBRARY='libpython$(VERSION).dylib'
	BLDLIBRARY='-L. -lpython$(VERSION)'
	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'

=> RUNSHARED=DYLD_LIBRARY_PATH=`pwd`:

and, test_build_ext.py:
 53     def _fixup_command(self, cmd):
 ...
 63         if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
 64             not sys.platform.startswith('win')):
 65             runshared = sysconfig.get_config_var('RUNSHARED')
 66             if runshared is None:
 67                 cmd.library_dirs = ['.']
 68             else:
 69                 name, equals, value = runshared.partition('=')
 70                 cmd.library_dirs = value.split(os.pathsep)

=> library_dirs=['`pwd`', '']

link command via unixccompier:
gcc-4.2 ... /foo.o -L`pwd` -L (empty!) -o /private/var/folder.../foo.so

=> "No such file or directory"
---

(for 2.7.2) https://raw.github.com/toggtc/python-patch/master/2.7.2/distutils_test_fixup_command_2.7.2.patch
(for 3.2.2) https://raw.github.com/toggtc/python-patch/master/3.2.2/distutils_test_fixup_command_3.2.2.patch

----------
assignee: tarek
components: Distutils, Macintosh
messages: 152219
nosy: eric.araujo, tarek, toggtc
priority: normal
severity: normal
status: open
title: test_get_outputs (test_distutils) failure with --enable-shared on Mac OS X
versions: Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13901>
_______________________________________

From report at bugs.python.org  Sun Jan 29 10:23:00 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 09:23:00 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out
	of	frame into generator/threadstate
In-Reply-To: <1327802425.89.0.202313759093.issue13897@psf.upfronthosting.co.za>
Message-ID: <4F250FF2.2080007@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

Nick Coghlan wrote:
> Nick Coghlan <ncoghlan at gmail.com> added the comment:
> 
> The division of responsibilities between generator objects and the eval loop is currently a little messy. The eval loop deals almost entirely with frame objects and also handles swapping exception states around on behalf of generators, which is why the generator specific exception state currently lives on frame objects (i.e. to avoid the eval loop needing to know too much about the internal structure of generator objects).

In my patch the exception state does not need swapping at all, so the 
question of whose responsibility it is become mute.
Neither generator nor ceval need to do it.

> 
> Before generator related state can reasonably be moved out of the frame objects, those responsibilities should be divided more cleanly. Ron Adam has made an initially attempt at tackling that problem in issue 13607.
> 
> Once the responsibilities are divided appropriately, *then* we can look at the fields that ceval no longer touches and see if they should be moved somewhere else (whether that's directly into the generator struct or into a new struct referenced from the generator struct).

This patch is largely orthogonal to that proposed for issue 13607.
In fact the new CoState struct would be a better place to pass data 
between PyEval_EvalFrame() and gen_send(), rather than adding yet 
another field to the frame.

----------
title: Move fields relevant to coroutine/generators out of frame into generator/threadstate -> Move fields relevant to coroutine/generators out of	frame into generator/threadstate

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 10:26:05 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 09:26:05 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out
	of	frame into generator/threadstate
In-Reply-To: <1327794836.94.0.29143192128.issue13897@psf.upfronthosting.co.za>
Message-ID: <4F2510AB.5080906@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

Amaury Forgeot d'Arc wrote:
> 
> - Why is it called CoState?  is it related to coroutines?

Yes it is related to coroutines, threads and generators *are*
(a limited form of) asymmetric coroutines, even if we don't usually 
think of them that way).

Perhaps GenState would be a better name
(treating the threadstate as a sort of top level pseudo-generator)?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 12:50:14 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sun, 29 Jan 2012 11:50:14 +0000
Subject: [issue1625] bz2.BZ2File doesn't support multiple streams
In-Reply-To: <1197624030.23.0.503522059328.issue1625@psf.upfronthosting.co.za>
Message-ID: <1327837814.55.0.59373010466.issue1625@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

> I am just recompressing a 77GB file because of this :-(.

Sorry to hear that :(

> I would consider that a bug, not a feature request.

Semantic issues aside, my concern here is that the patch for 2.7 is considerably
larger than the one for 3.3, and the code it modifies is more fragile.

An alternative solution I'd like to pursue is to backport 3.3's BZ2File
implementation to run on 2.7, and release it on PyPI. That way there's no risk
of introducing regressions for people who don't need this facility, and those
who do can get it without needing to upgrade to 2.7.3 (or even wait for it to be
released). It shouldn't take much effort to get it working on 2.6 as well.

How does that sound?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1625>
_______________________________________

From report at bugs.python.org  Sun Jan 29 13:40:43 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sun, 29 Jan 2012 12:40:43 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1327792039.4376.7.camel@localhost.localdomain>
Message-ID: <20120129124041.GA6202@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Antoine Pitrou <report at bugs.python.org> wrote:
> > I thought the whole Py_buffer API was only temporarily removed from the
> > limited API until the issues were sorted out:
> > 
> > http://bugs.python.org/issue10181#msg125462
> 
> I'm talking about the memoryview access macros. It's like
> PyList_GET_SIZE, it isn't part of the limited API and you have to use
> PyList_GetSize instead.

You're right: I presumed that the macros were excluded temporarily
when in fact that had already happened in 62b61abd02b8.

> > The flags are primarily for the memoryview itself to avoid repeated calls
> > to PyBuffer_IsContiguous(). So when 3rd-party uses PyMemoryView_GET_BUFFER
> > to get the view and also needs to determine the contiguity type, that
> > code could also use the flags.
> 
> But why would 3rd-party code use PyMemoryView_GET_BUFFER instead of, for
> example, PyObject_GetBuffer? You seldom have APIs which *expect* a
> memoryview, I think. Instead, they would expect a buffer-compatible
> object.

That's a good question. It looks to me that the macro was present as
PyMemoryView() initially. You renamed it in #3560 (with Guido's approval),
and later PyMemoryView_GET_BUFFER appeared in the docs.

I think 3rd-party code uses the macros mainly because they are
present and, in the case of PyMemoryView_GET_BUFFER, documented.
In most situations PyObject_GetBuffer() could be used indeed.

Most use cases I can think of would also involve having access to
the managed buffer API. As an example, here's a technique that is
similar to what goes on in PyMemoryView_GetContiguous():

Suppose you have an initialized bytes object that you want to
wrap as a multi-dimensional exporter. Then:

   - Base the memoryview on the bytes object and keep exactly one
     reference to it.

   - Adjust the shape, strides etc. to get the structure you want.

   - Return the view: You now have a fully compliant exporter that
     only needs a single Py_DECREF() to disappear and do all cleanup.

Of course this could also be exposed as a function, e.g.:

   /* stealing a reference to bytes */
   PyMemoryView_FromBytesAndInfo(PyObject *bytes, Py_buffer *info);

So let's make the flags private. What do you prefer?

  1) Leave them in memoryview.h, but with a leading underscore:

       _Py_MEMORYVIEW_C
       [...]

  2) Move them to memoryobject.c, with a leading underscore.

  3) Move them to memoryobject.c, without a leading underscore (I find
     this more readable).

  4) Move them to memoryobject.c as MV_C, MV_FORTRAN, etc.

Also, I'll add a note to the docs that PyMemoryView_GET_BUFFER can
probably be avoided by using PyObject_GetBuffer() directly.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 13:47:17 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sun, 29 Jan 2012 12:47:17 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327841237.04.0.878421127807.issue13848@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

I have fixed the refleak, added a _PyUnicode_HasNULChars and integrated it into the Win32-unicode-if-branch. Couldn't test it due to lack of win32 ? the function itself is tested though.

----------
Added file: http://bugs.python.org/file24355/open-nul.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sun Jan 29 14:00:08 2012
From: report at bugs.python.org (Ross Lagerwall)
Date: Sun, 29 Jan 2012 13:00:08 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1327842008.79.0.493155977187.issue13817@psf.upfronthosting.co.za>


Changes by Ross Lagerwall <rosslagerwall at gmail.com>:


----------
nosy: +rosslagerwall

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Sun Jan 29 14:33:44 2012
From: report at bugs.python.org (Hynek Schlawack)
Date: Sun, 29 Jan 2012 13:33:44 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327844024.04.0.206801866008.issue13848@psf.upfronthosting.co.za>


Hynek Schlawack <hs at ox.cx> added the comment:

With Georg's kind help I added some improvements:

 - I've been reluctant to waste heap for caching the nul string but he convinced me that I was being ridiculous ;)
 - For some reason there was a stray character inside, that should be fixed too.

In related news, I'm also adding tests for fileio since the last patch.

----------
Added file: http://bugs.python.org/file24356/open-nul.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sun Jan 29 14:53:31 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sun, 29 Jan 2012 13:53:31 +0000
Subject: [issue13902] Sporadic test_threading failure on FreeBSD 6.4 buildbot
Message-ID: <1327845211.23.0.0250327265855.issue13902@psf.upfronthosting.co.za>


New submission from Nadeem Vawda <nadeem.vawda at gmail.com>:

http://www.python.org/dev/buildbot/all/builders/x86%20FreeBSD%206.4%203.x/builds/2206/steps/test/logs/stdio

    FAIL: test_6_daemon_threads (test.test_threading.ThreadJoinOnShutdown)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_threading.py", line 675, in test_6_daemon_threads
        self.assertFalse(err)
    AssertionError: b'[52879 refs]\nT' is not false

----------
components: Tests
keywords: buildbot
messages: 152226
nosy: nadeem.vawda, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: Sporadic test_threading failure on FreeBSD 6.4 buildbot
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13902>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:11:18 2012
From: report at bugs.python.org (Ethan Furman)
Date: Sun, 29 Jan 2012 14:11:18 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327846278.71.0.104906899167.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Current semantics (before patch):

cause is not None  --> cause is set, display it instead of context
cause is None  --> no cause, try to display context

context is not None  --> no context
context is None  --> context set, display it (unless cause already displayed)

---

Proposed semantics (after patch)

cause is True  --> context set, but no display
cause is not None  --> cause set, display it instead of context
cause is None  --> no cause, try to display context

context is None --> no context
context is not None --> context set, display it (unless cause already displayed)

---

I prefer to go with True for cause, instead of False, as a way of saying "Yes, there was an exception before this one, but I'm not going to display it" as opposed to None meaning "No there is no cause, and I'm not going to show you the context either".

Using True instead of False, and leaving the None's as they are now, preserves the behavior of None meaning none, as in "there isn't one".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:24:07 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:24:07 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327847047.52.0.28566294966.issue6210@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Hmm, so "from None" sets cause to "True", while all other "from X" sets cause to "X".  That does not sound like a good idea to me.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:26:47 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 14:26:47 +0000
Subject: [issue13903] New shared-keys dictionary implementation
Message-ID: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

The proposed dictionary implementation allows sharing of keys & hashes between dictionaries. This leads to substantial memory savings for object-oriented programs. For non-OO programs the impact is negligible.

----------
components: Interpreter Core
hgrepos: 109
messages: 152229
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: New shared-keys dictionary implementation
type: performance
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:29:43 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:29:43 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327847383.83.0.179757313393.issue13857@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

BTW, the short spelling looks like it wouldn't indent the first line.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:29:58 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:29:58 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327847398.12.0.810367846643.issue13857@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Otherwise +1.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:35:57 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:35:57 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327847757.59.0.900675021568.issue13899@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

r'[\w]' also matches word chars.  I find that a very useful property, since you can easily build classes like '[\w.]'  It's also impossible to change this without breaking lots of regexes.  It's also explicitly documented, although IMO it's not clear it extends to \A and \Z, since it talks about "character classes".  So this is a docs issue.

----------
assignee:  -> docs at python
components: +Documentation
nosy: +docs at python, georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:38:58 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 14:38:58 +0000
Subject: [issue13900] documentation page on email.parser contains
	self-referential non-definition of headersonly parameter
In-Reply-To: <1327815089.31.0.591392873917.issue13900@psf.upfronthosting.co.za>
Message-ID: <E1RrVuD-0001XE-Ef@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 0d5667171356 by Georg Brandl in branch '3.2':
Fix #13900: resolve self-referential description of a parameter.
http://hg.python.org/cpython/rev/0d5667171356

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:39:38 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 14:39:38 +0000
Subject: [issue13900] documentation page on email.parser contains
	self-referential non-definition of headersonly parameter
In-Reply-To: <1327815089.31.0.591392873917.issue13900@psf.upfronthosting.co.za>
Message-ID: <E1RrVur-0001ci-3b@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 690d5978bd21 by Georg Brandl in branch '2.7':
Fix #13900: resolve self-referential description of a parameter.
http://hg.python.org/cpython/rev/690d5978bd21

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:39:46 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:39:46 +0000
Subject: [issue13900] documentation page on email.parser contains
	self-referential non-definition of headersonly parameter
In-Reply-To: <1327815089.31.0.591392873917.issue13900@psf.upfronthosting.co.za>
Message-ID: <1327847986.81.0.680013898777.issue13900@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Should now be fixed. Thanks for the report.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:39:51 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:39:51 +0000
Subject: [issue13900] documentation page on email.parser contains
	self-referential non-definition of headersonly parameter
In-Reply-To: <1327815089.31.0.591392873917.issue13900@psf.upfronthosting.co.za>
Message-ID: <1327847991.87.0.434893424485.issue13900@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13900>
_______________________________________

From report at bugs.python.org  Sun Jan 29 15:42:26 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 14:42:26 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327848146.82.0.169071156713.issue13903@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


----------
keywords: +patch
Added file: http://bugs.python.org/file24357/061f8573af54.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:29:02 2012
From: report at bugs.python.org (=?utf-8?q?Filip_Gruszczy=C5=84ski?=)
Date: Sun, 29 Jan 2012 15:29:02 +0000
Subject: [issue13896] shelf doesn't work with 'with'
In-Reply-To: <1327749150.13.0.147725255938.issue13896@psf.upfronthosting.co.za>
Message-ID: <1327850942.51.0.000934301978162.issue13896@psf.upfronthosting.co.za>


Filip Gruszczy?ski <gruszczy at gmail.com> added the comment:

Oh, I haven't noticed that. Using contextlib.closing solves my problem. Thanks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13896>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:32:27 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sun, 29 Jan 2012 15:32:27 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327851147.32.0.752780075223.issue13899@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

[\w] should definitely work, but [\B] doesn't seem to match anything useful, and it just fails silently because it's neither equivalent to \B nor to [B]:
>>> re.match(r'foo\B', 'foobar')  # on a non-word-boundary -- matches fine
<_sre.SRE_Match object at 0xb76dd3a0>
>>> re.match(r'foo[B]', 'fooBar')  # same as r'fooB'
<_sre.SRE_Match object at 0xb76dd1e0>
>>> re.match(r'foo[\B]', 'foobar')  # not equivalent to \B
>>> re.match(r'foo[\B]', 'fooBar')  # not equivalent to [B]

The same is true for \Z and \A:
>>> re.match(r'foo\Z', 'foo')  # end of the string -- matches fine
<_sre.SRE_Match object at 0xb76dd3a0>
>>> re.match(r'foo[Z]', 'fooZ')  # same as r'fooZ'
<_sre.SRE_Match object at 0xb76dd1e0>
>>> re.match(r'foo[\Z]', 'foo')  # not equivalent to \Z
>>> re.match(r'foo[\Z]', 'fooZ')  # not equivalent to [Z]
>>>
>>> re.match(r'\Afoo', 'foo')  # beginning of the string -- matches fine
<_sre.SRE_Match object at 0xb76dd1e0>
>>> re.match(r'[A]foo', 'Afoo')  # same as r'Afoo'
<_sre.SRE_Match object at 0xb76dd3a0>
>>> re.match(r'[\A]foo', 'foo')  # not equivalent to \A
>>> re.match(r'[\A]foo', 'Afoo')  # not equivalent to [A]

Inside [], \b switches from word boundary to backspace:
>>> re.match(r'foo\b', 'foobar')  # not on a word boundary -- no matches
>>> re.match(r'foo\b', 'foo bar')  # on a word boundary  -- matches fine
<_sre.SRE_Match object at 0xb74a4ec8>
>>> re.match(r'foo[\b]', 'foo bar')  # not equivalent to \b
>>> re.match(r'foo[\b]', 'foo\bbar')  # matches backspace
<_sre.SRE_Match object at 0xb76dd3d8>
>>> re.match(r'foo([\b])', 'foo\bbar').group(1)
'\x08'

Given that \b doesn't keep its word boundary meaning inside the [], \B (and \A and \Z) shouldn't keep it either (also because I can't see how having these inside [] would be of any use).
On the other hand I'm not sure they should be equivalent to B, A, Z either.  There are several escape sequences in the form \X (where X is an upper- or lower-case letter) that are not equivalent to X (\a\b\d\f\s\x\w\D\S\W...).
Raising an error that says something like "I don't think [\A] does what you think it does, use [A] instead." might be a better option (and in case anyone is wondering about re.escape, I just checked and it doesn't escape letters).  Even if this is technically backward incompatible, any string that has \A, \B, \Z inside [] can be considered buggy IMHO (unless someone can come up with a valid use case where they do something useful).

----------
assignee: docs at python -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:35:24 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 15:35:24 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327851324.34.0.488004909159.issue13899@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Interesting. That shifts the issue, since the current behavior is neither of the two that make sense. Then it would indeed make the most sense to raise in these cases.

(I wonder what these patterns actually would match, but I have no time to look in the sre sources right now...)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:37:50 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 29 Jan 2012 15:37:50 +0000
Subject: [issue13872] socket.detach doesn't mark socket._closed
In-Reply-To: <1327582652.34.0.856044764189.issue13872@psf.upfronthosting.co.za>
Message-ID: <1327851470.75.0.338569200437.issue13872@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13872>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:43:53 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 15:43:53 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <E1RrWv1-0000kO-Rt@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 9be82f458b79 by Charles-Fran?ois Natali in branch 'default':
Issue #6774: Back out c8b77efe8b56, which only brings confusion.
http://hg.python.org/cpython/rev/9be82f458b79

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:44:56 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Sun, 29 Jan 2012 15:44:56 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327851896.51.0.955901588177.issue4966@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

> One other point... the branch is actually now relative to default, not
> 3.2. While that was due to a merging mistake on my part, it also means
> I can legitimately ignore the narrow/wide build distinction in the
> section on strings.

So will this go on 3.3 only or are you planning to push it on 3.2(/2.7) too?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:46:01 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Sun, 29 Jan 2012 15:46:01 +0000
Subject: [issue6774] socket.shutdown documentation: on some platforms,
	closing one half closes the other half
In-Reply-To: <1251103521.34.0.917057608479.issue6774@psf.upfronthosting.co.za>
Message-ID: <1327851961.5.0.18746001407.issue6774@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

I've reverted the commit.

----------
resolution:  -> rejected
stage:  -> committed/rejected
status: open -> closed
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6774>
_______________________________________

From report at bugs.python.org  Sun Jan 29 16:50:55 2012
From: report at bugs.python.org (Patrick Westerhoff)
Date: Sun, 29 Jan 2012 15:50:55 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327852255.94.0.110946606175.issue6210@psf.upfronthosting.co.za>


Patrick Westerhoff <PatrickWesterhoff at gmail.com> added the comment:

I have to agree with Georg on that. I think it would make more sense to introduce some internal flag/variable that keeps track of if the cause was explicitely set. So if cause was set (i.e. `from X` syntax is used), then always display it in favor of the context ? except that a None-cause causes nothing to display.

Regardless of that I?m actually not sure if just changing the way the cause is displayed is a correct way to handle the context. If I explicitely raise an exception in an except-handler, I usually don?t expect that new exception to get the previous exception attached to. In the original example, I want to completely replace the ?context? by a new exception without implicitely keeping over the original exception.

So even if using `from None` will prevent the context from being displayed (as the explicitely set cause will override it), the `__context__` will probably still be set by the `raise` statement, and I think that shouldn?t happen. Hence the `raise X instead` or `raise as X` idea that simply does not set the context but ?destroys? it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 17:38:41 2012
From: report at bugs.python.org (July Tikhonov)
Date: Sun, 29 Jan 2012 16:38:41 +0000
Subject: [issue13904] Generator as *args: TypeError replaced
Message-ID: <1327855120.91.0.5569840716.issue13904@psf.upfronthosting.co.za>


New submission from July Tikhonov <july.tikh at gmail.com>:

>>> set().union(*(None[k] for k in range(5)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: union() argument after * must be a sequence, not generator

Clearly, exception in not relevant, since next line works:

>>> set().union(*([k] for k in range(5)))
{0, 1, 2, 3, 4}

Correct exception would be

>>> None[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

Problem is in python function call mechanics.
set().union can be replaced by any callable;
Generator can be replaced by any TypeError-raising iterable. Exceptions other then TypeError are handled correctly.

Python/ceval.c:4322
ext_do_call() converts stararg to tuple.
If any TypeError is raised, it is replaced with
TypeError("%s argument after * must be a sequence, not %s")


Proposed solution:

Probably, we can avoid replacing TypeError. Exceptions in the above cases would become relevant, and

>>> int(*None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type object argument after * must be a sequence, not NoneType

would become

>>> int(*None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not iterable

so exception is still recognizable (and, may be, even more relevant, since we don't actually need _sequence_ as stararg, _iterable_ would be enough).

----------
components: Interpreter Core
files: typeerror-replaced-in-stararg.diff
keywords: patch
messages: 152243
nosy: july
priority: normal
severity: normal
status: open
title: Generator as *args: TypeError replaced
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file24358/typeerror-replaced-in-stararg.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13904>
_______________________________________

From report at bugs.python.org  Sun Jan 29 17:39:09 2012
From: report at bugs.python.org (July Tikhonov)
Date: Sun, 29 Jan 2012 16:39:09 +0000
Subject: [issue13904] Generator as *args: TypeError replaced
In-Reply-To: <1327855120.91.0.5569840716.issue13904@psf.upfronthosting.co.za>
Message-ID: <1327855149.64.0.675724539614.issue13904@psf.upfronthosting.co.za>


Changes by July Tikhonov <july.tikh at gmail.com>:


Added file: http://bugs.python.org/file24359/typeerror-replaced-in-stararg-test.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13904>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:06:23 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 17:06:23 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327856783.63.0.983147680173.issue13903@psf.upfronthosting.co.za>


Changes by Mark Shannon <mark at hotpy.org>:


Added file: http://bugs.python.org/file24360/6a21f3b35e20.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:10:29 2012
From: report at bugs.python.org (Georg Brandl)
Date: Sun, 29 Jan 2012 17:10:29 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327857029.95.0.484735162181.issue13903@psf.upfronthosting.co.za>


Changes by Georg Brandl <georg at python.org>:


Removed file: http://bugs.python.org/file24357/061f8573af54.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:15:24 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:15:24 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327857324.91.0.200802645949.issue13848@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

The patch works under Windows here (on branch default).

----------
stage: needs patch -> patch review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:19:06 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:19:06 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <20120129124041.GA6202@sleipnir.bytereef.org>
Message-ID: <1327857406.3635.0.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Of course this could also be exposed as a function, e.g.:
> 
>    /* stealing a reference to bytes */
>    PyMemoryView_FromBytesAndInfo(PyObject *bytes, Py_buffer *info);

I think we should minimize the number of reference-stealing functions.

> So let's make the flags private. What do you prefer?

I don't really mind, whatever you think is best :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:21:19 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:21:19 +0000
Subject: [issue2636] Adding a new regex module (compatible with re)
In-Reply-To: <1327826273.02.0.709366491224.issue2636@psf.upfronthosting.co.za>
Message-ID: <1327857541.3635.2.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> It'd be nice if we had some sort of representative benchmark for
> real-world uses of Python regexps. The JS guys have all pitched in to
> create such a thing for uses of regexps on thew web. I don't know of
> any such thing for Python.

See http://hg.python.org/benchmarks/, there are regex benchmarks there.

> I agree that a Python implementation wouldn't be useful for some
> cases. On the other hand, I believe it would be fine (or at least
> tolerable) for some others. I don't know the ratio between the two.

I think the ratio would be something like 2% tolerable :)

As I said to Ezio and Georg, I think adding the regex module needs a
PEP, even if it ends up non-controversial.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2636>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:23:16 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:23:16 +0000
Subject: [issue13872] socket.detach doesn't mark socket._closed
In-Reply-To: <1327582652.34.0.856044764189.issue13872@psf.upfronthosting.co.za>
Message-ID: <1327857796.35.0.327790412546.issue13872@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> socket.socket.detach doesn't mark the socket._closed flag.

Well, does it have to? It's only an internal detail, it's not exposed as a public API.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13872>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:47:33 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 17:47:33 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <E1RrYqd-0007vG-7o@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 572bb8c265c0 by Antoine Pitrou in branch '3.2':
Issue #13848: open() and the FileIO constructor now check for NUL characters in the file name.
http://hg.python.org/cpython/rev/572bb8c265c0

New changeset 6bb05ce1cd1f by Antoine Pitrou in branch 'default':
Issue #13848: open() and the FileIO constructor now check for NUL characters in the file name.
http://hg.python.org/cpython/rev/6bb05ce1cd1f

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:48:25 2012
From: report at bugs.python.org (Brett Cannon)
Date: Sun, 29 Jan 2012 17:48:25 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327700568.22.0.963154831136.issue13890@psf.upfronthosting.co.za>
Message-ID: <CAP1=2W596BUFoXdsxEoB8Bwb2uAxJAt8DNg_SHeCvHcJQwGjCQ@mail.gmail.com>


Brett Cannon <brett at python.org> added the comment:

Is there a technological reason environ is not updated, or is it simply
oversight?

Lib/os.py: under POXIX, os.environ reflects posix.environ (it uses the same
underlying dict), while under Windows, os.environ uses a distinct dict from
nt.environ.
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue13890>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:50:11 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:50:11 +0000
Subject: [issue13848] io.open() doesn't check for embedded NUL characters
In-Reply-To: <1327370600.14.0.389323922323.issue13848@psf.upfronthosting.co.za>
Message-ID: <1327859411.12.0.619053052029.issue13848@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I've made small changes and committed the patch in 3.2 and 3.3.
2.7 would need further changes and I don't think it's worth the bother.
Thanks!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13848>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:52:56 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:52:56 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <1327859576.01.0.615947624537.issue13890@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

I think it's more laziness. _Environ.__setitem__ could also update the original mapping.

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Sun Jan 29 18:55:10 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 17:55:10 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327859710.62.0.674823990057.issue13903@psf.upfronthosting.co.za>


Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +benjamin.peterson, pitrou
stage:  -> patch review
versions: +Python 3.3 -Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 19:10:23 2012
From: report at bugs.python.org (Ethan Furman)
Date: Sun, 29 Jan 2012 18:10:23 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327860623.34.0.845447108659.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Patrick: The value in this enhancement is in not displaying the chained exception.  I do not see any value in throwing it away completely.  If you don't care about __context__ you can safely ignore it.  On the other hand, if it is completely removed, and you do care about it... well, too bad.

Georg, Nick:  On further thought, I agree that having 'from None' set cause from None to True is counter-intuitive, and there is consistency in having __context__ as None and __cause__ as False.

Thanks, Nick, for the pointers on the necessary changes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Sun Jan 29 19:43:53 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Sun, 29 Jan 2012 18:43:53 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327862633.8.0.201318780134.issue13903@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

In the initial comment, 'Dummy' to 'Deleted' here but only here:
-   Holds an active (key, value) pair.  Active can transition to Dummy 
+   Holds an active (key, value) pair.  Active can transition to Deleted 

Im Lib/test/test_pprint.py
     def test_set_reprs(self): ...
         # Consequently, this test is fragile and ...
+        # XXX So why include this "test" in the first place?
Raymond, I believe you added this 44927 and revised for 3.x in 45067.
I imagine it will also be a problem with randomized hashes. Should it be removed or somehow revised?

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 20:06:33 2012
From: report at bugs.python.org (Giampaolo Rodola')
Date: Sun, 29 Jan 2012 19:06:33 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327863993.58.0.89260571142.issue13903@psf.upfronthosting.co.za>


Changes by Giampaolo Rodola' <g.rodola at gmail.com>:


----------
nosy: +giampaolo.rodola

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 20:57:59 2012
From: report at bugs.python.org (Philip Jenvey)
Date: Sun, 29 Jan 2012 19:57:59 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327867079.49.0.738959020415.issue13903@psf.upfronthosting.co.za>


Changes by Philip Jenvey <pjenvey at underboss.org>:


----------
nosy: +pjenvey

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Sun Jan 29 20:58:52 2012
From: report at bugs.python.org (Catalin Iacob)
Date: Sun, 29 Jan 2012 19:58:52 +0000
Subject: [issue13905] Built-in Types Comparisons should mention rich
	comparison methods
Message-ID: <1327867132.3.0.913390477966.issue13905@psf.upfronthosting.co.za>


New submission from Catalin Iacob <iacobcatalin at gmail.com>:

In 2.7 the Comparisons section of stdtypes.rst only talks about __cmp__ and never mentions the rich comparison methods:
"Instances of a class normally compare as non-equal unless the class defines the __cmp__() method. Refer to Basic customization) for information on the use of this method to effect object comparisons."

The first sentence is false, instances can also compare as equal if they define __eq__. And since __cmp__ is gone in Python3, I think the rich comparison methods should at least be mentioned, or even emphasized over __cmp__ to help people write more forward compatible code.

----------
assignee: docs at python
components: Documentation
messages: 152254
nosy: catalin.iacob, docs at python
priority: normal
severity: normal
status: open
title: Built-in Types Comparisons should mention rich comparison methods
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13905>
_______________________________________

From report at bugs.python.org  Sun Jan 29 21:01:57 2012
From: report at bugs.python.org (Catalin Iacob)
Date: Sun, 29 Jan 2012 20:01:57 +0000
Subject: [issue13905] Built-in Types Comparisons should mention rich
	comparison methods
In-Reply-To: <1327867132.3.0.913390477966.issue13905@psf.upfronthosting.co.za>
Message-ID: <1327867317.28.0.417722515795.issue13905@psf.upfronthosting.co.za>


Catalin Iacob <iacobcatalin at gmail.com> added the comment:

Here's my attempt at a patch. It mostly takes the text from the default branch and adds references to __cmp__.

----------
keywords: +patch
Added file: http://bugs.python.org/file24361/issue13905v1.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13905>
_______________________________________

From report at bugs.python.org  Sun Jan 29 21:13:12 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 20:13:12 +0000
Subject: [issue5231] Change format of a memoryview
In-Reply-To: <1234474743.76.0.794318305676.issue5231@psf.upfronthosting.co.za>
Message-ID: <1327867992.62.0.741883436673.issue5231@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Shouldn't this be closed in favour of #10181?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5231>
_______________________________________

From report at bugs.python.org  Sun Jan 29 21:40:03 2012
From: report at bugs.python.org (Nadeem Vawda)
Date: Sun, 29 Jan 2012 20:40:03 +0000
Subject: [issue1625] bz2.BZ2File doesn't support multiple streams
In-Reply-To: <1197624030.23.0.503522059328.issue1625@psf.upfronthosting.co.za>
Message-ID: <1327869603.49.0.793871328677.issue1625@psf.upfronthosting.co.za>


Nadeem Vawda <nadeem.vawda at gmail.com> added the comment:

> An alternative solution I'd like to pursue is to backport 3.3's BZ2File
> implementation to run on 2.7, and release it on PyPI.

Well, that was easier than I expected. It didn't take much work to get it
working under 2.6, 2.7 and 3.2. I've put up this "bz2file" module on
GitHub <https://github.com/nvawda/bz2file>. I'll package it up and upload
it to PyPI sometime soon.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1625>
_______________________________________

From report at bugs.python.org  Sun Jan 29 21:52:13 2012
From: report at bugs.python.org (David Bolen)
Date: Sun, 29 Jan 2012 20:52:13 +0000
Subject: [issue13507] Modify OS X installer builds to package liblzma for the
	new lzma module
In-Reply-To: <1322647679.29.0.297920473861.issue13507@psf.upfronthosting.co.za>
Message-ID: <1327870333.29.0.727263784515.issue13507@psf.upfronthosting.co.za>


David Bolen <db3l.net at gmail.com> added the comment:

I recently built the xz library on my OSX Tiger buildbot (that also does the daily DMGs via the build script), and Nadeem mentioned this ticket.

As an FYI, I wasn't able to get the xz library (5.0.3) to configure/build as a universal build (i386/ppc) in a single step (as in the build script recipes).  It appears to use some compile options incompatible with multiple -arch.

For my buildbot, I built it twice (as i386 and ppc) and then combined the libraries (static and shared) using lipo.  This appears to work for both the regular buildbot and the DMG creation.

In peeking at the build script this approach is a bit beyond the current recipe process, so would need more work than just a new recipe.  Assuming a build/lipo process is considered legitimate for release installers, of course, I'm not that sure how much I ended up "cheating".

----------
nosy: +db3l

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13507>
_______________________________________

From report at bugs.python.org  Sun Jan 29 21:59:17 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sun, 29 Jan 2012 20:59:17 +0000
Subject: [issue5231] Change format of a memoryview
In-Reply-To: <1234474743.76.0.794318305676.issue5231@psf.upfronthosting.co.za>
Message-ID: <1327870757.27.0.460803551947.issue5231@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Yes, it's really superseded by #10181 now. I'm closing as 'duplicate',
since technically it'll be fixed once the patch for #10181 is committed.

----------
dependencies:  -Problems with Py_buffer management in memoryobject.c (and elsewhere?)
resolution:  -> duplicate
stage: needs patch -> committed/rejected
status: open -> closed
superseder:  -> Problems with Py_buffer management in memoryobject.c (and elsewhere?)

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5231>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:09:02 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Sun, 29 Jan 2012 21:09:02 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out
	of	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1327871342.9.0.737153143975.issue13897@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

IMO "tstate->exc_value" has nothing to do with generators.  Changing its name seems gratuitous breakage to me.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:23:10 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 21:23:10 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out
	of	frame into generator/threadstate
In-Reply-To: <1327871342.9.0.737153143975.issue13897@psf.upfronthosting.co.za>
Message-ID: <4F25B8C5.9070905@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

The important part is not the change of name, but wrapping them in a 
struct which can be embedded in both PyThreadState and PyGenObject.

The state->exc_XXX trio of values are the currently handled exception
(sys.exc_info()) and are shadowed by generator exception handlers.
My patch models that shadowing rather than swapping the values in and 
out. This allows me to eliminate save_exc_state(), swap_exc_state()
and restore_and_clear_exc_state() completely.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:34:37 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Sun, 29 Jan 2012 21:34:37 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327872877.79.0.500263215451.issue13868@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

If "floating point" in a sentence is in a role of an adjective, then it must be written as "floating-point" (with a hyphen), otherwise not.

Example: "The number 3.5 is a floating-point number."

Please consult some English orthography book and start writing correct English.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:41:11 2012
From: report at bugs.python.org (John Machin)
Date: Sun, 29 Jan 2012 21:41:11 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327873271.27.0.428289539817.issue13899@psf.upfronthosting.co.za>


John Machin <sjmachin at lexicon.net> added the comment:

@Ezio: Comparison of the behaviour of \letter inside/outside character classes is irrelevant. The rules for inside can be expressed simply as:

1. Letters dDsSwW are special; they represent categories as documented, and do in fact have a similar meaning outside character classes.

2. Otherwise normal Python rules for backslash escapes in string literals should be followed. This means automatically that \a -> \x07, \A -> A, \b -> backspace, \B -> B, \z -> z and \Z -> Z.

@Georg: No need to read the source, just read my initial posting: It's compiled as a zero-length matcher ("at") inside a character class ("in") i.e. a nonsense, then at runtime the illegality is deliberately ignored.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:50:46 2012
From: report at bugs.python.org (John Machin)
Date: Sun, 29 Jan 2012 21:50:46 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1327873846.76.0.0931496471827.issue13899@psf.upfronthosting.co.za>


John Machin <sjmachin at lexicon.net> added the comment:

Whoops: "normal Python rules for backslash escapes" should have had a note "but revert to the C behaviour of stripping the \ from unrecognised escapes" which is what re appears to do in its own \ handling.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Sun Jan 29 22:57:19 2012
From: report at bugs.python.org (Samuel Iseli)
Date: Sun, 29 Jan 2012 21:57:19 +0000
Subject: [issue13889] str(float) and round(float) issues with FPU precision
In-Reply-To: <1327676303.66.0.487112802864.issue13889@psf.upfronthosting.co.za>
Message-ID: <1327874239.77.0.938731957652.issue13889@psf.upfronthosting.co.za>


Samuel Iseli <samuel.iseli at gmail.com> added the comment:

Hi Marc, the changes to the pythoncore.vcproj Visual-Studio file define the HAVE_VC_FUNC_FOR_X87 symbol. I use this symbol to enable the precision-setting macros in pyport.h. I made this similar to the existing code for gcc (linux).

You can change this but currently this symbol has to be defined somewhere for the macros to have an effect.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13889>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:16:26 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sun, 29 Jan 2012 22:16:26 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>
Message-ID: <1327875386.21.0.543656005968.issue13856@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

Thanks for submitting the patch. Couple of comments.

1. This is a new feature, so the patch should be addressed against 3.x. 
2. The patch lacks tests and documentation and hence it is not complete.

You could take a look at http/client.py or ssl.py (with test_ssl.py) for for passing the key and cert from the client.

Antoine - I fail to recollect, but is there any reason for the clients in the stdlib are not carrying a ca_file and doing a certificate validation of the server connection? Is it required (or a good idea) ?

----------
nosy: +pitrou
versions: +Python 3.3 -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:18:07 2012
From: report at bugs.python.org (Alexander Maksimenko)
Date: Sun, 29 Jan 2012 22:18:07 +0000
Subject: [issue13906] mimetypes.py under windows - bad exception catch
Message-ID: <1327875487.38.0.0330076110954.issue13906@psf.upfronthosting.co.za>


New submission from Alexander Maksimenko <alex.mclander at gmail.com>:

mimetypes.py(249) expectts Unicode*En*codeError, but Unicode*De*codeError happens when registry has non latin symbols (Vista Home 64).

I just change cathc jn next line to UnicodeDecodeError and all now works fine. But may be error not here, but on encode method which raise "negative" exception


F:\>c:\python27\python -m SimpleHTTPServer
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import SimpleHTTPServer
  File "c:\python27\lib\SimpleHTTPServer.py", line 27, in <module>
    class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  File "c:\python27\lib\SimpleHTTPServer.py", line 204, in SimpleHTTPRequestHandler
    mimetypes.init() # try to read system mime.types
  File "c:\python27\lib\mimetypes.py", line 355, in init
    db.read_windows_registry()
  File "c:\python27\lib\mimetypes.py", line 259, in read_windows_registry
    for ctype in enum_types(mimedb):
  File "c:\python27\lib\mimetypes.py", line 249, in enum_types
    ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)


---------- here after changes -----------------

F:\>c:\python27\python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
mc-quad - - [30/Jan/2012 02:02:18] "GET / HTTP/1.1" 200 -
mc-quad - - [30/Jan/2012 02:02:18] code 404, message File not found
mc-quad - - [30/Jan/2012 02:02:18] "GET /favicon.ico HTTP/1.1" 404 -

----------
components: Library (Lib)
messages: 152267
nosy: mclander
priority: normal
severity: normal
status: open
title: mimetypes.py under windows - bad exception catch
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13906>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:27:56 2012
From: report at bugs.python.org (R. David Murray)
Date: Sun, 29 Jan 2012 22:27:56 +0000
Subject: [issue13906] mimetypes.py under windows - bad exception catch
In-Reply-To: <1327875487.38.0.0330076110954.issue13906@psf.upfronthosting.co.za>
Message-ID: <1327876076.98.0.482356841029.issue13906@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

This is a duplicate of issue 9291.

----------
nosy: +r.david.murray
resolution:  -> duplicate
stage:  -> committed/rejected
status: open -> closed
superseder:  -> mimetypes initialization fails on Windows because of non-Latin characters in registry

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13906>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:30:45 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Sun, 29 Jan 2012 22:30:45 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>
Message-ID: <1327876245.67.0.23679510461.issue13856@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

> Antoine - I fail to recollect, but is there any reason for the clients 
> in the stdlib are not carrying a ca_file and doing a certificate
> validation of the server connection?

Well, if you are a security expert you can volunteer to maintain a trusted certificates' file in the Python repository :) I think nobody else amongst us is qualified.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:37:00 2012
From: report at bugs.python.org (Barry A. Warsaw)
Date: Sun, 29 Jan 2012 22:37:00 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327778728.2219.115.camel@surprise>
Message-ID: <20120129173654.581d8a5d@resist.wooz.org>


Barry A. Warsaw <barry at python.org> added the comment:

On Jan 28, 2012, at 07:26 PM, Dave Malcolm wrote:

>This turns out to pass without PYTHONHASHRANDOMIZATION in the
>environment, and fail intermittently with it.
>
>Note that "make test" invokes the built python with "-E", so that it
>ignores the setting of PYTHONHASHRANDOMIZATION in the environment.
>
>Barry, Benjamin: does fixing this bug require getting the full test
>suite to pass with randomization enabled (and fixing the intermittent
>failures due to ordering issues), or is it acceptable to "merely" have
>full passes without randomizing the hashes?

I think we at least need to identify (to the best of our ability) the tests
that fail and include them in release notes.  If they're easy to fix, we
should fix them.  Maybe also open a bug report for each failure.

I'm okay though with some tests failing in 2.6 with this environment variable
set.  We needn't go back and fix them in 2.6 (since we're in security-fix only
mode), but I'll bet you'll get almost the same set for 2.7 and there we
*should* fix them, even if it happens after the release.

>What do the buildbots do?

I'm not sure, but as long as the buildbots are green, I'm happy. :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:39:16 2012
From: report at bugs.python.org (Jim Jewett)
Date: Sun, 29 Jan 2012 22:39:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327795553.2219.128.camel@surprise>
Message-ID: <CA+OGgf6mWQ+nRdO8gt6-EdX5uoLdPps6We-b-RWf61UFuPNiMg@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

Given PYTHONHASHSEED, what is the point of PYTHONHASHRANDOMIZATION?

Alternative:

On startup, python reads a config file with the seed (which defaults to zero).

Add a function to write a random value to that config file for the next startup.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:45:27 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 22:45:27 +0000
Subject: [issue13907] test_pprint relies on set/dictionary repr() ordering
Message-ID: <1327877127.5.0.775790034164.issue13907@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

The function test_set_reprs() includes the comment:
"Consequently, this test is fragile and implementation-dependent"
Changing the dictionary iteration ordering breaks it.

----------
components: Tests
messages: 152272
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: test_pprint relies on set/dictionary repr() ordering

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13907>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:46:58 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sun, 29 Jan 2012 22:46:58 +0000
Subject: [issue7097] urlparse.urljoin of simple "http://" and "somedomain.com"
	produces incorrect result
In-Reply-To: <1255196414.88.0.948496525016.issue7097@psf.upfronthosting.co.za>
Message-ID: <1327877218.61.0.647816047398.issue7097@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

>>> urljoin("http://", "//somedomain.com")
results in "http://somedomain.com"

So, I wonder if this  way to specify the relative url properly and not the base-url. 

The test suite of urlparse tries to follow all the advertised scenarios for RFC3986 and also some more tests (which are usually discovered by de-facto scenarios of how other clients (mainly browsers) deal with it. If there is a browser behavior which we should emulate, without breaking existing code, we should consider this, otherwise we could leave this in invalid state.  Thanks!

----------
nosy: +orsenthil

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7097>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:48:35 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 29 Jan 2012 22:48:35 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>
Message-ID: <1327877315.66.0.157783616911.issue13856@psf.upfronthosting.co.za>


Martin v. L?wis <martin at v.loewis.de> added the comment:

For 3.x, xmlrpc.client should just pass-through the SSL context. Since the code to do so will be quite different from the current patch, I'm tempted to close this issue as rejected, unless Nathanael indicates that he would like to redo the patch for 3.x; this issue could then be recycled for that.

orsenthil: I don't fully understand your question (what kind of "carrying" should the clients do); the standard library most certainly supports validation of the server certificate.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:50:20 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 22:50:20 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <20120129173654.581d8a5d@resist.wooz.org>
Message-ID: <4F25CD34.7050209@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

Barry A. Warsaw wrote:
> Barry A. Warsaw <barry at python.org> added the comment:
> 
> On Jan 28, 2012, at 07:26 PM, Dave Malcolm wrote:
> 
>> This turns out to pass without PYTHONHASHRANDOMIZATION in the
>> environment, and fail intermittently with it.
>>
>> Note that "make test" invokes the built python with "-E", so that it
>> ignores the setting of PYTHONHASHRANDOMIZATION in the environment.
>>
>> Barry, Benjamin: does fixing this bug require getting the full test
>> suite to pass with randomization enabled (and fixing the intermittent
>> failures due to ordering issues), or is it acceptable to "merely" have
>> full passes without randomizing the hashes?
> 
> I think we at least need to identify (to the best of our ability) the tests
> that fail and include them in release notes.  If they're easy to fix, we
> should fix them.  Maybe also open a bug report for each failure.

http://bugs.python.org/issue13903 causes even more tests to fail,
so I'm submitting bug reports for most of the failing tests already.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:51:26 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Sun, 29 Jan 2012 22:51:26 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CA+OGgf6mWQ+nRdO8gt6-EdX5uoLdPps6We-b-RWf61UFuPNiMg@mail.gmail.com>
Message-ID: <4F25CD6C.1040002@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> Given PYTHONHASHSEED, what is the point of PYTHONHASHRANDOMIZATION?

How would you do what it does without it? I.e. how would you indicate
that it should randomize the seed, rather than fixing the seed value?

> On startup, python reads a config file with the seed (which defaults to zero).

-1 on configuration files that Python reads at startup (let alone in a
bugfix release).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Sun Jan 29 23:55:28 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Sun, 29 Jan 2012 22:55:28 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327877728.01.0.820388568182.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Trying to make this change in 2.7 would actually be a bit of a nightmare - how do you cleanly split documentation of the binary data and text processing sequence types when "str" is used for both?

The change would be *mostly* feasible in 3.2 (that's why I started my branch from there), but there are still some sharp edges that go away in 3.3 (mainly the narrow/wide Unicode split).

So unless anyone is really keen to see the update in 3.2, my current plan is to leave the maintenance versions alone and only update it for 3.3. Going that way also provides better opportunities for post-checkin feedback from folks that aren't set up to build the docs themselves (rebuilding the docs is fairly straightforward on *nix, but Terry tells me that using Windows complicates that process quite a bit).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:04:35 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Sun, 29 Jan 2012 23:04:35 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327878275.17.0.159955463319.issue13903@psf.upfronthosting.co.za>


Changes by Gregory P. Smith <greg at krypto.org>:


----------
nosy: +gregory.p.smith

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:07:39 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 23:07:39 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <E1RrdqP-0002lY-KZ@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset f71249d785d6 by Victor Stinner in branch 'default':
Issue #13874: read_null() of faulthandler uses volatile to avoid optimisation
http://hg.python.org/cpython/rev/f71249d785d6

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:07:56 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 29 Jan 2012 23:07:56 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327878476.15.0.0503395950062.issue13874@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Does my commit fix the issue?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:09:39 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 29 Jan 2012 23:09:39 +0000
Subject: [issue13846] Add time.monotonic() function
In-Reply-To: <1327706195.24.0.939528205838.issue13846@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwZOStz_9qaJfwZoNMwdG6=yGVwDcoff1TXnFMfgJyuXrA@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Do we actually yet another function, or could this be covered by adding a parameter such as monotonic=False, perhaps to wallclock().

A monotonic is a different clock, it would be surprising that an
argument uses another clock.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13846>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:14:03 2012
From: report at bugs.python.org (Ethan Furman)
Date: Sun, 29 Jan 2012 23:14:03 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327878843.05.0.169042792642.issue6210@psf.upfronthosting.co.za>


Ethan Furman <ethan at stoneleaf.us> added the comment:

Not sure I have traceback._iter_chain() patched correctly, but all the tests pass.

Here's the latest code.

----------
Added file: http://bugs.python.org/file24362/raise_from_none_v3.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:17:09 2012
From: report at bugs.python.org (=?utf-8?q?Andreas_St=C3=BChrk?=)
Date: Sun, 29 Jan 2012 23:17:09 +0000
Subject: [issue13908] PyType_FromSpec() lacks PyType_Ready() call
Message-ID: <1327879029.72.0.4665491693.issue13908@psf.upfronthosting.co.za>


New submission from Andreas St?hrk <andy-python at hammerhartes.de>:

As already stated by Amaury in http://mail.python.org/pipermail/python-dev/2011-October/113829.html, that leads to crashes:

>>> import xxlimited
>>> repr(xxlimited.Str)
[1]    19575 segmentation fault (core dumped)  ./python

----------
files: pytype_fromspec.patch
keywords: patch
messages: 152282
nosy: Trundle, amaury.forgeotdarc, loewis
priority: normal
severity: normal
status: open
title: PyType_FromSpec() lacks PyType_Ready() call
type: crash
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file24363/pytype_fromspec.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13908>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:23:35 2012
From: report at bugs.python.org (Roundup Robot)
Date: Sun, 29 Jan 2012 23:23:35 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <E1Rre5s-0004fA-Dm@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 5b42aefb8969 by Victor Stinner in branch 'default':
Issue #13847: Fix test_time, time.gmtime() doesn't use localtime()
http://hg.python.org/cpython/rev/5b42aefb8969

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:25:34 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 23:25:34 +0000
Subject: [issue13909] Ordering of free variables in dis is dependent on dict
	ordering.
Message-ID: <1327879534.06.0.507724097117.issue13909@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

This only seems to apply to free variables, not local or cell variables.

The offending tests are lines 429 & 430 of Lib/test/test_dis.py
      (tricky, code_info_tricky),
      (co_tricky_nested_f, code_info_tricky_nested_f),


Changing PyDict_MINSIZE to 4 provokes this failure
python3 -m test.test_dis

----------
components: Tests
messages: 152284
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Ordering of free variables in dis is dependent on dict ordering.
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13909>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:30:50 2012
From: report at bugs.python.org (Steven D'Aprano)
Date: Sun, 29 Jan 2012 23:30:50 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1327852255.94.0.110946606175.issue6210@psf.upfronthosting.co.za>
Message-ID: <4F25D6A5.1080102@pearwood.info>


Steven D'Aprano <steve+python at pearwood.info> added the comment:

Patrick Westerhoff wrote:
> Patrick Westerhoff <PatrickWesterhoff at gmail.com> added the comment:
> 
> I have to agree with Georg on that. I think it would make more sense to
> introduce some internal flag/variable that keeps track of if the cause was
> explicitely set. So if cause was set (i.e. `from X` syntax is used), then
> always display it in favor of the context ? except that a None-cause causes
> nothing to display.
> 
> Regardless of that I?m actually not sure if just changing the way the cause
> is displayed is a correct way to handle the context. If I explicitely raise
> an exception in an except-handler, I usually don?t expect that new
> exception to get the previous exception attached to.

I'm inclined to agree. I'm not convinced that "raise MyError from None" should 
leave any trace of the old exception at all.

Suppose you have a couple of functions like this:

def spam(x):    # LBYL
     if hasattr(x, "__len__"):
         y = len(x)+1
     else:
         raise MyError("x has no length")
     do_stuff_with(y)

def ham(x):    # EAFP
     try:
         y = len(x)+1
     except TypeError:
         raise MyError("x has no length") from None
     do_stuff_with(y)

It is entirely an irrelevant implementation detail whether you happen to write 
spam or ham. The exception that the caller gets should, in my opinion, be the 
same. I can't see any benefit to exposing the TypeError, even if the caller 
has to overtly go looking for it in order to see it.

But having said that, if others have a specific scenario in mind where they 
would need to distinguish between spam and ham, I'm happy for the context to 
be set. But I am curious to learn what the scenario is. Is it just a matter of 
"just in case"?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:31:12 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 23:31:12 +0000
Subject: [issue13910] test_packaging is dependent on dict ordering.
Message-ID: <1327879872.45.0.0954403986846.issue13910@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

Changing PyDict_MINSIZE to 4 causes the following failure:

python -m test.test_packaging
...
Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/mark/python/cpython/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/mark/python/cpython/Lib/test/test_packaging.py", line 5, in <module>
    test_main()
  File "/home/mark/python/cpython/Lib/test/support.py", line 1496, in decorator
    return func(*args)
  File "/home/mark/python/cpython/Lib/packaging/tests/__main__.py", line 18, in test_main
    run_unittest(test_loader.discover(start_dir, top_level_dir=top_dir))
  File "/home/mark/python/cpython/Lib/test/support.py", line 1403, in run_unittest
    _run_suite(suite)
  File "/home/mark/python/cpython/Lib/test/support.py", line 1378, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/packaging/tests/test_create.py", line 168, in test_convert_setup_py_to_cfg
    """))
AssertionError: '[metadata]\nname = pyxfoil\nversion = 0.2\nsummary = Python bindings for the Xf [truncated]... != '[metadata]\nname = pyxfoil\nversion = 0.2\nsummary = Python bindings for the Xf [truncated]...
  [metadata]
  name = pyxfoil
  version = 0.2
  summary = Python bindings for the Xfoil engine
  download_url = UNKNOWN
  home_page = http://www.python-science.org/project/pyxfoil
  maintainer = Andr? Espaze
  maintainer_email = andre.espaze at logilab.fr
  description = My super Death-scription
         |barbar is now on the public domain,
         |ho, baby !

  [files]
  packages = pyxfoil
      babar
      me
  modules = my_lib
      mymodule
  scripts = my_script
      bin/run
  extra_files = Martinique/Lamentin/dady
      Martinique/Lamentin/mumy
      Martinique/Lamentin/sys
      Martinique/Lamentin/bro
      setup.py
      README
-     pyxfoil/fengine.so
      Pom
      Flora
      Alexander
+     pyxfoil/fengine.so

  resources =
      README.rst = {doc}
      pyxfoil.1 = {man}

----------
messages: 152286
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: test_packaging is dependent on dict ordering.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13910>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:32:24 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Sun, 29 Jan 2012 23:32:24 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327879944.41.0.908548650508.issue13868@psf.upfronthosting.co.za>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

Wow.. do you really expect any positive outcome from you reply style?

I'll pretend I didn't read your reply and let me rephrase my question like this: there are several occurrences of 'floating point' in python doc, are you going to fix all the ones that are wrong?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:35:52 2012
From: report at bugs.python.org (Mark Shannon)
Date: Sun, 29 Jan 2012 23:35:52 +0000
Subject: [issue13911] test_trace  depends on dict repr() ordering
Message-ID: <1327880152.89.0.0711859595035.issue13911@psf.upfronthosting.co.za>


New submission from Mark Shannon <mark at hotpy.org>:

Changing PyDict_MINSIZE to 4 causes the following failure
python -m test.test_trace  
... 
======================================================================
ERROR: test_coverage (__main__.TestCoverage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/test/test_trace.py", line 307, in test_coverage
    self._coverage(tracer)
  File "/home/mark/python/cpython/Lib/test/test_trace.py", line 300, in _coverage
    tracer.run(cmd)
  File "/home/mark/python/cpython/Lib/trace.py", line 501, in run
    self.runctx(cmd, dict, dict)
  File "/home/mark/python/cpython/Lib/trace.py", line 509, in runctx
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/mark/python/cpython/Lib/test/test_pprint.py", line 479, in test_main
    test.support.run_unittest(QueryTestCase)
  File "/home/mark/python/cpython/Lib/test/support.py", line 1403, in run_unittest
    _run_suite(suite)
  File "/home/mark/python/cpython/Lib/test/support.py", line 1378, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/test/test_pprint.py", line 273, in test_set_reprs
    self.assertEqual(pprint.pformat(cube), cube_repr_tgt)
AssertionError: '{frozenset(): frozenset({frozenset({2}), frozenset({0}), frozenset({1})}),\n fr [truncated]... != '{frozenset(): frozenset({frozenset({2}), frozenset({0}), frozenset({1})}),\n fr [truncated]...
Diff is 1436 characters long. Set self.maxDiff to None to see it.


======================================================================
ERROR: test_coverage_ignore (__main__.TestCoverage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/test/test_trace.py", line 322, in test_coverage_ignore
    self._coverage(tracer)
  File "/home/mark/python/cpython/Lib/test/test_trace.py", line 300, in _coverage
    tracer.run(cmd)
  File "/home/mark/python/cpython/Lib/trace.py", line 501, in run
    self.runctx(cmd, dict, dict)
  File "/home/mark/python/cpython/Lib/trace.py", line 509, in runctx
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/mark/python/cpython/Lib/test/test_pprint.py", line 479, in test_main
    test.support.run_unittest(QueryTestCase)
  File "/home/mark/python/cpython/Lib/test/support.py", line 1403, in run_unittest
    _run_suite(suite)
  File "/home/mark/python/cpython/Lib/test/support.py", line 1378, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "/home/mark/python/cpython/Lib/test/test_pprint.py", line 273, in test_set_reprs
    self.assertEqual(pprint.pformat(cube), cube_repr_tgt)
AssertionError: '{frozenset(): frozenset({frozenset({2}), frozenset({0}), frozenset({1})}),\n fr [truncated]... != '{frozenset(): frozenset({frozenset({2}), frozenset({0}), frozenset({1})}),\n fr [truncated]...
Diff is 1436 characters long. Set self.maxDiff to None to see it.

----------
components: Tests
messages: 152288
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: test_trace  depends on dict repr() ordering
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13911>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:39:16 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sun, 29 Jan 2012 23:39:16 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327877315.66.0.157783616911.issue13856@psf.upfronthosting.co.za>
Message-ID: <20120129233906.GA6478@mathmagic>


Senthil Kumaran <senthil at uthcode.com> added the comment:

On Sun, Jan 29, 2012 at 10:48:35PM +0000, Martin v. L?wis wrote:
> orsenthil: I don't fully understand your question (what kind of "carrying" should the clients do); 

By that I mean, sending the ca_file and cert_reqs from the client,
which I believe would be required if you want to verify the server
certificate from the client end [1]. The other clients send only
the cert_file and the key_file.

[1]: http://code.activestate.com/recipes/577548-https-httplib-client-connection-with-certificate-v/

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:42:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Sun, 29 Jan 2012 23:42:15 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327880535.51.0.390396847127.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Constant arguments

What do you call a constant argument? "float" and "decimal"? You would prefer a constant like time.FLOAT_FORMAT? Or maybe a boolean (decimal=True)?

I chose a string because my first idea was to add a registry to support other format, maybe user defined formats, like the one used by Unicode codecs.

If we choose to not support other formats, but only float and decimal, a simpler API can be designed.

Another possible format would be "tuple": (intpart: int, floatpart: int, divisor: int), a low level type used to "implement" other user-defined types. Using such tuple, you have all information (clock value and clock resolution) without losing information.

> varying return type

I agree that it is something uncommon in Python. I know os.listdir(bytes)->bytes and os.listdir(str)->str. I suppose that there are other functions with a different result type depending on the input.

I am not attached to my API, it was just a proposition.

> hidden import

Ah? I wouldn't call it hidden because I don't see how a function can return a decimal.Decimal object without importing it. If you consider that it is surprising (unexepected), it can be documented.

> and the list can go on.

What else?

> What is wrong with simply creating a new module, say "hirestime"
> with functions called decimal_time(), float_time(), datetime_time()
> and whatever else you would like.

Hum, adding a new module would need to duplicate code. The idea of adding an argument is also to simplify the implementation: most code is shared. We can still share a lot of code if we choose to add a new function in th time module instead of adding a new argument to existing functions.

> Let's keep the good old 'time' module simple.

What is complex in my patch? It doesn't break backward compatibility and should have a low (or null) overhead in runtime speed if the format is not set.

--

I notified something surprising in my patch: "t1=time.time("decimal"); t2=time.time("decimal"); t2-t1" returns something bigger than 20 ms... That's because the "import decimal" is done after reading the first clock value, and not before.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:55:02 2012
From: report at bugs.python.org (Stefan Krah)
Date: Sun, 29 Jan 2012 23:55:02 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327878476.15.0.0503395950062.issue13874@psf.upfronthosting.co.za>
Message-ID: <20120129235501.GA8883@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

STINNER Victor <report at bugs.python.org> wrote:
> Does my commit fix the issue?

Yes, perfectly.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Mon Jan 30 00:55:29 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Sun, 29 Jan 2012 23:55:29 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327876245.67.0.23679510461.issue13856@psf.upfronthosting.co.za>
Message-ID: <20120129235520.GB6478@mathmagic>


Senthil Kumaran <senthil at uthcode.com> added the comment:

On Sun, Jan 29, 2012 at 10:30:45PM +0000, Antoine Pitrou wrote:
> 
> Well, if you are a security expert you can volunteer to maintain a
> trusted certificates' file in the Python repository :) I think
> nobody else amongst us is qualified.

:-) haha. Neither am I (and I didn't figure out immediately that,
that's the requirement to go forward, so definitely not the person.)

But given that curl [1] has adopted the approach the user will know
where the CA cert will be and let's the user specify the details, I
wondered if the just having the provision to give ca_cert details is a
good thing.

Yeah, I recollect a discussion that there is not a standard
file-system path where OS store the certs and relying on them is not a
good idea.

[1] http://curl.haxx.se/docs/sslcerts.html

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Mon Jan 30 01:05:56 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 00:05:56 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327881956.05.0.941632758958.issue13903@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I see two unrelated parts in your patch:
 - change dictionary structure in memory
 - change many constants linked to optimization: PyDICT_MAXFREELIST: 80->40, 2/3->5/8, etc.

You may open a new issue for the second part, except if I am wrong and you need to change constants for the first part?

(I don't understand why you changed constants and how you chose new values.)

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Mon Jan 30 01:22:25 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 30 Jan 2012 00:22:25 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327882945.46.0.467662392754.issue6210@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

This was discussed a little more in the python-dev thread for PEP 409, but both Guido and I have been burned in the past by badly written libraries that replaced detailed exceptions that explained *exactly* what was going wrong with bland, generic "it broke!" exceptions that told us nothing. What should have been a 5 minute fix turns into a long bug hunt because useful information was being thrown away.

With __context__ always being set, all you need to do to cope with inappropriate use of "raise X from None" by libraries is write your own exception handler that always reports the entire exception chain, regardless of the __cause__ setting. If "raise X from None" actually *clobbers* the context, though, you instead have to go in and try to get hold of the detailed exception information *before* it gets clobbered (which is a lot harder to do).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Mon Jan 30 01:38:09 2012
From: report at bugs.python.org (Patrick Westerhoff)
Date: Mon, 30 Jan 2012 00:38:09 +0000
Subject: [issue6210] Exception Chaining missing method for suppressing context
In-Reply-To: <1244228665.64.0.918101455838.issue6210@psf.upfronthosting.co.za>
Message-ID: <1327883889.39.0.548946275775.issue6210@psf.upfronthosting.co.za>


Patrick Westerhoff <PatrickWesterhoff at gmail.com> added the comment:

Oh, where did that PEP come from? ^^ Also thanks for hinting at python-dev, didn?t realize that there was a discussion going on about this!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6210>
_______________________________________

From report at bugs.python.org  Mon Jan 30 01:42:32 2012
From: report at bugs.python.org (John O'Connor)
Date: Mon, 30 Jan 2012 00:42:32 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327884152.43.0.0319384526793.issue13903@psf.upfronthosting.co.za>


Changes by John O'Connor <tehjcon at gmail.com>:


----------
nosy: +jcon

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Mon Jan 30 01:45:23 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 30 Jan 2012 00:45:23 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <20120129233906.GA6478@mathmagic>
Message-ID: <4F25E822.8050804@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> By that I mean, sending the ca_file and cert_reqs from the client,
> which I believe would be required if you want to verify the server
> certificate from the client end [1]. The other clients send only
> the cert_file and the key_file.

Ah, you are probably referring to the "server name indication",
which is a fairly recent TLS feature.

Python does support it, in the server_hostname context field.

You never ever send the CA certificate to the server in TLS,
AFAIK (except as part of a chain including your own client
certificate).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Mon Jan 30 02:17:27 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 30 Jan 2012 01:17:27 +0000
Subject: [issue13908] PyType_FromSpec() lacks PyType_Ready() call
In-Reply-To: <1327879029.72.0.4665491693.issue13908@psf.upfronthosting.co.za>
Message-ID: <E1Rrfru-0005QD-Bo@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 53b8f55e08bd by Benjamin Peterson in branch 'default':
merge 3.2 (closes #13908)
http://hg.python.org/cpython/rev/53b8f55e08bd

----------
nosy: +python-dev
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13908>
_______________________________________

From report at bugs.python.org  Mon Jan 30 02:38:40 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 01:38:40 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1327887520.15.0.329658528191.issue13845@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Two articles (Microsoft and IBM) about high resolution time on Windows:
http://msdn.microsoft.com/en-us/magazine/cc163996.aspx
http://www.ibm.com/developerworks/library/i-seconds/

I installed the Windows port of the NTP daemon:
http://www.meinberg.de/english/sw/ntp.htm

Using the NTP daemon, the resolution is 1 ms instead of 15 ms (on Windows Seven). It looks like it is possible to have a resolution of 0.5 ms, but I failed to get this resolution.

Attached patch is much more simple than the previous one: it only changes _PyTime_gettimeofday().

----------
Added file: http://bugs.python.org/file24364/GetSystemTimeAsFileTime.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Mon Jan 30 02:39:24 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 30 Jan 2012 01:39:24 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327795553.2219.128.camel@surprise>
Message-ID: <1327887514.2444.8.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sun, 2012-01-29 at 00:06 +0000, Dave Malcolm wrote:

I went ahead and added the flag to sys.flags, so now
  $ make test TESTPYTHONOPTS=-R
shows:
Testing with flags: sys.flags(debug=0, division_warning=0, inspect=0,
interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0,
no_site=0, ignore_environment=1, verbose=0, bytes_warning=2,
hash_randomization=1)

...note the:
  hash_randomization=1
at the end of sys.flags.  (This seems useful for making it absolutely
clear if you're getting randomization or not).  Hopefully I'm not
creating too much work for the other Python implementations.

Am attaching new version of patch for 3.1:
  optin-hash-randomization-for-3.1-dmalcolm-2012-01-29-001.patch

----------
Added file: http://bugs.python.org/file24365/optin-hash-randomization-for-3.1-dmalcolm-2012-01-29-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 73dad4940b88 Doc/library/sys.rst
--- a/Doc/library/sys.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/library/sys.rst	Sun Jan 29 20:19:11 2012 -0500
@@ -220,6 +220,7 @@
    :const:`ignore_environment`   :option:`-E`
    :const:`verbose`              :option:`-v`
    :const:`bytes_warning`        :option:`-b`
+   :const:`hash_randomization`   :option:`-R`
    ============================= =============================
 
 
diff -r 73dad4940b88 Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/reference/datamodel.rst	Sun Jan 29 20:19:11 2012 -0500
@@ -1265,6 +1265,8 @@
    inheritance of :meth:`__hash__` will be blocked, just as if :attr:`__hash__`
    had been explicitly set to :const:`None`.
 
+   See also the :option:`-R` command-line option.
+
 
 .. method:: object.__bool__(self)
 
diff -r 73dad4940b88 Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/using/cmdline.rst	Sun Jan 29 20:19:11 2012 -0500
@@ -21,7 +21,7 @@
 
 When invoking Python, you may specify any of these options::
 
-    python [-bBdEhiOsSuvVWx?] [-c command | -m module-name | script | - ] [args]
+    python [-bBdEhiORsSuvVWx?] [-c command | -m module-name | script | - ] [args]
 
 The most common use case is, of course, a simple invocation of a script::
 
@@ -215,6 +215,30 @@
    Discard docstrings in addition to the :option:`-O` optimizations.
 
 
+.. cmdoption:: -R
+
+   Turn on "hash randomization, so that the :meth:`__hash__` values of str,
+   bytes and datetime objects are "salted" with an unpredictable random value.
+   Although they remain constant within an individual Python process, they
+   are not predictable between repeated invocations of Python.
+
+   This is intended to provide protection against a denial-of-service
+   caused by carefully-chosen inputs that exploit the worst case performance
+   of a dict lookup, O(n^2) complexity.  See:
+
+       http://www.ocert.org/advisories/ocert-2011-003.html
+
+   for details.
+
+   Changing hash values affects the order in which keys are retrieved from
+   a dict.  Although Python has never made guarantees about this ordering
+   (and it typically varies between 32-bit and 64-bit builds), enough
+   real-world code implicitly relies on this non-guaranteed behavior that
+   the randomization is disabled by default.
+
+   See also :envvar:`PYTHONHASHRANDOMIZATION`.
+
+
 .. cmdoption:: -s
 
    Don't add user site directory to sys.path
@@ -435,6 +459,25 @@
    import of source modules.
 
 
+.. envvar:: PYTHONHASHRANDOMIZATION
+
+   If this is set to a non-empty string it is equivalent to specifying the
+   :option:`-R` option.
+
+
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for generating the hash() of
+   the types covered by the :option:`-R` option (or its equivalent,
+   :envvar:`PYTHONHASHRANDOMIZATION`.
+
+   It is primarily intended for use in selftests for the interpreter, but
+   may perhaps be of use for reproducing a specific dict ordering.
+
+   It should be a decimal number in the range [0; 4294967295].  Specifying
+   the value 0 overrides the other setting, disabling the hash random salt.
+
+
 .. envvar:: PYTHONIOENCODING
 
    Overrides the encoding used for stdin/stdout/stderr, in the syntax
diff -r 73dad4940b88 Include/object.h
--- a/Include/object.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/object.h	Sun Jan 29 20:19:11 2012 -0500
@@ -473,6 +473,12 @@
 PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
+typedef struct {
+    long prefix;
+    long suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
+
 /* Helper for passing objects to printf and the like */
 #define PyObject_REPR(obj) _PyUnicode_AsString(PyObject_Repr(obj))
 
diff -r 73dad4940b88 Include/pydebug.h
--- a/Include/pydebug.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pydebug.h	Sun Jan 29 20:19:11 2012 -0500
@@ -19,6 +19,7 @@
 PyAPI_DATA(int) Py_DontWriteBytecodeFlag;
 PyAPI_DATA(int) Py_NoUserSiteDirectory;
 PyAPI_DATA(int) Py_UnbufferedStdioFlag;
+PyAPI_DATA(int) Py_HashRandomizationFlag;
 
 /* this is a wrapper around getenv() that pays attention to
    Py_IgnoreEnvironmentFlag.  It should be used for getting variables like
diff -r 73dad4940b88 Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pythonrun.h	Sun Jan 29 20:19:11 2012 -0500
@@ -174,6 +174,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r 73dad4940b88 Lib/os.py
--- a/Lib/os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/os.py	Sun Jan 29 20:19:11 2012 -0500
@@ -611,23 +611,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r 73dad4940b88 Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/regrtest.py	Sun Jan 29 20:19:11 2012 -0500
@@ -428,6 +428,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(tests)
diff -r 73dad4940b88 Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_cmd_line.py	Sun Jan 29 20:19:11 2012 -0500
@@ -190,6 +190,22 @@
             self.assertTrue(path1.encode('ascii') in stdout)
             self.assertTrue(path2.encode('ascii') in stdout)
 
+    def test_hash_randomization(self):
+        # Verify that -R enables hash randomization:
+        self.verify_valid_flag('-R')
+        hashes = []
+        for i in range(2):
+            code = 'print(hash("spam"))'
+            data, rc = self.start_python_and_exit_code('-R', '-c', code)
+            self.assertEqual(rc, 0)
+            hashes.append(data)
+        self.assertNotEqual(hashes[0], hashes[1])
+
+        # Verify that sys.flags contains hash_randomization
+        code = 'import sys; print(sys.flags)'
+        data, rc = self.start_python_and_exit_code('-R', '-c', code)
+        self.assertEqual(rc,0)
+        self.assertIn(b'hash_randomization=1', data)
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff -r 73dad4940b88 Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_hash.py	Sun Jan 29 20:19:11 2012 -0500
@@ -3,10 +3,14 @@
 #
 # Also test that hash implementations are inherited as expected
 
+import struct
 import unittest
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -118,10 +122,65 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+class RandomizationTestCase(unittest.TestCase):
+
+    # Examples of the various types having randomized hash:
+    test_reprs = [repr('abc'), repr(b'abc')]
+
+    def get_hash(self, repr_, randomization=None, seed=None):
+        env = {}
+        if randomization is not None:
+            env['PYTHONHASHRANDOMIZATION'] = str(randomization)
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', 'print(hash(%s))' % repr_,
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+        self.assertEqual(hash(b""), 0)
+
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            known_hash_of_obj = 1453079729188098211
+        else:
+            known_hash_of_obj = -1600925533
+        for repr_ in self.test_reprs:
+            # Randomization is disabled by default:
+            self.assertEqual(self.get_hash(repr_), known_hash_of_obj)
+
+            # If enabled, it can still be disabled by setting the seed to 0:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=0),
+                             known_hash_of_obj)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        # Note that all types share the same values:
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        for repr_ in self.test_reprs:
+            self.assertEqual(self.get_hash(repr_, randomization=1, seed=42),
+                             h)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        for repr_ in self.test_reprs:
+            run1 = self.get_hash(repr_, randomization=1)
+            run2 = self.get_hash(repr_, randomization=1)
+            self.assertNotEqual(run1, run2)
+
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
                               HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                              HashBuiltinsTestCase,
+                         RandomizationTestCase)
 
 
 if __name__ == "__main__":
diff -r 73dad4940b88 Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_os.py	Sun Jan 29 20:19:11 2012 -0500
@@ -9,6 +9,7 @@
 import sys
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 
 # Detect whether we're on a Linux system that uses the (now outdated
 # and unmaintained) linuxthreads threading library.  There's an issue
@@ -574,14 +575,33 @@
         f.close()
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(0)), 0)
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 class ExecTests(unittest.TestCase):
     @unittest.skipIf(USING_LINUXTHREADS,
diff -r 73dad4940b88 Lib/test/test_set.py
--- a/Lib/test/test_set.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_set.py	Sun Jan 29 20:19:11 2012 -0500
@@ -734,6 +734,17 @@
         if self.repr is not None:
             self.assertEqual(repr(self.set), self.repr)
 
+    def check_repr_against_values(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
     def test_print(self):
         try:
             fo = open(support.TESTFN, "w")
@@ -892,7 +903,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -903,7 +916,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{b'a', b'c', b'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +931,13 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
 
     def tearDown(self):
         warnings.filters = self.warning_filters
 
+    def test_repr(self):
+        self.check_repr_against_values()
+
 #==============================================================================
 
 def baditer():
diff -r 73dad4940b88 Lib/test/test_sys.py
--- a/Lib/test/test_sys.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_sys.py	Sun Jan 29 20:19:11 2012 -0500
@@ -446,7 +446,7 @@
         attrs = ("debug", "division_warning",
                  "inspect", "interactive", "optimize", "dont_write_bytecode",
                  "no_user_site", "no_site", "ignore_environment", "verbose",
-                 "bytes_warning")
+                 "bytes_warning", "hash_randomization")
         for attr in attrs:
             self.assertTrue(hasattr(sys.flags, attr), attr)
             self.assertEqual(type(getattr(sys.flags, attr)), int, attr)
diff -r 73dad4940b88 Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 20 11:23:02 2012 +0000
+++ b/Makefile.pre.in	Sun Jan 29 20:19:11 2012 -0500
@@ -305,6 +305,7 @@
 		Python/pymath.o \
 		Python/pystate.o \
 		Python/pythonrun.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r 73dad4940b88 Misc/NEWS
--- a/Misc/NEWS	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/NEWS	Sun Jan 29 20:19:11 2012 -0500
@@ -10,6 +10,12 @@
 Core and Builtins
 -----------------
 
+- Issue #13703: add -R command-line option and PYTHONHASHRANDOMIZATION (and
+  PYTHONHASHSEED) environment variables, to provide an opt-in way to protect
+  against denial of service attacks due to hash collisions within the dict
+  and set types. See http://www.ocert.org/advisories/ocert-2011-003.html
+  Patch by David Malcolm, based on work by Victor Stinner.
+
 Library
 -------
 
diff -r 73dad4940b88 Misc/python.man
--- a/Misc/python.man	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/python.man	Sun Jan 29 20:19:11 2012 -0500
@@ -34,6 +34,9 @@
 .B \-OO
 ]
 [
+.B \-R
+]
+[
 .B -Q
 .I argument
 ]
@@ -145,6 +148,18 @@
 .B \-OO
 Discard docstrings in addition to the \fB-O\fP optimizations.
 .TP
+.B \-R
+Turn on "hash randomization", so that the hash() values of str, bytes and
+datetime objects are "salted" with an unpredictable pseudo-random value.
+Although they remain constant within an individual Python process, they are
+not predictable between repeated invocations of Python.
+.IP
+This is intended to provide protection against a denial of service
+caused by carefully-chosen inputs that exploit the worst case performance
+of a dict lookup, O(n^2) complexity.  See
+http://www.ocert.org/advisories/ocert-2011-003.html
+for details.
+.TP
 .BI "\-Q " argument
 Division control; see PEP 238.  The argument must be one of "old" (the
 default, int/int and long/long return an int or long), "new" (new
@@ -403,6 +418,17 @@
 If this is set to a non-empty string it is equivalent to specifying
 the \fB\-v\fP option. If set to an integer, it is equivalent to
 specifying \fB\-v\fP multiple times. 
+.IP PYTHONHASHRANDOMIZATION
+If this is set to a non-empty string it is equivalent to specifying the
+\fB\-R\fP option.
+.IP PYTHONHASHSEED
+If this is set, it is used as a fixed seed for generating the hash() of
+the types covered by the \fB\-R\fP  (or its equivalent environment variable,
+\fB\PYTHONHASHRANDOMIZATION\fP).
+It is primarily intended for use in selftests for the interpreter, but
+may perhaps be of use for reproducing a specific dict ordering.
+It should be a decimal number in the range [0; 4294967295].  Specifying
+the value 0 overrides the other setting, disabling the hash random salt.
 .SH AUTHOR
 The Python Software Foundation: http://www.python.org/psf
 .SH INTERNET RESOURCES
diff -r 73dad4940b88 Modules/datetimemodule.c
--- a/Modules/datetimemodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/datetimemodule.c	Sun Jan 29 20:19:11 2012 -0500
@@ -2566,10 +2566,12 @@
     register long x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= len;
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
 
diff -r 73dad4940b88 Modules/main.c
--- a/Modules/main.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/main.c	Sun Jan 29 20:19:11 2012 -0500
@@ -47,7 +47,7 @@
 static int  orig_argc;
 
 /* command line options */
-#define BASE_OPTS L"bBc:dEhiJm:OsStuvVW:xX?"
+#define BASE_OPTS L"bBc:dEhiJm:ORsStuvVW:xX?"
 
 #define PROGRAM_OPTS BASE_OPTS
 
@@ -72,6 +72,9 @@
 -m mod : run library module as a script (terminates option list)\n\
 -O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x\n\
 -OO    : remove doc-strings in addition to the -O optimizations\n\
+-R     : use a pseudo-random salt to make hash() values of various types be\n\
+         unpredictable between separate invocations of the interpreter, as\n\
+         a defence against denial-of-service attacks\n\
 -s     : don't add user site directory to sys.path; also PYTHONNOUSERSITE\n\
 -S     : don't imply 'import site' on initialization\n\
 ";
@@ -99,6 +102,11 @@
 PYTHONCASEOK : ignore case in 'import' statements (Windows).\n\
 PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n\
 ";
+static char *usage_6 = "\
+PYTHONHASHRANDOMIZATION : make hash() values of various types be unpredictable\n\
+                          between separate invocations of the interpreter, as\n\
+                          a defence against denial-of-service attacks\n\
+";
 
 #ifndef MS_WINDOWS
 static FILE*
@@ -136,6 +144,7 @@
         fputs(usage_3, f);
         fprintf(f, usage_4, DELIM);
         fprintf(f, usage_5, DELIM, PYTHONHOMEHELP);
+        fputs(usage_6, f);
     }
 #if defined(__VMS)
     if (exitcode == 0) {
@@ -373,6 +382,10 @@
             PySys_AddWarnOption(_PyOS_optarg);
             break;
 
+        case 'R':
+            Py_HashRandomizationFlag++;
+            break;
+
         /* This space reserved for other options */
 
         default:
diff -r 73dad4940b88 Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/posixmodule.c	Sun Jan 29 20:19:11 2012 -0500
@@ -6942,82 +6942,6 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
-static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
-    }
-    return result;
-}
-#endif
-
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
 Return a string describing the encoding of the device\n\
@@ -7055,41 +6979,35 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+Return n pseudo-random bytes.");
+
+static PyObject *
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
+ 
+     /* Read arguments */
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -7374,12 +7292,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
     {NULL,              NULL}            /* Sentinel */
 };
 
diff -r 73dad4940b88 Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/bytesobject.c	Sun Jan 29 20:19:11 2012 -0500
@@ -899,11 +899,17 @@
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
+    if (len == 0) {
+        a->ob_shash = 0;
+        return 0;
+    }
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(a);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff -r 73dad4940b88 Objects/object.c
--- a/Objects/object.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/object.c	Sun Jan 29 20:19:11 2012 -0500
@@ -712,6 +712,8 @@
     return -1;
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
 long
 PyObject_Hash(PyObject *v)
 {
diff -r 73dad4940b88 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/unicodeobject.c	Sun Jan 29 20:19:11 2012 -0500
@@ -7344,11 +7344,17 @@
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
+    if (len == 0) {
+        self->hash = 0;
+        return 0;
+    }
     p = self->str;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(self);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     self->hash = x;
diff -r 73dad4940b88 PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 20 11:23:02 2012 +0000
+++ b/PCbuild/pythoncore.vcproj	Sun Jan 29 20:19:11 2012 -0500
@@ -1778,6 +1778,10 @@
 				RelativePath="..\Python\pythonrun.c"
 				>
 			</File>
+ 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
 			<File
 				RelativePath="..\Python\structmember.c"
 				>
diff -r 73dad4940b88 Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/pythonrun.c	Sun Jan 29 20:19:11 2012 -0500
@@ -73,6 +73,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 
 #ifdef WITH_THREAD
 extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@@ -91,6 +92,7 @@
 int Py_IgnoreEnvironmentFlag; /* e.g. PYTHONPATH, PYTHONHOME */
 int Py_NoUserSiteDirectory = 0; /* for -s and site.py */
 int Py_UnbufferedStdioFlag = 0; /* Unbuffered binary std{in,out,err} */
+int Py_HashRandomizationFlag = 0; /* for -R and PYTHONHASHRANDOMIZATION */
 
 /* PyModule_GetWarningsModule is no longer necessary as of 2.6
 since _warnings is builtin.  This API should not be used. */
@@ -195,6 +197,10 @@
         Py_OptimizeFlag = add_flag(Py_OptimizeFlag, p);
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
+    if ((p = Py_GETENV("PYTHONHASHRANDOMIZATION")) && *p != '\0')
+        Py_HashRandomizationFlag = add_flag(Py_HashRandomizationFlag, p);
+    
+    _PyRandom_Init();
 
     interp = PyInterpreterState_New();
     if (interp == NULL)
diff -r 73dad4940b88 Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Sun Jan 29 20:19:11 2012 -0500
@@ -0,0 +1,284 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Failed to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Failed to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Failed to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Failed to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Failed to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    /*
+      By default, hash randomization is disabled, and only
+      enabled if PYTHONHASHRANDOMIZATION is set to non-empty
+      or if "-R" is provided at the command line:
+    */
+    if (!Py_HashRandomizationFlag) {
+        /* Disable the randomized hash: */
+        memset(secret, 0, secret_size);
+        return;
+    }
+
+    /*
+      Hash randomization is enabled.  Generate a per-process secret,
+      using PYTHONHASHSEED if provided.
+    */
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else /* #ifdef MS_WINDOWS */
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+
diff -r 73dad4940b88 Python/sysmodule.c
--- a/Python/sysmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/sysmodule.c	Sun Jan 29 20:19:11 2012 -0500
@@ -1126,6 +1126,7 @@
     /* {"unbuffered",                   "-u"}, */
     /* {"skip_first",                   "-x"}, */
     {"bytes_warning", "-b"},
+    {"hash_randomization", "-R"},
     {0}
 };
 
@@ -1134,9 +1135,9 @@
     flags__doc__,       /* doc */
     flags_fields,       /* fields */
 #ifdef RISCOS
+    13
+#else
     12
-#else
-    11
 #endif
 };
 
@@ -1169,6 +1170,7 @@
     /* SetFlag(saw_unbuffered_flag); */
     /* SetFlag(skipfirstline); */
     SetFlag(Py_BytesWarningFlag);
+    SetFlag(Py_HashRandomizationFlag);
 #undef SetFlag
 
     if (PyErr_Occurred()) {

From report at bugs.python.org  Mon Jan 30 02:44:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 01:44:15 +0000
Subject: [issue13874] test_faulthandler: read_null test fails with current
	clang
In-Reply-To: <1327584887.45.0.833971151143.issue13874@psf.upfronthosting.co.za>
Message-ID: <1327887855.65.0.937950992473.issue13874@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13874>
_______________________________________

From report at bugs.python.org  Mon Jan 30 02:44:15 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 30 Jan 2012 01:44:15 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F248B01.4050702@udel.edu>
Message-ID: <1327887817.2444.12.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

On Sat, 2012-01-28 at 23:56 +0000, Terry J. Reedy wrote:
> Terry J. Reedy <tjreedy at udel.edu> added the comment:
> 
> > I think you should check with randomization enabled, if only to see the
> > nature of the failures and if they are expected.
> 
> Including the list of when-enabled expected failures in the release 
> notes would help those who compile and test.

Am attaching a patch which fixes various problems that are clearly just
assumptions about dict ordering:
  fix-unittests-broken-by-randomization-dmalcolm-2012-01-29-001.patch

 json/__init__.py                        |    4 +++-
 test/mapping_tests.py                   |    2 +-
 test/test_descr.py                      |   12 +++++++++++-
 test/test_urllib.py                     |    4 +++-
 tkinter/test/test_ttk/test_functions.py |    2 +-
 5 files changed, 19 insertions(+), 5 deletions(-)

Here are the issues that it fixes:
Lib/test/test_descr.py: fix for intermittent failure due to dict repr:
      File "Lib/test/test_descr.py", line 4304, in test_repr
        self.assertEqual(repr(self.C.__dict__), 'dict_proxy({!r})'.format(dict_))
    AssertionError: "dict_proxy({'__module__': 'test.test_descr', '__dict__': <attribute '__dict__' of 'C' objects>, '__doc__': None, '__weakref__': <attribute '__weakref__' of 'C' objects>, 'meth': <function meth at 0x5834be0>})"
                 != "dict_proxy({'__module__': 'test.test_descr', '__doc__': None, '__weakref__': <attribute '__weakref__' of 'C' objects>, 'meth': <function meth at 0x5834be0>, '__dict__': <attribute '__dict__' of 'C' objects>})"

Lib/json/__init__.py: fix (based on haypo's work) for intermittent failure:
    Failed example:
        json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',', ':'))
    Expected:
        '[1,2,3,{"4":5,"6":7}]'
    Got:
        '[1,2,3,{"6":7,"4":5}]'

Lib/test/mapping_tests.py: fix (based on haypo's work) for intermittent failures of test_collections, test_dict, and test_userdict seen here:
    ======================================================================
    ERROR: test_update (__main__.GeneralMappingTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "Lib/test/mapping_tests.py", line 207, in test_update
        i1 = sorted(d.items())
    TypeError: unorderable types: str() < int()

Lib/test/test_urllib.py: fix (based on haypo's work) for intermittent failure:
    ======================================================================
    FAIL: test_nonstring_seq_values (__main__.urlencode_Tests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "Lib/test/test_urllib.py", line 844, in test_nonstring_seq_values
        urllib.parse.urlencode({"a": {"a": 1, "b": 1}}, True))
    AssertionError: 'a=a&a=b' != 'a=b&a=a'
    ----------------------------------------------------------------------

Lib/tkinter/test/test_ttk/test_functions.py: fix from haypo's patch for intermittent failure:
    Traceback (most recent call last):
      File "Lib/tkinter/test/test_ttk/test_functions.py", line 146, in test_format_elemcreate
        ('a', 'b'), a='x', b='y'), ("test a b", ("-a", "x", "-b", "y")))
    AssertionError: Tuples differ: ('test a b', ('-b', 'y', '-a',... != ('test a b', ('-a', 'x', '-b',...

I see two remaining issues (which this patch doesn't address):
test test_module failed -- Traceback (most recent call last):
  File "Lib/test/test_module.py", line 79, in test_clear_dict_in_ref_cycle
    self.assertEqual(destroyed, [1])
AssertionError: Lists differ: [] != [1]

test_multiprocessing
Exception AssertionError: AssertionError() in <Finalize object, dead> ignored

----------
Added file: http://bugs.python.org/file24366/unnamed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unnamed
Type: text/x-patch
Size: 3683 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-bugs-list/attachments/20120130/90821f5a/attachment.bin>

From report at bugs.python.org  Mon Jan 30 02:58:52 2012
From: report at bugs.python.org (Ned Deily)
Date: Mon, 30 Jan 2012 01:58:52 +0000
Subject: [issue13901] test_get_outputs (test_distutils) failure with
	--enable-shared on Mac OS X
In-Reply-To: <1327826805.0.0.304314009246.issue13901@psf.upfronthosting.co.za>
Message-ID: <1327888732.58.0.664618905142.issue13901@psf.upfronthosting.co.za>


Ned Deily <nad at acm.org> added the comment:

On OS X, the linker includes in executables the absolute path to referenced shared libraries and normally --enable-shared builds are only usable from their installed location, i.e. after doing 'make install'.  RUNSHARED is defined as it is on OS X so that during the build of the standard library, the newly built interpreter uses its shared library before being installed.  After a 'make install', RUNSHARED should not be used on OS X; the shared library will always be found via the absolute path linked in the executable.  So I think the right solution for the problem here is to bypass the fixup code, so something like this (note the current 2.7.x is now similar to 3.2.x and differs from 2.7.2):


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -211,5 +211,8 @@
         if runshared is None:
             cmd.library_dirs = ['.']
         else:
-            name, equals, value = runshared.partition('=')
-            cmd.library_dirs = value.split(os.pathsep)
+            if sys.platform == 'darwin':
+                cmd.library_dirs = []
+            else:
+                name, equals, value = runshared.partition('=')
+                cmd.library_dirs = value.split(os.pathsep)

----------
nosy: +ned.deily, ronaldoussoren
versions: +Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13901>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:08:18 2012
From: report at bugs.python.org (Senthil Kumaran)
Date: Mon, 30 Jan 2012 02:08:18 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327446175.17.0.819972733751.issue13856@psf.upfronthosting.co.za>
Message-ID: <1327889298.58.0.320437579153.issue13856@psf.upfronthosting.co.za>


Senthil Kumaran <senthil at uthcode.com> added the comment:

I think, I was using wrong terminology, by 'sending' I meant, 'using' the ca_file in the client to verify Server's certificates.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:42:49 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:42:49 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891369.97.0.0942190814227.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 4, minor update:
 - remove the resolution field of _PyTime_t and remove int_log10(): use 1/divisior as the resolution to support divisor different than a power of 10 (e.g. the cpu frequency on Windows)
 - inline and remove _PyTime_FromTimespec()

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:43:23 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:43:23 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891403.72.0.24914316904.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
assignee:  -> Arfrever
Added file: http://bugs.python.org/file24367/time_decimal-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:43:31 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:43:31 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891411.53.0.358558341878.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24330/time_decimal-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:43:32 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:43:32 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891412.67.0.277247495381.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24331/time_decimal-3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:47:57 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:47:57 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891677.14.0.251665222351.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Another possible format would be "tuple"

Or, I forgot an obvious format: "datetime"!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:49:24 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:49:24 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327891764.74.0.121113708337.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
assignee: Arfrever -> 

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 03:54:02 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 02:54:02 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327892042.72.0.434168472096.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 04:04:30 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 03:04:30 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327892670.69.0.18680052418.issue11457@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I don't like the idea of adding new fields to os.stat() *by default* because it may break backward compatibility. And if the new fields are decimal.Decimal objects, the module has to be imported and it means that any call to os.stat() would be slower just to provide timestamps with a finer resolution: this is not acceptable if you just want to check if a file exists.

In the issue #13882, I propose to add a format argument to functions getting time (time.time(), time.clock(), etc.). My patch doesn't change the default type, but add a "decimal" format to get time as a decimal.Decimal object.

The option (format) value is a string to be able to add other formats without having to change the API later:

 - "decimal": decimal.Decimal object
 - "datetime": datetime.datetime object
 - "tuple": low-level value, (intpart: int, floatpart: int, divisor: int)
 - "float128"
 - etc.

For os.stat(), the optional argument can be called "timestamp".

So if you want timestamps in the best available resolution, use timestamp="decimal". If you prefer the datetime API, use timestamp="datetime". If you don't care of timestamps, just call os.stat() without setting the timestamp option ;-)

We might add a registry to add user-defined types, but the "tuple" format should be enough. (I don't know if we need to expose the low level "tuple" format.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 04:21:59 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Mon, 30 Jan 2012 03:21:59 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327893719.5.0.977023684576.issue11457@psf.upfronthosting.co.za>


Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com> added the comment:

I think that one of available types of time values returned by os.stat() should allow to directly pass these values to os.futimens() and os.utimensat(), which expect (time_sec, time_nsec) tuples.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 07:02:38 2012
From: report at bugs.python.org (Paul Nasrat)
Date: Mon, 30 Jan 2012 06:02:38 +0000
Subject: [issue9207] multiprocessing occasionally spits out exception during
	shutdown (_handle_workers)
In-Reply-To: <1278619533.83.0.0853433126095.issue9207@psf.upfronthosting.co.za>
Message-ID: <1327903358.85.0.0318011615946.issue9207@psf.upfronthosting.co.za>


Changes by Paul Nasrat <pnasrat at gmail.com>:


----------
nosy: +pnasrat

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9207>
_______________________________________

From report at bugs.python.org  Mon Jan 30 07:44:38 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Mon, 30 Jan 2012 06:44:38 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327905878.57.0.0470063957574.issue4966@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I agree with 3.3 only. This might not be ready for 3.2.3 anyway, depending on how soon hash patch is ready, and if not, it becomes a somewhat moot point as new people should then download 3.3.0 instead of 3.2.4 next August.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 30 07:51:21 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 30 Jan 2012 06:51:21 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327906281.61.0.730714304485.issue4966@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

ISTM that not doing this will make maintenance harder.  For 2.7 I agree that there is no clear boundary to make, but 3.2 should be split up as well to ease merging of updates.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 30 08:15:04 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 30 Jan 2012 07:15:04 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F25CD6C.1040002@v.loewis.de>
Message-ID: <4F264371.7010908@in.waw.pl>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

What about PYTHONHASHSEED= -> off, PYTHONHASHSEED=0 -> random, 
PYTHONHASHSEED=n -> n ? I agree with Jim that it's better to have one 
env. variable than two.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 30 08:30:00 2012
From: report at bugs.python.org (Kang-Hao (Kenny) Lu)
Date: Mon, 30 Jan 2012 07:30:00 +0000
Subject: [issue12892] UTF-16 and UTF-32 codecs should reject (lone) surrogates
In-Reply-To: <1315133370.95.0.855318807974.issue12892@psf.upfronthosting.co.za>
Message-ID: <1327908600.25.0.550432988702.issue12892@psf.upfronthosting.co.za>


Kang-Hao (Kenny) Lu <kennyluck at csail.mit.edu> added the comment:

Attached patch does the following beyond what the patch from haypo does:
  * call the error handler
  * reject 0xd800~0xdfff when decoding utf-32

The followings are on my TODO list, although this patch doesn't depend on any of these and can be reviewed and landed separately:
  * make the surrogatepass error handler work for utf-16 and utf-32. (I should be able to finish this by today)
  * fix an error in the error handler for utf-16-le. (In, Python3.2 b'\xdc\x80\x00\x41'.decode('utf-16-be', 'ignore') returns "\x00" instead of "A" for some reason)
  * make unicode_encode_call_errorhandler return bytes so that we can simplify this patch. (This arguably belongs to a separate bug so I'll file it when needed)

> All UTF codecs should reject lone surrogates in strict error mode,

Should we really reject lone surrogates for UTF-7? There's a test in test_codecs.py that tests "\udc80" to be encoded b"+3IA-" (. Given that UTF-7 is not really part of the Unicode Standard and it is more like a "data encoding" than a "text encoding" to me, I am not sure it's a good idea.

> but let them pass using the surrogatepass error handler (the UTF-8
> codec already does) and apply the usual error handling for ignore
> and replace.

For 'replace', the patch now emits b"\x00?" instead of b"?" so that UTF-16 stream doesn't get corrupted. It is not "usual" and not matching

  # Implements the ``replace`` error handling: malformed data is replaced
  # with a suitable replacement character such as ``'?'`` in bytestrings 
  # and ``'\ufffd'`` in Unicode strings.

in the documentation. What do we do? Are there other encodings that are not ASCII compatible besides UTF-7, UTF-16 and UTF-32 that Python supports? I think it would be better to use encoded U+fffd whenever possible and fall back to '?'. What do you think?

Some other self comments on my patch:
  * In the STORECHAR macro for utf-16 and utf-32, I change all instances of "ch & 0xFF" to (unsigned char) ch. I don't have enough C knowledge to know if this is actually better or if this makes any difference at all.
  * The code for utf-16 and utf-32 are duplicates of the uft-8 one. That one's complexity comes from issue #8092 . Not sure if there are ways to simplify these. For example, are there suitable functions there so that we don't need to check integer overflow at these places?

----------
nosy: +kennyluck
Added file: http://bugs.python.org/file24368/utf-16&32_reject_surrogates.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12892>
_______________________________________

From report at bugs.python.org  Mon Jan 30 08:45:49 2012
From: report at bugs.python.org (Gregory P. Smith)
Date: Mon, 30 Jan 2012 07:45:49 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F264371.7010908@in.waw.pl>
Message-ID: <CAGE7PNLWro6vmN=RL3u6EftPO2TmwHmEXc0WdmDHctsxQdZnDg@mail.gmail.com>


Gregory P. Smith <greg at krypto.org> added the comment:

> What about PYTHONHASHSEED= -> off, PYTHONHASHSEED=0 -> random,
> PYTHONHASHSEED=n -> n ? I agree with Jim that it's better to have one
> env. variable than two.

Rather than the "" empty string for off I suggest an explicit string
that makes it clear what the meaning is.  PYTHONHASHSEED="disabled"
perhaps.

Agreed, if we can have a single env var that is preferred.  It is more
obvious that the PYTHONHASHSEED env var. has no effect when it is set
to a special value rather than when it is set to something but it is
configured to be ignored by a _different_ env var.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 30 09:08:01 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 30 Jan 2012 08:08:01 +0000
Subject: [issue4966] Improving Lib Doc Sequence Types Section
In-Reply-To: <1232149418.64.0.0450574767427.issue4966@psf.upfronthosting.co.za>
Message-ID: <1327910881.06.0.805173780643.issue4966@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Good point, without doing the split in both, any doc merges in this section will be a nightmare. OK, with the caveat that the initial 3.2 version may gloss over some issues that no longer apply in 3.3 (specifically the narrow/wide split), I'll make a new branch in the sandbox so the changes will be once again based on 3.2.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4966>
_______________________________________

From report at bugs.python.org  Mon Jan 30 09:09:29 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 30 Jan 2012 08:09:29 +0000
Subject: [issue13856] xmlrpc / httplib changes to allow for certificate
	verification
In-Reply-To: <1327889298.58.0.320437579153.issue13856@psf.upfronthosting.co.za>
Message-ID: <4F265037.3030901@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> I think, I was using wrong terminology, by 'sending' I meant, 'using' the ca_file in the client to verify Server's certificates.

Then I still don't understand your remarks. You said "is there any
reason for the clients in the stdlib are not carrying a ca_file",
but this isn't actually true: clients in the stdlib *do* carry
the CA file and *do* perform verification, see ftplib, nntplib,
imaplib, http.client, etc. It's just that it was forgotten in
xmlrpc.client.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13856>
_______________________________________

From report at bugs.python.org  Mon Jan 30 09:14:05 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 08:14:05 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1327893719.5.0.977023684576.issue11457@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwbC-cGJVWctdqLjTgehNYiBjsAPYRpZQCmKqi+3d-SAcg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I think that one of available types of time values returned by os.stat() should
> allow to directly pass these values to os.futimens() and os.utimensat(), which
> expect (time_sec, time_nsec) tuples.

If we choose to give the possibility to get decimal.Decimal objects,
we should also patch some functions to support Decimal objects, like
datetime.datetime.fromtimestamp() (today, it "just works", because a
Decimal object can be converted to float). We may accept Decimal in
os.futimens() and os.utimensat(), but I am not sure for this
particular case.

To come back to my format solution, we can also support classical C
structures to interact with C functions (in Python or more directly
using ctypes):
 - "timeval": (secs, usec) where sec and usec are int
 - "timespec": (secs, nsec) where sec and nsec are int

Or we may introduce conversion functions from other types like float,
Decimal or another type.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 09:16:06 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 30 Jan 2012 08:16:06 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <CAGE7PNLWro6vmN=RL3u6EftPO2TmwHmEXc0WdmDHctsxQdZnDg@mail.gmail.com>
Message-ID: <4F2651C4.2030903@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> Rather than the "" empty string for off I suggest an explicit string
> that makes it clear what the meaning is.  PYTHONHASHSEED="disabled"
> perhaps.
> 
> Agreed, if we can have a single env var that is preferred.  It is more
> obvious that the PYTHONHASHSEED env var. has no effect when it is set
> to a special value rather than when it is set to something but it is
> configured to be ignored by a _different_ env var.

I think this is bike-shedding. The requirements for environment
variables are
a) with no variable set, it must not do randomization
b) there must be a way to seed from the platform's RNG
Having an explicit seed actually is no requirement, so I'd propose
to drop PYTHONHASHSEED instead.

However, I really suggest to let the patch author (Dave Malcolm)
design the API within the constraints.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 30 09:51:06 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Mon, 30 Jan 2012 08:51:06 +0000
Subject: [issue12892] UTF-16 and UTF-32 codecs should reject (lone) surrogates
In-Reply-To: <1315133370.95.0.855318807974.issue12892@psf.upfronthosting.co.za>
Message-ID: <1327913466.02.0.834023803478.issue12892@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

Thanks for the patch!

>  * fix an error in the error handler for utf-16-le. (In, Python3.2 
> b'\xdc\x80\x00\x41'.decode('utf-16-be', 'ignore') returns "\x00" 
> instead of "A" for some reason)

This should probably be done on a separate patch that will be applied to 3.2/3.3 (assuming that it can go to 3.2).  Rejecting surrogates will go in 3.3 only.  (Note that lot of Unicode-related code changed between 3.2 and 3.3.)

> Should we really reject lone surrogates for UTF-7?

No, I meant only UTF-8/16/32; UTF-7 is fine as is.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12892>
_______________________________________

From report at bugs.python.org  Mon Jan 30 14:14:21 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 13:14:21 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1327893719.5.0.977023684576.issue11457@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwZvBkzAqNsBJLextiT2kPdZZ-fsFB1GrziEA7kQ_pX72g@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I think that one of available types of time values returned by os.stat() should allow to directly pass these values to os.futimens() and os.utimensat(), which expect (time_sec, time_nsec) tuples.

Oh, I realized that these two functions were added to Python 3.3, so
it is not too late to change their API. I would prefer to limit the
number of timestamp formats: Python 3.2 has float and datetime, I (and
Martin) propose to add Decimal to Python 3.3 (to get nanosecond
resolution). (sec, nsec) is a new format, except if Python 3.2 has
already functions expecting such tuple?

I know that the underlying C function expects a timespec structure,
but Python can try to use a higher level API, isn't it?

Decimal is more practical than a tuple because you can just write :
t2-t1 to compute a time delta. Decimal has other advantages (read the
issue for the full list ;-)).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:08:08 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 30 Jan 2012 14:08:08 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327932488.93.0.670096021775.issue13868@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

I?know?that?there?are?a?lot?of?non-hyphenated?occurences?of?"floating point"?in?Python?docs,?so?I?just?want to?raise?awareness?about?those occurences. Can?you fix?all?occurences?of?"floating?point"?(when?in a?role?of an?adjective)?to?"floating-point" throughout Python docs?

As a matter of fact, everything that is an adjective is hyphenated if there are 2 words that form the adjective.

Example: "This is built in." versus "This is a built-in method."

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:15:08 2012
From: report at bugs.python.org (Sandro Tosi)
Date: Mon, 30 Jan 2012 14:15:08 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327932488.93.0.670096021775.issue13868@psf.upfronthosting.co.za>
Message-ID: <CAPdtAj3nxWRXn63r-xq3s-DxU8kXmN8GxmJakFyQfEowvvQWUA@mail.gmail.com>


Sandro Tosi <sandro.tosi at gmail.com> added the comment:

On Mon, Jan 30, 2012 at 15:08, Bo?tjan Mejak <report at bugs.python.org> wrote:
> Can?you fix?all?occurences?of?"floating?point"?(when?in a?role?of an?adjective)?to?"floating-point" throughout Python docs?

What I was asking is if *you* (since you care about the topic) would
provide a patch fixing all that needs fixing. At that point there will
be people interesting reviewing your patch, instead of people been
told what to do.

Regards,
Sandro

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:20:58 2012
From: report at bugs.python.org (Arfrever Frehtes Taifersar Arahesis)
Date: Mon, 30 Jan 2012 14:20:58 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327933258.01.0.0057281920586.issue11457@psf.upfronthosting.co.za>


Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA at GMail.Com> added the comment:

(secs, nsecs) tuples are more practical in performance-critical applications (e.g. synchronization of timestamps between 2 trees with large number of files).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:24:09 2012
From: report at bugs.python.org (Stefan Krah)
Date: Mon, 30 Jan 2012 14:24:09 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327933449.61.0.40436135593.issue13868@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Congratulations: You (again) have found a fundamental error in TAOCP.
Example (Third Edition, page 233):

"The following definitions seem to be appropriate for base b, excess q,
floating point numbers ..."

----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:38:25 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 14:38:25 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1327933258.01.0.0057281920586.issue11457@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwa=r3aUDXWU=eXThftr6kJ21U0YFH_8mPARs8AS2YFvEg@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> (secs, nsecs) tuples are more practical in performance-critical applications
> (e.g. synchronization of timestamps between 2 trees with large number of files).

This is also why I propose an argument to choose the format: everyone
has a different use case and use cases are incompatible. Tuples are
quick to create but has not a practical API, datetime has a nice API
but also issues listed before by Martin (and don't support nanosecond
resolution currently), Decimal has a nice API but is expensive to
create, etc.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 15:43:42 2012
From: report at bugs.python.org (R. David Murray)
Date: Mon, 30 Jan 2012 14:43:42 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327934622.91.0.223644416559.issue11457@psf.upfronthosting.co.za>


R. David Murray <rdmurray at bitdance.com> added the comment:

There is also the fact that we have traditionally exposed thin wrappers around posix functions (and then were practical provided Windows emulations).  We aren't 100% consistent about this, but we are pretty consistent about it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 16:14:26 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 30 Jan 2012 15:14:26 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327936466.12.0.217400395105.issue13868@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

For the time being I don't find any other "floating point" occurences in Python docs. I found one and made a patch for it. Please apply the patch.  Thanks.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 16:14:57 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Mon, 30 Jan 2012 15:14:57 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327936497.2.0.466283682896.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Please apply this patch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 30 16:18:47 2012
From: report at bugs.python.org (Brian Curtin)
Date: Mon, 30 Jan 2012 15:18:47 +0000
Subject: [issue10580] Installer sentence in bold
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327936727.93.0.786738516218.issue10580@psf.upfronthosting.co.za>


Changes by Brian Curtin <brian at python.org>:


----------
nosy:  -brian.curtin

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 30 16:20:08 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 30 Jan 2012 15:20:08 +0000
Subject: [issue13901] test_get_outputs (test_distutils) failure with
	--enable-shared on Mac OS X
In-Reply-To: <1327826805.0.0.304314009246.issue13901@psf.upfronthosting.co.za>
Message-ID: <1327936808.23.0.338985549766.issue13901@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for the analysis Ned.  Can you apply your patch to distutils and packaging?  Also, what do you think about adding a buildbot configured with --enable-shared?

----------
components: +Distutils2
nosy: +alexis

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13901>
_______________________________________

From report at bugs.python.org  Mon Jan 30 16:58:08 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 30 Jan 2012 15:58:08 +0000
Subject: [issue1625] bz2.BZ2File doesn't support multiple streams
In-Reply-To: <1197624030.23.0.503522059328.issue1625@psf.upfronthosting.co.za>
Message-ID: <1327939088.08.0.516710569841.issue1625@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

> I think this support should be backported to Python 2.7 and 3.2.
I think our policy is pretty clear: the module docs did not say multiple streams were supported, so when support for them was added it was clearly a new feature.

> Current code can't decompress files generated by "pbzip2", fairly popular.
That is unfortunate, but I don?t think that the features of one tool, even popular, should make us break the policy.  That?s why I concur with Nadeem and think that a PyPI backport is the only way to provide multi-streams bz2 to previous versions.

Best regards

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1625>
_______________________________________

From report at bugs.python.org  Mon Jan 30 17:43:17 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Mon, 30 Jan 2012 16:43:17 +0000
Subject: [issue13901] test_get_outputs (test_distutils) failure with
	--enable-shared on Mac OS X
In-Reply-To: <1327826805.0.0.304314009246.issue13901@psf.upfronthosting.co.za>
Message-ID: <1327941797.27.0.590785598226.issue13901@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

I'd prefer a buildbot with --enable-framework and --enable-universalsdk (that is, one closer to the options used to build the installer for OSX).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13901>
_______________________________________

From report at bugs.python.org  Mon Jan 30 17:49:14 2012
From: report at bugs.python.org (Ronald Oussoren)
Date: Mon, 30 Jan 2012 16:49:14 +0000
Subject: [issue10910] pyport.h FreeBSD/Mac OS X "fix" causes errors in C++
	compilation
In-Reply-To: <1295040100.28.0.0907494738582.issue10910@psf.upfronthosting.co.za>
Message-ID: <1327942154.16.0.820901437951.issue10910@psf.upfronthosting.co.za>


Ronald Oussoren <ronaldoussoren at mac.com> added the comment:

The only real fix I found is to introduce "Py_" prefixed versions of all definitions in ctypes.h that are used in Python (that is Py_isalnum) and use that throughout the python source tree.

That's a pretty invasive patch though and would probably be off-limit for 2.7 and 3.2.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10910>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:00:32 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 30 Jan 2012 17:00:32 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327942832.88.0.78847860178.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Here's is an updated version: termsize.diff.6

Following Antoine Pitrou's comment get_terminal_size_raw is renamed back to query_terminal_size.

Tests are moved to test_os.py, and there's a new test for query_terminal_size -- the output from 'stty size' is parsed. If something fails (e.g. stty is not available), the test is skipped.

I noticed that bash uses $LINES, not $ROWS. I have renamed rows to lines everywhere, to follow existing convention.

The big remaining question seems to be one function vs. two functions.
I'm still convinced that the high-level two function is better. This issue originated in the argparse module, which can use proper terminal size to efficiently format messages. Of course argparse must use a function with a fallback -- if it cannot query, some default must be used. So argparse would implement the high-level function anyway. It might just as well be in os, available for others to be used. Anyway, the second function is 18 lines of python code (excluding docstring and whitespace) -- a bit too much to copy and paste.

----------
Added file: http://bugs.python.org/file24369/termsize.diff.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:00:48 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 30 Jan 2012 17:00:48 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
Message-ID: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>


New submission from Jason R. Coombs <jaraco at jaraco.com>:

The Python 2.7.2 docs say this about __import__:

    Positive values for level indicate the number of parent directories to search relative to the directory of the module calling __import__().

But I find that even when setting level=1, the parent directory is not searched.

I've created this project tree to replicate the issue:

    jaraco at devjaraco:~$ tree master
    master
    ??? __init__.py
    ??? pkgA
    ??? ??? foo.py
    ??? ??? __init__.py
    ??? pkgB
        ??? bar.py
        ??? __init.py

    2 directories, 5 files
    jaraco at devjaraco:~$ cat master/pkgA/foo.py

    bar = __import__('pkgB', level=1).bar
    jaraco at devjaraco:~$ cat master/pkgB/bar.py
    var = "success"

It fails as so with python 2.7.2:

    jaraco at devjaraco:~$ python2.7 -c "import master.pkgA.foo"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "master/pkgA/foo.py", line 2, in <module>
        bar = __import__('pkgB', level=1).bar
    ImportError: No module named pkgB


It's conceivable I'm not using this correctly, but if so, I'm unable to find my mistake. I've confirmed that foo.__name__ is 'master.pkgA.foo'. I've tried using level=2 (in case I was off by one, but that wasn't the case).

----------
components: Interpreter Core
messages: 152331
nosy: jason.coombs
priority: normal
severity: normal
status: open
title: ImportError using __import__ and relative level 1
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:05:44 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 30 Jan 2012 17:05:44 +0000
Subject: =?utf-8?q?=5Bissue10580=5D_Minor_grammar_change_in_Python=E2=80=99s_MSI_i?=
	=?utf-8?q?nstaller?=
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1327943144.6.0.244261677005.issue10580@psf.upfronthosting.co.za>


Changes by ?ric Araujo <merwok at netwok.org>:


----------
title: Installer sentence in bold -> Minor grammar change in Python?s MSI installer
versions:  -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:10:17 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 30 Jan 2012 17:10:17 +0000
Subject: [issue13851] Packaging distutils2 for Fedora
In-Reply-To: <1327409259.69.0.346795065424.issue13851@psf.upfronthosting.co.za>
Message-ID: <1327943417.31.0.681978148185.issue13851@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks for your support of distutils2, but this tracker is the wrong place for this bug tracker, as no RPM spec file will be added to the distutils2 repo.  If there are bugs in distutils2 that prevent you from making a RPM, please say so, otherwise I will close this report (but stay available via private email or the distutils-sig mailing list).

----------
assignee: tarek -> eric.araujo
status: open -> pending

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13851>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:22:22 2012
From: report at bugs.python.org (=?utf-8?q?=C3=89ric_Araujo?=)
Date: Mon, 30 Jan 2012 17:22:22 +0000
Subject: [issue12804] "make test" fails on systems without internet access
In-Reply-To: <1313935702.14.0.838210022398.issue12804@psf.upfronthosting.co.za>
Message-ID: <1327944142.43.0.562961113131.issue12804@psf.upfronthosting.co.za>


?ric Araujo <merwok at netwok.org> added the comment:

Thanks!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12804>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:24:23 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 17:24:23 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1327942832.88.0.78847860178.issue13609@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwZUeSvG69+gdqP+6XUndFY-d4Ox3LbYRS8X0E+nAyP-Xw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> I noticed that bash uses $LINES, not $ROWS. I have renamed rows to lines everywhere, to follow existing convention.

The stty program has "rows" and "columns" commands to set the terminal
size. The tput programs has "cols" and "lines" commands to get the
terminal size. The curses library uses (y, x), e.g. it has a
getmaxyx() function. I don't know the most common term, lines or rows.

Extract of http://stackoverflow.com/questions/1780483/lines-and-columns-environmental-variables-lost-in-a-script
"The variables $LINES and $COLUMNS are shell variables, not
environmental variables, and thus are not exported to the child
process (...)"

If I understood correctly, LINES and COLUMNS environment variables are
only set by the user. In a bash script, these variables are wrapper to
ioctl().

The tput program reads TERM environment variable to decide if LINES
and COLUMNS are used or not:
"-Ttype indicates  the type of terminal.  Normally this option is
unnecessary, because the default is taken from the environment
variable TERM.  If -T is specified, then the shell variables LINES and
COLUMNS will be ignored,and the operating system will not be queried
for the actual screen size."

Extract of http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html :
------
COLUMNS
    A decimal integer > 0 used to indicate the user's preferred width
in column positions for the terminal screen or window. (See column
position .) If this variable is unset or null, the implementation
determines the number of columns, appropriate for the terminal or
window, in an unspecified manner. When COLUMNS is set, any
terminal-width information implied by TERM will be overridden. Users
and portable applications should not set COLUMNS unless they wish to
override the system selection and produce output unrelated to the
terminal characteristics. The default value for the number of column
positions is unspecified because historical implementations use
different methods to determine values corresponding to the size of the
screen in which the utility is run. This size is typically known to
the implementation through the value of TERM or by more elaborate
methods such as extensions to the stty utility, or knowledge of how
the user is dynamically resizing windows on a bit-mapped display
terminal. Users should not need to set this variable in the
environment unless there is a specific reason to override the
implementation's default behaviour, such as to display data in an area
arbitrarily smaller than the terminal or window.

LINES
    A decimal integer > 0 used to indicate the user's preferred number
of lines on a page or the vertical screen or window size in lines. A
line in this case is a vertical measure large enough to hold the
tallest character in the character set being displayed. If this
variable is unset or null, the implementation determines the number of
lines, appropriate for the terminal or window (size, terminal baud
rate, and so forth), in an unspecified manner. When LINES is set, any
terminal-height information implied by TERM will be overridden. Users
and portable applications should not set LINES unless they wish to
override the system selection and produce output unrelated to the
terminal characteristics. The default value for the number of lines is
unspecified because historical implementations use different methods
to determine values corresponding to the size of the screen in which
the utility is run. This size is typically known to the implementation
through the value of TERM or by more elaborate methods such as
extensions to the stty utility, or knowledge of how the user is
dynamically resizing windows on a bit-mapped display terminal. Users
should not need to set this variable in the environment unless there
is a specific reason to override the implementation's default
behaviour, such as to display data in an area arbitrarily smaller than
the terminal or window.
------

> The big remaining question seems to be one function vs. two functions.

Not exactly. I suggested to keep only the simple function in the os
module, and add maybe a function with a higher level API (using
environment variables) in another module (e.g. in shutil).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:31:17 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 30 Jan 2012 17:31:17 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <4F2651C4.2030903@v.loewis.de>
Message-ID: <1327944638.2444.16.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

It's useful for the selftests, so I've kept PYTHONHASHSEED.  However,
I've removed it from the man page; the only other place it's mentioned
(in Doc/using/cmdline.rst) I now explicitly say that it exists just to
serve the interpreter's own selftests.

Am attaching a revised patch, which has the above change, plus some
tweaks to Lib/test/test_hash.py (adds test coverage for the datetime
hash randomization):
  optin-hash-randomization-for-3.1-dmalcolm-2012-01-30-001.patch

Has anyone had a chance to try this patch on Windows?  Martin?  I'm
hoping that it doesn't impose a startup cost in the default
no-randomization cost, and that any startup cost in the -R case is
acceptable.

----------
Added file: http://bugs.python.org/file24370/optin-hash-randomization-for-3.1-dmalcolm-2012-01-30-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r 73dad4940b88 Doc/library/sys.rst
--- a/Doc/library/sys.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/library/sys.rst	Mon Jan 30 12:29:09 2012 -0500
@@ -220,6 +220,7 @@
    :const:`ignore_environment`   :option:`-E`
    :const:`verbose`              :option:`-v`
    :const:`bytes_warning`        :option:`-b`
+   :const:`hash_randomization`   :option:`-R`
    ============================= =============================
 
 
diff -r 73dad4940b88 Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/reference/datamodel.rst	Mon Jan 30 12:29:09 2012 -0500
@@ -1265,6 +1265,8 @@
    inheritance of :meth:`__hash__` will be blocked, just as if :attr:`__hash__`
    had been explicitly set to :const:`None`.
 
+   See also the :option:`-R` command-line option.
+
 
 .. method:: object.__bool__(self)
 
diff -r 73dad4940b88 Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 20 11:23:02 2012 +0000
+++ b/Doc/using/cmdline.rst	Mon Jan 30 12:29:09 2012 -0500
@@ -21,7 +21,7 @@
 
 When invoking Python, you may specify any of these options::
 
-    python [-bBdEhiOsSuvVWx?] [-c command | -m module-name | script | - ] [args]
+    python [-bBdEhiORsSuvVWx?] [-c command | -m module-name | script | - ] [args]
 
 The most common use case is, of course, a simple invocation of a script::
 
@@ -215,6 +215,30 @@
    Discard docstrings in addition to the :option:`-O` optimizations.
 
 
+.. cmdoption:: -R
+
+   Turn on "hash randomization, so that the :meth:`__hash__` values of str,
+   bytes and datetime objects are "salted" with an unpredictable random value.
+   Although they remain constant within an individual Python process, they
+   are not predictable between repeated invocations of Python.
+
+   This is intended to provide protection against a denial-of-service
+   caused by carefully-chosen inputs that exploit the worst case performance
+   of a dict lookup, O(n^2) complexity.  See:
+
+       http://www.ocert.org/advisories/ocert-2011-003.html
+
+   for details.
+
+   Changing hash values affects the order in which keys are retrieved from
+   a dict.  Although Python has never made guarantees about this ordering
+   (and it typically varies between 32-bit and 64-bit builds), enough
+   real-world code implicitly relies on this non-guaranteed behavior that
+   the randomization is disabled by default.
+
+   See also :envvar:`PYTHONHASHRANDOMIZATION`.
+
+
 .. cmdoption:: -s
 
    Don't add user site directory to sys.path
@@ -435,6 +459,24 @@
    import of source modules.
 
 
+.. envvar:: PYTHONHASHRANDOMIZATION
+
+   If this is set to a non-empty string it is equivalent to specifying the
+   :option:`-R` option.
+
+
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for generating the hash() of
+   the types covered by the :option:`-R` option (or its equivalent,
+   :envvar:`PYTHONHASHRANDOMIZATION`.
+
+   Its purpose is for use in selftests for the interpreter.
+
+   It should be a decimal number in the range [0; 4294967295].  Specifying
+   the value 0 overrides the other setting, disabling the hash random salt.
+
+
 .. envvar:: PYTHONIOENCODING
 
    Overrides the encoding used for stdin/stdout/stderr, in the syntax
diff -r 73dad4940b88 Include/object.h
--- a/Include/object.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/object.h	Mon Jan 30 12:29:09 2012 -0500
@@ -473,6 +473,12 @@
 PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
+typedef struct {
+    long prefix;
+    long suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
+
 /* Helper for passing objects to printf and the like */
 #define PyObject_REPR(obj) _PyUnicode_AsString(PyObject_Repr(obj))
 
diff -r 73dad4940b88 Include/pydebug.h
--- a/Include/pydebug.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pydebug.h	Mon Jan 30 12:29:09 2012 -0500
@@ -19,6 +19,7 @@
 PyAPI_DATA(int) Py_DontWriteBytecodeFlag;
 PyAPI_DATA(int) Py_NoUserSiteDirectory;
 PyAPI_DATA(int) Py_UnbufferedStdioFlag;
+PyAPI_DATA(int) Py_HashRandomizationFlag;
 
 /* this is a wrapper around getenv() that pays attention to
    Py_IgnoreEnvironmentFlag.  It should be used for getting variables like
diff -r 73dad4940b88 Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 20 11:23:02 2012 +0000
+++ b/Include/pythonrun.h	Mon Jan 30 12:29:09 2012 -0500
@@ -174,6 +174,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r 73dad4940b88 Lib/os.py
--- a/Lib/os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/os.py	Mon Jan 30 12:29:09 2012 -0500
@@ -611,23 +611,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r 73dad4940b88 Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/regrtest.py	Mon Jan 30 12:29:09 2012 -0500
@@ -428,6 +428,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(tests)
diff -r 73dad4940b88 Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_cmd_line.py	Mon Jan 30 12:29:09 2012 -0500
@@ -190,6 +190,22 @@
             self.assertTrue(path1.encode('ascii') in stdout)
             self.assertTrue(path2.encode('ascii') in stdout)
 
+    def test_hash_randomization(self):
+        # Verify that -R enables hash randomization:
+        self.verify_valid_flag('-R')
+        hashes = []
+        for i in range(2):
+            code = 'print(hash("spam"))'
+            data, rc = self.start_python_and_exit_code('-R', '-c', code)
+            self.assertEqual(rc, 0)
+            hashes.append(data)
+        self.assertNotEqual(hashes[0], hashes[1])
+
+        # Verify that sys.flags contains hash_randomization
+        code = 'import sys; print(sys.flags)'
+        data, rc = self.start_python_and_exit_code('-R', '-c', code)
+        self.assertEqual(rc,0)
+        self.assertIn(b'hash_randomization=1', data)
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff -r 73dad4940b88 Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_hash.py	Mon Jan 30 12:29:09 2012 -0500
@@ -3,10 +3,15 @@
 #
 # Also test that hash implementations are inherited as expected
 
+import datetime
+import struct
 import unittest
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -118,10 +123,92 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+class HashRandomizationTests(unittest.TestCase):
+
+    # Each subclass should define a field "repr_", containing the repr() of
+    # an object to be tested
+
+    def get_hash_command(self, repr_):
+        return 'print(hash(%s))' % repr_
+
+    def get_hash(self, repr_, randomization=None, seed=None):
+        env = {}
+        if randomization is not None:
+            env['PYTHONHASHRANDOMIZATION'] = str(randomization)
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', self.get_hash_command(repr_),
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        run1 = self.get_hash(self.repr_, randomization=1)
+        run2 = self.get_hash(self.repr_, randomization=1)
+        self.assertNotEqual(run1, run2)
+
+class StringlikeHashRandomizationTests(HashRandomizationTests):
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            known_hash_of_obj = 1453079729188098211
+        else:
+            known_hash_of_obj = -1600925533
+
+        # Randomization is disabled by default:
+        self.assertEqual(self.get_hash(self.repr_), known_hash_of_obj)
+
+        # If enabled, it can still be disabled by setting the seed to 0:
+        self.assertEqual(self.get_hash(self.repr_, randomization=1, seed=0),
+                         known_hash_of_obj)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        # Note that all types share the same values:
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        self.assertEqual(self.get_hash(self.repr_, randomization=1, seed=42),
+                         h)
+
+class StrHashRandomizationTests(StringlikeHashRandomizationTests):
+    repr_ = repr('abc')
+
+    def test_empty_string(self):
+        self.assertEqual(hash(b""), 0)
+
+class BytesHashRandomizationTests(StringlikeHashRandomizationTests):
+    repr_ = repr('abc')
+
+    def test_empty_string(self):
+        self.assertEqual(hash(b""), 0)
+
+class DatetimeTests(HashRandomizationTests):
+    def get_hash_command(self, repr_):
+        return 'import datetime; print(hash(%s))' % repr_
+
+class DatetimeDateTests(DatetimeTests):
+    repr_ = repr(datetime.date.today())
+
+class DatetimeDatetimeTests(DatetimeTests):
+    repr_ = repr(datetime.datetime.now())
+
+class DatetimeTimeTests(DatetimeTests):
+    repr_ = repr(datetime.time(0))
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
                               HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                              HashBuiltinsTestCase,
+                         StrHashRandomizationTests,
+                         BytesHashRandomizationTests,
+                         DatetimeDateTests,
+                         DatetimeDatetimeTests,
+                         DatetimeTimeTests)
+
 
 
 if __name__ == "__main__":
diff -r 73dad4940b88 Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_os.py	Mon Jan 30 12:29:09 2012 -0500
@@ -9,6 +9,7 @@
 import sys
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 
 # Detect whether we're on a Linux system that uses the (now outdated
 # and unmaintained) linuxthreads threading library.  There's an issue
@@ -574,14 +575,33 @@
         f.close()
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(0)), 0)
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 class ExecTests(unittest.TestCase):
     @unittest.skipIf(USING_LINUXTHREADS,
diff -r 73dad4940b88 Lib/test/test_set.py
--- a/Lib/test/test_set.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_set.py	Mon Jan 30 12:29:09 2012 -0500
@@ -734,6 +734,17 @@
         if self.repr is not None:
             self.assertEqual(repr(self.set), self.repr)
 
+    def check_repr_against_values(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
     def test_print(self):
         try:
             fo = open(support.TESTFN, "w")
@@ -892,7 +903,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -903,7 +916,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{b'a', b'c', b'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +931,13 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
 
     def tearDown(self):
         warnings.filters = self.warning_filters
 
+    def test_repr(self):
+        self.check_repr_against_values()
+
 #==============================================================================
 
 def baditer():
diff -r 73dad4940b88 Lib/test/test_sys.py
--- a/Lib/test/test_sys.py	Fri Jan 20 11:23:02 2012 +0000
+++ b/Lib/test/test_sys.py	Mon Jan 30 12:29:09 2012 -0500
@@ -446,7 +446,7 @@
         attrs = ("debug", "division_warning",
                  "inspect", "interactive", "optimize", "dont_write_bytecode",
                  "no_user_site", "no_site", "ignore_environment", "verbose",
-                 "bytes_warning")
+                 "bytes_warning", "hash_randomization")
         for attr in attrs:
             self.assertTrue(hasattr(sys.flags, attr), attr)
             self.assertEqual(type(getattr(sys.flags, attr)), int, attr)
diff -r 73dad4940b88 Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 20 11:23:02 2012 +0000
+++ b/Makefile.pre.in	Mon Jan 30 12:29:09 2012 -0500
@@ -305,6 +305,7 @@
 		Python/pymath.o \
 		Python/pystate.o \
 		Python/pythonrun.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r 73dad4940b88 Misc/NEWS
--- a/Misc/NEWS	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/NEWS	Mon Jan 30 12:29:09 2012 -0500
@@ -10,6 +10,12 @@
 Core and Builtins
 -----------------
 
+- Issue #13703: add -R command-line option and PYTHONHASHRANDOMIZATION (and
+  PYTHONHASHSEED) environment variables, to provide an opt-in way to protect
+  against denial of service attacks due to hash collisions within the dict
+  and set types. See http://www.ocert.org/advisories/ocert-2011-003.html
+  Patch by David Malcolm, based on work by Victor Stinner.
+
 Library
 -------
 
diff -r 73dad4940b88 Misc/python.man
--- a/Misc/python.man	Fri Jan 20 11:23:02 2012 +0000
+++ b/Misc/python.man	Mon Jan 30 12:29:09 2012 -0500
@@ -34,6 +34,9 @@
 .B \-OO
 ]
 [
+.B \-R
+]
+[
 .B -Q
 .I argument
 ]
@@ -145,6 +148,18 @@
 .B \-OO
 Discard docstrings in addition to the \fB-O\fP optimizations.
 .TP
+.B \-R
+Turn on "hash randomization", so that the hash() values of str, bytes and
+datetime objects are "salted" with an unpredictable pseudo-random value.
+Although they remain constant within an individual Python process, they are
+not predictable between repeated invocations of Python.
+.IP
+This is intended to provide protection against a denial of service
+caused by carefully-chosen inputs that exploit the worst case performance
+of a dict lookup, O(n^2) complexity.  See
+http://www.ocert.org/advisories/ocert-2011-003.html
+for details.
+.TP
 .BI "\-Q " argument
 Division control; see PEP 238.  The argument must be one of "old" (the
 default, int/int and long/long return an int or long), "new" (new
@@ -403,6 +418,9 @@
 If this is set to a non-empty string it is equivalent to specifying
 the \fB\-v\fP option. If set to an integer, it is equivalent to
 specifying \fB\-v\fP multiple times. 
+.IP PYTHONHASHRANDOMIZATION
+If this is set to a non-empty string it is equivalent to specifying the
+\fB\-R\fP option.
 .SH AUTHOR
 The Python Software Foundation: http://www.python.org/psf
 .SH INTERNET RESOURCES
diff -r 73dad4940b88 Modules/datetimemodule.c
--- a/Modules/datetimemodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/datetimemodule.c	Mon Jan 30 12:29:09 2012 -0500
@@ -2566,10 +2566,12 @@
     register long x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= len;
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
 
diff -r 73dad4940b88 Modules/main.c
--- a/Modules/main.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/main.c	Mon Jan 30 12:29:09 2012 -0500
@@ -47,7 +47,7 @@
 static int  orig_argc;
 
 /* command line options */
-#define BASE_OPTS L"bBc:dEhiJm:OsStuvVW:xX?"
+#define BASE_OPTS L"bBc:dEhiJm:ORsStuvVW:xX?"
 
 #define PROGRAM_OPTS BASE_OPTS
 
@@ -72,6 +72,9 @@
 -m mod : run library module as a script (terminates option list)\n\
 -O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x\n\
 -OO    : remove doc-strings in addition to the -O optimizations\n\
+-R     : use a pseudo-random salt to make hash() values of various types be\n\
+         unpredictable between separate invocations of the interpreter, as\n\
+         a defence against denial-of-service attacks\n\
 -s     : don't add user site directory to sys.path; also PYTHONNOUSERSITE\n\
 -S     : don't imply 'import site' on initialization\n\
 ";
@@ -99,6 +102,11 @@
 PYTHONCASEOK : ignore case in 'import' statements (Windows).\n\
 PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n\
 ";
+static char *usage_6 = "\
+PYTHONHASHRANDOMIZATION : make hash() values of various types be unpredictable\n\
+                          between separate invocations of the interpreter, as\n\
+                          a defence against denial-of-service attacks\n\
+";
 
 #ifndef MS_WINDOWS
 static FILE*
@@ -136,6 +144,7 @@
         fputs(usage_3, f);
         fprintf(f, usage_4, DELIM);
         fprintf(f, usage_5, DELIM, PYTHONHOMEHELP);
+        fputs(usage_6, f);
     }
 #if defined(__VMS)
     if (exitcode == 0) {
@@ -373,6 +382,10 @@
             PySys_AddWarnOption(_PyOS_optarg);
             break;
 
+        case 'R':
+            Py_HashRandomizationFlag++;
+            break;
+
         /* This space reserved for other options */
 
         default:
diff -r 73dad4940b88 Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Modules/posixmodule.c	Mon Jan 30 12:29:09 2012 -0500
@@ -6942,82 +6942,6 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
-static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
-    }
-    return result;
-}
-#endif
-
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
 Return a string describing the encoding of the device\n\
@@ -7055,41 +6979,35 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+Return n pseudo-random bytes.");
+
+static PyObject *
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
+ 
+     /* Read arguments */
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -7374,12 +7292,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
     {NULL,              NULL}            /* Sentinel */
 };
 
diff -r 73dad4940b88 Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/bytesobject.c	Mon Jan 30 12:29:09 2012 -0500
@@ -899,11 +899,17 @@
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
+    if (len == 0) {
+        a->ob_shash = 0;
+        return 0;
+    }
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(a);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff -r 73dad4940b88 Objects/object.c
--- a/Objects/object.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/object.c	Mon Jan 30 12:29:09 2012 -0500
@@ -712,6 +712,8 @@
     return -1;
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
 long
 PyObject_Hash(PyObject *v)
 {
diff -r 73dad4940b88 Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Objects/unicodeobject.c	Mon Jan 30 12:29:09 2012 -0500
@@ -7344,11 +7344,17 @@
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
+    if (len == 0) {
+        self->hash = 0;
+        return 0;
+    }
     p = self->str;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(self);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     self->hash = x;
diff -r 73dad4940b88 PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 20 11:23:02 2012 +0000
+++ b/PCbuild/pythoncore.vcproj	Mon Jan 30 12:29:09 2012 -0500
@@ -1778,6 +1778,10 @@
 				RelativePath="..\Python\pythonrun.c"
 				>
 			</File>
+ 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
 			<File
 				RelativePath="..\Python\structmember.c"
 				>
diff -r 73dad4940b88 Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/pythonrun.c	Mon Jan 30 12:29:09 2012 -0500
@@ -73,6 +73,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 
 #ifdef WITH_THREAD
 extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@@ -91,6 +92,7 @@
 int Py_IgnoreEnvironmentFlag; /* e.g. PYTHONPATH, PYTHONHOME */
 int Py_NoUserSiteDirectory = 0; /* for -s and site.py */
 int Py_UnbufferedStdioFlag = 0; /* Unbuffered binary std{in,out,err} */
+int Py_HashRandomizationFlag = 0; /* for -R and PYTHONHASHRANDOMIZATION */
 
 /* PyModule_GetWarningsModule is no longer necessary as of 2.6
 since _warnings is builtin.  This API should not be used. */
@@ -195,6 +197,10 @@
         Py_OptimizeFlag = add_flag(Py_OptimizeFlag, p);
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
+    if ((p = Py_GETENV("PYTHONHASHRANDOMIZATION")) && *p != '\0')
+        Py_HashRandomizationFlag = add_flag(Py_HashRandomizationFlag, p);
+    
+    _PyRandom_Init();
 
     interp = PyInterpreterState_New();
     if (interp == NULL)
diff -r 73dad4940b88 Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Mon Jan 30 12:29:09 2012 -0500
@@ -0,0 +1,284 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Failed to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Failed to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Failed to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Failed to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Failed to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    /*
+      By default, hash randomization is disabled, and only
+      enabled if PYTHONHASHRANDOMIZATION is set to non-empty
+      or if "-R" is provided at the command line:
+    */
+    if (!Py_HashRandomizationFlag) {
+        /* Disable the randomized hash: */
+        memset(secret, 0, secret_size);
+        return;
+    }
+
+    /*
+      Hash randomization is enabled.  Generate a per-process secret,
+      using PYTHONHASHSEED if provided.
+    */
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else /* #ifdef MS_WINDOWS */
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+
diff -r 73dad4940b88 Python/sysmodule.c
--- a/Python/sysmodule.c	Fri Jan 20 11:23:02 2012 +0000
+++ b/Python/sysmodule.c	Mon Jan 30 12:29:09 2012 -0500
@@ -1126,6 +1126,7 @@
     /* {"unbuffered",                   "-u"}, */
     /* {"skip_first",                   "-x"}, */
     {"bytes_warning", "-b"},
+    {"hash_randomization", "-R"},
     {0}
 };
 
@@ -1134,9 +1135,9 @@
     flags__doc__,       /* doc */
     flags_fields,       /* fields */
 #ifdef RISCOS
+    13
+#else
     12
-#else
-    11
 #endif
 };
 
@@ -1169,6 +1170,7 @@
     /* SetFlag(saw_unbuffered_flag); */
     /* SetFlag(skipfirstline); */
     SetFlag(Py_BytesWarningFlag);
+    SetFlag(Py_HashRandomizationFlag);
 #undef SetFlag
 
     if (PyErr_Occurred()) {

From report at bugs.python.org  Mon Jan 30 18:35:26 2012
From: report at bugs.python.org (Eric Snow)
Date: Mon, 30 Jan 2012 17:35:26 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327944926.11.0.00686225819228.issue13912@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

what value do you see for __package__ in pkgA?

----------
nosy: +eric.snow

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:46:27 2012
From: report at bugs.python.org (Vikash Agrawal)
Date: Mon, 30 Jan 2012 17:46:27 +0000
Subject: [issue13851] Packaging distutils2 for Fedora
In-Reply-To: <1327409259.69.0.346795065424.issue13851@psf.upfronthosting.co.za>
Message-ID: <1327945587.45.0.14163833989.issue13851@psf.upfronthosting.co.za>


Vikash Agrawal <vikashagrawal1990 at gmail.com> added the comment:

hi merwok, 
I didn't know this is the right place for that bug/query. I am still in the process of learning packaging, and at this moment I wont be able to tell that if its a problem of distutils2 or not, though I do feel that pysetup should have installed directly, considering my spec file is correct.

So as o now, the bug is closed, and later I will come with concrete results.

I will be available via private email and I will discuss this issue with dmalcolm over IRC

----------
status: pending -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13851>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:49:45 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 30 Jan 2012 17:49:45 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <E1RrvMN-0008Qi-Es@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 54d7823ec488 by Brett Cannon in branch 'default':
Issue #13890: Fix importlib case-sensitivity tests to not run on Windows.
http://hg.python.org/cpython/rev/54d7823ec488

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:51:58 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 30 Jan 2012 17:51:58 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <E1RrvOW-0000Kn-Hd@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 2914ce82bf89 by Brett Cannon in branch 'default':
Issue #13890: Also fix for extension module tests for case-insensitivity.
http://hg.python.org/cpython/rev/2914ce82bf89

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:52:15 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 30 Jan 2012 17:52:15 +0000
Subject: [issue13890] test_importlib failures under Windows
In-Reply-To: <1327683057.17.0.810855079387.issue13890@psf.upfronthosting.co.za>
Message-ID: <1327945935.92.0.534052991838.issue13890@psf.upfronthosting.co.za>


Changes by Brett Cannon <brett at python.org>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13890>
_______________________________________

From report at bugs.python.org  Mon Jan 30 18:52:18 2012
From: report at bugs.python.org (Bert JW Regeer)
Date: Mon, 30 Jan 2012 17:52:18 +0000
Subject: [issue10910] pyport.h FreeBSD/Mac OS X "fix" causes errors in C++
	compilation
In-Reply-To: <1295040100.28.0.0907494738582.issue10910@psf.upfronthosting.co.za>
Message-ID: <1327945938.08.0.483661200437.issue10910@psf.upfronthosting.co.za>


Bert JW Regeer <bertjw at regeer.org> added the comment:

In my first comment on this bug post I posted what project has issues with this, Botan with Boost.Python on FreeBSD and Mac OS X.

If required I will post how to reproduce this error using that project. If you would prefer a simplified test case, unfortunately I don't have the time to dig into that and find one.

One way to solve it is by including the Python header before any other file, however this is not an option for most 3rd party software that never sees this bug because those functions are not redefined for Linux...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10910>
_______________________________________

From report at bugs.python.org  Mon Jan 30 19:05:07 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 30 Jan 2012 18:05:07 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327946707.53.0.510476407054.issue13912@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

You have a typo in a filename: it should be pkgB/__init__.py (not the missing trailing underscores).

----------
nosy: +brett.cannon
resolution:  -> invalid
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Mon Jan 30 20:31:06 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 30 Jan 2012 19:31:06 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327951866.42.0.131744955264.issue13912@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

Sorry for the mistake.

I corrected the pkgB package, but the result is the same:

    jaraco at devjaraco:~$ tree master
    master
    ??? __init__.py
    ??? __init__.pyc
    ??? pkgA
    ??? ??? foo.py
    ??? ??? foo.pyc
    ??? ??? __init__.py
    ??? ??? __init__.pyc
    ??? pkgB
        ??? bar.py
        ??? __init__.py

    2 directories, 8 files
    jaraco at devjaraco:~$ python2.7 -c 'import master.pkgA; print("pkgA.__package__ is {}".format(master.pkgA.__package__)); import master.pkgA.foo'
    pkgA.__package__ is None
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "master/pkgA/foo.py", line 2, in <module>
        bar = __import__('pkgB', level=1).bar
    ImportError: No module named pkgB

As you can see, __package__ for pkgA is None. Same is true for master.__package__.

----------
resolution: invalid -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Mon Jan 30 20:44:33 2012
From: report at bugs.python.org (Justin Wehnes)
Date: Mon, 30 Jan 2012 19:44:33 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327952673.16.0.106737864358.issue13857@psf.upfronthosting.co.za>


Justin Wehnes <jwehnes at gmail.com> added the comment:

Just wondering if someone is already working on this or am I free to supply a patch?

----------
nosy: +jwehnes

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Mon Jan 30 20:55:53 2012
From: report at bugs.python.org (Jim Jewett)
Date: Mon, 30 Jan 2012 19:55:53 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327944638.2444.16.camel@surprise>
Message-ID: <CA+OGgf4mEn=EL=w9V1ojcUKyYUq7FOMwsSSTshB57WX2D241WA@mail.gmail.com>


Jim Jewett <jimjjewett at gmail.com> added the comment:

On Mon, Jan 30, 2012 at 12:31 PM,  Dave Malcolm <dmalcolm at redhat.com>
added the comment:

> It's useful for the selftests, so I've kept PYTHONHASHSEED.

The reason to read PYTHONHASHSEED was so that multiple members of a
cluster could use the same hash.

It would have been nice to have fewer environment variables, but I'll
grant that it is hard to say "use something random that we have *not*
precomputed" without either a config file or a magic value for
PYTHONHASHSEED.

-jJ

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Mon Jan 30 21:22:14 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Mon, 30 Jan 2012 20:22:14 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327954934.34.0.541798317479.issue13903@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

This would likely require a PEP before having a chance of being considered for inclusion.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:03:03 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Mon, 30 Jan 2012 21:03:03 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327957383.54.0.360942395872.issue13857@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Please go ahead!

And Georg is right - the short spelling doesn't handle the first line correctly. It also suffers from the "trailing whitespace" problem that Amaury pointed out in my original version.

The tests for the new function should check both those cases (i.e. don't indent blank lines, ensure the first line is correctly indented).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:09:17 2012
From: report at bugs.python.org (Georg Brandl)
Date: Mon, 30 Jan 2012 21:09:17 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1327957757.9.0.307948324.issue13868@psf.upfronthosting.co.za>


Georg Brandl <georg at python.org> added the comment:

Wow, does that mean he can cash in a Knuth check?

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:11:22 2012
From: report at bugs.python.org (Roundup Robot)
Date: Mon, 30 Jan 2012 21:11:22 +0000
Subject: [issue8828] Atomic function to rename a file
In-Reply-To: <1274917301.93.0.503310941815.issue8828@psf.upfronthosting.co.za>
Message-ID: <E1RryVU-0005yI-1o@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 80ddbd822227 by Antoine Pitrou in branch 'default':
Issue #8828: Add new function os.replace(), for cross-platform renaming with overwriting.
http://hg.python.org/cpython/rev/80ddbd822227

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8828>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:13:08 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Mon, 30 Jan 2012 21:13:08 +0000
Subject: [issue8828] Atomic function to rename a file
In-Reply-To: <1274917301.93.0.503310941815.issue8828@psf.upfronthosting.co.za>
Message-ID: <1327957988.37.0.702470541494.issue8828@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

os.replace() committed in 3.3!

----------
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8828>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:14:46 2012
From: report at bugs.python.org (=?utf-8?q?Martin_v=2E_L=C3=B6wis?=)
Date: Mon, 30 Jan 2012 21:14:46 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <CAMpsgwZvBkzAqNsBJLextiT2kPdZZ-fsFB1GrziEA7kQ_pX72g@mail.gmail.com>
Message-ID: <4F270844.50605@v.loewis.de>


Martin v. L?wis <martin at v.loewis.de> added the comment:

> I know that the underlying C function expects a timespec structure,
> but Python can try to use a higher level API, isn't it?

I agree entirely.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 22:26:49 2012
From: report at bugs.python.org (Mark Shannon)
Date: Mon, 30 Jan 2012 21:26:49 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327958809.7.0.175799536942.issue13903@psf.upfronthosting.co.za>


Mark Shannon <mark at hotpy.org> added the comment:

Does this really need a PEP?
There is no new language feature and no change to any API.
It is just saving some memory.

The only possible issue is changing dict repr() ordering, 
but issue 13703 will do that anyway.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:22:46 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Mon, 30 Jan 2012 22:22:46 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1327944638.2444.16.camel@surprise>
Message-ID: <1327962128.2893.7.camel@surprise>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

I slightly messed up the test_hash.py changes.

Revised patch attached:
  optin-hash-randomization-for-3.1-dmalcolm-2012-01-30-002.patch

----------
Added file: http://bugs.python.org/file24371/optin-hash-randomization-for-3.1-dmalcolm-2012-01-30-002.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________
-------------- next part --------------
diff -r e7706bdaaa0d Doc/library/sys.rst
--- a/Doc/library/sys.rst	Fri Jan 27 09:48:47 2012 +0100
+++ b/Doc/library/sys.rst	Mon Jan 30 17:21:17 2012 -0500
@@ -220,6 +220,7 @@
    :const:`ignore_environment`   :option:`-E`
    :const:`verbose`              :option:`-v`
    :const:`bytes_warning`        :option:`-b`
+   :const:`hash_randomization`   :option:`-R`
    ============================= =============================
 
 
diff -r e7706bdaaa0d Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst	Fri Jan 27 09:48:47 2012 +0100
+++ b/Doc/reference/datamodel.rst	Mon Jan 30 17:21:17 2012 -0500
@@ -1265,6 +1265,8 @@
    inheritance of :meth:`__hash__` will be blocked, just as if :attr:`__hash__`
    had been explicitly set to :const:`None`.
 
+   See also the :option:`-R` command-line option.
+
 
 .. method:: object.__bool__(self)
 
diff -r e7706bdaaa0d Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst	Fri Jan 27 09:48:47 2012 +0100
+++ b/Doc/using/cmdline.rst	Mon Jan 30 17:21:17 2012 -0500
@@ -21,7 +21,7 @@
 
 When invoking Python, you may specify any of these options::
 
-    python [-bBdEhiOsSuvVWx?] [-c command | -m module-name | script | - ] [args]
+    python [-bBdEhiORsSuvVWx?] [-c command | -m module-name | script | - ] [args]
 
 The most common use case is, of course, a simple invocation of a script::
 
@@ -215,6 +215,30 @@
    Discard docstrings in addition to the :option:`-O` optimizations.
 
 
+.. cmdoption:: -R
+
+   Turn on "hash randomization, so that the :meth:`__hash__` values of str,
+   bytes and datetime objects are "salted" with an unpredictable random value.
+   Although they remain constant within an individual Python process, they
+   are not predictable between repeated invocations of Python.
+
+   This is intended to provide protection against a denial-of-service
+   caused by carefully-chosen inputs that exploit the worst case performance
+   of a dict lookup, O(n^2) complexity.  See:
+
+       http://www.ocert.org/advisories/ocert-2011-003.html
+
+   for details.
+
+   Changing hash values affects the order in which keys are retrieved from
+   a dict.  Although Python has never made guarantees about this ordering
+   (and it typically varies between 32-bit and 64-bit builds), enough
+   real-world code implicitly relies on this non-guaranteed behavior that
+   the randomization is disabled by default.
+
+   See also :envvar:`PYTHONHASHRANDOMIZATION`.
+
+
 .. cmdoption:: -s
 
    Don't add user site directory to sys.path
@@ -435,6 +459,24 @@
    import of source modules.
 
 
+.. envvar:: PYTHONHASHRANDOMIZATION
+
+   If this is set to a non-empty string it is equivalent to specifying the
+   :option:`-R` option.
+
+
+.. envvar:: PYTHONHASHSEED
+
+   If this is set, it is used as a fixed seed for generating the hash() of
+   the types covered by the :option:`-R` option (or its equivalent,
+   :envvar:`PYTHONHASHRANDOMIZATION`.
+
+   Its purpose is for use in selftests for the interpreter.
+
+   It should be a decimal number in the range [0; 4294967295].  Specifying
+   the value 0 overrides the other setting, disabling the hash random salt.
+
+
 .. envvar:: PYTHONIOENCODING
 
    Overrides the encoding used for stdin/stdout/stderr, in the syntax
diff -r e7706bdaaa0d Include/object.h
--- a/Include/object.h	Fri Jan 27 09:48:47 2012 +0100
+++ b/Include/object.h	Mon Jan 30 17:21:17 2012 -0500
@@ -473,6 +473,12 @@
 PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
+typedef struct {
+    long prefix;
+    long suffix;
+} _Py_HashSecret_t;
+PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;
+
 /* Helper for passing objects to printf and the like */
 #define PyObject_REPR(obj) _PyUnicode_AsString(PyObject_Repr(obj))
 
diff -r e7706bdaaa0d Include/pydebug.h
--- a/Include/pydebug.h	Fri Jan 27 09:48:47 2012 +0100
+++ b/Include/pydebug.h	Mon Jan 30 17:21:17 2012 -0500
@@ -19,6 +19,7 @@
 PyAPI_DATA(int) Py_DontWriteBytecodeFlag;
 PyAPI_DATA(int) Py_NoUserSiteDirectory;
 PyAPI_DATA(int) Py_UnbufferedStdioFlag;
+PyAPI_DATA(int) Py_HashRandomizationFlag;
 
 /* this is a wrapper around getenv() that pays attention to
    Py_IgnoreEnvironmentFlag.  It should be used for getting variables like
diff -r e7706bdaaa0d Include/pythonrun.h
--- a/Include/pythonrun.h	Fri Jan 27 09:48:47 2012 +0100
+++ b/Include/pythonrun.h	Mon Jan 30 17:21:17 2012 -0500
@@ -174,6 +174,8 @@
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_getsig(int);
 PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t);
 
+/* Random */
+PyAPI_FUNC(int) _PyOS_URandom (void *buffer, Py_ssize_t size);
 
 #ifdef __cplusplus
 }
diff -r e7706bdaaa0d Lib/os.py
--- a/Lib/os.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/os.py	Mon Jan 30 17:21:17 2012 -0500
@@ -611,23 +611,6 @@
 except NameError: # statvfs_result may not exist
     pass
 
-if not _exists("urandom"):
-    def urandom(n):
-        """urandom(n) -> str
-
-        Return a string of n random bytes suitable for cryptographic use.
-
-        """
-        try:
-            _urandomfd = open("/dev/urandom", O_RDONLY)
-        except (OSError, IOError):
-            raise NotImplementedError("/dev/urandom (or equivalent) not found")
-        bs = b""
-        while len(bs) < n:
-            bs += read(_urandomfd, n - len(bs))
-        close(_urandomfd)
-        return bs
-
 # Supply os.popen()
 def popen(cmd, mode="r", buffering=-1):
     if not isinstance(cmd, str):
diff -r e7706bdaaa0d Lib/test/regrtest.py
--- a/Lib/test/regrtest.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/regrtest.py	Mon Jan 30 17:21:17 2012 -0500
@@ -428,6 +428,11 @@
         except ValueError:
             print("Couldn't find starting test (%s), using all tests" % start)
     if randomize:
+        hashseed = os.getenv('PYTHONHASHSEED')
+        if not hashseed:
+            os.environ['PYTHONHASHSEED'] = str(random_seed)
+            os.execv(sys.executable, [sys.executable] + sys.argv)
+            return
         random.seed(random_seed)
         print("Using random seed", random_seed)
         random.shuffle(tests)
diff -r e7706bdaaa0d Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/test_cmd_line.py	Mon Jan 30 17:21:17 2012 -0500
@@ -190,6 +190,22 @@
             self.assertTrue(path1.encode('ascii') in stdout)
             self.assertTrue(path2.encode('ascii') in stdout)
 
+    def test_hash_randomization(self):
+        # Verify that -R enables hash randomization:
+        self.verify_valid_flag('-R')
+        hashes = []
+        for i in range(2):
+            code = 'print(hash("spam"))'
+            data, rc = self.start_python_and_exit_code('-R', '-c', code)
+            self.assertEqual(rc, 0)
+            hashes.append(data)
+        self.assertNotEqual(hashes[0], hashes[1])
+
+        # Verify that sys.flags contains hash_randomization
+        code = 'import sys; print(sys.flags)'
+        data, rc = self.start_python_and_exit_code('-R', '-c', code)
+        self.assertEqual(rc,0)
+        self.assertIn(b'hash_randomization=1', data)
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff -r e7706bdaaa0d Lib/test/test_hash.py
--- a/Lib/test/test_hash.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/test_hash.py	Mon Jan 30 17:21:17 2012 -0500
@@ -3,10 +3,15 @@
 #
 # Also test that hash implementations are inherited as expected
 
+import datetime
+import struct
 import unittest
 from test import support
+from test.script_helper import assert_python_ok
 from collections import Hashable
 
+IS_64BIT = (struct.calcsize('l') == 8)
+
 
 class HashEqualityTestCase(unittest.TestCase):
 
@@ -118,10 +123,92 @@
         for obj in self.hashes_to_check:
             self.assertEqual(hash(obj), _default_hash(obj))
 
+class HashRandomizationTests(unittest.TestCase):
+
+    # Each subclass should define a field "repr_", containing the repr() of
+    # an object to be tested
+
+    def get_hash_command(self, repr_):
+        return 'print(hash(%s))' % repr_
+
+    def get_hash(self, repr_, randomization=None, seed=None):
+        env = {}
+        if randomization is not None:
+            env['PYTHONHASHRANDOMIZATION'] = str(randomization)
+        if seed is not None:
+            env['PYTHONHASHSEED'] = str(seed)
+        out = assert_python_ok(
+            '-c', self.get_hash_command(repr_),
+            **env)
+        stdout = out[1].strip()
+        return int(stdout)
+
+    def test_randomized_hash(self):
+        # two runs should return different hashes
+        run1 = self.get_hash(self.repr_, randomization=1)
+        run2 = self.get_hash(self.repr_, randomization=1)
+        self.assertNotEqual(run1, run2)
+
+class StringlikeHashRandomizationTests(HashRandomizationTests):
+    def test_null_hash(self):
+        # PYTHONHASHSEED=0 disables the randomized hash
+        if IS_64BIT:
+            known_hash_of_obj = 1453079729188098211
+        else:
+            known_hash_of_obj = -1600925533
+
+        # Randomization is disabled by default:
+        self.assertEqual(self.get_hash(self.repr_), known_hash_of_obj)
+
+        # If enabled, it can still be disabled by setting the seed to 0:
+        self.assertEqual(self.get_hash(self.repr_, randomization=1, seed=0),
+                         known_hash_of_obj)
+
+    def test_fixed_hash(self):
+        # test a fixed seed for the randomized hash
+        # Note that all types share the same values:
+        if IS_64BIT:
+            h = -4410911502303878509
+        else:
+            h = -206076799
+        self.assertEqual(self.get_hash(self.repr_, randomization=1, seed=42),
+                         h)
+
+class StrHashRandomizationTests(StringlikeHashRandomizationTests):
+    repr_ = repr('abc')
+
+    def test_empty_string(self):
+        self.assertEqual(hash(""), 0)
+
+class BytesHashRandomizationTests(StringlikeHashRandomizationTests):
+    repr_ = repr(b'abc')
+
+    def test_empty_string(self):
+        self.assertEqual(hash(b""), 0)
+
+class DatetimeTests(HashRandomizationTests):
+    def get_hash_command(self, repr_):
+        return 'import datetime; print(hash(%s))' % repr_
+
+class DatetimeDateTests(DatetimeTests):
+    repr_ = repr(datetime.date(1066, 10, 14))
+
+class DatetimeDatetimeTests(DatetimeTests):
+    repr_ = repr(datetime.datetime(1, 2, 3, 4, 5, 6, 7))
+
+class DatetimeTimeTests(DatetimeTests):
+    repr_ = repr(datetime.time(0))
+
 def test_main():
     support.run_unittest(HashEqualityTestCase,
                               HashInheritanceTestCase,
-                              HashBuiltinsTestCase)
+                              HashBuiltinsTestCase,
+                         StrHashRandomizationTests,
+                         BytesHashRandomizationTests,
+                         DatetimeDateTests,
+                         DatetimeDatetimeTests,
+                         DatetimeTimeTests)
+
 
 
 if __name__ == "__main__":
diff -r e7706bdaaa0d Lib/test/test_os.py
--- a/Lib/test/test_os.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/test_os.py	Mon Jan 30 17:21:17 2012 -0500
@@ -9,6 +9,7 @@
 import sys
 import shutil
 from test import support
+from test.script_helper import assert_python_ok
 
 # Detect whether we're on a Linux system that uses the (now outdated
 # and unmaintained) linuxthreads threading library.  There's an issue
@@ -574,14 +575,33 @@
         f.close()
 
 class URandomTests(unittest.TestCase):
-    def test_urandom(self):
-        try:
-            self.assertEqual(len(os.urandom(1)), 1)
-            self.assertEqual(len(os.urandom(10)), 10)
-            self.assertEqual(len(os.urandom(100)), 100)
-            self.assertEqual(len(os.urandom(1000)), 1000)
-        except NotImplementedError:
-            pass
+    def test_urandom_length(self):
+        self.assertEqual(len(os.urandom(0)), 0)
+        self.assertEqual(len(os.urandom(1)), 1)
+        self.assertEqual(len(os.urandom(10)), 10)
+        self.assertEqual(len(os.urandom(100)), 100)
+        self.assertEqual(len(os.urandom(1000)), 1000)
+
+    def test_urandom_value(self):
+        data1 = os.urandom(16)
+        data2 = os.urandom(16)
+        self.assertNotEqual(data1, data2)
+
+    def get_urandom_subprocess(self, count):
+        code = '\n'.join((
+            'import os, sys',
+            'data = os.urandom(%s)' % count,
+            'sys.stdout.buffer.write(data)',
+            'sys.stdout.buffer.flush()'))
+        out = assert_python_ok('-c', code)
+        stdout = out[1]
+        self.assertEqual(len(stdout), 16)
+        return stdout
+
+    def test_urandom_subprocess(self):
+        data1 = self.get_urandom_subprocess(16)
+        data2 = self.get_urandom_subprocess(16)
+        self.assertNotEqual(data1, data2)
 
 class ExecTests(unittest.TestCase):
     @unittest.skipIf(USING_LINUXTHREADS,
diff -r e7706bdaaa0d Lib/test/test_set.py
--- a/Lib/test/test_set.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/test_set.py	Mon Jan 30 17:21:17 2012 -0500
@@ -734,6 +734,17 @@
         if self.repr is not None:
             self.assertEqual(repr(self.set), self.repr)
 
+    def check_repr_against_values(self):
+        text = repr(self.set)
+        self.assertTrue(text.startswith('{'))
+        self.assertTrue(text.endswith('}'))
+
+        result = text[1:-1].split(', ')
+        result.sort()
+        sorted_repr_values = [repr(value) for value in self.values]
+        sorted_repr_values.sort()
+        self.assertEqual(result, sorted_repr_values)
+
     def test_print(self):
         try:
             fo = open(support.TESTFN, "w")
@@ -892,7 +903,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{'a', 'c', 'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -903,7 +916,9 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 3
-        self.repr   = "{b'a', b'c', b'b'}"
+
+    def test_repr(self):
+        self.check_repr_against_values()
 
 #------------------------------------------------------------------------------
 
@@ -916,11 +931,13 @@
         self.set    = set(self.values)
         self.dup    = set(self.values)
         self.length = 4
-        self.repr   = "{'a', b'a', 'b', b'b'}"
 
     def tearDown(self):
         warnings.filters = self.warning_filters
 
+    def test_repr(self):
+        self.check_repr_against_values()
+
 #==============================================================================
 
 def baditer():
diff -r e7706bdaaa0d Lib/test/test_sys.py
--- a/Lib/test/test_sys.py	Fri Jan 27 09:48:47 2012 +0100
+++ b/Lib/test/test_sys.py	Mon Jan 30 17:21:17 2012 -0500
@@ -446,7 +446,7 @@
         attrs = ("debug", "division_warning",
                  "inspect", "interactive", "optimize", "dont_write_bytecode",
                  "no_user_site", "no_site", "ignore_environment", "verbose",
-                 "bytes_warning")
+                 "bytes_warning", "hash_randomization")
         for attr in attrs:
             self.assertTrue(hasattr(sys.flags, attr), attr)
             self.assertEqual(type(getattr(sys.flags, attr)), int, attr)
diff -r e7706bdaaa0d Makefile.pre.in
--- a/Makefile.pre.in	Fri Jan 27 09:48:47 2012 +0100
+++ b/Makefile.pre.in	Mon Jan 30 17:21:17 2012 -0500
@@ -305,6 +305,7 @@
 		Python/pymath.o \
 		Python/pystate.o \
 		Python/pythonrun.o \
+		Python/random.o \
 		Python/structmember.o \
 		Python/symtable.o \
 		Python/sysmodule.o \
diff -r e7706bdaaa0d Misc/NEWS
--- a/Misc/NEWS	Fri Jan 27 09:48:47 2012 +0100
+++ b/Misc/NEWS	Mon Jan 30 17:21:17 2012 -0500
@@ -10,6 +10,12 @@
 Core and Builtins
 -----------------
 
+- Issue #13703: add -R command-line option and PYTHONHASHRANDOMIZATION (and
+  PYTHONHASHSEED) environment variables, to provide an opt-in way to protect
+  against denial of service attacks due to hash collisions within the dict
+  and set types. See http://www.ocert.org/advisories/ocert-2011-003.html
+  Patch by David Malcolm, based on work by Victor Stinner.
+
 Library
 -------
 
diff -r e7706bdaaa0d Misc/python.man
--- a/Misc/python.man	Fri Jan 27 09:48:47 2012 +0100
+++ b/Misc/python.man	Mon Jan 30 17:21:17 2012 -0500
@@ -34,6 +34,9 @@
 .B \-OO
 ]
 [
+.B \-R
+]
+[
 .B -Q
 .I argument
 ]
@@ -145,6 +148,18 @@
 .B \-OO
 Discard docstrings in addition to the \fB-O\fP optimizations.
 .TP
+.B \-R
+Turn on "hash randomization", so that the hash() values of str, bytes and
+datetime objects are "salted" with an unpredictable pseudo-random value.
+Although they remain constant within an individual Python process, they are
+not predictable between repeated invocations of Python.
+.IP
+This is intended to provide protection against a denial of service
+caused by carefully-chosen inputs that exploit the worst case performance
+of a dict lookup, O(n^2) complexity.  See
+http://www.ocert.org/advisories/ocert-2011-003.html
+for details.
+.TP
 .BI "\-Q " argument
 Division control; see PEP 238.  The argument must be one of "old" (the
 default, int/int and long/long return an int or long), "new" (new
@@ -403,6 +418,9 @@
 If this is set to a non-empty string it is equivalent to specifying
 the \fB\-v\fP option. If set to an integer, it is equivalent to
 specifying \fB\-v\fP multiple times. 
+.IP PYTHONHASHRANDOMIZATION
+If this is set to a non-empty string it is equivalent to specifying the
+\fB\-R\fP option.
 .SH AUTHOR
 The Python Software Foundation: http://www.python.org/psf
 .SH INTERNET RESOURCES
diff -r e7706bdaaa0d Modules/datetimemodule.c
--- a/Modules/datetimemodule.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Modules/datetimemodule.c	Mon Jan 30 17:21:17 2012 -0500
@@ -2566,10 +2566,12 @@
     register long x;
 
     p = (unsigned char *) data;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= len;
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
 
diff -r e7706bdaaa0d Modules/main.c
--- a/Modules/main.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Modules/main.c	Mon Jan 30 17:21:17 2012 -0500
@@ -47,7 +47,7 @@
 static int  orig_argc;
 
 /* command line options */
-#define BASE_OPTS L"bBc:dEhiJm:OsStuvVW:xX?"
+#define BASE_OPTS L"bBc:dEhiJm:ORsStuvVW:xX?"
 
 #define PROGRAM_OPTS BASE_OPTS
 
@@ -72,6 +72,9 @@
 -m mod : run library module as a script (terminates option list)\n\
 -O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x\n\
 -OO    : remove doc-strings in addition to the -O optimizations\n\
+-R     : use a pseudo-random salt to make hash() values of various types be\n\
+         unpredictable between separate invocations of the interpreter, as\n\
+         a defence against denial-of-service attacks\n\
 -s     : don't add user site directory to sys.path; also PYTHONNOUSERSITE\n\
 -S     : don't imply 'import site' on initialization\n\
 ";
@@ -99,6 +102,11 @@
 PYTHONCASEOK : ignore case in 'import' statements (Windows).\n\
 PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n\
 ";
+static char *usage_6 = "\
+PYTHONHASHRANDOMIZATION : make hash() values of various types be unpredictable\n\
+                          between separate invocations of the interpreter, as\n\
+                          a defence against denial-of-service attacks\n\
+";
 
 #ifndef MS_WINDOWS
 static FILE*
@@ -136,6 +144,7 @@
         fputs(usage_3, f);
         fprintf(f, usage_4, DELIM);
         fprintf(f, usage_5, DELIM, PYTHONHOMEHELP);
+        fputs(usage_6, f);
     }
 #if defined(__VMS)
     if (exitcode == 0) {
@@ -373,6 +382,10 @@
             PySys_AddWarnOption(_PyOS_optarg);
             break;
 
+        case 'R':
+            Py_HashRandomizationFlag++;
+            break;
+
         /* This space reserved for other options */
 
         default:
diff -r e7706bdaaa0d Modules/posixmodule.c
--- a/Modules/posixmodule.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Modules/posixmodule.c	Mon Jan 30 17:21:17 2012 -0500
@@ -6942,82 +6942,6 @@
 }
 #endif
 
-#ifdef MS_WINDOWS
-
-PyDoc_STRVAR(win32_urandom__doc__,
-"urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
-              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
-              DWORD dwFlags );
-typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
-              BYTE *pbBuffer );
-
-static CRYPTGENRANDOM pCryptGenRandom = NULL;
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
-static HCRYPTPROV hCryptProv = 0;
-
-static PyObject*
-win32_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
-        return PyErr_Format(PyExc_ValueError,
-                            "negative argument not allowed");
-
-    if (hCryptProv == 0) {
-        HINSTANCE hAdvAPI32 = NULL;
-        CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
-
-        /* Obtain handle to the DLL containing CryptoAPI
-           This should not fail         */
-        hAdvAPI32 = GetModuleHandle("advapi32.dll");
-        if(hAdvAPI32 == NULL)
-            return win32_error("GetModuleHandle", NULL);
-
-        /* Obtain pointers to the CryptoAPI functions
-           This will fail on some early versions of Win95 */
-        pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
-                                        hAdvAPI32,
-                                        "CryptAcquireContextA");
-        if (pCryptAcquireContext == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptAcquireContextA not found");
-
-        pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(
-                                        hAdvAPI32, "CryptGenRandom");
-        if (pCryptGenRandom == NULL)
-            return PyErr_Format(PyExc_NotImplementedError,
-                                "CryptGenRandom not found");
-
-        /* Acquire context */
-        if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
-                                   PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
-            return win32_error("CryptAcquireContext", NULL);
-    }
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        memset(PyBytes_AS_STRING(result), 0, howMany); /* zero seed */
-        if (! pCryptGenRandom(hCryptProv, howMany, (unsigned char*)
-                              PyBytes_AS_STRING(result))) {
-            Py_DECREF(result);
-            return win32_error("CryptGenRandom", NULL);
-        }
-    }
-    return result;
-}
-#endif
-
 PyDoc_STRVAR(device_encoding__doc__,
 "device_encoding(fd) -> str\n\n\
 Return a string describing the encoding of the device\n\
@@ -7055,41 +6979,35 @@
     return Py_None;
 }
 
-#ifdef __VMS
-/* Use openssl random routine */
-#include <openssl/rand.h>
-PyDoc_STRVAR(vms_urandom__doc__,
+PyDoc_STRVAR(posix_urandom__doc__,
 "urandom(n) -> str\n\n\
-Return n random bytes suitable for cryptographic use.");
-
-static PyObject*
-vms_urandom(PyObject *self, PyObject *args)
-{
-    int howMany;
-    PyObject* result;
-
-    /* Read arguments */
-    if (! PyArg_ParseTuple(args, "i:urandom", &howMany))
-        return NULL;
-    if (howMany < 0)
+Return n pseudo-random bytes.");
+
+static PyObject *
+posix_urandom(PyObject *self, PyObject *args)
+{
+    Py_ssize_t size;
+    PyObject *result;
+    int ret;
+ 
+     /* Read arguments */
+    if (!PyArg_ParseTuple(args, "n:urandom", &size))
+        return NULL;
+    if (size < 0)
         return PyErr_Format(PyExc_ValueError,
                             "negative argument not allowed");
-
-    /* Allocate bytes */
-    result = PyBytes_FromStringAndSize(NULL, howMany);
-    if (result != NULL) {
-        /* Get random data */
-        if (RAND_pseudo_bytes((unsigned char*)
-                              PyBytes_AS_STRING(result),
-                              howMany) < 0) {
-            Py_DECREF(result);
-            return PyErr_Format(PyExc_ValueError,
-                                "RAND_pseudo_bytes");
-        }
+    result = PyBytes_FromStringAndSize(NULL, size);
+    if (result == NULL)
+        return NULL;
+
+    ret = _PyOS_URandom(PyBytes_AS_STRING(result),
+                        PyBytes_GET_SIZE(result));
+    if (ret == -1) {
+        Py_DECREF(result);
+        return NULL;
     }
     return result;
 }
-#endif
 
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
@@ -7374,12 +7292,7 @@
 #ifdef HAVE_GETLOADAVG
     {"getloadavg",      posix_getloadavg, METH_NOARGS, posix_getloadavg__doc__},
 #endif
- #ifdef MS_WINDOWS
-    {"urandom", win32_urandom, METH_VARARGS, win32_urandom__doc__},
- #endif
- #ifdef __VMS
-    {"urandom", vms_urandom, METH_VARARGS, vms_urandom__doc__},
- #endif
+    {"urandom",         posix_urandom,   METH_VARARGS, posix_urandom__doc__},
     {NULL,              NULL}            /* Sentinel */
 };
 
diff -r e7706bdaaa0d Objects/bytesobject.c
--- a/Objects/bytesobject.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Objects/bytesobject.c	Mon Jan 30 17:21:17 2012 -0500
@@ -899,11 +899,17 @@
     if (a->ob_shash != -1)
         return a->ob_shash;
     len = Py_SIZE(a);
+    if (len == 0) {
+        a->ob_shash = 0;
+        return 0;
+    }
     p = (unsigned char *) a->ob_sval;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(a);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     a->ob_shash = x;
diff -r e7706bdaaa0d Objects/object.c
--- a/Objects/object.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Objects/object.c	Mon Jan 30 17:21:17 2012 -0500
@@ -712,6 +712,8 @@
     return -1;
 }
 
+_Py_HashSecret_t _Py_HashSecret;
+
 long
 PyObject_Hash(PyObject *v)
 {
diff -r e7706bdaaa0d Objects/unicodeobject.c
--- a/Objects/unicodeobject.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Objects/unicodeobject.c	Mon Jan 30 17:21:17 2012 -0500
@@ -7344,11 +7344,17 @@
     if (self->hash != -1)
         return self->hash;
     len = Py_SIZE(self);
+    if (len == 0) {
+        self->hash = 0;
+        return 0;
+    }
     p = self->str;
-    x = *p << 7;
+    x = _Py_HashSecret.prefix;
+    x ^= *p << 7;
     while (--len >= 0)
         x = (1000003*x) ^ *p++;
     x ^= Py_SIZE(self);
+    x ^= _Py_HashSecret.suffix;
     if (x == -1)
         x = -2;
     self->hash = x;
diff -r e7706bdaaa0d PCbuild/pythoncore.vcproj
--- a/PCbuild/pythoncore.vcproj	Fri Jan 27 09:48:47 2012 +0100
+++ b/PCbuild/pythoncore.vcproj	Mon Jan 30 17:21:17 2012 -0500
@@ -1778,6 +1778,10 @@
 				RelativePath="..\Python\pythonrun.c"
 				>
 			</File>
+ 			<File
+				RelativePath="..\Python\random.c"
+				>
+			</File>
 			<File
 				RelativePath="..\Python\structmember.c"
 				>
diff -r e7706bdaaa0d Python/pythonrun.c
--- a/Python/pythonrun.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Python/pythonrun.c	Mon Jan 30 17:21:17 2012 -0500
@@ -73,6 +73,7 @@
 extern void _PyUnicode_Fini(void);
 extern int _PyLong_Init(void);
 extern void PyLong_Fini(void);
+extern void _PyRandom_Init(void);
 
 #ifdef WITH_THREAD
 extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@@ -91,6 +92,7 @@
 int Py_IgnoreEnvironmentFlag; /* e.g. PYTHONPATH, PYTHONHOME */
 int Py_NoUserSiteDirectory = 0; /* for -s and site.py */
 int Py_UnbufferedStdioFlag = 0; /* Unbuffered binary std{in,out,err} */
+int Py_HashRandomizationFlag = 0; /* for -R and PYTHONHASHRANDOMIZATION */
 
 /* PyModule_GetWarningsModule is no longer necessary as of 2.6
 since _warnings is builtin.  This API should not be used. */
@@ -195,6 +197,10 @@
         Py_OptimizeFlag = add_flag(Py_OptimizeFlag, p);
     if ((p = Py_GETENV("PYTHONDONTWRITEBYTECODE")) && *p != '\0')
         Py_DontWriteBytecodeFlag = add_flag(Py_DontWriteBytecodeFlag, p);
+    if ((p = Py_GETENV("PYTHONHASHRANDOMIZATION")) && *p != '\0')
+        Py_HashRandomizationFlag = add_flag(Py_HashRandomizationFlag, p);
+    
+    _PyRandom_Init();
 
     interp = PyInterpreterState_New();
     if (interp == NULL)
diff -r e7706bdaaa0d Python/random.c
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Python/random.c	Mon Jan 30 17:21:17 2012 -0500
@@ -0,0 +1,284 @@
+#include "Python.h"
+#ifdef MS_WINDOWS
+#include <windows.h>
+#else
+#include <fcntl.h>
+#endif
+
+static int random_initialized = 0;
+
+#ifdef MS_WINDOWS
+typedef BOOL (WINAPI *CRYPTACQUIRECONTEXTA)(HCRYPTPROV *phProv,\
+              LPCSTR pszContainer, LPCSTR pszProvider, DWORD dwProvType,\
+              DWORD dwFlags );
+typedef BOOL (WINAPI *CRYPTGENRANDOM)(HCRYPTPROV hProv, DWORD dwLen,\
+              BYTE *pbBuffer );
+
+static CRYPTGENRANDOM pCryptGenRandom = NULL;
+/* This handle is never explicitly released. Instead, the operating
+   system will release it when the process terminates. */
+static HCRYPTPROV hCryptProv = 0;
+
+static int
+win32_urandom_init(int raise)
+{
+    HINSTANCE hAdvAPI32 = NULL;
+    CRYPTACQUIRECONTEXTA pCryptAcquireContext = NULL;
+
+    /* Obtain handle to the DLL containing CryptoAPI. This should not fail. */
+    hAdvAPI32 = GetModuleHandle("advapi32.dll");
+    if(hAdvAPI32 == NULL)
+        goto error;
+
+    /* Obtain pointers to the CryptoAPI functions. This will fail on some early
+       versions of Win95. */
+    pCryptAcquireContext = (CRYPTACQUIRECONTEXTA)GetProcAddress(
+                               hAdvAPI32, "CryptAcquireContextA");
+    if (pCryptAcquireContext == NULL)
+        goto error;
+
+    pCryptGenRandom = (CRYPTGENRANDOM)GetProcAddress(hAdvAPI32,
+                                                     "CryptGenRandom");
+    if (pCryptGenRandom == NULL)
+        goto error;
+
+    /* Acquire context */
+    if (! pCryptAcquireContext(&hCryptProv, NULL, NULL,
+                               PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
+        goto error;
+
+    return 0;
+
+error:
+    if (raise)
+        PyErr_SetFromWindowsErr(0);
+    else
+        Py_FatalError("Failed to initialize Windows random API (CryptoGen)");
+    return -1;
+}
+
+/* Fill buffer with size pseudo-random bytes generated by the Windows CryptoGen
+   API. Return 0 on success, or -1 on error. */
+static int
+win32_urandom(unsigned char *buffer, Py_ssize_t size, int raise)
+{
+    Py_ssize_t orig_size = size;
+    Py_ssize_t chunk;
+
+    if (hCryptProv == 0)
+    {
+        if (win32_urandom_init(raise) == -1)
+            return -1;
+    }
+
+    while (size > 0)
+    {
+        chunk = Py_MIN(size, INT_MAX);
+        if (!pCryptGenRandom(hCryptProv, chunk, buffer))
+        {
+            /* CryptGenRandom() failed */
+            if (raise)
+                PyErr_SetFromWindowsErr(0);
+            else
+                Py_FatalError("Failed to initialized the randomized hash "
+                        "secret using CryptoGen)");
+            return -1;
+        }
+        buffer += chunk;
+        size -= chunk;
+    }
+    return 0;
+}
+
+#else
+
+/* Read size bytes from /dev/urandom into buffer.
+   Call Py_FatalError() on error. */
+static void
+dev_urandom_noraise(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    assert (0 < size);
+
+    fd = open("/dev/urandom", O_RDONLY);
+    if (fd < 0)
+        Py_FatalError("Failed to open /dev/urandom");
+
+    while (0 < size)
+    {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+        {
+            /* stop on error or if read(size) returned 0 */
+            Py_FatalError("Failed to read bytes from /dev/urandom");
+            break;
+        }
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    }
+    close(fd);
+}
+
+/* Read size bytes from /dev/urandom into buffer.
+   Return 0 on success, raise an exception and return -1 on error. */
+static int
+dev_urandom_python(char *buffer, Py_ssize_t size)
+{
+    int fd;
+    Py_ssize_t n;
+
+    if (size <= 0)
+        return 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    fd = open("/dev/urandom", O_RDONLY);
+    Py_END_ALLOW_THREADS
+    if (fd < 0)
+    {
+        PyErr_SetFromErrnoWithFilename(PyExc_OSError, "/dev/urandom");
+        return -1;
+    }
+
+    Py_BEGIN_ALLOW_THREADS
+    do {
+        do {
+            n = read(fd, buffer, (size_t)size);
+        } while (n < 0 && errno == EINTR);
+        if (n <= 0)
+            break;
+        buffer += n;
+        size -= (Py_ssize_t)n;
+    } while (0 < size);
+    Py_END_ALLOW_THREADS
+
+    if (n <= 0)
+    {
+        /* stop on error or if read(size) returned 0 */
+        if (n < 0)
+            PyErr_SetFromErrno(PyExc_OSError);
+        else
+            PyErr_Format(PyExc_RuntimeError,
+                         "Failed to read %zi bytes from /dev/urandom",
+                         size);
+        close(fd);
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
+/* Fill buffer with pseudo-random bytes generated by a linear congruent
+   generator (LCG):
+
+       x(n+1) = (x(n) * 214013 + 2531011) % 2^32
+
+   Use bits 23..16 of x(n) to generate a byte. */
+static void
+lcg_urandom(unsigned int x0, unsigned char *buffer, size_t size)
+{
+    size_t index;
+    unsigned int x;
+
+    x = x0;
+    for (index=0; index < size; index++) {
+        x *= 214013;
+        x += 2531011;
+        /* modulo 2 ^ (8 * sizeof(int)) */
+        buffer[index] = (x >> 16) & 0xff;
+    }
+}
+
+/* Fill buffer with size pseudo-random bytes, not suitable for cryptographic
+   use, from the operating random number generator (RNG).
+
+   Return 0 on success, raise an exception and return -1 on error. */
+int
+_PyOS_URandom(void *buffer, Py_ssize_t size)
+{
+    if (size < 0) {
+        PyErr_Format(PyExc_ValueError,
+                     "negative argument not allowed");
+        return -1;
+    }
+    if (size == 0)
+        return 0;
+
+#ifdef MS_WINDOWS
+    return win32_urandom((unsigned char *)buffer, size, 1);
+#else
+    return dev_urandom_python((char*)buffer, size);
+#endif
+}
+
+void
+_PyRandom_Init(void)
+{
+    char *env;
+    void *secret = &_Py_HashSecret;
+    Py_ssize_t secret_size = sizeof(_Py_HashSecret);
+
+    if (random_initialized)
+        return;
+    random_initialized = 1;
+
+    /*
+      By default, hash randomization is disabled, and only
+      enabled if PYTHONHASHRANDOMIZATION is set to non-empty
+      or if "-R" is provided at the command line:
+    */
+    if (!Py_HashRandomizationFlag) {
+        /* Disable the randomized hash: */
+        memset(secret, 0, secret_size);
+        return;
+    }
+
+    /*
+      Hash randomization is enabled.  Generate a per-process secret,
+      using PYTHONHASHSEED if provided.
+    */
+
+    env = Py_GETENV("PYTHONHASHSEED");
+    if (env && *env != '\0') {
+        char *endptr = env;
+        unsigned long seed;
+        seed = strtoul(env, &endptr, 10);
+        if (*endptr != '\0'
+            || seed > 4294967295UL
+            || (errno == ERANGE && seed == ULONG_MAX))
+        {
+            Py_FatalError("PYTHONHASHSEED must be an integer "
+                          "in range [0; 4294967295]");
+        }
+        if (seed == 0) {
+            /* disable the randomized hash */
+            memset(secret, 0, secret_size);
+        }
+        else {
+            lcg_urandom(seed, (unsigned char*)secret, secret_size);
+        }
+    }
+    else {
+#ifdef MS_WINDOWS
+#if 1
+        (void)win32_urandom((unsigned char *)secret, secret_size, 0);
+#else
+        /* fast but weak RNG (fast initialization, weak seed) */
+        _PyTime_timeval t;
+        unsigned int seed;
+        _PyTime_gettimeofday(&t);
+        seed = (unsigned int)t.tv_sec;
+        seed ^= t.tv_usec;
+        seed ^= getpid();
+        lcg_urandom(seed, (unsigned char*)secret, secret_size);
+#endif
+#else /* #ifdef MS_WINDOWS */
+        dev_urandom_noraise((char*)secret, secret_size);
+#endif
+    }
+}
+
diff -r e7706bdaaa0d Python/sysmodule.c
--- a/Python/sysmodule.c	Fri Jan 27 09:48:47 2012 +0100
+++ b/Python/sysmodule.c	Mon Jan 30 17:21:17 2012 -0500
@@ -1126,6 +1126,7 @@
     /* {"unbuffered",                   "-u"}, */
     /* {"skip_first",                   "-x"}, */
     {"bytes_warning", "-b"},
+    {"hash_randomization", "-R"},
     {0}
 };
 
@@ -1134,9 +1135,9 @@
     flags__doc__,       /* doc */
     flags_fields,       /* fields */
 #ifdef RISCOS
+    13
+#else
     12
-#else
-    11
 #endif
 };
 
@@ -1169,6 +1170,7 @@
     /* SetFlag(saw_unbuffered_flag); */
     /* SetFlag(skipfirstline); */
     SetFlag(Py_BytesWarningFlag);
+    SetFlag(Py_HashRandomizationFlag);
 #undef SetFlag
 
     if (PyErr_Occurred()) {

From report at bugs.python.org  Mon Jan 30 23:25:52 2012
From: report at bugs.python.org (Stefan Krah)
Date: Mon, 30 Jan 2012 22:25:52 +0000
Subject: [issue13837] test_shutil fails with symlinks enabled under Windows
In-Reply-To: <1327255807.39.0.63998442373.issue13837@psf.upfronthosting.co.za>
Message-ID: <1327962352.7.0.524309022663.issue13837@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


----------
nosy: +skrah

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13837>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:27:45 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 22:27:45 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327962465.55.0.41108849259.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Version 5:
 - add "datetime" and "timespec" formats: datetime.datetime object and  (sec: int, nsec: int)
 - add timestamp optional format to os.stat(), os.lstat(), os.fstat(), os.fstatat()
 - support passing the timestamp format as a keyword: time.time(format="decimal")

I am not really conviced by the usefulness of "timespec" format, but it was just an example for #11457.

The "datetime" format is surprising for time.clock() and time.wallclock(), these timestamps use an arbitrary start. I suppose that time.clock(format="datetime") and time.wallclock(format="datetime") should raise a ValueError.

----------
Added file: http://bugs.python.org/file24372/time_decimal-5.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:34:46 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 22:34:46 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327962886.23.0.344246491422.issue13882@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24367/time_decimal-4.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:42:25 2012
From: report at bugs.python.org (Brett Cannon)
Date: Mon, 30 Jan 2012 22:42:25 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327963345.73.0.864227177747.issue13912@psf.upfronthosting.co.za>


Brett Cannon <brett at python.org> added the comment:

I see your mistake now: you need to call it as __import__('pkgB', globals(), index=1), else __import__ has no clue how to anchor your import in the relative package space.

Try that and let me know if it makes it work.

And why exactly are you trying to call __import__ directly? You should be using importlib.import_module() for programmatic imports.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:44:00 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 22:44:00 +0000
Subject: [issue11457] os.stat(): add new fields to get timestamps as Decimal
	objects with nanosecond resolution
In-Reply-To: <1299715528.65.0.317038011667.issue11457@psf.upfronthosting.co.za>
Message-ID: <1327963440.72.0.787817674063.issue11457@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

I attached a more complete patch to the issue #13882: it adds an optional timestamp format to os.stat(), os.lstat(), os.fstat(), os.fstatat().

Examples:

$ ./python 
Python 3.3.0a0 (default:2914ce82bf89+, Jan 30 2012, 23:07:24) 
>>> import os
>>> s=os.stat("setup.py", timestamp="datetime")
>>> s.st_mtime - s.st_ctime
datetime.timedelta(0)
>>> print(s.st_atime - s.st_ctime)
52 days, 1:44:06.191293
>>> os.stat("setup.py", timestamp="timespec").st_ctime
(1323458640, 702327236)
>>> os.stat("setup.py", timestamp="decimal").st_ctime
Decimal('1323458640.702327236')

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11457>
_______________________________________

From report at bugs.python.org  Mon Jan 30 23:49:30 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Mon, 30 Jan 2012 22:49:30 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327880535.51.0.390396847127.issue13882@psf.upfronthosting.co.za>
Message-ID: <CAP7h-xZBcrmnHGzLX-cztE3oZx2K9Rmwzb2GB1e2zhPi_ahK+A@mail.gmail.com>


Alexander Belopolsky <alexander.belopolsky at gmail.com> added the comment:

On Sun, Jan 29, 2012 at 6:42 PM, STINNER Victor <report at bugs.python.org> wrote:
..
> What do you call a constant argument? "float" and "decimal"?
> You would prefer a constant like time.FLOAT_FORMAT?
> Or maybe a boolean (decimal=True)?

Yes. This was explained on python-dev not so long ago:

http://mail.python.org/pipermail/python-dev/2010-July/102100.html

The problem is not with the type of the argument (although boolean or
enum argument type is often a symptom pointing to the issue.)  The
issue is that an argument that is never given a variable value at the
call site is usually a sign of an awkward API.  For example, what
would you prefer:

math('log', x)

or

math.log(x)

?

>
> I chose a string because my first idea was to add a registry to support other format,
> maybe user defined formats, like the one used by Unicode codecs.

With all my respect for MAL, codecs are not my favorite part of the
python library.

One possibility (still awkward IMO) would be to use the return type as
the format specifier.  This would at least require the user to import
datetime or decimal before calling time() with corresponding format.
Few users would tolerate I/O delay when they want to get time with
nanosecond precision.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:05:31 2012
From: report at bugs.python.org (Jason R. Coombs)
Date: Mon, 30 Jan 2012 23:05:31 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327964731.13.0.311264274877.issue13912@psf.upfronthosting.co.za>


Jason R. Coombs <jaraco at jaraco.com> added the comment:

Thanks for the tip Brent. Still, no luck.

    jaraco at devjaraco:~$ python2.7 -c 'import master.pkgA; print("pkgA.__package__ is {}".format(master.pkgA.__package__)); import master.pkgA.foo'
    pkgA.__package__ is None
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "master/pkgA/foo.py", line 2, in <module>
        bar = __import__('pkgB', globals(), level=1).bar
    ImportError: No module named pkgB

I considered importlib, but it is only included for Python 2.7+ and I have to support Python 2.6. I'm using programmatic imports for a few  reasons:

1) I'm importing the modules for the side effects, so I don't need or want the name in the namespace.
2) If I import it naturally, I get pyflakes warnings that it's an used imports.
3) The order of imports is important. If I place it too early in the imports, it doesn't have the state it needs and fails.
4) By placing it in the top of the file with other imports, it's not obvious why it's being imported, so requires comments.
5) The most appropriate place to invoke this functionality is in a specific function, "run_app".

I see now that there is an importlib backport in the cheeseshop for earlier versions of Python. Given your recommendation, I'll give that a try.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:15:29 2012
From: report at bugs.python.org (STINNER Victor)
Date: Mon, 30 Jan 2012 23:15:29 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327611637.79.0.902972785006.issue13882@psf.upfronthosting.co.za>
Message-ID: <1327965329.96.0.695497919885.issue13882@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> One possibility (still awkward IMO) would be to use the return type as
> the format specifier.

Yeah, I already thaught to this idea. The API would be:
 - time.time(format=float)
 - time.time(format=decimal.Decimal)
 - time.time(format=datetime.datetime)
 - time.time(format=?) # for timespec, but I don't think that we need timespec in Python which is a object oriented language, we can use better than low level strutures
 - os.stat(path, format=decimal.Decimal)
 - etc.

I have to write a function checking that obj is decimal.Decimal or datetime.datetime without importing the module. I suppose that it is possible by checking obj type (it must be a class) and then obj.__module__.

> This would at least require the user to import
> datetime or decimal before calling time() with corresponding
> format.

Another possibility is what I proposed before in the issue #11457: take a callback argument.
http://bugs.python.org/issue11457#msg143738

The callback prototype would be:

def myformat(seconds, floatpart, divisor):
    return ...

Each module can implements its own converter and time can provide some builtin converts (because I don't want to add something related to time in the decimal module for example).

But I don't really like this idea because it requires to decide the API of the low level structure of a timestamp (which may change later), and it doesn't really solve the issue of "import decimal" if the converter is in the time module.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:24:26 2012
From: report at bugs.python.org (Alexander Belopolsky)
Date: Mon, 30 Jan 2012 23:24:26 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <1327965329.96.0.695497919885.issue13882@psf.upfronthosting.co.za>
Message-ID: <CAP7h-xYZWhrDSfzb9RLKhFiX9dLHfLPtUmaiqpp2wCVOD5fCfQ@mail.gmail.com>


Alexander Belopolsky <alexander.belopolsky at gmail.com> added the comment:

On Mon, Jan 30, 2012 at 6:15 PM, STINNER Victor <report at bugs.python.org> wrote:
> Another possibility is what I proposed before in the issue #11457: take a callback argument.
> http://bugs.python.org/issue11457#msg143738

I think you are over-engineering a complicated solution to a simple
problem.  If you want to add functionality to time module, add a
function with a new name that would mimic appropriate POSIX API and
return a (named) tuple of integers where C function would populate a
struct.  If you write a new high resolution time module - import
decimal at the top of the module and make all your clocks return
seconds as Decimal objects.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:25:45 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Mon, 30 Jan 2012 23:25:45 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1327965945.56.0.885419783195.issue13609@psf.upfronthosting.co.za>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Updated version following comments by Victor Stinner: termsize.diff.7

- os.get_terminal_size() is moved to shutil.get_terminal_size()
- some small doc updates

Victor STINNER wrote:
>> I noticed that bash uses $LINES, not $ROWS. I have renamed rows to
>> lines everywhere, to follow existing convention.
> The stty program has "rows" and "columns" commands to set the terminal
Yes, also struct winsize uses 'rows' and 'cols'. But since we use the variable $LINES, it seems better to use 'lines'.

Thanks for the link to SUSv2! I've added a link in the docs, since it adds a nice justification to the COLUMNS/query/fallback behaviour.

Now get_terminal_size() lives in shutil. I don't think it matters much, but I see the point of keeping os clean.

I also looked at some programs, to see who behaves how.
dpkg -> COLUMNS first
aptitude -> ignores COLUMNS
git -> COLUMNS first
systemctl, loginctl, systemd-cgls (systemd cmdline interface) -> COLUMNS first
less, w ->  ioctl, COLUMNS as fallback

(Of course this is very unscientific, because I just picked some programs at random which I remembered to care about the terminal size).
I guess that the case of 'less' is special, because less is very
much dependent on the terminal and does quite a lot of manipulations on it. Otherwise, taking COLUMNS as highest priority seems well established, even if not universal.

----------
Added file: http://bugs.python.org/file24373/termsize.diff.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:33:31 2012
From: report at bugs.python.org (Eric Snow)
Date: Mon, 30 Jan 2012 23:33:31 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327966411.3.0.862648677538.issue13912@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

Jason: just a warning.  importlib_backport is a relatively naive tool for generating the backport from the py3k source.  It's also relatively fragile and at this point probably doesn't work with the default branch.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Tue Jan 31 00:41:44 2012
From: report at bugs.python.org (Martin)
Date: Mon, 30 Jan 2012 23:41:44 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327966904.94.0.273785927847.issue13703@psf.upfronthosting.co.za>


Martin <gzlist at googlemail.com> added the comment:

> Has anyone had a chance to try this patch on Windows?  Martin?  I'm
> hoping that it doesn't impose a startup cost in the default
> no-randomization cost, and that any startup cost in the -R case is
> acceptable.

Just tested as requested. Is the patch against 3.1 for a reason? Can't
really be compared to earlier results, but get enough weird outliers
that that may not be useful anyway. Also needed the following change:

-+        chunk = Py_MIN(size, INT_MAX);
++        chunk = size > INT_MAX ? INT_MAX : size;

Summary, looks like extra work in the default case is avoided and
isn't crippling otherwise, though there were a couple of very odd runs
not presented probably due to other disk access.

Vanilla:

>timeit PCbuild\python.exe -c "import sys;print(sys.version)"
3.1.4+ (default, Jan 30 2012, 22:38:52) [MSC v.1500 32 bit (Intel)]

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        10:42 pm, Monday, January 30 2012
Elapsed Time:     0:00:00.218
Process Time:     0:00:00.187
System Calls:     3974
Context Switches: 574
Page Faults:      1696
Bytes Read:       480331
Bytes Written:    0
Bytes Other:      190860


Patched:

>timeit PCbuild\python.exe -c "import sys;print(sys.version)"
3.1.4+ (default, Jan 30 2012, 22:55:06) [MSC v.1500 32 bit (Intel)]

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        10:55 pm, Monday, January 30 2012
Elapsed Time:     0:00:00.218
Process Time:     0:00:00.187
System Calls:     3560
Context Switches: 441
Page Faults:      1660
Bytes Read:       461956
Bytes Written:    0
Bytes Other:      24926


>timeit PCbuild\python.exe -Rc "import sys;print(sys.version)"
3.1.4+ (default, Jan 30 2012, 22:55:06) [MSC v.1500 32 bit (Intel)]

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        11:05 pm, Monday, January 30 2012
Elapsed Time:     0:00:00.249
Process Time:     0:00:00.234
System Calls:     3959
Context Switches: 483
Page Faults:      1847
Bytes Read:       892464
Bytes Written:    0
Bytes Other:      27090

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 31 01:03:48 2012
From: report at bugs.python.org (Ezra Berch)
Date: Tue, 31 Jan 2012 00:03:48 +0000
Subject: [issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
In-Reply-To: <1327462635.84.0.618726818638.issue13857@psf.upfronthosting.co.za>
Message-ID: <1327968228.44.0.564322788453.issue13857@psf.upfronthosting.co.za>


Ezra Berch <ezraberch at mac.com> added the comment:

I've created a patch using the conditional expression in msg151945. The one problem I found with it is that when the input string is terminated by a newline it removes that newline.

I've added an optional third argument: a function which determines which lines are indented. If omitted, the default behavior is to indent non-empty lines.

----------
keywords: +patch
nosy: +ezberch
Added file: http://bugs.python.org/file24374/indent.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13857>
_______________________________________

From report at bugs.python.org  Tue Jan 31 02:34:16 2012
From: report at bugs.python.org (Dave Malcolm)
Date: Tue, 31 Jan 2012 01:34:16 +0000
Subject: [issue13703] Hash collision security issue
In-Reply-To: <1325619410.45.0.862265945379.issue13703@psf.upfronthosting.co.za>
Message-ID: <1327973656.07.0.527772935894.issue13703@psf.upfronthosting.co.za>


Dave Malcolm <dmalcolm at redhat.com> added the comment:

Am attaching a backport of optin-hash-randomization-for-3.1-dmalcolm-2012-01-30-002.patch to 2.6

Randomization covers the str, unicode and buffer types; equality of hashes is preserved for these types.

----------
Added file: http://bugs.python.org/file24375/optin-hash-randomization-for-2.6-dmalcolm-2012-01-30-001.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13703>
_______________________________________

From report at bugs.python.org  Tue Jan 31 02:46:57 2012
From: report at bugs.python.org (Eric Snow)
Date: Tue, 31 Jan 2012 01:46:57 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1327974417.05.0.00912499830975.issue13912@psf.upfronthosting.co.za>


Eric Snow <ericsnowcurrently at gmail.com> added the comment:

The problem is your level is off and the name is incomplete.  master.pkgA.foo.py should have the following:

__import__('pkgB.bar', master.pkgA.__dict__, level=2).bar

or 

__import__('pkgB.bar', master.pkgA.__dict__, fromlist=['-'], level=2)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Tue Jan 31 03:10:33 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Tue, 31 Jan 2012 02:10:33 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327975833.16.0.359148926652.issue13903@psf.upfronthosting.co.za>


Changes by Raymond Hettinger <raymond.hettinger at gmail.com>:


----------
assignee:  -> rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Tue Jan 31 03:17:21 2012
From: report at bugs.python.org (Raymond Hettinger)
Date: Tue, 31 Jan 2012 02:17:21 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1327976241.58.0.481478004965.issue13903@psf.upfronthosting.co.za>


Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Changing dictionaries is a big deal.  You're changing many pieces at once (not a good idea) including changing tunable parameters that are well-studied (I spent a month testing whether 5/8 was better idea that 2/3 for resizing or when the ideal small dict size was 4, 8, or 16).  You're changing the meaning of the fields in dictobject.h which will likely break any code that relied on those.

The ideas may be good ones but they warrant a good deal of thought.  Dicts weren't just slapped together -- the current code is the product to two decades of tweaking by engineers who devoted significant time to the task.  It would be easy to unknowingly undo some of their work.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Tue Jan 31 04:47:52 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 31 Jan 2012 03:47:52 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <1327981672.3.0.109080716239.issue13506@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I tested on 3.2, Win 7. sys.path starts with '' on startup, after restart, and after ^c interrupt of 'while True: pass'. It start with absolute path when running a file from the editor. I believe this is as should be.

I simplified the patch a bit. Its basic idea is to add a parameter with_cwd to PyShell.ModifiedInterpreter.transfer_path. That method is called by methods start_subprocess and restart_subprocess. start_process is only called by PyShell.PyShell.begin. So I think there is no need to propagate the new parameter back past the call in start_subprocess.

.restart_subprocess is called in .poll_subprocess, .runcode, PyShell.cancel_process, and PyShell.restart_shell. (I am a little puzzled why ModifiedInterpreter.runcode calls self.interp.restart_subprocess, as in the PyShell methods, instead of self.restart_subprocess, as in poll_subprocess, but moving on...)

restart_process needs the new parameter to pass to transfer_path because it should be True for F6 Restart Shell and not for F5 Run Module. Roger's patch also left the default False for the other calls. I do not know what they do so I cannot judge them. Roger, do you know, and did you consider each? (Leaving the current behavior for those will at worst not fix something that could be, so it is safe.)

restart_shell is *bound* to F6 Restart Shell (which passes the ignored even), so the default with_cwd must be True for that binding. It is *called* by a function ScriptBinding.py that gets bound, so the call can be altered to pass the correct False. I prefer that to making two functions. That call could be replaces by a call directly to .interp.restart_subprocess(False) (and restart_shell slightly simplified).

I retested after these changes to the patch. It is from TortoiseHG, so should apply (after I modified the file paths). Since hacking on idlelib is new for me, I would like other eyes and tests before I apply.

----------
assignee:  -> terry.reedy
Added file: http://bugs.python.org/file24376/issue13506c.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 04:51:44 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 31 Jan 2012 03:51:44 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <1327981904.39.0.761292818206.issue13506@psf.upfronthosting.co.za>


Changes by Terry J. Reedy <tjreedy at udel.edu>:


Removed file: http://bugs.python.org/file24376/issue13506c.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 04:53:57 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 31 Jan 2012 03:53:57 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <1327982037.18.0.423537358136.issue13506@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

Whoops, hit submit too soon, before saving tab to space change.

----------
Added file: http://bugs.python.org/file24377/issue13506d.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 05:54:43 2012
From: report at bugs.python.org (Roger Serwy)
Date: Tue, 31 Jan 2012 04:54:43 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <1327985683.92.0.539014045852.issue13506@psf.upfronthosting.co.za>


Roger Serwy <roger.serwy at gmail.com> added the comment:

I tested your patch and it works. For the sake of completeness, here's what I did:

Test 1: Start IDLE in shell mode and run "import sys; print(sys.path)". The first entry should be ''

This is consistent with the behavior of the regular python shell.

Test 2: Restart the shell with Ctrl+F6 (and View->Restart Shell) and repeat Test 1. This should give the same result.

Test 3: Create a script in /tmp/a.py with "import sys; print(sys.path)" as its contents. Running this script from the editor gives '/tmp' as the first entry.

This is consistent with the behavior of running "python3 /tmp/a.py".

I'd make one minor change to the patch for sake of being explicit in ScriptBinding.py:
    shell.restart_shell(event, with_cwd=False)

My rationale was to modify only PyShell.py to get the correct behavior. Clearly, making a simple modification to ScriptBinding.py gives a much shorter solution. I also like your simplification for "start_subprocess". 

I did consider each case of default arguments. I developed my patch iteratively starting by having "with_cwd=False" for all function calls that called "transfer_path" so that the existing (incorrect) behavior remained. I then modified how these functions were called, setting "with_cwd=True" in the cases where '' needed to be transferred, namely on the shell start and restart from within the PyShell window. That is why I introduced "restart_shell_interactive". I do agree with your reasoning that one function call is better than two.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 06:02:20 2012
From: report at bugs.python.org (Ramchandra Apte)
Date: Tue, 31 Jan 2012 05:02:20 +0000
Subject: [issue13701] Remove Decimal Python 2.3 Compatibility
In-Reply-To: <1325559706.07.0.31676641509.issue13701@psf.upfronthosting.co.za>
Message-ID: <1327986140.76.0.0784454923339.issue13701@psf.upfronthosting.co.za>


Ramchandra Apte <maniandram01 at gmail.com> added the comment:

There is another problem with the code it creates a dummy module when threading is not available but now threading imports dummy_threading when threading is not available.

Thanks

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13701>
_______________________________________

From report at bugs.python.org  Tue Jan 31 08:59:27 2012
From: report at bugs.python.org (Roundup Robot)
Date: Tue, 31 Jan 2012 07:59:27 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <E1Rs8cf-00020v-Sm@dinsdale.python.org>


Roundup Robot <devnull at psf.upfronthosting.co.za> added the comment:

New changeset 1b5abba0c808 by Terry Jan Reedy in branch '2.7':
#13506 Add '' to path for interactive interpreter by adding with_cwd parameter
http://hg.python.org/cpython/rev/1b5abba0c808

New changeset 1993aa091d89 by Terry Jan Reedy in branch '3.2':
#13506 Add '' to path for interactive interpreter by adding with_cwd parameter
http://hg.python.org/cpython/rev/1993aa091d89

New changeset acedd92086c5 by Terry Jan Reedy in branch 'default':
Merge 3.2
http://hg.python.org/cpython/rev/acedd92086c5

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 09:06:38 2012
From: report at bugs.python.org (Terry J. Reedy)
Date: Tue, 31 Jan 2012 08:06:38 +0000
Subject: [issue13506] IDLE sys.path does not contain Current Working Directory
In-Reply-To: <1322621465.07.0.137768445913.issue13506@psf.upfronthosting.co.za>
Message-ID: <1327997198.06.0.952408064878.issue13506@psf.upfronthosting.co.za>


Terry J. Reedy <tjreedy at udel.edu> added the comment:

I noticed that the script in ScriptBinding.py already dereferences interp and calls 3 interp functions directly, so I decided that calling a 4th on interp was fine. That lets restart_shell be specialized to the one purpose of being bound to Restart Shell on the menu. I added a docstring to that effect so no will will try to generalize it again to use when running a module.

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13506>
_______________________________________

From report at bugs.python.org  Tue Jan 31 11:38:17 2012
From: report at bugs.python.org (Mark Shannon)
Date: Tue, 31 Jan 2012 10:38:17 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327976241.58.0.481478004965.issue13903@psf.upfronthosting.co.za>
Message-ID: <4F27C496.4030800@hotpy.org>


Mark Shannon <mark at hotpy.org> added the comment:

Raymond Hettinger wrote:
> Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:
> 
> Changing dictionaries is a big deal.  You're changing many pieces at once (not a good idea) including changing tunable parameters that are well-studied (I spent a month testing whether 5/8 was better idea that 2/3 for resizing or when the ideal small dict size was 4, 8, or 16).  You're changing the meaning of the fields in dictobject.h which will likely break any code that relied on those.
> 
> The ideas may be good ones but they warrant a good deal of thought.  Dicts weren't just slapped together -- the current code is the product to two decades of tweaking by engineers who devoted significant time to the task.  It would be easy to unknowingly undo some of their work.
> 

OK.
I'll write a PEP.

By the way, I'm trying not changing the tunable parameters for the 
unshared-keys case, just the shared-keys case. Of course, they do 
interact with each other.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:02:09 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 31 Jan 2012 12:02:09 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327942847.95.0.649353892202.issue13912@psf.upfronthosting.co.za>
Message-ID: <1328011329.93.0.52252821481.issue13912@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

It sounds like you may want runpy.run_module [1], rather than using imports at all. If you know how many levels up you want to go, it isn't hard to do your own munging of __name__ to create absolute module references to pass to runpy.

The signature of __import__ is known to be unintuitive to the point of being insane - it's really designed for the convenience of the compiler and the interpreter, not for direct use by humans.

[1] http://docs.python.org/release/2.6.7/library/runpy#runpy.run_module

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:00:25 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 12:00:25 +0000
Subject: [issue13882] Add format argument for time.time(), time.clock(),
	... to get a timestamp as a Decimal object
In-Reply-To: <CAP7h-xYZWhrDSfzb9RLKhFiX9dLHfLPtUmaiqpp2wCVOD5fCfQ@mail.gmail.com>
Message-ID: <CAMpsgwagRdhaav6JamVHYbbxF7ddG7tpzjRv7O4tdahqH5kbqw@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch version 6:

 - timestamp format is now a type instead of a string, e.g. time.time(int)
 - add int and datetime.timedelta formats, remove timespec format
 - complete the documentation
 - fix integer overflows, convert correctly time_t to PyLong

There are now 5 timestamp formats:

 - int
 - float
 - decimal.Decimal
 - datetime.datetime
 - datetime.timedelta

I consider the patch as ready to be commited, or at least ready for a
review ;-) There is no more FIXME or known limitation. Well, now the
most important part is to decide the API and the list of timestamp
formats.

The patch should be tested on Linux, FreeBSD and Windows, 32 and 64
bits to check assertions on type sizes:

assert(sizeof(clock_t) <= sizeof(size_t));
assert(sizeof(LONGLONG) <= sizeof(size_t));
assert(sizeof(time_t) <= sizeof(PY_LONG_LONG));

----------
Added file: http://bugs.python.org/file24378/time_decimal-6.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13882>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -808,13 +808,16 @@ as internal buffering of data.
    Availability: Unix.
 
 
-.. function:: fstat(fd)
+.. function:: fstat(fd, timestamp=None)
 
    Return status for file descriptor *fd*, like :func:`~os.stat`.
 
    Availability: Unix, Windows.
 
-.. function:: fstatat(dirfd, path, flags=0)
+   .. versionchanged:: 3.3
+      Added the *timestamp* argument.
+
+.. function:: fstatat(dirfd, path, flags=0, timestamp="float")
 
    Like :func:`stat` but if *path* is relative, it is taken as relative to *dirfd*.
    *flags* is optional and may be 0 or :data:`AT_SYMLINK_NOFOLLOW`.
@@ -1696,7 +1699,7 @@ Files and Directories
    .. versionadded:: 3.3
 
 
-.. function:: lstat(path)
+.. function:: lstat(path, timestamp=None)
 
    Perform the equivalent of an :c:func:`lstat` system call on the given path.
    Similar to :func:`~os.stat`, but does not follow symbolic links.  On
@@ -1706,6 +1709,9 @@ Files and Directories
    .. versionchanged:: 3.2
       Added support for Windows 6.0 (Vista) symbolic links.
 
+   .. versionchanged:: 3.3
+      The *timestamp* argument was added.
+
 
 .. function:: lutimes(path[, times])
 
@@ -1955,7 +1961,7 @@ Files and Directories
    .. versionadded:: 3.3
 
 
-.. function:: stat(path)
+.. function:: stat(path, timestamp=None)
 
    Perform the equivalent of a :c:func:`stat` system call on the given path.
    (This function follows symlinks; to stat a symlink use :func:`lstat`.)
@@ -1975,6 +1981,11 @@ Files and Directories
    * :attr:`st_ctime` - platform dependent; time of most recent metadata change on
      Unix, or the time of creation on Windows)
 
+   :attr:`st_atime`, :attr:`st_mtime` and :attr:`st_ctime` values type is
+   :class:`float` by default, or :class:`int` if :func:`os.stat_float_times` is
+   ``False``. Set the optional *timestamp* argument to get another
+   :ref:`timestamp format <timestamp-formats>`.
+
    On some Unix systems (such as Linux), the following attributes may also be
    available:
 
@@ -2030,6 +2041,9 @@ Files and Directories
 
    Availability: Unix, Windows.
 
+   .. versionchanged:: 3.3
+      Added the *timestamp* argument.
+
 
 .. function:: stat_float_times([newvalue])
 
@@ -2055,6 +2069,9 @@ Files and Directories
    are processed, this application should turn the feature off until the library
    has been corrected.
 
+   .. deprecated:: 3.3
+      Use *timestamp* argument of stat functions instead.
+
 
 .. function:: statvfs(path)
 
diff --git a/Doc/library/time.rst b/Doc/library/time.rst
--- a/Doc/library/time.rst
+++ b/Doc/library/time.rst
@@ -95,6 +95,16 @@ An explanation of some terminology and c
   | local time              |                         |                         |
   +-------------------------+-------------------------+-------------------------+
 
+.. _timestamp-formats:
+
+* Python supports the following timestamp formats:
+
+  * :class:`int`
+  * :class:`float`
+  * :class:`datetime.datetime`
+  * :class:`datetime.timedelta`
+  * :class:`decimal.Decimal`
+
 
 The module defines the following functions and data items:
 
@@ -118,7 +128,7 @@ The module defines the following functio
       Unlike the C function of the same name, there is no trailing newline.
 
 
-.. function:: clock()
+.. function:: clock(format=float)
 
    .. index::
       single: CPU time
@@ -135,16 +145,27 @@ The module defines the following functio
    :c:func:`QueryPerformanceCounter`. The resolution is typically better than one
    microsecond.
 
+   Return the time as a floating point number by default, set the optional
+   *format* argument to get another :ref:`timestamp format <timestamp-formats>`.
 
-.. function:: clock_getres(clk_id)
+   .. versionchanged:: 3.3
+      Added the *format* argument.
+
+
+.. function:: clock_getres(clk_id, format=float)
 
    Return the resolution (precision) of the specified clock *clk_id*.
+   Return the resolution as a floating point number by default, set the
+   optional *format* argument to get another :ref:`timestamp format <timestamp-formats>`.
+
 
    .. versionadded:: 3.3
 
-.. function:: clock_gettime(clk_id)
+.. function:: clock_gettime(clk_id, format=float)
 
    Return the time of the specified clock *clk_id*.
+   Return the time as a floating point number by default, set the optional
+   *format* argument to get another :ref:`timestamp format <timestamp-formats>`.
 
    .. versionadded:: 3.3
 
@@ -431,15 +452,20 @@ The module defines the following functio
    :exc:`TypeError` is raised.
 
 
-.. function:: time()
+.. function:: time(format=float)
 
-   Return the time as a floating point number expressed in seconds since the epoch,
-   in UTC.  Note that even though the time is always returned as a floating point
+   Return the time expressed in seconds since the epoch in UTC. Return
+   the time as a floating point number by default, set the optional *format*
+   argument to get another :ref:`timestamp format <timestamp-formats>`.
+   Note that even though the time is always returned as a floating point
    number, not all systems provide time with a better precision than 1 second.
    While this function normally returns non-decreasing values, it can return a
    lower value than a previous call if the system clock has been set back between
    the two calls.
 
+   .. versionchanged:: 3.3
+      Added the *format* argument.
+
 
 .. data:: timezone
 
@@ -537,13 +563,15 @@ The module defines the following functio
       ('EET', 'EEST')
 
 
-.. function:: wallclock()
+.. function:: wallclock(format=float)
 
    .. index::
       single: Wallclock
       single: benchmarking
 
    Return the current time in fractions of a second to the system's best ability.
+   Return the time as a floating point number by default, set the optional
+   *format* argument to get another :ref:`timestamp format <timestamp-formats>`.
    Use this when the most accurate representation of wall-clock is required, i.e.
    when "processor time" is inappropriate.  The reference point of the returned
    value is undefined so only the difference of consecutive calls is valid.
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -270,6 +270,34 @@ new, more precise information::
    '<function C.D.meth at 0x7f46b9fe31e0>'
 
 
+Timestamp formats
+=================
+
+:func:`time.clock`, :func:`time.clock_gettime`, :func:`time.clock_getres`,
+:func:`time.time` and :func:`time.wallclock` now have an optional *format*
+argument to choose the format of the timestamp. See the list of available
+:ref:`timestamp formats <timestamp-formats>`.
+
+Use :class:`decimal.Decimal` to support clock using a nanosecond resolution.
+
+:func:`os.fstat`, :func:`os.fstatat`, :func:`os.lstat` and :func:`os.stat` have
+also a *timestamp* optional argument to choose the timestamp format of
+``st_atime``, ``st_ctime`` and ``st_mtime`` fields.
+
+Example::
+
+    >>> time.time()
+    1328006975.681211
+    >>> time.time(format=int)
+    1328006979
+    >>> time.time(format=decimal.Decimal)
+    Decimal('1328006983.761119')
+    >>> time.time(format=datetime.datetime)
+    datetime.datetime(2012, 1, 31, 11, 49, 49, 409831)
+    >>> print(time.time(format=datetime.timedelta))
+    15370 days, 10:49:52.842116
+
+
 Other Language Changes
 ======================
 
diff --git a/Include/pytime.h b/Include/pytime.h
--- a/Include/pytime.h
+++ b/Include/pytime.h
@@ -12,6 +12,8 @@ functions and constants
 extern "C" {
 #endif
 
+#include "object.h"
+
 #ifdef HAVE_GETTIMEOFDAY
 typedef struct timeval _PyTime_timeval;
 #else
@@ -37,6 +39,21 @@ do { \
     ((tv_end.tv_sec - tv_start.tv_sec) + \
      (tv_end.tv_usec - tv_start.tv_usec) * 0.000001)
 
+typedef struct {
+    time_t seconds;
+    /* floatpart can be zero */
+    size_t floatpart;
+    /* divisor cannot be zero */
+    size_t divisor;
+    /* the resolution is 1/divisor */
+} _PyTime_t;
+
+/* Similar to POSIX gettimeofday.  If system gettimeofday
+   fails or is not available, fall back to lower resolution clocks.  */
+PyAPI_FUNC(void) _PyTime_get(_PyTime_t *tp);
+
+PyAPI_FUNC(PyObject*) _PyTime_AsFormat(_PyTime_t *ts, PyObject *format);
+
 /* Dummy to force linking. */
 PyAPI_FUNC(void) _PyTime_Init(void);
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -217,6 +217,30 @@ class StatAttributeTests(unittest.TestCa
             warnings.simplefilter("ignore", DeprecationWarning)
             self.check_stat_attributes(fname)
 
+    def test_stat_timestamp(self):
+        import datetime
+        import decimal
+
+        old_value = os.stat_float_times()
+        try:
+            # test invalid timestamp format
+            self.assertRaises(ValueError, os.stat, self.fname, timestamp="abc")
+            self.assertRaises(ValueError, os.stat, self.fname, timestamp=decimal.Context)
+
+            for float_times in (False, True):
+                os.stat_float_times(float_times)
+                t = os.stat(self.fname).st_mtime
+                if float_times:
+                    self.assertIsInstance(t, float)
+                else:
+                    self.assertIsInstance(t, int)
+
+                for type in (int, float, decimal.Decimal, datetime.datetime, datetime.timedelta):
+                    t = os.stat(self.fname, timestamp=type).st_mtime
+                    self.assertIsInstance(t, type)
+        finally:
+            os.stat_float_times(old_value)
+
     def test_statvfs_attributes(self):
         if not hasattr(os, "statvfs"):
             return
diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -1,10 +1,10 @@
 from test import support
+import locale
+import platform
+import sys
+import sysconfig
 import time
 import unittest
-import locale
-import sysconfig
-import sys
-import platform
 
 # Max year is only limited by the size of C int.
 SIZEOF_INT = sysconfig.get_config_var('SIZEOF_INT') or 4
@@ -331,6 +331,41 @@ class TimeTestCase(unittest.TestCase):
             pass
         self.assertEqual(time.strftime('%Z', tt), tzname)
 
+    def test_format(self):
+        import datetime, decimal
+        calls = [(time.time,)]
+        if hasattr(time, 'wallclock'):
+            calls.append((time.wallclock,))
+        if hasattr(time, 'CLOCK_REALTIME'):
+            if hasattr(time, 'clock_gettime'):
+                calls.append((time.clock_gettime, time.CLOCK_REALTIME))
+            if hasattr(time, 'clock_getres'):
+                calls.append((time.clock_getres, time.CLOCK_REALTIME))
+        for call in calls:
+            func, *args = call
+
+            # test invalid format
+            self.assertRaises(ValueError, func, *args, format="abc")
+            self.assertRaises(ValueError, func, *args, format=decimal.Context)
+
+            # test float
+            timestamp = func(*args, format=float)
+            self.assertIsInstance(timestamp, float)
+
+            # test int
+            timestamp = func(*args, format=int)
+            self.assertIsInstance(timestamp, int)
+
+            # test decimal
+            timestamp = func(*args, format=decimal.Decimal)
+            self.assertIsInstance(timestamp, decimal.Decimal)
+
+            # test datetime
+            timestamp = func(*args, format=datetime.datetime)
+            self.assertIsInstance(timestamp, datetime.datetime)
+            timestamp = func(*args, format=datetime.timedelta)
+            self.assertIsInstance(timestamp, datetime.timedelta)
+
     def test_wallclock(self):
         t1 = time.wallclock()
         t2 = time.wallclock()
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -1708,9 +1708,12 @@ stat_float_times(PyObject* self, PyObjec
 }
 
 static void
-fill_time(PyObject *v, int index, time_t sec, unsigned long nsec)
+fill_time(PyObject *v, int index, time_t sec, unsigned long nsec,
+          PyObject *timestamp)
 {
     PyObject *fval,*ival;
+    _PyTime_t ts;
+
 #if SIZEOF_TIME_T > SIZEOF_LONG
     ival = PyLong_FromLongLong((PY_LONG_LONG)sec);
 #else
@@ -1718,9 +1721,15 @@ fill_time(PyObject *v, int index, time_t
 #endif
     if (!ival)
         return;
-    if (_stat_float_times) {
-        fval = PyFloat_FromDouble(sec + 1e-9*nsec);
-    } else {
+    if (timestamp == NULL && _stat_float_times)
+        timestamp = (PyObject*)&PyFloat_Type;
+    if (timestamp != NULL) {
+        ts.seconds = sec;
+        ts.floatpart = nsec;
+        ts.divisor = 1000000000;
+        fval = _PyTime_AsFormat(&ts, timestamp);
+    }
+    else {
         fval = ival;
         Py_INCREF(fval);
     }
@@ -1731,7 +1740,7 @@ fill_time(PyObject *v, int index, time_t
 /* pack a system stat C structure into the Python stat tuple
    (used by posix_stat() and posix_fstat()) */
 static PyObject*
-_pystat_fromstructstat(STRUCT_STAT *st)
+_pystat_fromstructstat(STRUCT_STAT *st, PyObject *timestamp)
 {
     unsigned long ansec, mnsec, cnsec;
     PyObject *v = PyStructSequence_New(&StatResultType);
@@ -1776,9 +1785,9 @@ _pystat_fromstructstat(STRUCT_STAT *st)
 #else
     ansec = mnsec = cnsec = 0;
 #endif
-    fill_time(v, 7, st->st_atime, ansec);
-    fill_time(v, 8, st->st_mtime, mnsec);
-    fill_time(v, 9, st->st_ctime, cnsec);
+    fill_time(v, 7, st->st_atime, ansec, timestamp);
+    fill_time(v, 8, st->st_mtime, mnsec, timestamp);
+    fill_time(v, 9, st->st_ctime, cnsec, timestamp);
 
 #ifdef HAVE_STRUCT_STAT_ST_BLKSIZE
     PyStructSequence_SET_ITEM(v, ST_BLKSIZE_IDX,
@@ -1829,7 +1838,7 @@ _pystat_fromstructstat(STRUCT_STAT *st)
 }
 
 static PyObject *
-posix_do_stat(PyObject *self, PyObject *args,
+posix_do_stat(PyObject *self, PyObject *args, PyObject *kw,
               char *format,
 #ifdef __VMS
               int (*statfunc)(const char *, STRUCT_STAT *, ...),
@@ -1839,15 +1848,18 @@ posix_do_stat(PyObject *self, PyObject *
               char *wformat,
               int (*wstatfunc)(const wchar_t *, STRUCT_STAT *))
 {
+    static char *kwlist[] = {"path", "timestamp", NULL};
     STRUCT_STAT st;
     PyObject *opath;
     char *path;
     int res;
     PyObject *result;
+    PyObject *timestamp = NULL;
 
 #ifdef MS_WINDOWS
     PyObject *po;
-    if (PyArg_ParseTuple(args, wformat, &po)) {
+    if (PyArg_ParseTupleAndKeywords(args, kw, wformat, kwlist,
+                                    &po, &timestamp)) {
         wchar_t *wpath = PyUnicode_AsUnicode(po);
         if (wpath == NULL)
             return NULL;
@@ -1858,15 +1870,17 @@ posix_do_stat(PyObject *self, PyObject *
 
         if (res != 0)
             return win32_error_object("stat", po);
-        return _pystat_fromstructstat(&st);
+        return _pystat_fromstructstat(&st, timestamp);
     }
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-#endif
-
-    if (!PyArg_ParseTuple(args, format,
-                          PyUnicode_FSConverter, &opath))
+    timestamp = NULL;
+#endif
+
+    if (!PyArg_ParseTupleAndKeywords(args, kw, format, kwlist,
+                                     PyUnicode_FSConverter, &opath,
+                                     &timestamp))
         return NULL;
 #ifdef MS_WINDOWS
     if (win32_warn_bytes_api()) {
@@ -1887,7 +1901,7 @@ posix_do_stat(PyObject *self, PyObject *
 #endif
     }
     else
-        result = _pystat_fromstructstat(&st);
+        result = _pystat_fromstructstat(&st, timestamp);
 
     Py_DECREF(opath);
     return result;
@@ -3362,12 +3376,12 @@ PyDoc_STRVAR(posix_stat__doc__,
 Perform a stat system call on the given path.");
 
 static PyObject *
-posix_stat(PyObject *self, PyObject *args)
+posix_stat(PyObject *self, PyObject *args, PyObject *kw)
 {
 #ifdef MS_WINDOWS
-    return posix_do_stat(self, args, "O&:stat", STAT, "U:stat", win32_stat_w);
+    return posix_do_stat(self, args, kw, "O&|O:stat", STAT, "U:stat", win32_stat_w);
 #else
-    return posix_do_stat(self, args, "O&:stat", STAT, NULL, NULL);
+    return posix_do_stat(self, args, kw, "O&|O:stat", STAT, NULL, NULL);
 #endif
 }
 
@@ -6367,16 +6381,16 @@ PyDoc_STRVAR(posix_lstat__doc__,
 Like stat(path), but do not follow symbolic links.");
 
 static PyObject *
-posix_lstat(PyObject *self, PyObject *args)
+posix_lstat(PyObject *self, PyObject *args, PyObject *kw)
 {
 #ifdef HAVE_LSTAT
-    return posix_do_stat(self, args, "O&:lstat", lstat, NULL, NULL);
+    return posix_do_stat(self, args, kw, "O&|O:lstat", lstat, NULL, NULL);
 #else /* !HAVE_LSTAT */
 #ifdef MS_WINDOWS
-    return posix_do_stat(self, args, "O&:lstat", win32_lstat, "U:lstat",
+    return posix_do_stat(self, args, kw, "O&|O:lstat", win32_lstat, "U|O:lstat",
                          win32_lstat_w);
 #else
-    return posix_do_stat(self, args, "O&:lstat", STAT, NULL, NULL);
+    return posix_do_stat(self, args, "kw, O&|O:lstat", STAT, NULL, NULL);
 #endif
 #endif /* !HAVE_LSTAT */
 }
@@ -7339,12 +7353,15 @@ PyDoc_STRVAR(posix_fstat__doc__,
 Like stat(), but for an open file descriptor.");
 
 static PyObject *
-posix_fstat(PyObject *self, PyObject *args)
-{
+posix_fstat(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    static char *kwlist[] = {"fd", "timestamp", NULL};
     int fd;
     STRUCT_STAT st;
     int res;
-    if (!PyArg_ParseTuple(args, "i:fstat", &fd))
+    PyObject *timestamp = NULL;
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "i|O:fstat", kwlist,
+                                     &fd, &timestamp))
         return NULL;
 #ifdef __VMS
     /* on OpenVMS we must ensure that all bytes are written to the file */
@@ -7363,7 +7380,7 @@ posix_fstat(PyObject *self, PyObject *ar
 #endif
     }
 
-    return _pystat_fromstructstat(&st);
+    return _pystat_fromstructstat(&st, timestamp);
 }
 
 PyDoc_STRVAR(posix_isatty__doc__,
@@ -9654,15 +9671,18 @@ If path is relative and dirfd is the spe
 is interpreted relative to the current working directory.");
 
 static PyObject *
-posix_fstatat(PyObject *self, PyObject *args)
-{
+posix_fstatat(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    static char *kwlist[] = {"dirfd", "path", "flags", "timestamp", NULL};
     PyObject *opath;
     char *path;
     STRUCT_STAT st;
     int dirfd, res, flags = 0;
-
-    if (!PyArg_ParseTuple(args, "iO&|i:fstatat",
-            &dirfd, PyUnicode_FSConverter, &opath, &flags))
+    PyObject *timestamp = NULL;
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "iO&|iO:fstatat", kwlist,
+                                     &dirfd, PyUnicode_FSConverter, &opath,
+                                     &flags, &timestamp))
         return NULL;
     path = PyBytes_AsString(opath);
 
@@ -9673,7 +9693,7 @@ posix_fstatat(PyObject *self, PyObject *
     if (res != 0)
         return posix_error();
 
-    return _pystat_fromstructstat(&st);
+    return _pystat_fromstructstat(&st, timestamp);
 }
 #endif
 
@@ -10537,7 +10557,7 @@ static PyMethodDef posix_methods[] = {
 #ifdef HAVE_FDOPENDIR
     {"fdlistdir",       posix_fdlistdir, METH_VARARGS, posix_fdlistdir__doc__},
 #endif
-    {"lstat",           posix_lstat, METH_VARARGS, posix_lstat__doc__},
+    {"lstat",           (PyCFunction)posix_lstat, METH_VARARGS | METH_KEYWORDS, posix_lstat__doc__},
     {"mkdir",           posix_mkdir, METH_VARARGS, posix_mkdir__doc__},
 #ifdef HAVE_NICE
     {"nice",            posix_nice, METH_VARARGS, posix_nice__doc__},
@@ -10556,7 +10576,7 @@ static PyMethodDef posix_methods[] = {
 #endif /* !defined(HAVE_READLINK) && defined(MS_WINDOWS) */
     {"rename",          posix_rename, METH_VARARGS, posix_rename__doc__},
     {"rmdir",           posix_rmdir, METH_VARARGS, posix_rmdir__doc__},
-    {"stat",            posix_stat, METH_VARARGS, posix_stat__doc__},
+    {"stat",            (PyCFunction)posix_stat, METH_VARARGS | METH_KEYWORDS, posix_stat__doc__},
     {"stat_float_times", stat_float_times, METH_VARARGS, stat_float_times__doc__},
 #if defined(HAVE_SYMLINK) && !defined(MS_WINDOWS)
     {"symlink",         posix_symlink, METH_VARARGS, posix_symlink__doc__},
@@ -10771,7 +10791,8 @@ static PyMethodDef posix_methods[] = {
     {"sendfile",        (PyCFunction)posix_sendfile, METH_VARARGS | METH_KEYWORDS,
                             posix_sendfile__doc__},
 #endif
-    {"fstat",           posix_fstat, METH_VARARGS, posix_fstat__doc__},
+    {"fstat",           (PyCFunction)posix_fstat, METH_VARARGS | METH_KEYWORDS,
+                            posix_fstat__doc__},
     {"isatty",          posix_isatty, METH_VARARGS, posix_isatty__doc__},
 #ifdef HAVE_PIPE
     {"pipe",            posix_pipe, METH_NOARGS, posix_pipe__doc__},
@@ -10906,7 +10927,8 @@ static PyMethodDef posix_methods[] = {
     {"fchownat",        posix_fchownat, METH_VARARGS, posix_fchownat__doc__},
 #endif /* HAVE_FCHOWNAT */
 #ifdef HAVE_FSTATAT
-    {"fstatat",         posix_fstatat, METH_VARARGS, posix_fstatat__doc__},
+    {"fstatat",         (PyCFunction)posix_fstatat, METH_VARARGS | METH_KEYWORDS,
+                            posix_fstatat__doc__},
 #endif
 #ifdef HAVE_FUTIMESAT
     {"futimesat",       posix_futimesat, METH_VARARGS, posix_futimesat__doc__},
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -40,20 +40,26 @@
 #include <sys/time.h>
 #endif
 
+#if (defined(MS_WINDOWS) && !defined(__BORLANDC__)) || defined(HAVE_CLOCK)
+#  define HAVE_PYCLOCK
+#endif
+
 /* Forward declarations */
 static int floatsleep(double);
-static double floattime(void);
 
 static PyObject *
-time_time(PyObject *self, PyObject *unused)
+time_time(PyObject *self, PyObject *args, PyObject *kwargs)
 {
-    double secs;
-    secs = floattime();
-    if (secs == 0.0) {
-        PyErr_SetFromErrno(PyExc_IOError);
+    static char *kwlist[] = {"format", NULL};
+    PyObject *format = NULL;
+    _PyTime_t ts;
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|O:time", kwlist,
+                                     &format))
         return NULL;
-    }
-    return PyFloat_FromDouble(secs);
+
+    _PyTime_get(&ts);
+    return _PyTime_AsFormat(&ts, format);
 }
 
 PyDoc_STRVAR(time_doc,
@@ -72,55 +78,80 @@ Fractions of a second may be present if 
 #endif
 #endif
 
-static PyObject *
-pyclock(void)
+static int
+pyclock(_PyTime_t *ts)
 {
-    clock_t value;
-    value = clock();
-    if (value == (clock_t)-1) {
+    clock_t processor_time;
+    processor_time = clock();
+    if (processor_time == (clock_t)-1) {
         PyErr_SetString(PyExc_RuntimeError,
                 "the processor time used is not available "
                 "or its value cannot be represented");
-        return NULL;
+        return -1;
     }
-    return PyFloat_FromDouble((double)value / CLOCKS_PER_SEC);
+    ts->seconds = 0;
+    assert(sizeof(clock_t) <= sizeof(size_t));
+    ts->floatpart = Py_SAFE_DOWNCAST(processor_time, clock_t, size_t);
+    ts->divisor = CLOCKS_PER_SEC;
+    return 0;
 }
 #endif /* HAVE_CLOCK */
 
 #if defined(MS_WINDOWS) && !defined(__BORLANDC__)
 /* Win32 has better clock replacement; we have our own version, due to Mark
    Hammond and Tim Peters */
-static PyObject *
-time_clock(PyObject *self, PyObject *unused)
+static int
+win32_pyclock(_PyTime_t *ts)
 {
     static LARGE_INTEGER ctrStart;
-    static double divisor = 0.0;
+    static LONGLONG cpu_frequency = 0;
     LARGE_INTEGER now;
-    double diff;
+    LONGLONG dt;
 
-    if (divisor == 0.0) {
+    if (cpu_frequency == 0) {
         LARGE_INTEGER freq;
         QueryPerformanceCounter(&ctrStart);
-        if (!QueryPerformanceFrequency(&freq) || freq.QuadPart == 0) {
+        if (!QueryPerformanceFrequency(&freq) || freq.QuadPart == 0)
+        {
             /* Unlikely to happen - this works on all intel
                machines at least!  Revert to clock() */
-            return pyclock();
+            return pyclock(ts);
         }
-        divisor = (double)freq.QuadPart;
+        cpu_frequency = freq.QuadPart;
     }
     QueryPerformanceCounter(&now);
-    diff = (double)(now.QuadPart - ctrStart.QuadPart);
-    return PyFloat_FromDouble(diff / divisor);
+    dt = now.QuadPart - ctrStart.QuadPart;
+
+    ts->seconds = 0;
+    assert(sizeof(LONGLONG) <= sizeof(size_t));
+    ts->floatpart = Py_SAFE_DOWNCAST(dt, LONGLONG, size_t);
+    ts->divisor = Py_SAFE_DOWNCAST(cpu_frequency, LONGLONG, size_t);
+    return 0;
 }
+#endif
 
-#elif defined(HAVE_CLOCK)
+#ifdef HAVE_PYCLOCK
+static PyObject *
+time_clock(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    static char *kwlist[] = {"format", NULL};
+    _PyTime_t ts;
+    PyObject *format = NULL;
 
-static PyObject *
-time_clock(PyObject *self, PyObject *unused)
-{
-    return pyclock();
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|O:clock", kwlist,
+                                     &format))
+        return NULL;
+
+#if defined(MS_WINDOWS) && !defined(__BORLANDC__)
+    if (win32_pyclock(&ts))
+        return NULL;
+#else
+    if (pyclock(&ts))
+        return NULL;
+#endif
+    return _PyTime_AsFormat(&ts, format);
 }
-#endif /* HAVE_CLOCK */
+#endif
 
 
 #ifdef HAVE_CLOCK
@@ -134,13 +165,17 @@ records.");
 
 #ifdef HAVE_CLOCK_GETTIME
 static PyObject *
-time_clock_gettime(PyObject *self, PyObject *args)
+time_clock_gettime(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"clk_id", "format", NULL};
+    PyObject *format = NULL;
     int ret;
     clockid_t clk_id;
     struct timespec tp;
+    _PyTime_t ts;
 
-    if (!PyArg_ParseTuple(args, "i:clock_gettime", &clk_id))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "i|O:clock_gettime", kwlist,
+                                     &clk_id, &format))
         return NULL;
 
     ret = clock_gettime((clockid_t)clk_id, &tp);
@@ -148,8 +183,10 @@ time_clock_gettime(PyObject *self, PyObj
         PyErr_SetFromErrno(PyExc_IOError);
         return NULL;
     }
-
-    return PyFloat_FromDouble(tp.tv_sec + tp.tv_nsec * 1e-9);
+    ts.seconds = tp.tv_sec;
+    ts.floatpart = tp.tv_nsec;
+    ts.divisor = 1000000000;
+    return _PyTime_AsFormat(&ts, format);
 }
 
 PyDoc_STRVAR(clock_gettime_doc,
@@ -160,13 +197,17 @@ Return the time of the specified clock c
 
 #ifdef HAVE_CLOCK_GETRES
 static PyObject *
-time_clock_getres(PyObject *self, PyObject *args)
+time_clock_getres(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"clk_id", "format", NULL};
+    PyObject *format = NULL;
     int ret;
     clockid_t clk_id;
     struct timespec tp;
+    _PyTime_t ts;
 
-    if (!PyArg_ParseTuple(args, "i:clock_getres", &clk_id))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "i|O:clock_getres", kwlist,
+                                     &clk_id, &format))
         return NULL;
 
     ret = clock_getres((clockid_t)clk_id, &tp);
@@ -174,8 +215,10 @@ time_clock_getres(PyObject *self, PyObje
         PyErr_SetFromErrno(PyExc_IOError);
         return NULL;
     }
-
-    return PyFloat_FromDouble(tp.tv_sec + tp.tv_nsec * 1e-9);
+    ts.seconds = tp.tv_sec;
+    ts.floatpart = tp.tv_nsec;
+    ts.divisor = 1000000000;
+    return _PyTime_AsFormat(&ts, format);
 }
 
 PyDoc_STRVAR(clock_getres_doc,
@@ -763,12 +806,14 @@ the local timezone used by methods such 
 should not be relied on.");
 #endif /* HAVE_WORKING_TZSET */
 
-static PyObject *
-time_wallclock(PyObject *self, PyObject *unused)
+static int
+pywallclock(_PyTime_t *ts)
 {
 #if defined(MS_WINDOWS) && !defined(__BORLANDC__)
-    return time_clock(self, NULL);
-#elif defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC)
+    return win32_pyclock(ts);
+#else
+
+#if defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC)
     static int clk_index = 0;
     clockid_t clk_ids[] = {
 #ifdef CLOCK_MONOTONIC_RAW
@@ -788,18 +833,38 @@ time_wallclock(PyObject *self, PyObject 
         clockid_t clk_id = clk_ids[clk_index];
         ret = clock_gettime(clk_id, &tp);
         if (ret == 0)
-            return PyFloat_FromDouble(tp.tv_sec + tp.tv_nsec * 1e-9);
+        {
+            ts->seconds = tp.tv_sec;
+            ts->floatpart = tp.tv_nsec;
+            ts->divisor = 1000000000;
+            return 0;
+        }
 
         clk_index++;
         if (Py_ARRAY_LENGTH(clk_ids) <= clk_index)
             clk_index = -1;
     }
-    return time_time(self, NULL);
-#else
-    return time_time(self, NULL);
+#endif /* defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) */
+    _PyTime_get(ts);
+    return 0;
 #endif
 }
 
+static PyObject *
+time_wallclock(PyObject *self, PyObject *args, PyObject *kwargs)
+{
+    static char *kwlist[] = {"format", NULL};
+    PyObject *format = NULL;
+    _PyTime_t ts;
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|O:wallclock", kwlist,
+                                     &format))
+        return NULL;
+    if (pywallclock(&ts))
+        return NULL;
+    return _PyTime_AsFormat(&ts, format);
+}
+
 PyDoc_STRVAR(wallclock_doc,
 "wallclock() -> float\n\
 \n\
@@ -919,15 +984,17 @@ PyInit_timezone(PyObject *m) {
 
 
 static PyMethodDef time_methods[] = {
-    {"time",            time_time, METH_NOARGS, time_doc},
-#if (defined(MS_WINDOWS) && !defined(__BORLANDC__)) || defined(HAVE_CLOCK)
-    {"clock",           time_clock, METH_NOARGS, clock_doc},
+    {"time",            (PyCFunction)time_time, METH_VARARGS | METH_KEYWORDS, time_doc},
+#ifdef HAVE_PYCLOCK
+    {"clock",           (PyCFunction)time_clock, METH_VARARGS | METH_KEYWORDS, clock_doc},
 #endif
 #ifdef HAVE_CLOCK_GETTIME
-    {"clock_gettime",   time_clock_gettime, METH_VARARGS, clock_gettime_doc},
+    {"clock_gettime",   (PyCFunction)time_clock_gettime,
+                            METH_VARARGS | METH_KEYWORDS, clock_gettime_doc},
 #endif
 #ifdef HAVE_CLOCK_GETRES
-    {"clock_getres",    time_clock_getres, METH_VARARGS, clock_getres_doc},
+    {"clock_getres",    (PyCFunction)time_clock_getres,
+                            METH_VARARGS | METH_KEYWORDS, clock_getres_doc},
 #endif
     {"sleep",           time_sleep, METH_VARARGS, sleep_doc},
     {"gmtime",          time_gmtime, METH_VARARGS, gmtime_doc},
@@ -944,7 +1011,8 @@ static PyMethodDef time_methods[] = {
 #ifdef HAVE_WORKING_TZSET
     {"tzset",           time_tzset, METH_NOARGS, tzset_doc},
 #endif
-    {"wallclock",       time_wallclock, METH_NOARGS, wallclock_doc},
+    {"wallclock",       (PyCFunction)time_wallclock,
+                            METH_VARARGS | METH_KEYWORDS, wallclock_doc},
     {NULL,              NULL}           /* sentinel */
 };
 
@@ -1029,15 +1097,6 @@ PyInit_time(void)
     return m;
 }
 
-static double
-floattime(void)
-{
-    _PyTime_timeval t;
-    _PyTime_gettimeofday(&t);
-    return (double)t.tv_sec + t.tv_usec*0.000001;
-}
-
-
 /* Implement floatsleep() for various platforms.
    When interrupted (or when another error occurs), return -1 and
    set an exception; else return 0. */
diff --git a/Python/pytime.c b/Python/pytime.c
--- a/Python/pytime.c
+++ b/Python/pytime.c
@@ -19,8 +19,14 @@ extern int ftime(struct timeb *);
 #endif /* MS_WINDOWS */
 #endif /* HAVE_FTIME */
 
+#define MILLISECONDS       1000
+#define MICROSECONDS    1000000
+#define NANOSECONDS  1000000000
+
+static PyObject* datetime_module = NULL;
+
 void
-_PyTime_gettimeofday(_PyTime_timeval *tp)
+_PyTime_get(_PyTime_t *ts)
 {
     /* There are three ways to get the time:
       (1) gettimeofday() -- resolution in microseconds
@@ -30,27 +36,409 @@ _PyTime_gettimeofday(_PyTime_timeval *tp
       Since on some systems (e.g. SCO ODT 3.0) gettimeofday() may
       fail, so we fall back on ftime() or time().
       Note: clock resolution does not imply clock accuracy! */
+
+#ifdef HAVE_GETTIMEOFDAY
+    struct timeval tv;
+    int err;
+#endif
+#if defined(HAVE_FTIME)
+    struct timeb t;
+#endif
+
 #ifdef HAVE_GETTIMEOFDAY
 #ifdef GETTIMEOFDAY_NO_TZ
-    if (gettimeofday(tp) == 0)
+    err = gettimeofday(&tv);
+#else /* !GETTIMEOFDAY_NO_TZ */
+    err = gettimeofday(&tv, (struct timezone *)NULL);
+#endif /* !GETTIMEOFDAY_NO_TZ */
+    if (err == 0)
+    {
+        ts->seconds = tv.tv_sec;
+        ts->floatpart = tv.tv_usec;
+        ts->divisor = MICROSECONDS;
         return;
-#else /* !GETTIMEOFDAY_NO_TZ */
-    if (gettimeofday(tp, (struct timezone *)NULL) == 0)
-        return;
-#endif /* !GETTIMEOFDAY_NO_TZ */
+    }
 #endif /* !HAVE_GETTIMEOFDAY */
+
 #if defined(HAVE_FTIME)
+    ftime(&t);
+    ts->seconds = t.time;
+    ts->floatpart = t.millitm;
+    ts->divisor = MILLISECONDS;
+#else /* !HAVE_FTIME */
+    ts->seconds = time(NULL);
+    ts->floatpart = 0;
+    ts->divisor = 1;
+#endif /* !HAVE_FTIME */
+}
+
+void
+_PyTime_gettimeofday(_PyTime_timeval *tv)
+{
+    _PyTime_t ts;
+    size_t k;
+
+    _PyTime_get(&ts);
+    tv->tv_sec = ts.seconds;
+    if (ts.floatpart) {
+        if (ts.floatpart > ts.divisor) {
+            tv->tv_sec += ts.floatpart / ts.divisor;
+            ts.floatpart = ts.floatpart % ts.divisor;
+        }
+        if (MICROSECONDS >= ts.divisor) {
+            k = MICROSECONDS / ts.divisor;
+            tv->tv_usec = ts.floatpart * k;
+        }
+        else {
+            k = ts.divisor / MICROSECONDS;
+            tv->tv_usec = ts.floatpart / k;
+        }
+    }
+    else {
+        tv->tv_usec = 0;
+    }
+}
+
+static PyObject*
+PyLong_FromTime_t(time_t value)
+{
+#ifdef SIZEOF_TIME_T <= SIZEOF_LONG
+    return PyLong_FromLong(value);
+#else
+    assert(sizeof(time_t) <= sizeof(PY_LONG_LONG));
+    return PyLong_FromLongLong(value);
+#endif
+}
+
+/* Convert a timestamp to a PyFloat object */
+static PyObject*
+_PyTime_AsFloat(_PyTime_t *ts)
+{
+    double d;
+    d = (double)ts->seconds;
+    d += (double)ts->floatpart / (double)ts->divisor;
+    return PyFloat_FromDouble(d);
+}
+
+/* Convert a timestamp to a PyLong object */
+static PyObject*
+_PyTime_AsLong(_PyTime_t *ts)
+{
+    PyObject *a, *b, *c;
+
+    a = PyLong_FromTime_t(ts->seconds);
+    if (a == NULL)
+        return NULL;
+    b = PyLong_FromSize_t(ts->floatpart / ts->divisor);
+    if (b == NULL)
     {
-        struct timeb t;
-        ftime(&t);
-        tp->tv_sec = t.time;
-        tp->tv_usec = t.millitm * 1000;
+        Py_DECREF(a);
+        return NULL;
     }
-#else /* !HAVE_FTIME */
-    tp->tv_sec = time(NULL);
-    tp->tv_usec = 0;
-#endif /* !HAVE_FTIME */
-    return;
+    c = PyNumber_Add(a, b);
+    Py_DECREF(a);
+    Py_DECREF(b);
+    return c;
+}
+
+/* Convert a timestamp to a decimal.Decimal object */
+static PyObject*
+_PyTime_AsDecimal(_PyTime_t *ts)
+{
+    static PyObject* module = NULL;
+    static PyObject* decimal = NULL;
+    static PyObject* context = NULL;
+    /* resolution cache (1/divisor), dictionary of int (divisor)=>Decimal */
+    static PyObject* exponents = NULL;
+    PyObject *t = NULL;
+    PyObject *key, *exponent, *quantized;
+    _Py_IDENTIFIER(quantize);
+    _Py_IDENTIFIER(__truediv__);
+
+    if (!module) {
+        module = PyImport_ImportModuleNoBlock("decimal");
+        if (module == NULL)
+            return NULL;
+    }
+
+    if (!decimal) {
+        decimal = PyObject_GetAttrString(module, "Decimal");
+        if (decimal == NULL)
+            return NULL;
+    }
+
+    if (context == NULL)
+    {
+        /* context = decimal.Context(22, rounding=decimal.ROUND_HALF_EVEN) */
+        PyObject *context_class, *rounding;
+        /* Use 12 decimal digits to store 10,000 years in seconds + 9
+           decimal digits for the floating part in nanoseconds + 1 decimal
+           digit to round correctly */
+        context_class = PyObject_GetAttrString(module, "Context");
+        if (context_class == NULL)
+            return NULL;
+        rounding = PyObject_GetAttrString(module, "ROUND_HALF_EVEN");
+        if (rounding == NULL)
+        {
+            Py_DECREF(context_class);
+            return NULL;
+        }
+        context = PyObject_CallFunction(context_class, "iO", 22, rounding);
+        Py_DECREF(context_class);
+        Py_DECREF(rounding);
+        if (context == NULL)
+            return NULL;
+    }
+
+    /* t = decimal.Decimal(value, context) */
+    if (ts->seconds) {
+        PyObject *f = PyLong_FromTime_t(ts->seconds);
+        t = PyObject_CallFunction(decimal, "OO", f, context);
+        Py_CLEAR(f);
+    }
+    else {
+        t = PyObject_CallFunction(decimal, "iO", 0, context);
+    }
+    if (t == NULL)
+        return NULL;
+
+    if (ts->floatpart)
+    {
+        /* t += decimal.Decimal(floatpart, ctx) / decimal.Decimal(divisor, ctx) */
+        PyObject *a, *b, *c, *d, *x;
+
+        x = PyLong_FromSize_t(ts->floatpart);
+        if (x == NULL)
+            goto error;
+        a = PyObject_CallFunction(decimal, "OO", x, context);
+        Py_CLEAR(x);
+        if (a == NULL)
+            goto error;
+
+        x = PyLong_FromSize_t(ts->divisor);
+        if (x == NULL)
+        {
+            Py_DECREF(a);
+            goto error;
+        }
+        b = PyObject_CallFunction(decimal, "OO", x, context);
+        Py_CLEAR(x);
+        if (b == NULL)
+        {
+            Py_DECREF(a);
+            goto error;
+        }
+
+        c = _PyObject_CallMethodId(a, &PyId___truediv__, "OO",
+                                   b, context);
+        Py_DECREF(a);
+        Py_DECREF(b);
+        if (c == NULL)
+            goto error;
+
+        d = PyNumber_Add(t, c);
+        Py_DECREF(c);
+        if (d == NULL)
+            goto error;
+        Py_DECREF(t);
+        t = d;
+    }
+
+    if (exponents == NULL) {
+        exponents = PyDict_New();
+        if (exponents == NULL)
+            goto error;
+    }
+
+    key = PyLong_FromSize_t(ts->divisor);
+    if (key == NULL)
+        goto error;
+    exponent = PyDict_GetItem(exponents, key);
+    if (exponent == NULL) {
+        /* exponent = decimal.Decimal(1) / decimal.Decimal(resolution) */
+        PyObject *one, *divisor;
+
+        one = PyObject_CallFunction(decimal, "i", 1);
+        if (one == NULL) {
+            Py_DECREF(key);
+            goto error;
+        }
+
+        divisor = PyObject_CallFunction(decimal, "O", key);
+        if (divisor == NULL) {
+            Py_DECREF(key);
+            Py_DECREF(one);
+            goto error;
+        }
+
+        exponent = _PyObject_CallMethodId(one, &PyId___truediv__, "OO",
+                                          divisor, context);
+        Py_DECREF(one);
+        Py_DECREF(divisor);
+        if (exponent == NULL) {
+            Py_DECREF(key);
+            goto error;
+        }
+
+        if (PyDict_SetItem(exponents, key, exponent) < 0) {
+            Py_DECREF(key);
+            Py_DECREF(exponent);
+            goto error;
+        }
+        Py_DECREF(key);
+    }
+
+    /* t = t.quantize(exponent, None, context) */
+    quantized = _PyObject_CallMethodId(t, &PyId_quantize, "OOO",
+                                       exponent, Py_None, context);
+    if (quantized == NULL)
+        goto error;
+    Py_DECREF(t);
+    t = quantized;
+
+    return t;
+
+error:
+    Py_XDECREF(t);
+    return NULL;
+}
+
+/* Convert a timestamp to a datetime.datetime object */
+PyObject*
+_PyTime_AsDatetime(_PyTime_t *ts)
+{
+    static PyObject* datetime = NULL;
+    _Py_IDENTIFIER(fromtimestamp);
+    PyObject *decimal, *result;
+
+    if (!datetime_module) {
+        datetime_module = PyImport_ImportModuleNoBlock("datetime");
+        if (datetime_module == NULL)
+            return NULL;
+    }
+
+    if (!datetime) {
+        datetime = PyObject_GetAttrString(datetime_module, "datetime");
+        if (datetime == NULL)
+            return NULL;
+    }
+
+    decimal = _PyTime_AsDecimal(ts);
+    if (decimal == NULL)
+        return NULL;
+    result = _PyObject_CallMethodId(datetime, &PyId_fromtimestamp,
+                                    "O", decimal);
+    Py_DECREF(decimal);
+    return result;
+}
+
+/* Convert a timestamp to a datetime.timedelta object */
+PyObject*
+_PyTime_AsTimedelta(_PyTime_t *ts)
+{
+    static PyObject* timedelta = NULL;
+    unsigned long microseconds;
+    PyObject *seconds_obj, *microseconds_obj, *result;
+
+    if (!datetime_module) {
+        datetime_module = PyImport_ImportModuleNoBlock("datetime");
+        if (datetime_module == NULL)
+            return NULL;
+    }
+
+    if (!timedelta) {
+        timedelta = PyObject_GetAttrString(datetime_module, "timedelta");
+        if (timedelta == NULL)
+            return NULL;
+    }
+
+    seconds_obj = PyLong_FromTime_t(ts->seconds);
+    if (seconds_obj == NULL)
+        return NULL;
+    if (ts->floatpart) {
+        size_t k;
+        if (ts->floatpart > ts->divisor) {
+            PyObject *b, *c;
+            b = PyLong_FromSize_t(ts->floatpart / ts->divisor);
+            ts->floatpart = ts->floatpart % ts->divisor;
+            if (b == NULL) {
+                Py_DECREF(seconds_obj);
+                return NULL;
+            }
+            c = PyNumber_Add(seconds_obj, b);
+            Py_DECREF(seconds_obj);
+            Py_DECREF(b);
+            seconds_obj = c;
+        }
+        if (MICROSECONDS >= ts->divisor) {
+            k = MICROSECONDS / ts->divisor;
+            microseconds = Py_SAFE_DOWNCAST(ts->floatpart * k,
+                                            size_t, unsigned long);
+        }
+        else {
+            k = ts->divisor / MICROSECONDS;
+            microseconds = Py_SAFE_DOWNCAST(ts->floatpart / k,
+                                            size_t, unsigned long);
+        }
+    }
+    else {
+        microseconds = 0;
+    }
+    microseconds_obj = PyLong_FromSize_t(microseconds);
+    if (microseconds_obj == NULL) {
+        Py_DECREF(seconds_obj);
+        return NULL;
+    }
+    result = PyObject_CallFunction(timedelta, "iOO",
+                                   0, seconds_obj, microseconds_obj);
+    Py_DECREF(seconds_obj);
+    Py_DECREF(microseconds_obj);
+    return result;
+}
+
+/* Convert a timestamp to the specified format.
+
+   The result type depends on the requested format.
+
+   Raise a ValueError if the format is unknown. */
+PyObject*
+_PyTime_AsFormat(_PyTime_t *ts, PyObject *format)
+{
+    assert(ts->divisor != 0);
+
+    if (format == NULL || (PyTypeObject *)format == &PyFloat_Type)
+        return _PyTime_AsFloat(ts);
+    if ((PyTypeObject *)format == &PyLong_Type)
+        return _PyTime_AsLong(ts);
+
+    if (PyType_Check(format))
+    {
+        PyObject *module, *name;
+        _Py_IDENTIFIER(__name__);
+        _Py_IDENTIFIER(__module__);
+
+        module = _PyObject_GetAttrId(format, &PyId___module__);
+        name = _PyObject_GetAttrId(format, &PyId___name__);
+        if (module != NULL && PyUnicode_Check(module)
+            && name != NULL && PyUnicode_Check(name))
+        {
+            if (PyUnicode_CompareWithASCIIString(module, "decimal") == 0
+                && PyUnicode_CompareWithASCIIString(name, "Decimal") == 0)
+                return _PyTime_AsDecimal(ts);
+            if (PyUnicode_CompareWithASCIIString(module, "datetime") == 0)
+            {
+                if (PyUnicode_CompareWithASCIIString(name, "datetime") == 0)
+                    return _PyTime_AsDatetime(ts);
+                if (PyUnicode_CompareWithASCIIString(name, "timedelta") == 0)
+                    return _PyTime_AsTimedelta(ts);
+            }
+        }
+        else
+            PyErr_Clear();
+    }
+
+    PyErr_Format(PyExc_ValueError, "Unknown timestamp format: %R", format);
+    return NULL;
 }
 
 void

From report at bugs.python.org  Tue Jan 31 13:09:00 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 31 Jan 2012 12:09:00 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1328011740.73.0.407829463509.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Hmm, given the various *at() APIs that sort alphabetically next to their path based counterparts, perhaps we can make the naming consistency change go the other way? (i.e. listdirfd() and walkfd()). Even if POSIX puts the fd at the front, do we really have to follow them in a silly naming scheme?

And as per the python-dev discussion, +1 for being consistent with os.walk() when it comes to symlinks to directories.

Aside from the naming question, is there anything else holding this up?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:27:16 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 31 Jan 2012 12:27:16 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1328012836.57.0.84162505274.issue13734@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

Taking a closer look at the current naming scheme in the os module, fdopen() appears to be the only current function that uses the 'fd' prefix. All the other operations that accept a file descriptor just use 'f' as the prefix (fchmod, fchown, faccess, etc).

Given that, flistdir() and fwalk() seem like the most consistent choices of name for APIs that aren't directly matching an underlying POSIX function name.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:31:47 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 12:31:47 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1328013107.93.0.811471967475.issue13734@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

There's something I don't understand in the patch: why does _are_same_file examine st_mode?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:53:57 2012
From: report at bugs.python.org (Nick Coghlan)
Date: Tue, 31 Jan 2012 12:53:57 +0000
Subject: [issue13229] Improve tools for iterating over filesystem directories
In-Reply-To: <1319076330.7.0.757546324527.issue13229@psf.upfronthosting.co.za>
Message-ID: <1328014437.69.0.747039569657.issue13229@psf.upfronthosting.co.za>


Nick Coghlan <ncoghlan at gmail.com> added the comment:

With the release of 0.3, I'm pretty happy with the WalkDir design (previous versions coerced the output values to ordinary 3-tuples, now it will pass along whatever the underlying iterable produces with changing the type at all).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13229>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:55:33 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 12:55:33 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1328014533.04.0.464082635562.issue13903@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

As Victor I think the tunables could be changed separately, unless they truely get in the way of the shared-keys optimization.

By the way, there will need a "protection" for the case where instances are used as bags of key/value pairs (dict-alikes with attribute access). Perhaps bound the size of the keys array to 100 entries and then switch into unshared mode.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Tue Jan 31 13:57:52 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 12:57:52 +0000
Subject: [issue13229] Improve tools for iterating over filesystem directories
In-Reply-To: <1319076330.7.0.757546324527.issue13229@psf.upfronthosting.co.za>
Message-ID: <1328014672.34.0.565050581786.issue13229@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

(speaking of which, I'd just mention I've published pathlib as a standalone package: http://pypi.python.org/pypi/pathlib/ )

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13229>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:03:32 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 31 Jan 2012 13:03:32 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327535890.98.0.936666267948.issue13868@psf.upfronthosting.co.za>
Message-ID: <1328015012.57.0.150001332652.issue13868@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

What's?the?purpose?of?this?tracker?if?the?author?of?an?issue?report?has to?beg?for?something?to?be?fixed,?despite?the?fact?that (s)he?even made?a patch ?????

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:06:28 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 31 Jan 2012 13:06:28 +0000
Subject: =?utf-8?q?=5Bissue10580=5D_Minor_grammar_change_in_Python=E2=80=99s_MSI_i?=
	=?utf-8?q?nstaller?=
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1328015188.46.0.706182997194.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Rather apply the patch, not change the title.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:13:10 2012
From: report at bugs.python.org (Ezio Melotti)
Date: Tue, 31 Jan 2012 13:13:10 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1328015590.57.0.567266081962.issue13899@psf.upfronthosting.co.za>


Ezio Melotti <ezio.melotti at gmail.com> added the comment:

The rule 1 makes sense, but it's not entirely obvious (people might consider bBaAzZ special too).

The "normal Python rules for backslash escapes but revert to the C behaviour of stripping the \ from unrecognised escapes" is not obvious either, and from r'[\A]' people might expect:
  1) same as \A, (beginning of the string);
  2) a letter 'A';
  3) a '\' or a letter 'A' (especially if they write it as '[\\A]');

This is why I suggested to raise an error (and refuse the temptation to guess), but on the other hand, if you consider 'A' a "normal" letter like 'C', having an error for \A would be incoherent.
It would have been better if \C raised an error too (I don't see why that would appear in a regex, since re.escape doesn't escape C and the user has no reason to add the \), but now it's too late for that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:45:50 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 31 Jan 2012 13:45:50 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1328013107.93.0.811471967475.issue13734@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM2xr-wgzi0O7pYf5hNobCNPL2XVHdzH-x2q9zbkPJco7A@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

> Given that, flistdir() and fwalk() seem like the most consistent choices of name for APIs that aren't directly
> matching an underlying POSIX function name.

Well, that seems OK for me.
I guess the only reason fdlistdir() is named that way is because of
fdopendir(3).
I can make the change for fwalk(), and since 3.3 hasn't been released
yet, I guess we can rename fdlistdir() too.

> There's something I don't understand in the patch: why does _are_same_file examine st_mode?

It doesn't have to, that's actually useless.

The only thing that bothers me is that it needs O(height of directory
tree), so with really deep directory trees, we could run out of FDs.
Not sure that could be a problem in practice, but that's something to
keep in mind.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:51:37 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 13:51:37 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1328017897.73.0.0450166705029.issue13734@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

One other thing: is it deliberate to silence errors in the following snippet?

+        try:
+            names = fdlistdir(topfd)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 14:54:51 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 31 Jan 2012 13:54:51 +0000
Subject: [issue13903] New shared-keys dictionary implementation
In-Reply-To: <1327847206.97.0.499643364955.issue13903@psf.upfronthosting.co.za>
Message-ID: <1328018091.11.0.812238082078.issue13903@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13903>
_______________________________________

From report at bugs.python.org  Tue Jan 31 15:04:54 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 31 Jan 2012 14:04:54 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1325981643.51.0.923790442233.issue13734@psf.upfronthosting.co.za>
Message-ID: <1328018694.87.0.217712973532.issue13734@psf.upfronthosting.co.za>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

It's to mimic os.walk()'s behavior:

http://hg.python.org/cpython/file/bf31815548c9/Lib/os.py#l268

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 15:22:27 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Tue, 31 Jan 2012 14:22:27 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1328019747.05.0.0543783154098.issue13609@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

Terminal stuff is irrelevant to `shutil`, which is the module for 'High-level file operations' and deserve a separate module named 'console'.

Why? Because terminal size is only relevant when you want to page output. The next step would be to detect if terminal supports color to see if ANSI sequences will be collapsed. Then you'll want to get user input without pressing '<enter>' key.

It is a whole new module.

----------
nosy: +techtonik

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 15:31:24 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 14:31:24 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1328020284.04.0.399465459256.issue13609@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

"except Exception" clauses in the tests are too broad. Otherwise, looks fine.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 16:03:28 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 15:03:28 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1328020284.04.0.399465459256.issue13609@psf.upfronthosting.co.za>
Message-ID: <CAMpsgwYpRpLFZQUAw4GvhyOY0N8n8Y8Ta9out8M_NYJ0_F+d1w@mail.gmail.com>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

termsize.diff.7:
 - shutil.get_terminal_size(): I would prefer columns and lines
variable names instead of "ccc" and "rrr"
 - "Right now the values are not cached, but this might change." why
would it be cached? this sentence can just be removed
 - I would prefer os.get_terminal_size() instead of
os.query_terminal_size(), I didn't know other function using the verb
"query" in Python
 - To keep os simple, os.query_terminal_size() can return a simple,
whereas shutil.get_terminal_size() returns a namedtuple
 - test_os.py: use @unittest.skipUnless() on TermsizeTests to check if
os.query_terminal_size() is available
 - test.py, test_does_not_crash() catchs OSError, you may only ignore
some error codes, not any. For example, skip the test if stdout is not
a tty

+        try:
+            size = os.query_terminal_size(sys.__stdout__.fileno())
+        except (NameError, OSError):
+            size = os.terminal_size(fallback)

AttributeError should be catched here, not NameError. Or better, check
if os has a query_terminal_size() function.

+.. function:: get_terminal_size(fallback=(columns, lines))

Hum, you may document the default value: (80, 24).

shutil.get_terminal_size() doesn't allow to choose the file
descriptor. Would it be useful to allow to get the size of sys.stderr
or another TTY?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:01:05 2012
From: report at bugs.python.org (Brett Cannon)
Date: Tue, 31 Jan 2012 16:01:05 +0000
Subject: [issue13912] ImportError using __import__ and relative level 1
In-Reply-To: <1327966411.3.0.862648677538.issue13912@psf.upfronthosting.co.za>
Message-ID: <CAP1=2W72A_rwJstAtsHAWoXLfAWPcOexuEa54ocO454LrbcykA@mail.gmail.com>


Brett Cannon <brett at python.org> added the comment:

On Mon, Jan 30, 2012 at 18:33, Eric Snow <report at bugs.python.org> wrote:

>
> Eric Snow <ericsnowcurrently at gmail.com> added the comment:
>
> Jason: just a warning.  importlib_backport is a relatively naive tool for
> generating the backport from the py3k source.  It's also relatively fragile
> and at this point probably doesn't work with the default branch.
>

I think Jason was thinking of http://pypi.python.org/pypi/importlib/ which
is my personal backport of importlib from Python 2.7 all the way back to
Python 2.3.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13912>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:01:07 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 31 Jan 2012 16:01:07 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1328025667.71.0.412529262479.issue13496@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

I think the patch is *NOT* correct. Does it work with http://bugs.python.org/msg148567, in a 32 bits python?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:28:52 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 31 Jan 2012 16:28:52 +0000
Subject: [issue1625] bz2.BZ2File doesn't support multiple streams
In-Reply-To: <1197624030.23.0.503522059328.issue1625@psf.upfronthosting.co.za>
Message-ID: <1328027332.96.0.35763667831.issue1625@psf.upfronthosting.co.za>


Jes?s Cea Avi?n <jcea at jcea.es> added the comment:

?ric, bz2 module in Python is documented as able to manage bz2 files. But that is not true, since it is unable to manage popular bz2 files. That looks like a bug to me. In fact, you can create those files from python, files that python can not unpack later.

Moreover, Python 2.7 is going to live for a long time.

If the refusal of backporting this to 3.2 and 2.7 is firm, I would beg to document this limitation in the 2.7/3.2 docs. It is serious enough.

Please, reconsider backporting this to 2.7, at least.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1625>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:36:33 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Tue, 31 Jan 2012 16:36:33 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1328019747.05.0.0543783154098.issue13609@psf.upfronthosting.co.za>
Message-ID: <4F281887.2010906@in.waw.pl>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

Well, right now it's just one function. Functionality which you propose
could of course be useful, but let's leave if for another day, another 
issue.

See also http://bugs.python.org/issue13609#msg149627 and #444582 and 
#12442 -- shutil is growing in different directions, and terminal size 
can be one of them.

 > Antoine Pitrou added the comment:
 > "except Exception" clauses in the tests are too broad.
Fixed.
 > Otherwise, looks fine.

 > STINNER Victor added the comment:
 > - shutil.get_terminal_size(): I would prefer columns and lines
 > variable names instead of "ccc" and "rrr"
Changed (this was a leftover from the version when there was no named 
tuple).

 > - I would prefer os.get_terminal_size() instead of
 > os.query_terminal_size(), I didn't know other function using the verb
 > "query" in Python
Changed. I used os.g_t_s and shutil.g_t_s in docs so it should be clear 
which is which.

 > - To keep os simple, os.query_terminal_size() can return a simple,
 > whereas shutil.get_terminal_size() returns a namedtuple
We would have two very similar functions returning something different. 
Also, the amount of code saved will be negligible, because the named 
tuple is mostly docs. I prefer to keep it in os to keep both functions 
similar.

 > - test_os.py: use @unittest.skipUnless() on TermsizeTests to check if
 > os.query_terminal_size() is available
> - To keep os simple, os.query_terminal_size() can return a simple,
> whereas shutil.get_terminal_size() returns a namedtuple
> AttributeError should be catched here, not NameError. Or better, check
> if os has a query_terminal_size() function.
Fixed. I changed the tests a bit, e.g. to take into account that stty 
queries stdin, not stdout.

 > Hum, you may document the default value: (80, 24).
Done.

shutil.get_terminal_size() is tied to COLUMNS and ROWS and thus makes 
most sense for stdout. But I guess that an optional parameter could be 
added:
   shutil.get_terminal_size(fd=None, fallback=(80, 24))
where fd could be either an integer, or an object with .fileno().
I don't know.

 > - "Right now the values are not cached, but this might change." why
 > would it be cached? this sentence can just be removed
Done. In theory it could be cached with watching for SIGWINCH, but I'll 
just remove the comment for now.

Thanks for the review and comments!

Patch version nine attached: termsize.diff.8

----------
Added file: http://bugs.python.org/file24379/termsize.diff.8

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -1408,6 +1408,44 @@
    .. versionadded:: 3.3
 
 
+.. _terminal-size:
+
+Querying the size of the output terminal
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 3.3
+
+.. function:: get_terminal_size(fd=STDOUT_FILENO)
+
+   Return the size of the terminal window as ``(columns, lines)``,
+   tuple of type :class:`terminal_size`.
+
+   The optional argument ``fd`` (default ``STDOUT_FILENO``, or standard
+   output) specifies which file descriptor should be queried.
+
+   If the file descriptor is not connected to a terminal, an :exc:`OSError`
+   is thrown.
+
+   This function is only present if an implementation is available for
+   this system (currently POSIX and Windows).
+
+   :func:`shutil.get_terminal_size` is the high-level function which
+   should normally be used, ``os.get_terminal_size`` is the low-level
+   implementation.
+
+.. class:: terminal_size(tuple)
+
+   A tuple of ``(columns, lines)`` for holding terminal window size.
+
+   .. attribute:: columns
+
+      Width of the terminal window in characters.
+
+   .. attribute:: lines
+
+      Height of the terminal window in characters.
+
+
 .. _os-file-dir:
 
 Files and Directories
diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -450,3 +450,33 @@
     -rw-r--r-- tarek/staff   37192 2010-02-06 18:23:10 ./known_hosts
 
 
+Querying the size of the output terminal
+----------------------------------------
+
+.. versionadded:: 3.3
+
+.. function:: get_terminal_size(fallback=(columns, lines))
+
+   Get the size of the terminal window.
+
+   For each of the two dimensions, the environment variable, ``COLUMNS``
+   and ``LINES`` respectively, is checked. If the variable is defined and
+   the value is a positive integer, it is used.
+
+   When ``COLUMNS`` or ``LINES`` is not defined, which is the common case,
+   the terminal connected to :data:`sys.__stdout__` is queried
+   by invoking :func:`os.get_terminal_size`.
+
+   If the terminal size cannot be successfully queried, either because
+   the system doesn't support querying, or because we are not
+   connected to a terminal, the value given in ``fallback`` parameter
+   is used. ``fallback`` defaults to ``(80, 24)`` which is the default
+   size used by many terminal emulators.
+
+   The value returned is a named tuple of type :class:`os.terminal_size`.
+
+   See also: The Single UNIX Specification, Version 2,
+   `Other Environment Variables`_.
+
+.. _`Other Environment Variables`:
+   http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html#tag_002_003
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -469,6 +469,12 @@
 
   (Patch submitted by Giampaolo Rodol?? in :issue:`10784`.)
 
+* XXX The :mod:`shutil` module has a new function
+  :func:`~shutil.get_terminal_size` to check the size of output terminal
+  window.
+
+  (Patch submitted by Zbigniew J??drzejewski-Szmek in :issue:`13609`.)
+
 * "at" functions (:issue:`4761`):
 
   * :func:`~os.faccessat`
diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -872,3 +872,46 @@
             raise LookupError("no such group: {!r}".format(group))
 
     os.chown(path, _user, _group)
+
+def get_terminal_size(fallback=(80, 24)):
+    """Get the size of the terminal window.
+
+    For each of the two dimensions, the environment variable, COLUMNS
+    and LINES respectively, is checked. If the variable is defined and
+    the value is a positive integer, it is used.
+
+    When COLUMNS or LINES is not defined, which is the common case,
+    the terminal connected to sys.__stdout__ is queried
+    by invoking os.get_terminal_size.
+
+    If the terminal size cannot be successfully queried, either because
+    the system doesn't support querying, or because we are not
+    connected to a terminal, the value given in fallback parameter
+    is used. Fallback defaults to (80, 24) which is the default
+    size used by many terminal emulators.
+
+    The value returned is a named tuple of type os.terminal_size.
+    """
+    # columns, lines are the working values
+    try:
+        columns = int(os.environ['COLUMNS'])
+    except (KeyError, ValueError):
+        columns = 0
+
+    try:
+        lines = int(os.environ['LINES'])
+    except (KeyError, ValueError):
+        lines = 0
+
+    # only query if necessary
+    if columns <= 0 or lines <= 0:
+        try:
+            size = os.get_terminal_size(sys.__stdout__.fileno())
+        except (NameError, OSError):
+            size = os.terminal_size(fallback)
+        if columns <= 0:
+            columns = size.columns
+        if lines <= 0:
+            lines = size.lines
+
+    return os.terminal_size((columns, lines))
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -1676,6 +1676,43 @@
                               os.symlink, filename, filename)
 
 
+ at unittest.skipUnless(hasattr(os, 'get_terminal_size'), "requires os.get_terminal_size")
+class TermsizeTests(unittest.TestCase):
+    def test_does_not_crash(self):
+        """Check if get_terminal_size() returns a meaningful value.
+
+        There's no easy portable way to actually check the size of the
+        terminal, so let's check if it returns something sensible instead.
+        """
+        try:
+            size = os.get_terminal_size()
+        except OSError as e:
+            if e.errno == errno.EINVAL or sys.platform == "win32":
+                # Under win32 a generic OSError can be thrown if the
+                # handle cannot be retrieved
+                self.skipTest("failed to query terminal size")
+            raise
+
+        self.assertGreater(size.columns, 0)
+        self.assertGreater(size.lines, 0)
+
+    def test_stty_match(self):
+        """Check if stty returns the same results
+
+        stty actually tests stdin, so get_terminal_size is invoked on
+        stdin explicitly. If stty succeeded, then get_terminal_size()
+        should work too.
+        """
+        try:
+            size = subprocess.check_output(['stty', 'size']).decode().split()
+        except (FileNotFoundError, subprocess.CalledProcessError):
+            self.skipTest("stty invocation failed")
+        expected = (int(size[1]), int(size[0])) # reversed order
+
+        actual = os.get_terminal_size(sys.__stdin__.fileno())
+        self.assertEqual(expected, actual)
+
+
 @support.reap_threads
 def test_main():
     support.run_unittest(
@@ -1701,6 +1738,7 @@
         ProgramPriorityTests,
         ExtendedAttributeTests,
         Win32DeprecatedBytesAPI,
+        TermsizeTests,
     )
 
 if __name__ == "__main__":
diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py
--- a/Lib/test/test_shutil.py
+++ b/Lib/test/test_shutil.py
@@ -8,6 +8,7 @@
 import os
 import os.path
 import functools
+import subprocess
 from test import support
 from test.support import TESTFN
 from os.path import splitdrive
@@ -1223,10 +1224,55 @@
         finally:
             os.rmdir(dst_dir)
 
+class TermsizeTests(unittest.TestCase):
+    def test_does_not_crash(self):
+        """Check if get_terminal_size() returns a meaningful value.
+
+        There's no easy portable way to actually check the size of the
+        terminal, so let's check if it returns something sensible instead.
+        """
+        size = shutil.get_terminal_size()
+        self.assertGreater(size.columns, 0)
+        self.assertGreater(size.lines, 0)
+
+    def test_os_environ_first(self):
+        "Check if environment variables have precedence"
+
+        with support.EnvironmentVarGuard() as env:
+            env['COLUMNS'] = '777'
+            size = shutil.get_terminal_size()
+        self.assertEqual(size.columns, 777)
+
+        with support.EnvironmentVarGuard() as env:
+            env['LINES'] = '888'
+            size = shutil.get_terminal_size()
+        self.assertEqual(size.lines, 888)
+
+    @unittest.skipUnless(os.isatty(sys.__stdout__.fileno()), "not on tty")
+    def test_stty_match(self):
+        """Check if stty returns the same results ignoring env
+
+        This test will fail if stdin and stdout are connected to
+        different terminals with different sizes. Nevertheless, such
+        situations should be pretty rare.
+        """
+        try:
+            size = subprocess.check_output(['stty', 'size']).decode().split()
+        except (FileNotFoundError, subprocess.CalledProcessError):
+            self.skipTest("stty invocation failed")
+        expected = (int(size[1]), int(size[0])) # reversed order
+
+        with support.EnvironmentVarGuard() as env:
+            del env['LINES']
+            del env['COLUMNS']
+            actual = shutil.get_terminal_size()
+
+        self.assertEqual(expected, actual)
 
 
 def test_main():
-    support.run_unittest(TestShutil, TestMove, TestCopyFile)
+    support.run_unittest(TestShutil, TestMove, TestCopyFile,
+                         TermsizeTests)
 
 if __name__ == '__main__':
     test_main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -7289,6 +7289,9 @@
 
 - Windows locale mapping updated to Vista.
 
+- Issue #13609: shutil.get_terminal_size() and os.get_terminal_size()
+  functions are added to check the size of the terminal window.
+
 Tools/Demos
 -----------
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -125,6 +125,18 @@
 #include <dlfcn.h>
 #endif
 
+#if defined(MS_WINDOWS)
+#  define TERMSIZE_USE_CONIO
+#elif defined(HAVE_SYS_IOCTL_H)
+#  include <sys/ioctl.h>
+#  if defined(HAVE_TERMIOS_H)
+#    include <termios.h>
+#  endif
+#  if defined(TIOCGWINSZ)
+#    define TERMSIZE_USE_IOCTL
+#  endif
+#endif /* MS_WINDOWS */
+
 /* Various compilers have only certain posix functions */
 /* XXX Gosh I wish these were all moved into pyconfig.h */
 #if defined(PYCC_VACPP) && defined(PYOS_OS2)
@@ -10496,6 +10508,114 @@
 
 #endif /* USE_XATTRS */
 
+
+/* Terminal size querying */
+
+static PyTypeObject TerminalSizeType;
+
+PyDoc_STRVAR(TerminalSize_docstring,
+    "A tuple of (columns, lines) for holding terminal window size");
+
+static PyStructSequence_Field TerminalSize_fields[] = {
+    {"columns", "width of the terminal window in characters"},
+    {"lines", "height of the terminal window in characters"},
+    {NULL, NULL}
+};
+
+static PyStructSequence_Desc TerminalSize_desc = {
+    "os.terminal_size",
+    TerminalSize_docstring,
+    TerminalSize_fields,
+    2,
+};
+
+#if defined(TERMSIZE_USE_CONIO) || defined(TERMSIZE_USE_IOCTL)
+PyDoc_STRVAR(termsize__doc__,
+    "Return the size of the terminal window as (columns, lines).\n"        \
+    "\n"                                                                   \
+    "The optional argument fd (default standard output) specifies\n"       \
+    "which file descriptor should be queried.\n"                           \
+    "\n"                                                                   \
+    "If the file descriptor is not connected to a terminal, an OSError\n"  \
+    "is thrown.\n"                                                         \
+    "\n"                                                                   \
+    "This function will only be defined if an implementation is\n"         \
+    "available for this system.\n"                                         \
+    "\n"                                                                   \
+    "shutil.get_terminal_size is the high-level function which should \n"  \
+    "normally be used, os.get_terminal_size is the low-level implementation.");
+
+static PyObject*
+get_terminal_size(PyObject *self, PyObject *args)
+{
+    int columns, lines;
+    PyObject *termsize;
+
+    int fd = fileno(stdout);
+    /* Under some conditions stdout may not be connected and
+     * fileno(stdout) may point to an invalid file descriptor. For example
+     * GUI apps don't have valid standard streams by default.
+     *
+     * If this happens, and the optional fd argument is not present,
+     * the ioctl below will fail returning EBADF. This is what we want.
+     */
+
+    if (!PyArg_ParseTuple(args, "|i", &fd))
+        return NULL;
+
+#ifdef TERMSIZE_USE_IOCTL
+    {
+        struct winsize w;
+        if (ioctl(fd, TIOCGWINSZ, &w))
+            return PyErr_SetFromErrno(PyExc_OSError);
+        columns = w.ws_col;
+        lines = w.ws_row;
+    }
+#endif /* TERMSIZE_USE_IOCTL */
+
+#ifdef TERMSIZE_USE_CONIO
+    {
+        DWORD nhandle;
+        HANDLE handle;
+        CONSOLE_SCREEN_BUFFER_INFO csbi;
+        switch (fd) {
+        case 0: nhandle = STD_INPUT_HANDLE;
+            break;
+        case 1: nhandle = STD_OUTPUT_HANDLE;
+            break;
+        case 2: nhandle = STD_ERROR_HANDLE;
+            break;
+        default:
+            return PyErr_Format(PyExc_ValueError, "bad file descriptor");
+        }
+        handle = GetStdHandle(nhandle);
+        if (handle == NULL)
+            return PyErr_Format(PyExc_OSError, "handle cannot be retrieved");
+        if (handle == INVALID_HANDLE_VALUE)
+            return PyErr_SetFromWindowsErr(0);
+
+        if (!GetConsoleScreenBufferInfo(handle, &csbi))
+            return PyErr_SetFromWindowsErr(0);
+
+        columns = csbi.srWindow.Right - csbi.srWindow.Left + 1;
+        lines = csbi.srWindow.Bottom - csbi.srWindow.Top + 1;
+    }
+#endif /* TERMSIZE_USE_CONIO */
+
+    termsize = PyStructSequence_New(&TerminalSizeType);
+    if (termsize == NULL)
+        return NULL;
+    PyStructSequence_SET_ITEM(termsize, 0, PyLong_FromLong(columns));
+    PyStructSequence_SET_ITEM(termsize, 1, PyLong_FromLong(lines));
+    if (PyErr_Occurred()) {
+        Py_DECREF(termsize);
+        return NULL;
+    }
+    return termsize;
+}
+#endif /* defined(TERMSIZE_USE_CONIO) || defined(TERMSIZE_USE_IOCTL) */
+
+
 static PyMethodDef posix_methods[] = {
     {"access",          posix_access, METH_VARARGS, posix_access__doc__},
 #ifdef HAVE_TTYNAME
@@ -10963,6 +11083,9 @@
     {"llistxattr", posix_llistxattr, METH_VARARGS, posix_llistxattr__doc__},
     {"flistxattr", posix_flistxattr, METH_VARARGS, posix_flistxattr__doc__},
 #endif
+#if defined(TERMSIZE_USE_CONIO) || defined(TERMSIZE_USE_IOCTL)
+    {"get_terminal_size", get_terminal_size, METH_VARARGS, termsize__doc__},
+#endif
     {NULL,              NULL}            /* Sentinel */
 };
 
@@ -11557,6 +11680,10 @@
         PyStructSequence_InitType(&SchedParamType, &sched_param_desc);
         SchedParamType.tp_new = sched_param_new;
 #endif
+
+        /* initialize TerminalSize_info */
+        PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc);
+        Py_INCREF(&TerminalSizeType);
     }
 #if defined(HAVE_WAITID) && !defined(__APPLE__)
     Py_INCREF((PyObject*) &WaitidResultType);
@@ -11611,6 +11738,9 @@
 
 
 #endif /* __APPLE__ */
+
+    PyModule_AddObject(m, "terminal_size", (PyObject*) &TerminalSizeType);
+
     return m;
 
 }
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -6147,7 +6147,7 @@
 sys/audioio.h sys/xattr.h sys/bsdtty.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
 sys/param.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
-sys/stat.h sys/termio.h sys/time.h \
+sys/stat.h sys/termio.h sys/time.h sys/ioctl.h \
 sys/times.h sys/types.h sys/uio.h sys/un.h sys/utsname.h sys/wait.h pty.h \
 libutil.h sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \
 bluetooth/bluetooth.h linux/tipc.h spawn.h util.h
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -1337,7 +1337,7 @@
 sys/audioio.h sys/xattr.h sys/bsdtty.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
 sys/param.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
-sys/stat.h sys/termio.h sys/time.h \
+sys/stat.h sys/termio.h sys/time.h sys/ioctl.h \
 sys/times.h sys/types.h sys/uio.h sys/un.h sys/utsname.h sys/wait.h pty.h \
 libutil.h sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \
 bluetooth/bluetooth.h linux/tipc.h spawn.h util.h)
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -905,6 +905,9 @@
 /* Define to 1 if you have the <sys/file.h> header file. */
 #undef HAVE_SYS_FILE_H
 
+/* Define to 1 if you have the <sys/ioctl.h> header file. */
+#undef HAVE_SYS_IOCTL_H
+
 /* Define to 1 if you have the <sys/loadavg.h> header file. */
 #undef HAVE_SYS_LOADAVG_H
 

From report at bugs.python.org  Tue Jan 31 17:36:41 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 31 Jan 2012 16:36:41 +0000
Subject: [issue13897] Move fields relevant to coroutine/generators out
	of	frame into generator/threadstate
In-Reply-To: <1327767887.13.0.0676848614688.issue13897@psf.upfronthosting.co.za>
Message-ID: <1328027801.3.0.0541738272977.issue13897@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13897>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:39:18 2012
From: report at bugs.python.org (=?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?=)
Date: Tue, 31 Jan 2012 16:39:18 +0000
Subject: [issue13899] re pattern r"[\A]" should work like "A" but matches
	nothing. Ditto B and Z.
In-Reply-To: <1327789733.95.0.48339527102.issue13899@psf.upfronthosting.co.za>
Message-ID: <1328027958.37.0.575790483773.issue13899@psf.upfronthosting.co.za>


Changes by Jes?s Cea Avi?n <jcea at jcea.es>:


----------
nosy: +jcea

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13899>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:41:14 2012
From: report at bugs.python.org (Kang-Hao (Kenny) Lu)
Date: Tue, 31 Jan 2012 16:41:14 +0000
Subject: [issue7856] cannot decode from or encode to big5 \xf9\xd8
In-Reply-To: <1265346469.79.0.269704961777.issue7856@psf.upfronthosting.co.za>
Message-ID: <1328028074.75.0.730815134498.issue7856@psf.upfronthosting.co.za>


Changes by Kang-Hao (Kenny) Lu <kennyluck at csail.mit.edu>:


----------
nosy: +kennyluck

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7856>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:45:13 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Tue, 31 Jan 2012 16:45:13 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1323997298.98.0.00319779142763.issue13609@psf.upfronthosting.co.za>
Message-ID: <1328028313.57.0.48309181166.issue13609@psf.upfronthosting.co.za>


anatoly techtonik <techtonik at gmail.com> added the comment:

What happens with COLUMNS env variables if terminal is resized after the Python script is executed. Will get_terminal_size() return new value?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:48:32 2012
From: report at bugs.python.org (Zbyszek Szmek)
Date: Tue, 31 Jan 2012 16:48:32 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <1328028313.57.0.48309181166.issue13609@psf.upfronthosting.co.za>
Message-ID: <4F281B59.8050401@in.waw.pl>


Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:

> the Python script is executed. Will get_terminal_size()
 > return new value?
Please see previous discussion and the docs (and the SUSv2 specs).
The env. var. is for overriding.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 17:53:30 2012
From: report at bugs.python.org (anatoly techtonik)
Date: Tue, 31 Jan 2012 16:53:30 +0000
Subject: [issue13609] Add "os.get_terminal_size()" function
In-Reply-To: <4F281B59.8050401@in.waw.pl>
Message-ID: <CAPkN8xLLyhx+mC-+8uBtrEKrNr+Y6=0HohmrtDZT-87+nt_tNQ@mail.gmail.com>


anatoly techtonik <techtonik at gmail.com> added the comment:

On Tue, Jan 31, 2012 at 7:48 PM, Zbyszek Szmek <report at bugs.python.org>wrote:

>
> Zbyszek Szmek <zbyszek at in.waw.pl> added the comment:
>
> > the Python script is executed. Will get_terminal_size()
>  > return new value?
> Please see previous discussion and the docs (and the SUSv2 specs).
> The env. var. is for overriding.
>

Could you just answer the question or provide a relevant link
(unfortunately I don't have time to read SUSv2 specs)?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13609>
_______________________________________

From report at bugs.python.org  Tue Jan 31 18:27:56 2012
From: report at bugs.python.org (Kang-Hao (Kenny) Lu)
Date: Tue, 31 Jan 2012 17:27:56 +0000
Subject: [issue13913] utf-8 or utf8 or utf-8 (codec display name inconsistency)
Message-ID: <1328030876.68.0.627537179228.issue13913@psf.upfronthosting.co.za>


New submission from Kang-Hao (Kenny) Lu <kennyluck at csail.mit.edu>:

Since Python 3.2.2 (I don't have earlier version to test with),

>>> "\udc80".encode("utf-8")
UnicodeEncodeError: *utf-8* codec can't encode character '\udc80'...

but

>>> b"\xff".decode("utf-8")
UnicodeDecodeError: *utf8* codec can't decode byte 0xff in position 0

and the table on the documentation of the codec module suggests *utf_8* as the name of the codec, which I believe to be equivalent to "utf_8" because '-' is not a valid character of an identifier.

Can we at least make the above two consistent? I would go for "utf-8", which was probably introduced for rejecting surrogates, but "utf8" has been there for years. What do we do? I am happy to submit patches for all branches. These are one-liners anyway.

The backward compatibility risk should be pretty low as usually you don't get encoding from these errors and I don't see any use of PyUnicode(Encode|Decode)Error_GetEncoding in trunk, although I'm using it for issue #12892. 

Also, "latin_1" displays as *latin-1* but "iso2022-jp" displays as *iso2022_jp*. I care less about this nit though.

----------
components: Unicode
messages: 152399
nosy: ezio.melotti, kennyluck
priority: normal
severity: normal
status: open
title: utf-8 or utf8 or utf-8 (codec display name inconsistency)
versions: Python 2.7, Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13913>
_______________________________________

From report at bugs.python.org  Tue Jan 31 18:28:42 2012
From: report at bugs.python.org (Kang-Hao (Kenny) Lu)
Date: Tue, 31 Jan 2012 17:28:42 +0000
Subject: [issue13913] utf-8 or utf8 or utf-8 (codec display name inconsistency)
In-Reply-To: <1328030876.68.0.627537179228.issue13913@psf.upfronthosting.co.za>
Message-ID: <1328030922.5.0.611865473715.issue13913@psf.upfronthosting.co.za>


Changes by Kang-Hao (Kenny) Lu <kennyluck at csail.mit.edu>:


----------
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13913>
_______________________________________

From report at bugs.python.org  Tue Jan 31 18:35:07 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 31 Jan 2012 17:35:07 +0000
Subject: [issue13817] deadlock in subprocess while running several threads
	using Popen
In-Reply-To: <1326909636.44.0.892345276723.issue13817@psf.upfronthosting.co.za>
Message-ID: <1328031307.97.0.710607661812.issue13817@psf.upfronthosting.co.za>


Changes by Charles-Fran?ois Natali <neologix at free.fr>:


----------
stage: patch review -> commit review

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13817>
_______________________________________

From report at bugs.python.org  Tue Jan 31 18:50:57 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Tue, 31 Jan 2012 17:50:57 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1328032257.57.0.46607816978.issue13496@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

> I think the patch is *NOT* correct.

Can you elaborate?  It works fine for that example on a 64-bit build;  not sure why 32-bit would be any different.

The idea is just that the single cast forces the addition to be done as an addition of integers of type size_t.  Since the integers being added are (a) nonnegative and (b) both fit into a Py_ssize_t, the sum fits into a size_t without overflow, and the result of dividing the sum by 2 fits into a size_t.

(This is all assuming that 2*Py_SSIZE_T_MAX fits into a size_t, but that's a fairly safe assumption in practice.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Tue Jan 31 18:59:14 2012
From: report at bugs.python.org (Mark Dickinson)
Date: Tue, 31 Jan 2012 17:59:14 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1328032754.68.0.683448535553.issue13496@psf.upfronthosting.co.za>


Mark Dickinson <dickinsm at gmail.com> added the comment:

> Does it work with http://bugs.python.org/msg148567, in a 32 bits python?

Yes.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Tue Jan 31 19:11:50 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 18:11:50 +0000
Subject: [issue13496] bisect module: Overflow at index computation
In-Reply-To: <1322519672.22.0.173147243777.issue13496@psf.upfronthosting.co.za>
Message-ID: <1328033510.38.0.479446984583.issue13496@psf.upfronthosting.co.za>


Antoine Pitrou <pitrou at free.fr> added the comment:

Perhaps adding a comment before those two lines would make the reason for the cast clearer.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13496>
_______________________________________

From report at bugs.python.org  Tue Jan 31 20:10:37 2012
From: report at bugs.python.org (Petri Lehtinen)
Date: Tue, 31 Jan 2012 19:10:37 +0000
Subject: [issue13402] Document absoluteness of sys.executable
In-Reply-To: <1321289437.84.0.982753479785.issue13402@psf.upfronthosting.co.za>
Message-ID: <1328037037.3.0.0510637119209.issue13402@psf.upfronthosting.co.za>


Petri Lehtinen <petri at digip.org> added the comment:

Attached one more patch. The documentation now also mentions None as a possible value.

----------
Added file: http://bugs.python.org/file24380/issue13402_v3.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13402>
_______________________________________

From report at bugs.python.org  Tue Jan 31 21:33:08 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 31 Jan 2012 20:33:08 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1328041988.59.0.302902426645.issue10181@psf.upfronthosting.co.za>


Changes by Stefan Krah <stefan-usenet at bytereef.org>:


Added file: http://bugs.python.org/file24381/a5c4a96dc981.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Tue Jan 31 21:34:30 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 31 Jan 2012 20:34:30 +0000
Subject: [issue10181] Problems with Py_buffer management in memoryobject.c
	(and elsewhere?)
In-Reply-To: <1287858158.95.0.0752296181045.issue10181@psf.upfronthosting.co.za>
Message-ID: <1328042070.59.0.932149953197.issue10181@psf.upfronthosting.co.za>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

I've uploaded a new patch that should address the remaining issues:

   o In the documentation _testbuffer has been replaced by
     m.cast() + the now multi-dimensional m.tolist().

   o I restored the state of the limited API. If we want
     to include Py_buffer again, I think this should be done
     in a separate patch.

   o Flags of the memoryview object are private.


Additionally, because NumPy allows non-aligned array accesses,
I changed the packing functions to use memcpy for multi-byte types.

On x86/amd64 gcc is smart enough to produce almost exactly the same
asm output as before, with a slowdown of 0-1%, depending on the
benchmark.

On other platforms the situation might be worse, but I don't have
access to real hardware where alignment actually matters.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10181>
_______________________________________

From report at bugs.python.org  Tue Jan 31 21:50:52 2012
From: report at bugs.python.org (=?utf-8?q?Bo=C5=A1tjan_Mejak?=)
Date: Tue, 31 Jan 2012 20:50:52 +0000
Subject: =?utf-8?q?=5Bissue10580=5D_Minor_grammar_change_in_Python=E2=80=99s_MSI_i?=
	=?utf-8?q?nstaller?=
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1328043052.61.0.0387467880217.issue10580@psf.upfronthosting.co.za>


Bo?tjan Mejak <bostjan.mejak at gmail.com> added the comment:

Apply this patch. It is not necessary to change the title of this issue report, it is of most importance that this patch gets applied already after more than a year. Come on people, move your butts on this.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Tue Jan 31 21:57:21 2012
From: report at bugs.python.org (Amaury Forgeot d'Arc)
Date: Tue, 31 Jan 2012 20:57:21 +0000
Subject: =?utf-8?q?=5Bissue10580=5D_Minor_grammar_change_in_Python=E2=80=99s_MSI_i?=
	=?utf-8?q?nstaller?=
In-Reply-To: <1291053115.29.0.21132112584.issue10580@psf.upfronthosting.co.za>
Message-ID: <1328043441.05.0.621960911579.issue10580@psf.upfronthosting.co.za>


Amaury Forgeot d'Arc <amauryfa at gmail.com> added the comment:

Retro, this kind of tone is not appropriate for the bug tracker, and not appreciated at all by core developers.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10580>
_______________________________________

From report at bugs.python.org  Tue Jan 31 22:28:44 2012
From: report at bugs.python.org (Stefan Krah)
Date: Tue, 31 Jan 2012 21:28:44 +0000
Subject: [issue13868] Add hyphen doc fix
In-Reply-To: <1327957757.9.0.307948324.issue13868@psf.upfronthosting.co.za>
Message-ID: <20120131212843.GA8245@sleipnir.bytereef.org>


Stefan Krah <stefan-usenet at bytereef.org> added the comment:

Georg Brandl <report at bugs.python.org> wrote:
> Wow, does that mean he can cash in a Knuth check?

Two already: One for the insight that "i-th" is the only true spelling,
and now this one.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13868>
_______________________________________

From report at bugs.python.org  Tue Jan 31 22:42:03 2012
From: report at bugs.python.org (=?utf-8?q?Charles-Fran=C3=A7ois_Natali?=)
Date: Tue, 31 Jan 2012 21:42:03 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <1328018694.87.0.217712973532.issue13734@psf.upfronthosting.co.za>
Message-ID: <CAH_1eM3xYiGgA4q70Y=-15W=HRNsbgqZ5uzt4SA1rq6EshyPbw@mail.gmail.com>


Charles-Fran?ois Natali <neologix at free.fr> added the comment:

Here are two new versions, both addressing Antoine's and Nick's comments:
- fwalk-3.diff is just an updated version
- fwalk-single_fd.diff doesn't use more than 2 FDs to walk a directory
tree, instead of the depth of directory tree. It's not as simple and
clean as I'd like it to be, but it should be much more robust, and
still safe (please make sure about that :-).
I was a little worried about the performance impact, so I did some
trivial benchmarks:
- O(depth) fwalk() is actually a tiny bit faster than walk() (it may
be because we don't do as much path lookup)
- O(1) fwalk() is around 20% slower, on a pure-traversal benchmark (so
in a realistic use case where we would actually do something with the
values returned by fwalk() the difference shouldn't be that
noticeable)

----------
Added file: http://bugs.python.org/file24382/fwalk-3.diff
Added file: http://bugs.python.org/file24383/fwalk-single_fd.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2251,6 +2251,56 @@
               os.rmdir(os.path.join(root, name))
 
 
+.. function:: fwalk(top, topdown=True, onerror=None, followlinks=False)
+
+   .. index::
+      single: directory; walking
+      single: directory; traversal
+
+    This behaves exactly like :func:`walk`, except that it yields a 4-tuple
+    ``(dirpath, dirnames, filenames, dirfd)``.
+
+   *dirpath*, *dirnames* and *filenames* are identical to :func:`walk` output,
+   and *dirfd* is a file descriptor referring to the directory *dirpath*.
+
+   .. note::
+
+      Since :func:`fwalk` yields file descriptors, those are only valid until
+      the next iteration step, so you should duplicate them (e.g. with
+      :func:`dup`) if you want to keep them longer.
+
+   This example displays the number of bytes taken by non-directory files in each
+   directory under the starting directory, except that it doesn't look under any
+   CVS subdirectory::
+
+      import os
+      for root, dirs, files, rootfd in os.fwalk('python/Lib/email'):
+          print(root, "consumes", end="")
+          print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                end="")
+          print("bytes in", len(files), "non-directory files")
+          if 'CVS' in dirs:
+              dirs.remove('CVS')  # don't visit CVS directories
+
+   In the next example, walking the tree bottom-up is essential:
+   :func:`unlinkat` doesn't allow deleting a directory before the directory is
+   empty::
+
+      # Delete everything reachable from the directory named in "top",
+      # assuming there are no symbolic links.
+      # CAUTION:  This is dangerous!  For example, if top == '/', it
+      # could delete all your disk files.
+      import os
+      for root, dirs, files, rootfd in os.fwalk(top, topdown=False):
+          for name in files:
+              os.unlinkat(rootfd, name)
+          for name in dirs:
+              os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+
+   Availability: Unix.
+
+   .. versionadded:: 3.3
+
 .. _os-process:
 
 Process Management
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -497,6 +497,10 @@
 
   (Patch submitted by Giampaolo Rodol?? in :issue:`10784`.)
 
+* The :mod:`os` module has a new :func:`~os.fwalk` function similar to
+  :func:`~os.walk` except that it also yields file descriptors referring to the
+  directories visited. This is especially useful to avoid symlink races.
+
 * "at" functions (:issue:`4761`):
 
   * :func:`~os.faccessat`
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -24,6 +24,7 @@
 #'
 
 import sys, errno
+import stat as st
 
 _names = sys.builtin_module_names
 
@@ -32,6 +33,9 @@
            "defpath", "name", "path", "devnull",
            "SEEK_SET", "SEEK_CUR", "SEEK_END"]
 
+def _exists(name):
+    return name in globals()
+
 def _get_exports_list(module):
     try:
         return list(module.__all__)
@@ -120,7 +124,12 @@
     umask(mask)
     return mode & ~mask
 
-#'
+def _are_same_file(stat1, stat2):
+    """Helper function that checks whether two stat results refer to the same
+    file.
+    """
+    return (stat1.st_ino == stat2.st_ino and stat1.st_dev == stat2.st_dev)
+#
 
 # Super directory utilities.
 # (Inspired by Eric Raymond; the doc strings are mostly his)
@@ -151,7 +160,6 @@
     try:
         mkdir(name, mode)
     except OSError as e:
-        import stat as st
         if not (e.errno == errno.EEXIST and exist_ok and path.isdir(name) and
                 st.S_IMODE(lstat(name).st_mode) == _get_masked_mode(mode)):
             raise
@@ -298,6 +306,94 @@
 
 __all__.append("walk")
 
+if _exists("openat"):
+
+    def fwalk(top, topdown=True, onerror=None, followlinks=False):
+        """Directory tree generator.
+
+        This behaves exactly like walk(), except that it yields a 4-tuple
+
+            dirpath, dirnames, filenames, dirfd
+
+        `dirpath`, `dirnames` and `filenames` are identical to walk() output,
+        and `dirfd` is a file descriptor referring to the directory `dirpath`.
+
+        The advantage of walkfd() over walk() is that it's safe against symlink
+        races (when followlinks is False).
+
+        Caution:
+        Since fwalk() yields file descriptors, those are only valid until the
+        next iteration step, so you should dup() them if you want to keep them
+        for a longer period.
+
+        Example:
+
+        import os
+        for root, dirs, files, rootfd in os.fwalk('python/Lib/email'):
+            print(root, "consumes", end="")
+            print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                  end="")
+            print("bytes in", len(files), "non-directory files")
+            if 'CVS' in dirs:
+                dirs.remove('CVS')  # don't visit CVS directories
+        """
+        # Note: To guard against symlink races, we use the standard
+        # lstat()/open()/fstat() trick.
+        orig_st = lstat(top)
+        topfd = open(top, O_RDONLY)
+        try:
+            if (followlinks or (st.S_ISDIR(orig_st.st_mode) and
+                               _are_same_file(orig_st, fstat(topfd)))):
+                for x in _fwalk(topfd, top, topdown, onerror, followlinks):
+                    yield x
+        finally:
+            close(topfd)
+
+    def _fwalk(topfd, toppath, topdown, onerror, followlinks):
+        try:
+            names = fdlistdir(topfd)
+        except error as err:
+            if onerror is not None:
+                onerror(err)
+            return
+
+        dirs, nondirs = [], []
+        for name in names:
+            # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
+            # walk() which reports symlinks to directories as directories. We do
+            # however check for symlinks before recursing into a subdirectory.
+            if st.S_ISDIR(fstatat(topfd, name).st_mode):
+                dirs.append(name)
+            else:
+                nondirs.append(name)
+
+        # whether to follow symlinks
+        flag = 0 if followlinks else AT_SYMLINK_NOFOLLOW
+
+        if topdown:
+            yield toppath, dirs, nondirs, topfd
+
+        for name in dirs:
+            try:
+                orig_st = fstatat(topfd, name, flag)
+                dirfd = openat(topfd, name, O_RDONLY)
+            except error as err:
+                if onerror is not None:
+                    onerror(err)
+                return
+            try:
+                if followlinks or _are_same_file(orig_st, fstat(dirfd)):
+                    dirpath = path.join(toppath, name)
+                    for x in _fwalk(dirfd, dirpath, topdown, onerror, followlinks):
+                        yield x
+            finally:
+                close(dirfd)
+
+        if not topdown:
+            yield toppath, dirs, nondirs, topfd
+
+    __all__.append("fwalk")
+
 # Make sure os.environ exists, at least
 try:
     environ
@@ -598,9 +694,6 @@
 fsencode, fsdecode = _fscodec()
 del _fscodec
 
-def _exists(name):
-    return name in globals()
-
 # Supply spawn*() (probably only for Unix)
 if _exists("fork") and not _exists("spawnv") and _exists("execv"):
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -20,6 +20,8 @@
 import asyncore
 import asynchat
 import socket
+import itertools
+import stat
 try:
     import threading
 except ImportError:
@@ -159,7 +161,6 @@
         if not hasattr(os, "stat"):
             return
 
-        import stat
         result = os.stat(fname)
 
         # Make sure direct access works
@@ -476,7 +477,7 @@
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
-    def test_traversal(self):
+    def setUp(self):
         import os
         from os.path import join
 
@@ -586,6 +587,58 @@
                     os.remove(dirname)
         os.rmdir(support.TESTFN)
 
+ at unittest.skipUnless(hasattr(os, 'fwalk'), "Test needs os.fwalk()")
+class fwalkTests(WalkTests):
+    """Tests for os.fwalk()."""
+
+    def test_compare_to_walk(self):
+        # compare with walk() results
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            expected = {}
+            for root, dirs, files in os.walk(*args):
+                expected[root] = (set(dirs), set(files))
+
+            for root, dirs, files, rootfd in os.fwalk(*args):
+                self.assertIn(root, expected)
+                self.assertEqual(expected[root], (set(dirs), set(files)))
+
+    def test_dir_fd(self):
+        # check returned file descriptors
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            for root, dirs, files, rootfd in os.fwalk(*args):
+                # check that the FD is valid
+                os.fstat(rootfd)
+                # check that fdlistdir() returns consistent information
+                self.assertEqual(set(os.fdlistdir(rootfd)), set(dirs) | set(files))
+
+    def test_fd_leak(self):
+        # Since we're opening a lot of FDs, we must be careful to avoid leaks:
+        # we both check that calling fwalk() a large number of times doesn't
+        # yield EMFILE, and that the minimum allocated FD hasn't changed.
+        minfd = os.dup(1)
+        os.close(minfd)
+        for i in range(512):
+            for x in os.fwalk(support.TESTFN):
+                pass
+        newfd = os.dup(1)
+        self.addCleanup(os.close, newfd)
+        self.assertEqual(newfd, minfd)
+
+    def tearDown(self):
+        # cleanup
+        for root, dirs, files, rootfd in os.fwalk(support.TESTFN, topdown=False):
+            for name in files:
+                os.unlinkat(rootfd, name)
+            for name in dirs:
+                st = os.fstatat(rootfd, name, os.AT_SYMLINK_NOFOLLOW)
+                if stat.S_ISDIR(st.st_mode):
+                    os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+                else:
+                    os.unlinkat(rootfd, name)
+        os.rmdir(support.TESTFN)
+
 class MakedirTests(unittest.TestCase):
     def setUp(self):
         os.mkdir(support.TESTFN)
@@ -1700,6 +1753,7 @@
         StatAttributeTests,
         EnvironTests,
         WalkTests,
+        fwalkTests,
         MakedirTests,
         DevNullTests,
         URandomTests,
-------------- next part --------------
diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2251,6 +2251,56 @@
               os.rmdir(os.path.join(root, name))
 
 
+.. function:: fwalk(top, topdown=True, onerror=None, followlinks=False)
+
+   .. index::
+      single: directory; walking
+      single: directory; traversal
+
+    This behaves exactly like :func:`walk`, except that it yields a 4-tuple
+    ``(dirpath, dirnames, filenames, dirfd)``.
+
+   *dirpath*, *dirnames* and *filenames* are identical to :func:`walk` output,
+   and *dirfd* is a file descriptor referring to the directory *dirpath*.
+
+   .. note::
+
+      Since :func:`fwalk` yields file descriptors, those are only valid until
+      the next iteration step, so you should duplicate them (e.g. with
+      :func:`dup`) if you want to keep them longer.
+
+   This example displays the number of bytes taken by non-directory files in each
+   directory under the starting directory, except that it doesn't look under any
+   CVS subdirectory::
+
+      import os
+      for root, dirs, files, rootfd in os.fwalk('python/Lib/email'):
+          print(root, "consumes", end="")
+          print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                end="")
+          print("bytes in", len(files), "non-directory files")
+          if 'CVS' in dirs:
+              dirs.remove('CVS')  # don't visit CVS directories
+
+   In the next example, walking the tree bottom-up is essential:
+   :func:`unlinkat` doesn't allow deleting a directory before the directory is
+   empty::
+
+      # Delete everything reachable from the directory named in "top",
+      # assuming there are no symbolic links.
+      # CAUTION:  This is dangerous!  For example, if top == '/', it
+      # could delete all your disk files.
+      import os
+      for root, dirs, files, rootfd in os.fwalk(top, topdown=False):
+          for name in files:
+              os.unlinkat(rootfd, name)
+          for name in dirs:
+              os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+
+   Availability: Unix.
+
+   .. versionadded:: 3.3
+
 .. _os-process:
 
 Process Management
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -497,6 +497,10 @@
 
   (Patch submitted by Giampaolo Rodol?? in :issue:`10784`.)
 
+* The :mod:`os` module has a new :func:`~os.fwalk` function similar to
+  :func:`~os.walk` except that it also yields file descriptors referring to the
+  directories visited. This is especially useful to avoid symlink races.
+
 * "at" functions (:issue:`4761`):
 
   * :func:`~os.faccessat`
diff --git a/Lib/os.py b/Lib/os.py
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -24,6 +24,7 @@
 #'
 
 import sys, errno
+import stat as st
 
 _names = sys.builtin_module_names
 
@@ -32,6 +33,9 @@
            "defpath", "name", "path", "devnull",
            "SEEK_SET", "SEEK_CUR", "SEEK_END"]
 
+def _exists(name):
+    return name in globals()
+
 def _get_exports_list(module):
     try:
         return list(module.__all__)
@@ -120,7 +124,12 @@
     umask(mask)
     return mode & ~mask
 
-#'
+def _are_same_file(stat1, stat2):
+    """Helper function that checks whether two stat results refer to the same
+    file.
+    """
+    return (stat1.st_ino == stat2.st_ino and stat1.st_dev == stat2.st_dev)
+#
 
 # Super directory utilities.
 # (Inspired by Eric Raymond; the doc strings are mostly his)
@@ -151,7 +160,6 @@
     try:
         mkdir(name, mode)
     except OSError as e:
-        import stat as st
         if not (e.errno == errno.EEXIST and exist_ok and path.isdir(name) and
                 st.S_IMODE(lstat(name).st_mode) == _get_masked_mode(mode)):
             raise
@@ -298,6 +306,115 @@
 
 __all__.append("walk")
 
+if _exists("openat"):
+
+    def fwalk(top, topdown=True, onerror=None, followlinks=False):
+        """Directory tree generator.
+
+        This behaves exactly like walk(), except that it yields a 4-tuple
+
+            dirpath, dirnames, filenames, dirfd
+
+        `dirpath`, `dirnames` and `filenames` are identical to walk() output,
+        and `dirfd` is a file descriptor referring to the directory `dirpath`.
+
+        The advantage of walkfd() over walk() is that it's safe against symlink
+        races (when followlinks is False).
+
+        Caution:
+        Since fwalk() yields file descriptors, those are only valid until the
+        next iteration step, so you should dup() them if you want to keep them
+        for a longer period.
+
+        Example:
+
+        import os
+        for root, dirs, files, rootfd in os.fwalk('python/Lib/email'):
+            print(root, "consumes", end="")
+            print(sum([os.fstatat(rootfd, name).st_size for name in files]),
+                  end="")
+            print("bytes in", len(files), "non-directory files")
+            if 'CVS' in dirs:
+                dirs.remove('CVS')  # don't visit CVS directories
+        """
+        # Note: To guard against symlink races, we use the standard
+        # lstat()/open()/fstat() trick.
+        orig_st = lstat(top)
+        topfd = open(top, O_RDONLY)
+        if (followlinks or (st.S_ISDIR(orig_st.st_mode) and
+                            _are_same_file(orig_st, fstat(topfd)))):
+            for x in _fwalk(topfd, top, topdown, onerror, followlinks):
+                yield x
+        else:
+            close(topfd)
+
+    def _fwalk(topfd, toppath, topdown, onerror, followlinks):
+        """fwalk() backend - `topfd` is always closed."""
+        # Note: We use openat()/fdlistdir()/fstatat() to recurse into
+        # subdirectories in a safe way. However, we don't want to keep an open
+        # FD at each directory level, because we could run out of FDs on deeply
+        # nested directory hierarchies. So, we have to close the current `topfd`
+        # FD before recursing into a subdirectory, and re-open it right after.
+        # Of couse, we must check that the directory hasn't changed in between.
+
+        try:
+            orig_top_st = fstat(topfd)
+            names = fdlistdir(topfd)
+        except error as err:
+            close(topfd)
+            if onerror is not None:
+                onerror(err)
+            return
+
+        dirs, nondirs = [], []
+        for name in names:
+            # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
+            # walk() which reports symlinks to directories as directories. We do
+            # however check for symlinks before recursing into a subdirectory.
+            if st.S_ISDIR(fstatat(topfd, name).st_mode):
+                dirs.append(name)
+            else:
+                nondirs.append(name)
+
+        # whether to follow symlinks
+        flag = 0 if followlinks else AT_SYMLINK_NOFOLLOW
+
+        if topdown:
+            yield toppath, dirs, nondirs, topfd
+
+        for name in dirs:
+            try:
+                orig_st = fstatat(topfd, name, flag)
+                dirfd = openat(topfd, name, O_RDONLY)
+            except error as err:
+                close(topfd)
+                if onerror is not None:
+                    onerror(err)
+                return
+
+            if followlinks or _are_same_file(orig_st, fstat(dirfd)):
+                # close the current directory's FD before recursing into
+                # subdirectory
+                close(topfd)
+
+                dirpath = path.join(toppath, name)
+                for x in _fwalk(dirfd, dirpath, topdown, onerror, followlinks):
+                    yield x
+
+                # re-open it, and check it hasn't changed
+                topfd = open(toppath, O_RDONLY)
+                if not (followlinks or _are_same_file(orig_top_st, fstat(topfd))):
+                    break
+            else:
+                close(dirfd)
+
+        if not topdown:
+            yield toppath, dirs, nondirs, topfd
+
+        close(topfd)
+
+    __all__.append("fwalk")
+
 # Make sure os.environ exists, at least
 try:
     environ
@@ -598,9 +715,6 @@
 fsencode, fsdecode = _fscodec()
 del _fscodec
 
-def _exists(name):
-    return name in globals()
-
 # Supply spawn*() (probably only for Unix)
 if _exists("fork") and not _exists("spawnv") and _exists("execv"):
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -20,6 +20,8 @@
 import asyncore
 import asynchat
 import socket
+import itertools
+import stat
 try:
     import threading
 except ImportError:
@@ -159,7 +161,6 @@
         if not hasattr(os, "stat"):
             return
 
-        import stat
         result = os.stat(fname)
 
         # Make sure direct access works
@@ -476,7 +477,7 @@
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
-    def test_traversal(self):
+    def setUp(self):
         import os
         from os.path import join
 
@@ -586,6 +587,58 @@
                     os.remove(dirname)
         os.rmdir(support.TESTFN)
 
+ at unittest.skipUnless(hasattr(os, 'fwalk'), "Test needs os.fwalk()")
+class FwalkTests(WalkTests):
+    """Tests for os.fwalk()."""
+
+    def test_compare_to_walk(self):
+        # compare with walk() results
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            expected = {}
+            for root, dirs, files in os.walk(*args):
+                expected[root] = (set(dirs), set(files))
+
+            for root, dirs, files, rootfd in os.fwalk(*args):
+                self.assertIn(root, expected)
+                self.assertEqual(expected[root], (set(dirs), set(files)))
+
+    def test_dir_fd(self):
+        # check returned file descriptors
+        for topdown, followlinks in itertools.product((True, False), repeat=2):
+            args = support.TESTFN, topdown, None, followlinks
+            for root, dirs, files, rootfd in os.fwalk(*args):
+                # check that the FD is valid
+                os.fstat(rootfd)
+                # check that fdlistdir() returns consistent information
+                self.assertEqual(set(os.fdlistdir(rootfd)), set(dirs) | set(files))
+
+    def test_fd_leak(self):
+        # Since we're opening a lot of FDs, we must be careful to avoid leaks:
+        # we both check that calling fwalk() a large number of times doesn't
+        # yield EMFILE, and that the minimum allocated FD hasn't changed.
+        minfd = os.dup(1)
+        os.close(minfd)
+        for i in range(512):
+            for x in os.fwalk(support.TESTFN):
+                pass
+        newfd = os.dup(1)
+        self.addCleanup(os.close, newfd)
+        self.assertEqual(newfd, minfd)
+
+    def tearDown(self):
+        # cleanup
+        for root, dirs, files, rootfd in os.fwalk(support.TESTFN, topdown=False):
+            for name in files:
+                os.unlinkat(rootfd, name)
+            for name in dirs:
+                st = os.fstatat(rootfd, name, os.AT_SYMLINK_NOFOLLOW)
+                if stat.S_ISDIR(st.st_mode):
+                    os.unlinkat(rootfd, name, os.AT_REMOVEDIR)
+                else:
+                    os.unlinkat(rootfd, name)
+        os.rmdir(support.TESTFN)
+
 class MakedirTests(unittest.TestCase):
     def setUp(self):
         os.mkdir(support.TESTFN)
@@ -1700,6 +1753,7 @@
         StatAttributeTests,
         EnvironTests,
         WalkTests,
+        FwalkTests,
         MakedirTests,
         DevNullTests,
         URandomTests,

From report at bugs.python.org  Tue Jan 31 22:45:41 2012
From: report at bugs.python.org (py.user)
Date: Tue, 31 Jan 2012 21:45:41 +0000
Subject: [issue13914] In module re the repeat interval {} doesn't accept
	numbers greater than 65535
Message-ID: <1328046341.18.0.439922472722.issue13914@psf.upfronthosting.co.za>


New submission from py.user <port139 at yandex.ru>:

>>> import re
>>> len(re.search(r'a+', 'a' * 100000).group())
100000
>>>
>>> re.search(r'a{65536,}', 'a' * 100000)
Traceback (most recent call last):
  File "/usr/local/lib/python3.2/functools.py", line 176, in wrapper
    result = cache[key]
KeyError: (<class 'str'>, 'a{65536,}', 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/re.py", line 158, in search
    return _compile(pattern, flags).search(string)
  File "/usr/local/lib/python3.2/re.py", line 255, in _compile
    return _compile_typed(type(pattern), pattern, flags)
  File "/usr/local/lib/python3.2/functools.py", line 180, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/local/lib/python3.2/re.py", line 267, in _compile_typed
    return sre_compile.compile(pattern, flags)
  File "/usr/local/lib/python3.2/sre_compile.py", line 491, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/local/lib/python3.2/sre_parse.py", line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File "/usr/local/lib/python3.2/sre_parse.py", line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File "/usr/local/lib/python3.2/sre_parse.py", line 511, in _parse
    raise error("bad repeat interval")
sre_constants.error: bad repeat interval
>>>
>>>
>>> re.search(r'a{65536}', 'a' * 100000)
Traceback (most recent call last):
  File "/usr/local/lib/python3.2/functools.py", line 176, in wrapper
    result = cache[key]
KeyError: (<class 'str'>, 'a{65536}', 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/re.py", line 158, in search
    return _compile(pattern, flags).search(string)
  File "/usr/local/lib/python3.2/re.py", line 255, in _compile
    return _compile_typed(type(pattern), pattern, flags)
  File "/usr/local/lib/python3.2/functools.py", line 180, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/local/lib/python3.2/re.py", line 267, in _compile_typed
    return sre_compile.compile(pattern, flags)
  File "/usr/local/lib/python3.2/sre_compile.py", line 514, in compile
    groupindex, indexgroup
OverflowError: regular expression code size limit exceeded
>>>

----------
components: Library (Lib), Regular Expressions
messages: 152409
nosy: ezio.melotti, py.user
priority: normal
severity: normal
status: open
title: In module re the repeat interval {} doesn't accept numbers greater than 65535
type: behavior
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13914>
_______________________________________

From report at bugs.python.org  Tue Jan 31 22:56:14 2012
From: report at bugs.python.org (Antoine Pitrou)
Date: Tue, 31 Jan 2012 21:56:14 +0000
Subject: [issue13734] Add a generic directory walker method to avoid symlink
	attacks
In-Reply-To: <CAH_1eM3xYiGgA4q70Y=-15W=HRNsbgqZ5uzt4SA1rq6EshyPbw@mail.gmail.com>
Message-ID: <1328046829.20847.1.camel@localhost.localdomain>


Antoine Pitrou <pitrou at free.fr> added the comment:

> I was a little worried about the performance impact, so I did some
> trivial benchmarks:
> - O(depth) fwalk() is actually a tiny bit faster than walk() (it may
> be because we don't do as much path lookup)
> - O(1) fwalk() is around 20% slower, on a pure-traversal benchmark (so
> in a realistic use case where we would actually do something with the
> values returned by fwalk() the difference shouldn't be that
> noticeable)

I think the O(depth) version is fine. The O(1) version is quite more
complicated, difficult to follow, and it seems less robust (it doesn't
use try/finally and therefore might leak fds if the generator isn't
exhausted before being destroyed).

On modern systems you have at least 1024 fds, so the restriction
shouldn't be a problem.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13734>
_______________________________________

From report at bugs.python.org  Tue Jan 31 23:36:01 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 22:36:01 +0000
Subject: [issue13914] In module re the repeat interval {} doesn't accept
	numbers greater than 65535
In-Reply-To: <1328046341.18.0.439922472722.issue13914@psf.upfronthosting.co.za>
Message-ID: <1328049361.56.0.0594063092305.issue13914@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

This issue is a duplicate of #13169.

----------
nosy: +haypo
resolution:  -> duplicate
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13914>
_______________________________________

From report at bugs.python.org  Tue Jan 31 23:36:21 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 22:36:21 +0000
Subject: [issue13169] Regular expressions with 0 to 65536 repetitions raises
	OverflowError
In-Reply-To: <1318523427.81.0.476768111228.issue13169@psf.upfronthosting.co.za>
Message-ID: <1328049381.58.0.915616895574.issue13169@psf.upfronthosting.co.za>


STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Issue #13914 has been marked as a duplicate of this issue.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13169>
_______________________________________

From report at bugs.python.org  Tue Jan 31 23:37:13 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 22:37:13 +0000
Subject: [issue13845] Use GetSystemTimeAsFileTime() to get a resolution of 100
	ns on Windows
In-Reply-To: <1327363365.73.0.00740710968562.issue13845@psf.upfronthosting.co.za>
Message-ID: <1328049433.68.0.294534807811.issue13845@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


Removed file: http://bugs.python.org/file24306/timespec-2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13845>
_______________________________________

From report at bugs.python.org  Tue Jan 31 23:38:15 2012
From: report at bugs.python.org (STINNER Victor)
Date: Tue, 31 Jan 2012 22:38:15 +0000
Subject: [issue13847] Catch time(), ftime(), localtime() and clock() errors
In-Reply-To: <1327367189.33.0.271402103517.issue13847@psf.upfronthosting.co.za>
Message-ID: <1328049495.3.0.860876063369.issue13847@psf.upfronthosting.co.za>


Changes by STINNER Victor <victor.stinner at haypocalc.com>:


----------
resolution:  -> fixed
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13847>
_______________________________________