[ python-Bugs-1115379 ] Built-in compile function with PEP 0263 encoding bug

[SourceForge.net]

Bugs item #1115379, was opened at 2005-02-03 05:11
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1115379&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Parser/Compiler
Group: Python 2.4
Status: Open
Resolution: None
Priority: 7
Submitted By: Christoph Zwerschke (cito)
>Assigned to: Martin v. Löwis (loewis)
Summary: Built-in compile function with PEP 0263 encoding bug

Initial Comment:
a = 'print "Hello, World"'
u = '# -*- coding: utf-8 -*-\n' + a

print compile(a, '<string>', 'exec') # ok
print compile(u, '<string>', 'exec') # ok
print compile(unicode(a), '<string>', 'exec') # ok
print compile(unicode(u), '<string>', 'exec') # error

# The last line gives a SystemError.
# Think this is a bug.

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-03-20 00:28

Message:
Logged In: YES 
user_id=33168

Martin, the attached patches (2.4 and 2.5) fix the problem.
 However, it seems that the patches would violate the PEP
according to one of your notes.  I'm not sure about all the
details, but ISTM based on your comment that if (flags &&
flags->cf_flags & PyCF_SOURCE_IS_UTF8) and (TYPE(n) ==
encoding_decl) this is an error that should be returned?

I would like to get this fixed for 2.4.3, so we need to move
fast for it.  2.5 can wait and is trivial to fix once we
know what this is supposed to do.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-02-20 13:37

Message:
Logged In: YES 
user_id=849994

This even aborts the interpreter in 2.5 HEAD with a failing
assertion.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-09-27 22:48

Message:
Logged In: YES 
user_id=21627

If you load the files manually, why is it that you want to
decode them to Unicode before compile()ing them? Couldn't
you just pass the bytes you read from the file to compile()?

----------------------------------------------------------------------

Comment By: Vágvölgyi Attila (wigy)
Date: 2005-09-27 21:29

Message:
Logged In: YES 
user_id=156682

If this special case is a feature, not a bug, than it breaks
some symmetry for sure.

If I run a script having utf-8 encoding from a file with

  python script.py

then it has to have an encoding declaration. Now if I would
like to load the same file manually, decode it to a unicode
object, I also have to remove the encoding declaration at
the beginning of the file before I can give it to the
compile() function.

What special advantage comes from the fact that the compiler
does not simply ignore encoding declaration nodes from
unicode objects? Does this error message catch some possible
errors or does it make the compiler code simpler?

----------------------------------------------------------------------

Comment By: Vágvölgyi Attila (wigy)
Date: 2005-09-27 21:20

Message:
Logged In: YES 
user_id=156682

If this special case is a feature, not a bug, than it breaks
some symmetry for sure.

If I run a script having utf-8 encoding from a file with

  python script.py

then it has to have an encoding declaration. Now if I would
like to load the same file manually, decode it to a unicode
object, I also have to remove the encoding declaration at
the beginning of the file before I can give it to the
compile() function.

What special advantage comes from the fact that the compiler
does not simply ignore encoding declaration nodes from
unicode objects? Does this error message catch some possible
errors or does it make the compiler code simpler?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-02-09 16:37

Message:
Logged In: YES 
user_id=21627

There is a bug somewhere, certainly. However, I believe it
is in PEP 263, which should point out that unicode strings
in compile are only legal if they do *not* contain an
encoding declaration, as such strings are implicitly encoded
as UTF-8.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1115379&group_id=5470