[Python-Dev] Criticism of execfile() removal in Python3

[Fabio Zadrozny]

On Sat, Jun 14, 2014 at 6:00 PM, Markus Unterwaditzer <
markus at unterwaditzer.net> wrote:

> On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
> > Hello,
> >
> > I was pleasantly surprised with the response to recent post about
> > MicroPython implementation details
> > (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I
> > hope that discussion means that posts about alternative implementations
> > are not unwelcome here, so I would like to bring up another (of many)
> > issues we faced while implementing MicroPython.
> >
> > execfile() builtin function was removed in 3.0. This brings few
> > problems:
> >
> > 1. It hampers interactive mode - instead of short and easy to type
> > execfile("file.py") one needs to use exec(open("file.py").read()). I'm
> > sure that's not going to bother a lot of people - after all, the
> > easiest way to execute a Python file is to drop back to shell and
> > restart python with file name, using all wonders of tab completion. But
> > now imagine that Python interpreter runs on bare hardware, and its REPL
> > is the only shell. That's exactly what we have with MicroPython's
> > Cortex-M port. But it's not really MicroPython-specific, there's
> > CPython port to baremetal either - http://www.pycorn.org/ .
>
> As far as i can see, minimizing the amount of characters to type was never
> a
> design goal of the Python language. And because that goal never mattered as
> much for the designers as it seems to do for you, the reason for it to get
> removed -- reducing the amount of builtins without reducing functionality
> --
> was the only one left.
>
> > 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> > there's a problem - it requires to load entire file to memory. But
> > there can be not enough memory. Consider 1Mb file with 900Kb comments
> > (autogenerated, for example). execfile() could easily parse it, using
> > small buffer. But exec() requires to slurp entire file into memory, and
> > 1Mb is much more than heap sizes that we target.
>
> That is a valid concern, but i believe violating the language
> specification and
> adding your own execfile implementation (either as a builtin or in a new
> stdlib
> module) here is justified, even if it means you will have to modify your
> existing Python 3 code to use it -- i don't think the majority of software
> written in Python will be able to run under such memory constraints without
> major modifications anyway.
>
> > Comments, suggestions? Just to set a productive direction, please
> > kindly don't consider the problems above as MicroPython's.
>
> A new (not MicroPython-specific) stdlib module containing functions such as
> execfile could be considered. Not really for Python-2-compatibility, but
> for
> performance-critical situations.
>
> I am not sure if this is a good solution. Not at all. Even though it's
> separated from the builtins, i think it would still sacrifice the purity
> of the
> the language (by which i mean having a minimal composable API), because
> people
> are going to use it anyway. It reminds me of the situation in Python 2
> where
> developers are trying to use cStringIO with a fallback to StringIO as a
> matter
> of principle, not because they actually need that kind of performance.
>
> Another, IMO better idea which shifts the problem to the MicroPython devs
> is to
> "just" detect code using
>
>     exec(open(...).read())
>
> and transparently rewrite it to something more memory-efficient. This is
> the
> idea i actually think is a good one.
>
>
> > I very much liked how last discussion went: I was pointed that
> > https://docs.python.org/3/reference/index.html is not really a CPython
> > reference, it's a *Python* reference, and there were even motion to
> > clarify in it some points which came out from MicroPython discussion.
> > So, what about https://docs.python.org/3/library/index.html - is it
> > CPython, or Python standard library specification? Assuming the latter,
> > what we have is that, by removal of previously available feature,
> > *Python* became less friendly for interactive usage and less scalable.
>
> "Less friendly for interactive usage" is a strong and vague statement. If
> you're going after the amount of characters required to type, yes,
> absolutely,
> but by that terms one could declare Bash and Perl to be superior languages.
> Look at it from a different perspective: There are fewer builtins to
> remember.
>
> >
>

Well, I must say that the exec(open().read()) is not really a proper
execfile implementation because it may fail because of encoding issues...
(i.e.: one has to check the file encoding to do the open with the proper
encoding, otherwise it's possible to end up with gibberish).

The PyDev debugger has an implementation (see:
https://github.com/fabioz/Pydev/blob/development/plugins/org.python.pydev/pysrc/_pydev_execfile.py)
which considers the encoding so that the result is ok (but it still has a
bug related to utf-8 with bom:
https://sw-brainwy.rhcloud.com/tracker/PyDev/346 which I plan to fix
soon...)

Personally, it's one thing that I think should be restored as the proper
implementation is actually quite tricky and the default recommended
solution does not work properly on some situations (and if micropython can
provide an optimized implementation which'd conform to Python, that'd be
one more point to add it back)...

Best Regards,

Fabio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140614/9a9f845e/attachment-0001.html>