From dan at omacneil.org  Sun Aug  4 19:17:15 2019
From: dan at omacneil.org (Dan MacNeil)
Date: Sun, 4 Aug 2019 19:17:15 -0400
Subject: [Python-porting] futurize double testing ? nta
Message-ID: <CAOugbJ6SDROTpOi1xHbsbSq2kwvp9fTscoCz9y_bZgqFQf8DZA@mail.gmail.com>

Absent unit tests, does using Futurize to require twice as much manual
testing (tests for both p2 and p3) as just converting to python 3 ?

How much kruft does Futurize leave in the python3 code , once you no longer
need python2 compatibility ?

Is Futurize as likely to generate "wrong" python2 code as 2to3 is to
generate "wrong" python 3 code?  (I am grateful for 2to3 , but it is only a
"mostly" solution.)

BACKGROUND

We're working on porting about 700 python files, most of which are plugins
for an internal application.

We've got the main part of the application (43 files) mostly ported to
python 3 via with 2to3 and various manual changes.

Our original thought  was to run futurize on the approximately 650 module
files maintained by people outside our group. In theory:

(1) They could test and then do their development work on the ported code
on our existing python2 application for a couple months and be ready for
our python 3 cut-over without too much trouble.

(2) We wouldn't have the danger of cutting everything over to python 3 all
at once.

Based on recent experience with 2to3 , I'm now guessing that getting things
to work on python2 in future mode is going to require roughly the same
amount of work as getting it to work in python 3 and instead of cutting
massive  changes over in the python3 change we'll do that in an earlier
python 2 change.

Thoughts ?
-- 
cell: 978-455-1885
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190804/58aaeb80/attachment.html>

From regebro at gmail.com  Mon Aug  5 16:53:52 2019
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 5 Aug 2019 22:53:52 +0200
Subject: [Python-porting] futurize double testing ? nta
In-Reply-To: <CAOugbJ6SDROTpOi1xHbsbSq2kwvp9fTscoCz9y_bZgqFQf8DZA@mail.gmail.com>
References: <CAOugbJ6SDROTpOi1xHbsbSq2kwvp9fTscoCz9y_bZgqFQf8DZA@mail.gmail.com>
Message-ID: <CAL0kPAUsoKKz3A6w1=QTf2Sv+HmCda_ndR1p6yG8UScUR=7wkg@mail.gmail.com>

On Mon, Aug 5, 2019 at 4:31 PM Dan MacNeil <dan at omacneil.org> wrote:

> Absent unit tests, does using Futurize to require twice as much manual
> testing (tests for both p2 and p3) as just converting to python 3 ?
>

Yes.


> How much kruft does Futurize leave in the python3 code , once you no
> longer need python2 compatibility ?
>

Too much for my liking, I had more problem with futurize than it was worth,
but obviously other people have a different experience.
I generally use Modernize instead. It doesn't try to solve bytes/unicode
issues, like Futurize does, but that's one of the reasons I had problems.

Thoughts ?
>

My thought is that moving to Python 3 without unit tests requires a lot of
bravery. :-)
But since it's plugins, perhaps each of the 700 files are small and easy to
test?

Also, my thoughts on moving larger projects are recently recorded here:
https://youtu.be/HdtBT3yXUeE?t=6312
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190805/2f40791e/attachment.html>

From justinisrael at gmail.com  Mon Aug  5 15:55:33 2019
From: justinisrael at gmail.com (Justin Israel)
Date: Tue, 6 Aug 2019 07:55:33 +1200
Subject: [Python-porting] futurize double testing ? nta
In-Reply-To: <mailman.44.1565020804.10191.python-porting@python.org>
References: <mailman.44.1565020804.10191.python-porting@python.org>
Message-ID: <CAPGFgA2m9ncwbuN7RX944mvikNzhVxmr-Bjpu-OVwPPhFm6ezQ@mail.gmail.com>

On Tue, Aug 6, 2019, 4:02 AM <python-porting-request at python.org> wrote:

> Send Python-porting mailing list submissions to
>         python-porting at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/python-porting
> or, via email, send a message with subject or body 'help' to
>         python-porting-request at python.org
>
> You can reach the person managing the list at
>         python-porting-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-porting digest..."
> Today's Topics:
>
>    1. Re: Porting Python 2 comments to Python 3 (Fenil Mehta)
>    2. futurize double testing ? nta (Dan MacNeil)
>
>
>
> ---------- Forwarded message ----------
> From: Fenil Mehta <fenilgmehta at gmail.com>
> To: Brett Cannon <brett at python.org>
> Cc: Python porting <python-porting at python.org>
> Bcc:
> Date: Thu, 18 Jul 2019 08:34:09 +0530
> Subject: Re: [Python-porting] Porting Python 2 comments to Python 3
> Hi,
>
> Unfortunately it's the latter. So, I will have to write a module for it.
> Thanks for the suggestion.
>
> Regards,
> Fenil Mehta
>
>
> On Tue, Jul 16, 2019 at 12:04 AM Brett Cannon <brett at python.org> wrote:
>
>> Are these doctests or just normal comments? If it's the the former I
>> thought 2to3 fixed those. If it's the latter then you will probably have to
>> take 2to3 and use it to write your own code rewriter to update the comments.
>>
>> On Mon, Jul 15, 2019 at 4:59 AM Fenil Mehta <fenilgmehta at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am working on a large project with lots of comments having Python 2
>>> program along with program explanations.
>>> I want to port all the Python 2 statements in comments to Python 3
>>> automatically.
>>> Please suggest some ways to do this. Thanks in advance.
>>>
>>> Regards,
>>> Fenil Mehta
>>> _______________________________________________
>>> Python-porting mailing list
>>> Python-porting at python.org
>>> https://mail.python.org/mailman/listinfo/python-porting
>>>
>>
>
>
> ---------- Forwarded message ----------
> From: Dan MacNeil <dan at omacneil.org>
> To: python-porting at python.org
> Cc:
> Bcc:
> Date: Sun, 4 Aug 2019 19:17:15 -0400
> Subject: [Python-porting] futurize double testing ? nta
> Absent unit tests, does using Futurize to require twice as much manual
> testing (tests for both p2 and p3) as just converting to python 3 ?
>
> How much kruft does Futurize leave in the python3 code , once you no
> longer need python2 compatibility ?
>
> Is Futurize as likely to generate "wrong" python2 code as 2to3 is to
> generate "wrong" python 3 code?  (I am grateful for 2to3 , but it is only a
> "mostly" solution.)
>
> BACKGROUND
>
> We're working on porting about 700 python files, most of which are plugins
> for an internal application.
>
> We've got the main part of the application (43 files) mostly ported to
> python 3 via with 2to3 and various manual changes.
>
> Our original thought  was to run futurize on the approximately 650 module
> files maintained by people outside our group. In theory:
>
> (1) They could test and then do their development work on the ported code
> on our existing python2 application for a couple months and be ready for
> our python 3 cut-over without too much trouble.
>
> (2) We wouldn't have the danger of cutting everything over to python 3 all
> at once.
>
> Based on recent experience with 2to3 , I'm now guessing that getting
> things to work on python2 in future mode is going to require roughly the
> same amount of work as getting it to work in python 3 and instead of
> cutting massive  changes over in the python3 change we'll do that in an
> earlier python 2 change.
>
> Thoughts ?
>

My experience so far, after having ported about 7 different projects using
futurize, is that it gets you somewhere between 50-90% of the way. That
huge window is due to factors like how much string/bytes processing you
need to do, or more edge case differences. And also it depends on how good
your tests already are. For some of the projects I have done so far, it has
been minimal to the point where I could just remove the future imports
after py2 isn't needed. Other project have relied a lot more on the
future.utils module.
As for the quality of code that is generated, it mostly seems to be correct
python3 stuff, with an occasional breaking change with stage2
transformations. We have been keeping an internal developers document to
list the "gotchas" as we find them.
So yea a good amount of time has been spent actually porting tests and
getting them to run correctly. But usually I have seen good python3 code
transformations, and code to be removed can easily be text searched for use
of the future module. We have avoided the forward ports.

Justin

-- 
> cell: 978-455-1885
> _______________________________________________
> Python-porting mailing list
> Python-porting at python.org
> https://mail.python.org/mailman/listinfo/python-porting
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190806/1deac82d/attachment.html>