From alan.gauld at yahoo.co.uk  Sat Dec  1 04:12:27 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 1 Dec 2018 09:12:27 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <pttj5o$4ca$1@blaine.gmane.org>

On 01/12/2018 00:43, Henrique Castro wrote:
>a cluster with 4 nodes each running Linux (our Fedora-based distro), 

OK, The good news is you should be able to get a pre-built distro
rather than try to build your own.

But I still think the Conda users are more likely to be able to assist.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From henriquecsj at outlook.com  Sat Dec  1 06:41:27 2018
From: henriquecsj at outlook.com (Henrique Castro)
Date: Sat, 1 Dec 2018 11:41:27 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <pttj5o$4ca$1@blaine.gmane.org>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>,
 <pttj5o$4ca$1@blaine.gmane.org>
Message-ID: <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>

Thank you guys, I'll try to contact the Conda community.
Alan Gauld, good to know. I suggest that you try to spread the news to Fermi Lab, CERN and other research centers.
Sometimes when you need a specific setting it is just easier to pre-build your distro than to repeat the same configuration hundreds of times.


--
Henrique C. S. Junior

________________________________
From: Tutor <tutor-bounces+henriquecsj=outlook.com at python.org> on behalf of Alan Gauld via Tutor <tutor at python.org>
Sent: Saturday, December 1, 2018 07:12
To: tutor at python.org
Subject: Re: [Tutor] Moving a conda environment to an off-line computer

On 01/12/2018 00:43, Henrique Castro wrote:
>a cluster with 4 nodes each running Linux (our Fedora-based distro),

OK, The good news is you should be able to get a pre-built distro
rather than try to build your own.

But I still think the Conda users are more likely to be able to assist.

--
Alan G
Author of the Learn to Program web site
https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alan-g.me.uk%2F&amp;data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636792525248838704&amp;sdata=0xTl3WSOMgfGih2FYN8AYSzXORCRy9fKLN6iLk3txlk%3D&amp;reserved=0
https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.amazon.com%2Fauthor%2Falan_gauld&amp;data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636792525248838704&amp;sdata=FVJGUemjFp3fRyu%2FK7PpkefITCLyjLmZ1DGuhmfLqtU%3D&amp;reserved=0
Follow my photo-blog on Flickr at:
https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.flickr.com%2Fphotos%2Falangauldphotos&amp;data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636792525248838704&amp;sdata=UTIgec8%2BlrRQ75j8j%2FFla5CYsU7%2FWywZb0D4kY6Gefs%3D&amp;reserved=0


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Ftutor&amp;data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636792525248838704&amp;sdata=lR6inUjJWRGm9TZ2MwyoWXLkXSinM1lCbkCcJV6MGuw%3D&amp;reserved=0

From david at graniteweb.com  Sat Dec  1 14:10:54 2018
From: david at graniteweb.com (David Rock)
Date: Sat, 1 Dec 2018 13:10:54 -0600
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <pttj5o$4ca$1@blaine.gmane.org>
 <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <8EE98D33-8982-4522-9374-1CFDBF470FF3@graniteweb.com>


> On Dec 1, 2018, at 05:41, Henrique Castro <henriquecsj at outlook.com> wrote:
> 
> Thank you guys, I'll try to contact the Conda community.
> Alan Gauld, good to know. I suggest that you try to spread the news to Fermi Lab, CERN and other research centers.
> Sometimes when you need a specific setting it is just easier to pre-build your distro than to repeat the same configuration hundreds of times.


So it sounds like you have local network connectivity, but do not have external internet access?  I ask, because I wonder how you are copying the data to the system from your computer that does have an internet connection.

Is there no internet access at all?  By that, I mean is there also no web proxy that can be used?

If a proxy exists, it may be possible to do something like define an http_proxy

os.environ['http_proxy?]=?yourproxy:port'


It looks to me like you have a bigger problem than being able to ?install conda.?  Even after conda is functioning, errors like :

Traceback (most recent call last):
 File "qm7_ANI.py", line 15, in <module>
   featurizer='BPSymmetryFunction', split='stratified', move_mean=False)
 File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/molnet/load_function/qm7_datasets.py", line 50, in load_qm7_from_mat
   'http://deepchem.io.s3-website-us-west-1.amazonaws.com/datasets/qm7.mat'


tell me it is always going to want to use the internet to access datasets in AWS.  Unless you can use a web proxy, or possibly get a local mirror of the deepchem.io data on a university system you _can_ reach, this may not be possible to resolve.


? 
David Rock
david at graniteweb.com





From alan.gauld at yahoo.co.uk  Sat Dec  1 14:20:18 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 1 Dec 2018 19:20:18 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <pttj5o$4ca$1@blaine.gmane.org>
 <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <ptumpf$nak$1@blaine.gmane.org>

On 01/12/2018 11:41, Henrique Castro wrote:
> Thank you guys, I'll try to contact the Conda community.


> Alan Gauld, good to know. I suggest that you try to spread the news

I only meant that because it was Fedora (rather than a bespoke
Linux build) that you should find a binary package someplace.
You still need to figure out how to install it on your cluster,
but that's still much easier than building everything from scratch.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From mats at wichmann.us  Sat Dec  1 15:04:17 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Sat, 1 Dec 2018 13:04:17 -0700
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <ptumpf$nak$1@blaine.gmane.org>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <pttj5o$4ca$1@blaine.gmane.org>
 <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptumpf$nak$1@blaine.gmane.org>
Message-ID: <de5556d2-0d76-1ff5-7ec9-46b6eb2a8762@wichmann.us>

On 12/1/18 12:20 PM, Alan Gauld via Tutor wrote:
> On 01/12/2018 11:41, Henrique Castro wrote:
>> Thank you guys, I'll try to contact the Conda community.
> 
> 
>> Alan Gauld, good to know. I suggest that you try to spread the news
> 
> I only meant that because it was Fedora (rather than a bespoke
> Linux build) that you should find a binary package someplace.
> You still need to figure out how to install it on your cluster,
> but that's still much easier than building everything from scratch.
> 
> 

yes, fedora has an anaconda package.  but since anaconda itself is a
kind of package manager, the interesting thing is getting it to install
the stuff you want, and that stuff is not going to come from Fedora
packages, even if there may in fact be packages of the things you want
to install - the fedpkgs wouldn't put stuff in the places anaconda will
expect.

conda caches packages it has downloaded.  I don't know exactly where,
but I think it will tell you if you ask it to dump its full
configuration.  if you replicate the cache directory from a test machine
which is internet-connected over to the cluster, it seems likely
installs from inside conda will then work. hopefully :)




From oscar.j.benjamin at gmail.com  Sat Dec  1 17:38:34 2018
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sat, 1 Dec 2018 22:38:34 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <CAHVvXxRAyeNNo5MP=DBcqsfcP6qk2r3jsnLZZRxB4RfrfeHLbA@mail.gmail.com>

On Sat, 1 Dec 2018 at 00:10, Henrique Castro <henriquecsj at outlook.com> wrote:
>
> Dear colleagues,
> Soon I'll start to use one of the powerful computers on my university as a tool in my Ph.D. The computer does not have an internet connection and I need to find a way to install a conda environment on it.

When you say that you need to install a conda environment I imagine
that what you mean is you need to install some Python packages and you
think the easiest way is to install a conda environment. Is that
correct?

Having used a similar setup at my University the situation we have is
that there is a big cluster with thousands of nodes that don't have
internet access but there is internet access on certain nodes called
the "login nodes" which you can ssh into. They have a filesystem that
is shared with all other nodes which means you can use git, pip etc to
get your code set up and working before submitting a job to be run on
the main cluster. Is that not the case for the setup you're using?

Also the setup we have actually provides many Python versions and
packages: I just have to activate them with a command that's something
like "module activate python-3.6 python-3.6-numpy ...".

The other aspect of our setup is that it is well equipped with
compilers, scientific libraries etc. so that it's usually straight
forward to compile e.g. numpy from source with "python setup.py
install". I don't know if I could also just transfer manylinux wheels
in there to avoid compiling as well...

So there are potentially a number of ways of achieving what you want.
With the information you've provided so far, I'm not clear what the
best way forward is. The first question is why is it that you want a
conda environment?

--
Oscar

From alan.gauld at yahoo.co.uk  Sat Dec  1 18:29:37 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 1 Dec 2018 23:29:37 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <ptskp5$jsc$1@blaine.gmane.org>
 <RO1P152MB127311066DEFFC8AAF90C949BFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
 <pttj5o$4ca$1@blaine.gmane.org>
 <RO1P152MB127350C215EBC5F554F1F4ECBFAC0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <ptv5cu$u5m$1@blaine.gmane.org>

On 01/12/2018 11:41, Henrique Castro wrote:
> Thank you guys, I'll try to contact the Conda community.

Try:

https://support.anaconda.com/

"Community Support" looks a likely option....


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avigross at verizon.net  Sat Dec  1 17:02:14 2018
From: avigross at verizon.net (Avi Gross)
Date: Sat, 1 Dec 2018 17:02:14 -0500
Subject: [Tutor] Wrap a pipe to get text mode
Message-ID: <001001d489c1$84e21710$8ea64530$@verizon.net>

Brief(er) message on topic:

 

Someone was having a problem that was traced to their use of a pipe to
communicate between tasks that normally passes data in binary mode.

 

They were offered solutions on how to convert the data that should work
fine. I am offering a different solution from the Lutz Book I am currently
reading:

 

Programming Python: Powerful Object-Oriented Programming

By Mark Lutz

 

This is the snippet on-line:

 

https://books.google.com/books?id=q8W3WQbNWmkC
<https://books.google.com/books?id=q8W3WQbNWmkC&pg=PA226&lpg=PA226&dq=python
+wrap+a+pipe&source=bl&ots=Y8ddhRkA2E&sig=NW2YygRxI9qUtJjMQx77Xhwfy88&hl=en&
sa=X&ved=2ahUKEwj-sZygzv_eAhUCnFkKHbbxBu8Q6AEwCnoECAQQAQ#v=onepage&q=python%
20wrap%20a%20pipe&f=false>
&pg=PA226&lpg=PA226&dq=python+wrap+a+pipe&source=bl&ots=Y8ddhRkA2E&sig=NW2Yy
gRxI9qUtJjMQx77Xhwfy88&hl=en&sa=X&ved=2ahUKEwj-sZygzv_eAhUCnFkKHbbxBu8Q6AEwC
noECAQQAQ#v=onepage&q=python%20wrap%20a%20pipe&f=false

 

The key is to take one end of the pipe and wrap a file around it that then
provides the interface that processes and returns raw data as text:

 

In the parent, take the side of the pipe you read from, called pipein as in:

 

pipe, pipeout = os.pipe

 

and do this:

 

pipein = os.fdopen(pipein)

 

NOTE that their method does not use Popen() to sun a command. Your child
process would need to start that program on their own. But the text received
would not require further processing from binary.

 

Many other solutions exist, such as on some systems using named pipes which
can be opened in text mode. Even simpler is to run the external process with
an argument like ">file" and have python open that file in text mode after
it completes. These solutions do require having permission to create files
and perhaps require additional effort to create a unique temp file.

 

 


From dave at the-hills.org.uk  Sun Dec  2 05:29:43 2018
From: dave at the-hills.org.uk (Dave Hill)
Date: Sun, 2 Dec 2018 10:29:43 +0000
Subject: [Tutor] subprocess.Popen()
Message-ID: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>

I am a volunteer at a Heritage Railway in N.Wales and, amongst other 
things, I provide electronics and software for various exhibits in the 
museum. I use the Raspberry Pi to provide various video presentations, 
employing the omxplayer.

I am in the process of updating an application known as the 'Runaway 
Train', originally written some 4ish years ago, as it has started to 
exhibit intermittent faults. This involves playing a video (at 8 times 
normal speed), via two TV screens, of a narrow gauge train ride from the 
view of the locomotive driver, on request, with the visitor located in a 
replica footplate, in which the floor vibrates and smoke emanates.

As I learn more Python I wanted to improve the interaction with 
omxplayer. In the original application I called omxplayer using Popen

__________________________________________________________________________________

 ??? omxp = Popen(['omxplayer', MOVIE_PATH])

followed by

 ??? # wait for video process to finish
 ??? omxp.wait()

__________________________________________________________________________________

Having 'graduated' to Python 3.7, I thought I would explore 
subprocess.Popen, and put the code in a Class, see code below. The video 
runs, but an error occurs, which I do not understand, see further below

__________________________________________________________________________________

# Import subprocess
import subprocess

from io import StringIO

class Player:

 ??? def __init__(self, path, timeout ):

 ??????? self.path = path
 ??????? self.timeout = timeout

 ??? def playVideo(self, filename, audio):

 ??????? MOVIE_PATH = self.path + filename

 ??????? if ( audio == "HDMI" ):
 ??????????? omxp = subprocess.Popen(['omxplayer', '-s', '-o', 'hdmi',\
MOVIE_PATH],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
 ??????? else:
 ??????????? # Call omxplayer - audio to Analog Port
 ??????????? omxp = subprocess.Popen(['omxplayer', '-s', '-o', 'local',\
MOVIE_PATH],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

 ??????? try:
 ??????????? out = omxp.communicate(timeout=self.timeout)
 ??????????? print("Try outs =? ", StringIO(out))
 ??????? except:
 ??????????? omxp.kill()
 ??????????? out = omxp.communicate()
 ??????????? print("Except outs =? ", StringIO(out))

__________________________________________________________________________________

Traceback (most recent call last):
   File "/home/pi/Code/TestVideo#4.py", line 31, in <module>
     player.playVideo(FILE, 'HDMI')
   File "/home/pi/Code/VideoPlayer.py", line 51, in playVideo
     out = omxp.communicate()
   File "/usr/lib/python3.5/subprocess.py", line 801, in communicate
     stdout, stderr = self._communicate(input, endtime, timeout)
   File "/usr/lib/python3.5/subprocess.py", line 1437, in _communicate
     selector.register(self.stdout, selectors.EVENT_READ)
   File "/usr/lib/python3.5/selectors.py", line 351, in register
     key = super().register(fileobj, events, data)
   File "/usr/lib/python3.5/selectors.py", line 237, in register
     key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
   File "/usr/lib/python3.5/selectors.py", line 224, in _fileobj_lookup
     return _fileobj_to_fd(fileobj)
   File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
     "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=8>

__________________________________________________________________________________

 From previous discussions, where do I sit in the demographics! I 
graduated from a degree, that was 95% physics with a small amount of 
electronics, spent 30 years as essentially an electronics engineer, 
during which time I wrote software using Fortran, Basic, Pascal (HP) and 
C++, typically for embedded applications. I have been retired for 10 
years, and occasionally play trains.


From matt.ruffalo at gmail.com  Sun Dec  2 10:27:18 2018
From: matt.ruffalo at gmail.com (Matt Ruffalo)
Date: Sun, 2 Dec 2018 10:27:18 -0500
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>
Message-ID: <447967e6-7c16-4b95-a6e0-2399345a9de1@gmail.com>

Hi Henrique-

It is quite easy to transfer an Anaconda installation from one machine
to the other by copying all of the files -- I have done this repeatedly
with cluster compute environments. It is sometimes nicer to run `conda
upgrade --all` in a local VM and then `rsync` the updated Anaconda
installation between machines, since (as you mentioned) internet access
can sometimes be an issue.

It looks like you did everything correctly, and everything is "working"
as well as you would expect. As Alan mentioned, though, it looks like
the 'deepchem' package is trying to access the internet to load one of
its data sets, and this is what is failing. You could perhaps download
that data set and put it somewhere on the cluster where deepchem would
know where to look for it, to avoid having to download it, but I am
completely unfamiliar with deepchem so I can't offer any advice about
how to do that.

MMR...

On 30/11/18 08:47, Henrique Castro wrote:
> Dear colleagues,
> Soon I'll start to use one of the powerful computers on my university as a tool in my Ph.D. The computer does not have an internet connection and I need to find a way to install a conda environment on it.
> At first I tried to install and set the conda environment that I need in a computer with internet connection and taking care to copy everything in a path that is similar in the off-line computer (i.e I installed everything on /home/henrique/bin/anaconda3 at home and tried to copy everything to /home/henrique/bin/anaconda3 in the off-line computer - with the same .bashrc) but when I run conda I get an error(it works on my home computer):
>
> (deepchem) [henrique at europio qm7] $ python qm7_ANI.py
> /home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
>   from numpy.core.umath_tests import inner1d
> Traceback (most recent call last):
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1318, in do_open
>     encode_chunked=req.has_header('Transfer-encoding'))
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1239, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1285, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1234, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1026, in _send_output
>     self.send(msg)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 964, in send
>     self.connect()
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 936, in connect
>     (self.host,self.port), self.timeout, self.source_address)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", line 704, in create_connection
>     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", line 745, in getaddrinfo
>     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
> socket.gaierror: [Errno -2] Name or service not known
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "qm7_ANI.py", line 15, in <module>
>     featurizer='BPSymmetryFunction', split='stratified', move_mean=False)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/molnet/load_function/qm7_datasets.py", line 50, in load_qm7_from_mat
>     'http://deepchem.io.s3-website-us-west-1.amazonaws.com/datasets/qm7.mat'
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/utils/__init__.py", line 85, in download_url
>     urlretrieve(url, os.path.join(dest_dir, name))
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 248, in urlretrieve
>     with contextlib.closing(urlopen(url, data)) as fp:
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 223, in urlopen
>     return opener.open(url, data, timeout)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 526, in open
>     response = self._open(req, data)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 544, in _open
>     '_open', req)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 504, in _call_chain
>     result = func(*args)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1346, in http_open
>     return self.do_open(http.client.HTTPConnection, req)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1320, in do_open
>     raise URLError(err)
> urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
>
> Any help is much appreciated
>
>
> --
> Henrique C. S. Junior
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



From alan.gauld at yahoo.co.uk  Mon Dec  3 05:12:30 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 3 Dec 2018 10:12:30 +0000
Subject: [Tutor] subprocess.Popen()
In-Reply-To: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
References: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
Message-ID: <pu2veb$d28$1@blaine.gmane.org>

On 02/12/2018 10:29, Dave Hill wrote:

> Having 'graduated' to Python 3.7, I thought I would explore 
> Traceback (most recent call last):
>    File "/home/pi/Code/TestVideo#4.py", line 31, in <module>
>      player.playVideo(FILE, 'HDMI')
>    File "/home/pi/Code/VideoPlayer.py", line 51, in playVideo
>      out = omxp.communicate()
>    File "/usr/lib/python3.5/subprocess.py", line 801, in communicate
>      stdout, stderr = self._communicate(input, endtime, timeout)

The first thing I notice is that you say you are using
Python 3.7 yet the libs appear to be 3.5.

It could be some kind of path problem?

Or maybe how you start Python. Do you explicitly call python37?

I don't have the time to go through the code in detail just now.
If nobody else picks it up I'll take a closer look later.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From henriquecsj at outlook.com  Mon Dec  3 05:54:48 2018
From: henriquecsj at outlook.com (Henrique Castro)
Date: Mon, 3 Dec 2018 10:54:48 +0000
Subject: [Tutor] Moving a conda environment to an off-line computer
In-Reply-To: <447967e6-7c16-4b95-a6e0-2399345a9de1@gmail.com>
References: <RO1P152MB127340EFD9C8AE4DDDBB6FAEBFD30@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>,
 <447967e6-7c16-4b95-a6e0-2399345a9de1@gmail.com>
Message-ID: <RO1P152MB1273AC45B182A84415DB23B7BFAE0@RO1P152MB1273.LAMP152.PROD.OUTLOOK.COM>

Thank you for all the insightful replies!
It looks like the problem is exactly that deepchem is looking for an internet connection and crashing.
I'll see what I can do and provide some feedback here.

Again, thank you for using some of your time to take a look at this.

--
Henrique C. S. Junior

________________________________
From: Matt Ruffalo <matt.ruffalo at gmail.com>
Sent: Sunday, December 2, 2018 13:27
To: Henrique Castro; tutor at python.org
Subject: Re: Moving a conda environment to an off-line computer

Hi Henrique-

It is quite easy to transfer an Anaconda installation from one machine
to the other by copying all of the files -- I have done this repeatedly
with cluster compute environments. It is sometimes nicer to run `conda
upgrade --all` in a local VM and then `rsync` the updated Anaconda
installation between machines, since (as you mentioned) internet access
can sometimes be an issue.

It looks like you did everything correctly, and everything is "working"
as well as you would expect. As Alan mentioned, though, it looks like
the 'deepchem' package is trying to access the internet to load one of
its data sets, and this is what is failing. You could perhaps download
that data set and put it somewhere on the cluster where deepchem would
know where to look for it, to avoid having to download it, but I am
completely unfamiliar with deepchem so I can't offer any advice about
how to do that.

MMR...

On 30/11/18 08:47, Henrique Castro wrote:
> Dear colleagues,
> Soon I'll start to use one of the powerful computers on my university as a tool in my Ph.D. The computer does not have an internet connection and I need to find a way to install a conda environment on it.
> At first I tried to install and set the conda environment that I need in a computer with internet connection and taking care to copy everything in a path that is similar in the off-line computer (i.e I installed everything on /home/henrique/bin/anaconda3 at home and tried to copy everything to /home/henrique/bin/anaconda3 in the off-line computer - with the same .bashrc) but when I run conda I get an error(it works on my home computer):
>
> (deepchem) [henrique at europio qm7] $ python qm7_ANI.py
> /home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
>   from numpy.core.umath_tests import inner1d
> Traceback (most recent call last):
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1318, in do_open
>     encode_chunked=req.has_header('Transfer-encoding'))
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1239, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1285, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1234, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 1026, in _send_output
>     self.send(msg)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 964, in send
>     self.connect()
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", line 936, in connect
>     (self.host,self.port), self.timeout, self.source_address)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", line 704, in create_connection
>     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", line 745, in getaddrinfo
>     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
> socket.gaierror: [Errno -2] Name or service not known
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "qm7_ANI.py", line 15, in <module>
>     featurizer='BPSymmetryFunction', split='stratified', move_mean=False)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/molnet/load_function/qm7_datasets.py", line 50, in load_qm7_from_mat
>     'https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdeepchem.io.s3-website-us-west-1.amazonaws.com%2Fdatasets%2Fqm7.mat&amp;data=02%7C01%7C%7C76013bc4f1c946bc00ea08d6586aa6b0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636793612418543403&amp;sdata=v5exjG1hIZlYhQ9zNis87L4sMf%2BXmhIIBF4lGAC8hWU%3D&amp;reserved=0'
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/utils/__init__.py", line 85, in download_url
>     urlretrieve(url, os.path.join(dest_dir, name))
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 248, in urlretrieve
>     with contextlib.closing(urlopen(url, data)) as fp:
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 223, in urlopen
>     return opener.open(url, data, timeout)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 526, in open
>     response = self._open(req, data)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 544, in _open
>     '_open', req)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 504, in _call_chain
>     result = func(*args)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1346, in http_open
>     return self.do_open(http.client.HTTPConnection, req)
>   File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/urllib/request.py", line 1320, in do_open
>     raise URLError(err)
> urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
>
> Any help is much appreciated
>
>
> --
> Henrique C. S. Junior
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Ftutor&amp;data=02%7C01%7C%7C76013bc4f1c946bc00ea08d6586aa6b0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636793612418543403&amp;sdata=c1luHrdUYbzCfn5%2BNc3c8awFxIJfkESVyNKZiAtCaa4%3D&amp;reserved=0
>



From jstapcot at gmail.com  Mon Dec  3 05:35:29 2018
From: jstapcot at gmail.com (James Stapleton-Cotton)
Date: Mon, 3 Dec 2018 12:35:29 +0200
Subject: [Tutor] Beginners Book, Python and PyScripter
Message-ID: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>

Hello,

On this page (
http://openbookproject.net/thinkcs/python/english3e/way_of_the_program.html)
- a book for learning Computer Science using Python - I am directed to (
http://code.google.com/p/pyscripter) in order to access the appropriate
program development environment, PyScripter. I have downloaded the latest
version of Python from Python.org and PyScripter from a site that I am
directed to from the original site mentioned in the beginning of this email
- (https://sourceforge.net/projects/pyscripter/files/), however I can't
seem to be able to run PyScripter on my Mac.
The only file that opens is the Python Shell. I am really confused as to
whether or not PyScripter works on Mac OS or if I'm missing a step in the
installation process?

I hope I have been clear in my query. I look forward to hearing from you.

Regards,
James

From steve at pearwood.info  Mon Dec  3 08:07:52 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 4 Dec 2018 00:07:52 +1100
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
Message-ID: <20181203130751.GC13061@ando.pearwood.info>

Hi James, and welcome.

On Mon, Dec 03, 2018 at 12:35:29PM +0200, James Stapleton-Cotton wrote:
[...]
> I have downloaded the latest
> version of Python from Python.org and PyScripter from a site that I am
> directed to from the original site mentioned in the beginning of this email
> - (https://sourceforge.net/projects/pyscripter/files/), however I can't
> seem to be able to run PyScripter on my Mac.
> The only file that opens is the Python Shell. I am really confused as to
> whether or not PyScripter works on Mac OS or if I'm missing a step in the
> installation process?

I don't think there are a lot of Mac experts here, but let's see what we 
can do...

You say "the only file that opens is the Python Shell", but you don't 
tell us *precisely* what you did to get it to open. Go through it step 
by step, e.g. if you were a Windows user you might say:

"I clicked on the Start Menu, click on PyScripter, the shell opens"

"I right-clicked on a .py file and choose Open from the menu"

"I double-clicked a .py file on the desktop"

etc. If any of the steps are Mac-specific, you might need to go into a 
bit more detail, but we generally understand things like double- 
clicking, control-click to get the context menu, etc.

When you say "the Python Shell", you are probably seeing a text-based 
console that starts with something similar to this:

Python 3.5.2 (default, Oct 12 2016, 10:47:40)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-55)] on linux
Type "help", "copyright", "credits" or "license" for more information.


(the details will be different, of course) and then presents you with a 
prompt >>> where you can type commands. Is this correct?

If not, please explain what you get.

Don't bother with a screen shot at this stage: they're not helpful to 
those who are visually impaired or blind (screen readers can't work with 
them) and this mailing list will strip them out of your email. You ought 
to be able to copy any visible text in the shell and paste it into your 
email, if necessary.

Another test you can do is to open the Terminal from Mac OS (I think it 
is found under Applications ? Utilities ? Terminal) and then enter the 
command "pyscripter" (without the quotes) at the $ or % prompt. What 
happens?

(If you get an error, please copy and paste the entire error, including 
the command you entered. Don't try to summarise it or retype it from 
memory.)


P.S. when replying, reply to the list, not to me personally.


-- 
Steve

From mats at wichmann.us  Mon Dec  3 09:02:53 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 3 Dec 2018 07:02:53 -0700
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
Message-ID: <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>

On 12/3/18 3:35 AM, James Stapleton-Cotton wrote:
> Hello,
> 
> On this page (
> http://openbookproject.net/thinkcs/python/english3e/way_of_the_program.html)
> - a book for learning Computer Science using Python - I am directed to (
> http://code.google.com/p/pyscripter) in order to access the appropriate
> program development environment, PyScripter. I have downloaded the latest
> version of Python from Python.org and PyScripter from a site that I am
> directed to from the original site mentioned in the beginning of this email
> - (https://sourceforge.net/projects/pyscripter/files/), however I can't
> seem to be able to run PyScripter on my Mac.

it looks like from the project's own description that PyScripter is
intended solely for the Windows environment.

But there are a TON of other IDE products, free and commercial, out
there, many of which do run on the Mac.

This may be information overload, but try looking here:

https://wiki.python.org/moin/IntegratedDevelopmentEnvironments



From mats at wichmann.us  Mon Dec  3 11:27:26 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 3 Dec 2018 09:27:26 -0700
Subject: [Tutor] subprocess.Popen()
In-Reply-To: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
References: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
Message-ID: <68da77a5-d7ac-c0e4-1a95-371e916c7139@wichmann.us>

On 12/2/18 3:29 AM, Dave Hill wrote:

> Having 'graduated' to Python 3.7, I thought I would explore
> subprocess.Popen, and put the code in a Class, see code below. The video
> runs, but an error occurs, which I do not understand, see further below

the error happens in the except clause of your try block ...

> Traceback (most recent call last):
...
>   File "/home/pi/Code/VideoPlayer.py", line 51, in playVideo
>     out = omxp.communicate()
...
>   File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
>     "{!r}".format(fileobj)) from None
> ValueError: Invalid file object: <_io.BufferedReader name=8>

You can see that the call to communicate() is the one without arguments,
so it's in your cleanup code.   You should catch the specific exception
you expect could happen here, which is probably a timeout - but you
can't tell because you don't specify it.

> ??????? try:
> ??????????? out = omxp.communicate(timeout=self.timeout)
> ??????????? print("Try outs =? ", StringIO(out))
> ??????? except:
> ??????????? omxp.kill()
> ??????????? out = omxp.communicate()
> ??????????? print("Except outs =? ", StringIO(out))

use

      except TimeoutExpired:

then it looks like the file descriptor for the second communicate is no
longer valid... here's a larger chunk from subprocess that shows what it
is raising and how it got there:

    if isinstance(fileobj, int):
        fd = fileobj
    else:
        try:
            fd = int(fileobj.fileno())
        except (AttributeError, TypeError, ValueError):
            raise ValueError("Invalid file object: "
                             "{!r}".format(fileobj)) from None


In other words, it's raising the ValueError you see  because it's not a
file object. The "from None" part suppresses the context from the
original exception.

Note that communicate returns a tuple so your 'out' is actually a tuple,
and this is likely what is going wrong - by calling StringIO(out) in the
try: you are causing an exception which isn't the exception one would
expect; by not being explicit in your except: you catch that error and
take cleanup steps that are not appropriate - your video probably
finished normally.

hopefully this will help debug things...


From dave at the-hills.org.uk  Mon Dec  3 08:31:18 2018
From: dave at the-hills.org.uk (Dave Hill)
Date: Mon, 3 Dec 2018 13:31:18 +0000
Subject: [Tutor] subprocess.Popen()
In-Reply-To: <pu2veb$d28$1@blaine.gmane.org>
References: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
 <pu2veb$d28$1@blaine.gmane.org>
Message-ID: <aea85cb4-1d44-6500-3469-94eaefcf9b16@the-hills.org.uk>

I had not spotted that! It appears that, although I have 3.7 on my 
laptop, 3.7 is not a part of the latest standard Raspbian release. I 
will install this later today and try again.

Dave

On 03/12/2018 10:12, Alan Gauld via Tutor wrote:
> On 02/12/2018 10:29, Dave Hill wrote:
>
>> Having 'graduated' to Python 3.7, I thought I would explore
>> Traceback (most recent call last):
>>     File "/home/pi/Code/TestVideo#4.py", line 31, in <module>
>>       player.playVideo(FILE, 'HDMI')
>>     File "/home/pi/Code/VideoPlayer.py", line 51, in playVideo
>>       out = omxp.communicate()
>>     File "/usr/lib/python3.5/subprocess.py", line 801, in communicate
>>       stdout, stderr = self._communicate(input, endtime, timeout)
> The first thing I notice is that you say you are using
> Python 3.7 yet the libs appear to be 3.5.
>
> It could be some kind of path problem?
>
> Or maybe how you start Python. Do you explicitly call python37?
>
> I don't have the time to go through the code in detail just now.
> If nobody else picks it up I'll take a closer look later.
>

From dave at the-hills.org.uk  Mon Dec  3 14:15:36 2018
From: dave at the-hills.org.uk (Dave Hill)
Date: Mon, 3 Dec 2018 19:15:36 +0000
Subject: [Tutor] subprocess.Popen()
In-Reply-To: <68da77a5-d7ac-c0e4-1a95-371e916c7139@wichmann.us>
References: <c5e3cbb8-a385-488d-38c3-55cec48768fd@the-hills.org.uk>
 <68da77a5-d7ac-c0e4-1a95-371e916c7139@wichmann.us>
Message-ID: <97d31b8c-16d9-5a85-4b1f-11e6e58a3a06@the-hills.org.uk>

The combination of installing Python 3.7 on RasPi and removing the 
StringIO() cures the error.

I now get the statistics and the 'have a nice day' output from omxplayer.

I just need to find out how to invoke the idle 3.7, rather than idle 
3.5, but I think that is a question for the RasPi forum.

Thank you.

Dave

On 03/12/2018 16:27, Mats Wichmann wrote:
> On 12/2/18 3:29 AM, Dave Hill wrote:
>
>> Having 'graduated' to Python 3.7, I thought I would explore
>> subprocess.Popen, and put the code in a Class, see code below. The video
>> runs, but an error occurs, which I do not understand, see further below
> the error happens in the except clause of your try block ...
>
>> Traceback (most recent call last):
> ...
>>    File "/home/pi/Code/VideoPlayer.py", line 51, in playVideo
>>      out = omxp.communicate()
> ...
>>    File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
>>      "{!r}".format(fileobj)) from None
>> ValueError: Invalid file object: <_io.BufferedReader name=8>
> You can see that the call to communicate() is the one without arguments,
> so it's in your cleanup code.   You should catch the specific exception
> you expect could happen here, which is probably a timeout - but you
> can't tell because you don't specify it.
>
>>  ??????? try:
>>  ??????????? out = omxp.communicate(timeout=self.timeout)
>>  ??????????? print("Try outs =? ", StringIO(out))
>>  ??????? except:
>>  ??????????? omxp.kill()
>>  ??????????? out = omxp.communicate()
>>  ??????????? print("Except outs =? ", StringIO(out))
> use
>
>        except TimeoutExpired:
>
> then it looks like the file descriptor for the second communicate is no
> longer valid... here's a larger chunk from subprocess that shows what it
> is raising and how it got there:
>
>      if isinstance(fileobj, int):
>          fd = fileobj
>      else:
>          try:
>              fd = int(fileobj.fileno())
>          except (AttributeError, TypeError, ValueError):
>              raise ValueError("Invalid file object: "
>                               "{!r}".format(fileobj)) from None
>
>
> In other words, it's raising the ValueError you see  because it's not a
> file object. The "from None" part suppresses the context from the
> original exception.
>
> Note that communicate returns a tuple so your 'out' is actually a tuple,
> and this is likely what is going wrong - by calling StringIO(out) in the
> try: you are causing an exception which isn't the exception one would
> expect; by not being explicit in your except: you catch that error and
> take cleanup steps that are not appropriate - your video probably
> finished normally.
>
> hopefully this will help debug things...
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor

From cs at cskk.id.au  Mon Dec  3 16:29:53 2018
From: cs at cskk.id.au (Cameron Simpson)
Date: Tue, 4 Dec 2018 08:29:53 +1100
Subject: [Tutor] To check for empty string after a portion of the string
 in python 3.6
In-Reply-To: <CAFsTbWdFjdvkW+2Z26OZNo2A9hrQVmPt+4yqssdPfGgqoa3j3Q@mail.gmail.com>
References: <CAFsTbWdFjdvkW+2Z26OZNo2A9hrQVmPt+4yqssdPfGgqoa3j3Q@mail.gmail.com>
Message-ID: <20181203212953.GA63325@cskk.homeip.net>

Note: post returned to the tutor list.

Please DO NOT cross post to multiple lists (i.e. tutor and python-list, 
as you have). This makes things difficult for people who are not on both 
lists. Pick a _single_ list, and use that.

On 04Dec2018 02:46, srinivasan <srinivasan.rns at gmail.com> wrote:
>Could you please help me, as am still learning python syntax, how can
>I add conditional check for empty string after running "hcitool scan"
>(ie., when there is no Bluetooth devices discovered) ie., after the
>word  "Scanning..." , when there are no Bluetooth discover-able
>devices and then return False in the below python code snippet?
>
>
>Command:
>~$ sudo hcitool scan
>Scanning ...                              -------------------------->
>When there are no BT devices
>~$ sudo hcitool scan
>Scanning ...
>64:A2:F9:06:63:79 OnePlus 6 ---------------------------> When there
>are BT devices
>~$
>
>Python code snippet:
>    def bluetooth_scan(self):
>        """
>            Start bluetooth scanning process.
>
>            :return: Return device info by mac address and name.
>        """
>        cmd = "sudo hciconfig hci0 up"
>        self._helper.execute_cmd_return_code(cmd)
>
>        cmd = "sudo hcitool scan"
>        res = self._helper.execute_cmd_output_string(cmd)
>
>        return res

Well you document your function as returning device info by MAC. So part 
the output if "hcitool scan" and for MAC address and name and store that 
in a dictionary (key MAC, value the name). Return the dictionary.

If the dictionary is empty, there were no devices.

Not that like almost all collections, empty dictionaries are "false", so 
you can go:

  bluetooth_devices = scanner.bluetooth_scan()
  if not bluetooth_devices:
    ... no devices found ...

Cheers,
Cameron Simpson <cs at cskk.id.au>

From cs at cskk.id.au  Mon Dec  3 18:58:00 2018
From: cs at cskk.id.au (Cameron Simpson)
Date: Tue, 4 Dec 2018 10:58:00 +1100
Subject: [Tutor] To check for empty string after a portion of the string
 in python 3.6
In-Reply-To: <CAFsTbWd6JVDs=nemLMurbMqy5Y-EzC1V9_ZTyf_u1chw3qhX6g@mail.gmail.com>
References: <CAFsTbWd6JVDs=nemLMurbMqy5Y-EzC1V9_ZTyf_u1chw3qhX6g@mail.gmail.com>
Message-ID: <20181203235800.GA27762@cskk.homeip.net>

Again, _please_ use just one list: tutor or python-list. I've have 
directed replies to the tutor list. - Cameron

On 04Dec2018 04:05, srinivasan <srinivasan.rns at gmail.com> wrote:
>Thanks a lot for your quick responses, Could you please let me know
>when the device is not enabled I get the error " I get the below error
>saying "IndexError: list index out of range""

>Code snippet:
[...]
>res = self._helper.execute_cmd_output_string(cmd)
>print("The value of res!!!!", res)
>res = self._helper.execute_cmd_output_string(cmd).split("\n", 2)
>print("the value!!!!!!!!!!!!", res)
>print("the value!!!!!!!!!!!!", res[1])
>if __name__ == "__main__":
>    m = Bt()
>    print(m.bluetooth_scan())
>
>1. Output when device is enabled:
>The value of res!!!! Scanning ...
>64:A2:F9:06:63:79 OnePlus 6
>the value!!!!!!!!!!!! ['Scanning ...', '\t64:A2:F9:06:63:79\tOnePlus 6']
>the value!!!!!!!!!!!! 64:A2:F9:06:63:79 OnePlus 6
>None

The "None" is because your bluetooth_scan method has no return value (I 
think), which in Python it means it returns None.

>2. Output when device is not enabled
>When the device is not connected, I get the below error saying
>"IndexError: list index out of range"
>The value of res!!!! Scanning ...
>Traceback (most recent call last):
>the value!!!!!!!!!!!! ['Scanning ...']
>  File "/home/srinivasan/Downloads/bt_tests/qa/test_library/Bt.py",
>line 96, in <module>
>    print(m.bluetooth_scan())
>  File "/home/srinivasan/Downloads/bt_tests/qa/test_library/Bt.py",
>line 74, in bluetooth_scan
>    print("the value!!!!!!!!!!!!", res[1])
>IndexError: list index out of range

Well, you've printed your list:

  ['Scanning ...']

It is a single element list. So it has an element at index 0. There is 
no element 1. Therefor Python raises an IndexError if you try to access 
that element.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From 2hendrixtp at stu.bps-ok.org  Mon Dec  3 17:02:32 2018
From: 2hendrixtp at stu.bps-ok.org (Treyton Hendrix)
Date: Mon, 3 Dec 2018 16:02:32 -0600
Subject: [Tutor] I need help with my project
In-Reply-To: <20181130231402.GA10155@cskk.homeip.net>
References: <ptm4it$7s2$1@blaine.gmane.org>
 <006b01d48739$a13d6cf0$e3b846d0$@verizon.net>
 <20181130231402.GA10155@cskk.homeip.net>
Message-ID: <CA+uao+HdV+0ub62ok-egpRP+fJctgkBTwu=9GGc-3ouZm4gpkg@mail.gmail.com>

Awesome! Thanks to your help, I finally got my program done. Click here if
you want to see it! ---->

On Fri, Nov 30, 2018 at 5:14 PM Cameron Simpson <cs at cskk.id.au> wrote:

> Avi and Alan and Sibylle, you're making this a bit hard on the OP
> (Treyton).
>
> Yes he's supplied no context, but it is easy to make some suggestions.
> Each of yours suggests he design a much wider system (menu entry, web
> interface, some kind of GUI). All of which is (a) beyond him and (b)
> irrelevant.
>
> Why not pretend he _has_ the existing order, from wherever.
>
> Suggest ways to store that order (in a list, or a dict mapping ordable
> items to counts, or something). Then ask him to write a little Python,
> or even detailed English prose.
>
> Treyton: you seem to have recitied a homework question:
> >If the user selected a sandwich, french fries, and a beverage, reduce
> >the
> >total cost of the order by $1.00.
> >This is what I have to do and I don't know where to start.
>
> Ok, this is clear: Treyton can't get off the ground, very common for
> beginning programmers.
>
> The core challenge is to break your problem into a sequence of tasks.
> How would _you_, a person, do this if you had a food order given to you?
>
> Think about a food order. It is usually a list of standard food items, a
> count of how many of each. And each item will have a cost.
>
> The total cost is the sum of (each item's cost * its price * its count),
> for each item in the order. Or for all possible items, by presuming that
> unordered items just have a count of 0.
>
> So you need:
>
> A label for each item, so you can talk about it. You can just use a
> string for this, eg "sandwich" or "fries". Make the strings simple to
> start with to avoid spelling mistakes. You can always associate better
> names with the short strings later.
>
> You need a table of items and their costs. It is normal to make a
> mapping for this, such a Python's dict type. You can write dicts
> literally:
>
>   costs = {
>     "sandwich": 200,
>     "fries": 100,
>   }
>
> In the example above, I'm imagining you have dollars and cents, and
> making prices in cents.
>
> You also need a representation of the order, being the item type and the
> count. You could use a Python list for this. Example:
>
>   [ "fries", 2 ]
>
> The whole order might be a list of those, example:
>
>   [ ["fries", 2 ], [ "sandwich", 3 ] ]
>
> So, a list of lists.
>
> For purposes of your program you can just set all this stuff up at the
> beginning, not worrying about GUIs or input forma or any complication.
>
>   whole_order = [
>     ["fries", 2 ],
>     [ "sandwich", 3 ]
>   ]
>
> Now comes the part you need to do:
>
> - write some Python code to compute the total cost of the order (item
>   cost * item count), summed for all the items. Print this raw total so
>   that you can see it is correct.
>
> - write some totally separate code to look at the order and decide if
>   the client met your special condition (sandwich, fries, beverage) and
>   get a true/false result. Print this, too.
>
> - write a Python statement to subtract $1.00 (or 100 cents) from the
>   total if that condition is true. Print that.
>
> Then fiddle the order and run your programme several times to check that
> it is behaving the way it should.
>
> If you find difficulties you cannot surmount, come back here (by
> replying directly to one of the messages in your discussion) with:
>
> - your complete code
>
> - your expected output, and the output from your programme
>
> - a complete transcript of any error message, for example if your
>   programme raised an exception
>
> Make sure these are inline in your message, _not_ attachments. We drop
> attachments in this list.
>
> Cheers,
> Cameron Simpson <cs at cskk.id.au>
>

From asad.hasan2004 at gmail.com  Mon Dec  3 17:01:53 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Tue, 4 Dec 2018 03:31:53 +0530
Subject: [Tutor] sftp using subprocess
Message-ID: <CAG3LsKHgtYTHCMvsp+XrGLOzpJ1VZNa48VbRf+W=3Avjb85Kvg@mail.gmail.com>

Hi All ,

        I am looking for a solution to automate downloading file using sftp
process using :

batch.txt :contains the following two lines
cd 12345678
get 12345678_1.zip


import subprocess
host="abc.com"
user="sa"
user_host="%s@%s" % (user, host)
sftp_process = subprocess.Popen(['sftp', '-b', 'batch.txt', user_host],
shell=False)

however it gives error even unable to provide password .

Please advise.

Thank you,
-- 
Asad Hasan
+91 9582111698

From smravikumardonbosco at gmail.com  Mon Dec  3 17:15:51 2018
From: smravikumardonbosco at gmail.com (Ravi Kumar)
Date: Mon, 3 Dec 2018 16:15:51 -0600
Subject: [Tutor] Regarding Python api script
Message-ID: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>

Hi,

I have developed a python script to get api calls for meraki
clientlogevents I am wanting the output of apicalls which is in xml format
integrated to sql server management  studio how do i do that?

Thanks

From srinivasan.rns at gmail.com  Mon Dec  3 16:16:30 2018
From: srinivasan.rns at gmail.com (srinivasan)
Date: Tue, 4 Dec 2018 02:46:30 +0530
Subject: [Tutor] To check for empty string after a portion of the string in
 python 3.6
Message-ID: <CAFsTbWdFjdvkW+2Z26OZNo2A9hrQVmPt+4yqssdPfGgqoa3j3Q@mail.gmail.com>

Dear Python Experts,

Could you please help me, as am still learning python syntax, how can
I add conditional check for empty string after running "hcitool scan"
(ie., when there is no Bluetooth devices discovered) ie., after the
word  "Scanning..." , when there are no Bluetooth discover-able
devices and then return False in the below python code snippet?


Command:
~$ sudo hcitool scan
Scanning ...                              -------------------------->
When there are no BT devices
~$ sudo hcitool scan
Scanning ...
64:A2:F9:06:63:79 OnePlus 6 ---------------------------> When there
are BT devices
~$

Python code snippet:
    def bluetooth_scan(self):
        """
            Start bluetooth scanning process.

            :return: Return device info by mac address and name.
        """
        cmd = "sudo hciconfig hci0 up"
        self._helper.execute_cmd_return_code(cmd)

        cmd = "sudo hcitool scan"
        res = self._helper.execute_cmd_output_string(cmd)

        return res

Many Thanks in advance,

From srinivasan.rns at gmail.com  Mon Dec  3 17:35:11 2018
From: srinivasan.rns at gmail.com (srinivasan)
Date: Tue, 4 Dec 2018 04:05:11 +0530
Subject: [Tutor] To check for empty string after a portion of the string
 in python 3.6
In-Reply-To: <20181203212953.GA63325@cskk.homeip.net>
References: <CAFsTbWdFjdvkW+2Z26OZNo2A9hrQVmPt+4yqssdPfGgqoa3j3Q@mail.gmail.com>
 <20181203212953.GA63325@cskk.homeip.net>
Message-ID: <CAFsTbWd6JVDs=nemLMurbMqy5Y-EzC1V9_ZTyf_u1chw3qhX6g@mail.gmail.com>

Dear Mr. Cameron

Thanks a lot for your quick responses, Could you please let me know
when the device is not enabled I get the error " I get the below error
saying "IndexError: list index out of range""

Code snippet:
cmd = "sudo hcitool scan"
res = self._helper.execute_cmd_output_string(cmd)
print("The value of res!!!!", res)
res = self._helper.execute_cmd_output_string(cmd).split("\n", 2)
print("the value!!!!!!!!!!!!", res)
print("the value!!!!!!!!!!!!", res[1])

if __name__ == "__main__":
    m = Bt()
    print(m.bluetooth_scan())


1. Output when device is enabled:
The value of res!!!! Scanning ...
64:A2:F9:06:63:79 OnePlus 6
the value!!!!!!!!!!!! ['Scanning ...', '\t64:A2:F9:06:63:79\tOnePlus 6']
the value!!!!!!!!!!!! 64:A2:F9:06:63:79 OnePlus 6
None

Process finished with exit code 0

2. Output when device is not enabled

When the device is not connected, I get the below error saying
"IndexError: list index out of range"
The value of res!!!! Scanning ...
Traceback (most recent call last):
the value!!!!!!!!!!!! ['Scanning ...']
  File "/home/srinivasan/Downloads/bt_tests/qa/test_library/Bt.py",
line 96, in <module>
    print(m.bluetooth_scan())
  File "/home/srinivasan/Downloads/bt_tests/qa/test_library/Bt.py",
line 74, in bluetooth_scan
    print("the value!!!!!!!!!!!!", res[1])
IndexError: list index out of range

Process finished with exit code 1

Could you please do the needful

Thanks in advance



















On Tue, Dec 4, 2018 at 3:09 AM Cameron Simpson <cs at cskk.id.au> wrote:
>
> Note: post returned to the tutor list.
>
> Please DO NOT cross post to multiple lists (i.e. tutor and python-list,
> as you have). This makes things difficult for people who are not on both
> lists. Pick a _single_ list, and use that.
>
> On 04Dec2018 02:46, srinivasan <srinivasan.rns at gmail.com> wrote:
> >Could you please help me, as am still learning python syntax, how can
> >I add conditional check for empty string after running "hcitool scan"
> >(ie., when there is no Bluetooth devices discovered) ie., after the
> >word  "Scanning..." , when there are no Bluetooth discover-able
> >devices and then return False in the below python code snippet?
> >
> >
> >Command:
> >~$ sudo hcitool scan
> >Scanning ...                              -------------------------->
> >When there are no BT devices
> >~$ sudo hcitool scan
> >Scanning ...
> >64:A2:F9:06:63:79 OnePlus 6 ---------------------------> When there
> >are BT devices
> >~$
> >
> >Python code snippet:
> >    def bluetooth_scan(self):
> >        """
> >            Start bluetooth scanning process.
> >
> >            :return: Return device info by mac address and name.
> >        """
> >        cmd = "sudo hciconfig hci0 up"
> >        self._helper.execute_cmd_return_code(cmd)
> >
> >        cmd = "sudo hcitool scan"
> >        res = self._helper.execute_cmd_output_string(cmd)
> >
> >        return res
>
> Well you document your function as returning device info by MAC. So part
> the output if "hcitool scan" and for MAC address and name and store that
> in a dictionary (key MAC, value the name). Return the dictionary.
>
> If the dictionary is empty, there were no devices.
>
> Not that like almost all collections, empty dictionaries are "false", so
> you can go:
>
>   bluetooth_devices = scanner.bluetooth_scan()
>   if not bluetooth_devices:
>     ... no devices found ...
>
> Cheers,
> Cameron Simpson <cs at cskk.id.au>

From david at graniteweb.com  Mon Dec  3 19:12:05 2018
From: david at graniteweb.com (David Rock)
Date: Mon, 3 Dec 2018 18:12:05 -0600
Subject: [Tutor] sftp using subprocess
In-Reply-To: <CAG3LsKHgtYTHCMvsp+XrGLOzpJ1VZNa48VbRf+W=3Avjb85Kvg@mail.gmail.com>
References: <CAG3LsKHgtYTHCMvsp+XrGLOzpJ1VZNa48VbRf+W=3Avjb85Kvg@mail.gmail.com>
Message-ID: <A218A11B-5812-4679-AC08-0DC5929EF405@graniteweb.com>


> On Dec 3, 2018, at 16:01, Asad <asad.hasan2004 at gmail.com> wrote:
> 
> Hi All ,
> 
>        I am looking for a solution to automate downloading file using sftp
> process using :
> 
> batch.txt :contains the following two lines
> cd 12345678
> get 12345678_1.zip
> 
> 
> import subprocess
> host="abc.com"
> user="sa"
> user_host="%s@%s" % (user, host)
> sftp_process = subprocess.Popen(['sftp', '-b', 'batch.txt', user_host],
> shell=False)
> 
> however it gives error even unable to provide password .

batch mode for sftp assumes you don?t need a password.  From the manage:

     -b batchfile
             Batch mode reads a series of commands from an input batchfile instead of stdin.
             Since it lacks user interaction it should be used in conjunction with non-inter?
             active authentication.  

You would need to set up an ssh key with a null passphrase, or set up an ssh-agent in order for that to work.

Another option would be to use pexpect to automate console input


? 
David Rock
david at graniteweb.com





From steve at pearwood.info  Mon Dec  3 19:12:56 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 4 Dec 2018 11:12:56 +1100
Subject: [Tutor] sftp using subprocess
In-Reply-To: <CAG3LsKHgtYTHCMvsp+XrGLOzpJ1VZNa48VbRf+W=3Avjb85Kvg@mail.gmail.com>
References: <CAG3LsKHgtYTHCMvsp+XrGLOzpJ1VZNa48VbRf+W=3Avjb85Kvg@mail.gmail.com>
Message-ID: <20181204001255.GI13061@ando.pearwood.info>

On Tue, Dec 04, 2018 at 03:31:53AM +0530, Asad wrote:

> however it gives error even unable to provide password .

Is the error a secret? Would you like us to guess what it is?


-- 
Steve

From steve at pearwood.info  Mon Dec  3 19:15:16 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 4 Dec 2018 11:15:16 +1100
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
Message-ID: <20181204001514.GJ13061@ando.pearwood.info>

On Mon, Dec 03, 2018 at 04:15:51PM -0600, Ravi Kumar wrote:
> Hi,
> 
> I have developed a python script to get api calls for meraki
> clientlogevents I am wanting the output of apicalls which is in xml format
> integrated to sql server management  studio how do i do that?

What have you tried?


-- 
Steve

From alan.gauld at yahoo.co.uk  Mon Dec  3 19:36:58 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 00:36:58 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
Message-ID: <pu4i38$qrg$1@blaine.gmane.org>

On 03/12/2018 22:15, Ravi Kumar wrote:

> I have developed a python script to get api calls for meraki
> clientlogevents I am wanting the output of apicalls which is in xml format
> integrated to sql server management  studio how do i do that?

XML is such a flexible format that without seeing the
specific schema and data its near impossible to give
specific answers.

However, the etree module that comes with Python is
designed for parsing XML so if you take a look at
its documentation (and search for ElementTree
tutorials) you should get something.

There are other third party modules that are slightly
easier to use but provided its straightforward,
well formed XML, etree should be fine.

If you need to access the SQL Server database to
fetch the XML then the Python DBAPI has modules
that work with SQLserver. The ODBC one if nothing
else works!

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Mon Dec  3 20:24:58 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 01:24:58 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
Message-ID: <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>

CCd the list, please use Reply All when responding to the tutor list.

On 04/12/2018 00:52, Ravi Kumar wrote:
> Thanks a lot! I was wondering is it easier to access JSON format into
> Sql Server from python rather than XML If so how do I format my output
> from xml to Json?
>
JSON is generally easier than XML using the json module.


However, I confess I don't know what you mean by

"...access JSON format into Sql Server from python..."

Is the JSON/XML put into the database before you access it?

Or are you trying to access data from JSON/XML and then
store it in the database?

Or are you just trying to store JSON/XML directly into the database?

I really don't understand what it is you are trying to do.


-- 

Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


From asad.hasan2004 at gmail.com  Tue Dec  4 01:25:56 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Tue, 4 Dec 2018 11:55:56 +0530
Subject: [Tutor] sftp using subprocess
In-Reply-To: <mailman.1488.1543882389.25467.tutor@python.org>
References: <mailman.1488.1543882389.25467.tutor@python.org>
Message-ID: <CAG3LsKFgn3rtxikpCsFTvdVBbAmGYDdn3U3-NE-AOA33bj+3Gw@mail.gmail.com>

Hi All,

        I get the following error :

>>> sftp_process = subprocess.Popen(['sftp', '-b', 'batch.txt', user_host],
shell=False)
>>>

Permission denied (keyboard-interactive,publickey,password).
Connection closed

Is there any other way to acheive this because I cannot import pexcept
module .

Thanks,


>
>
> ---------- Forwarded message ----------
> From: David Rock <david at graniteweb.com>
> To: Tutor Python <tutor at python.org>
> Cc:
> Bcc:
> Date: Mon, 3 Dec 2018 18:12:05 -0600
> Subject: Re: [Tutor] sftp using subprocess
>
> > On Dec 3, 2018, at 16:01, Asad <asad.hasan2004 at gmail.com> wrote:
> >
> > Hi All ,
> >
> >        I am looking for a solution to automate downloading file using
> sftp
> > process using :
> >
> > batch.txt :contains the following two lines
> > cd 12345678
> > get 12345678_1.zip
> >
> >
> > import subprocess
> > host="abc.com"
> > user="sa"
> > user_host="%s@%s" % (user, host)
> > sftp_process = subprocess.Popen(['sftp', '-b', 'batch.txt', user_host],
> > shell=False)
> >
> > however it gives error even unable to provide password .
>
> batch mode for sftp assumes you don?t need a password.  From the manage:
>
>      -b batchfile
>              Batch mode reads a series of commands from an input batchfile
> instead of stdin.
>              Since it lacks user interaction it should be used in
> conjunction with non-inter?
>              active authentication.
>
> You would need to set up an ssh key with a null passphrase, or set up an
> ssh-agent in order for that to work.
>
> Another option would be to use pexpect to automate console input
>
>
> ?
> David Rock
> david at graniteweb.com
>
>
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: "Steven D'Aprano" <steve at pearwood.info>
> To: tutor at python.org
> Cc:
> Bcc:
> Date: Tue, 4 Dec 2018 11:12:56 +1100
> Subject: Re: [Tutor] sftp using subprocess
> On Tue, Dec 04, 2018 at 03:31:53AM +0530, Asad wrote:
>
> > however it gives error even unable to provide password .
>
> Is the error a secret? Would you like us to guess what it is?
>
>
> --
> Steve
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
Asad Hasan
+91 9582111698

From smravikumardonbosco at gmail.com  Mon Dec  3 20:31:26 2018
From: smravikumardonbosco at gmail.com (Ravi Kumar)
Date: Mon, 3 Dec 2018 19:31:26 -0600
Subject: [Tutor] Regarding Python api script
In-Reply-To: <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
Message-ID: <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>

My output api calls in python is in xml so I want the output to be in the
database

So is there a way where I can alter my code to get api responses in json
format in python and then later   move the output   to  the database

Basically at the end of it I want the all the data to be in database

Thanks

From alan.gauld at yahoo.co.uk  Tue Dec  4 03:36:50 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 08:36:50 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
Message-ID: <pu5e6v$r69$1@blaine.gmane.org>

On 04/12/2018 01:31, Ravi Kumar wrote:
> My output api calls in python is in xml so I want the output to be in the
> database

I'm still not 100% clear but I think you are saying
that your API currently returns XML when you call it
from Python. And you want to extract the data from
the XML and store it in a SQL Server database.
Is that correct?

If so the sequence of events is:

1) Connect to web site and retrieve the data (It
   sounds like you can already do this bit)
2) Parse the XML into usable data - you need
   etree (or another parser) to do that
3) Store the data in the database. You need
   a DBAPI module to do that. You will need
   to know some SQL.

Also, for step 3 you will need to either create
a new database or use an existing one. Do you
already have a target database? Or do you need
to design and create the table(s)? That may
require more SQL.

> So is there a way where I can alter my code to get
> api responses in json format in python and then
> later move the output to  the database

The answer to that lies in the API.
Assuming it is not under your control then you need
to read the API documentation to see if there is an
option to change the output to JSON. If not you are
stuck with whatever the API gives you.

Before we can give you any more help you will need
to share some code with us. What have you got
so far?

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Tue Dec  4 03:41:03 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 08:41:03 +0000
Subject: [Tutor] sftp using subprocess
In-Reply-To: <CAG3LsKFgn3rtxikpCsFTvdVBbAmGYDdn3U3-NE-AOA33bj+3Gw@mail.gmail.com>
References: <mailman.1488.1543882389.25467.tutor@python.org>
 <CAG3LsKFgn3rtxikpCsFTvdVBbAmGYDdn3U3-NE-AOA33bj+3Gw@mail.gmail.com>
Message-ID: <pu5ees$ejf$1@blaine.gmane.org>

On 04/12/2018 06:25, Asad wrote:

> Permission denied (keyboard-interactive,publickey,password).
> Connection closed
> 
> Is there any other way to acheive this because I cannot import pexcept
> module .

That looks like the problem that David already highlighted
with sftp.

When you say you cannot import pexpect, is that because
you are not allowed to? Or because of some technical issue?
If so what? We might be able to help fix that.

Also, you do realize that pexpect is not part of the
standard library? You need to install it first.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From david at graniteweb.com  Tue Dec  4 12:36:17 2018
From: david at graniteweb.com (David Rock)
Date: Tue, 4 Dec 2018 11:36:17 -0600
Subject: [Tutor] sftp using subprocess
In-Reply-To: <pu5ees$ejf$1@blaine.gmane.org>
References: <mailman.1488.1543882389.25467.tutor@python.org>
 <CAG3LsKFgn3rtxikpCsFTvdVBbAmGYDdn3U3-NE-AOA33bj+3Gw@mail.gmail.com>
 <pu5ees$ejf$1@blaine.gmane.org>
Message-ID: <012E7B40-E5F3-4B23-AF76-6FD2FB5CE389@graniteweb.com>


> On Dec 4, 2018, at 02:41, Alan Gauld via Tutor <tutor at python.org> wrote:
> 
> On 04/12/2018 06:25, Asad wrote:
> 
>> Permission denied (keyboard-interactive,publickey,password).
>> Connection closed
>> 
>> Is there any other way to acheive this because I cannot import pexcept
>> module .
> 
> That looks like the problem that David already highlighted
> with sftp.

Yup.  So, back to my other question regarding ssh key access?

Do you have the ability to ssh to an account on the target system (i.e., use ssh to log in, not use sftp to grab a file)?
If you do, then the ?easiest? solution is to create an ssh key and set up passwordless authentication.  If you do not, then maybe you can upload the key through some web service.  What is the actual target system to which you are trying to connect?  Is there an admin that can help configure ssh key-based access for you?

If you can?t set up key access, then you will need to automate sending the login information (hence the pexpect recommendation, since this is a python list, after all).  If you can?t install pexpect, you may have expect available (on what OS are you running your script?); but writing an expect script is starting to drift away from a python tutor list a bit.

If you are on linux, you may want to forego python altogether if you can?t get the ssh key access enabled and use a bash here document for automation.  
http://www.tldp.org/LDP/abs/html/here-docs.html

The disadvantage to this (and to using expect, for that matter) is plaintext storage of credentials on your system, but it?s pretty simple to use.


? 
David Rock
david at graniteweb.com





From asad.hasan2004 at gmail.com  Tue Dec  4 13:05:33 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Tue, 4 Dec 2018 23:35:33 +0530
Subject: [Tutor] sftp using subprocess
In-Reply-To: <mailman.10.1543942802.26085.tutor@python.org>
References: <mailman.10.1543942802.26085.tutor@python.org>
Message-ID: <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>

Hi All ,

           I am not allowed to import pexcept   .Therefore only option I
have is to implement a solution using the standard libraries in python .

            However I am using subprocess.Popen for sftp I am unable to
pass the password.

I tried the following :

>>> import subprocess
>>> subprocess.Popen(['sftp','username at server.com', 'stop'],
stdin=PIPE).communicate(input='abc')


Enter password for username
Password:

  Here I am unable to proceed , I am sure someone would have cracked this
problem earlier if they can share the code

Thanks in advance,       .

On Tue, Dec 4, 2018 at 10:30 PM <tutor-request at python.org> wrote:

> Send Tutor mailing list submissions to
>         tutor at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/tutor
> or, via email, send a message with subject or body 'help' to
>         tutor-request at python.org
>
> You can reach the person managing the list at
>         tutor-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tutor digest..."
> Today's Topics:
>
>    1. Re: sftp using subprocess (Alan Gauld)
>
>
>
> ---------- Forwarded message ----------
> From: Alan Gauld <alan.gauld at yahoo.co.uk>
> To: tutor at python.org
> Cc:
> Bcc:
> Date: Tue, 4 Dec 2018 08:41:03 +0000
> Subject: Re: [Tutor] sftp using subprocess
> On 04/12/2018 06:25, Asad wrote:
>
> > Permission denied (keyboard-interactive,publickey,password).
> > Connection closed
> >
> > Is there any other way to acheive this because I cannot import pexcept
> > module .
>
> That looks like the problem that David already highlighted
> with sftp.
>
> When you say you cannot import pexpect, is that because
> you are not allowed to? Or because of some technical issue?
> If so what? We might be able to help fix that.
>
> Also, you do realize that pexpect is not part of the
> standard library? You need to install it first.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
Asad Hasan
+91 9582111698

From jstapcot at gmail.com  Tue Dec  4 06:02:48 2018
From: jstapcot at gmail.com (James Stapleton-Cotton)
Date: Tue, 4 Dec 2018 13:02:48 +0200
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
 <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
Message-ID: <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>

Thank you Mats and Steven. I'm back on track now with a different tutorial
which has lead to me to the relevant coding tools.
https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
https://python.swaroopch.com

On Mon, Dec 3, 2018 at 4:04 PM Mats Wichmann <mats at wichmann.us> wrote:

> On 12/3/18 3:35 AM, James Stapleton-Cotton wrote:
> > Hello,
> >
> > On this page (
> >
> http://openbookproject.net/thinkcs/python/english3e/way_of_the_program.html
> )
> > - a book for learning Computer Science using Python - I am directed to (
> > http://code.google.com/p/pyscripter) in order to access the appropriate
> > program development environment, PyScripter. I have downloaded the latest
> > version of Python from Python.org and PyScripter from a site that I am
> > directed to from the original site mentioned in the beginning of this
> email
> > - (https://sourceforge.net/projects/pyscripter/files/), however I can't
> > seem to be able to run PyScripter on my Mac.
>
> it looks like from the project's own description that PyScripter is
> intended solely for the Windows environment.
>
> But there are a TON of other IDE products, free and commercial, out
> there, many of which do run on the Mac.
>
> This may be information overload, but try looking here:
>
> https://wiki.python.org/moin/IntegratedDevelopmentEnvironments
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From david at graniteweb.com  Tue Dec  4 13:30:55 2018
From: david at graniteweb.com (David Rock)
Date: Tue, 4 Dec 2018 12:30:55 -0600
Subject: [Tutor] sftp using subprocess
In-Reply-To: <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>
References: <mailman.10.1543942802.26085.tutor@python.org>
 <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>
Message-ID: <7CDE2CDC-2C43-464D-AA52-C62A85A389B0@graniteweb.com>


> On Dec 4, 2018, at 12:05, Asad <asad.hasan2004 at gmail.com> wrote:
> 
> Hi All ,
> 
>           I am not allowed to import pexcept   .Therefore only option I
> have is to implement a solution using the standard libraries in python .

I?m only suggesting it because it?s an easier way to interface with expect.  If you aren?t allowed to install it, then we?ll stop talking about it.

>            However I am using subprocess.Popen for sftp I am unable to
> pass the password.
> 

You still are not understanding my point.  using sftp in batch mode CAN?T USE a password.  If you look at the manpage for sftp, batch mode prohibits it.

     -b batchfile
             Batch mode reads a series of commands from an input batchfile instead of stdin.
             Since it lacks user interaction it should be used in conjunction with non-inter?
             active authentication.   <<?????

?non-interactive authentication? means you must use an ssh key to authenticate instead of a password.  In other words, trying to figure out how to send a login to the subprocess is not the issue; even if you can, it won?t work because sftp *in batch mode* will not accept it.  If you _can?t_ set up an ssh key for this access, then you MUST stop trying to use batch mode, because it will never work.

>  Here I am unable to proceed , I am sure someone would have cracked this
> problem earlier if they can share the code

There is no code for this because it is not possible to do within the constraints you have proposed up to this point.

1. Can you set up ssh key passwordless authentication?
   if yes, then do it and what you have now will start working
   if no, then you can?t use sftp the way you are currently trying (the -b ?batch mode?) and find a different solution (expect, here docs, etc).


? 
David Rock
david at graniteweb.com





From alan.gauld at yahoo.co.uk  Tue Dec  4 16:37:38 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 21:37:38 +0000
Subject: [Tutor] sftp using subprocess
In-Reply-To: <7CDE2CDC-2C43-464D-AA52-C62A85A389B0@graniteweb.com>
References: <mailman.10.1543942802.26085.tutor@python.org>
 <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>
 <7CDE2CDC-2C43-464D-AA52-C62A85A389B0@graniteweb.com>
Message-ID: <pu6ruv$t9k$1@blaine.gmane.org>

On 04/12/2018 18:30, David Rock wrote:
> you MUST stop trying to use batch mode, because it will never work.

But you could presumably use interactive mode via Popen
by taking control of the stdin/out pipes.

But then you are basically re writing expect!

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From david at graniteweb.com  Tue Dec  4 16:44:45 2018
From: david at graniteweb.com (David Rock)
Date: Tue, 4 Dec 2018 15:44:45 -0600
Subject: [Tutor] sftp using subprocess
In-Reply-To: <pu6ruv$t9k$1@blaine.gmane.org>
References: <mailman.10.1543942802.26085.tutor@python.org>
 <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>
 <7CDE2CDC-2C43-464D-AA52-C62A85A389B0@graniteweb.com>
 <pu6ruv$t9k$1@blaine.gmane.org>
Message-ID: <9E8B073F-5702-4B95-A61D-F277789DA526@graniteweb.com>


> On Dec 4, 2018, at 15:37, Alan Gauld via Tutor <tutor at python.org> wrote:
> 
> On 04/12/2018 18:30, David Rock wrote:
>> you MUST stop trying to use batch mode, because it will never work.
> 
> But you could presumably use interactive mode via Popen
> by taking control of the stdin/out pipes.
> 
> But then you are basically re writing expect!

Right.  and what?s worse, sftp doesn?t like here documents (by design, for fairly obvious reasons).  The best option (if it?s possible) really is setting up ssh key authentication.



? 
David Rock
david at graniteweb.com





From mats at wichmann.us  Tue Dec  4 17:49:41 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Tue, 4 Dec 2018 15:49:41 -0700
Subject: [Tutor] sftp using subprocess
In-Reply-To: <9E8B073F-5702-4B95-A61D-F277789DA526@graniteweb.com>
References: <mailman.10.1543942802.26085.tutor@python.org>
 <CAG3LsKF4CwUH=knT1qjN00DeaSAY5C0dK13oN=RZLh8WA95HJg@mail.gmail.com>
 <7CDE2CDC-2C43-464D-AA52-C62A85A389B0@graniteweb.com>
 <pu6ruv$t9k$1@blaine.gmane.org>
 <9E8B073F-5702-4B95-A61D-F277789DA526@graniteweb.com>
Message-ID: <9d0f512a-3210-c5c6-83a2-32c398e45830@wichmann.us>

On 12/4/18 2:44 PM, David Rock wrote:
> 
>> On Dec 4, 2018, at 15:37, Alan Gauld via Tutor <tutor at python.org> wrote:
>>
>> On 04/12/2018 18:30, David Rock wrote:
>>> you MUST stop trying to use batch mode, because it will never work.
>>
>> But you could presumably use interactive mode via Popen
>> by taking control of the stdin/out pipes.
>>
>> But then you are basically re writing expect!
> 
> Right.  and what?s worse, sftp doesn?t like here documents (by design, for fairly obvious reasons).  The best option (if it?s possible) really is setting up ssh key authentication.

yes, programs which need to prompt for a secret are intentionally
paranoid and hard to work with programmatically. often they flush the
input stream before prompting, so you have to synchronize things across
your pipe to get it to work. it's certainly a reason a program like
expect is non-trivial. you're really better off not trying to reinvent
this logic, as experts have spent a lot of time getting it to work
safely and correctly.

"I am sure someone would have cracked this problem earlier if they can
share the code".  Yup.  The problem is you're telling us you're not
allowed to use the solutions that exist - I presume if pexpect is out,
then other things will be as well?

Here's a project you could look at, though I'm expecting you can't use
it either...

https://bitbucket.org/dundeemt/pysftp/overview



From adameyring at gmail.com  Tue Dec  4 14:14:44 2018
From: adameyring at gmail.com (Adam Eyring)
Date: Tue, 4 Dec 2018 14:14:44 -0500
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
 <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
 <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>
Message-ID: <CAPStRW_XJze2LSf_BHscDQ-vtR52fv0M=rSVJJ+v+A4mCLTpEw@mail.gmail.com>

I haven't gone through many python books, but have been using a copy of
Automating the Boring Stuff with Python. It covers lists, dictionaries,
scraping data from websites, etc.
https://automatetheboringstuff.com/
The PDF is free.

Adam

On Tue, Dec 4, 2018 at 1:09 PM James Stapleton-Cotton <jstapcot at gmail.com>
wrote:

> Thank you Mats and Steven. I'm back on track now with a different tutorial
> which has lead to me to the relevant coding tools.
> https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
> https://python.swaroopch.com
>
> On Mon, Dec 3, 2018 at 4:04 PM Mats Wichmann <mats at wichmann.us> wrote:
>
> > On 12/3/18 3:35 AM, James Stapleton-Cotton wrote:
> > > Hello,
> > >
> > > On this page (
> > >
> >
> http://openbookproject.net/thinkcs/python/english3e/way_of_the_program.html
> > )
> > > - a book for learning Computer Science using Python - I am directed to
> (
> > > http://code.google.com/p/pyscripter) in order to access the
> appropriate
> > > program development environment, PyScripter. I have downloaded the
> latest
> > > version of Python from Python.org and PyScripter from a site that I am
> > > directed to from the original site mentioned in the beginning of this
> > email
> > > - (https://sourceforge.net/projects/pyscripter/files/), however I
> can't
> > > seem to be able to run PyScripter on my Mac.
> >
> > it looks like from the project's own description that PyScripter is
> > intended solely for the Windows environment.
> >
> > But there are a TON of other IDE products, free and commercial, out
> > there, many of which do run on the Mac.
> >
> > This may be information overload, but try looking here:
> >
> > https://wiki.python.org/moin/IntegratedDevelopmentEnvironments
> >
> >
> > _______________________________________________
> > Tutor maillist  -  Tutor at python.org
> > To unsubscribe or change subscription options:
> > https://mail.python.org/mailman/listinfo/tutor
> >
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From avigross at verizon.net  Tue Dec  4 14:31:46 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 4 Dec 2018 19:31:46 +0000 (UTC)
Subject: [Tutor] Borrowing free code
References: <788878217.1782528.1543951906972.ref@mail.yahoo.com>
Message-ID: <788878217.1782528.1543951906972@mail.yahoo.com>


David,

What does it mean when someone says they cannot install a module? I can see how a school assignment might require using only some limited set of functionality. I note some installations add more than others or let you designate optional components to include.

Some companies may have security concerns or portability considerations. The finished product may be used on machines lacking what is needed or even cut off from the ability to get them.

But some packages are simply python code that you can simply insert into your own python files. If they have few external dependencies and no need to compile C code, can you copy them without violating rules?

I am thinking for example of a module that defines a class you may want to subclass. You could find the source code you would have imported and take relevant parts into your main code file or into any local files you import. 

I do not know if the module being discussed meets the criteria or if this would not? be considered honest. But it is a step to be considered. Another thread I have seen related to a user with no internet connection for downloading. But sneakernet is trivial if the only thing needed for an install might be copying a file or three into any valid location on the machine including the directory your main program is in.

And, yes, this means you do not get updates if the module changes.? 

Just a thought.


Sent from AOL Mobile Mail
On Tuesday, December 4, 2018 David Rock <david at graniteweb.com> wrote:

> On Dec 4, 2018, at 12:05, Asad <asad.hasan2004 at gmail.com> wrote:
> 
> Hi All ,
> 
>? ? ? ? ? I am not allowed to import pexcept? .Therefore only option I
> have is to implement a solution using the standard libraries in python .

I?m only suggesting it because it?s an easier way to interface with expect.? If you aren?t allowed to install it, then we?ll stop talking about it.

>? ? ? ? ? ? However I am using subprocess.Popen for sftp I am unable to
> pass the password.
> 

You still are not understanding my point.? using sftp in batch mode CAN?T USE a password.? If you look at the manpage for sftp, batch mode prohibits it.

? ? -b batchfile
? ? ? ? ? ? Batch mode reads a series of commands from an input batchfile instead of stdin.
? ? ? ? ? ? Since it lacks user interaction it should be used in conjunction with non-inter?
? ? ? ? ? ? active authentication.? <<?????

?non-interactive authentication? means you must use an ssh key to authenticate instead of a password.? In other words, trying to figure out how to send a login to the subprocess is not the issue; even if you can, it won?t work because sftp *in batch mode* will not accept it.? If you _can?t_ set up an ssh key for this access, then you MUST stop trying to use batch mode, because it will never work.

>? Here I am unable to proceed , I am sure someone would have cracked this
> problem earlier if they can share the code

There is no code for this because it is not possible to do within the constraints you have proposed up to this point.

1. Can you set up ssh key passwordless authentication?
? if yes, then do it and what you have now will start working
? if no, then you can?t use sftp the way you are currently trying (the -b ?batch mode?) and find a different solution (expect, here docs, etc).


? 
David Rock
david at graniteweb.com




_______________________________________________
Tutor maillist? -? Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

From alan.gauld at yahoo.co.uk  Tue Dec  4 18:43:05 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Tue, 4 Dec 2018 23:43:05 +0000
Subject: [Tutor] Borrowing free code
In-Reply-To: <788878217.1782528.1543951906972@mail.yahoo.com>
References: <788878217.1782528.1543951906972.ref@mail.yahoo.com>
 <788878217.1782528.1543951906972@mail.yahoo.com>
Message-ID: <pu73a6$qhr$1@blaine.gmane.org>

On 04/12/2018 19:31, Avi Gross wrote:

> But some packages are simply python code that you can 
> simply insert into your own python files. 

If they are fully public domain that's probably true.
If they are copyright (even if open/free) you would be
potentially liable for prosecution since you are copying
someone else's work.

Even if it is open source then at the very least
you should include a comment to the effect that
the code is based on, say, M Palin's file parrot.py
or whatever.

> And, yes, this means you do not get updates if the module changes.? 

And this is a big drawback for any non trivial code
unless you are a significantly better programmer
than whoever wrote it in the first place. (Since
understanding somebody else's code is much harder
than understanding your own!)

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From david at graniteweb.com  Tue Dec  4 18:56:09 2018
From: david at graniteweb.com (David Rock)
Date: Tue, 4 Dec 2018 17:56:09 -0600
Subject: [Tutor] Borrowing free code
In-Reply-To: <788878217.1782528.1543951906972@mail.yahoo.com>
References: <788878217.1782528.1543951906972.ref@mail.yahoo.com>
 <788878217.1782528.1543951906972@mail.yahoo.com>
Message-ID: <7CF9C8CF-F346-431F-A49D-23AE532A35F4@graniteweb.com>


> On Dec 4, 2018, at 13:31, Avi Gross <avigross at verizon.net> wrote:
> 
> David,
> 
> What does it mean when someone says they cannot install a module? I can see how a school assignment might require using only some limited set of functionality. I note some installations add more than others or let you designate optional components to include.

In the context of what the OP posted, I don?t think it actually matters if they mean ?not allowed,? ?not internet accessible,? or something else.  The statement is enough to move on in the conversation to try and be productive rather than belabor the point.

That said, hopefully we will get some feedback on the OS and ssh setup options.

? 
David Rock
david at graniteweb.com





From steve at pearwood.info  Wed Dec  5 01:14:46 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 5 Dec 2018 17:14:46 +1100
Subject: [Tutor] Borrowing free code
In-Reply-To: <pu73a6$qhr$1@blaine.gmane.org>
References: <788878217.1782528.1543951906972.ref@mail.yahoo.com>
 <788878217.1782528.1543951906972@mail.yahoo.com>
 <pu73a6$qhr$1@blaine.gmane.org>
Message-ID: <20181205061446.GO13061@ando.pearwood.info>

On Tue, Dec 04, 2018 at 11:43:05PM +0000, Alan Gauld via Tutor wrote:
> On 04/12/2018 19:31, Avi Gross wrote:
> 
> > But some packages are simply python code that you can 
> > simply insert into your own python files. 
> 
> If they are fully public domain that's probably true.

Almost nothing younger than 70 years is in the public domain. Nearly all 
countries share certain copyright laws, including automatic granting of 
copyright for the life of the author + many years (exactly how many 
depends on the country) whether you want it or not.

According the lawyers behind the Creative Commons, many jurisdictions 
make it difficult if not impossible to voluntarily give up copyright and 
put a work in the public domain:

https://creativecommons.org/share-your-work/public-domain/cc0/

One of the few exceptions is that many (but not all) works by the US 
government are required to be placed in the public domain by law.

Having said that, all is not *quite* lost (yet). There are still ways to 
get equivalent freedom as if the work was in the public domain, and 
another Mickey Mouse Copyright Extension Act is unlikely to occur any 
time soon.


> If they are copyright (even if open/free) you would be
> potentially liable for prosecution since you are copying
> someone else's work.

That's a rather negative way of looking at it.

Copyright infringement is typically handled through the civil courts: 
the legal authorities don't press charges for copying (except, perhaps, 
in the case of large-scale piracy of manufactured goods like fake 
designer handbags, movie DVDs, etc). Rather, the copyright owner has to 
sue you. If the copyright owner is explicitly giving you the right to 
copy and modify the software, which is the case for FOSS (Free Open 
Source Software), then you have nothing to fear so long as you have a 
valid licence.

If you have a valid licence to use and copy the software, and you obey 
the licence terms, then you are in no danger of being prosecuted for 
copyright infringement because you are licenced to do so.


> Even if it is open source then at the very least
> you should include a comment to the effect that
> the code is based on, say, M Palin's file parrot.py
> or whatever.

Giving credit might be good from an ethical point of view, but it may 
not be either necessary or sufficient (depending on the licence).

Some examples: 

The MIT and BSD licences do not explicitly require you to credit the 
author, but they do require you to include a copy of the original 
author's copyright and licence in your work:

https://en.wikipedia.org/wiki/BSD_licenses
https://en.wikipedia.org/wiki/MIT_License

If you choose to give credit as a courtesy, that's fine, but you must be 
careful as to do so in such a way that you avoid giving the impression 
that the author endorses or is responsible for your work.

    Good: "MyApp, by me, with a big thank you to J. Cleese for 
    his spam library."

    Bad: "MyApp with spam by J. Cleese!"


On the other hand, the GPL requires more than just credit, it requires 
that (with some exceptions, including fair use) if you duplicate their 
code, that your code in turn must be released under the GPL as well. 
Merely giving credit is in no way sufficient.

And on the gripping hand, you have ultra-permissive public-domain like 
licences such as the creative commons CC0 or the Toybox licence:

https://en.wikipedia.org/wiki/Public-domain-equivalent_license

neither of which require credit be given.

Regardless of the licence, you must obey the conditions (whether they 
are minimal, as in the MIT licence, or onerous and difficult as in most 
proprietary licences) in order to be legally permitted to copy the work. 
But if you have a licence to copy, then of course you may copy according 
to the terms of the licence.

In a practical sense, copying trivial amounts of code from software 
licenced under the MIT licence, or similar, would be highly unlikely to 
get you sued. For starters, the author would have to know you copied it, 
and care, and track you down in real life, and sue you. And if the code 
snippet was small enough, you could defend on the basis of fair use.

(Although more and more jurisdictions are taking a hard-line, minimalist 
approach to fair use, allowing it effectively only for the purposes of 
satire. Or even not acknowledging a fair use right at all.)

Another factor may be that, regardless of the *actual* legal risk, some 
employers may not allow copying or use of (some or all) FOSS software 
even if licenced, because they may fear the unknown, or they believe 
anti-GPL propoganda ("its a virus that means you will have to give up 
your intellectual property"), ethical reasons (copying without 
attribution may be plagiarism), or possibly even for legitimate legal 
reasons ("the GPL is not compatible with our licence").

Bottom line: yes, you can legally copy FOSS software, under certain 
conditions, and need not fear being prosecuted.

(Actually, you could even copy closed-source proprietary software, if 
you have a licence allowing you to do so.)


-- 
Steve

From avigross at verizon.net  Tue Dec  4 19:50:21 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 4 Dec 2018 19:50:21 -0500
Subject: [Tutor] Copyleft
Message-ID: <005e01d48c34$801a9200$804fb600$@verizon.net>

Alan,

Just a reminder. I am NOT suggesting the first way to solve a problem is what I described. I am saying that sometimes a roadblock (real or imagined) can be gotten around using methods like that.

Agreed. There are copyrights that have to be considered, albeit often SMALL amounts are considered OK. Attribution is a good idea in general.

The problem that began this is when someone puts roadblocks in the way of what works.

If someone for example wants to be able to use something as trivial as math.pi but for some reason won't load the math module, you could suggest they look up the value of pi to umpteen decimal places and create an object called math Perhaps a class) containing nothing but a value called pi ....

This might even satisfy other code they have imported that expects to be able to invoke math.pi. But if that other code does an import too, then you might need to create an (almost) empty file called math.py containing pi=3.1415926535...

Just to make the rest work.

Or, I suggest you can copy the real math module into the same place. You may want to edit it with a comment.

But, yes, bad example as pretty much any implementation of python will include the math module as it is standard.

Your post does remind me that there are many versions of copyright including some with the silly naming of "copyleft" that allow rather free use but with limits. They may charge large businesses or government agencies. They may demand you leave in some text as in comments or not make changes or notify them to get permission so they can keep track or ...

I brought up some questions a while ago that boiled down to how a program can know what is already available on the machine the code is being run on. Clearly if you use an unfamiliar package, one solution would be to bring the needed files with you in the distribution of the program. Yes, it may not be the latest or greatest, but you can imagine many ways where it can conditionally fall back on the copies included if a regular import fails. 

This may be one reason someone would be asked to limit their code to the standard distribution. 



-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of Alan Gauld via Tutor
Sent: Tuesday, December 4, 2018 6:43 PM
To: tutor at python.org
Subject: Re: [Tutor] Borrowing free code

On 04/12/2018 19:31, Avi Gross wrote:

> But some packages are simply python code that you can simply insert 
> into your own python files.

If they are fully public domain that's probably true.
If they are copyright (even if open/free) you would be potentially liable for prosecution since you are copying someone else's work.

Even if it is open source then at the very least you should include a comment to the effect that the code is based on, say, M Palin's file parrot.py or whatever.

> And, yes, this means you do not get updates if the module changes.

And this is a big drawback for any non trivial code unless you are a significantly better programmer than whoever wrote it in the first place. (Since understanding somebody else's code is much harder than understanding your own!)

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From jstapcot at gmail.com  Tue Dec  4 18:52:47 2018
From: jstapcot at gmail.com (James Stapleton-Cotton)
Date: Wed, 5 Dec 2018 01:52:47 +0200
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <CAPStRW_XJze2LSf_BHscDQ-vtR52fv0M=rSVJJ+v+A4mCLTpEw@mail.gmail.com>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
 <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
 <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>
 <CAPStRW_XJze2LSf_BHscDQ-vtR52fv0M=rSVJJ+v+A4mCLTpEw@mail.gmail.com>
Message-ID: <CAMDhPpePwGqSVZWQsypmfA5stRa4OkZ0AbwE7AKc099E5z+kqg@mail.gmail.com>

Thank you Adam.
I am currently going through the book, 'A Byte of Python' - found here -
https://python.swaroopch.com.
On pages 29-30 it writes about 'Using A Source File', creating a file
called '/tmp/py' on a Mac OS using the 'bash-terminal' and running the
saved PyCharm file, 'hello.py'. I type 'mkdir /tmp/py' into the terminal
and press return. This supposedly creates a new folder. I type 'mkdir
/tmp/py' a second time and press return as so:

USERs-MacBook-Pro:~ user$ mkdir /tmp/py

mkdir: /tmp/py: File exists


This shows that the file is indeed created and 'exists'. In the next
instructed step (typing 'python hello.py' into the terminal) there shows an
error as such:

USERs-MacBook-Pro:~ user$ mkdir /tmp/py

mkdir: /tmp/py: File exists

USERs-MacBook-Pro:~ user$ python hello.py
python: can't open file 'hello.py': [Errno 2] No such file or directory

I previously created a python file called 'hello.py' using the PyCharmEdu
editor (as instructed). This file is saved in my 'Documents' in a folder
called 'Pycharm Projects' [I tried changing the location of the file to
'/tmp/py', however this created an error during my 'print' function -
print("hello world") - in the editor, therefore I changed the location back
to the original, 'PycharmProjects'.]

I have found the /tmp/py folder within my 'Users' folder. I dragged the
'hello.py' file into the tmp folder and attempted the above process of
'running' the 'python hello.py' file from the terminal, however the same
result occurred.

Could you please help clarify where I am going wrong in the instructed
process given by the book mentioned above.

Regards

On Tue, Dec 4, 2018 at 9:16 PM Adam Eyring <adameyring at gmail.com> wrote:

> I haven't gone through many python books, but have been using a copy of
> Automating the Boring Stuff with Python. It covers lists, dictionaries,
> scraping data from websites, etc.
> https://automatetheboringstuff.com/
> The PDF is free.
>
> Adam
>
> On Tue, Dec 4, 2018 at 1:09 PM James Stapleton-Cotton <jstapcot at gmail.com>
> wrote:
>
>> Thank you Mats and Steven. I'm back on track now with a different tutorial
>> which has lead to me to the relevant coding tools.
>> https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
>> https://python.swaroopch.com
>>
>> On Mon, Dec 3, 2018 at 4:04 PM Mats Wichmann <mats at wichmann.us> wrote:
>>
>> > On 12/3/18 3:35 AM, James Stapleton-Cotton wrote:
>> > > Hello,
>> > >
>> > > On this page (
>> > >
>> >
>> http://openbookproject.net/thinkcs/python/english3e/way_of_the_program.html
>> > )
>> > > - a book for learning Computer Science using Python - I am directed
>> to (
>> > > http://code.google.com/p/pyscripter) in order to access the
>> appropriate
>> > > program development environment, PyScripter. I have downloaded the
>> latest
>> > > version of Python from Python.org and PyScripter from a site that I am
>> > > directed to from the original site mentioned in the beginning of this
>> > email
>> > > - (https://sourceforge.net/projects/pyscripter/files/), however I
>> can't
>> > > seem to be able to run PyScripter on my Mac.
>> >
>> > it looks like from the project's own description that PyScripter is
>> > intended solely for the Windows environment.
>> >
>> > But there are a TON of other IDE products, free and commercial, out
>> > there, many of which do run on the Mac.
>> >
>> > This may be information overload, but try looking here:
>> >
>> > https://wiki.python.org/moin/IntegratedDevelopmentEnvironments
>> >
>> >
>> > _______________________________________________
>> > Tutor maillist  -  Tutor at python.org
>> > To unsubscribe or change subscription options:
>> > https://mail.python.org/mailman/listinfo/tutor
>> >
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>

From matthew.polack at htlc.vic.edu.au  Tue Dec  4 19:39:49 2018
From: matthew.polack at htlc.vic.edu.au (Matthew Polack)
Date: Wed, 5 Dec 2018 11:39:49 +1100
Subject: [Tutor] Any 'graphical' ways of learning Python
Message-ID: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>

Hi All,

We're using Python with our Year 9 and 10 students to teach programming.
I've started with basic console programming...doing simple games like a
quiz game etc.

Students always seem to like 'graphics'..one of the reasons things like
'Scratch' are so popular in schools is because of  the ready made GUI.

My concern with things like 'Scratch' and 'Game Engines' is that perhaps
kids can miss out on  learning core fundamentals...but can appreciate some
visuals can be very motivating...

Can anyone recommend any ways of integrating 'graphics' but in a simpler
way.

I've had some experience with TKinter...(which is still quite a jump for
beginners)....also can see that 'Pygame' might offer some of this...

also have seen some tutorials on Udemy that encourages using the 'Turtle'.

Does anything else come to find for helping kids stay engaged when they
start to get tired of just text based console programming?

Thanks for any suggestions.


Matthew Polack | Teacher

-- 
**Disclaimer: *Whilst every attempt has been made to ensure that material 
contained in this email is free from computer viruses or other defects, the 
attached files are provided, and may only be used, on the basis that the 
user assumes all responsibility for use of the material transmitted. This 
email is intended only for the use of the individual or entity named above 
and may contain information that is confidential and privileged. If you are 
not the intended recipient, please note that any dissemination, 
distribution or copying of this email is strictly prohibited. If you have 
received this email in error, please notify us immediately by return email 
or telephone +61 3 5382 2529**?and destroy the original message.*

From alan.gauld at yahoo.co.uk  Wed Dec  5 03:37:25 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 5 Dec 2018 08:37:25 +0000
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
References: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
Message-ID: <pu82k1$l1d$1@blaine.gmane.org>

On 05/12/2018 00:39, Matthew Polack wrote:

> Can anyone recommend any ways of integrating 'graphics' but in a simpler
> way.
> 

Have you considered the turtle module.

Its limited to drawing shapes but does give some immediate results.
You can of course create functions to draw more sophisticated shapes
etc. And you write the code in an ordinary IDE/editor so its a good
transition from fully graphical scratch to fully text mode Python.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Wed Dec  5 03:44:14 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 5 Dec 2018 08:44:14 +0000
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <CAMDhPpePwGqSVZWQsypmfA5stRa4OkZ0AbwE7AKc099E5z+kqg@mail.gmail.com>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
 <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
 <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>
 <CAPStRW_XJze2LSf_BHscDQ-vtR52fv0M=rSVJJ+v+A4mCLTpEw@mail.gmail.com>
 <CAMDhPpePwGqSVZWQsypmfA5stRa4OkZ0AbwE7AKc099E5z+kqg@mail.gmail.com>
Message-ID: <pu830r$39k$1@blaine.gmane.org>

On 04/12/2018 23:52, James Stapleton-Cotton wrote:

> USERs-MacBook-Pro:~ user$ python hello.py
> python: can't open file 'hello.py': [Errno 2] No such file or directory

You need to provide the full path to the Python file.

> I previously created a python file called 'hello.py' using the PyCharmEdu
> editor (as instructed). This file is saved in my 'Documents' in a folder
> called 'Pycharm Projects' 

So you would have needed to type

user$ python "~/Documents/Pycharm Projects/hello.py"

> to the original, 'PycharmProjects'.]

Note that here you omit the space.
The OS is very particular about such details, you need
to get it exactly right. (Although in bash using the
tab key will usually fill things in the way you need it)

> 'hello.py' file into the tmp folder and attempted the above process of
> 'running' the 'python hello.py' file from the terminal, however the same
> result occurred.

Again you needthe patg:

user$ python /tmp/py/hello.py

The other option is to change directory to where the code is and then
just specify the name:

user$ cd /tmp/py
user$ python hello.py

Should work too.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Wed Dec  5 04:01:08 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 5 Dec 2018 09:01:08 +0000
Subject: [Tutor] Borrowing free code
In-Reply-To: <20181205061446.GO13061@ando.pearwood.info>
References: <788878217.1782528.1543951906972.ref@mail.yahoo.com>
 <788878217.1782528.1543951906972@mail.yahoo.com>
 <pu73a6$qhr$1@blaine.gmane.org> <20181205061446.GO13061@ando.pearwood.info>
Message-ID: <pu840h$p1o$1@blaine.gmane.org>

On 05/12/2018 06:14, Steven D'Aprano wrote:
> On Tue, Dec 04, 2018 at 11:43:05PM +0000, Alan Gauld via Tutor wrote:
>> On 04/12/2018 19:31, Avi Gross wrote:
>>
>>> But some packages are simply python code that you can 
>>> simply insert into your own python files. 
>>
>> If they are fully public domain that's probably true.
> 
> Almost nothing younger than 70 years is in the public domain. 

True and anything not public domain throws up lots
of issues as you point out! :-)

> Copyright infringement is typically handled through the civil courts: 

You say that as if "civil courts" were somehow not like
higher courts but in these parts there is no distinction
except in the processes used. They hold the same weight
and its all considered criminal activity.

> If you have a valid licence to use and copy the software, and you obey 
> the licence terms, then you are in no danger of being prosecuted for 
> copyright infringement because you are licenced to do so.

But it depends on the copyright. You may be free to use
it provided you include the original copyright/license
notice, or you may only need to give accreditation
(possibly in some fixed format), or maybe you can just use it.
Its all very murky and you need a lawyer to sort it out.

And that's really the point I was making.
Avi's suggestion is technically correct but if used
in any piece of production software can lead to a
world of pain.

> Regardless of the licence, you must obey the conditions

And that was my point.
It's misleading to say on a beginners list that you can
just copy code into your own files. Its much more
complicated than that.

> In a practical sense, copying trivial amounts of code from software 
> licenced under the MIT licence, or similar, would be highly unlikely to 
> get you sued.

Agreed, especially if its for private or limited use. If released into
the wider market then its a differentstory.
And some companies (I know of two at least) actively police
the market (Git hub and source forge inclyded) for corporate
code being abused(even if it's been GPL'd)

> Another factor may be that, regardless of the *actual* legal risk, some 
> employers may not allow copying or use of (some or all) FOSS software 
> even if licenced, because they may fear the unknown,

Again true and was the case in my old company.
To use FOSS code we had to jump through several
hoops with the company lawyers.
It had to be a major bit of work to make that worthwhile!

> Bottom line: yes, you can legally copy FOSS software, under certain 
> conditions, and need not fear being prosecuted.

Yes, the point being you need to find out what the conditions are.
And there are many permutations.

> (Actually, you could even copy closed-source proprietary software, if 
> you have a licence allowing you to do so.)

True.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Wed Dec  5 03:32:24 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 5 Dec 2018 08:32:24 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
Message-ID: <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>

CCing the list, please use Reply All when responding to the tutor list.


On 05/12/2018 03:44, Ravi Kumar wrote:
> Yes? thats right I want to extract the xml and store into database(SQL
> Server) and I will have to cteate a new table
>
> Here is the sample output I am getting similarly there bulk data which
> I want to store in database format is as shown below
>
>
> b'[{"deviceSerial":"Q2XD-333X-G8Q8","occurredAt":1537565.640085,"type":"802.11
> association","details":{"radio":"1","vap":"0","clientMac":"C8:21:6679:B6:16","channel":"44","rssi":"57","aid":"31645681"}},{"deviceSerial":"Q2XD-97gX-G8Q8","occurredAt":153765.700095,"type":"WPA
> deauthentication","details":{"radio":"1","vap":"0","clientMac":"C621:58:79:B6:16","aid":"316681"}},{"deviceSerial":"Q2XD-97gX-G8Q8","occurredAt":1563369.780085,"type":"WPA
> deauthentication","details":{"radio":"1","vap":"0","clientMac":"C8:21:58:9:B6:16","aid":"31645681"}},{"deviceSerial":"Q297JX-G8Q8"
>
>
> Please let me whether this can be formatted into json in the code

It looks to me like it is already in JSON format.

It is a list of objects.

The json module will let you extract the objects into Python data objects.


> and also any other tips or links to refer if available? to store in
> database

Do you know SQL?

If so using the Python DBAPI is very easy.

If not you will need to learn some basic SQL.

You can try reading the database topic in my Python tutorial(see below)
which is based on SQLite rather than SQL Server but should be 90%
compatible.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


From steve at pearwood.info  Wed Dec  5 09:08:45 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 01:08:45 +1100
Subject: [Tutor] Copyleft
In-Reply-To: <005e01d48c34$801a9200$804fb600$@verizon.net>
References: <005e01d48c34$801a9200$804fb600$@verizon.net>
Message-ID: <20181205140844.GP13061@ando.pearwood.info>

Avi,

Why do you keep changing the subject line? You started a new thread 
about "borrowing" (copying) code, people responded to that thread, and 
you've immediately changed the subject to "Copyleft" even though you 
talk about much more than copyleft licences.

(You barely mention copyleft at all -- and what you say about it is 
very inaccurate.)


On Tue, Dec 04, 2018 at 07:50:21PM -0500, Avi Gross wrote:
> Alan,
> 
> Just a reminder. I am NOT suggesting the first way to solve a problem 
> is what I described. I am saying that sometimes a roadblock (real or 
> imagined) can be gotten around using methods like that.

In my experience, the sorts of organisations which make it difficult to 
install third-party software (whether FOSS or proprietary) are also the 
sorts of organisations which will fire you immediately for copying 
someone else's code.

YMMV.


> Agreed. There are copyrights that have to be considered, albeit often 
> SMALL amounts are considered OK. Attribution is a good idea in 
> general.

There's another factor to consider: software patents. If you're going to 
copy a small amount of code, you may still violate their patent. 
(Disclaimer: I am not a patent lawyer.) In that case, giving credit to 
them will be tantamount to admitting to willful infringement, which in 
most jurisdictions that I know of receives triple damages.

That's how broken the patent system is: the penalties for infringement 
encourage people to avoid doing a patent search. Its better to say "We 
couldn't be bothered doing a patent search" than to say "We tried to do 
due diligence but failed to spot this patent", because when it comes to 
patents, if you make the effort but fail, you are deemed to have 
willfully infringed, whereas if you intentionally make no attempt to 
avoid infringing, your infringement is deemed to have been an innocent 
mistake.

Crazy but true.



> The problem that began this is when someone puts roadblocks in the way of what works.

There may be very good reasons for this policy that you call a 
roadblock.

And we only have the Original Poster's word that he is not allowed 
to install pexpect. That could be his misunderstanding, or perhaps he 
can't be bothered going through the process to get approval.

In fairness, some places do make that process rather gruelling... in a 
previous position, I worked with a client whose change request process 
for installing new software on a production server could easily take 
three or four months for approval to come through, if it was given at 
all. Now, you might be tempted to roll your eyes and call that 
bureaucracy gone mad, but they had very good reasons for that level of 
caution and conservativeness.

A colleague got a figurative spanking after he accidentally brought 
their production server down (fortunately only for ten minutes, although 
that was ten minutes too long) by rolling out a buggy "bug fix" directly 
onto the production server without testing it on a staging server first, 
or getting approval for the change.

https://i.chzbgr.com/full/7756111616/hED5FBCC9/


 
> If someone for example wants to be able to use something as trivial as 
> math.pi but for some reason won't load the math module, you could 
> suggest they look up the value of pi to umpteen decimal places and 
> create an object called math Perhaps a class) containing nothing but a 
> value called pi ....

[...]
> Or, I suggest you can copy the real math module into the same place. 
> You may want to edit it with a comment.

So rather than adding a single constant, you're going to risk infringing 
the copyright and moral rights of other programmers by copying a 75 
kilobyte C library that you don't need, just to get the value of pi 
which you could legally and easily copy from your calculator or Google 
in about five seconds.

Are you being paid by the hour? *wink*


> But, yes, bad example as pretty much any implementation of python will 
> include the math module as it is standard.

Indeed. Let's take a better example: the pexpect library.

The OP is not able (not permitted? not physically able? can't be 
bothered?) to install the pexpect library. So your advice for him is 
to... install the pexpect library, potentially ignoring its licence 
terms, making a copy of the source (hoping that its all Python and not C 
that needs compiling) and edit the source to "add a comment" (saying 
what?).

I think you haven't really considered that if the OP is not able to 
install the pexpect library, he probably won't be able to copy it into 
his application either. To say nothing of the ethical issues of copying 
software without obeying the licence terms.

Fortunately pexpect itself is licenced under a very permissive licence 
that allows copying with (almost) no restriction:

https://github.com/pexpect/pexpect/blob/master/LICENSE

but as a general matter of principle, suggesting people just copy the 
source code they want and stick it in their application is rather risky 
and potentially dubious advice.

Not risky for you, of course, its no skin off your nose if somebody 
takes your advice and gets sued or fired from their job.



> Your post does remind me that there are many versions of copyright 
> including some with the silly naming of "copyleft" that allow rather 
> free use but with limits.

That's incorrect: there's only one "copyright" (although of course each 
nation has its own laws enforcing it, and may grant its own slightly 
different monopoly rights under the copyright banner).

"Copyleft" is, of course, a pun on copyright. But it isn't "a version of 
copyright". Only governments can create copyrights. It is a software 
licence.

Copyright is a government-granted, and enforced, monopoly on the legal 
right to make copies of a work (among other privileges).

A licence, on the other hand, is an agreement between two private 
parties where the copyright owner agrees to permit the other party to do 
certain things which they otherwise would not be allowed to do. Such as 
making copies or modifications to the software.



> They may charge large businesses or 
> government agencies. They may demand you leave in some text as in 
> comments or not make changes or notify them to get permission so they 
> can keep track or ...

This is a completely inaccurate description of what copyleft licences 
do. Any licence which required such conditions would not be copyleft. It 
is pretty much as far from the copyleft ethos as it is possible to get.

(Aside from "some text" if that is a copy of the licence itself.)



> I brought up some questions a while ago that boiled down to how a 
> program can know what is already available on the machine the code is 
> being run on. Clearly if you use an unfamiliar package, one solution 
> would be to bring the needed files with you in the distribution of the 
> program. Yes, it may not be the latest or greatest, but you can 
> imagine many ways where it can conditionally fall back on the copies 
> included if a regular import fails.

How could a "regular import" (as opposed to what?) succeed *except* by 
means of having a copy of the library available somewhere accessible to 
the application?

The only distinction here is whether the library is installed within the 
application itself, using whatever mechanism is used to bundle libraries 
within an application, or installed externally to the application. 
Either way, there must be a copy.

In my experience, it is usual that bundling a library bundled in an 
application will shadow the system library, rather than being used as a 
fallback if and only if the system library is missing. Things might be 
different in the C world, or other languages, but that's how the Python 
import system typically works.


For example, try running this at the shell:

touch math.py
python -c "import math; print(math.pi)"


Believe it or not, that's a feature, not a bug :-)



-- 
Steve

From steve at pearwood.info  Wed Dec  5 09:13:16 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 01:13:16 +1100
Subject: [Tutor] Beginners Book, Python and PyScripter
In-Reply-To: <pu830r$39k$1@blaine.gmane.org>
References: <CAMDhPpcMpJCB0QthqQuQPC0AgC-h-S_wZvf29jorW_qTS4tGXw@mail.gmail.com>
 <efcb9550-f8f8-2b9a-c141-9012c0a8327e@wichmann.us>
 <CAMDhPpe59ay50ZVtjUu6yyN0_sKMHmF5Ghw6afUhkAr0umV02g@mail.gmail.com>
 <CAPStRW_XJze2LSf_BHscDQ-vtR52fv0M=rSVJJ+v+A4mCLTpEw@mail.gmail.com>
 <CAMDhPpePwGqSVZWQsypmfA5stRa4OkZ0AbwE7AKc099E5z+kqg@mail.gmail.com>
 <pu830r$39k$1@blaine.gmane.org>
Message-ID: <20181205141315.GQ13061@ando.pearwood.info>

On Wed, Dec 05, 2018 at 08:44:14AM +0000, Alan Gauld via Tutor wrote:
> On 04/12/2018 23:52, James Stapleton-Cotton wrote:
> 
> > USERs-MacBook-Pro:~ user$ python hello.py
> > python: can't open file 'hello.py': [Errno 2] No such file or directory
> 
> You need to provide the full path to the Python file.
> 
> > I previously created a python file called 'hello.py' using the PyCharmEdu
> > editor (as instructed). This file is saved in my 'Documents' in a folder
> > called 'Pycharm Projects' 
> 
> So you would have needed to type
> 
> user$ python "~/Documents/Pycharm Projects/hello.py"

To be absolutely clear here, the "user$" part is the prompt, you don't 
type that.



-- 
Steve

From steve at pearwood.info  Wed Dec  5 09:28:49 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 01:28:49 +1100
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
References: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
Message-ID: <20181205142848.GR13061@ando.pearwood.info>

On Wed, Dec 05, 2018 at 11:39:49AM +1100, Matthew Polack wrote:
> Hi All,
> 
> We're using Python with our Year 9 and 10 students to teach programming.

Yay! And I see you're a fellow Aussie :-)


> I've started with basic console programming...doing simple games like a
> quiz game etc.
> 
> Students always seem to like 'graphics'..one of the reasons things like
> 'Scratch' are so popular in schools is because of  the ready made GUI.

Indeed.

Alas, graphics is one of Python's weaknesses, especially at the 
beginners' level.

The easiest to use will be the turtle module, although its fairly 
limited in what it does. Its basically a simulation of the turtle 
from the Logo programming language. Logo was cutting edge about half a 
century ago.

You might try PyGame, although I don't know how easy it will be for you 
to pre-install it for the students, or how difficult it is to use (I've 
never used it myself).

https://www.pygame.org/


Another option is PySimpleGUI:

https://pypi.org/project/PySimpleGUI/




-- 
Steve

From adameyring at gmail.com  Wed Dec  5 09:02:21 2018
From: adameyring at gmail.com (Adam Eyring)
Date: Wed, 5 Dec 2018 09:02:21 -0500
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <pu82k1$l1d$1@blaine.gmane.org>
References: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
 <pu82k1$l1d$1@blaine.gmane.org>
Message-ID: <CAPStRW8shTyaEY=gvBE1aM5sjhGLYq5w+GeZhqEMynOAtNGhCA@mail.gmail.com>

I've liked turtle and make my graphing more interesting by asking for user
input such as dimensions, then graph automatically.

One starter source for using pygame graphics is
https://inventwithpython.com/pygame/
It jumps into game writing very quickly, but provides explanations of
commands.

The one I'm working on now is using the arcade library developed by a
python college professor:
http://arcade.academy
Many many examples of code (including videos of student projects) are
provided to help you write games that resemble many of our favorites.


On Wed, Dec 5, 2018, 3:40 AM Alan Gauld via Tutor <tutor at python.org> wrote:

> On 05/12/2018 00:39, Matthew Polack wrote:
>
> > Can anyone recommend any ways of integrating 'graphics' but in a simpler
> > way.
> >
>
> Have you considered the turtle module.
>
> Its limited to drawing shapes but does give some immediate results.
> You can of course create functions to draw more sophisticated shapes
> etc. And you write the code in an ordinary IDE/editor so its a good
> transition from fully graphical scratch to fully text mode Python.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

On Dec 5, 2018 3:40 AM, "Alan Gauld via Tutor" <tutor at python.org> wrote:

On 05/12/2018 00:39, Matthew Polack wrote:

> Can anyone recommend any ways of integrating 'graphics' but in a simpler
> way.
>

Have you considered the turtle module.

Its limited to drawing shapes but does give some immediate results.
You can of course create functions to draw more sophisticated shapes
etc. And you write the code in an ordinary IDE/editor so its a good
transition from fully graphical scratch to fully text mode Python.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

From avigross at verizon.net  Wed Dec  5 11:22:35 2018
From: avigross at verizon.net (Avi Gross)
Date: Wed, 5 Dec 2018 11:22:35 -0500
Subject: [Tutor] Borrowing restricted code
Message-ID: <007201d48cb6$bbaf2060$330d6120$@verizon.net>

I don't want to start (and continue) another sideways discussion. Alan and
Steven (not Stephen) both made very good points and helped clarify my
understanding.

I am NOT advocating copying code. Not even "free" code.

I am saying there may be times you want to package the code for special
purposes.

Perhaps someone can enlighten me on a subtle aspect here.

What does it mean to include something?

If we return to the days of yesteryear, albeit much of it exists this year,
a major programming paradigm was to compile a language like C or PASCAL into
some machine language or a binary form. But you only wrote a part of the
final program and inserted parts from what had been written, mainly, by
others.

One inclusion in languages like C was variants of lines like this:

#include "stdio.h"

What would happen was when a phase of the compile process read your file and
saw a request to include, it would pause, try to find the file in the usual
places, and then copy the text and continue. This could be recursive if that
file included another. It is in one sense similar to how "import" works in
python or "library" in R and similar features in other languages.

What was different was that one large file was created which was a flattened
version of this tree of code. Another pass of the compiler would then
process it. Nothing was RUN until later and only if there were no
compilation errors. Along the way, another form of inclusion had to happen.
If your code (directly or indirectly) called functions not defined in your
own code, and not already compiled in, it had to search assorted libraries
of such code, find the segments needed, and link them in. Eventually, it got
even worse and shared libraries would be mounted in memory and when your
code ran, it would link in what was needed.

The above is not meant to be a precise description but indicates that your
code was rarely all your own. You could literally set it up so if someone
else left their code unguarded, you could arrange to grab text or library
code into your finished executable.

In python, you can generally have access to the python code of what you can
import or at least to the byte code. But there may be different rules
attached to several categories.

If you use "import" you don't so much copy as USE the code. I mean the
interpreter pauses evaluating the current file and opens the one you asked
for and reads it as if it were your code. Some actions are transient. A line
like "5+3" evaluates to 8 and you ignore it. But many and perhaps most lines
end up creating instantiations of objects (yes, classes are also objects)
which then sit in memory. Functions are also objects and pretty much contain
within them everything you need to reconstruct the function if you know
where to look. Is what is copied really different than the original text
used? 

If you import the base modules that come with python, can you assume it is
freely given with few strings other than the kind discussed? If it is
outside but available from a standard download location, perhaps you need to
see the documentation for each item. If it is sold to you, I suspect you
need to be very careful about including any.

Back to the main topic, for me. What kind of uses might be considered legal
or at least not worth prosecuting? The law in various countries likely
differs, Alan and I are definitely using different legal systems but
possibly not as different as where Asad is. (His phone number would be in
India.) Clearly if you make a product and sell it, then borrowing code can
be a serious thing. If you are a student doing a homework assignment and
will never use it again, few might care especially if you use small amounts
and don't claim it is your own work.

But the question I would like answered is DOES IT MAKE A DIFFERENCE how you
borrow the use of the code? 

We talked about the normal intended method:

Load the file into a place on your local machine.
Write one or more python files and one of them imports it.

A second method I suggested WHEN NEEDED AND ALLOWED is to find the code you
need, perhaps on another machine where you can download it, or in some other
public repository. Copy the minimum you need into your own code, with
perhaps enough attribution and explanation.

Now what about a third way? No edit needed. Simply use a program like "cat"
to concatenate multiple files including the one with your code into a bigger
lump. Feed that to the python interpreter.  Included would be the code you
want. True, the code would now all be in one namespace. But is this form of
copying all that much different than was intended? 

I am not going to continue in any further discussion. But I do think this
could be useful to people learning python as often the best way to find out
how to do something is to study the code of someone who did it. You might
want to play with it and see if you can improve it or vary some part ...

So it would be good to know what can be done legally and what to avoid.

As an aside, I note that in C, the included files had no concept of a
namespace so for efficiency, it could make sense to gather all your code
into a simple file. In python, there may be non-trivial overhead in
searching for all the files to import and opening and evaluating each. But I
can still see how putting every single function or class definition in
another file in some huge tree may be overkill and make for a slow runtime.
If you end up importing "*" all the time so it is all in your global
namespace, why not bring them into the main program. I am, of course,
talking about your own code.








From breamoreboy at gmail.com  Wed Dec  5 14:49:58 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Wed, 5 Dec 2018 19:49:58 +0000
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
Message-ID: <pu9a13$nqq$1@blaine.gmane.org>

On 05/12/2018 16:22, Avi Gross wrote:

[huge snip]

Please take yourself to another forum, your ramblings have no place on 
the *PYTHON TUTOR* mailing list.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From sarfraaz at gmail.com  Wed Dec  5 12:43:37 2018
From: sarfraaz at gmail.com (Sarfraaz Ahmed)
Date: Wed, 5 Dec 2018 23:13:37 +0530
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
References: <CAFkpTinu6h_URRRn5rvZ6Y94t0eqAU6LNRWMgLQoCi66mGLApA@mail.gmail.com>
Message-ID: <CAKvdLD7dusZ4SbeJL=C5M26K9DDVkj7JDT9CBm6pAb5mrza1Rg@mail.gmail.com>

Hello Matthew,

Although, its not for Graphics, I have noticed that
http://www.pythontutor.com is a good place that can come in handy for
students who are new to programming.

It gives a graphical view of how memory is allocated for variables and how
functions are invoked in a program.

I have seen students strengthen their fundamental programming concepts
through the visual representation of code flow on this website.

You could try this as well.

On Wed, Dec 5, 2018 at 1:54 PM Matthew Polack <
matthew.polack at htlc.vic.edu.au> wrote:

> Hi All,
>
> We're using Python with our Year 9 and 10 students to teach programming.
> I've started with basic console programming...doing simple games like a
> quiz game etc.
>
> Students always seem to like 'graphics'..one of the reasons things like
> 'Scratch' are so popular in schools is because of  the ready made GUI.
>
> My concern with things like 'Scratch' and 'Game Engines' is that perhaps
> kids can miss out on  learning core fundamentals...but can appreciate some
> visuals can be very motivating...
>
> Can anyone recommend any ways of integrating 'graphics' but in a simpler
> way.
>
> I've had some experience with TKinter...(which is still quite a jump for
> beginners)....also can see that 'Pygame' might offer some of this...
>
> also have seen some tutorials on Udemy that encourages using the 'Turtle'.
>
> Does anything else come to find for helping kids stay engaged when they
> start to get tired of just text based console programming?
>
> Thanks for any suggestions.
>
>
> Matthew Polack | Teacher
>
> --
> **Disclaimer: *Whilst every attempt has been made to ensure that material
> contained in this email is free from computer viruses or other defects,
> the
> attached files are provided, and may only be used, on the basis that the
> user assumes all responsibility for use of the material transmitted. This
> email is intended only for the use of the individual or entity named above
> and may contain information that is confidential and privileged. If you
> are
> not the intended recipient, please note that any dissemination,
> distribution or copying of this email is strictly prohibited. If you have
> received this email in error, please notify us immediately by return email
> or telephone +61 3 5382 2529** and destroy the original message.*
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
        Thanks
               -- Sarfraaz Ahmed

From steve at pearwood.info  Wed Dec  5 19:45:43 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 11:45:43 +1100
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
Message-ID: <20181206004541.GS13061@ando.pearwood.info>

On Wed, Dec 05, 2018 at 11:22:35AM -0500, Avi Gross wrote:

> I am NOT advocating copying code. Not even "free" code.
> 
> I am saying there may be times you want to package the code for special
> purposes.

"Packaging" the code IS copying the code.


> Perhaps someone can enlighten me on a subtle aspect here.
> 
> What does it mean to include something?

https://en.wiktionary.org/wiki/include


[... description of how #include works in C ...]
> The above is not meant to be a precise description but indicates that your
> code was rarely all your own. You could literally set it up so if someone
> else left their code unguarded, you could arrange to grab text or library
> code into your finished executable.

The precise details of what an individual programming language means by 
"including" code will, naturally, depend on the language in question.

This is not really the place to go into a detailed description of all 
the possible variations (this is supposed to be a *Python* forum after 
all) but briefly there are at least two kinds of code which we might 
"include": source code and compiled object code.

When "including" source code, the interpreter might read a command like 
"load spam", look up a file "spam", read the contents into memory, parse 
it and run it through the interpreter as if it had been copied and 
pasted into the original file in place of the "load" line.

When "including" object code, the compiler (or often a separate program 
called a linker) can make a copy of the object code and insert that code 
directly into the object code it is generating. This is called "static 
linking".

An alternative is to leave the object code where it is, but instead 
insert instructions for how to access it at runtime. This is called 
"dynamic linking".

https://kb.iu.edu/d/akqn

Python's import is (I think) closer to dynamic linking than the others.


> In python, you can generally have access to the python code of what you can
> import or at least to the byte code. But there may be different rules
> attached to several categories.
> 
> If you use "import" you don't so much copy as USE the code. I mean the
> interpreter pauses evaluating the current file and opens the one you asked
> for and reads it as if it were your code.

That is not entirely wrong, but its not quite right either.

The action of the import command is surprisingly complex, and so I may 
have got some subtle details wrong. But the high-level overview of what 
the interpreter does when you run "import spam" is as follows.

1. Look for an already loaded module "spam"; if the interpreter finds 
one, it creates an new variable called "spam" in the current namespace, 
and assigns that module object to that name. The import is complete.

# Pseudo-code
if "spam" in sys.modules:
    spam = sys.modules["spam"]
    return


2. If no such already loaded module, then search a set of known 
locations for a library called "spam":

# Pseudo-code
for location in sys.path:
    for filename in os.listdir(location):
        name, ext = path.splitext(filename)
        if name == "spam":
            if ext == ".py":
                read the source code from spam.py into memory
                parse it into bytecode into memory
                write out the bytecode to spam.pyc
            elif ext == ".pyc":
                read the bytecode from spam.pyc into memory
            else:
                # other cases handled here, e.g. packages, zip files,
                # C libraries (.dll or .so), other extensions etc.
            # Assuming we get to here...
            create a module object in memory
            run that bytecode, using that module object as the namespace
            cache the module object in sys.modules['spam']
            spam = module object
            return
# no such "spam" module or package found
raise ImportError


So you can see that modules are only executed the first time the 
interpreter imports them, not on subsequent imports.

There really isn't a distinction to make between code treated "as if it 
were your code" and other code. All code is treated the same.

How could it not be? The interpreter cannot know which modules or 
libraries you wrote, which were collaborative efforts between you and 
other people, and which were written by other people.



> Some actions are transient. A line
> like "5+3" evaluates to 8 and you ignore it. But many and perhaps most lines
> end up creating instantiations of objects (yes, classes are also objects)
> which then sit in memory.

Or get used and then disposed of by the garbage collector.


> Functions are also objects and pretty much contain
> within them everything you need to reconstruct the function if you know
> where to look. Is what is copied really different than the original text
> used? 

Yes.


> If you import the base modules that come with python, can you assume it is
> freely given with few strings other than the kind discussed?

There is no need to assume anything, you can read the licence. When you 
start the interactive interpeter, it says:

Type "help", "copyright", "credits" or "license" for more information.


so the licence is always at your fingertips.

If that's too much trouble, then you can trust the millions of other 
people who use Python every day, and believe them when they say the 
interpreter and the standard library are available under a permissive 
licence.

Unless you are distributing a copy of Python or its libraries to others, 
you don't need to care about the details. That's the point of using a 
permissive licence.


[...]
> Back to the main topic, for me. What kind of uses might be considered legal
> or at least not worth prosecuting?

Under the current copyright maximalist domain, virtually nothing is 
legal unless you are the copyright owner, or have a licence from the 
copyright owner. (Disclaimer: I am not a lawyer.)

You may have the right to *read* the code (if it isn't a Trade Secret), 
but that's about as far as it goes. In principle, copying the code in 
any way (whether by a literal "copy and paste" into your text editor, or 
by merely re-implementing the code from memory after reading it) could 
be deemed to be either copyright infringement or plagiarism or both.

Especially under the current academic standards of zero-tolerance for 
so-called "plagiarism" in the US, which are extreme and hypocritical. 
But that's off-topic.

There's a theoretical "Fair Use" right in some countries (but not in 
Australia!) that could allow you to copy small, trivial chunks, perhaps 
as much as 10% of the work, under certain circumstances, but lawyers 
LOVE arguing about what is or isn't Fair Use because such arguments can 
go on and on for months.

There are also theoretical arguments that the code in question was so 
trivial that there should be no copyright on it in the first place. Just 
as you can't copyright the sentence "I ate a spam sandwich" in isolation 
(only as part of a more substantial work), so you can't copyright a 
trivial isolated line of code. Again, lawyers love arguing about what is 
or isn't trivial.

In principle, you might even argue independent invention. If you 
happened to come up with substantially the same work independently, 
perhaps because there is only a single obvious way to do something, then 
no copying too place and so you may be deemed to have not infringed 
copyright. (But independent invention is no defence against patent 
infringement.)

So you can see why many organisations are so paranoid about having 
licences for every line of code they use. Failure to be fully licenced 
could be *incredibly* time-consuming and expensive if they get into a 
legal dispute. The only way to win is to avoid getting into a legal 
dispute in the first place.

As for what is "not worth prosecuting", there are no copyright police 
who troll the internet looking for copied lines of code. Nobody is going 
to be scanning your Github repos looking for infringement (at least not 
yet, and I'm sure that by now the anti-plagiarism software industry is 
looking to do something like that...)

There generally needs to be a civil complaint where the copyright owner 
needs to sue you.

In the USA, the TSA and immigration do look for infringing software (at 
least sometimes) but they mostly care about commercial-scale piracy. If 
the customs official is feeling especially obnoxious and/or diligent, 
they might demand to know why you don't have a Windows licence key on 
your laptop[1] or poke around looking for pirated videos on behalf of 
Hollywood and co. Deep pockets speak loudly.

But they're not going to open up your Python folder and demand to see 
licences for everything or question whether or not you copy code from 
Stackoverflow without permission.


> The law in various countries likely
> differs, Alan and I are definitely using different legal systems but
> possibly not as different as where Asad is. (His phone number would be in
> India.)

Most countries in the world abide by the Berne convention on 
copyrights, so probably not as different as you may be imagining.

India, in particular, is a signatory to the Berne Convention, the 
Universal Copyright Convention of both Geneva and Paris, and the TRIPS 
agreement (but not the WIPO Copyright Treaty) so the differences will be 
relatively minor.

https://en.wikipedia.org/wiki/Berne_Convention

https://en.wikipedia.org/wiki/List_of_parties_to_international_copyright_agreements



> Clearly if you make a product and sell it, then borrowing code can
> be a serious thing. 

Indeed. Most FOSS (Free Open Source Software) projects turn a blind eye 
to trivial non-compliance unless it involves commercial products. Even 
Richard Stallman is probably not going to chase you for copying a few 
lines of GPLed software


> If you are a student doing a homework assignment and
> will never use it again, few might care especially if you use small amounts
> and don't claim it is your own work.

I wouldn't be so sure about that, especially in US macademia. (Warning: 
may contain nuts.) Their rules for plagiarism are insanely strict and 
exceedingly hypocritical, because the macademics enforcing those rules 
against students don't even come close to living up to the same 
standards themselves.


> But the question I would like answered is DOES IT MAKE A DIFFERENCE how you
> borrow the use of the code? 

Not legally. Copying is copying, whether you cut and paste in a text 
editor, drag and drop a file from one disk to another, download it from 
the Pirate Bay, or retype it from memory.


> We talked about the normal intended method:
> 
> Load the file into a place on your local machine.

That's a copy. If you are not permitted to make that copy, you are 
already infringing right there.


> Write one or more python files and one of them imports it.
> 
> A second method I suggested WHEN NEEDED AND ALLOWED is to find the code you
> need, perhaps on another machine where you can download it, or in some other
> public repository. Copy the minimum you need into your own code, with
> perhaps enough attribution and explanation.

That's a copy. If you are not permitted to make that copy, you are
already infringing right there.

In addition, "enough attribution" may not be sufficient to avoid charges 
of plagiarism.

Example: I've seen at least one case of a student given a reprimand for 
plagiarism after giving a one-line quote, acknowledging the author, but 
failing to give a formal reference including page number. 



> Now what about a third way? No edit needed. Simply use a program like "cat"
> to concatenate multiple files

That's a copy. If you are not permitted to make that copy, you are
already infringing right there.

(How do you get the copy of the file you pass to cat, if not by making a 
copy?)






[1] Because it is unthinkable that you might be running some OS other 
than Windows, one which doesn't require a licence key.


-- 
Steve

From steve at pearwood.info  Wed Dec  5 22:58:03 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 14:58:03 +1100
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <pu9a13$nqq$1@blaine.gmane.org>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
 <pu9a13$nqq$1@blaine.gmane.org>
Message-ID: <20181206035803.GT13061@ando.pearwood.info>

On Wed, Dec 05, 2018 at 07:49:58PM +0000, Mark Lawrence wrote:
> On 05/12/2018 16:22, Avi Gross wrote:
> 
> [huge snip]
> 
> Please take yourself to another forum, your ramblings have no place on 
> the *PYTHON TUTOR* mailing list.

Steady on Mark, a lot of what Avi says is misinformed or close to 
stream-of-consciousness irrelevant chatter (no offense Avi, but you do 
go on and on with no apparent point sometimes), but some of it is 
relevant to Python programming.

Every programmer who distributes or copies software needs to have at 
least a basic understanding of copyright and licencing.

We allow discussions about IDEs or the best graphical libraries, and 
they too are only peripherally related to Python. But they are related, 
just as licencing and copyright are.



-- 
Steve

From alan.gauld at yahoo.co.uk  Thu Dec  6 05:41:37 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Thu, 6 Dec 2018 10:41:37 +0000
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <20181206004541.GS13061@ando.pearwood.info>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
 <20181206004541.GS13061@ando.pearwood.info>
Message-ID: <puau8u$jmo$1@blaine.gmane.org>

On 06/12/2018 00:45, Steven D'Aprano wrote:

> So you can see why many organisations are so paranoid about having 
> licences for every line of code they use. Failure to be fully licenced 
> could be *incredibly* time-consuming and expensive if they get into a 
> legal dispute. The only way to win is to avoid getting into a legal 
> dispute in the first place.
> 
> As for what is "not worth prosecuting", there are no copyright police 
> who troll the internet looking for copied lines of code. Nobody is going 
> to be scanning your Github repos looking for infringement (at least not 

Sorry, that's not strictly true. I know of at least two large companies
who have full time teams whose job is to trawl Github, sourceforge
and a few others looking at new checkins for unlicensed use of
corporate code. And one of those teams is not even a technical
team, they are corporate lawyers... And they do prosecute (or at least
threaten to).

> But they're not going to open up your Python folder and demand to see 
> licences for everything or question whether or not you copy code from 
> Stackoverflow without permission.

Again bodies like FAST(*) certainly do that (with police cooperation
of course - they need a search warrant). But they have been known to
litigate and fines of several thousand pounds have been issued to
infringers(?) But FAST is rarely interested in FOSS software its
commercial code they worry about. And usually it's unlicensed
binary code but they will pursue source code infringements if
asked. But having been on the receiving end of a FAST raid its an
unnerving experience (even though they didn't find anything).

(*)I wasn't sure if FAST are still active because I haven't
heard of any big prosecutions for about 10 years but their web
site suggests they are still very much around.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Thu Dec  6 06:38:07 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 6 Dec 2018 22:38:07 +1100
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <puau8u$jmo$1@blaine.gmane.org>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
 <20181206004541.GS13061@ando.pearwood.info> <puau8u$jmo$1@blaine.gmane.org>
Message-ID: <20181206113807.GU13061@ando.pearwood.info>

On Thu, Dec 06, 2018 at 10:41:37AM +0000, Alan Gauld via Tutor wrote:
> On 06/12/2018 00:45, Steven D'Aprano wrote:
>
> > As for what is "not worth prosecuting", there are no copyright police 
> > who troll the internet looking for copied lines of code. Nobody is going 
> > to be scanning your Github repos looking for infringement (at least not 
> 
> Sorry, that's not strictly true. I know of at least two large companies
> who have full time teams whose job is to trawl Github, sourceforge
> and a few others looking at new checkins for unlicensed use of
> corporate code. And one of those teams is not even a technical
> team, they are corporate lawyers... And they do prosecute (or at least
> threaten to).

Ah, that's my error. When I said "copyright police", I was actually 
talking about *actual* law enforcement, not private companies. Sorry for 
not being more clear.


> > But they're not going to open up your Python folder and demand to see 
> > licences for everything or question whether or not you copy code from 
> > Stackoverflow without permission.
> 
> Again bodies like FAST(*) certainly do that (with police cooperation
> of course - they need a search warrant).

Again, I was specifically talking about customs agents.

I don't know who FAST is, but if they're like the Business Software 
Alliance they're effectively a front for Microsoft to strong-arm 
companies into buying Microsoft software under the threat of copyright 
infringement lawsuits.

https://en.wikipedia.org/wiki/BSA_%28The_Software_Alliance%29

If they're not in the pocket of Microsoft as the BSA is (was?) then 
they're in the pocket of their members. They're not White Knight 
palladins searching for pirated software out of their sense of ethical 
outrage, they're doing it because they're paid to.


> But they have been known to
> litigate and fines of several thousand pounds have been issued to
> infringers(?) But FAST is rarely interested in FOSS software its
> commercial code they worry about. 

Point of order: FOSS *is* commercial code (or at least can be). Just ask 
Red Hat, Apple, Mozilla and even Microsoft.

I think you mean that it is *proprietary* code they care about, and 
even then, I daresay they only litigate on behalf of their clients.



-- 
Steve

From akleider at sonic.net  Thu Dec  6 11:39:06 2018
From: akleider at sonic.net (Alex Kleider)
Date: Thu, 06 Dec 2018 08:39:06 -0800
Subject: [Tutor] Borrowing restricted code
In-Reply-To: <20181206004541.GS13061@ando.pearwood.info>
References: <007201d48cb6$bbaf2060$330d6120$@verizon.net>
 <20181206004541.GS13061@ando.pearwood.info>
Message-ID: <565ff762c948b5f822cf2bd6b5cbfcb8@sonic.net>

On 2018-12-05 16:45, Steven D'Aprano wrote:
> On Wed, Dec 05, 2018 at 11:22:35AM -0500, Avi Gross wrote:
> 

Those following this thread might like to google "code V2"- the book by 
Laurence Lessig is relevant to many if not all of the disagreements.  
It's also a very interesting read (and free.)

>> I am NOT advocating copying code. Not even "free" code.
>> 
>> I am saying there may be times you want to package the code for 
>> special
>> purposes.
> 
> "Packaging" the code IS copying the code.
> 
> 
>> Perhaps someone can enlighten me on a subtle aspect here.
>> 
>> What does it mean to include something?
> 
> https://en.wiktionary.org/wiki/include
> 
> 
> [... description of how #include works in C ...]
>> The above is not meant to be a precise description but indicates that 
>> your
>> code was rarely all your own. You could literally set it up so if 
>> someone
>> else left their code unguarded, you could arrange to grab text or 
>> library
>> code into your finished executable.
> 
> The precise details of what an individual programming language means by
> "including" code will, naturally, depend on the language in question.
> 
> This is not really the place to go into a detailed description of all
> the possible variations (this is supposed to be a *Python* forum after
> all) but briefly there are at least two kinds of code which we might
> "include": source code and compiled object code.
> 
> When "including" source code, the interpreter might read a command like
> "load spam", look up a file "spam", read the contents into memory, 
> parse
> it and run it through the interpreter as if it had been copied and
> pasted into the original file in place of the "load" line.
> 
> When "including" object code, the compiler (or often a separate program
> called a linker) can make a copy of the object code and insert that 
> code
> directly into the object code it is generating. This is called "static
> linking".
> 
> An alternative is to leave the object code where it is, but instead
> insert instructions for how to access it at runtime. This is called
> "dynamic linking".
> 
> https://kb.iu.edu/d/akqn
> 
> Python's import is (I think) closer to dynamic linking than the others.
> 
> 
>> In python, you can generally have access to the python code of what 
>> you can
>> import or at least to the byte code. But there may be different rules
>> attached to several categories.
>> 
>> If you use "import" you don't so much copy as USE the code. I mean the
>> interpreter pauses evaluating the current file and opens the one you 
>> asked
>> for and reads it as if it were your code.
> 
> That is not entirely wrong, but its not quite right either.
> 
> The action of the import command is surprisingly complex, and so I may
> have got some subtle details wrong. But the high-level overview of what
> the interpreter does when you run "import spam" is as follows.
> 
> 1. Look for an already loaded module "spam"; if the interpreter finds
> one, it creates an new variable called "spam" in the current namespace,
> and assigns that module object to that name. The import is complete.
> 
> # Pseudo-code
> if "spam" in sys.modules:
>     spam = sys.modules["spam"]
>     return
> 
> 
> 2. If no such already loaded module, then search a set of known
> locations for a library called "spam":
> 
> # Pseudo-code
> for location in sys.path:
>     for filename in os.listdir(location):
>         name, ext = path.splitext(filename)
>         if name == "spam":
>             if ext == ".py":
>                 read the source code from spam.py into memory
>                 parse it into bytecode into memory
>                 write out the bytecode to spam.pyc
>             elif ext == ".pyc":
>                 read the bytecode from spam.pyc into memory
>             else:
>                 # other cases handled here, e.g. packages, zip files,
>                 # C libraries (.dll or .so), other extensions etc.
>             # Assuming we get to here...
>             create a module object in memory
>             run that bytecode, using that module object as the 
> namespace
>             cache the module object in sys.modules['spam']
>             spam = module object
>             return
> # no such "spam" module or package found
> raise ImportError
> 
> 
> So you can see that modules are only executed the first time the
> interpreter imports them, not on subsequent imports.
> 
> There really isn't a distinction to make between code treated "as if it
> were your code" and other code. All code is treated the same.
> 
> How could it not be? The interpreter cannot know which modules or
> libraries you wrote, which were collaborative efforts between you and
> other people, and which were written by other people.
> 
> 
> 
>> Some actions are transient. A line
>> like "5+3" evaluates to 8 and you ignore it. But many and perhaps most 
>> lines
>> end up creating instantiations of objects (yes, classes are also 
>> objects)
>> which then sit in memory.
> 
> Or get used and then disposed of by the garbage collector.
> 
> 
>> Functions are also objects and pretty much contain
>> within them everything you need to reconstruct the function if you 
>> know
>> where to look. Is what is copied really different than the original 
>> text
>> used?
> 
> Yes.
> 
> 
>> If you import the base modules that come with python, can you assume 
>> it is
>> freely given with few strings other than the kind discussed?
> 
> There is no need to assume anything, you can read the licence. When you
> start the interactive interpeter, it says:
> 
> Type "help", "copyright", "credits" or "license" for more information.
> 
> 
> so the licence is always at your fingertips.
> 
> If that's too much trouble, then you can trust the millions of other
> people who use Python every day, and believe them when they say the
> interpreter and the standard library are available under a permissive
> licence.
> 
> Unless you are distributing a copy of Python or its libraries to 
> others,
> you don't need to care about the details. That's the point of using a
> permissive licence.
> 
> 
> [...]
>> Back to the main topic, for me. What kind of uses might be considered 
>> legal
>> or at least not worth prosecuting?
> 
> Under the current copyright maximalist domain, virtually nothing is
> legal unless you are the copyright owner, or have a licence from the
> copyright owner. (Disclaimer: I am not a lawyer.)
> 
> You may have the right to *read* the code (if it isn't a Trade Secret),
> but that's about as far as it goes. In principle, copying the code in
> any way (whether by a literal "copy and paste" into your text editor, 
> or
> by merely re-implementing the code from memory after reading it) could
> be deemed to be either copyright infringement or plagiarism or both.
> 
> Especially under the current academic standards of zero-tolerance for
> so-called "plagiarism" in the US, which are extreme and hypocritical.
> But that's off-topic.
> 
> There's a theoretical "Fair Use" right in some countries (but not in
> Australia!) that could allow you to copy small, trivial chunks, perhaps
> as much as 10% of the work, under certain circumstances, but lawyers
> LOVE arguing about what is or isn't Fair Use because such arguments can
> go on and on for months.
> 
> There are also theoretical arguments that the code in question was so
> trivial that there should be no copyright on it in the first place. 
> Just
> as you can't copyright the sentence "I ate a spam sandwich" in 
> isolation
> (only as part of a more substantial work), so you can't copyright a
> trivial isolated line of code. Again, lawyers love arguing about what 
> is
> or isn't trivial.
> 
> In principle, you might even argue independent invention. If you
> happened to come up with substantially the same work independently,
> perhaps because there is only a single obvious way to do something, 
> then
> no copying too place and so you may be deemed to have not infringed
> copyright. (But independent invention is no defence against patent
> infringement.)
> 
> So you can see why many organisations are so paranoid about having
> licences for every line of code they use. Failure to be fully licenced
> could be *incredibly* time-consuming and expensive if they get into a
> legal dispute. The only way to win is to avoid getting into a legal
> dispute in the first place.
> 
> As for what is "not worth prosecuting", there are no copyright police
> who troll the internet looking for copied lines of code. Nobody is 
> going
> to be scanning your Github repos looking for infringement (at least not
> yet, and I'm sure that by now the anti-plagiarism software industry is
> looking to do something like that...)
> 
> There generally needs to be a civil complaint where the copyright owner
> needs to sue you.
> 
> In the USA, the TSA and immigration do look for infringing software (at
> least sometimes) but they mostly care about commercial-scale piracy. If
> the customs official is feeling especially obnoxious and/or diligent,
> they might demand to know why you don't have a Windows licence key on
> your laptop[1] or poke around looking for pirated videos on behalf of
> Hollywood and co. Deep pockets speak loudly.
> 
> But they're not going to open up your Python folder and demand to see
> licences for everything or question whether or not you copy code from
> Stackoverflow without permission.
> 
> 
>> The law in various countries likely
>> differs, Alan and I are definitely using different legal systems but
>> possibly not as different as where Asad is. (His phone number would be 
>> in
>> India.)
> 
> Most countries in the world abide by the Berne convention on
> copyrights, so probably not as different as you may be imagining.
> 
> India, in particular, is a signatory to the Berne Convention, the
> Universal Copyright Convention of both Geneva and Paris, and the TRIPS
> agreement (but not the WIPO Copyright Treaty) so the differences will 
> be
> relatively minor.
> 
> https://en.wikipedia.org/wiki/Berne_Convention
> 
> https://en.wikipedia.org/wiki/List_of_parties_to_international_copyright_agreements
> 
> 
> 
>> Clearly if you make a product and sell it, then borrowing code can
>> be a serious thing.
> 
> Indeed. Most FOSS (Free Open Source Software) projects turn a blind eye
> to trivial non-compliance unless it involves commercial products. Even
> Richard Stallman is probably not going to chase you for copying a few
> lines of GPLed software
> 
> 
>> If you are a student doing a homework assignment and
>> will never use it again, few might care especially if you use small 
>> amounts
>> and don't claim it is your own work.
> 
> I wouldn't be so sure about that, especially in US macademia. (Warning:
> may contain nuts.) Their rules for plagiarism are insanely strict and
> exceedingly hypocritical, because the macademics enforcing those rules
> against students don't even come close to living up to the same
> standards themselves.
> 
> 
>> But the question I would like answered is DOES IT MAKE A DIFFERENCE 
>> how you
>> borrow the use of the code?
> 
> Not legally. Copying is copying, whether you cut and paste in a text
> editor, drag and drop a file from one disk to another, download it from
> the Pirate Bay, or retype it from memory.
> 
> 
>> We talked about the normal intended method:
>> 
>> Load the file into a place on your local machine.
> 
> That's a copy. If you are not permitted to make that copy, you are
> already infringing right there.
> 
> 
>> Write one or more python files and one of them imports it.
>> 
>> A second method I suggested WHEN NEEDED AND ALLOWED is to find the 
>> code you
>> need, perhaps on another machine where you can download it, or in some 
>> other
>> public repository. Copy the minimum you need into your own code, with
>> perhaps enough attribution and explanation.
> 
> That's a copy. If you are not permitted to make that copy, you are
> already infringing right there.
> 
> In addition, "enough attribution" may not be sufficient to avoid 
> charges
> of plagiarism.
> 
> Example: I've seen at least one case of a student given a reprimand for
> plagiarism after giving a one-line quote, acknowledging the author, but
> failing to give a formal reference including page number.
> 
> 
> 
>> Now what about a third way? No edit needed. Simply use a program like 
>> "cat"
>> to concatenate multiple files
> 
> That's a copy. If you are not permitted to make that copy, you are
> already infringing right there.
> 
> (How do you get the copy of the file you pass to cat, if not by making 
> a
> copy?)
> 
> 
> 
> 
> 
> 
> [1] Because it is unthinkable that you might be running some OS other
> than Windows, one which doesn't require a licence key.

From mike_barnett at hotmail.com  Thu Dec  6 12:22:55 2018
From: mike_barnett at hotmail.com (Mike Barnett)
Date: Thu, 6 Dec 2018 17:22:55 +0000
Subject: [Tutor] Any 'graphical' ways of learning Python
Message-ID: <MWHPR11MB17108CB14624CD48891131AEE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>

I stumbled onto this mailing list when Google found a mention of PySimpleGUI in a thread titled: Any 'graphical' ways of learning Python.

I wanted to respond with information on PySimpleGUI and why it's ideal for students.

If you're teaching kids and they want to learn how to do GUIs so that their programs run and look like Windows programs, then the PySimpleGUI package is the right choice.  It runs on top of either tkinter or Qt.  With tkinter it's easy to also package up the program into an EXE file that they kids can give to friends.  There are instructions in the PySimpleGUI documentation located at http://www.PySimpleGUI.org.

The GitHub http://www.PySimpleGUI.com has over 130 demo programs and there is a Cookbook filled with Recipes that you can copy, pate and run. There is a section with a bunch of exercises that are part of a course that an Aussie teacher developed.  You'll find them posted here:
https://github.com/MikeTheWatchGuy/PySimpleGUI/tree/master/ProgrammingClassExamples

There is extensive documentation for the package http://www.PySimpleGUI.org

The experience for the first time user was designed to be a positive one, filled with success, with the goal of having a GUI up on the screen within FIVE MINUTES of starting the process of installing.

PySimpleGUI was written with students and beginners in mind.  The idea was to enable the creation of any GUI layout using the same full-featured GUI widgets that are available if you coded directly in tkinter or Qt, but done without the boilerplate code.  The package does all the messy work for you.  It also hides the concepts like callbacks that are confusing to beginners.

Take a look at a small sampling of the GUIs made.
https://github.com/MikeTheWatchGuy/PySimpleGUI/issues/1
The better ones are towards the bottom. It's a long list dating back to the 1.0 release so the early stuff looks a little primitive.

The reviews from users have been extremely positive on Reddit and on the GitHub site.

Today this was posted by a user on Reddit:
I'm finding it so easy to add a quick GUI front-end to some of my Python scripts with PySimpleGUI. There's so much less "boilerplate" needed than with more traditional GUI frameworks that I find myself adding simple GUIs to scripts which I would just run in console mode otherwise.

There are currently two flavors of PySimpleGUI, one that uses tkinter and the other Qt.  The great thing about PySimpleGUI is that you can create the same layout that you would if you coded directly in tkinter or Qt.  All of the same widgets are available for use.  They are just presented in an easier to understand manner.  The source code created in PySimpleGUI will run on either tkinter or Qt platform.  The only thing you have to change is the import statement.

PySimpleGUI has plenty of depth of features.  The result is a powerful GIU framework capable of running a large variety of user interfaces including running in the system tray.

PySimpleGUI is actively developed and maintained.  Should you run into a problem, it's typically addressed the same day.

There was a clear, large gap in the GUI landscape for Python.  In order to write even the most primitive of GUIs, you are required to know and use skills that are advanced beginner / intermediate.  GUIs are often taught towards the end of the curriculum for this reason.

With PySimpleGUI GUIs can be taught from the beginning of the course.  Rather than using the command line for input/output, it's easy to use windows instead.

If you want to learn more about the overall architecture, this recent post will help:
https://www.reddit.com/r/Python/comments/a1mj0p/pysimplegui_under_the_hood/



@mike<mailto:mike_barnett at hotmail.com>


From mike_barnett at hotmail.com  Thu Dec  6 12:56:22 2018
From: mike_barnett at hotmail.com (Mike Barnett)
Date: Thu, 6 Dec 2018 17:56:22 +0000
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <MWHPR11MB17108CB14624CD48891131AEE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>
References: <MWHPR11MB17108CB14624CD48891131AEE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>
Message-ID: <MWHPR11MB1710B68D72219956C42B039DE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>

Oh, one more thing on this topic... there are tutorial videos available for PySimpleGUI, both basic and advanced.

Basic 5 video series:
https://www.youtube.com/playlist?list=PLl8dD0doyrvHMoJGTdMtgLuHymaqJVjzt

Additional 9 in-depth videos:
https://www.youtube.com/playlist?list=PLl8dD0doyrvHMoJGTdMtgLuHymaqJVjzt


-mike


From smravikumardonbosco at gmail.com  Thu Dec  6 09:17:23 2018
From: smravikumardonbosco at gmail.com (Ravi Kumar)
Date: Thu, 6 Dec 2018 08:17:23 -0600
Subject: [Tutor] Regarding Python api script
In-Reply-To: <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
Message-ID: <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>

Ya i do know SQL thanks I will look into it but before proceeding  I had
couple of concerns about my code to geneeate the client log events my
concerns


1)The for loops that have written I am able to access all the networks,able
to loop through  all access points(Devices) in the network,able to loop
through and get all clients in each access points but when it comea to
client log events I am able to loop through and get only  the last row of
accesspoints  list and get those particular and clients and its log events
I assume I am going wrong in the last for loop code and output as shown
below

import requests
import meraki
import json


serveraddress= 'meraki.com <http://n136.meraki.com/>'
merakiAPIkey='3****????ce879efeafa5'
organization='4
networkid='N_63951****2046'

print ("Currently, the following organizations exist:\n")
base_url = "https://" + serveraddress + "/api/v0/organizations"
headers = {'X-Cisco-Meraki-API-Key': merakiAPIkey}

# Send the query to the Prime Infrastructure Server
r = requests.get(base_url,headers=headers)
# Retrieve the results in JSON format
json_string = r.json()


for org in json_string:

  print ("Organization: {}, ID: {}".format(org.get("name"), org.get("id")))
  print ("Meraki Query Engine Starting...\n")



#### Section 2 Getting the networkID'S####
#Construct the url for querying Prime Infrastructure
base_url=("https://"+serveraddress+"/api/v0")
config_url=("/organizations/"+organization+"/networks")
headers = {'X-Cisco-Meraki-API-Key':merakiAPIkey}


r = requests.get(base_url+config_url,headers=headers)
json_string = r.json()

for network in json_string:
    print("Network:{}, ID:  {}".format(network.get("name")
,network.get("id")))


for item in json_string:

    if item['id'] == "N_630" :
        network = item['id']
        found_network = True

if not found_network:
    print ("No Valid Network Found...")
    exit()

print ("\n****************>>Querying Meraki for Access Points(Devices) on
Network:"+network)

#### Section 2 Getting the Accesspoints(Devices)####
config_url = "/networks/"+network+"/devices"
r = requests.get(base_url+config_url,headers=headers)
json_string = r.json()

print ('{0:20}   {1:20}   {2:20}   {3:30}   {4:20}'.format("Switch Name",
"MAC Address", "Serial Number", "IP Address","Device Type"))


serialnumlist=[]


for item in json_string:


    # Extract all the appropriate fields
    devicename = item.get("name")
    macAddress = item.get("mac")
    serialnum = item.get("serial")

    serialnumlist.append(item)

    devicetype = item.get("model")
    ipaddress = item.get("lanIp")

    # Print the resulting data to the screen
    print ('{0:20}   {1:20}   {2:20}    {3:30}   {4:20}'.format(devicename,
macAddress, serialnum,ipaddress, devicetype))

for item in serialnumlist:
    print ("\n*****************>>Querying Meraki for clients on Access
Points:" + item.get("name")+ " (" + item.get("serial") + ")")
    print ('{0:20}    {1:30}    {2:16}    {3:18}          {4:15}
{5:20}'.format("Client Name", "Description", "IP Address", "MAC Address",
"Sent KBytes", "Recv KBytes"))


    config_url = "/devices/" + item.get("serial") +
"/clients?timespan=86400"
    r = requests.get(base_url + config_url, headers=headers)
    json_string = r.json()

    for client in json_string:

        hostname = client.get("dhcpHostname")
        description = client.get("description")
        ipaddress = client.get("ip")
        macaddress = client.get("mac")

        usage = client.get("usage")
        sentbytes = usage.get("sent")
        recvbytes = usage.get("recv")

        print ('{0:20}   {1:20}   {2:20}    {3:30}   {4:20}
{5:20}'.format(hostname, description, ipaddress, macaddress,sentbytes,
recvbytes))



for  item in serialnumlist:
    print ("\nQuerying Meraki for clientlogevents on Devices: (" +
item.get("mac") + ")")
   # print ('{0:20}   {1:30}   {2:16}   {3:18}   {4:10}
{5:11}'.format("Hostname", "Description", "IP Address", "MAC Address","Sent
KBytes", "Recv KBytes"))

for item  in json_string:
    config_url = "/networks/"+"/N_63***1050/"+"/clients/"+ item.get("mac")
+ "/events?perPage=1000"
    r = requests.get(base_url + config_url, headers=headers)

    print ('{}  '.format(r.content))
**********************************>

OUTPUT


Network:Room, ID:  N_631140260
Network:Ofce, ID:  N_639540739

***>>Querying Meraki for Access Points(Devices) on Network:N_639511050

Switch Name            MAC Address

Serial Number

HA         e0:cb:b:3e:95      Q2XD-USN6          172.16.172.

AB.          e0:cb:7:01:f4      Q2XD-G8Q8          172.16.172.


***********>>Querying Meraki for clients on Access Points:HR (Q2XD-USN6)

Client Name                MAC Address
Kiosk                        55:77:6a:5
Chphone.               44:91:60:a2:0

*****************>>Querying Meraki for clients on Access Points:DerbyRoom
(Q2XD--GQ8)

Client Name               MAC Address
iPhone                      f0:98:9d:2c:                   arphone
        c8:21:58:79:b
it-min                        e:05:a4:c9:

Querying Meraki for clientlogevents on Devices: e0:cb:b::3e:95 (HA)

Not looping the devices to get api calls for this access point

Querying Meraki for clientlogevents on Devices: e0:cb:7:01:f4 (AB)
Getting api calls for list of devices in the access point
*******

I know I am asking a lot but I hit a road block on this i am assuming One
small change in my last for loop I can get this working for all devices and
I have hardcoded for one network how do I use the same code for different
networks as well
You help is really appreciated  Thank you all!








On Wed, Dec 5, 2018, 2:32 AM Alan Gauld <alan.gauld at yahoo.co.uk wrote:

> CCing the list, please use Reply All when responding to the tutor list.
>
>
> On 05/12/2018 03:44, Ravi Kumar wrote:
> > Yes  thats right I want to extract the xml and store into database(SQL
> > Server) and I will have to cteate a new table
> >
> > Here is the sample output I am getting similarly there bulk data which
> > I want to store in database format is as shown below
> >
> >
> >
> b'[{"deviceSerial":"Q2XD-333X-G8Q8","occurredAt":1537565.640085,"type":"802.11
> >
> association","details":{"radio":"1","vap":"0","clientMac":"C8:21:6679:B6:16","channel":"44","rssi":"57","aid":"31645681"}},{"deviceSerial":"Q2XD-97gX-G8Q8","occurredAt":153765.700095,"type":"WPA
> >
> deauthentication","details":{"radio":"1","vap":"0","clientMac":"C621:58:79:B6:16","aid":"316681"}},{"deviceSerial":"Q2XD-97gX-G8Q8","occurredAt":1563369.780085,"type":"WPA
> >
> deauthentication","details":{"radio":"1","vap":"0","clientMac":"C8:21:58:9:B6:16","aid":"31645681"}},{"deviceSerial":"Q297JX-G8Q8"
> >
> >
> > Please let me whether this can be formatted into json in the code
>
> It looks to me like it is already in JSON format.
>
> It is a list of objects.
>
> The json module will let you extract the objects into Python data objects.
>
>
> > and also any other tips or links to refer if available  to store in
> > database
>
> Do you know SQL?
>
> If so using the Python DBAPI is very easy.
>
> If not you will need to learn some basic SQL.
>
> You can try reading the database topic in my Python tutorial(see below)
> which is based on SQLite rather than SQL Server but should be 90%
> compatible.
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>

From alan.gauld at yahoo.co.uk  Thu Dec  6 19:25:40 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 7 Dec 2018 00:25:40 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
Message-ID: <pucei1$4ml$1@blaine.gmane.org>

On 06/12/2018 14:17, Ravi Kumar wrote:

> 1)The for loops that have written I am able to access all the networks,able
> to loop through  all access points(Devices) in the network,able to loop
> through and get all clients in each access points but when it comea to
> client log events I am able to loop through and get only  the last row of
> accesspoints  list and get those particular and clients and its log events

I don't know the cause, but did take a quick look at the code.
One thing that makes it very difficult to follow is that you use the
same variable names over and over making it hard to keep track of what
any given 'item' or 'json_string' or 'r' actually holds at any given time.

Buy using  a different name for each json_string reflecting the
expected data - such as json_networks or json_clients - it would
be very much easier to follow the flow of the code.

> I assume I am going wrong in the last for loop code and output as shown
> below

Probably but I confess I couldn't spot it from a casual read through.

>     for client in json_string:
> 
>         hostname = client.get("dhcpHostname")
>         description = client.get("description")
>         ipaddress = client.get("ip")
>         macaddress = client.get("mac")
> 
>         usage = client.get("usage")
>         sentbytes = usage.get("sent")
>         recvbytes = usage.get("recv")
> 
>         print ('{0:20}   {1:20}   {2:20}    {3:30}   {4:20}
> {5:20}'.format(hostname, description, ipaddress, macaddress,sentbytes,
> recvbytes))
> 
> 
> 
> for  item in serialnumlist:
>     print ("\nQuerying Meraki for clientlogevents on Devices: (" +
> item.get("mac") + ")")
> 
> for item  in json_string:
>     config_url = "/networks/"+"/N_63***1050/"+"/clients/"+ item.get("mac")
> + "/events?perPage=1000"
>     r = requests.get(base_url + config_url, headers=headers)
> 
>     print ('{}  '.format(r.content))
> **********************************>

The 'item's in the last loop appear to be the same as the 'client's
in the previous loop? Since 'json_string' never changes...

Also the setialnumlist only prints the result it never stores anything.

Then the last loop uses item.get(mac) which will not be the one printed
earlier since item is now read from json_string(back tome earlier
confusion over names!)

I suspect this lack of continuity may be the root of your problem
but the name collisions are twisting my brain and its late at night...


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Thu Dec  6 21:35:29 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Dec 2018 13:35:29 +1100
Subject: [Tutor] Regarding Python api script
In-Reply-To: <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
Message-ID: <20181207023527.GW13061@ando.pearwood.info>

On Thu, Dec 06, 2018 at 08:17:23AM -0600, Ravi Kumar wrote:

> I know I am asking a lot 

Yes you are. Please read this:

http://sscce.org/


It is written for Java programmers, but it applies equally to all 
languages, including Python.

Think about how difficult a job you are giving us: we don't have access 
to your network; we don't have the API key (which is understandable); we 
can't run the code; we can't even be confident that the code you give us 
is the same code you are running.

In fact we know that it cannot be the same code, because what you have 
given us has a Syntax Error in line 8:

> organization='4

(missing quote mark). Where there is one change to the code, there could 
be dozens for all we know.


> but I hit a road block on this i am assuming One
> small change in my last for loop I can get this working for all devices

That doesn't seem like a reasonable assumption to me. You might be 
right, or it could take many large changes.


> and
> I have hardcoded for one network how do I use the same code for different
> networks as well

You can read the network from the command line, or a config file, or an 
environment variable, or interactively by asking the user to enter the 
network using the "input" (Python 3) or "raw_input" (Python 2) 
functions.

I expect that you need more than one piece of information to change the 
network, it looks like you probably need at least four:

serveraddress
merakiAPIkey
organization
networkid

so it probably is best to read the details from a config file:

https://docs.python.org/3/library/configparser.html

is probably simplest, but you could use Property Lists as well:

https://docs.python.org/3/library/plistlib.html




-- 
Steve

From sunil.techspk at gmail.com  Fri Dec  7 02:58:18 2018
From: sunil.techspk at gmail.com (Sunil Tech)
Date: Fri, 7 Dec 2018 13:28:18 +0530
Subject: [Tutor] Trouble in dealing with special characters.
Message-ID: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>

Hi Tutor,

I have a trouble with dealing with special characters in Python Below is
the sentence with a special character(apostrophe) "MOUNTAIN VIEW WOMEN?S
HEALTH CLINIC" with actually should be "MOUNTAIN VIEW WOMEN'S HEALTH CLINIC
".

Please help, how to identify these kinds of special characters and replace
them with appropriate ASCII?

Thanks,
Sunil. G

From avigross at verizon.net  Thu Dec  6 21:13:01 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 6 Dec 2018 21:13:01 -0500
Subject: [Tutor] Regarding Python api script
In-Reply-To: <pucei1$4ml$1@blaine.gmane.org>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org>
Message-ID: <008901d48dd2$613e2be0$23ba83a0$@verizon.net>

I left the "subject" above to be the same, as requested.

The actual subject might have been "Pythonic variable name use"

Alan says he had a challenge evaluating code (below) because the same
variable names were reused and it made me wonder if the python community has
general ideas about name re-use.

I am keeping this short. In languages without garbage collection, reusing
the same name, like "index" repeatedly might save some small amount of
space. With garbage collection, reuse may be one way to hint it can be used
again but if that is important, you can use "del var" to remove var.

For short programs, or in a context like a "normal" function that goes away
when completed, it probably is best to make the code very readable. PLEASE
do not mention the many exceptions as I do not intend much of what I say to
be taken as completely accurate.

But so much code I see in python does not only reuse the same variable names
but in a confusing way.

file =  "some name"
file = open(file, "r")
file = some_wrapper(file)

mylist = [ ... ]
mylist = sorted(mylist)

for index in range(...):
  stuff

for index in open(...)
  more stuff

for index, index2  in enumerate(...)
  yet more stuff

I have often seen something like this done with methods, such as to emulate
decorator functionality where a method is created in an object with a name
and the very next method created has the same name with the same function
name as an argument to replace it.

So is there a guide on when reuse is good and when it just obfuscates? What
is good practice?

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Alan Gauld via Tutor
Sent: Thursday, December 6, 2018 7:26 PM
To: tutor at python.org
Subject: Re: [Tutor] Regarding Python api script

On 06/12/2018 14:17, Ravi Kumar wrote:

> 1)The for loops that have written I am able to access all the 
> networks,able to loop through  all access points(Devices) in the 
> network,able to loop through and get all clients in each access points 
> but when it comea to client log events I am able to loop through and 
> get only  the last row of accesspoints  list and get those particular 
> and clients and its log events

I don't know the cause, but did take a quick look at the code.
One thing that makes it very difficult to follow is that you use the same
variable names over and over making it hard to keep track of what any given
'item' or 'json_string' or 'r' actually holds at any given time.

Buy using  a different name for each json_string reflecting the expected
data - such as json_networks or json_clients - it would be very much easier
to follow the flow of the code.

> I assume I am going wrong in the last for loop code and output as 
> shown below

Probably but I confess I couldn't spot it from a casual read through.

>     for client in json_string:
> 
>         hostname = client.get("dhcpHostname")
>         description = client.get("description")
>         ipaddress = client.get("ip")
>         macaddress = client.get("mac")
> 
>         usage = client.get("usage")
>         sentbytes = usage.get("sent")
>         recvbytes = usage.get("recv")
> 
>         print ('{0:20}   {1:20}   {2:20}    {3:30}   {4:20}
> {5:20}'.format(hostname, description, ipaddress, macaddress,sentbytes,
> recvbytes))
> 
> 
> 
> for  item in serialnumlist:
>     print ("\nQuerying Meraki for clientlogevents on Devices: (" +
> item.get("mac") + ")")
> 
> for item  in json_string:
>     config_url = "/networks/"+"/N_63***1050/"+"/clients/"+ 
> item.get("mac")
> + "/events?perPage=1000"
>     r = requests.get(base_url + config_url, headers=headers)
> 
>     print ('{}  '.format(r.content))
> **********************************>

The 'item's in the last loop appear to be the same as the 'client's in the
previous loop? Since 'json_string' never changes...

Also the setialnumlist only prints the result it never stores anything.

Then the last loop uses item.get(mac) which will not be the one printed
earlier since item is now read from json_string(back tome earlier confusion
over names!)

I suspect this lack of continuity may be the root of your problem but the
name collisions are twisting my brain and its late at night...


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From nathan-tech at hotmail.com  Thu Dec  6 21:17:55 2018
From: nathan-tech at hotmail.com (nathan tech)
Date: Fri, 7 Dec 2018 02:17:55 +0000
Subject: [Tutor] playing sound files in python
Message-ID: <AM0PR05MB6225701DAD2A69C2CD186061E4AA0@AM0PR05MB6225.eurprd05.prod.outlook.com>

Hello all!

My name is nate, and I am relatively new to this list, relatively being 
just signed up.

I have a question that you would think would be obvious, but alas I have 
struggled to figure out.

How do I play sound files in python.

More specificly, I want to play ogg files especially, with wav and mp3 
also being a high priority.

I am using windows, and building for windows machines.

I am currently building my gui using wx python, and using something like 
pygame just seems overly blotey to me.

Thanks for any help you can offer.

Nate


From alan.gauld at yahoo.co.uk  Fri Dec  7 03:30:34 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 7 Dec 2018 08:30:34 +0000
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
Message-ID: <pudav7$nol$1@blaine.gmane.org>

On 07/12/2018 07:58, Sunil Tech wrote:

> I have a trouble with dealing with special characters in Python Below is
> the sentence with a special character(apostrophe) "MOUNTAIN VIEW WOMEN?S
> HEALTH CLINIC" with actually should be "MOUNTAIN VIEW WOMEN'S HEALTH CLINIC
> ".

How do you define "special characters"?
There is nothing special about the apostraphe. It is just as
valid a character as all the other characters.
What makes it special to you?

> Please help, how to identify these kinds of special characters and replace
> them with appropriate ASCII?

What is appropriate ASCII?
ASCII only has 127 characters.
Unicode has thousands of characters.
How do you want to map a unicode character into ASCII?
There are lots of options but we can't tell what you
think is appropriate.

Finally, character handling changed between Python 2 and 3
(where unicode became the default), so the solution will
likely depend on the Python version you are using.
Please tell us which.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From sunil.techspk at gmail.com  Fri Dec  7 03:36:16 2018
From: sunil.techspk at gmail.com (Sunil Tech)
Date: Fri, 7 Dec 2018 14:06:16 +0530
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <pudav7$nol$1@blaine.gmane.org>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
 <pudav7$nol$1@blaine.gmane.org>
Message-ID: <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>

Hi Alan,

I am using Python 2.7.8
>>> tx = "MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"
>>> tx.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19:
ordinal not in range(128)

How to know whether in a given string(sentence) is there any that is not
ASCII character and how to replace?

On Fri, Dec 7, 2018 at 2:01 PM Alan Gauld via Tutor <tutor at python.org>
wrote:

> On 07/12/2018 07:58, Sunil Tech wrote:
>
> > I have a trouble with dealing with special characters in Python Below is
> > the sentence with a special character(apostrophe) "MOUNTAIN VIEW WOMEN?S
> > HEALTH CLINIC" with actually should be "MOUNTAIN VIEW WOMEN'S HEALTH
> CLINIC
> > ".
>
> How do you define "special characters"?
> There is nothing special about the apostraphe. It is just as
> valid a character as all the other characters.
> What makes it special to you?
>
> > Please help, how to identify these kinds of special characters and
> replace
> > them with appropriate ASCII?
>
> What is appropriate ASCII?
> ASCII only has 127 characters.
> Unicode has thousands of characters.
> How do you want to map a unicode character into ASCII?
> There are lots of options but we can't tell what you
> think is appropriate.
>
> Finally, character handling changed between Python 2 and 3
> (where unicode became the default), so the solution will
> likely depend on the Python version you are using.
> Please tell us which.
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From alan.gauld at yahoo.co.uk  Fri Dec  7 03:57:35 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 7 Dec 2018 08:57:35 +0000
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
 <pudav7$nol$1@blaine.gmane.org>
 <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>
Message-ID: <pudchs$sjd$1@blaine.gmane.org>

On 07/12/2018 08:36, Sunil Tech wrote:

> I am using Python 2.7.8
>>>> tx = "MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"
>>>> tx.decode()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19:
> ordinal not in range(128)
> 
> How to know whether in a given string(sentence) is there any that is not
> ASCII character and how to replace?

How to detect is to do wat you just did but wrap a try/except around it:

try:
    tx.decode()
except UnicodeError:
    print " There were non ASCII characters in the data"

Now, how you replace the characters is up to you.
The location of the offending character is given in the error.
(Although there may be more, once you deal with that one!)
What would you like to replace it with from the ASCII subset?

But are you really sure you want to replace it with
an ASCII character? Most display devices these days
can cope with at least UTF-8 version of Unicode.
Maybe you really want to change your default character
set so it can handle those extra characters??

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Fri Dec  7 04:50:49 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Dec 2018 20:50:49 +1100
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
Message-ID: <20181207095048.GX13061@ando.pearwood.info>

On Fri, Dec 07, 2018 at 01:28:18PM +0530, Sunil Tech wrote:
> Hi Tutor,
> 
> I have a trouble with dealing with special characters in Python 

There are no special characters in Python. There are only Unicode 
characters. All characters are Unicode, including those which are also 
ASCII.

Start here:

https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

https://blog.codinghorror.com/there-aint-no-such-thing-as-plain-text/

https://www.youtube.com/watch?v=sgHbC6udIqc

https://nedbatchelder.com/text/unipain.html


https://docs.python.org/3/howto/unicode.html

https://docs.python.org/2/howto/unicode.html


Its less than a month away from 2019. It is sad and shameful to be 
forced to use only ASCII, and nearly always unnecessary. Writing code 
that only supports the 128 ASCII characters is like writing a calculator 
that only supports numbers from 1 to 10.

But if you really must do so, keep reading.


> Below is
> the sentence with a special character(apostrophe) "MOUNTAIN VIEW WOMEN?S
> HEALTH CLINIC" with actually should be "MOUNTAIN VIEW WOMEN'S HEALTH CLINIC
> ".

Actually, no, it should be precisely what it is: "WOMEN?S" is correct, 
since that is an apostrophe. Changing the ? to an inch-mark ' is not 
correct. But if you absolutely MUST change it:

mystring = "MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"
mystring = mystring.replace("?", "'")

will do it in Python 3. In Python 2 you have to write this instead:

# Python 2 only
mystring = u"MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"
mystring = mystring.replace(u"?", u"'")


to ensure Python uses Unicode strings.

What version of Python are you using, and what are you doing that gives 
you trouble?

It is very unlikely that the only way to solve the problem is to throw 
away the precise meaning of the text you are dealing with by reducing it 
to ASCII.

In Python 3, you can also do this:

mystring = ascii(mystring)

but the result will probably not be what you want.


> Please help, how to identify these kinds of special characters and replace
> them with appropriate ASCII?

For 99.99% of the characters, there is NO appropriate ASCII. What 
ASCII character do you expect for these?

    ? ? ? ? ? ? ? ? ? ??

ASCII, even when it was invented in 1963, wasn't sufficient even for 
American English (no cent sign, no proper quotes, missing punctuation 
marks) let alone British English or international text.

Unless you are stuck communicating with an ancient program written in 
the 1970s or 80s that cannot be upgraded, there are few good reasons to 
cripple your program by only supporting ASCII text.

But if you really need to, this might help:

http://code.activestate.com/recipes/251871-latin1-to-ascii-the-unicode-hammer/

http://code.activestate.com/recipes/578243-repair-common-unicode-mistakes-after-theyve-been-m/




-- 
Steve

From steve at pearwood.info  Fri Dec  7 05:20:04 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Dec 2018 21:20:04 +1100
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
 <pudav7$nol$1@blaine.gmane.org>
 <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>
Message-ID: <20181207102004.GY13061@ando.pearwood.info>

On Fri, Dec 07, 2018 at 02:06:16PM +0530, Sunil Tech wrote:
> Hi Alan,
> 
> I am using Python 2.7.8

That is important information.

Python 2 unfortunately predates Unicode, and when it was added some bad 
decisions were made. For example, we can write this in Python 2:

>>> txt = "abc?"

but it is a lie, because what we get isn't the string we typed, but the 
interpreters *bad guess* that we actually meant this:

>>> txt
'abc\xcf\x80'

Depending on your operating system, sometimes you can work with these 
not-really-text strings for a long time, but when it fails, it fails 
HARD with confusing errors. Just as you have here:

> >>> tx = "MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"
> >>> tx.decode()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19:
> ordinal not in range(128)


Here, Python tried to guess an encoding, and picked some 
platform-specific encoding like Latin-1 or CP-1252 or something even 
more exotic. That is the wrong thing to do.

But if you can guess which encoding it uses, you can make it work:

tx.decode("Latin1")

tx.decode("CP-1252")

But a better fix is to use actual text, by putting a "u" prefix outside 
the quote marks:

txt = u"MOUNTAIN VIEW WOMEN?S HEALTH CLINIC"


If you need to write this to a file, you can do this:

file.write(txt.encode('utf-8'))

To read it back again:

# from a file using UTF-8
txt = file.read().decode('utf-8')

(If you get a decoding error, it means your text file wasn't actually 
UTF-8. Ask the supplier what it really is.)


> How to know whether in a given string(sentence) is there any that is not
> ASCII character and how to replace?

That's usually the wrong solution. That's like saying, "My program can't 
add numbers greater than 100. How do I tell if a number is greater than 
100, and turn it into a number smaller than 100?"

You can do this:

mystring = "something"
if any(ord(c) > 127 for c in mystring):
    print "Contains non-ASCII"
    

But what you do then is hard to decide. Delete non-ASCII characters? 
Replace them with what?

If you are desperate, you can do this:

bytestring = "something"
text = bytestring.decode('ascii', errors='replace')
bytestring = text.encode('ascii', errors='replace')


but that will replace any non-ascii character with a question mark "?" 
which might not be what you want.



-- 
Steve

From steve at pearwood.info  Fri Dec  7 05:49:58 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Dec 2018 21:49:58 +1100
Subject: [Tutor] playing sound files in python
In-Reply-To: <AM0PR05MB6225701DAD2A69C2CD186061E4AA0@AM0PR05MB6225.eurprd05.prod.outlook.com>
References: <AM0PR05MB6225701DAD2A69C2CD186061E4AA0@AM0PR05MB6225.eurprd05.prod.outlook.com>
Message-ID: <20181207104958.GZ13061@ando.pearwood.info>

On Fri, Dec 07, 2018 at 02:17:55AM +0000, nathan tech wrote:
> Hello all!
> 
> My name is nate, and I am relatively new to this list, relatively being 
> just signed up.
> 
> I have a question that you would think would be obvious, but alas I have 
> struggled to figure out.
> 
> How do I play sound files in python.

Alas, its not obvious nor easy.

https://duckduckgo.com/?q=python+play+sound+files

 
> More specificly, I want to play ogg files especially, with wav and mp3 
> also being a high priority.

I believe that wxPython supports .wav files, but not .ogg or .mp3.

Playing sound in Python is a definite weakness unless you have a 
third-party library like PyGame that supports it in a platform- 
independent way.

You could try:

    print('\a')

but this requires that you are running in a terminal that supports the 
BEL character, that the system beep has not been turned off, and even if 
it works, its only a short beep.

If your OS (Windows?) has a sound file player, you could try 
calling out to it with os.system. Under Linux, I might try 
something like this:

os.system('mpg123 -q why_is_the_rum_gone-remix.mp3')

but that pauses until paying is over, and requires the mpg123 player. 
There are other players, like ogg123. What you get on Windows, I don't 
know.

Pygame is starting to sound more attractive :-)



-- 
Steve

From steve at pearwood.info  Fri Dec  7 05:59:33 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 7 Dec 2018 21:59:33 +1100
Subject: [Tutor] Regarding Python api script
In-Reply-To: <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
References: <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org> <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
Message-ID: <20181207105932.GA13061@ando.pearwood.info>

On Thu, Dec 06, 2018 at 09:13:01PM -0500, Avi Gross wrote:

> But so much code I see in python does not only reuse the same variable names
> but in a confusing way.
> 
> file =  "some name"
> file = open(file, "r")
> file = some_wrapper(file)

I agree this is confusing: you have the same name, "file", used to mean 
a file name, an open file object, and whatever some_wrapper returns. 
This is not the clearest code I've ever seen.


> mylist = [ ... ]
> mylist = sorted(mylist)

There's nothing wrong with that, except that it ought to be written:

mylist = [ ... ]
mylist.sort()

to sort in place instead of making a sorted copy.



> for index in range(...):
>   stuff
> 
> for index in open(...)
>   more stuff


Using "index" to iterate over the lines in a file is just a crappy 
choice of name.



[...]
> I have often seen something like this done with methods, such as to emulate
> decorator functionality where a method is created in an object with a name
> and the very next method created has the same name with the same function
> name as an argument to replace it.

I don't understand this. Can you show an example? Even if made up?


> So is there a guide on when reuse is good and when it just obfuscates? What
> is good practice?

If you pick *meaningful* names, it will be pretty obvious when to reuse 
the same name: is the name still meaningful? Then you MAY reuse. If the 
name is not meaningful, then you MUST NOT reuse it, unless you are 
deliberately writing obfuscated code.

If you have a name "column" which you use for a columns, then you 
shouldn't reuse it for rows, or lines of text read from a file, or the 
length of a list, or the date. But you can reuse it for another column, 
provided there's no confusion over which column is which.

The same applies to any other language.



-- 
Steve

From mats at wichmann.us  Fri Dec  7 10:28:31 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 7 Dec 2018 08:28:31 -0700
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <20181207102004.GY13061@ando.pearwood.info>
References: <CAExJxTPARm3ytJvh8mYxjuNMAQ_up087yRKDpecxeaLV3Te=Ew@mail.gmail.com>
 <pudav7$nol$1@blaine.gmane.org>
 <CAExJxTNRiDbeyS0G_93qqcfk_8jOWBuxe2X0K8n6-96qQaU-iw@mail.gmail.com>
 <20181207102004.GY13061@ando.pearwood.info>
Message-ID: <c03ea3e2-65ab-44ee-878d-fcd457d5f03b@wichmann.us>

On 12/7/18 3:20 AM, Steven D'Aprano wrote:

>> How to know whether in a given string(sentence) is there any that is not
>> ASCII character and how to replace?
> 
> That's usually the wrong solution. That's like saying, "My program can't 
> add numbers greater than 100. How do I tell if a number is greater than 
> 100, and turn it into a number smaller than 100?"

yes, it's usually the wrong solution, but in the case of quote marks it
is *possible* is is the wanted solution: certain text editing products
(cough cough Microsoft Word) are really prone to putting in typographic
quote marks.  Everyone knows not to use Word for editing your code, but
that doesn't mean some stuff doesn't make it into a data set we forced
to process, if someone exports some text from an editor, etc. There are
more quoting styles in the world than the English style, e.g. this one
is used in many languages: ?quoted text?  (I don't know if that will
survive the email system, but starts with a descended double-quote mark).

It's completely up to what the application needs; it *might* as I say be
appropriate to normalize text so that only a single double-quote and
only a single single-quote (or apostrophe) style is used.  Or it might not.



From asad.hasan2004 at gmail.com  Fri Dec  7 09:00:57 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Fri, 7 Dec 2018 19:30:57 +0530
Subject: [Tutor] unzip and connect to Oracle database
Message-ID: <CAG3LsKF7vqavDJfm=D8czoio+rd_c9Y=MSPX+Uysjz8OCMm8+w@mail.gmail.com>

Hi All ,

         I would like to unzip a file using python and then execute the sql
scripts  in the file on Oracle database .

 >>> from zipfile import ZipFile
>>> file_name = 'file.zip'
>>> z = ZipFile(file_name)
>>> print(z.namelist())
[]
>>> z = ZipFile('file.zip')
>>> print z.namelist()
[]
>>> print z
<zipfile.ZipFile object at 0xf9220fd0>
>>> z.extractall()
    Doesnot unzip the zip file

I unable to unzip the file what I am doing wrong ?

Also share some code or documentation to connect to Oracle DB and execute
sql scripts.

Thanks,

-- 
Asad Hasan
+91 9582111698

From avigross at verizon.net  Fri Dec  7 12:30:18 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 7 Dec 2018 12:30:18 -0500
Subject: [Tutor] Regarding Python api script
In-Reply-To: <20181207105932.GA13061@ando.pearwood.info>
References: <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org> <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
 <20181207105932.GA13061@ando.pearwood.info>
Message-ID: <007801d48e52$879f1800$96dd4800$@verizon.net>

[[ Real SUBJECT: emulating decorators ]]

Steven,

I am not suggesting that a particular way of reusing a name is a good idea.
I am asking if some ways are encouraged and others are discouraged in the
python community. 

Just to clarify one point you asked about:

#  > I have often seen something like this done with methods, such as to
emulate
#  > decorator functionality where a method is created in an object with a
name
#  > and the very next method created has the same name with the same
function
#  > name as an argument to replace it.
#  
# I don't understand this. Can you show an example? Even if made up?

The answer is that I saw that mentioned in a Lutz book that I no longer
have. What I wrote above should be clear if I give an example:

Say you have a method in a class definition called do_this() that does
whatever this is.

You may also have other methods in the same or different classes with names
like do_that() and your programs run fine.

At a later point you decide you want to see how long it takes the method to
run. So you make or borrow a function called timing_is_everything() that
looks like this in pseudocode:

def timing_is_everything(fun, args):
    save the current time
    call fun with args
    save the return value of calling fun
    get the current time
    calculate time elapsed and do whatever with it.
    Return saved return value from fun.

The argument "fun" above can be replaced by any function such as "do_this"
or "do_that" and produce the same side-effect while also running the
function unharmed.

Then you decide you want a function that logs when another function is
called and with what arguments:

Again, in PSEUDOCODE, it looks like:

def logistics(fun, args):
    with open(logfile):
         write info about the current date/time, name of the function and
arguments
    return result of calling the function with those args

Now to the point. You could write each and every module you want
independently. At the top of the body you could insert any similar code to
begin timing it or to log it. Then you place the code needed for that
function followed by anything you want done after such an action and finally
return a value, if needed.

But if you already have written and tested something like the two modules
above, you can do something like this:

class classy(object):
    ...
    # Original function first of this name
    def recreational_fun(args):
        ...

    # New function, second of this name
    recreational _fun = timing_is_everything(recreational _fun, args)

    # New function, third of this name
    recreational _fun = logistics(recreational _fun, args)

AGAIN, the above is for illustration only and I am well aware of the proper
ways to do the pseudocode such as using notations like *args, **kwargs and
so on. We are talking about an IDEA that perhaps predated the introduction
of decorators way back when. The point is that you could have used three
names here for the various functions but realistically, only the last was
needed so why carry around extra methods.

With "decorators" as a python feature allowing you to wrap multiple
functions around the one being defined, assuming you built the above helper
functions properly so they can be used as decorators, the above might look
like this:

class classified(object):
    ...
    # Original function first and only of this name 
    # and highly decorated bottom to top

    @ logistics
    @ timing_is_everything
    def recreational_fun(args):
        ...

Both ways of doing this, again if PROPERLY done, should have the same
result, albeit there may be subtle changes such as hidden within the dunder
properties. There will be a single function name visible to any user. When
called, it will log an entry, note what time it is, do whatever the
re-created method is supposed to do, then calculate how much time it took
and return some result.

And, yes, I know there is overhead added as you end up with layered function
calls and so on.

This was an answer to a request. I hope I answered it. I do not wish to now
continue on this topic. The topic was not decorators. But the decorator
functionality used above likely does not confuse anybody as to what the name
of the method is. That variable name is only seen once as compared to my
first example when it refers to three completely different functions and
keeps giving up a connection to what effectively becomes an anonymous
function that can only be remembered and called within the enclosing
function.

I tend to agree with the other points Steve made.



From alan.gauld at yahoo.co.uk  Fri Dec  7 12:59:22 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 7 Dec 2018 17:59:22 +0000
Subject: [Tutor] Regarding Python api script
In-Reply-To: <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
References: <CANUFOznsxbVAttAaXB61tJ8BCnXG+tdvohodNy+H1oF=NccbbQ@mail.gmail.com>
 <pu4i38$qrg$1@blaine.gmane.org>
 <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org> <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
Message-ID: <puec9n$99k$1@blaine.gmane.org>

On 07/12/2018 02:13, Avi Gross wrote:
> Alan says he had a challenge evaluating code (below) because the same
> variable names were reused 

It wasn't the reuse per se but the generic nature of
the names combined with reuse. Reusing names like
i,j,k for simple integer indices etc is not a problem.
But when values of domain significance are given
the same generic name (like item or file) then it
gets confusing. Its very easy to miss a significant
reassignment of the variable to a new value.

> ... In languages without garbage collection, reusing
> the same name, like "index" repeatedly might save some 
> small amount of space. 

Garbage collection only helps if the variable loses
its assignment. If the variable retains its value it
won't be garbage collected. So the same space saving applies.

However it is very rare that Python is used in the kind
of scenarios where that kind of space saving is
significant (usually embedded controllers with
tiny memory spaces).

> if that is important, you can use "del var" to remove var.

Exactly so.

> But so much code I see in python does not only > reuse the same variable names but in a confusing way.

Then its probably badly written Python!

> file =  "some name"
> file = open(file, "r")
> file = some_wrapper(file)

This is a good example of poor reuse since the objects
are all totally different and only the second it
truly a file.

> mylist = [ ... ]
> mylist = sorted(mylist)

That's not really reusing it, it's just a convoluted way
of modifying the value. (IN a purely functional view
of the world it is reuse but only a functional purist
would argue it that way I suspect)

> for index in range(...):
>   stuff

THis usage is possibly ok if range is used to generate
indexes into some collection. Although probably
iterating over the colection itself would be preferred.

> for index in open(...)
>   more stuff

But this is bad since index is not an index its a line,
and possibly even more specifically a domain object
representation which should get a domain related name.

> for index, index2  in enumerate(...)
>   yet more stuff

Again the index is OK here but index2 is very wrong
since it holds the value not an index.

But these issues are not Python issues they are
general naming issues in all languages.

It is good practice, regardless of language, to use
problem domain names for variables not generic
names or type related names. And also to not reuse
variables if the name is not applicable to the new
value.

> I have often seen something like this done with methods, such as to emulate
> decorator functionality

It's quite common to decorate a method and assign
the result to the same name but that is akin to the
sorted list example above. The name is being
augmented by the decorator not used for a
different purpose. It is specifically being
assigned to the same name to hide the original
non augmented function object.

> So is there a guide on when reuse is good > and when it just obfuscates?

It's the same in Python as in any language.
Use a name that makes sense in the domain.
If that name is applicable in multiple places
its ok to use the same name.

If a name is non domain specific then it
should only be used in very localised
contexts - such as a loop body - or where
it will be replaced by a domain specific
variable immediately - such as a tuple
from a database being used to instantiate
a domain related object.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Fri Dec  7 13:12:01 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 7 Dec 2018 18:12:01 +0000
Subject: [Tutor] unzip and connect to Oracle database
In-Reply-To: <CAG3LsKF7vqavDJfm=D8czoio+rd_c9Y=MSPX+Uysjz8OCMm8+w@mail.gmail.com>
References: <CAG3LsKF7vqavDJfm=D8czoio+rd_c9Y=MSPX+Uysjz8OCMm8+w@mail.gmail.com>
Message-ID: <pued1f$avf$1@blaine.gmane.org>

On 07/12/2018 14:00, Asad wrote:
> Hi All ,
> 
>          I would like to unzip a file using python and then execute the sql
> scripts  in the file on Oracle database .

Are you absolutely sure?
That's a very dangerous thing to do from a
security point of view. Potentially similar
to using exec() on user input...

You will need to ensure that the files inside
the zip file are very secure and from trusted
sources.

>  >>> from zipfile import ZipFile
>>>> file_name = 'file.zip'
>>>> z = ZipFile(file_name)
>>>> print(z.namelist())
> []

What did you expect to see? What does zip file contain?
Recall that zip files are not simply compressed files
they are also archives of files.

>>>> z.extractall()
>     Doesnot unzip the zip file

Are you sure you are looking in the right folder?
Can you use os.listdir() to print the file list
and verify that file.zip is where you expect and
after extracting that the extracted files don't exist?

Also recall that zip files may have absolute paths
so the extracted files may be in some completely
different location.

> Also share some code or documentation to connect to Oracle DB and execute
> sql scripts.

Lets deal with that separately.
There are several tutorials on YouTube and
assorted web pages showing how to do that.
If you don't understand them come back with
specific questions. Lets focus on ZipFile for now!


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Fri Dec  7 19:16:49 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 8 Dec 2018 11:16:49 +1100
Subject: [Tutor] Regarding Python api script
In-Reply-To: <puec9n$99k$1@blaine.gmane.org>
References: <CANUFOz=RmzrVac+nc4vT9rYvPGwuooW1t5G9Yj8_qJY=g3VOpA@mail.gmail.com>
 <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org> <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
 <puec9n$99k$1@blaine.gmane.org>
Message-ID: <20181208001648.GG13061@ando.pearwood.info>

On Fri, Dec 07, 2018 at 05:59:22PM +0000, Alan Gauld via Tutor wrote:

[...]
> > ... In languages without garbage collection, reusing
> > the same name, like "index" repeatedly might save some 
> > small amount of space. 
> 
> Garbage collection only helps if the variable loses
> its assignment. If the variable retains its value it
> won't be garbage collected. So the same space saving applies.
> 
> However it is very rare that Python is used in the kind
> of scenarios where that kind of space saving is
> significant (usually embedded controllers with
> tiny memory spaces).

People routinely use Python with seriously large amounts of data, 
hundreds of megabytes or even gigabytes, and freely make copies of it, 
secure in the knowledge that they don't have to do much to manage memory 
because the garbage collector will do so for them.

Consider a simple expression like:

text = '"' + (text.strip().lower().replace(',', ' ')) + '"'

That makes five slightly-modified copies of the original text. And 
that's just from a single short expression. Memory is cheap, but not so 
cheap that we can afford to keep around hundreds of redundant copies of 
large objects.

Generally speaking, we can afford to be profligate with copies of 
objects because we know they won't "leak" and survive for very long: at 
worst, they will be reclaimed when the current function returns. The 
only times we need worry about lifetimes of objects are when we are 
working in the global scope, or when adding attributes to long-lived 
objects. Regardless of whether we write the above as a single 
expression, or split it over many lines:

text = text.strip()
text = text.lower()
text = text.replace(',', ' ')
text = '"' + text
text = text + '"'

the result will be the same. And I don't think the above would be more 
understandable if we invented separate names for each of the 
intermediate results.


-- 
Steve

From steve at pearwood.info  Fri Dec  7 23:33:56 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 8 Dec 2018 15:33:56 +1100
Subject: [Tutor] Regarding Python api script
In-Reply-To: <007801d48e52$879f1800$96dd4800$@verizon.net>
References: <34862259-e9dd-17dd-ef8c-10a9cfc732c9@yahoo.co.uk>
 <CANUFOz=uj422ArHGuzfGKW7nMLg1Oe5BVo8VyeviLV_rCzDUEg@mail.gmail.com>
 <pu5e6v$r69$1@blaine.gmane.org>
 <CANUFOzkgV5dBdY3Nr+CaZXQqdCHuQE+cRjjFu88W2-AT28V_tg@mail.gmail.com>
 <4f99ead7-9fdd-b484-332d-0cc558d09ecf@yahoo.co.uk>
 <CANUFOzk8VjmguVoWN9J+YOz8QwWHhJvhZvF9OoU0gz7AnAt5BA@mail.gmail.com>
 <pucei1$4ml$1@blaine.gmane.org> <008901d48dd2$613e2be0$23ba83a0$@verizon.net>
 <20181207105932.GA13061@ando.pearwood.info>
 <007801d48e52$879f1800$96dd4800$@verizon.net>
Message-ID: <20181208043356.GJ13061@ando.pearwood.info>

On Fri, Dec 07, 2018 at 12:30:18PM -0500, Avi Gross wrote:

> #  > I have often seen something like this done with methods, such as to
> #  > emulate decorator functionality where a method is created in an 
> #  > object with a name
> #  > and the very next method created has the same name with the same
> #  > function name as an argument to replace it.
> #  

Then I asked:

> # I don't understand this. Can you show an example? Even if made up?
[...]

And Avi explained:

> class classy(object):
>     ...
>     # Original function first of this name
>     def recreational_fun(args):
>         ...
> 
>     # New function, second of this name
>     recreational _fun = timing_is_everything(recreational _fun, args)

Ah, now I understand!

That's a decorator. The only difference is that it predates the 
introduction of *decorator syntax* using the @ symbol.

The *decorator design pattern* is an idiom which uses a wrapper function 
to add functionality to an existing function. Given a language where 
functions are first-class values, you can pass a function to another 
function as argument, and return yet a third newly constructed function. 
So this gives us the decorate pattern. In pseudo-code:

# Decorator pattern
def decorate(input):
    output = wrap input with extra functionality
    return output

new_function = decorate(function)

But if you don't need to keep both the old and new versions of the 
function, you can just write this:

function = decorate(function)


https://en.wikipedia.org/wiki/Decorator_pattern

In Java, it is more common to apply the decorator pattern to instances 
rather than functions or classes, but the principle is the same. Even if 
Java makes something conceptually simple incredibly complex :-)

*Technically* speaking, such a design could have been used in Python 
going all the way back to Python 1.4 or older, since functions have 
always been first-class values. But it only got really useful from about 
Python 2.2, with the introduction of closures to the language, and the 
classmethod and staticmethod decorators.

But initially, we didn't have dedicated syntax for applying a decorator, 
and had to write:

def function():
    ...

function = decorate(function)


which repeats the function name three times. It wasn't until 2.4 that 
the @ syntax was approved:

@decorate
def function():
    ...


which is literally syntactic sugar for the original version.

See PEP 318 for the background:

https://www.python.org/dev/peps/pep-0318/




-- 
Steve

From cs at cskk.id.au  Sat Dec  8 17:23:59 2018
From: cs at cskk.id.au (Cameron Simpson)
Date: Sun, 9 Dec 2018 09:23:59 +1100
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <20181207102004.GY13061@ando.pearwood.info>
References: <20181207102004.GY13061@ando.pearwood.info>
Message-ID: <20181208222359.GA27991@cskk.homeip.net>

On 07Dec2018 21:20, Steven D'Aprano <steve at pearwood.info> wrote:
>On Fri, Dec 07, 2018 at 02:06:16PM +0530, Sunil Tech wrote:
>> I am using Python 2.7.8
>
>That is important information.
>
>Python 2 unfortunately predates Unicode, and when it was added some bad
>decisions were made. For example, we can write this in Python 2:
>
>>>> txt = "abc?"
>
>but it is a lie, because what we get isn't the string we typed, but the
>interpreters *bad guess* that we actually meant this:
>
>>>> txt
>'abc\xcf\x80'

Wow. I did not know that! I imagined Python 2 would have simply rejected 
such a string (out of range characters -- ordinals >= 256 -- in a "byte" 
string).

Cheers,
Cameron Simpson <cs at cskk.id.au>

From steve at pearwood.info  Sat Dec  8 18:40:10 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 9 Dec 2018 10:40:10 +1100
Subject: [Tutor] Trouble in dealing with special characters.
In-Reply-To: <20181208222359.GA27991@cskk.homeip.net>
References: <20181207102004.GY13061@ando.pearwood.info>
 <20181208222359.GA27991@cskk.homeip.net>
Message-ID: <20181208234010.GM13061@ando.pearwood.info>

On Sun, Dec 09, 2018 at 09:23:59AM +1100, Cameron Simpson wrote:
> On 07Dec2018 21:20, Steven D'Aprano <steve at pearwood.info> wrote:


# Python 2
> >>>>txt = "abc?"
> >
> >but it is a lie, because what we get isn't the string we typed, but the
> >interpreters *bad guess* that we actually meant this:
> >
> >>>>txt
> >'abc\xcf\x80'
> 
> Wow. I did not know that! I imagined Python 2 would have simply rejected 
> such a string (out of range characters -- ordinals >= 256 -- in a "byte" 
> string).

Nope.

Python 2 tries hard to make bytes and unicode text work together. If 
your strings are pure ASCII, it "Just Works" and it seems great but on 
trickier cases it can lead to really confusing errors.

Behind the scenes, what the interpreter is doing is using some platform- 
specific codec (ASCII, UTF-8, or similar) to automatically encode/decode 
from bytes to text or vise versa. This sort of "Do What I Mean" 
processing can work, up to the point that it doesn't, then it all goes 
pearshaped and you have silent failures and hard-to-diagnose errors.

That's why Python 3 takes a hard-line policy that you cannot mix text 
and bytes (except, possibly, if one is the empty string) except by 
explicitly converting from one to the other.

-- 
Steve

From steve at pearwood.info  Sat Dec  8 19:02:47 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 9 Dec 2018 11:02:47 +1100
Subject: [Tutor] Weird Unicode encode/decode errors in Python 2
Message-ID: <20181209000247.GN13061@ando.pearwood.info>

This is not a request for help, but a demonstration of what can go wrong 
with text processing in Python 2.

Following up on the "Special characters" thread, one of the design flaws 
of Python 2 is that byte strings and text strings offer BOTH decode and 
encode methods, even though only one is meaningful in each case.[1]

- text strings are ENCODED to bytes;
- byte are DECODED to text strings.

One of the symptoms of getting it wrong is when you take a Unicode text 
string and encode/decode it but get an error from the *opposite* 
operation:


py> u'?'.decode('latin1')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in 
position 0: ordinal not in range(128)


Look at what happens: I try to DECODE a string, but get an ENCODE error. 
And even though I specified Latin 1 as the codec, Python uses ASCII. 
What is going on here?

Behind the scenes, the interpreter takes my text u'?' (a Unicode string) 
and attempts to *encode* it to bytes first, using the default ASCII 
codec. That fails. Had it succeeded, it would have then attempted to 
*decode* those bytes using Latin 1.

Similarly:

py> b = u'?'.encode('latin1')
py> print repr(b)
'\xe4'
py> b.encode('latin1')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: 
ordinal not in range(128)


The error here is that I tried to encode a bunch of bytes, instead of 
decoding them. But the insidious thing about this error is if you are 
working with pure ASCII, it seems to work:

py> 'ascii'.encode('utf-16')
'\xff\xfea\x00s\x00c\x00i\x00i\x00'

That is, it *seems* to work because there's no error, but the result is 
pretty much meaningless: I *intended* to get a UTF-16 Unicode string, 
but instead I ended up with bytes just like I started with.

Python 3 fixes this bug magnet by removing the decode method from 
Unicode text strings, and the encode method from byte-strings.



[1] Technically this is not so, as there are codecs which can be used to 
convert bytes to bytes, or text to text. But the vast majority of common 
cases, codecs are used to convert bytes to text and vice versa. For the 
rare exception, we can use the "codecs" module.


-- 
Steve

From asad.hasan2004 at gmail.com  Sun Dec  9 05:15:07 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Sun, 9 Dec 2018 15:45:07 +0530
Subject: [Tutor] Increase performance of the script
Message-ID: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>

Hi All ,

          I have the following code to search for an error and prin the
solution .

/A/B/file1.log size may vary from 5MB -5 GB

f4 = open (r" /A/B/file1.log  ", 'r' )
string2=f4.readlines()
for i in range(len(string2)):
    position=i
    lastposition =position+1
    while True:
         if re.search('Calling rdbms/admin',string2[lastposition]):
          break
         elif lastposition==len(string2)-1:
          break
         else:
          lastposition += 1
    errorcheck=string2[position:lastposition]
    for i in range ( len ( errorcheck ) ):
        if re.search ( r'"error(.)*13?"', errorcheck[i] ):
            print "Reason of error \n", errorcheck[i]
            print "script \n" , string2[position]
            print "block of code \n"
            print errorcheck[i-3]
            print errorcheck[i-2]
            print errorcheck[i-1]
            print errorcheck[i]
            print "Solution :\n"
            print "Verify the list of objects belonging to Database "
            break
    else:
        continue
    break

The problem I am facing in performance issue it takes some minutes to print
out the solution . Please advice if there can be performance enhancements
to this script .

Thanks,

From alan.gauld at yahoo.co.uk  Sun Dec  9 11:14:10 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sun, 9 Dec 2018 16:14:10 +0000
Subject: [Tutor] Increase performance of the script
In-Reply-To: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
References: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
Message-ID: <pujesf$5go$1@blaine.gmane.org>

On 09/12/2018 10:15, Asad wrote:

> f4 = open (r" /A/B/file1.log  ", 'r' )

Are you sure you want that space at the start ofthe filename?


> string2=f4.readlines()

Here you read the entire file into memory. OK for small files
but if it really can be 5GB that's a lot of memory being used.

> for i in range(len(string2)):

This is usually the wrong thing to do in Python. Aside
from the loss of readability it requires the interpreter
to do a lot of indexing operations which is not the
fastest way to access things.

>     position=i
>     lastposition =position+1
>     while True:
>          if re.search('Calling rdbms/admin',string2[lastposition]):

You are using regex to search for a fixed string.
Its simpler and faster to use string methods
either foo in string or string.find(foo)

>           break
>          elif lastposition==len(string2)-1:
>           break
>          else:
>           lastposition += 1

This means you iterate over the whole file content
multiple times. Once for every line in the file.
If the file has 1000 lines that means you do these
tests close to 1000000/2 times!

This is probably your biggest performance issue.

>     errorcheck=string2[position:lastposition]
>     for i in range ( len ( errorcheck ) ):
>         if re.search ( r'"error(.)*13?"', errorcheck[i] )

This use of regex is valid since its a pattern.
But it might be more efficient to join the lines
and do a single regex search across lone boundaries.
But you need to test/time it to see.

But you also do another loop inside the outer loop.
You need to look at how/whether you can eliminate
all these inner loops and just loop over the file
once - ideally without reading the entire thing
into memory before you start.

Processing it as you read it will be much more efficient.
On a previous thread we showed you several ways you
could approach that.

>             print "Reason of error \n", errorcheck[i]
>             print "script \n" , string2[position]
>             print "block of code \n"
>             print errorcheck[i-3]
>             print errorcheck[i-2]
>             print errorcheck[i-1]
>             print errorcheck[i]
>             print "Solution :\n"
>             print "Verify the list of objects belonging to Database "
>             break
>     else:
>         continue
>     break



-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From __peter__ at web.de  Sun Dec  9 15:17:53 2018
From: __peter__ at web.de (Peter Otten)
Date: Sun, 09 Dec 2018 21:17:53 +0100
Subject: [Tutor] Increase performance of the script
References: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
Message-ID: <pujt5e$eus$1@blaine.gmane.org>

Asad wrote:

> Hi All ,
> 
>           I have the following code to search for an error and prin the
> solution .
> 
> /A/B/file1.log size may vary from 5MB -5 GB
> 
> f4 = open (r" /A/B/file1.log  ", 'r' )
> string2=f4.readlines()

Do not read the complete file into memory. Read one line at a time and keep 
only those lines around that you may have to look at again.

> for i in range(len(string2)):
>     position=i
>     lastposition =position+1
>     while True:
>          if re.search('Calling rdbms/admin',string2[lastposition]):
>           break
>          elif lastposition==len(string2)-1:
>           break
>          else:
>           lastposition += 1

You are trying to find a group of lines. The way you do it for a file of the 
structure

foo
bar
baz
end-of-group-1
ham
spam
end-of-group-2

you find the groups

foo
bar
baz
end-of-group-1

bar
baz
end-of-group-1

baz
end-of-group-1

ham
spam
end-of-group-2

spam
end-of-group-2

That looks like a lot of redundancy which you can probably avoid. But 
wait...


>     errorcheck=string2[position:lastposition]
>     for i in range ( len ( errorcheck ) ):
>         if re.search ( r'"error(.)*13?"', errorcheck[i] ):
>             print "Reason of error \n", errorcheck[i]
>             print "script \n" , string2[position]
>             print "block of code \n"
>             print errorcheck[i-3]
>             print errorcheck[i-2]
>             print errorcheck[i-1]
>             print errorcheck[i]
>             print "Solution :\n"
>             print "Verify the list of objects belonging to Database "
>             break
>     else:
>         continue
>     break

you throw away almost all the hard work to look for the line containing 
those four lines? It looks like you only need the 
"error...13" lines, the three lines that precede it and the last 
"Calling..." line occuring before the "error...13".

> The problem I am facing in performance issue it takes some minutes to
> print out the solution . Please advice if there can be performance
> enhancements to this script .

If you want to learn the Python way you should try hard to write your 
scripts without a single

for i in range(...):
    ...

loop. This style is usually the last resort, it may work for small datasets, 
but as soon as you have to deal with large files performance dives.
Even worse, these loops tend to make your code hard to debug.

Below is a suggestion for an implementation of what your code seems to be 
doing that only remembers the four recent lines and works with a single 
loop. If that saves you some time use that time to clean the scripts you 
have lying around from occurences of "for i in range(....): ..." ;)


from __future__ import print_function

import re
import sys
from collections import deque


def show(prompt, *values):
    print(prompt)
    for value in values:
        print(" {}".format(value.rstrip("\n")))


def process(filename):
    tail = deque(maxlen=4)  # the last four lines
    script = None
    with open(filename) as instream:
        for line in instream:
            tail.append(line)
            if "Calling rdbms/admin" in line:
                script = line
            elif re.search('"error(.)*13?"', line) is not None:
                show("Reason of error:", tail[-1])
                show("Script:", script)
                show("Block of code:", *tail)
                show(
                    "Solution",
                    "Verify the list of objects belonging to Database"
                )
                break


if __name__ == "__main__":
    filename = sys.argv[1]
    process(filename)



From steve at pearwood.info  Sun Dec  9 17:43:20 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 10 Dec 2018 09:43:20 +1100
Subject: [Tutor] Increase performance of the script
In-Reply-To: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
References: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
Message-ID: <20181209224318.GW13061@ando.pearwood.info>

On Sun, Dec 09, 2018 at 03:45:07PM +0530, Asad wrote:
> Hi All ,
> 
>           I have the following code to search for an error and prin the
> solution .
> 
> /A/B/file1.log size may vary from 5MB -5 GB
[...]

> The problem I am facing in performance issue it takes some minutes to print
> out the solution . Please advice if there can be performance enhancements
> to this script .

How many minutes is "some"? If it takes 2 minutes to analyse a 5GB file, 
that's not bad performance. If it takes 2 minutes to analyse a 5MB file, 
that's not so good.



-- 
Steve

From steve at pearwood.info  Sun Dec  9 19:00:58 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 10 Dec 2018 11:00:58 +1100
Subject: [Tutor] Increase performance of the script
In-Reply-To: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
References: <CAG3LsKE0Ne3awb8_ZFk=cxq_3OVB9vu8Siu2ivR6tPVbOuAKJw@mail.gmail.com>
Message-ID: <20181210000058.GY13061@ando.pearwood.info>

On Sun, Dec 09, 2018 at 03:45:07PM +0530, Asad wrote:
> Hi All ,
> 
>           I have the following code to search for an error and prin the
> solution .

Please tidy your code before asking for help optimizing it. We're 
volunteers, not being paid to work on your problem, and your code is too 
hard to understand.

Some comments:


> f4 = open (r" /A/B/file1.log  ", 'r' )
> string2=f4.readlines()

You have a variable "f4". Where are f1, f2 and f3?

You have a variable "string2", which is a lie, because it is not a 
string, it is a list.

I will be very surprised if the file name you show is correct. It has a 
leading space, and two trailing spaces.


> for i in range(len(string2)):
>     position=i

Poor style. In Python, you almost never need to write code that iterates 
over the indexes (this is not Pascal). You don't need the assignment 
position=i. Better:

for position, line in enumerate(lines):
    ...


>     lastposition =position+1

Poorly named variable. You call it "last position", but it is actually 
the NEXT position.


>     while True:
>          if re.search('Calling rdbms/admin',string2[lastposition]):

Unnecessary use of regex, which will be slow. Better:

    if 'Calling rdbms/admin' in line:
        break


>           break
>          elif lastposition==len(string2)-1:
>           break

If you iterate over the lines, you don't need to check for the end of 
the list yourself.


A better solution is to use the *accumulator* design pattern to collect 
a block of lines for further analysis:

# Untested.
with open(filename, 'r') as f:
    block = []
    inside_block = False
    for line in f:
        line = line.strip()
        if inside_block:
            if line == "End of block":
                inside_block = False
                process(block)
                block = []  # Reset to collect the next block.
            else:
                block.append(line)
        elif line == "Start of block":
            inside_block = True
    # At the end of the loop, we might have a partial block.
    if block:
         process(block)


Your process() function takes a single argument, the list of lines which 
makes up the block you care about.

If you need to know the line numbers, it is easy to adapt:

    for line in f:

becomes:

    for linenumber, line in enumerate(f):
        # The next line is not needed in Python 3.
        linenumber += 1  # Adjust to start line numbers at 1 instead of 0

and:
 
    block.append(line)

becomes 

    block.append((linenumber, line))


If you re-write your code using this accumulator pattern, using ordinary 
substring matching and equality instead of regular expressions whenever 
possible, I expect you will see greatly improved performance (as well as 
being much, much easier to understand and maintain).



-- 
Steve

From matthew.polack at htlc.vic.edu.au  Mon Dec 10 20:09:16 2018
From: matthew.polack at htlc.vic.edu.au (Matthew Polack)
Date: Tue, 11 Dec 2018 12:09:16 +1100
Subject: [Tutor] Any 'graphical' ways of learning Python
In-Reply-To: <MWHPR11MB1710B68D72219956C42B039DE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>
References: <MWHPR11MB17108CB14624CD48891131AEE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>
 <MWHPR11MB1710B68D72219956C42B039DE0A90@MWHPR11MB1710.namprd11.prod.outlook.com>
Message-ID: <CAFkpTikgN=OZNiHMdkOVogi+P2dfMSMUJfP5Y9HPE2ZGzaRExg@mail.gmail.com>

Hi Mike and everyone who replied to the tutor question re: GUI ideas for
Python.

Really value all the ideas and suggestions...that is very helpful! Thanks
for taking time to reply with so much great information.

We're just in wind down mode/rush for our close of school year ...but will
really enjoy looking through all the various links and ideas as I prepare
the coding course for 2019. Will use them, to upskill myself too.

Thanks so much!

- Matthew Polack



On Fri, Dec 7, 2018 at 6:05 AM Mike Barnett <mike_barnett at hotmail.com>
wrote:

> Oh, one more thing on this topic... there are tutorial videos available
> for PySimpleGUI, both basic and advanced.
>
> Basic 5 video series:
> https://www.youtube.com/playlist?list=PLl8dD0doyrvHMoJGTdMtgLuHymaqJVjzt
>
> Additional 9 in-depth videos:
> https://www.youtube.com/playlist?list=PLl8dD0doyrvHMoJGTdMtgLuHymaqJVjzt
>
>
> -mike
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

-- 
**Disclaimer: *Whilst every attempt has been made to ensure that material 
contained in this email is free from computer viruses or other defects, the 
attached files are provided, and may only be used, on the basis that the 
user assumes all responsibility for use of the material transmitted. This 
email is intended only for the use of the individual or entity named above 
and may contain information that is confidential and privileged. If you are 
not the intended recipient, please note that any dissemination, 
distribution or copying of this email is strictly prohibited. If you have 
received this email in error, please notify us immediately by return email 
or telephone +61 3 5382 2529**?and destroy the original message.*

From asad.hasan2004 at gmail.com  Tue Dec 11 10:37:58 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Tue, 11 Dec 2018 21:07:58 +0530
Subject: [Tutor] Increase performance of the script
In-Reply-To: <mailman.8.1544461202.31434.tutor@python.org>
References: <mailman.8.1544461202.31434.tutor@python.org>
Message-ID: <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>

Hi All,

          I used your solution , however found a strange issue with deque :

I am using python 2.6.6:

>>> import collections
>>> d = collections.deque('abcdefg')
>>> print 'Deque:', d
  File "<stdin>", line 1
    print 'Deque:', d
                 ^
SyntaxError: invalid syntax
>>> print ('Deque:', d)
Deque: deque(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>>> print d
  File "<stdin>", line 1
    print d
          ^
SyntaxError: invalid syntax
>>> print (d)
deque(['a', 'b', 'c', 'd', 'e', 'f', 'g'])

In python 2.6 print statement work as print "Solution"

however after import collection I have to use print with print("Solution")
is this a known issue ?

Please let me know .

Thanks,

On Mon, Dec 10, 2018 at 10:30 PM <tutor-request at python.org> wrote:

> Send Tutor mailing list submissions to
>         tutor at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/tutor
> or, via email, send a message with subject or body 'help' to
>         tutor-request at python.org
>
> You can reach the person managing the list at
>         tutor-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Tutor digest..."
> Today's Topics:
>
>    1. Re: Increase performance of the script (Peter Otten)
>    2. Re: Increase performance of the script (Steven D'Aprano)
>    3. Re: Increase performance of the script (Steven D'Aprano)
>
>
>
> ---------- Forwarded message ----------
> From: Peter Otten <__peter__ at web.de>
> To: tutor at python.org
> Cc:
> Bcc:
> Date: Sun, 09 Dec 2018 21:17:53 +0100
> Subject: Re: [Tutor] Increase performance of the script
> Asad wrote:
>
> > Hi All ,
> >
> >           I have the following code to search for an error and prin the
> > solution .
> >
> > /A/B/file1.log size may vary from 5MB -5 GB
> >
> > f4 = open (r" /A/B/file1.log  ", 'r' )
> > string2=f4.readlines()
>
> Do not read the complete file into memory. Read one line at a time and
> keep
> only those lines around that you may have to look at again.
>
> > for i in range(len(string2)):
> >     position=i
> >     lastposition =position+1
> >     while True:
> >          if re.search('Calling rdbms/admin',string2[lastposition]):
> >           break
> >          elif lastposition==len(string2)-1:
> >           break
> >          else:
> >           lastposition += 1
>
> You are trying to find a group of lines. The way you do it for a file of
> the
> structure
>
> foo
> bar
> baz
> end-of-group-1
> ham
> spam
> end-of-group-2
>
> you find the groups
>
> foo
> bar
> baz
> end-of-group-1
>
> bar
> baz
> end-of-group-1
>
> baz
> end-of-group-1
>
> ham
> spam
> end-of-group-2
>
> spam
> end-of-group-2
>
> That looks like a lot of redundancy which you can probably avoid. But
> wait...
>
>
> >     errorcheck=string2[position:lastposition]
> >     for i in range ( len ( errorcheck ) ):
> >         if re.search ( r'"error(.)*13?"', errorcheck[i] ):
> >             print "Reason of error \n", errorcheck[i]
> >             print "script \n" , string2[position]
> >             print "block of code \n"
> >             print errorcheck[i-3]
> >             print errorcheck[i-2]
> >             print errorcheck[i-1]
> >             print errorcheck[i]
> >             print "Solution :\n"
> >             print "Verify the list of objects belonging to Database "
> >             break
> >     else:
> >         continue
> >     break
>
> you throw away almost all the hard work to look for the line containing
> those four lines? It looks like you only need the
> "error...13" lines, the three lines that precede it and the last
> "Calling..." line occuring before the "error...13".
>
> > The problem I am facing in performance issue it takes some minutes to
> > print out the solution . Please advice if there can be performance
> > enhancements to this script .
>
> If you want to learn the Python way you should try hard to write your
> scripts without a single
>
> for i in range(...):
>     ...
>
> loop. This style is usually the last resort, it may work for small
> datasets,
> but as soon as you have to deal with large files performance dives.
> Even worse, these loops tend to make your code hard to debug.
>
> Below is a suggestion for an implementation of what your code seems to be
> doing that only remembers the four recent lines and works with a single
> loop. If that saves you some time use that time to clean the scripts you
> have lying around from occurences of "for i in range(....): ..." ;)
>
>
> from __future__ import print_function
>
> import re
> import sys
> from collections import deque
>
>
> def show(prompt, *values):
>     print(prompt)
>     for value in values:
>         print(" {}".format(value.rstrip("\n")))
>
>
> def process(filename):
>     tail = deque(maxlen=4)  # the last four lines
>     script = None
>     with open(filename) as instream:
>         for line in instream:
>             tail.append(line)
>             if "Calling rdbms/admin" in line:
>                 script = line
>             elif re.search('"error(.)*13?"', line) is not None:
>                 show("Reason of error:", tail[-1])
>                 show("Script:", script)
>                 show("Block of code:", *tail)
>                 show(
>                     "Solution",
>                     "Verify the list of objects belonging to Database"
>                 )
>                 break
>
>
> if __name__ == "__main__":
>     filename = sys.argv[1]
>     process(filename)
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: "Steven D'Aprano" <steve at pearwood.info>
> To: tutor at python.org
> Cc:
> Bcc:
> Date: Mon, 10 Dec 2018 09:43:20 +1100
> Subject: Re: [Tutor] Increase performance of the script
> On Sun, Dec 09, 2018 at 03:45:07PM +0530, Asad wrote:
> > Hi All ,
> >
> >           I have the following code to search for an error and prin the
> > solution .
> >
> > /A/B/file1.log size may vary from 5MB -5 GB
> [...]
>
> > The problem I am facing in performance issue it takes some minutes to
> print
> > out the solution . Please advice if there can be performance enhancements
> > to this script .
>
> How many minutes is "some"? If it takes 2 minutes to analyse a 5GB file,
> that's not bad performance. If it takes 2 minutes to analyse a 5MB file,
> that's not so good.
>
>
>
> --
> Steve
>
>
>
>
> ---------- Forwarded message ----------
> From: "Steven D'Aprano" <steve at pearwood.info>
> To: tutor at python.org
> Cc:
> Bcc:
> Date: Mon, 10 Dec 2018 11:00:58 +1100
> Subject: Re: [Tutor] Increase performance of the script
> On Sun, Dec 09, 2018 at 03:45:07PM +0530, Asad wrote:
> > Hi All ,
> >
> >           I have the following code to search for an error and prin the
> > solution .
>
> Please tidy your code before asking for help optimizing it. We're
> volunteers, not being paid to work on your problem, and your code is too
> hard to understand.
>
> Some comments:
>
>
> > f4 = open (r" /A/B/file1.log  ", 'r' )
> > string2=f4.readlines()
>
> You have a variable "f4". Where are f1, f2 and f3?
>
> You have a variable "string2", which is a lie, because it is not a
> string, it is a list.
>
> I will be very surprised if the file name you show is correct. It has a
> leading space, and two trailing spaces.
>
>
> > for i in range(len(string2)):
> >     position=i
>
> Poor style. In Python, you almost never need to write code that iterates
> over the indexes (this is not Pascal). You don't need the assignment
> position=i. Better:
>
> for position, line in enumerate(lines):
>     ...
>
>
> >     lastposition =position+1
>
> Poorly named variable. You call it "last position", but it is actually
> the NEXT position.
>
>
> >     while True:
> >          if re.search('Calling rdbms/admin',string2[lastposition]):
>
> Unnecessary use of regex, which will be slow. Better:
>
>     if 'Calling rdbms/admin' in line:
>         break
>
>
> >           break
> >          elif lastposition==len(string2)-1:
> >           break
>
> If you iterate over the lines, you don't need to check for the end of
> the list yourself.
>
>
> A better solution is to use the *accumulator* design pattern to collect
> a block of lines for further analysis:
>
> # Untested.
> with open(filename, 'r') as f:
>     block = []
>     inside_block = False
>     for line in f:
>         line = line.strip()
>         if inside_block:
>             if line == "End of block":
>                 inside_block = False
>                 process(block)
>                 block = []  # Reset to collect the next block.
>             else:
>                 block.append(line)
>         elif line == "Start of block":
>             inside_block = True
>     # At the end of the loop, we might have a partial block.
>     if block:
>          process(block)
>
>
> Your process() function takes a single argument, the list of lines which
> makes up the block you care about.
>
> If you need to know the line numbers, it is easy to adapt:
>
>     for line in f:
>
> becomes:
>
>     for linenumber, line in enumerate(f):
>         # The next line is not needed in Python 3.
>         linenumber += 1  # Adjust to start line numbers at 1 instead of 0
>
> and:
>
>     block.append(line)
>
> becomes
>
>     block.append((linenumber, line))
>
>
> If you re-write your code using this accumulator pattern, using ordinary
> substring matching and equality instead of regular expressions whenever
> possible, I expect you will see greatly improved performance (as well as
> being much, much easier to understand and maintain).
>
>
>
> --
> Steve
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
Asad Hasan
+91 9582111698

From __peter__ at web.de  Tue Dec 11 13:42:40 2018
From: __peter__ at web.de (Peter Otten)
Date: Tue, 11 Dec 2018 19:42:40 +0100
Subject: [Tutor] print statement vs print() function,
 was Re: Increase performance of the script
References: <mailman.8.1544461202.31434.tutor@python.org>
 <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
Message-ID: <pup0au$khb$1@blaine.gmane.org>

Asad wrote:

> Hi All,
> 
>           I used your solution , however found a strange issue with deque
>           :
> 
> I am using python 2.6.6:

Consider switching to Python 3; my code as posted already works with that.
 
>>>> import collections
>>>> d = collections.deque('abcdefg')
>>>> print 'Deque:', d
>   File "<stdin>", line 1
>     print 'Deque:', d
>                  ^
> SyntaxError: invalid syntax

This is the effect of a

from __future__ import print_function

import. Once that is used print becomes a function and has to be invoked as 
such. 

If you don't want this remove the "from __future__ print_function" statement 
and remove the parentheses from all print-s (or at least those with multiple 
arguments).

Note that in default Python 2

print("foo", "bar")

works both with and without and parens, but the parens constitute a tuple. 
So

>>> print "foo", "bar"  # print statement with two args
foo bar
>>> print ("foo", "bar")  # print statement with a single tuple arg
('foo', 'bar')
>>> from __future__ import print_function
>>> print "foo", "bar"  # illegal, print() is now a function
  File "<stdin>", line 1
    print "foo", "bar"
              ^
SyntaxError: invalid syntax
>>> print("foo", "bar")  # invoke print() function with two args
foo bar



From steve at pearwood.info  Tue Dec 11 17:21:17 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 12 Dec 2018 09:21:17 +1100
Subject: [Tutor] Increase performance of the script
In-Reply-To: <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
References: <mailman.8.1544461202.31434.tutor@python.org>
 <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
Message-ID: <20181211222106.GG13061@ando.pearwood.info>

On Tue, Dec 11, 2018 at 09:07:58PM +0530, Asad wrote:
> Hi All,
> 
>           I used your solution , however found a strange issue with deque :

No you haven't. You found a *syntax error*, as the exception says:

> >>> print 'Deque:', d
>   File "<stdin>", line 1
>     print 'Deque:', d
>                  ^
> SyntaxError: invalid syntax

which means the error occurs before the interpreter runs the code. You 
could replace the above line with any similar line:

print 'Not a deque', 1.2345

and you will get the same error.

When you are faced with an error in the interactive interpreter, you 
should try different things to see how they effect the problem. Does the 
problem go away if you use a float instead of a deque? If you change the 
string, does the problem go away? If you swap the order, does the 
problem go away? What if you use a single value instead of two?

This is called "debugging", and as a programmer, you need to learn how 
to do this.


[...]
> In python 2.6 print statement work as print "Solution"
> however after import collection I have to use print with print("Solution")
> is this a known issue ?

As Peter says, you must have run 

from __future__ import print_function

to see this behaviour. This has nothing to do with import collection. 
You can debug that for yourself by exiting the interactive interpreter, 
starting it up again, and trying to print before and after importing 
collection.



-- 
Steve

From avigross at verizon.net  Wed Dec 12 00:52:09 2018
From: avigross at verizon.net (Avi Gross)
Date: Wed, 12 Dec 2018 00:52:09 -0500
Subject: [Tutor] Increase performance of the script
In-Reply-To: <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
References: <mailman.8.1544461202.31434.tutor@python.org>
 <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
Message-ID: <001101d491de$d28b0380$77a10a80$@verizon.net>

Asad,

I wonder if an import from __future__ happened, perhaps in the version of
collections you used. Later versions of 2.x allow optional use of the 3.x
style of print.

When you redefine print, the old statement style is hidden or worse.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Asad
Sent: Tuesday, December 11, 2018 10:38 AM
To: tutor at python.org
Subject: [Tutor] Increase performance of the script

Hi All,

          I used your solution , however found a strange issue with deque :

I am using python 2.6.6:

>>> import collections
>>> d = collections.deque('abcdefg')
>>> print 'Deque:', d
  File "<stdin>", line 1
    print 'Deque:', d
                 ^
SyntaxError: invalid syntax
>>> print ('Deque:', d)
Deque: deque(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>>> print d
  File "<stdin>", line 1
    print d
          ^
SyntaxError: invalid syntax
>>> print (d)
deque(['a', 'b', 'c', 'd', 'e', 'f', 'g'])

In python 2.6 print statement work as print "Solution"

however after import collection I have to use print with print("Solution")
is this a known issue ?

Please let me know .

Thanks,



From hoangquocdat7 at gmail.com  Wed Dec 12 00:26:30 2018
From: hoangquocdat7 at gmail.com (Hoang Quoc Dat)
Date: Wed, 12 Dec 2018 11:26:30 +0600
Subject: [Tutor] Python_About "Guess my number" program
Message-ID: <CAC_wLw3beMSf1AjOWvd5XC4Wu0o-oCop2Pa0ictK=N8ArDO4hA@mail.gmail.com>

Dear all,

This is Dat, I am a newbie in learning Python. My recent attempt is to read
and practice the book of Michael Dawson-Python Programming for the Absolute
Beginner 3rd Edition. At the end of Chapter 3 there is a challenge of
creating "Guess my number" program where the computer guess your number
from 1-100.

I have been trying to get it through but stuck while attempting to figure
out the loop of how the computer will understand and narrow down the
guessing range. After a while, I went on and search on the internet and
find out 1 similar program but the coding logic is like: The computer keeps
guessing the middle number of the range and narrow down the range by
focusing on only the upper/lower part of the range after receving our
answer.

I somehow have come to believe that the Python language and our logical
thinking can do better than that option of choosing the middle number all
the time but I still cannot find out the proper answer (probably because I
am very much new to this). Could someone provide answer for this program?
or some sample code on how to make the computer understand it like human do?

Thank you in advance and have a nice day everyone.

From steve at pearwood.info  Wed Dec 12 03:22:14 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 12 Dec 2018 19:22:14 +1100
Subject: [Tutor] Increase performance of the script
In-Reply-To: <001101d491de$d28b0380$77a10a80$@verizon.net>
References: <mailman.8.1544461202.31434.tutor@python.org>
 <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
 <001101d491de$d28b0380$77a10a80$@verizon.net>
Message-ID: <20181212082213.GJ13061@ando.pearwood.info>

On Wed, Dec 12, 2018 at 12:52:09AM -0500, Avi Gross wrote:
> Asad,
> 
> I wonder if an import from __future__ happened, perhaps in the version of
> collections you used. Later versions of 2.x allow optional use of the 3.x
> style of print.


The effect of __future__ imports, like any other import, is only within 
the module that actually does the import. Even in the unlikely event 
that collections did such a future import, it would only effect print 
within that module, not globally or in the interactive interpreter.

Here's a demo:

# prfunc_demo.py
from __future__ import print_function

try:
    exec("print 123")
except SyntaxError:
    print("old style print failed, as expected")
    print("as print is now a function")



And importing it into the interactive interpreter shows that the effect 
of the future import is localised:

[steve at ando ~]$ python2.6
Python 2.6.7 (r267:88850, Mar 10 2012, 12:32:58)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
py> import prfunc_demo
old style print failed, as expected
as print is now a function
py> print "But here in the REPL, nothing has changed."
But here in the REPL, nothing has changed.







-- 
Steve

From alan.gauld at yahoo.co.uk  Wed Dec 12 03:28:24 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 12 Dec 2018 08:28:24 +0000
Subject: [Tutor] Python_About "Guess my number" program
In-Reply-To: <CAC_wLw3beMSf1AjOWvd5XC4Wu0o-oCop2Pa0ictK=N8ArDO4hA@mail.gmail.com>
References: <CAC_wLw3beMSf1AjOWvd5XC4Wu0o-oCop2Pa0ictK=N8ArDO4hA@mail.gmail.com>
Message-ID: <puqgn5$h7q$1@blaine.gmane.org>

On 12/12/2018 05:26, Hoang Quoc Dat wrote:

> find out 1 similar program but the coding logic is like: The computer keeps
> guessing the middle number of the range and narrow down the range by
> focusing on only the upper/lower part of the range after receiving our
> answer.
> 
> I somehow have come to believe that the Python language and our logical
> thinking can do better than that option of choosing the middle number all
> the time but I still cannot find out the proper answer

Actually, that is quite an efficient algorithm.
It even has a name: the "binary chop".

But can you think of a better one? Forget about
the computer. Ask how you would do it using pen
and paper. How would you guess the number?

Once you understand that you should be able to
program Python to do the same thing.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Wed Dec 12 03:52:52 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 12 Dec 2018 19:52:52 +1100
Subject: [Tutor] Python_About "Guess my number" program
In-Reply-To: <CAC_wLw3beMSf1AjOWvd5XC4Wu0o-oCop2Pa0ictK=N8ArDO4hA@mail.gmail.com>
References: <CAC_wLw3beMSf1AjOWvd5XC4Wu0o-oCop2Pa0ictK=N8ArDO4hA@mail.gmail.com>
Message-ID: <20181212085252.GK13061@ando.pearwood.info>

Hi Dat, and welcome!

On Wed, Dec 12, 2018 at 11:26:30AM +0600, Hoang Quoc Dat wrote:
[...]
> After a while, I went on and search on the internet and
> find out 1 similar program but the coding logic is like: The computer keeps
> guessing the middle number of the range and narrow down the range by
> focusing on only the upper/lower part of the range after receving our
> answer.

Yes, that is the classic algorithm. It is a variant of binary search.


> I somehow have come to believe that the Python language and our logical
> thinking can do better than that option of choosing the middle number all
> the time but I still cannot find out the proper answer (probably because I
> am very much new to this). Could someone provide answer for this program?

If the only information you have is whether the secret number is 
"higher" or "lower", then no, there is no better algorithm.

If you had more information, like "red hot, hot, warmer, warm, cool, 
cold, colder, ice cold" to indicate how close the guess is, then you 
might be able to do better, if you had some objective rule for deciding 
how close "warm" etc was.


-- 
Steve

From __peter__ at web.de  Wed Dec 12 05:04:01 2018
From: __peter__ at web.de (Peter Otten)
Date: Wed, 12 Dec 2018 11:04:01 +0100
Subject: [Tutor] Increase performance of the script
References: <mailman.8.1544461202.31434.tutor@python.org>
 <CAG3LsKFZ3+DkYT0CGkux8QwOuXfu0FYdZBf=oEry2PSJPfvo=Q@mail.gmail.com>
 <20181211222106.GG13061@ando.pearwood.info>
Message-ID: <puqmae$3be$1@blaine.gmane.org>

Steven D'Aprano wrote:

> [...]
>> In python 2.6 print statement work as print "Solution"
>> however after import collection I have to use print with
>> print("Solution") is this a known issue ?
> 
> As Peter says, you must have run
> 
> from __future__ import print_function
> 
> to see this behaviour. This has nothing to do with import collection.
> You can debug that for yourself by exiting the interactive interpreter,
> starting it up again, and trying to print before and after importing
> collection.

To be fair to Asad -- I sneaked in the __future__ import into my sample 
code. I did it to be able to write Python 3 code that would still run in his 
2.6 interpreter. 

In hindsight that was not a good idea as it can confuse someone who has 
never seen it, and the OP has yet to learn other more important things.

 



From smravikumardonbosco at gmail.com  Wed Dec 12 07:57:09 2018
From: smravikumardonbosco at gmail.com (Ravi Kumar)
Date: Wed, 12 Dec 2018 06:57:09 -0600
Subject: [Tutor] Python script errors
Message-ID: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>

Hi,

I have developed a python script to get api calls for meraki
clientlogevents Thanks for all the help previously  I am facing few errors
such as

Json_string=r.json()

raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
>>>

'{0:20}   {1:30}   {2:16}   {3:18}   {4:10}     {5:11} '.format(
deviceserial, type, macaddress,occurredAt, details))
TypeError: unsupported format string passed to NoneType.__format__

I know this occurs when the api response is nulls but how do I handle this?

Thanks

From alan.gauld at yahoo.co.uk  Wed Dec 12 11:37:18 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 12 Dec 2018 16:37:18 +0000
Subject: [Tutor] Python script errors
In-Reply-To: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>
References: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>
Message-ID: <purdbr$nke$1@blaine.gmane.org>

On 12/12/2018 12:57, Ravi Kumar wrote:

> '{0:20}   {1:30}   {2:16}   {3:18}   {4:10}     {5:11} '.format(
> deviceserial, type, macaddress,occurredAt, details))
> TypeError: unsupported format string passed to NoneType.__format__

You have 6 format holders in your string but you only
pass 5 values. Also you seem to have a spurious closing
parens at the end of the call to format (after details).

But without seeing the full code or error message
it is hard to know what is going on.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From __peter__ at web.de  Wed Dec 12 12:12:51 2018
From: __peter__ at web.de (Peter Otten)
Date: Wed, 12 Dec 2018 18:12:51 +0100
Subject: [Tutor] Python script errors
References: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>
Message-ID: <purfeh$65f$1@blaine.gmane.org>

Ravi Kumar wrote:

> Hi,
> 
> I have developed a python script to get api calls for meraki
> clientlogevents Thanks for all the help previously  I am facing few errors
> such as
> 
> Json_string=r.json()
> 
> raw_decode
>     raise JSONDecodeError("Expecting value", s, err.value) from None
> json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
>>>>
> 
> '{0:20}   {1:30}   {2:16}   {3:18}   {4:10}     {5:11} '.format(
> deviceserial, type, macaddress,occurredAt, details))
> TypeError: unsupported format string passed to NoneType.__format__
> 
> I know this occurs when the api response is nulls but how do I handle
> this?

Say x can be a string or None. 

>>> for x in ["a string", None]:
...     "{:20}".format(x)
... 
'a string            '
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: unsupported format string passed to NoneType.__format__

The string can be padded while None does not support that. The best solution 
is usually to replace None with a string that cannot occur normally, and 
then to use that instead of None. Example:

>>> for x in ["a string", None]:
...     "{:20}".format("#NONE" if x is None else x)
... 
'a string            '
'#NONE               '

Slightly more convenient: you can force string conversion before padding:

>>> for x in ["a string", None]:
...     "{!s:20}".format(x)
... 
'a string            '
'None                '





From steve at pearwood.info  Wed Dec 12 15:28:43 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 13 Dec 2018 07:28:43 +1100
Subject: [Tutor] Python script errors
In-Reply-To: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>
References: <CANUFOzk3Ln1NmLwXvYV53DrKdPwMbmeh4=p_Pr_F_NVqO4vkig@mail.gmail.com>
Message-ID: <20181212202843.GO13061@ando.pearwood.info>

On Wed, Dec 12, 2018 at 06:57:09AM -0600, Ravi Kumar wrote:

> I know this occurs when the api response is nulls but how do I handle this?

if response is None:
    handle None case
else:
    handle non-None case



-- 
Steve

From avigross at verizon.net  Thu Dec 13 12:36:27 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 13 Dec 2018 12:36:27 -0500
Subject: [Tutor] Long Lines techniques
Message-ID: <009101d4930a$60d1b860$22752920$@verizon.net>

Simple question:

 

When lines get long, what points does splitting them make sense and what
methods are preferred?

 

Details.

 

I am used to many languages where you can continue a statement across
multiple lines. What they share in common is the fact they do not use
indenting for the use Python makes of it. They often use something like
"{.}" to combine multiple statements into a single body and carriage returns
are generally ignored as multiple lines are parsed to search for thee end of
the compound statement at the "}" . In many languages, parentheses and
square brackets and even angle brackets can serve a similar function. Some
languages that insist on ending a statement with ";" are even looser. Many
treat a trailing comma or plus sign (and other symbols) as a hint to
continue reading another line.

 

So I would like some feedback on ways to deal with longer statements in
Python.

 

Yes, I am aware of ways to break up something long by breaking in into
multiple statements. It feels silly to do this:

 

A = "long string"

B = "another string"

.

Z = "last string"

 

When you meant to write one long string in the first place or connect a
bunch of them with adjacency or a "+"

 

By breaking them into snippets like above, and using short variable names,
you can combine them like this:

 

Combined = A + B + . + Z

 

Or even worse:

 

Combined = ""

Combined += A

Combined += B

.

Combined += Z

 

The above was ONE example of many where I bend my code in ways to make the
lines reasonably short so reading and editing are easy.

 

There are places you can break lines as in a comprehension such as this set
comprehension:

 

    letter_set = { letter

                   for word in (left_list + right_list)

                   for letter in word }

 

The above is an example where I know I can break because the {} is holding
it together. I know I can break at each "for" or "if" but can I break at
random places? Here is a bizarre example that worked:

 

>>> [ x *

      y

      for x in

      range(1,5,2)

      for y

      in range(2,

                       6

                       ,2)

      if x + y <

      9

      ]

[2, 4, 6, 12]

 

So clearly when you use [.] it ignores indentation until it gets to the end.
I see that parentheses sometimes offer a similar protection as in the new
print function:

 

>>> print(1,

                  2

                  ,

                  3)

1 2 3

 

But python has a few features like allowing a comma at the end of a tuple
that get in the way:

 

>>> x,y = 1,

Traceback (most recent call last):

  File "<pyshell#15>", line 1, in <module>

    x,y = 1,

ValueError: not enough values to unpack (expected 2, got 1)

 

DUH! I was going to put the rest on the next line!

 

So I added parentheses and that does the trick here.

 

>>> x,y = (1,

                   2

                   )

 

Similarly, it is a syntax error if I simply end with something like a +

 

>>> x = 1 +

SyntaxError: invalid syntax

>>> x = (1 +

                2)

>>> x

3

 

I AM NOT SAYING THIS IS HORRIBLE, just asking what pythonic ways are best to
use to get around things.

 

One that seems not to be favored is a backslash at the end of a line:

 

>>> x = 1 + \

                2 + \

                3

>>> x

6

 

Although my examples use short numbers, remember I might be using this with
long lines and not want to mess up the overall indentation. Strings have a
similar issue:

 

>>> x = "hello

SyntaxError: EOL while scanning string literal

>>> x = """hello

world"""

>>> x

'hello\nworld'

>>> x = "hello \

                world!"

>>> x

'hello \tworld!''

 

As shown, the triple quotes sort of work but introduce and keep carriage
returns and formatting inside. The backslash at the end suppressed the
former but keeps the latter. Yes, there are routines you can pass the string
through afterward to remove leading whitespace or carriage returns.

 

I will stop here with saying that unlike many languages, parentheses must be
used with care in python as they may create a tuple or even generator
expression. Curly braces may make a set or dictionary. Options for splitting
the code without messing up indenting cues would be nice to understand.

 

 

 

 

 


From sjl1357 at comcast.net  Thu Dec 13 11:56:23 2018
From: sjl1357 at comcast.net (Sammy Lee)
Date: Thu, 13 Dec 2018 11:56:23 -0500 (EST)
Subject: [Tutor] Import CSV
Message-ID: <823755185.32856.1544720183281@connect.xfinity.com>

I need help on how to open a webpage and save the HTML to a given file path, given a URL.

From sjl1357 at comcast.net  Thu Dec 13 11:56:59 2018
From: sjl1357 at comcast.net (Sammy Lee)
Date: Thu, 13 Dec 2018 11:56:59 -0500 (EST)
Subject: [Tutor] Import CSV
In-Reply-To: <823755185.32856.1544720183281@connect.xfinity.com>
References: <823755185.32856.1544720183281@connect.xfinity.com>
Message-ID: <699719575.32866.1544720219269@connect.xfinity.com>

def save_url_to_file(url, savefile):

This is the starting code.


> On December 13, 2018 at 11:56 AM Sammy Lee <sjl1357 at comcast.net> wrote:
> 
> 
>     I need help on how to open a webpage and save the HTML to a given file path, given a URL.
> 
 

From sjl1357 at comcast.net  Thu Dec 13 12:14:41 2018
From: sjl1357 at comcast.net (Sammy Lee)
Date: Thu, 13 Dec 2018 12:14:41 -0500 (EST)
Subject: [Tutor] Extract URL
Message-ID: <1043141371.33167.1544721281685@connect.xfinity.com>

I need help on the problem stated below.


Given a URL, open the webpage and return the first anchor link url (a href).


def extract_url_link(url):

From sjl1357 at comcast.net  Thu Dec 13 12:16:44 2018
From: sjl1357 at comcast.net (Sammy Lee)
Date: Thu, 13 Dec 2018 12:16:44 -0500 (EST)
Subject: [Tutor] Open webpage and save CSV
Message-ID: <104738747.33207.1544721404935@connect.xfinity.com>

I need help on the problem below,


Given a URL, open the webpage and save the CSV to a given file path.


def save_url_to_csv_file(url, savefile):

From sjl1357 at comcast.net  Thu Dec 13 12:21:36 2018
From: sjl1357 at comcast.net (Sammy Lee)
Date: Thu, 13 Dec 2018 12:21:36 -0500 (EST)
Subject: [Tutor] Python function
Message-ID: <582145781.33310.1544721697131@connect.xfinity.com>

How do I create a python function that opens a CSV file and determines how many columns
of data are in the file? The CSV files have been randomly generated from https://www.mockaroo.com/

def csv_column_count(openfile):

From bgailer at gmail.com  Thu Dec 13 14:08:02 2018
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 13 Dec 2018 14:08:02 -0500
Subject: [Tutor] Open webpage and save CSV
In-Reply-To: <104738747.33207.1544721404935@connect.xfinity.com>
References: <104738747.33207.1544721404935@connect.xfinity.com>
Message-ID: <CAP1rxO4xYbof=mbTrTMjR8ar1unOPxUM7tR4wgNF0YEvEMoN=A@mail.gmail.com>

On Dec 13, 2018 1:51 PM, "Sammy Lee" <sjl1357 at comcast.net> wrote:
>
> I need help on the problem below,

Could you be more specific? What kind of help do you need? Have you made
any attempt to write a program?

We are happy to help but we're not going to do all the work for you. So
tell us what you do know about the various aspects of this problem.

My personal guess is that this is a homework assignment. If that's the case
the class should have given you some of the information (ideally all the
information) you need to solve the problem.

Also please tell us what operating system you're using and what version of
python.

You will need the services of the urllib. request module to get the
contents of a web page.

So start with that.

> Given a URL, open the webpage and save the CSV to a given file path.
>
>
> def save_url_to_csv_file(url, savefile):

Bob Gailer

From bgailer at gmail.com  Thu Dec 13 14:33:12 2018
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 13 Dec 2018 14:33:12 -0500
Subject: [Tutor] Long Lines techniques
In-Reply-To: <009101d4930a$60d1b860$22752920$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
Message-ID: <CAP1rxO68Q8nq=ydHXNeG=+Vjrx_K-JBWygnOYzZhbhyOHMTXbg@mail.gmail.com>

On Dec 13, 2018 1:51 PM, "Avi Gross" <avigross at verizon.net> wrote:
>
> Simple question:

Avi: when I see an email from you I tend to ignore it because it always
seems to lead to something that is long, time consuming and complex. Would
you consider finding ways to make your questions or comments a lot briefer?

I will be more inclined to read them if they are briefer.

You have correctly determined the conditions that will lead to continuation
lines without backslash. I think we tend to use whatever is convenient.

In your example of a list comprehension over multiple lines there is no
indentation. There is just a lot of white space. You might look at it this
way: the compiler sees a left bracket with no corresponding right bracket
on that line. So it assumes that there's more to the statement on the next
line, it ignores the newline and just continues. Indentation is only
significant if it starts at the beginning of a statement.

Hope this helps

Bob gailer

From bgailer at gmail.com  Thu Dec 13 15:16:13 2018
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 13 Dec 2018 15:16:13 -0500
Subject: [Tutor] Python function
In-Reply-To: <582145781.33310.1544721697131@connect.xfinity.com>
References: <582145781.33310.1544721697131@connect.xfinity.com>
Message-ID: <CAP1rxO50hUnfAruCp=f-kpD89BLwHoiMyKHku0dw+evdfvptbA@mail.gmail.com>

On Dec 13, 2018 1:55 PM, "Sammy Lee" <sjl1357 at comcast.net> wrote:
>
> How do I create a python function that opens a CSV file and determines
how many columns
> of data are in the file? The CSV files have been randomly generated from
https://www.mockaroo.com/
>
> def csv_column_count(openfile):

Same comments as I made in response to your other question. What part of
this do you need help with? do need to know how to read a file? do you need
to understand what a CSV file is? do you need to know how to parse a
character string?

The more specific you are the easier it is for us to help you.

Have you ever written a Python program? What has your course taught you so
far? Do you know how to write pseudocode?

From bgailer at gmail.com  Thu Dec 13 15:43:59 2018
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 13 Dec 2018 15:43:59 -0500
Subject: [Tutor] Extract URL
In-Reply-To: <1043141371.33167.1544721281685@connect.xfinity.com>
References: <1043141371.33167.1544721281685@connect.xfinity.com>
Message-ID: <CAP1rxO708oWv54VJVztw-7cpbV-Z=CdEg4Sr8WBzHo678xp9dA@mail.gmail.com>

On Dec 13, 2018 2:01 PM, "Sammy Lee" <sjl1357 at comcast.net> wrote:
>
> I need help on the problem stated below.
>
>
> Given a URL, open the webpage and return the first anchor link url (a
href).
>
>
> def extract_url_link(url):

Same comments as my other two emails.

From breamoreboy at gmail.com  Thu Dec 13 16:26:29 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Thu, 13 Dec 2018 21:26:29 +0000
Subject: [Tutor] Open webpage and save CSV
In-Reply-To: <104738747.33207.1544721404935@connect.xfinity.com>
References: <104738747.33207.1544721404935@connect.xfinity.com>
Message-ID: <puuim2$bq7$1@blaine.gmane.org>

On 13/12/2018 17:16, Sammy Lee wrote:
> I need help on the problem below,
> 
> Given a URL, open the webpage and save the CSV to a given file path.
> 
> def save_url_to_csv_file(url, savefile):


Run up your favourite text editor and type up some code.  Try to run it. 
  Tell us what problems you get.  Failing that send a large cheque to 
the PSF and no doubt somebody will help you out.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From breamoreboy at gmail.com  Thu Dec 13 16:27:43 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Thu, 13 Dec 2018 21:27:43 +0000
Subject: [Tutor] Long Lines techniques
In-Reply-To: <CAP1rxO68Q8nq=ydHXNeG=+Vjrx_K-JBWygnOYzZhbhyOHMTXbg@mail.gmail.com>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <CAP1rxO68Q8nq=ydHXNeG=+Vjrx_K-JBWygnOYzZhbhyOHMTXbg@mail.gmail.com>
Message-ID: <puuioc$bq7$2@blaine.gmane.org>

On 13/12/2018 19:33, Bob Gailer wrote:
> On Dec 13, 2018 1:51 PM, "Avi Gross" <avigross at verizon.net> wrote:
>>
>> Simple question:
> 
> Avi: when I see an email from you I tend to ignore it because it always
> seems to lead to something that is long, time consuming and complex. Would
> you consider finding ways to make your questions or comments a lot briefer?
> 
> I will be more inclined to read them if they are briefer.
> 
> You have correctly determined the conditions that will lead to continuation
> lines without backslash. I think we tend to use whatever is convenient.
> 
> In your example of a list comprehension over multiple lines there is no
> indentation. There is just a lot of white space. You might look at it this
> way: the compiler sees a left bracket with no corresponding right bracket
> on that line. So it assumes that there's more to the statement on the next
> line, it ignores the newline and just continues. Indentation is only
> significant if it starts at the beginning of a statement.
> 
> Hope this helps
> 
> Bob gailer
> 

Big +1 from me :-)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From adameyring at gmail.com  Thu Dec 13 14:09:55 2018
From: adameyring at gmail.com (Adam Eyring)
Date: Thu, 13 Dec 2018 14:09:55 -0500
Subject: [Tutor] Import CSV
In-Reply-To: <823755185.32856.1544720183281@connect.xfinity.com>
References: <823755185.32856.1544720183281@connect.xfinity.com>
Message-ID: <CAPStRW-JEsLHvheCs-xH7R5Uckhvzt1D34fsZzAoS8WsqCCaMg@mail.gmail.com>

Take a look at the book Automating the Boring Stuff with Python (free PDF):
https://automatetheboringstuff.com/

There's a couple of chapters in there for downloading CSV files and
extracting the data.

On Thu, Dec 13, 2018 at 1:57 PM Sammy Lee <sjl1357 at comcast.net> wrote:

> I need help on how to open a webpage and save the HTML to a given file
> path, given a URL.
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>

From steve at pearwood.info  Thu Dec 13 19:27:14 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 14 Dec 2018 11:27:14 +1100
Subject: [Tutor] Long Lines techniques
In-Reply-To: <009101d4930a$60d1b860$22752920$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
Message-ID: <20181214002713.GU13061@ando.pearwood.info>

On Thu, Dec 13, 2018 at 12:36:27PM -0500, Avi Gross wrote:

> Simple question:
> 
> When lines get long, what points does splitting them make sense and what
> methods are preferred?

Good question!

First, some background:

Long lines are a potential code smell: a possible sign of excessively 
terse code. A long line may be a sign that you're doing too much in one 
line.

https://martinfowler.com/bliki/CodeSmell.html
http://wiki.c2.com/?CodeSmell
https://blog.codinghorror.com/code-smells/

Related: 
https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/

Note that merely splitting a logical line over two or more physical 
lines may still be a code-smell. Sure, your eyes don't get as tired 
reading fifteen lines of 50 characters each, compared to a single 750 
character line, but there's just as much processing going on in what is 
essentially a single operation.

Long lines are harder to read: your eyes have to scan across a long 
line, and beyond 60 or 70 characters, it becomes physically more 
difficult to scan across the line, and the error rate increases. 
[Citation required.]

But short lines don't include enough information, so the traditional 
compromise is 80 characters, the character width of the old-school 
green-screen terminals. The Python standard library uses 79 characters. 
(The odd number is to allow for scripts which count the newline at the 
end of the line as one of the 80.)

https://www.python.org/dev/peps/pep-0008/


Okay, so we have a style-guide that sets a maximum line length, whether 
it is 72 or 79 or 90 or 100 characters. What do you do when a line 
exceeds that length?

The only firm rule is that you must treat each case on its own merits. 
There is no one right or wrong answer. Every long line of code is 
different, and the solution will depend on the line itself. There is no 
getting away from human judgement.


(1) Long names. Do you really need to call the variable 
"number_of_characters" when "numchars" or even "n" would do?

The same applies to long function names: "get_data_from_database" is 
probably redundant, "get_data" will probably do.

Especially watch out for long dotted names that you use over and over 
again. Unlike static languages like Java, each dot represents a runtime 
lookup. Long names like:

    package.subpackage.module.object.method

requires four lookups. Look for oportunities to make an alias for a long 
name and avoid long chains of dots:

    for item in sequence:
        do_something_with(package.subpackage.module.object.method(arg, item))

can be refactored to:

    method = package.subpackage.module.object.method
    for item in sequence:
        do_something_with(method(arg, item))

and is both easier to read and more efficient. A double win!


(2) Temporary constants: sometimes it is good enough to just introduce a 
simple named constant used once. The cognitive load is low if it is 
defined immediately before it is used. Instead of the long line:

    raise ValueError("expected a list, string, dict or None, but instead got '%s'" % type(value).__name__)

I write:

    errmsg = "expected a list, string, dict or None, but instead got '%s'"
    raise ValueError(errmsg % type(value).__name__)


(3) Code refactoring. Maybe that long line is sign that you need to add 
a method or function? Especially if you are using that line, or similar, 
in multiple places. But refactoring is justified even if you use the 
line *once* if it is complicated enough.

Likewise, sometimes it is helpful to factor out separate sub-expressions 
onto their own lines, using their own variables, rather than doing 
everything in a single, complicated, expression.

Psychologists, educators and linguists call this "chunking", and it is 
often very helpful for simplifying complicated ideas, sentences and 
expressions.

The lack of chunks is why long Perl one-liners are so inpenetrable.


(4) Split the long logical line over multiple physical lines. This does 
nothing to reduce the inherent complexity of the line, but if that's 
fairly low to start with, it is often helpful.

Python gives us two ways to split a logical line over multiple physical 
lines: a backslash at the end of the line, and brackets of any sort.

The preferred way is to use round brackets for grouping:

    result = (some very long expression
              which can be split over
              many lines)

This is especially useful with function calls:

    result = function(first_argument, second_argument,
                      third_argument, fourth_argument)

If you are building a list or dict literal, there is no need for the 
parentheses, as square and curly brackets have the same effect. That's 
especially useful with two-dimensional nested lists:

    data = [[row, one, with, many, items],
            [row, two, with, many, items],
            [row, three, with, many, items]]

For long strings, I like to use *implicit string concatentation*. String 
literals which are separated by nothing except whitespace are 
concatenated at compile-time. So I can write a long string like this:

    long_string = ("this is a very long string which doesn't"
                   " fit on a single line but isn't appropriate"
                   " for a triple-quoted string")

Notice that I split the string at word breaks, and move the space to the 
beginning of the physical line rather than the end. I find that I'm less 
likely to forget the space if I put it at the start of the line rather 
than the end.

Not preferred, but allowed for backwards compatibility and still very 
occasionally useful, is to end the line with a bare backslash. I find it 
helpful in conjunction with triple quoted strings:

    text = """\
    body of the string
    is aligned
    including the first line
    """

but otherwise the backslash is problematic and error-prone. It must be 
*immediately* followed by a newline, if you accidentally add a space 
after the backslash it won't work.


And finally:

(5) Its just a style guide, not a law of physics. As Douglas Bader once 
said, "Rules are for the guidance of the wise and the obedience of 
fools." See also Raymond Hettinger's talk "Beyond PEP 8":

https://twitter.com/raymondh/status/589849947408703488

https://medium.com/@drb/pep-8-beautiful-code-and-the-tyranny-of-guidelines-f96499f5ac17

Better to go two or three characters beyond the maximum length than to 
make the code ugly.



[...]
> There are places you can break lines as in a comprehension such as this set
> comprehension:
> 
>     letter_set = { letter
>                    for word in (left_list + right_list)
>                    for letter in word }
> 
> The above is an example where I know I can break because the {} is holding
> it together. I know I can break at each "for" or "if" but can I break at
> random places?

Not quite random, you can't break in the middle of a word, but you 
can break between words.



[...]
> I will stop here with saying that unlike many languages, parentheses must be
> used with care in python as they may create a tuple or even generator
> expression.

But not by accident. You can't create a generator expression by accident 
by wrapping an arbitrary expression in round brackets, or turn a 
expression into a tuple. 

Remember, it isn't the parentheses which make tuples, its the commas. 
Except for the empty tuple special case, (), the parens are ALWAYS 
just there to either group the tuple so as to avoid ambiguity, or to 
visually emphasize that it is a tuple even if the interpreter doesn't 
need the hint.



-- 
Steve

From alan.gauld at yahoo.co.uk  Thu Dec 13 19:41:45 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 14 Dec 2018 00:41:45 +0000
Subject: [Tutor] Import CSV
In-Reply-To: <823755185.32856.1544720183281@connect.xfinity.com>
References: <823755185.32856.1544720183281@connect.xfinity.com>
Message-ID: <puuu46$1e9$1@blaine.gmane.org>

On 13/12/2018 16:56, Sammy Lee wrote:
> I need help on how to open a webpage and save the HTML to a given file path, given a URL.

You can use the standard library urllib module to fetch the
file but most folks find the requests module easier to use.
requests is a 3rd party module you will need to install.

You can read about how to use urllib in the Web Clients
topic of my tutorial(see below).

Saving the HTML is just the same as saving any other string
to a file. Do you know how to do that?
If not the file handling topic of my tutor covers it.

Your subject mentions CSV but your text only says HTML.
If you do have a CSV file then the Python csv module
will deal with parsing the CSV data for you. The module
documentation is quite helpful.

If you need to pare HTML to extract the data then the
standard http module can help there. See the Web Clients
topic in my tutorial for more on that too.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Thu Dec 13 19:47:03 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 14 Dec 2018 00:47:03 +0000
Subject: [Tutor] Python function
In-Reply-To: <582145781.33310.1544721697131@connect.xfinity.com>
References: <582145781.33310.1544721697131@connect.xfinity.com>
Message-ID: <puuue4$1e9$3@blaine.gmane.org>

On 13/12/2018 17:21, Sammy Lee wrote:
> How do I create a python function that opens a CSV file and determines how many columns
> of data are in the file? The CSV files have been randomly generated from https://www.mockaroo.com/
> 
> def csv_column_count(openfile):

You will find a bunch of stuff on creating functions in the Functions
and Modules topic of my tutorial.(see below)

You will find a bunch of stuff on handling files in the Handling Files
topic of my tutorial.(see below)

You will find a lot of help on using CSV files in the csv module
documentation in the standard library. (Or you could buy my book
"Python Projects" which has a section on the topic.)



-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Thu Dec 13 19:44:32 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 14 Dec 2018 00:44:32 +0000
Subject: [Tutor] Extract URL
In-Reply-To: <1043141371.33167.1544721281685@connect.xfinity.com>
References: <1043141371.33167.1544721281685@connect.xfinity.com>
Message-ID: <puuu9d$1e9$2@blaine.gmane.org>

On 13/12/2018 17:14, Sammy Lee wrote:
> I need help on the problem stated below.
> 
> 
> Given a URL, open the webpage and return the first anchor link url (a href).

You will find a very similar example in the Web Client
topic of my tutorial, see below...

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From alan.gauld at yahoo.co.uk  Thu Dec 13 20:03:55 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 14 Dec 2018 01:03:55 +0000
Subject: [Tutor] Long Lines techniques
In-Reply-To: <009101d4930a$60d1b860$22752920$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
Message-ID: <puuvdo$1b4$1@blaine.gmane.org>

On 13/12/2018 17:36, Avi Gross wrote:

> When lines get long, what points does splitting them make sense and what
> methods are preferred?

Its down to personal preference and convenience
plus a smidge of idiom.

> Yes, I am aware of ways to break up something long by breaking in into
> multiple statements. It feels silly to do this:
> 
> A = "long string"
> B = "another string"
> .
> Z = "last string"

I'd probably suggest

stgs = ''.join([
"long string",
"another string",
...
"last string"
])

> There are places you can break lines as in a comprehension such as this set
> comprehension:
>     letter_set = { letter
>                    for word in (left_list + right_list)
>                    for letter in word }

You can break anythiong anywhere (pretty much) inside a set of
parens/brackets/braces.




> So clearly when you use [.] it ignores indentation until it gets to the end.

Indentation only matters for blocks, inside parens etc it
doesn't signify.


> But python has a few features like allowing a comma at the end of a tuple
> that get in the way:
> 
>>>> x,y = 1,

There's no parens to hold the subsequent line.
You have expressed a valid single value tuple.

> One that seems not to be favored is a backslash at the end of a line:
>>>> x = 1 + \
>                 2 + \
>                 3

Its easy to get wrong and rarely needed but if its the
only way to get neat code its acceptable.

> I will stop here with saying that unlike many languages, parentheses must be
> used with care in python as they may create a tuple 

Not a tuple, they arecreated with commas.
But parens are used to disambiguate tuples
 - for example in argument lists. But its the comma that makes it a tiuple:


t = 5,6   # 5,6 is a tuple
a,b = t   # a->5, b->6


> or even generator expression. 

These are usually just parens in a function. The generator
expression is the bit inside (not including) the parens.

> Curly braces may make a set or dictionary. Options for splitting
> the code without messing up indenting cues would be nice to understand.

Braces and square brackets do create structures,
but you can break lines within those. Indeed its very
common to define large literal structures across
multiple lines (possibly including descriptive comments):

mydict = {
"key" : 42,  # the first key
"two" : 66   # another key
...
}


HTH

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avigross at verizon.net  Thu Dec 13 19:45:31 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 13 Dec 2018 19:45:31 -0500
Subject: [Tutor] Long Lines techniques
In-Reply-To: <CAP1rxO68Q8nq=ydHXNeG=+Vjrx_K-JBWygnOYzZhbhyOHMTXbg@mail.gmail.com>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <CAP1rxO68Q8nq=ydHXNeG=+Vjrx_K-JBWygnOYzZhbhyOHMTXbg@mail.gmail.com>
Message-ID: <009501d49346$513b46a0$f3b1d3e0$@verizon.net>

Just for you Bob, I will make this short.

 

Here is the most bizarre solution as you might agree. 

 

Edit a file of python code. Use lines as long as it takes. Make sure it works.

 

Edit the file again and split lines in a way that does not mess up indentation. Meaning, use a normal text editor, not one that understands python. 

 

At the end of each line you cut, place a marker that makes sense. For example ?!EOL? that may be easier to read or print (or harder) but not to edit.

 

Save all this to another plain file. It is not valid python.

 

Write a real python file that contains a function with a name like outclude(filename) that opens a file, reads in the entire file, replaces every place with the marker at the end of the line with NOTHING, effectively splicing it to the next line. It then returns a string containing your entire program with proper indentation.

 

Finally:

 

eval(outclude("filename"))

 

SKIPPING the other 98% this post could have had. Just an example:

 

DISCLAIMER: The above was posted as a sick form of humor, only. Any resemblance to reality is unintentional. But note, some languages do have a phase of scanning that would allow tricks like this in a pre-processor.

 

From: Bob Gailer <bgailer at gmail.com> 
Sent: Thursday, December 13, 2018 2:33 PM
To: Avi Gross <avigross at verizon.net>
Cc: tutor at python.org
Subject: Re: [Tutor] Long Lines techniques

 

On Dec 13, 2018 1:51 PM, "Avi Gross" <avigross at verizon.net <mailto:avigross at verizon.net> > wrote:
>
> Simple question:

Avi: when I see an email from you I tend to ignore it because it always seems to lead to something that is long, time consuming and complex. Would you consider finding ways to make your questions or comments a lot briefer?

I will be more inclined to read them if they are briefer.

You have correctly determined the conditions that will lead to continuation lines without backslash. I think we tend to use whatever is convenient.

In your example of a list comprehension over multiple lines there is no indentation. There is just a lot of white space. You might look at it this way: the compiler sees a left bracket with no corresponding right bracket on that line. So it assumes that there's more to the statement on the next line, it ignores the newline and just continues. Indentation is only significant if it starts at the beginning of a statement.

Hope this helps 

Bob gailer


From steve at pearwood.info  Fri Dec 14 00:17:12 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 14 Dec 2018 16:17:12 +1100
Subject: [Tutor] Long Lines techniques
In-Reply-To: <puuvdo$1b4$1@blaine.gmane.org>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <puuvdo$1b4$1@blaine.gmane.org>
Message-ID: <20181214051712.GV13061@ando.pearwood.info>

On Fri, Dec 14, 2018 at 01:03:55AM +0000, Alan Gauld via Tutor wrote:

> I'd probably suggest
> 
> stgs = ''.join([
> "long string",
> "another string",
> ...
> "last string"
> ])

That's certainly better than using the + operator, as that will be quite 
inefficient for large numbers of strings. But it still does unnecessary 
work at runtime that could be done at compile time.

Hence my preferred solution is implicit string concatenation:

stgs = ("long string"
        "another string"
        "last string")

Note the lack of commas.


[...]
> > or even generator expression. 
> 
> These are usually just parens in a function. The generator
> expression is the bit inside (not including) the parens.

Hmmm, well, yes, no, maybe. The definition of a generator comprehension 
*requires* it to be surrounded by parentheses. You cannot write this:

gen = x + 1 for x in items

you must use brackets of some form:

- [] for a list comprehension;
- () for a generator comprehension;
- {} for a dict or set comprehension;

But for the generator case, a short-cut is allowed. If the expression is 
already surrounded by parens, as in a one-argument function call, you 
can forego the inner brackets. So instead of:

    flag = any((condition(x) for x in items))

we can just write:

    flag = any(condition(x) for x in items)



-- 
Steve

From steve at pearwood.info  Fri Dec 14 00:22:15 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 14 Dec 2018 16:22:15 +1100
Subject: [Tutor] Long Lines techniques
In-Reply-To: <00ec01d49362$9a125dc0$ce371940$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
 <00ec01d49362$9a125dc0$ce371940$@verizon.net>
Message-ID: <20181214052214.GW13061@ando.pearwood.info>

On Thu, Dec 13, 2018 at 11:07:59PM -0500, Avi Gross wrote:

[...]
> There are cases where it may make sense to have a long like connected by AND
> or OR given how python does short-circuiting while returning the last thing
> or two it touched instead of an actual True/False. For example, you may want
> to take the first available queue that is not empty with something like
> this:
> 
> Using = A or B or C or ... or Z
> Handling = Using.pop()
> 
> Sure, that could be rewritten into multiple lines. 

using = (A or B
         or C or D 
         or E or F)


[...]
> I recently wrote some code and ran into error messages on lines I was trying
> to keep short:
> 
> A = 'text"
> A += "more text"
> A += object
> A+= ...

Without knowing the error message, its impossible to say what the 
problem is. My guess is that the object on line three wasn't a string. 
In which case, the obvious fix is:

A += str(object)



-- 
Steve

From steve at pearwood.info  Fri Dec 14 00:35:43 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 14 Dec 2018 16:35:43 +1100
Subject: [Tutor] Obfuscated Python [was Long Lines techniques]
In-Reply-To: <00ec01d49362$9a125dc0$ce371940$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
 <00ec01d49362$9a125dc0$ce371940$@verizon.net>
Message-ID: <20181214053543.GX13061@ando.pearwood.info>

On Thu, Dec 13, 2018 at 11:07:59PM -0500, Avi Gross wrote:

> Python may claim to be straightforward but I can easily see ways
> to fool people in python too with dunder methods or function closures or
> decorators or ...

Dunder methods shouldn't fool anyone. Each dunder method has a 
straightforward use, and most of the time you can guess that they do. 
__add__ implements the + operator, etc.

Closures and decorators can be abused, as can anything. One can write 
obfuscated code in any language. Here's some amusing obfuscated Python:

https://www.reddit.com/r/Python/comments/i1qgp/reduce_and_lambda_two_great_tastes_that_taste/

https://mail.python.org/pipermail/python-list/2013-October/658470.html

Anyone else want to share some obfuscated Python code?



-- 
Steve

From avigross at verizon.net  Thu Dec 13 23:07:59 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 13 Dec 2018 23:07:59 -0500
Subject: [Tutor] Long Lines techniques
In-Reply-To: <20181214002713.GU13061@ando.pearwood.info>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
Message-ID: <00ec01d49362$9a125dc0$ce371940$@verizon.net>

<[{SYNOPSIS: Many good answers. I am satisfied and we can move on.}]>

Steven,

I appreciate the many useful suggestions.

Many of them are what I already do. Some are in tension with other
considerations. Yes, it can be shorter and more efficient to not keep saying
module.this.that.something versus something like:

>From module.this.that import something as myname.

Of course you do that with care as you want to be careful about pulling too
many things into collisions in one namespace. Longer more descriptive names
are encouraged.

Based on reading quite a bit of code lately, I do see how common it is to
try to shorten names while not polluting the namespace as in the nearly
universal:

Import numpy as np, pandas as pd

The places I like to wrap lines tend to be, in reality, the places python
tolerates it. If you use a function that lets you set many options, it is
nice to see the options one per line. Since the entire argument list is in
parentheses, that works. Ditto for creating lists, sets and dictionaries
with MANY items at once. 

There are cases where it may make sense to have a long like connected by AND
or OR given how python does short-circuiting while returning the last thing
or two it touched instead of an actual True/False. For example, you may want
to take the first available queue that is not empty with something like
this:

Using = A or B or C or ... or Z
Handling = Using.pop()

Sure, that could be rewritten into multiple lines. 

I won't get sucked into a PERL discussion except to say that some people
love to write somethings so obscure they won't recognize it even a daylater.
PERL makes that very easy. I have done that myself a few times as I was an
early user. Python may claim to be straightforward but I can easily see ways
to fool people in python too with dunder methods or function closures or
decorators or ...

All in all, I think my question has been answered. I will add one more
concept.

I recently wrote some code and ran into error messages on lines I was trying
to keep short:

A = 'text"
A += "more text"
A += object
A+= ...

At one point, I decided to use a formatted string instead:

A = f"...{...}...{...}..."

Between curly braces I could insert variables holding various strings. As
long as those names were not long, and with some overhead, the line of code
was of reasonable size even if it expanded to much more.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Thursday, December 13, 2018 7:27 PM
To: tutor at python.org
Subject: Re: [Tutor] Long Lines techniques

On Thu, Dec 13, 2018 at 12:36:27PM -0500, Avi Gross wrote:

> Simple question:
> 
> When lines get long, what points does splitting them make sense and 
> what methods are preferred?

Good question!

First, some background:

Long lines are a potential code smell: a possible sign of excessively terse
code. A long line may be a sign that you're doing too much in one line.

https://martinfowler.com/bliki/CodeSmell.html
http://wiki.c2.com/?CodeSmell
https://blog.codinghorror.com/code-smells/

Related: 
https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/

Note that merely splitting a logical line over two or more physical lines
may still be a code-smell. Sure, your eyes don't get as tired reading
fifteen lines of 50 characters each, compared to a single 750 character
line, but there's just as much processing going on in what is essentially a
single operation.

Long lines are harder to read: your eyes have to scan across a long line,
and beyond 60 or 70 characters, it becomes physically more difficult to scan
across the line, and the error rate increases. 
[Citation required.]

But short lines don't include enough information, so the traditional
compromise is 80 characters, the character width of the old-school
green-screen terminals. The Python standard library uses 79 characters. 
(The odd number is to allow for scripts which count the newline at the end
of the line as one of the 80.)

https://www.python.org/dev/peps/pep-0008/


Okay, so we have a style-guide that sets a maximum line length, whether it
is 72 or 79 or 90 or 100 characters. What do you do when a line exceeds that
length?

The only firm rule is that you must treat each case on its own merits. 
There is no one right or wrong answer. Every long line of code is different,
and the solution will depend on the line itself. There is no getting away
from human judgement.


(1) Long names. Do you really need to call the variable
"number_of_characters" when "numchars" or even "n" would do?

The same applies to long function names: "get_data_from_database" is
probably redundant, "get_data" will probably do.

Especially watch out for long dotted names that you use over and over again.
Unlike static languages like Java, each dot represents a runtime lookup.
Long names like:

    package.subpackage.module.object.method

requires four lookups. Look for oportunities to make an alias for a long
name and avoid long chains of dots:

    for item in sequence:
        do_something_with(package.subpackage.module.object.method(arg,
item))

can be refactored to:

    method = package.subpackage.module.object.method
    for item in sequence:
        do_something_with(method(arg, item))

and is both easier to read and more efficient. A double win!


(2) Temporary constants: sometimes it is good enough to just introduce a
simple named constant used once. The cognitive load is low if it is defined
immediately before it is used. Instead of the long line:

    raise ValueError("expected a list, string, dict or None, but instead got
'%s'" % type(value).__name__)

I write:

    errmsg = "expected a list, string, dict or None, but instead got '%s'"
    raise ValueError(errmsg % type(value).__name__)


(3) Code refactoring. Maybe that long line is sign that you need to add a
method or function? Especially if you are using that line, or similar, in
multiple places. But refactoring is justified even if you use the line
*once* if it is complicated enough.

Likewise, sometimes it is helpful to factor out separate sub-expressions
onto their own lines, using their own variables, rather than doing
everything in a single, complicated, expression.

Psychologists, educators and linguists call this "chunking", and it is often
very helpful for simplifying complicated ideas, sentences and expressions.

The lack of chunks is why long Perl one-liners are so inpenetrable.


(4) Split the long logical line over multiple physical lines. This does
nothing to reduce the inherent complexity of the line, but if that's fairly
low to start with, it is often helpful.

Python gives us two ways to split a logical line over multiple physical
lines: a backslash at the end of the line, and brackets of any sort.

The preferred way is to use round brackets for grouping:

    result = (some very long expression
              which can be split over
              many lines)

This is especially useful with function calls:

    result = function(first_argument, second_argument,
                      third_argument, fourth_argument)

If you are building a list or dict literal, there is no need for the
parentheses, as square and curly brackets have the same effect. That's
especially useful with two-dimensional nested lists:

    data = [[row, one, with, many, items],
            [row, two, with, many, items],
            [row, three, with, many, items]]

For long strings, I like to use *implicit string concatentation*. String
literals which are separated by nothing except whitespace are concatenated
at compile-time. So I can write a long string like this:

    long_string = ("this is a very long string which doesn't"
                   " fit on a single line but isn't appropriate"
                   " for a triple-quoted string")

Notice that I split the string at word breaks, and move the space to the
beginning of the physical line rather than the end. I find that I'm less
likely to forget the space if I put it at the start of the line rather than
the end.

Not preferred, but allowed for backwards compatibility and still very
occasionally useful, is to end the line with a bare backslash. I find it
helpful in conjunction with triple quoted strings:

    text = """\
    body of the string
    is aligned
    including the first line
    """

but otherwise the backslash is problematic and error-prone. It must be
*immediately* followed by a newline, if you accidentally add a space after
the backslash it won't work.


And finally:

(5) Its just a style guide, not a law of physics. As Douglas Bader once
said, "Rules are for the guidance of the wise and the obedience of fools."
See also Raymond Hettinger's talk "Beyond PEP 8":

https://twitter.com/raymondh/status/589849947408703488

https://medium.com/@drb/pep-8-beautiful-code-and-the-tyranny-of-guidelines-f
96499f5ac17

Better to go two or three characters beyond the maximum length than to make
the code ugly.



[...]
> There are places you can break lines as in a comprehension such as this
set
> comprehension:
> 
>     letter_set = { letter
>                    for word in (left_list + right_list)
>                    for letter in word }
> 
> The above is an example where I know I can break because the {} is holding
> it together. I know I can break at each "for" or "if" but can I break at
> random places?

Not quite random, you can't break in the middle of a word, but you 
can break between words.



[...]
> I will stop here with saying that unlike many languages, parentheses must
be
> used with care in python as they may create a tuple or even generator
> expression.

But not by accident. You can't create a generator expression by accident 
by wrapping an arbitrary expression in round brackets, or turn a 
expression into a tuple. 

Remember, it isn't the parentheses which make tuples, its the commas. 
Except for the empty tuple special case, (), the parens are ALWAYS 
just there to either group the tuple so as to avoid ambiguity, or to 
visually emphasize that it is a tuple even if the interpreter doesn't 
need the hint.



-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Thu Dec 13 23:20:22 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 13 Dec 2018 23:20:22 -0500
Subject: [Tutor] Python function
In-Reply-To: <puuue4$1e9$3@blaine.gmane.org>
References: <582145781.33310.1544721697131@connect.xfinity.com>
 <puuue4$1e9$3@blaine.gmane.org>
Message-ID: <00ee01d49364$55369480$ffa3bd80$@verizon.net>

If you know what a CommaSeparatedValues file looks like, then reading ONE
LINE gives you enough info to answer the question about how many columns of
data it describes. 

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Alan Gauld via Tutor
Sent: Thursday, December 13, 2018 7:47 PM
To: tutor at python.org
Subject: Re: [Tutor] Python function

On 13/12/2018 17:21, Sammy Lee wrote:
> How do I create a python function that opens a CSV file and determines 
> how many columns of data are in the file? The CSV files have been 
> randomly generated from https://www.mockaroo.com/
> 
> def csv_column_count(openfile):

You will find a bunch of stuff on creating functions in the Functions and
Modules topic of my tutorial.(see below)

You will find a bunch of stuff on handling files in the Handling Files topic
of my tutorial.(see below)

You will find a lot of help on using CSV files in the csv module
documentation in the standard library. (Or you could buy my book "Python
Projects" which has a section on the topic.)



--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Fri Dec 14 00:51:05 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 14 Dec 2018 00:51:05 -0500
Subject: [Tutor] Long Lines techniques
In-Reply-To: <20181214052214.GW13061@ando.pearwood.info>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
 <00ec01d49362$9a125dc0$ce371940$@verizon.net>
 <20181214052214.GW13061@ando.pearwood.info>
Message-ID: <002001d49371$00f71f90$02e55eb0$@verizon.net>

Steve[n],

Yes, I figured out what the problem was but while evaluating I also realized
that a format string was an ALTERNATIVE that automatically called for an str
or repr so in a sense it would work without debugging. 

I find that sometimes the long lines make it harder to see the skeleton of
the logic as you get bogged down by things that may be of lesser importance.

So I close with another idea. I asked about functions with lots of
arguments. An example might be making a graph using a library that lets you
specify literally hundreds of parameters all at once. I realized that a
perfectly valid alternative to make the main purpose more reasonable would
be to stuff all the extra arguments in a dictionary like

Other_args = { this: value,
		That : value,
		...
		}
That is something in braces that can be wrapped for legibility.

Then the main function call can use the **  expansion like so:

Ret = function(from, to, **Other_args)

Of course, you can also use * with a list or other iterable for the
positional args when that makes it easier but at some point you no longer
have a feel as to what the function call is doing. But hiding less important
details this way seems to be good. Not sure about run-time efficiency, of
course.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 14, 2018 12:22 AM
To: tutor at python.org
Subject: Re: [Tutor] Long Lines techniques

On Thu, Dec 13, 2018 at 11:07:59PM -0500, Avi Gross wrote:

[...]
> There are cases where it may make sense to have a long like connected 
> by AND or OR given how python does short-circuiting while returning 
> the last thing or two it touched instead of an actual True/False. For 
> example, you may want to take the first available queue that is not 
> empty with something like
> this:
> 
> Using = A or B or C or ... or Z
> Handling = Using.pop()
> 
> Sure, that could be rewritten into multiple lines. 

using = (A or B
         or C or D 
         or E or F)


[...]
> I recently wrote some code and ran into error messages on lines I was 
> trying to keep short:
> 
> A = 'text"
> A += "more text"
> A += object
> A+= ...

Without knowing the error message, its impossible to say what the problem
is. My guess is that the object on line three wasn't a string. 
In which case, the obvious fix is:

A += str(object)



--
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Fri Dec 14 01:15:50 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 14 Dec 2018 01:15:50 -0500
Subject: [Tutor] Obfuscated Python [was Long Lines techniques]
In-Reply-To: <20181214053543.GX13061@ando.pearwood.info>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
 <00ec01d49362$9a125dc0$ce371940$@verizon.net>
 <20181214053543.GX13061@ando.pearwood.info>
Message-ID: <002101d49374$76391990$62ab4cb0$@verizon.net>

Steven,

There are dunderheads who will maliciously misuse things. Yes, we know what
__add__ is supposed to do. But if someone codes a class with one that
ignores addition to a collection if it has reached a maximum size, or does
addition modulo 16 or makes it play a happy birthday tune, while
subtracting, it may not be trivial for the innocent user of your code to
make much sense of it. 

There are documented IDEAS about what many of those dunder methods might
mean but no enforcement. 

I remember an Abstract Algebra course I took where we often talked about
operations like + and * in very abstract ways and came up with some
fascinating meanings to the operations. An example might be the various
definitions of multiplication for vectors or matrices. When you multiply,
are you multiplying corresponding parts or are you multiplying a row times a
column and summing the multiplications for each slot in the matrix? Or could
multiplication be a tad more complex and require also taking a transpose
first? Or perhaps getting a determinant or eigenvalues or eigenvectors? What
does it mean to add "1" to a complex number or quaternion or octonion? 

The reality is that two people can often try to make a similar class and
come up with different ideas and implementations. If I have a class
representing people who RSVP for a party, might a PLUS of a person result in
upping the party count by 2 since we assume they are bringing a date and if
their presence requires another table, upping that count and so on? A simple
__add__ can result in many things such as checking the added person for
dietary needs and adjusting more stuff just because adding a person can be
complex.

You re not going to sucker me into discussing obfuscation today. Some people
here got touchy last time.

I am not defending PERL nor slamming Python. I am saying people who behave
well will try to avoid writing code that is hard to read and may be
understood in multiple ways. There are cute ways to do things in many
languages.

Last comment, in what some might call stream of consciousness. Python allows
a function to return without an explicit return. Usually it returns None but
I not in some generators it may throw an exception instead. The problem is
that if you are not looking carefully at indentation, you might thing the
statements that follow may be part of the same function. So my personal
preference is to have an explicit return instead of letting it drop out of
the function or at least a comment saying this is the end of the function.
Similarly, many people won't close a file (I am not talking about in a with
statement) or delete variables no longer in use, and depend on these things
happening automatically eventually. There may be nothing wrong with that,
especially for smaller programs. But a part of me appreciates when the
scales are visibly balanced. But when overdone, as in a function where every
statement is in its own try/catch even for problems very unlikely to happen,
there is too muc detail to follow. Something in between seems more
comfortable. So interrupting an algorithm to del no longer needed variables
may not be best either. To each their own.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 14, 2018 12:36 AM
To: tutor at python.org
Subject: [Tutor] Obfuscated Python [was Long Lines techniques]

On Thu, Dec 13, 2018 at 11:07:59PM -0500, Avi Gross wrote:

> Python may claim to be straightforward but I can easily see ways to 
> fool people in python too with dunder methods or function closures or 
> decorators or ...

Dunder methods shouldn't fool anyone. Each dunder method has a
straightforward use, and most of the time you can guess that they do. 
__add__ implements the + operator, etc.

Closures and decorators can be abused, as can anything. One can write
obfuscated code in any language. Here's some amusing obfuscated Python:

https://www.reddit.com/r/Python/comments/i1qgp/reduce_and_lambda_two_great_t
astes_that_taste/

https://mail.python.org/pipermail/python-list/2013-October/658470.html

Anyone else want to share some obfuscated Python code?



--
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From mats at wichmann.us  Fri Dec 14 19:42:53 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 14 Dec 2018 17:42:53 -0700
Subject: [Tutor] Obfuscated Python [was Long Lines techniques]
In-Reply-To: <002101d49374$76391990$62ab4cb0$@verizon.net>
References: <009101d4930a$60d1b860$22752920$@verizon.net>
 <20181214002713.GU13061@ando.pearwood.info>
 <00ec01d49362$9a125dc0$ce371940$@verizon.net>
 <20181214053543.GX13061@ando.pearwood.info>
 <002101d49374$76391990$62ab4cb0$@verizon.net>
Message-ID: <78d4cdda-a659-c332-ae95-ced3ee9af3a5@wichmann.us>

On 12/13/18 11:15 PM, Avi Gross wrote:
> Steven,
> 
> There are dunderheads who will maliciously misuse things. Yes, we know what
> __add__ is supposed to do. But if someone codes a class with one that
> ignores addition to a collection if it has reached a maximum size, or does
> addition modulo 16 or makes it play a happy birthday tune, while
> subtracting, it may not be trivial for the innocent user of your code to
> make much sense of it. 
> 
> There are documented IDEAS about what many of those dunder methods might
> mean but no enforcement. 

As Raymond Hettinger likes to say, Python is a Consenting Adults
language.  You're not prevented from doing things, even if it might
outrage a few...



From msauerland815 at gmail.com  Thu Dec 20 10:49:25 2018
From: msauerland815 at gmail.com (Mary Sauerland)
Date: Thu, 20 Dec 2018 10:49:25 -0500
Subject: [Tutor] Python
Message-ID: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>

Hi, 

I want to get rid of words that are less than three characters but I keep getting errors. I tried multiple ways but keep getting errors. 

Here is my code:

f1_name = "/Users/marysauerland/Documents/file1.txt"
#the opinions
f2_name = "/Users/marysauerland/Documents/file2.txt"
#the constitution


def read_words(words_file):
    return [word.upper() for line in open(words_file, 'r') for word in line.split()]


read_words(f1_name)
#performs the function on the file
set1 = set(read_words(f1_name))
#makes each word into a set and removes duplicate words
read_words(f2_name)
set2 = set(read_words(f2_name))

count_same_words = 0

for word in set1:
    if word in set2:
        count_same_words += 1
#comparing the set1 (set of unique words in the opinions) with set2 (set of unique words in the constitution) and adding 1 for each matching word found which is just counting the words
print(count_same_words)


Best, 

Mary

From bgailer at gmail.com  Thu Dec 20 14:34:49 2018
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 20 Dec 2018 14:34:49 -0500
Subject: [Tutor] Python
In-Reply-To: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
References: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
Message-ID: <CAP1rxO4vRnpQTXqNBtQpNaEO7f1Skb_a97QSEOqqvbyrkeyc5g@mail.gmail.com>

On Dec 20, 2018 12:17 PM, "Mary Sauerland" <msauerland815 at gmail.com> wrote:
>
> Hi,
>
> I want to get rid of words that are less than three characters but I keep
getting errors. I tried multiple ways but keep getting errors.

Hi Mary welcome to the tutor list. We love to help. We are a few
volunteers. It is very difficult for us to be mind readers. So please give
us more information. Especially what the error is you are getting. I
presume it is what we call a trace back. It is important that you copy the
entire traceback and paste it into the email. It will also be very helpful
if you gave us a sample of the two text files and the output You're
Expecting.

>
> Here is my code:
>
> f1_name = "/Users/marysauerland/Documents/file1.txt"
> #the opinions
> f2_name = "/Users/marysauerland/Documents/file2.txt"
> #the constitution
>
>
> def read_words(words_file):
>     return [word.upper() for line in open(words_file, 'r') for word in
line.split()]
>
>
> read_words(f1_name)
> #performs the function on the file
> set1 = set(read_words(f1_name))
> #makes each word into a set and removes duplicate words
> read_words(f2_name)
> set2 = set(read_words(f2_name))
>
> count_same_words = 0
>
> for word in set1:
>     if word in set2:
>         count_same_words += 1
> #comparing the set1 (set of unique words in the opinions) with set2 (set
of unique words in the constitution) and adding 1 for each matching word
found which is just counting the words
> print(count_same_words)
>
>
> Best,
>
> Mary
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor

From mats at wichmann.us  Thu Dec 20 14:54:52 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Thu, 20 Dec 2018 12:54:52 -0700
Subject: [Tutor] Python
In-Reply-To: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
References: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
Message-ID: <dfd5b998-6e0a-d12c-244c-2882d53bb296@wichmann.us>

On 12/20/18 8:49 AM, Mary Sauerland wrote:
> Hi, 
> 
> I want to get rid of words that are less than three characters but I keep getting errors. I tried multiple ways but keep getting errors. 

Just a quick note or two:
> 
> Here is my code:
> 
> f1_name = "/Users/marysauerland/Documents/file1.txt"
> #the opinions
> f2_name = "/Users/marysauerland/Documents/file2.txt"
> #the constitution
> 
> 
> def read_words(words_file):
>     return [word.upper() for line in open(words_file, 'r') for word in line.split()]
> 
> 
> read_words(f1_name)

^^^ this line is meaningless.  "everything is an object" in Python. your
function returns a list object - which you don't do anything with. you
should assign a name to it, like:

constitution_words = read_words(f1_name)

Since no name is assigned to that object, Python sees it has no
references, and it is just lost.

and then...

> #performs the function on the file
> set1 = set(read_words(f1_name))
if you saved the object returned from the earlier call to the function,
then you don't need to call the function again, instead you convert the
saved list object to a set object.  We can't tell whether you have an
eventual use for the unfiltered list of words, or only the set of unique
words, the answer to that determines how you write this section.

picking a more descriptive name than set1 would be a good idea (and
f1_name as well, and others - when writing software, the hard part is
maintenance, where you or others have to go in later and fix or change
something.  using meaningful names really helps with that, so it's a
good habit to get into).

since you have sets consisting of words from your two documents, you may
as well use set operations to work with them.  Do you know the set
operation to find all of the members of one set that are also in another
set? hint: in set theory, that is called the intersection.

you say you are trying to remove short words, but there seems to be no
code to do that.  instead you seem to be solving a different problem?

From avigross at verizon.net  Thu Dec 20 16:54:16 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 20 Dec 2018 16:54:16 -0500
Subject: [Tutor] Python
In-Reply-To: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
References: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
Message-ID: <007201d498ae$8db7c730$a9275590$@verizon.net>

Mary,

Mary,

It is often best to develop and test small parts of the project where you
can easily play with it, then move it into more complex configurations like
a function body

Here is your code:

def read_words(words_file):
    return [word.upper() for line in open(words_file, 'r') for word in
line.split()]

I made a file on my local system and this works:

def read_words(words_file):
    return [word.upper() for line in open(words_file, 'r') for word in
line.split()]

now you are returning an uppercase version of the current 'word" stored in
word. So what is the length of that word?

Here is the modified variation on your code:

>>> [word.upper() for line in open('TESTINK.txt', 'r') for word in
line.split()]
['THIS', 'IS', 'LINE', 'ONE', 'AND', 'THIS', 'IS', 'ANOTHER', 'LINE',
'JUST', 'TO', 'TEST', 'WITH.']

Here is yet another modification showing the length, instead:

>>> [len(word) for line in open('TESTINK.txt', 'r') for word in
line.split()]
[4, 2, 4, 3, 3, 4, 2, 7, 4, 4, 2, 4, 5]

By your rules, you want to only keep those words where "len(word) > 3"

So where in the list comprehension would you add this an if condition to get
this?

['THIS', 'LINE', 'THIS', 'ANOTHER', 'LINE', 'JUST', 'TEST', 'WITH.']

Since you read in all your data using the same function, you might even make
it take an optional value to cut at, defaulting with 3 or even 0.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Mary Sauerland
Sent: Thursday, December 20, 2018 10:49 AM
To: tutor at python.org
Subject: [Tutor] Python

Hi, 

I want to get rid of words that are less than three characters but I keep
getting errors. I tried multiple ways but keep getting errors. 

Here is my code:

f1_name = "/Users/marysauerland/Documents/file1.txt"
#the opinions
f2_name = "/Users/marysauerland/Documents/file2.txt"
#the constitution


def read_words(words_file):
    return [word.upper() for line in open(words_file, 'r') for word in
line.split()]


read_words(f1_name)
#performs the function on the file
set1 = set(read_words(f1_name))
#makes each word into a set and removes duplicate words
read_words(f2_name)
set2 = set(read_words(f2_name))

count_same_words = 0

for word in set1:
    if word in set2:
        count_same_words += 1
#comparing the set1 (set of unique words in the opinions) with set2 (set of
unique words in the constitution) and adding 1 for each matching word found
which is just counting the words
print(count_same_words)


Best, 

Mary
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From steve at pearwood.info  Thu Dec 20 17:34:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 21 Dec 2018 09:34:06 +1100
Subject: [Tutor] Python
In-Reply-To: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
References: <7B53B45E-08B8-45CD-B3E1-AB492A1F73A1@gmail.com>
Message-ID: <20181220223406.GK13061@ando.pearwood.info>

On Thu, Dec 20, 2018 at 10:49:25AM -0500, Mary Sauerland wrote:

> I want to get rid of words that are less than three characters


> f1_name = "/Users/marysauerland/Documents/file1.txt"
> #the opinions
> f2_name = "/Users/marysauerland/Documents/file2.txt"
> #the constitution

Better than comments are meaningful file names:

opinions_filename = "/Users/marysauerland/Documents/file1.txt"
constitution_filename = "/Users/marysauerland/Documents/file2.txt"

> def read_words(words_file):
>     return [word.upper() for line in open(words_file, 'r') for word in line.split()]

Don't try to do too much in a single line of code. While technically 
that should work (I haven't tried it to see that it actually does) it 
would be better written as:

def read_words(words_file):
    with open(words_file, 'r') as f:
        return [word.upper() for line in f for word in line.split()]

This also has the advantage of ensuring that the file is closed after 
the words are read. In your earlier version, it is possible for the file 
to remain locked in an open state.

Note that in this case Python's definition of "word" may not agree with 
the human reader's definition of a word. For example, Python, being 
rather simple-minded, will include punctuation in words so that 

"HELLO"
"HELLO."

count as different words. Oh well, that's something that can be adjusted 
later. For now, let's just go with the simple-minded definition of a 
word, and worry about adjusting it to something more specialised later.



> read_words(f1_name)
> #performs the function on the file

The above line of code (and comment) are pointless. The function is 
called, the file is read, the words are generated, and then immediately 
thrown away. To use the words, you need to assign them to a variable, as 
you do below:


> set1 = set(read_words(f1_name))
> #makes each word into a set and removes duplicate words

A meaningful name is better. Also the comment is inaccurate: it is not 
that *each individual* word is turned into a set, but that the *list* of 
all the words are turned into a set. So better would be:

opinions_words = set(read_words(opinions_filename))
constitition_words = set(read_words(constitution_filename))

This gives us the perfect opportunity to skip short words:

opinions_words = set(
    word for word in read_words(opinions_filename) if len(word) >= 3)
constitition_words = set(
    word for word in read_words(constitution_filename) if len(word) >= 3)


Now you have two sets of unique words, each word guaranteed to be at 
least 3 characters long.

The next thing you try to do is count how many words appear in each set. 
You do it with a double loop:

> count_same_words = 0
> for word in set1:
>     if word in set2:
>         count_same_words += 1

but the brilliant thing about sets is that they already know how to do 
this themselves! Let's see the sorts of operations sets understand:


py> set1 = set("abcdefgh")
py> set2 = set("defghijk")
py> set1 & set2  # the intersection (overlap) of both sets
{'h', 'd', 'f', 'g', 'e'}
py> set1 | set2  # the union (combination) of both sets
{'f', 'd', 'c', 'b', 'h', 'i', 'k', 'j', 'a', 'g', 'e'}
py> set1 ^ set2  # items in one or the other but not both sets
{'i', 'k', 'c', 'b', 'j', 'a'}
py> set1 - set2  # items in set1 but not set2
{'c', 'b', 'a'}


(In the above, "py>" is the Python prompt. On your computer, your prompt 
is probably set to ">>>".)

Can you see which set operation, one of & | ^ or - , you would use to 
get the set of words which appear in both sets? Hint: it isn't the - 
operation. If you wanted to know how many words appear in the 
constitution but NOT in the opinions, you could write:


word_count = len(constitition_words - opinions_words)


Does that give you a hint how to approach this?



Steve

From aine.gormley at gmail.com  Thu Dec 20 16:47:44 2018
From: aine.gormley at gmail.com (Aine Gormley)
Date: Thu, 20 Dec 2018 22:47:44 +0100
Subject: [Tutor] loop error
Message-ID: <CACN0mQqvvTt5RqDY+ehnj5iF-A-UuBwhLV7o1+Y2kET5kvJzbA@mail.gmail.com>

Hello, could somebody take a quick look at my code? I am unsure why I am
getting a loop error?

From robertvstepp at gmail.com  Thu Dec 20 20:58:20 2018
From: robertvstepp at gmail.com (boB Stepp)
Date: Thu, 20 Dec 2018 19:58:20 -0600
Subject: [Tutor] loop error
In-Reply-To: <CACN0mQqvvTt5RqDY+ehnj5iF-A-UuBwhLV7o1+Y2kET5kvJzbA@mail.gmail.com>
References: <CACN0mQqvvTt5RqDY+ehnj5iF-A-UuBwhLV7o1+Y2kET5kvJzbA@mail.gmail.com>
Message-ID: <CANDiX9KrgKcaDtKZrb_G0WL81gULBDN4HpcSqxrY7W9-VA587g@mail.gmail.com>

Greetings Aine!

On Thu, Dec 20, 2018 at 6:57 PM Aine Gormley <aine.gormley at gmail.com> wrote:
>
> Hello, could somebody take a quick look at my code? I am unsure why I am
> getting a loop error?

This is a plain text only list that does not (typically) allow file
attachments.  So I do not see any code.  So if you wish for someone on
this list to assist you, you need to copy and paste the relevant code
into a plain text email, including a copy and paste of the error
messages you are receiving.  It is also helpful to mention your
operating system and what version of Python you are using.  An even
better approach would be to construct the smallest possible runnable
example code that reproduces your problem.

Good luck and better thinking!

-- 
boB

From steve at pearwood.info  Fri Dec 21 00:21:29 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 21 Dec 2018 16:21:29 +1100
Subject: [Tutor] loop error
In-Reply-To: <CACN0mQqvvTt5RqDY+ehnj5iF-A-UuBwhLV7o1+Y2kET5kvJzbA@mail.gmail.com>
References: <CACN0mQqvvTt5RqDY+ehnj5iF-A-UuBwhLV7o1+Y2kET5kvJzbA@mail.gmail.com>
Message-ID: <20181221052129.GL13061@ando.pearwood.info>

On Thu, Dec 20, 2018 at 10:47:44PM +0100, Aine Gormley wrote:
> Hello, could somebody take a quick look at my code? I am unsure why I am
> getting a loop error?

That's hard to do if you don't show us the code :-) 

Please COPY AND PASTE (don't try to retype it from memory) the MINIMUM 
amount of code which actually runs. If you're unsure about cutting the 
code down to the minimum that demonstrates the error, please feel free 
to ask. You can also read this:

http://www.sscce.org/


Its written for Java programmers, but applies to any programming 
language including Python.

And what is "a loop error"? Please COPY AND PASTE the full exception. It 
should start with a line:

    Traceback...

and end with some sort of error message.



-- 
Steve

From michaelmossey at gmail.com  Thu Dec 20 23:16:52 2018
From: michaelmossey at gmail.com (Michael Mossey)
Date: Thu, 20 Dec 2018 20:16:52 -0800
Subject: [Tutor] graphics library for teaching Python
Message-ID: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>

I'm a computer science tutor and I'm asking advice about a graphics or game
library that can be used with Python effectively for teaching purposes.

I've found that having my student pick a long-term project is a good way
for them to learn coding, and graphics or games make great projects that
both have stimulating results and bring computer-science-y topics into the
mix (i.e. they are natural vehicles for OO, data structures, and
algorithms).

There are two sub-topics I'm interested in - (1) graphics, as in drawing
interesting pictures or art, or using diagrams for data visualization. (2)
Simple games, with the use of sprites.

I've been using PyGame, but I'm not happy with it. It's not very well
organized or documented, and not very capable of general graphics.

So I'm looking into a few other possibilities. No library can be everything
to everyone, but I'd  like something that gives students exposure to a
variety of programming tasks, like constructing GUIs, events and event
loops, and 2-D graphics via stroking and filling common polygon shapes and
Bezier curves. (3D is not necessary for now.)

Here are some alternatives to PyGame:

- Pyglet. Doesn't look promising. I'm mainly interested in 2D, and I don't
see common data visualization tasks provided like filling and stroking
Bezier curve shapes (maybe I'm missing something).

- Cairo. Looks great for static 2D graphics, but not games .. again maybe
I'm missing something.

- Cocos2D. Good for games but not necessary general 2D graphics.

- PyQt. I used this extensively at my last regular desk job many years ago,
and if I recall it has the QCanvas element with pretty deep graphics
ability and also event handling. It has collision detection too, I think,
allowing for easy 2D game writing.

It looks to me like PyQt is the most capable program and most related to
what I want to do, but one thing I'm wondering about is how widely its used
and whether it would be good exposure for students to connect them to the
wider world of computing they will someday enter.

Any thoughts would be appreciated.

Mike

From alan.gauld at yahoo.co.uk  Fri Dec 21 03:38:47 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 21 Dec 2018 08:38:47 +0000
Subject: [Tutor] graphics library for teaching Python
In-Reply-To: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
References: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
Message-ID: <pvi8mk$rim$1@blaine.gmane.org>

On 21/12/2018 04:16, Michael Mossey wrote:
> I'm a computer science tutor and I'm asking advice about a graphics or game
> library that can be used with Python effectively for teaching purposes.

I'll start by saying that Python is not the berst programming language
for graphics. However there are some other options beyond those you mention:

> There are two sub-topics I'm interested in - (1) graphics, as in drawing
> interesting pictures or art, or using diagrams for data visualization. 

The turtle module gives a basic intro yto drawing shapes and
introduces an interesting alternative to traditional cartesian
coordinate geometry.

The Tkinter module includes a Canvas widget that supports graphic primitives

The gnuplot and matplotlib packages allow plotting of data.
Neither is in the standard library but can be easily installed.

The SciPy bundle comes with a neat tool for visualisaton
which might be useful. Especially for older students with a
knowledge of math - you don't specify the target age group.

Finally Dabo includes a graphical GUI builder and database
integration. I haven't used it for several years so I'm not
sure on its current development status but it seemed to work
well when I tried it.

(2) Simple games, with the use of sprites.

I'm not much of a games person so I'll leave others to comment.

HTH
-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From adameyring at gmail.com  Fri Dec 21 07:20:35 2018
From: adameyring at gmail.com (Adam Eyring)
Date: Fri, 21 Dec 2018 07:20:35 -0500
Subject: [Tutor] graphics library for teaching Python
In-Reply-To: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
References: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
Message-ID: <CAPStRW9PYeZ8yzQD7qpY98cQQfS6QTng3cX3H6p_nrqEgX7w-w@mail.gmail.com>

On Fri, Dec 21, 2018, 3:26 AM Michael Mossey <michaelmossey at gmail.com>
wrote:

>
> There are two sub-topics I'm interested in - (1) graphics, as in drawing
> interesting pictures or art, or using diagrams for data visualization. (2)
> Simple games, with the use of sprites.
>

One that hasn't been mention in yours and Alan's lists is Arcade. Developed
in recent years by a Simpson College professor, he's been using it for
teaching game development. Supports a lot of features you're asking for and
I've been enjoying playing with it.
Http://arcade.academy

From cranky.frankie at gmail.com  Fri Dec 21 10:04:26 2018
From: cranky.frankie at gmail.com (Cranky Frankie)
Date: Fri, 21 Dec 2018 10:04:26 -0500
Subject: [Tutor] graphics library for teaching Python
In-Reply-To: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
References: <CAHKS=zHat=hn0TDLQj6Gq6QdoFC3zJM6OkFnB__qv1g81bDygw@mail.gmail.com>
Message-ID: <CAON5Gn2GHqFX-idOdxtHT+RMcW84UuS0w6hMdP+v+Q0FB4x2cw@mail.gmail.com>

Tutor at python.org:
"I'm a computer science tutor and I'm asking advice about a graphics or game
library that can be used with Python effectively for teaching purposes."

Michael Dawson's "Python Programming for Absolute Beginners" uses PyGame.
Might be worth a look.



On Fri, Dec 21, 2018 at 3:26 AM Michael Mossey <michaelmossey at gmail.com>
wrote:

> I'm a computer science tutor and I'm asking advice about a graphics or game
> library that can be used with Python effectively for teaching purposes.
>
> I've found that having my student pick a long-term project is a good way
> for them to learn coding, and graphics or games make great projects that
> both have stimulating results and bring computer-science-y topics into the
> mix (i.e. they are natural vehicles for OO, data structures, and
> algorithms).
>
> There are two sub-topics I'm interested in - (1) graphics, as in drawing
> interesting pictures or art, or using diagrams for data visualization. (2)
> Simple games, with the use of sprites.
>
> I've been using PyGame, but I'm not happy with it. It's not very well
> organized or documented, and not very capable of general graphics.
>
> So I'm looking into a few other possibilities. No library can be everything
> to everyone, but I'd  like something that gives students exposure to a
> variety of programming tasks, like constructing GUIs, events and event
> loops, and 2-D graphics via stroking and filling common polygon shapes and
> Bezier curves. (3D is not necessary for now.)
>
> Here are some alternatives to PyGame:
>
> - Pyglet. Doesn't look promising. I'm mainly interested in 2D, and I don't
> see common data visualization tasks provided like filling and stroking
> Bezier curve shapes (maybe I'm missing something).
>
> - Cairo. Looks great for static 2D graphics, but not games .. again maybe
> I'm missing something.
>
> - Cocos2D. Good for games but not necessary general 2D graphics.
>
> - PyQt. I used this extensively at my last regular desk job many years ago,
> and if I recall it has the QCanvas element with pretty deep graphics
> ability and also event handling. It has collision detection too, I think,
> allowing for easy 2D game writing.
>
> It looks to me like PyQt is the most capable program and most related to
> what I want to do, but one thing I'm wondering about is how widely its used
> and whether it would be good exposure for students to connect them to the
> wider world of computing they will someday enter.
>
> Any thoughts would be appreciated.
>
> Mike
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
Frank L. "Cranky Frankie" Palmeri, Risible Riding Raconteur & Writer
"If you have a garden and a library, you have everything you need." - Cicero

From mike_barnett at hotmail.com  Fri Dec 21 12:15:02 2018
From: mike_barnett at hotmail.com (Mike Barnett)
Date: Fri, 21 Dec 2018 17:15:02 +0000
Subject: [Tutor] graphics library for teaching Python
Message-ID: <MWHPR11MB171030E7C9DECBB9B9A9C738E0B80@MWHPR11MB1710.namprd11.prod.outlook.com>

>There are two sub-topics I'm interested in - (1) graphics, as in drawing interesting pictures or art, or using diagrams for data visualization. (2) Simple games, with the use of sprites.

Can you post some images of similar programs / graphics you are wanting to create?  It would be extremely helpful to see concrete "targets".  Then I can better inform you if PySimpleGUI is capable of doing what you seek.

I've recently been writing more games using PySimpleGUI to see what's missing in the SDK, to determine new features that will help developers.

For data visualization, there's Matplotlib and Pyplot, both integrate with PySimpleGUI very well.  Someone mentioned Turtle, it too integrates well by using a tkinter canvas.

I've been doing simple graphs by using the drawing primitives and games using images that I often include in the source code.  The Uno Game for example has all of the cards included in the .py file.

Some examples:

Bar chart:
https://gist.github.com/MikeTheWatchGuy/fd79baad627ae879bb74437427ad4ff0

Simple Matplotlib:
https://gist.github.com/MikeTheWatchGuy/33579888cd976e2ef0231869bba38bb4

Simple plotting - sine wave:
https://gist.github.com/MikeTheWatchGuy/397b681d1b3863a74936724fdfd1ea34

Scrolling bar chart:
https://user-images.githubusercontent.com/13696193/47611749-18964c80-da42-11e8-93c4-6821a6fce488.gif

Matplotlib integration:
https://user-images.githubusercontent.com/13696193/50046965-cb664c80-007a-11e9-9470-b7d1ab3fa20e.gif


Some simple games

Conway's Game of Life:
https://user-images.githubusercontent.com/13696193/50244709-9e79a880-039e-11e9-9b51-733357a87c68.gif

Uno Card Game:
https://user-images.githubusercontent.com/13696193/49945232-67952580-feba-11e8-90c8-7dc31c5f7c67.gif

Turtle:
https://user-images.githubusercontent.com/13696193/49346588-b644f300-f662-11e8-8c83-44c74aedf89f.gif

A Crossword Puzzle GUI:
https://user-images.githubusercontent.com/13696193/47968340-98ba4480-e036-11e8-9d44-8a39ac174533.jpg

Chess GUI:
https://user-images.githubusercontent.com/13696193/47380967-b6012180-d6cc-11e8-946d-a646921fb8d0.gif


These are all tkinter based examples. 

While PySimpleGUI does have a Qt port, it's not as complete when it comes to these graphics primitives and integration with Matplotlib.


-mike


From michaelmossey at gmail.com  Fri Dec 21 13:39:47 2018
From: michaelmossey at gmail.com (Michael Mossey)
Date: Fri, 21 Dec 2018 10:39:47 -0800
Subject: [Tutor] graphics library for teaching Python
In-Reply-To: <MWHPR11MB171030E7C9DECBB9B9A9C738E0B80@MWHPR11MB1710.namprd11.prod.outlook.com>
References: <MWHPR11MB171030E7C9DECBB9B9A9C738E0B80@MWHPR11MB1710.namprd11.prod.outlook.com>
Message-ID: <CAHKS=zEB-Ojjn+byGp=co4K0zhGJ5rObzc6fTgHDc6w71RrQKg@mail.gmail.com>

On Fri, Dec 21, 2018 at 9:15 AM Mike Barnett <mike_barnett at hotmail.com>
wrote:

> >There are two sub-topics I'm interested in - (1) graphics, as in drawing
> interesting pictures or art, or using diagrams for data visualization. (2)
> Simple games, with the use of sprites.
>
> Can you post some images of similar programs / graphics you are wanting to
> create?  It would be extremely helpful to see concrete "targets".  Then I
> can better inform you if PySimpleGUI is capable of doing what you seek.
>

I think PySimpleGUI might be a great fit.

I'm getting my ideas from projects way in the past, so I can't easily find
screenshots. But what I'm looking for is pretty easy to describe.

for graphics: "Capabilities similar to JavaScript canvas plus collision
detection."

In other words, basic text filling/stroking, basic shape and line and
Bezier curve filling/stroking, and then collision detection to make simple
games easier.

for GUIs: "Basic dialogs, buttons, labels, checkboxes, and integrated
canvas."

My students range a lot in age and ability. Youngest is 9th grade, ranging
upward to professionals who are looking to learn programming as a new skill
to use in the workplace or put on their resume.



>
> I've recently been writing more games using PySimpleGUI to see what's
> missing in the SDK, to determine new features that will help developers.
>
> For data visualization, there's Matplotlib and Pyplot, both integrate with
> PySimpleGUI very well.  Someone mentioned Turtle, it too integrates well by
> using a tkinter canvas.
>

By data visualization, I don't mean necessarily plotting. Matplotlib might
be an awkward fit.

I'm thinking of visualizing math processes, algorithms, and so forth. Say,
we animate a cubic curve changing shape while the coefficients change. Or
we animate the inner workings of a tic tac toe strategy algorithm by having
it draw a tic tac toe board with changing colors or shapes to represent
positions or calculations in progress. I don't mean something really
complicated. Just picture the simplest thing that makes sense.

Thanks for the other links to examples.

Mike

From mike_barnett at hotmail.com  Fri Dec 21 14:37:39 2018
From: mike_barnett at hotmail.com (Mike Barnett)
Date: Fri, 21 Dec 2018 19:37:39 +0000
Subject: [Tutor] graphics library for teaching Python
In-Reply-To: <CAHKS=zEB-Ojjn+byGp=co4K0zhGJ5rObzc6fTgHDc6w71RrQKg@mail.gmail.com>
References: <MWHPR11MB171030E7C9DECBB9B9A9C738E0B80@MWHPR11MB1710.namprd11.prod.outlook.com>
 <CAHKS=zEB-Ojjn+byGp=co4K0zhGJ5rObzc6fTgHDc6w71RrQKg@mail.gmail.com>
Message-ID: <MWHPR11MB171027AB5FECEBEE23E59E0AE0B80@MWHPR11MB1710.namprd11.prod.outlook.com>

For PySimpleGUI?.
Things like ?collision detection? are up to the application.  PySimpleGUI gives you ability to place ?objects? at x,y coordinates.  The primitives allow images, circles, lines, rectangles, arcs, and text to be drawn and removed.  If one overlaps another you?re on the hook for determining that.

GUIs
PySimpleGUI rocks for doing custom GUIs.  At last count there were 28 widgets that you can place pretty much anywhere within a window.  These are the basic widgets (buttons, listbox, checkbox, radio button, etc.) as well as advanced Panes and Button-Menus.  It?s excellent for teaching GUIs and how to layout a GUI because all of the widgets are available for the developer, not a subset.

You can run sync and async event loops so doing things like polling hardware devices is easy.  No callbacks to worry about.
Here?s a description of the architecture
https://pysimplegui.readthedocs.io/architecture/

Determining which button was clicked is as easy as a comparison:
if button == ?Download?:
   do_download_stuff()

Animation is a super easy to do as all you?re doing is showing images in a loop.

Here?s a program that does a simple graph of a math function that allows you to change the variables using a couple of sliders.  It demonstrates the ?graphing? capability.
https://gist.github.com/MikeTheWatchGuy/8453e82fc65e24060775443e843ab80b


One big drawback for tkinter / PySimpleGUI is the inability to have an image as the background.  You can have an image for the background of a canvas, but of the entire window.  For that you?ll need to use PySimpleGUIQt.

Anyway, it?s worth a look to see if it fits some of your needs.


@mike<mailto:mike_barnett at hotmail.com>

From: Michael Mossey <michaelmossey at gmail.com>
Sent: Friday, December 21, 2018 1:40 PM
To: Mike Barnett <mike_barnett at hotmail.com>
Cc: tutor at python.org
Subject: Re: [Tutor] graphics library for teaching Python



On Fri, Dec 21, 2018 at 9:15 AM Mike Barnett <mike_barnett at hotmail.com<mailto:mike_barnett at hotmail.com>> wrote:
>There are two sub-topics I'm interested in - (1) graphics, as in drawing interesting pictures or art, or using diagrams for data visualization. (2) Simple games, with the use of sprites.

Can you post some images of similar programs / graphics you are wanting to create?  It would be extremely helpful to see concrete "targets".  Then I can better inform you if PySimpleGUI is capable of doing what you seek.

I think PySimpleGUI might be a great fit.

I'm getting my ideas from projects way in the past, so I can't easily find screenshots. But what I'm looking for is pretty easy to describe.

for graphics: "Capabilities similar to JavaScript canvas plus collision detection."

In other words, basic text filling/stroking, basic shape and line and Bezier curve filling/stroking, and then collision detection to make simple games easier.

for GUIs: "Basic dialogs, buttons, labels, checkboxes, and integrated canvas."

My students range a lot in age and ability. Youngest is 9th grade, ranging upward to professionals who are looking to learn programming as a new skill to use in the workplace or put on their resume.



I've recently been writing more games using PySimpleGUI to see what's missing in the SDK, to determine new features that will help developers.

For data visualization, there's Matplotlib and Pyplot, both integrate with PySimpleGUI very well.  Someone mentioned Turtle, it too integrates well by using a tkinter canvas.

By data visualization, I don't mean necessarily plotting. Matplotlib might be an awkward fit.

I'm thinking of visualizing math processes, algorithms, and so forth. Say, we animate a cubic curve changing shape while the coefficients change. Or we animate the inner workings of a tic tac toe strategy algorithm by having it draw a tic tac toe board with changing colors or shapes to represent positions or calculations in progress. I don't mean something really complicated. Just picture the simplest thing that makes sense.

Thanks for the other links to examples.

Mike



From mats at wichmann.us  Sun Dec 23 18:40:31 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Sun, 23 Dec 2018 16:40:31 -0700
Subject: [Tutor] look back comprehensively
In-Reply-To: <20181114220131.GI4071@ando.pearwood.info>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
Message-ID: <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>

On 11/14/18 3:01 PM, Steven D'Aprano wrote:
> On Tue, Nov 13, 2018 at 11:59:24PM -0500, Avi Gross wrote:
>> I have been thinking about the thread we have had where the job seemed to be
>> to read in a log file and if some string was found, process the line before
>> it and generate some report. Is that generally correct?
> 
> If that description is correct, then the solution is trivial: iterate 
> over the file, line by line, keeping the previous line:
> 
> previous_line = None
> for current_line in file:
>     process(current_line, previous_line)
>     previous_line = current_line
> 
> 
> No need for complex solutions, or memory-hungry solutions that require 
> reading the entire file into memory at once (okay for, say, a million 
> lines, but not if your logfile is 2GB in size). If you need the line 
> number:

Absolutely, let's not go reading everything in in bulk, Python has tried
very hard to build elegant iterators all over the place to avoid "doing
the whole thing" when you don't have to - and it has helped heaps with
what years ago used to be an indictment of Python as being "too slow":
not doing work you don't need to do is always a good thing.

The general problem is pretty common, I think, and expands a bit beyond
the trivial case.  Log files may have a start and end marker for a case
you have to examine, and the number of lines between those may be fixed
(0, 1, 2, whatever - 0 being the most trivial case) or variable - I
think that's the situation that started this thread way back, and it
comes up lot. You can have a search on Stack{Exchange,Overflow}, a
non-trivial number of people have asked.

I just now have a different scenario, similar requirement... I happen to
want to scan a bunch of Python code to locate instances of the Python
idiom for ignoring certain possible/expected error conditions:

try:
    block of code
except SomeError:
    pass


to experiment with replacing those with contextlib.suppress and see if
the team of a particular project thinks that makes code more readable:

from contextlib import suppress
...

with suppress(SomeError):
    block of code


This is pretty similar - I want to identify a multi-line sequence that
starts with "try:", has one or more lines, then ends with, in this case,
a two-line sequence where the first line starts with "except" and is
immediately followed by "pass" - but to make it more exciting, is can
then not then followed by either "else" or "finally", because if the try
block has either of those clauses, it is not a candidate for using
suppress instead.  Regexes aren't necessarily helpful on multiline
patterns, even if you ignore the jokes about regexes ("now you have two
problems")

As common as this is, I suspect there are elegant solutions that go
beyond everyone rolling their own.  I'm thinking that maybe pyparsing
has the tools to help with this kind of problem...  I may take a look
into that over then next few days since I just ended up with a personal
interest.

From pasokan at gmail.com  Mon Dec 24 00:25:34 2018
From: pasokan at gmail.com (Asokan Pichai)
Date: Mon, 24 Dec 2018 10:55:34 +0530
Subject: [Tutor] look back comprehensively
In-Reply-To: <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
Message-ID: <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>

On Mon, Dec 24, 2018 at 5:16 AM Mats Wichmann <mats at wichmann.us> wrote:

> On 11/14/18 3:01 PM, Steven D'Aprano wrote:
> > On Tue, Nov 13, 2018 at 11:59:24PM -0500, Avi Gross wrote:
> >> I have been thinking about the thread we have had where the job seemed
> to be
> >> to read in a log file and if some string was found, process the line
> before
> >> it and generate some report. Is that generally correct?
> >
> > If that description is correct, then the solution is trivial: iterate
> > over the file, line by line, keeping the previous line:
> >
> > previous_line = None
> > for current_line in file:
> >     process(current_line, previous_line)
> >     previous_line = current_line
> >
>

The classic COBOL programmers idiom :-)

That said, sometimes text processing at the shell can precede or even
replace some of these. Of course that assumes Unix/Linux OS.

In this speciic case grep with its -B option can be used to generate a file
that contains only the lines of interest.

HTH

Asokan Pichai

From alan.gauld at yahoo.co.uk  Mon Dec 24 04:14:38 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 24 Dec 2018 09:14:38 +0000
Subject: [Tutor] look back comprehensively
In-Reply-To: <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
 <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
Message-ID: <pvq7tq$pco$1@blaine.gmane.org>

On 24/12/2018 05:25, Asokan Pichai wrote:

> That said, sometimes text processing at the shell can precede or even
> replace some of these. Of course that assumes Unix/Linux OS.
> 

It's a good point except that any OS will do.
Even MS DOS had basic text processing commands suitable
for filtering lines of interest and OS' like DEC VMS
or IBM MVS are every bit as powerful as *nix for
text processing.

But the basic principle of using the best tool for the
job applies and often pre-processing in the OS before
running a Python script is sound advice.

In fact for most log file analysis I still use [ng]awk.
Its hard to beat the simplicity of regex based event
handling for slicing text files.


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From mats at wichmann.us  Mon Dec 24 10:14:50 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Mon, 24 Dec 2018 08:14:50 -0700
Subject: [Tutor] look back comprehensively
In-Reply-To: <pvq7tq$pco$1@blaine.gmane.org>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
 <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
 <pvq7tq$pco$1@blaine.gmane.org>
Message-ID: <98e7c585-530b-8092-2f11-75a100e07a8f@wichmann.us>

On 12/24/18 2:14 AM, Alan Gauld via Tutor wrote:
> On 24/12/2018 05:25, Asokan Pichai wrote:
> 
>> That said, sometimes text processing at the shell can precede or even
>> replace some of these. Of course that assumes Unix/Linux OS.

> In fact for most log file analysis I still use [ng]awk.
> Its hard to beat the simplicity of regex based event
> handling for slicing text files.

Sure... there's nothing wrong with using "the appropriate tool for the
job" rather than making everything look like a Python problem.  There's
Perl, too... the ultimate log analysis toolkit. If only I could read
what I wrote a week later :)



From avigross at verizon.net  Mon Dec 24 19:45:04 2018
From: avigross at verizon.net (Avi Gross)
Date: Mon, 24 Dec 2018 19:45:04 -0500
Subject: [Tutor] look back comprehensively
In-Reply-To: <98e7c585-530b-8092-2f11-75a100e07a8f@wichmann.us>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
 <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
 <pvq7tq$pco$1@blaine.gmane.org>
 <98e7c585-530b-8092-2f11-75a100e07a8f@wichmann.us>
Message-ID: <006c01d49beb$13d91280$3b8b3780$@verizon.net>

There is linear thinking and then there is more linear thinking.

As Alan, Mats and others said, there are often choices we can make and many
approaches.

If your goal is to solve a problem NOW and right where you are, any method
you can find is great.

If your goal is to solve it repeatedly or in many possible places or more
efficiently or are trying to learn more about a particular language/method
like python, then you have constraints to consider.

The linear solution might be to solve the entire problem in one way. That
may be python or it may be some UNIX tool like AWK.

The flexible solutions may include doing it in stages, perhaps switching
tools along the way.

So for the logfile solution, some see two paradigms. One is to read in the
entire file, no matter how big. The other is to read in no more than a line
at a time.

Bogus choice.

In the logfile example, there are many other choices. If you can recognize
the beginning of a region and then the end, sure, you can buffer lines. But
how about a solution where you simply read the entire file (a line at a
time) while writing a second file on only the lines you might need. When
done, if the file is empty, move on.

If not, open that smaller file, perhaps reading it all in at once, and
process that small file containing perhaps a few error logs. Or, heck, maybe
each error region was written into a different file and you process them one
at a time with even less complex code.

Unless efficiency is paramount, many schemes like these can result in a
better division of labor, easier to understand and perhaps even code. And
some of these schemes may even have other advantages.

Here is yet another weird idea. REWIND. If the log file is a real file, you
can wait till you reach the end condition that tells you that you also need
earlier lines. Make a note of your position in the file and calculate an
estimate of how far back in the file you need to go, perhaps a very generous
estimate. Say you rewind a thousand bytes and start reading lines again,
perhaps discarding the first one as likely to be incomplete. You can read
all these lines into a buffer, figure out which lines you need starting from
the end, do what you want with it, toss the buffer, reset the file pointer,
and continue!

That solution is not linear at all. If you have a huge log file with a
sparse (or nonexistent) number of errors to process, this may even be faster
than a scheme which buffers the last hundred lines and is constantly rolling
those lines over for naught.

I am not criticizing any approach but suggesting that one good approach to
problems is to not close in on one particular solution prematurely. Be open
to other solutions and even think outside that box. There may be many ways
to look back. 

As for the UNIX tools, one nice thing about them was using them in a
pipeline where each step made some modification and often that merely
allowed the next step to modify that. The solution did not depend on one
tool doing everything.

Even within python, you can find a way to combine many modules to get a job
done rather than building it from scratch.

Which leads to the question of how you would design a log file if you knew
you needed to be able to search it efficiently for your particular
application.

I would offer something like HTML as an example. 

To some extent, the design of many elements looks like this:

<BODY>
...
</BODY>

The idea is you can write code that starts saving info when it reaches the
first tag, and when it reaches the second tag that ends it, you make a
decision. If you are in the right region, process it. If not, toss it and
just move on.

Of course, this scheme does not actually work for many of the tags in HTML.
Many tags allow an optional close but tolerate not having one. Some things
may be nested within each other.

But when it comes to log files, if some line says:

***

And that marks any error and you now wait till the end of the region to see
which error on a line like:

#ERRNO: 26

Then you can ask your code to ignore lines till it sees the first, marker.
Buffer any subsequent lines till you recognize the second marker and process
the buffer. Then go back to ignoring till ...

A real logfile may contain many sections for many purposes. They may even
have interleaved lines from different processes to the point where your
design may require all lines to start with something unique like the process
ID of the writer. This would make parsing such a file very hard, perhaps
requiring multiple passes sort of like described above. So sometimes a
better design is multiple log files that can be merged if needed.

Of course if you must use what exists, ....

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Mats Wichmann
Sent: Monday, December 24, 2018 10:15 AM
To: tutor at python.org
Subject: Re: [Tutor] look back comprehensively

On 12/24/18 2:14 AM, Alan Gauld via Tutor wrote:
> On 24/12/2018 05:25, Asokan Pichai wrote:
> 
>> That said, sometimes text processing at the shell can precede or even 
>> replace some of these. Of course that assumes Unix/Linux OS.

> In fact for most log file analysis I still use [ng]awk.
> Its hard to beat the simplicity of regex based event handling for 
> slicing text files.

Sure... there's nothing wrong with using "the appropriate tool for the job"
rather than making everything look like a Python problem.  There's Perl,
too... the ultimate log analysis toolkit. If only I could read what I wrote
a week later :)


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From mats at wichmann.us  Tue Dec 25 11:04:20 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Tue, 25 Dec 2018 09:04:20 -0700
Subject: [Tutor] look back comprehensively
In-Reply-To: <006c01d49beb$13d91280$3b8b3780$@verizon.net>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
 <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
 <pvq7tq$pco$1@blaine.gmane.org>
 <98e7c585-530b-8092-2f11-75a100e07a8f@wichmann.us>
 <006c01d49beb$13d91280$3b8b3780$@verizon.net>
Message-ID: <7906b0a4-2f0a-c2eb-6fae-cabb66a315e7@wichmann.us>

On 12/24/18 5:45 PM, Avi Gross wrote:


> As for the UNIX tools, one nice thing about them was using them in a
> pipeline where each step made some modification and often that merely
> allowed the next step to modify that. The solution did not depend on one
> tool doing everything.

I know we're wondering off topic here, but I miss the days when this
philosophy was more prevalent - "do one thing well" and be prepared to
pass your results on in a way that a different tool could potentially
consume, doing its one thing well, and so on if needed.  Of course
equivalents of those old UNIX tools are still with us, mostly thanks to
the GNU umbrella of projects, but so many current tools have grown so
many capabilities they no longer can interact with with other tools in
any sane way.  "pipes and filters" seems destined to be constrained to
the dustbin of tech history.

I'll shut up now...



From avigross at verizon.net  Tue Dec 25 19:00:40 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 25 Dec 2018 19:00:40 -0500
Subject: [Tutor] decomposing a problem
Message-ID: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>

[Long enough that some should neither read nor comment on.]

Mats raised an issue that I think does relate to how to tutor people in
python.

The issue is learning how to take a PROBLEM to solve that looks massive and
find ways to look at it as a series of steps where each step can be easily
solved using available tools and techniques OR can recursively be decomposed
into smaller parts that can. Many people learn to program without learning
first how to write down several levels of requirements that spell out how
each part of the overall result needs to look and finally how each part will
be developed and tested. I worked in organizations with a division of labor
to try to get this waterfall method in place. At times I would write
higher-level architecture documents followed by Systems Engineering
documents and Developer documents and Unit Test and System Test and even
Field Support. The goal was to move from abstract to concrete so that the
actual development was mainly writing fairly small functions, often used
multiple times,  and gluing them together.

I looked back at the kind of tools used in UNIX and realize how limited they
were relative to what is easily done in languages like python especially
given a huge tool set you can import. The support for passing the output of
one program to another made it easy to build pipelines. You can do that in
python too but rarely need to.

And I claim there are many easy ways to do things even better in python.

Many UNIX tools were simple filters. One would read a file or two and pass
through some of the lines, perhaps altered, to the standard output. The next
process in the pipeline would often do the same, with a twist and sometimes
new lines might even be added. The simple tools like cat and grep and sed
and so on loosely fit the filter analogy. They worked on a line at a time,
mostly. The more flexible tools like AWK and PERL are frankly more like
Python than the simple tools.

So if you had a similar task to do in python, is there really much
difference? I claim not so much.

Python has quite a few ways to do a filter. One simple one is a list
comprehension and its relatives. Other variations are the map and filter
functions and even reduce. Among other things, they can accept a list of
lines of text and apply changes to them or just keep a subset or even
calculate a result from them.

Let me be concrete. You have a set of lines to process. You want to find all
lines that pass through a gauntlet, perhaps with changes along the way.

So assume you read an entire file (all at once at THIS point) into a list of
lines.

stuff = open(...).readlines()

Condition 1 might be to keep only lines that had some word or pattern in
them. You might have used sed or grep in the UNIX shell to specify a fixed
string or pattern to search for.

So in python, what might you do? Since stuff is a list, something like a
list comprehension can handle many such needs. For a fixed string like
"this" you can do something like this.

stuff2 = [some_function(line) for line in stuff if some_condition(line)]

The condition might be: "this" in line
Or it might be a phrase than the line ends with something.
Or it might be a regular expression type search.
Or it might be the length is long enough or the number of words short
enough. Every such condition can be some of the same things used in a UNIX
pipeline or brand new ideas not available there like does a line translate
into a set of numbers that are all prime!

And, the function applied to what is kept can be to transform it to
uppercase, or replace it with something else looked up in a dictionary and
so on. You might even be able to apply multiple filters with each step.
Python allows phrases like line.strip().upper() and conditions like: this or
(that and not something_else)

The point is a single line like the list comprehension above may already do
what a pipeline of 8 simple commands in UNIX did, and more.

Some of the other things UNIX tools did might involve taking a line and
breaking it into chunks such as at a comma or tab or space and then keeping
just the third and fifth and eighth but in reverse order. We sometimes used
commands like cut or very brief AWK scripts to do that. Again, this can be
trivial to do in python. Built in to character strings are functions that
let you split a line like the above into a list of fields on a separator and
perhaps rearrange and even rejoin them. In the above list comprehension
method, if you are expecting eight regions that are comma separated

>>> line1 = "f1,f2,f3,f4,f5,f6,f7,f8"
>>> line2 = "g1,g2,g3,g4,g5,g6,g7,g8"
>>> lines=[line1, line2]
>>> splitsville = [line.split(',') for line in lines]
>>> splitsville
[['f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8'], ['g1', 'g2', 'g3', 'g4',
'g5', 'g6', 'g7', 'g8']]
>>> items8_5_3 = [(h8, h5, h3) for (h1,h2,h3,h4,h5,h6,h7,h8) in splitsville]
>>> items8_5_3
[('f8', 'f5', 'f3'), ('g8', 'g5', 'g3')]

Or if you want them back as character with an underscore between:

>>> items8_5_3 = ['_'.join([h8, h5, h3]) for (h1,h2,h3,h4,h5,h6,h7,h8) in
splitsville]
>>> items8_5_3
['f8_f5_f3', 'g8_g5_g3']

The point is that we have oodles of little tools we can combine to solve
bigger problems, sometimes in a big complicated mess and sometimes a simple
step at a time. Not all can be easily chained the same way but then we have
a bit more complex topic like generators and queues that can be chained
together in ways even more complex than UNIX pipelines. Each generator would
only be called to produce one result when another needs it.

And I assume it might be possible to make a series of methods that are
placed in an object that extends an object type like "list" that maintain an
internal representation like a list of strings and changes it in place just
like .sort() does.

So to apply a UNIX-style pipeline may be as simple as:

mystdout = mystdin("LIST OF STRINGS TO
INITIALIZE).method1(args).method2(args)....methodn(args)

The initializer will set the current "lines" and each method will loop on
the lines and replace them with the output it wants. Perhaps the initializer
or first method may actually read all lines from stdin. Perhaps the last
method will write to stdout. All methods will effectively follow in sequence
as they massage the data but will not actually run in parallel.

And you can even write a generic method that accepts any external function
designed to accept such a list of lines and return another to replace it.

My point to Mats is that the goal is to learn to divide and conquer a
problem. Using small and well defined methods that can fit together is
great. Many things in python can be made to fit and some need work. Dumb
example is that sorting something internally returns None and not the object
itself. So you cannot chain something like object.upper().sort() any
further. You may be able to chain this:

>>> list(reversed(sorted(lines))).pop()
'f1,f2,f3,f4,f5,f6,f7,f8'

Why the odd syntax? Because the developers of python in their wisdom may not
have chosen to enhance some methods to do things another way.

Object.sort() and Object.reverse() will change the internals and return
nothing. They are not designed to be piped. If there was a method that
performed a sort AND returned the object or performed a reverse and returned
the object, then we might see:

lines.sort(show=True).reverse(show=True).pop()

Or some other similar stratagem. Then we could write a fairly complex
sequence in a pipelined mode.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Mats Wichmann
Sent: Tuesday, December 25, 2018 11:04 AM
To: tutor at python.org
Subject: Re: [Tutor] look back comprehensively

On 12/24/18 5:45 PM, Avi Gross wrote:


> As for the UNIX tools, one nice thing about them was using them in a 
> pipeline where each step made some modification and often that merely 
> allowed the next step to modify that. The solution did not depend on 
> one tool doing everything.

I know we're wondering off topic here, but I miss the days when this
philosophy was more prevalent - "do one thing well" and be prepared to pass
your results on in a way that a different tool could potentially consume,
doing its one thing well, and so on if needed.  Of course equivalents of
those old UNIX tools are still with us, mostly thanks to the GNU umbrella of
projects, but so many current tools have grown so many capabilities they no
longer can interact with with other tools in any sane way.  "pipes and
filters" seems destined to be constrained to the dustbin of tech history.

I'll shut up now...



From alan.gauld at yahoo.co.uk  Tue Dec 25 20:06:04 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Wed, 26 Dec 2018 01:06:04 +0000
Subject: [Tutor] decomposing a problem
In-Reply-To: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
Message-ID: <pvuk1p$kig$1@blaine.gmane.org>

On 26/12/2018 00:00, Avi Gross wrote:

> great. Many things in python can be made to fit and some need work. Dumb
> example is that sorting something internally returns None and not the object
> itself. 

This is one of my few complaints about Python.
In Smalltalk the default return value from
any method is self. In Python it is None.

self allows chaining of methods, None does not.
Introducing features like reversed() and sorted()
partially addresses the issue but leads to
inconsistent and ugly syntax.

Smalltalk uses this technique so much it has
its own code layout idiom (Pythonised as
follows):

object
   .method1()
   .method2()
   .method3()
   ....
   .lastone()

We can do this with some methods but not all.
And of course methods that return a different
type of value require careful handling (eg. an
index() call in the middle of a set of list
operations means the subsequent methods are
being called on an int not a list - which if
handled correctly can be confusing and if
not handled correctly produces errors! (The
idiomatic way says don't chain with methods
not returning self!)

In practice I (and the Smalltalk community) don't
find that an issue in real world usage, but it
may have been why Guido chose not to do it that way.
But I still curse the decision every time I hit it!

But as I said, it's about the only thing in Python
I dislike... a small price to pay.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Tue Dec 25 20:29:59 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 Dec 2018 12:29:59 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <pvuk1p$kig$1@blaine.gmane.org>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <pvuk1p$kig$1@blaine.gmane.org>
Message-ID: <20181226012958.GT13061@ando.pearwood.info>

On Wed, Dec 26, 2018 at 01:06:04AM +0000, Alan Gauld via Tutor wrote:

> In Smalltalk the default return value from
> any method is self. In Python it is None.
> 
> self allows chaining of methods, None does not.


You might be interested in this simple recipe for retrofitting method 
chaining onto any class:

http://code.activestate.com/recipes/578770-method-chaining-or-cascading/


-- 
Steve

From cs at cskk.id.au  Tue Dec 25 20:44:04 2018
From: cs at cskk.id.au (Cameron Simpson)
Date: Wed, 26 Dec 2018 12:44:04 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <pvuk1p$kig$1@blaine.gmane.org>
References: <pvuk1p$kig$1@blaine.gmane.org>
Message-ID: <20181226014404.GA17671@cskk.homeip.net>

On 26Dec2018 01:06, Alan Gauld <alan.gauld at yahoo.co.uk> wrote:
>On 26/12/2018 00:00, Avi Gross wrote:
>> great. Many things in python can be made to fit and some need work. 
>> Dumb example is that sorting something internally returns None and 
>> not the object itself.
>
>This is one of my few complaints about Python.
>In Smalltalk the default return value from
>any method is self. In Python it is None.
>self allows chaining of methods, None does not.
[...]
>Smalltalk uses this technique so much it has
>its own code layout idiom (Pythonised as
>follows):
>
>object
>   .method1()
>   .method2()
>   .method3()
>   ....
>   .lastone()

While I see your point, the Python distinction is that methods returning 
values tend to return _independent_ values; the original object is not 
normally semanticly changed. As you know.

To take the builtin sorted() example, let us soppose object is a 
collection, such as a list. I would not want:

  object.sort()

to return the list because that method has a side effect on object.

By contract, I'd be happy with a:

  object.sorted()

method returning a new list because it hasn't changes object, and it 
returns a nice chaining capable object for continued use.

But that way lies a suite of doubled methods for most classes: one to 
apply some operation to an object, modifying it, and its partner to 
produce a new object (normally of the same type) being a copy of the 
first object with the operation applied.

To me it is the side effect on the original object which weighs against 
modification methods returning self.

Here's a shiny counter example for chaining.

    thread1:
      print(object.sorted())
    thread2:
      print(object.sorted(reverse=True))

The above employs composable methods. And they conflict. When methods 
return a copy the above operation is, loosely speaking, safe:

    thread1:
      print(sorted(object))
    thread2:
      print(sorted(object,reverse=True))

Cheers,
Cameron Simpson <cs at cskk.id.au>

From avigross at verizon.net  Tue Dec 25 22:25:50 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 25 Dec 2018 22:25:50 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <pvuk1p$kig$1@blaine.gmane.org>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <pvuk1p$kig$1@blaine.gmane.org>
Message-ID: <004701d49cca$b33419d0$199c4d70$@verizon.net>

Alan,

Your thoughts were helpful and gave me a hint.

Just an idea. What if you sub-classed an object type like list with a name
like chainable_list?

For most things it would be left alone. But if you isolated specific named
methods like sort() and reverse() you could over-ride them with the same
name or a new name.

If you override the function, you need to call list.sort() with whatever
arguments you had passed and then return this. If you choose a new name,
call this.sort() and then return this.

I tried it and it seems to work fine when I use a new name:

"""Module to create a version of list that is more chainable"""

class chainable_list(list):
    """Same as list but sort() can now be chained"""
    def chainsort(this, *args, **kwargs):
        this.sort(*args, **kwargs)
        return this

Here it is on a list of ints:

>>> testink = chainable_list([3,5,1,7])
>>> testink
[3, 5, 1, 7]
>>> testink.chainsort()
[1, 3, 5, 7]
>>> testink.chainsort(reverse=True)
[7, 5, 3, 1]

Here it is on a list of strings that sort differently unless coerced back
into an int to show keyword arguments are passed:

>>> testink = chainable_list(["3","15","1","7"])
>>> testink.chainsort()
['1', '15', '3', '7']
>>> testink.chainsort(reverse=True)
['7', '3', '15', '1']
>>> testink.chainsort(key=int,reverse=True)
['15', '7', '3', '1']

I then tested the second method using the same name but asking the original
list sort to do things:

"""Module to create a version of list that is more chainable"""

class chainable_list(list):
    """Same as list but sort() can now be chained"""
    def sort(this, *args, **kwargs):
        list.sort(this, *args, **kwargs)
        return this

>>> testink = chainable_list(["3","15","1","7"])
>>> testink.sort()
['1', '15', '3', '7']
>>> testink.sort().sort(reverse=true)
Traceback (most recent call last):
  File "<pyshell#18>", line 1, in <module>
    testink.sort().sort(reverse=true)
NameError: name 'true' is not defined
>>> testink.sort().sort(reverse=True)
['7', '3', '15', '1']
>>> testink.sort().sort(reverse=True).sort(key=int)
['1', '3', '7', '15']

Again, it works fine. So if someone did something similar to many of the
methods that now return None, you could use the new class when needed.

This seems too simple so it must have been done. Obviously not in the
standard distribution but perhaps elsewhere. And, no, I do not expect a
method like pop() to suddenly return the list with a member dropped but it
would be nice to fix some like this one:

>>> testink.remove('7')
>>> testink
['1', '3', '15']

Meanwhile, I hear Beethoven is decomp..., well never mind! It was probably
Liszt!

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Alan Gauld via Tutor
Sent: Tuesday, December 25, 2018 8:06 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On 26/12/2018 00:00, Avi Gross wrote:

> great. Many things in python can be made to fit and some need work. 
> Dumb example is that sorting something internally returns None and not 
> the object itself.

This is one of my few complaints about Python.
In Smalltalk the default return value from any method is self. In Python it
is None.

self allows chaining of methods, None does not.
Introducing features like reversed() and sorted() partially addresses the
issue but leads to inconsistent and ugly syntax.

Smalltalk uses this technique so much it has its own code layout idiom
(Pythonised as
follows):

object
   .method1()
   .method2()
   .method3()
   ....
   .lastone()

We can do this with some methods but not all.
And of course methods that return a different type of value require careful
handling (eg. an
index() call in the middle of a set of list operations means the subsequent
methods are being called on an int not a list - which if handled correctly
can be confusing and if not handled correctly produces errors! (The
idiomatic way says don't chain with methods not returning self!)

In practice I (and the Smalltalk community) don't find that an issue in real
world usage, but it may have been why Guido chose not to do it that way.
But I still curse the decision every time I hit it!

But as I said, it's about the only thing in Python I dislike... a small
price to pay.

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From steve at pearwood.info  Tue Dec 25 23:38:50 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 Dec 2018 15:38:50 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <004701d49cca$b33419d0$199c4d70$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <pvuk1p$kig$1@blaine.gmane.org> <004701d49cca$b33419d0$199c4d70$@verizon.net>
Message-ID: <20181226043849.GU13061@ando.pearwood.info>

On Tue, Dec 25, 2018 at 10:25:50PM -0500, Avi Gross wrote:

> class chainable_list(list):
>     """Same as list but sort() can now be chained"""
>     def chainsort(this, *args, **kwargs):
>         this.sort(*args, **kwargs)
>         return this

In Python, it is traditional to use "self" rather than "this" as the 
instance parameter.

Using "this" is not an error, but you can expect a lot of strange looks. 
Like a Scotsman in a kilt wandering down the middle of Main Street, 
Pleasantville USA.



-- 
Steve

From avigross at verizon.net  Tue Dec 25 23:56:21 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 25 Dec 2018 23:56:21 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
Message-ID: <005301d49cd7$585c1b40$091451c0$@verizon.net>

Mike,

Excellent advice.

I find that many people are fairly uncomfortable with abstraction and tend to resist a pure top down approach by diving to any solutions they may envision. For example, if you say things like create a data structure that can hold as many kinds of information as will be needed. The data should be able to be viewed in several ways and adding a new item should be fast even if the number of items grows large ...

Some will have stopped reading (or creating) and will jump to deciding then need a dictionary. Others may want a deque. Some may insist they need a new class. 

But wait, if you continue reading or designing, it may be clear that some choices are not optimal. Heck, it may turn out some design elements are contradictory. As someone asked on another python list, is there a better way to get a random key for a dictionary. Well, not easily without expanding all keys into a list of perhaps huge length. Followed by a search of much of that list to get the nth index. So maybe a plain dictionary does not make that easy or efficient so do you give up that need or use some other data structure that makes that fast? Perhaps you need a hybrid data structure. One weird idea is to use the dictionary but every time you generate a new key/value pair you also store a second pair that looks like "findkey666": key so that a random key of the first kind can be found in constant time by picking a random number up to half the number of items, concatenate it to "findkey" and look up the value which is a key.

When you try to work bottom up with students, some see no point as they are missing the big picture. I used to work during graduate school writing PASCAL code for a company making flexible manufacturing systems and my job often was to read a man page describing some function that did something minor. I often had no clue why it was needed or where it would be used? I was sometimes told it had to FIT into a certain amount of memory because of the overlay technique used and if it was compiled to something larger, was asked to break the function down into multiple functions that were called alternately .... Sometimes an entire section had to be redesigned because it had to fit into the same footprint as another. That was the limit of the big picture. A shadow!

What I found works for me is a combination. I mean teaching. You give them just enough of the top-down view for motivation. Then you say that we need to figure out what kinds of things might be needed to support the functionality. This includes modules to import as well as objects or functions to build. But that too can be hard unless you move back into the middle and explain a bit about the subunit you are building so you know what kind of support it needs closer to the bottom.

I admit that my personal style is the wrong one for most people. I do top down and bottom up simultaneously as well as jump into the middle to see both ways to try to make sure the parts will meet fairly seamlessly. Does not always work.

How often have we seen a project where some function is designed with three arguments. Much later, you find out some uses of the function only have and need two but some may have additional arguments, perhaps to pass along to yet another function the second will conditionally invoke? It may turn out that the bottom up approach starting from one corner assumed that the function would easily meet multiple needs when the needs elsewhere are not identical enough. If they keep demanding one function to master all, you can end up with fairly awful spaghetti code. Of course python is not a compiled language like C/C++ and PASCAL and many others were. It is often fairly easy in python to have a variable number of arguments or for the same function to do something reasonable with multiple types and do something reasonable for each.

One thing I warn people about is mission creep. When asked to do something, try not to add lots of nice features at least until you have developed and tested the main event. I have seen many projects that did feel the need to add every feature they could imagine as there remained keys on the keyboard that did not yet invoke some command, even if no customer ever asked for it or would ever use it. Amazing how often these projects took too long and came to market too late to catch on ...

Some of the people asking questions here do not even tell us much about what is needed, let alone their initial design plan. It can take multiple interactions back and forth and I wonder how many give up long before as they just want an ANSWER. 

In case you wonder, I am reliably told the answer to life, the universe and everything  is 2*21.

-----Original Message-----
From: Mike Mossey <mike at pathtomathclarity.com> 
Sent: Tuesday, December 25, 2018 9:49 PM
To: Avi Gross <avigross at verizon.net>
Subject: Re: [Tutor] decomposing a problem


> On Dec 25, 2018, at 4:00 PM, Avi Gross <avigross at verizon.net> wrote:
> 
> [Long enough that some should neither read nor comment on.]
> 
> Mats raised an issue that I think does relate to how to tutor people 
> in python.
> 
> The issue is learning how to take a PROBLEM to solve that looks 
> massive and find ways to look at it as a series of steps where each 
> step can be easily solved using available tools and techniques OR can 
> recursively be decomposed into smaller parts that can. Many people 
> learn to program without learning first how to write down several 
> levels of requirements that spell out how each part of the overall 
> result needs to look and finally how each part will be developed and 
> tested. I worked in organizations with a division of labor to try to 
> get this waterfall method in place. At times I would write 
> higher-level architecture documents followed by Systems Engineering 
> documents and Developer documents and Unit Test and System Test and 
> even Field Support. The goal was to move from abstract to concrete so 
> that the actual development was mainly writing fairly small functions, often used multiple times,  and gluing them together.
> 
> 

It?s an interesting topic, getting students to decompose problems. 

As a tutor of programming (I use mainly Python), I try to get students to describe a problem at a high level with pseudocode and/or English before attacking the details. 

Both my programming and math students tend to overcomplicate things. They can benefit greatly from using simpler ideas. 

When they write an English description of a problem and their proposed solution, I try to get them to use fewer words and to use big fuzzy concepts. I try to get them to describe a problem in terms like ?we take 2 lists, combine them, and filter the results.? Their normal instinct is to describe a problem like that with several sentences or even several paragraphs, meaning they are bogged down in the details.

Then I have them break down the problem further and gradually write English documentation or pseudocode with more details. 

We also use bottom-up design sometimes, like taking just one piece of the problem and solving it as a learning experience. With bottom-up, again I try to simplify it for them by having them not worry about the big picture. Don?t try to relate everything to the whole problem but just take details one at a time, and assume that you are writing a little test algorithm. We can discover the relationship to the whole problem over time, and we might rewrite or even abandon our experiment.

Mike





From avigross at verizon.net  Wed Dec 26 00:10:00 2018
From: avigross at verizon.net (Avi Gross)
Date: Wed, 26 Dec 2018 00:10:00 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181226043849.GU13061@ando.pearwood.info>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <pvuk1p$kig$1@blaine.gmane.org> <004701d49cca$b33419d0$199c4d70$@verizon.net>
 <20181226043849.GU13061@ando.pearwood.info>
Message-ID: <005401d49cd9$40cbc780$c2635680$@verizon.net>

[REAL SUBJECT: What's this?]

Steven,

I am afraid you are right. I was not selfish enough about this. I have done
object-oriented programming in many other languages and I am afraid today it
showed. Think C++ or Java. Part of me continues to think in every language I
ever used, including human languages. So since the name of this variable is
a suggestion, it was not enforced by the interpreter and I was not reminded.

Be happy I even used an English word and not  something like idempotent or
eponymous
.
P.S. just to confuse the issue, some in JavaScript confusingly use both this
and self near each other.
P.P.S. Please pardon my puns, especially the ones you did not notice.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Tuesday, December 25, 2018 11:39 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Tue, Dec 25, 2018 at 10:25:50PM -0500, Avi Gross wrote:

> class chainable_list(list):
>     """Same as list but sort() can now be chained"""
>     def chainsort(this, *args, **kwargs):
>         this.sort(*args, **kwargs)
>         return this

In Python, it is traditional to use "self" rather than "this" as the
instance parameter.

Using "this" is not an error, but you can expect a lot of strange looks. 
Like a Scotsman in a kilt wandering down the middle of Main Street,
Pleasantville USA.



-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From steve at pearwood.info  Wed Dec 26 00:45:13 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 Dec 2018 16:45:13 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <005301d49cd7$585c1b40$091451c0$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
Message-ID: <20181226054513.GW13061@ando.pearwood.info>

On Tue, Dec 25, 2018 at 11:56:21PM -0500, Avi Gross wrote:

> I find that many people are fairly uncomfortable with abstraction and 
> tend to resist a pure top down approach by diving to any solutions 
> they may envision.

https://blog.codinghorror.com/it-came-from-planet-architecture/

> As someone asked on another python list, 
> is there a better way to get a random key for a dictionary. Well, not 
> easily without expanding all keys into a list of perhaps huge length. 

Define "better".

What do you value? Time, space, simplicity or something else?

One of the most harmful things to value is "cleverness" for its own 
sake. Some people tend to value a "clever" solution even when it wastes 
time, space and is over complex and therefore hard to maintain or debug.

Even when the practical response to the "clever" solution is "YAGNI".

What counts as "huge"? To me, picking a random key from a list of 100 
keys is "huge". Copy out 100 keys to a list by hand and then pick one? 
What a PITA that would be.

But to your computer, chances are that ten million keys is "small". One 
hundred million might be pushing "largish". A billion, or perhaps ten 
billion, could be "large". Fifty, a hundred, maybe even a thousand 
billion (a trillion) would be "huge".

Unless you expect to be handling at least a billion keys, there's 
probably no justification for anything more complex than:

    random.choose(list(dict.keys())

Chances are that it will be faster *and* use less memory than any clever 
solution you come up with -- and even if it does use more memory, it 
uses it for a few milliseconds, only when needed, unlike a more complex 
solution that inflates the size of the data structure all the time, 
whether you need it or not.

Of course there may be use-cases where we really do need a more complex, 
clever solution, and are willing to trade off space for time (or 
sometimes time for space). But chances are YAGNI.


> Followed by a search of much of that list to get the nth index.

That's incorrect. Despite the name, Python lists aren't linked lists[1] 
where you have to traverse N items to get to the Nth item. They're 
arrays, where indexing requires constant time.


[...]
> If they keep demanding one function to master all, you can end up with 
> fairly awful spaghetti code.

https://en.wikipedia.org/wiki/God_object




[1] Technically speaking, this is not a requirement of the language, 
only a "quality of implementation" question. A Python interpreter could 
offer built-in lists using linked lists under the hood, with O(N) 
indexing. But all the major implementations -- CPython, Stackless, PyPy, 
Jython, IronPython, Cython, Nuitka, even (I think) MicroPython -- use 
arrays as the list implementation. Given how simple arrays are, I think 
it is fair to assume that any reasonable Python interpreter will do the 
same.



-- 
Steve

From avigross at verizon.net  Tue Dec 25 23:05:57 2018
From: avigross at verizon.net (Avi Gross)
Date: Tue, 25 Dec 2018 23:05:57 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181226014404.GA17671@cskk.homeip.net>
References: <pvuk1p$kig$1@blaine.gmane.org>
 <20181226014404.GA17671@cskk.homeip.net>
Message-ID: <004b01d49cd0$4e01f9f0$ea05edd0$@verizon.net>

Steven showed a more abstract solution than the one I tried but Cameron is
making some good points on whether it might not be a great idea to chain
some side-effect operations.

I have seen languages where everything seems to be immutable. Python does
this in places like with tuples. The idea is that every change results in a
new copy of things, sort of. But in such a world, there is no real concept
of making a change internally. The name pointing to the object is all that
remains the same. The underlying object is a new copy if any changes are
needed.

So if I made a deep copy of "this" and returned that, then what would happen
to the original? Would it still be changed for anything else that cared?

When I am doing a pipeline, I really may not care about the original. At
every point I care about propagating my changes.

"  Hello World  ".lower() becomes "  hello world  " at that point. If I then
add a .rstrip() and a .lstrip() each produces a new string without some
whitespace. I don't care if the original is affected. If I then add a
request to get just part of the string, again. The original is intact. The
only time it is changed is when I assign the result back to the original
variable and even then, anything else also pointing to it is unchanged.

Python has plenty of operators of multiple kinds. Some return a shallow copy
and some a deep copy and some an altered copy and some an altered original
and some change NOTHING whatsoever. Some make subtle changes in the parent
class for example which may later impact the child but are not stored in the
child.

And efficiency is also a concern. Returning something when not needed is not
efficient and can result in having to do something to suppress.

Brief digression to make the point. R defaults to returning the last
evaluated item in a function with no explicit return statement. Python
returns None. So sometimes a line typed at the console generates a print of
the returned value. In some cases, the automatic print generates a graph, as
in a ggplot object. So some functions take care to mark the returned value
as invisible. It is there if you ask for it by saving the result to a
variable but does not otherwise print by default.

So I can easily see why the design of some features is to not do more than
you have to. If the goal is to change the current object, you can simply
show the darn object afterwards, right? Well, no, not easy when using a
pipeline method. 

Still, how much would it hurt to allow a keyword option on those methods
people WANT to call in a pipeline when it makes sense. Why not let me say
object.sort(key=int, reverse=True,displayCopy=True) or something like that.
If that messes up threads, fine. Don't use them there.

The side effect issue is not to be taken lightly. I believe that may be
similar to why there is no ++ operator. But they are adding := which
arguably is also a side effect.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Cameron Simpson
Sent: Tuesday, December 25, 2018 8:44 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On 26Dec2018 01:06, Alan Gauld <alan.gauld at yahoo.co.uk> wrote:
>On 26/12/2018 00:00, Avi Gross wrote:
>> great. Many things in python can be made to fit and some need work. 
>> Dumb example is that sorting something internally returns None and 
>> not the object itself.
>
>This is one of my few complaints about Python.
>In Smalltalk the default return value from any method is self. In 
>Python it is None.
>self allows chaining of methods, None does not.
[...]
>Smalltalk uses this technique so much it has its own code layout idiom 
>(Pythonised as
>follows):
>
>object
>   .method1()
>   .method2()
>   .method3()
>   ....
>   .lastone()

While I see your point, the Python distinction is that methods returning
values tend to return _independent_ values; the original object is not
normally semanticly changed. As you know.

To take the builtin sorted() example, let us soppose object is a collection,
such as a list. I would not want:

  object.sort()

to return the list because that method has a side effect on object.

By contract, I'd be happy with a:

  object.sorted()

method returning a new list because it hasn't changes object, and it returns
a nice chaining capable object for continued use.

But that way lies a suite of doubled methods for most classes: one to apply
some operation to an object, modifying it, and its partner to produce a new
object (normally of the same type) being a copy of the first object with the
operation applied.

To me it is the side effect on the original object which weighs against
modification methods returning self.

Here's a shiny counter example for chaining.

    thread1:
      print(object.sorted())
    thread2:
      print(object.sorted(reverse=True))

The above employs composable methods. And they conflict. When methods return
a copy the above operation is, loosely speaking, safe:

    thread1:
      print(sorted(object))
    thread2:
      print(sorted(object,reverse=True))

Cheers,
Cameron Simpson <cs at cskk.id.au>
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Wed Dec 26 11:02:07 2018
From: avigross at verizon.net (Avi Gross)
Date: Wed, 26 Dec 2018 11:02:07 -0500
Subject: [Tutor] decomposing a problem
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info> 
Message-ID: <004d01d49d34$5a36d470$0ea47d50$@verizon.net>

[REAL SUBJECT: Beats me]
Steven,

I often find that I try to make a main point ad people then focus on
something else, like an example.

So, do we agree on the main point that choosing a specific data structure or
algorithm (or even computer language) too soon can lead to problems that can
be avoided if we first map out the problem and understand it better?

So when someone asks a question here like how do I get a data structure to
do this, often the answer might include asking if that is the right data
structure for the job. And yes, there may be no RIGHT as compared to degrees
of fit that include subjective components.

The rest of your post really is a rehash of the periodic efficiency
discussions.

I do not concede that efficiency can be ignored because computers are fast.
I do concede that it is often not worth the effort or that you can
inadvertently make things worse and there are tradeoffs.

Let me be specific. The side topic was asking how to get a random key from
an existing dictionary. If you do this ONCE, it may be no big deal to make a
list of all keys, index it by a random number, and move on. I did supply a
solution that might(or might not) run faster by using a generator to get one
item at a time and stopping when found. Less space but not sure if less
time.

But what I often need to do is to segment lots of data into two piles. One
is for training purposes using some machine learning algorithm and the
remainder is to be used for verifications. The choice must be random or the
entire project may become meaningless. So if your data structure was a
dictionary with key names promptly abandoned, you cannot just call pop()
umpteen times to get supposedly random results as they may come in a very
specific order. If you want to have 75% of the data in the training section,
and 25% reserved, and you have millions of records, what is a good way to
go? Worse, some algorithms like this build forests and take numerous random
samples for each tree as they go along.

So you can wonder if the dictionary is the best way to go if there is no
faster way to get random records. If you are told that while doing this you
also need to allow simultaneous access to the data structure so new
key/value pairs are being inserted or modified or removed, that may make you
wonder if other choices would perhaps be best overall.

One quite obvious solution is to not create a full list each and every time
you want one random number. There are many ways to arrange either to have
the list of keys standing around all the time or to be created and saved on
the first need and kept around as long as needed. A simple generator that
stays around can do that. So asking for 2 million keys one after another can
slow things down if you do it from scratch each time even as garbage
collection works overtime. But other solutions might simply make a copy of
the dictionary as a data frame or whatever it takes. There may be merits to
each solution as well as drawbacks. In any case, spending too much time on
it may not be worthwhile. And as is often noted, using something already in
existence and often already highly optimized and debugged, is often a
deciding factor. 

I believe I mentioned that the implementation that creates a list from a
generator may be faster than your own algorithm that may appear to be more
efficient in some way. My earlier mention is an example. I agree that the
implementation of lists usually results in an indexed search for the nth
item and is rapid. I was referring to algorithms that generate one item at a
time and find it on average halfway through and in the worst case end up
reading the entire list. In theory they can be faster than creating the
entire list at once but in practice, maybe not, and in any case, maybe not
the best place to optimize.

I tend to avoid extremes so let me make one thing very clear.

I neither like it when people make snap decisions about how to do something
with software and refuse to reconsider when reality intervenes NOR do I like
it when they need endless (often fairly meaningless) steps to plan it all
out so they may never actually start it. I like a prototyping approach with
flexibility. Do some planning, try some things out, perhaps with a set of
tools that lets you put a skeleton of the idea together to see if it fits.
If not, adjust. When it looks reasonable, flesh it out including lots more
rigor and bulletproofing.

For this forum, the questions that come in range from people who have no
idea how to even start a project to those that are well along and get stuck
at some point. The former may need help in making a general plan of attack.
The latter may just need someone to point to an error or design flaw. One
size does not fit all.

And, as Steven points out, there are tradeoffs. For a student exercise
intended to learn how to use a feature of the language, it is not that
important that the function they write checks to make sure they were called
with sensible arguments like a positive number when asked how many ice cream
scoops they want. For production code, much more may be required. And
extremely clever solutions being offered that may not even be understood,
often defeat the purpose.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Wednesday, December 26, 2018 12:45 AM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Tue, Dec 25, 2018 at 11:56:21PM -0500, Avi Gross wrote:

> I find that many people are fairly uncomfortable with abstraction and 
> tend to resist a pure top down approach by diving to any solutions 
> they may envision.

https://blog.codinghorror.com/it-came-from-planet-architecture/

> As someone asked on another python list, is there a better way to get 
> a random key for a dictionary. Well, not easily without expanding all 
> keys into a list of perhaps huge length.

Define "better".

What do you value? Time, space, simplicity or something else?

One of the most harmful things to value is "cleverness" for its own sake.
Some people tend to value a "clever" solution even when it wastes time,
space and is over complex and therefore hard to maintain or debug.

Even when the practical response to the "clever" solution is "YAGNI".

What counts as "huge"? To me, picking a random key from a list of 100 keys
is "huge". Copy out 100 keys to a list by hand and then pick one? 
What a PITA that would be.

But to your computer, chances are that ten million keys is "small". One
hundred million might be pushing "largish". A billion, or perhaps ten
billion, could be "large". Fifty, a hundred, maybe even a thousand billion
(a trillion) would be "huge".

Unless you expect to be handling at least a billion keys, there's probably
no justification for anything more complex than:

    random.choose(list(dict.keys())

Chances are that it will be faster *and* use less memory than any clever
solution you come up with -- and even if it does use more memory, it uses it
for a few milliseconds, only when needed, unlike a more complex solution
that inflates the size of the data structure all the time, whether you need
it or not.

Of course there may be use-cases where we really do need a more complex,
clever solution, and are willing to trade off space for time (or sometimes
time for space). But chances are YAGNI.


> Followed by a search of much of that list to get the nth index.

That's incorrect. Despite the name, Python lists aren't linked lists[1]
where you have to traverse N items to get to the Nth item. They're arrays,
where indexing requires constant time.


[...]
> If they keep demanding one function to master all, you can end up with 
> fairly awful spaghetti code.

https://en.wikipedia.org/wiki/God_object




[1] Technically speaking, this is not a requirement of the language, only a
"quality of implementation" question. A Python interpreter could offer
built-in lists using linked lists under the hood, with O(N) indexing. But
all the major implementations -- CPython, Stackless, PyPy, Jython,
IronPython, Cython, Nuitka, even (I think) MicroPython -- use arrays as the
list implementation. Given how simple arrays are, I think it is fair to
assume that any reasonable Python interpreter will do the same.



--
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From asad.hasan2004 at gmail.com  Thu Dec 27 10:10:12 2018
From: asad.hasan2004 at gmail.com (Asad)
Date: Thu, 27 Dec 2018 20:40:12 +0530
Subject: [Tutor] Re Module
Message-ID: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>

Hi All ,

          I trying find a solution for my script , I have two files :

file1 - I need a search a error say x if the error matches

Look for the same error x in other file 2

Here is the code :
I have 10 different patterns therefore I used list comprehension and
compiling the pattern so I loop over and find the exact pattern matching

re_comp1 = [re.compile(pattern) for pattern in str1]

for pat in re_comp1:
    if pat.search(st,re.IGNORECASE):
        x = pat.pattern
print x                ===> here it gives the expected output it correct
match
print type(x)

<type 'unicode'>

if re.search('x', line, re.IGNORECASE) is not None:  ===> Gives a wrong
match
      print line

Instead if I use :

if re.search(x, line, re.IGNORECASE) is not None: then no match occurs
      print line

Please advice where I going wrong or what can be done to make it better .

Thanks,


-- 
Asad Hasan
+91 9582111698

From breamoreboy at gmail.com  Thu Dec 27 14:01:33 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Thu, 27 Dec 2018 19:01:33 +0000
Subject: [Tutor] look back comprehensively
In-Reply-To: <006c01d49beb$13d91280$3b8b3780$@verizon.net>
References: <000001d47bd6$d04752e0$70d5f8a0$@verizon.net>
 <20181114220131.GI4071@ando.pearwood.info>
 <a0e94465-c4fa-9d95-0b8b-29b2422cc2e4@wichmann.us>
 <CAB3TsCj7xaB7VATZhzjoqLUsdqGeuU0GjzCF6dTnn3u4KoaLLw@mail.gmail.com>
 <pvq7tq$pco$1@blaine.gmane.org>
 <98e7c585-530b-8092-2f11-75a100e07a8f@wichmann.us>
 <006c01d49beb$13d91280$3b8b3780$@verizon.net>
Message-ID: <q037ea$dkg$1@blaine.gmane.org>

On 25/12/2018 00:45, Avi Gross wrote:


Please go away as you are so boring :-(

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From breamoreboy at gmail.com  Thu Dec 27 14:03:18 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Thu, 27 Dec 2018 19:03:18 +0000
Subject: [Tutor] decomposing a problem
In-Reply-To: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
Message-ID: <q037hj$dkg$2@blaine.gmane.org>

On 26/12/2018 00:00, Avi Gross wrote:
> [Long enough that some should neither read nor comment on.]
> 

PLEASE GO AWAY YOU ARE REALLY IRRITATING.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From steve at pearwood.info  Thu Dec 27 17:04:51 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 28 Dec 2018 09:04:51 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <q037hj$dkg$2@blaine.gmane.org>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <q037hj$dkg$2@blaine.gmane.org>
Message-ID: <20181227220451.GC13061@ando.pearwood.info>

On Thu, Dec 27, 2018 at 07:03:18PM +0000, Mark Lawrence wrote:
> On 26/12/2018 00:00, Avi Gross wrote:
> >[Long enough that some should neither read nor comment on.]
> >
> 
> PLEASE GO AWAY YOU ARE REALLY IRRITATING.

People in glass houses...

Mark, you're not the arbiter of who is allowed to post here. You are 
being obnoxious. Please settle down and perhaps chill a bit. If you 
don't want to read Avi's posts, you know how to hit delete in your mail 
reader don't you?


-- 
Steve

From steve at pearwood.info  Thu Dec 27 17:38:13 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 28 Dec 2018 09:38:13 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
Message-ID: <20181227223813.GD13061@ando.pearwood.info>

On Wed, Dec 26, 2018 at 11:02:07AM -0500, Avi Gross wrote:

> I often find that I try to make a main point ad people then focus on
> something else, like an example.

I can't speak for others, but for me, that could be because of a number 
of reasons:

- I agree with what you say, but don't feel like adding "I agree!!!!" 
after each paragraph of yours;

- I disagree, but can't be bothered arguing;

- I don't understand the point you intend to make, so just move on.

But when you make an obvious error, I tend to respond. This is supposed 
to be a list for teaching people to use Python better, after all.


> So, do we agree on the main point that choosing a specific data structure or
> algorithm (or even computer language) too soon can lead to problems that can
> be avoided if we first map out the problem and understand it better?

Sure, why not? That's vague and generic enough that it has to be true.

But if its meant as advice, you don't really offer anything concrete. 
How does one decide what is "too soon"? How does one avoid design 
paralysis?


> I do not concede that efficiency can be ignored because computers are fast.

That's good, but I'm not sure why you think it is relevant as I never 
suggested that efficiency can be ignored. Only that what people *guess* 
is "lots of data" and what actually *is* lots of data may not be the 
same thing.


> I do concede that it is often not worth the effort or that you can
> inadvertently make things worse and there are tradeoffs.

Okay.


> Let me be specific. The side topic was asking how to get a random key from
> an existing dictionary. If you do this ONCE, it may be no big deal to make a
> list of all keys, index it by a random number, and move on. I did supply a
> solution that might(or might not) run faster by using a generator to get one
> item at a time and stopping when found. Less space but not sure if less
> time.

Why don't you try it and find out?


> But what I often need to do is to segment lots of data into two piles. One
> is for training purposes using some machine learning algorithm and the
> remainder is to be used for verifications. The choice must be random or the
> entire project may become meaningless. So if your data structure was a
> dictionary with key names promptly abandoned, you cannot just call pop()
> umpteen times to get supposedly random results as they may come in a very
> specific order.

Fortunately I never suggested doing that.


> If you want to have 75% of the data in the training section,
> and 25% reserved, and you have millions of records, what is a good way to
> go? 

The obvious solution:

keys = list(mydict.keys())
random.shuffle(keys)
index = len(keys)*3//4
training_data = keys[:index]
reserved = keys[index:]

Now you have the keys split into training data and reserved data. To 
extract the value, you can just call mydict[some_key]. If you prefer, 
you can generate two distinct dicts:

training_data = {key: mydict[key] for key in training_data}

and similarly for the reserved data, and then mydict becomes redundant 
and you are free to delete it (or just ignore it).

Anything more complex than this solution should not even be attempted 
until you have tried the simple, obvious solution and discovered that it 
isn't satisfactory.

Keep it simple. Try the simplest thing that works first, and don't add 
complexity until you know that you need it.

By the way, your comments would be more credible if you had actual 
working code that demonstrates your point, rather than making vague 
comments that something "may" be faster. Sure, anything "may" be faster. 
We can say that about literally anything. Walking to Alaska from the 
southernmost tip of Chile while dragging a grand piano behind you "may" 
be faster than flying, but probably isn't. Unless you have actual code 
backing up your assertions, they're pretty meaningless.

And the advantage of working code is that people might actually learn 
some Python too.



-- 
Steve

From avigross at verizon.net  Thu Dec 27 18:32:51 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 27 Dec 2018 18:32:51 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181227223813.GD13061@ando.pearwood.info>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
Message-ID: <008f01d49e3c$7c9e9120$75dbb360$@verizon.net>

[Mark Lawrence please press DELETE now in case the rest of this message is
all about you.]
[[If that is not working, if on Windows, try Control-ALT-DELETE as that will
really get rid of my message.]]

Back to replying to Steven,

Of course I want to be corrected when wrong.

I think everyone here knows I tend to be quite expansive in my thoughts and
sometimes to the point where they suggest I am free-associating. I am trying
to get to the point faster and stay there.

So if what I write is not wrong as a general point and you want to bring up
every exception, fine. I reserve the right not to follow you there,
especially not on the forum. I may continue a discussion with you in
private, of course.

I often have a problem in real life (not talking about you, let alone
whoever Mark is) where I think I said something clearly by using phrases
like "if" and find the other person simply acts as if I had left that out.
You know, we can go to the park IF it is not raining tomorrow. Reply is to
tell me the weather report says it will rain so why am I suggesting we go to
the park. Duh. I was not aware of the weather report directly BUT clearly
suggested it was a consideration we should look at before deciding. 

Now a more obvious error should be pointed out. EXAMPLE, I am driving to
Pennsylvania this weekend not far from a National Park and will have some
hours to kill. I suggested we might visit Valley Forge National Historic
Park and did not say only if it was open. Well, in the U.S. we happen to
have the very real possibility the Park will be closed due to it being
deemed optional during a so-called Government Shutdown so such a reply IS
reasonable. I did not consider that and stand corrected.

But Chris, you point out I reacted similarly to what you said. Indeed, you
said that sometimes we don't need to focus on efficiency as compared to
saying we should always ignore it or something like that. I think we
actually are in relative agreement in how we might approach a problem like
this. We might try to solve it in a reasonable way first and not worry at
first about efficiency especially now that some equipment runs so fast and
with so much memory that results appear faster than we can get to them. But,
with experience, and need, we may fine tune code that is causing issues. As
I have mentioned, I have applications that regularly need huge samples taken
at random so a list of millions being created millions of times and the
above being done thousands of times, adds up. Many cheaper methods might
then be considered including, especially, just switching to a better data
structure ONCE.

I will stop this message here as I suspect Mark is still reading and fuming.
Note, I do not intend to mention Mark again in future messages. I do not
actually want to annoy him and wish he would live and let live.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Thursday, December 27, 2018 5:38 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Wed, Dec 26, 2018 at 11:02:07AM -0500, Avi Gross wrote:

> I often find that I try to make a main point ad people then focus on 
> something else, like an example.

I can't speak for others, but for me, that could be because of a number of
reasons:

- I agree with what you say, but don't feel like adding "I agree!!!!" 
after each paragraph of yours;

- I disagree, but can't be bothered arguing;

- I don't understand the point you intend to make, so just move on.

But when you make an obvious error, I tend to respond. This is supposed to
be a list for teaching people to use Python better, after all.


> So, do we agree on the main point that choosing a specific data structure
or
> algorithm (or even computer language) too soon can lead to problems that
can
> be avoided if we first map out the problem and understand it better?

Sure, why not? That's vague and generic enough that it has to be true.

But if its meant as advice, you don't really offer anything concrete. 
How does one decide what is "too soon"? How does one avoid design 
paralysis?


> I do not concede that efficiency can be ignored because computers are
fast.

That's good, but I'm not sure why you think it is relevant as I never 
suggested that efficiency can be ignored. Only that what people *guess* 
is "lots of data" and what actually *is* lots of data may not be the 
same thing.


> I do concede that it is often not worth the effort or that you can
> inadvertently make things worse and there are tradeoffs.

Okay.


> Let me be specific. The side topic was asking how to get a random key from
> an existing dictionary. If you do this ONCE, it may be no big deal to make
a
> list of all keys, index it by a random number, and move on. I did supply a
> solution that might(or might not) run faster by using a generator to get
one
> item at a time and stopping when found. Less space but not sure if less
> time.

Why don't you try it and find out?


> But what I often need to do is to segment lots of data into two piles. One
> is for training purposes using some machine learning algorithm and the
> remainder is to be used for verifications. The choice must be random or
the
> entire project may become meaningless. So if your data structure was a
> dictionary with key names promptly abandoned, you cannot just call pop()
> umpteen times to get supposedly random results as they may come in a very
> specific order.

Fortunately I never suggested doing that.


> If you want to have 75% of the data in the training section,
> and 25% reserved, and you have millions of records, what is a good way to
> go? 

The obvious solution:

keys = list(mydict.keys())
random.shuffle(keys)
index = len(keys)*3//4
training_data = keys[:index]
reserved = keys[index:]

Now you have the keys split into training data and reserved data. To 
extract the value, you can just call mydict[some_key]. If you prefer, 
you can generate two distinct dicts:

training_data = {key: mydict[key] for key in training_data}

and similarly for the reserved data, and then mydict becomes redundant 
and you are free to delete it (or just ignore it).

Anything more complex than this solution should not even be attempted 
until you have tried the simple, obvious solution and discovered that it 
isn't satisfactory.

Keep it simple. Try the simplest thing that works first, and don't add 
complexity until you know that you need it.

By the way, your comments would be more credible if you had actual 
working code that demonstrates your point, rather than making vague 
comments that something "may" be faster. Sure, anything "may" be faster. 
We can say that about literally anything. Walking to Alaska from the 
southernmost tip of Chile while dragging a grand piano behind you "may" 
be faster than flying, but probably isn't. Unless you have actual code 
backing up your assertions, they're pretty meaningless.

And the advantage of working code is that people might actually learn 
some Python too.



-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From alan.gauld at yahoo.co.uk  Thu Dec 27 20:02:10 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 28 Dec 2018 01:02:10 +0000
Subject: [Tutor] Re Module
In-Reply-To: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
References: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
Message-ID: <q03sif$hrb$1@blaine.gmane.org>

On 27/12/2018 15:10, Asad wrote:

> file1 - I need a search a error say x if the error matches
> 
> Look for the same error x in other file 2
> 
> Here is the code :
> I have 10 different patterns therefore I used list comprehension and
> compiling the pattern so I loop over and find the exact pattern matching
> 
> re_comp1 = [re.compile(pattern) for pattern in str1]

I assume str1 is actually a list of strings? You don't
show the definition but since you say it gives the
expected output I'll hope that its correct.

> for pat in re_comp1:
>     if pat.search(st,re.IGNORECASE):
>         x = pat.pattern
> print x                ===> here it gives the expected output it correct

I assume st comes from your file1? You don't show us that
bit of code either...

But you do realize that the print only shows the last result.
If there is more than one matching pattern the previous results
get thrown away. And if you only care about one match you
could just use a single regex.
On the other hand, if you do only want the last matching
pattern then what you have works.

> if re.search('x', line, re.IGNORECASE) is not None:  ===> Gives a wrong
> match
>       print line

Notice that you pass the string 'x' into the search.
I assume it is meant to be x? That means you are searching
for the single character 'x' in line. You also don't show
us where line comes from I assume its the other file?

But why do you switch from using the compiled pattern?
Why not just assign x to the pattern object pat? This can
then be used to search line directly and with greater
efficiency.


> if re.search(x, line, re.IGNORECASE) is not None: then no match occurs
>       print line

And are you sure a match should occur?
It would help debug this if you showed us some sample data.
Such as the value of x and the value of line.

Given you are obviously only showing us a selected segment
of your code its hard to be sure. But as written here you
are searching line even if no pattern matches in file1.
That is, you could loop through all your patterns, never
assign anything to x and then go ahead and try to search
for 'x' in line. You should probably check x first.

Also, since you don't show the file looping code we don't
know whether you break out whenever you find a match or
whether the rest of the code is all inside the first
loop over file1. Trying to debug someone else's code
is hard enough. When we only have half the code we are
reduced to guesswork.

Finally, do you get any error messages? If so, please
post them in their entirety. Based on your code I'm
assuming you are working on Python v2.? but its always
worth posting the python version and OS.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Thu Dec 27 21:39:34 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 28 Dec 2018 13:39:34 +1100
Subject: [Tutor] Re Module
In-Reply-To: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
References: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
Message-ID: <20181228023934.GB13616@ando.pearwood.info>

On Thu, Dec 27, 2018 at 08:40:12PM +0530, Asad wrote:
> Hi All ,
> 
>           I trying find a solution for my script , I have two files :
> 
> file1 - I need a search a error say x if the error matches
> 
> Look for the same error x in other file 2
> 
> Here is the code :
> I have 10 different patterns therefore I used list comprehension and
> compiling the pattern so I loop over and find the exact pattern matching
> 
> re_comp1 = [re.compile(pattern) for pattern in str1]


You can move the IGNORECASE flag into the call to compile. Also, perhaps 
you can use better names instead of "str1" (one string?).

patterns = [re.compile(pattern, re.IGNORECASE) for pattern in string_patterns]
 
> for pat in re_comp1:
>     if pat.search(st,re.IGNORECASE):
>         x = pat.pattern
> print x                ===> here it gives the expected output it correct
> match
> print type(x)
> <type 'unicode'>

Be careful here: even though you have ten different patterns, only *one* 
will be stored in x. If three patterns match, x will only get the last 
of the three and the others will be ignored.

 
> if re.search('x', line, re.IGNORECASE) is not None:  ===> Gives a wrong match

That's because you are trying to match the literal string "x", so it 
will match anything with the letter "x":

box, text, ax, equinox, except, hexadecimal, fix, Kleenex, sixteen ...


> Instead if I use :
> 
> if re.search(x, line, re.IGNORECASE) is not None: then no match occurs
>       print line

Here you are trying to match the variable called x. That is a very bad 
name for a variable (what does "x" mean?) but it should work.

If no match occurs, it probably means that the value of x doesn't occur 
in the line you are looking at.

Try printing x and line and see if they are what you expect them to be:

print x
print line


-- 
Steve

From avigross at verizon.net  Thu Dec 27 21:48:02 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 27 Dec 2018 21:48:02 -0500
Subject: [Tutor] dangerous class neighborhood
In-Reply-To: <CAPTjJmpyxNQeUuzMh3qo9f6VQj2dnvsWz0TTyNk38S75Ji4mWg@mail.gmail.com>
References: <005f01d49e2d$8ed49380$ac7dba80$@verizon.net>
 <CAPTjJmpyxNQeUuzMh3qo9f6VQj2dnvsWz0TTyNk38S75Ji4mWg@mail.gmail.com>
Message-ID: <00c501d49e57$c047c7a0$40d756e0$@verizon.net>

Sometimes when I post something I get back comments and evaluate them and
learn quite a bit. I then reply and debate every little point and it can
continue for a few rounds.

I don't seem to be in that mood today so let me simply restate my entire
post in a few sentences with no examples, no lectures, no advice on what
anyone else can do and very little for anyone to bother replying to. Here
goes:

Sometimes when I run up against a wall and find that a solution to a problem
does not work because things may not work as I expected, I pause. I
reconsider what I actually need to get done. Then I look to see if I can
come up with other ways to do it that will work while still getting the
important parts done. Failing that, I ask if perhaps there is another tool,
such as another programming language that is a better fit for the task. And,
if the work needed seems excessive, I ask if perhaps the problem does not
really need to be solved by me and I move on.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net at python.org> On
Behalf Of Chris Angelico
Sent: Thursday, December 27, 2018 5:11 PM
To: Python <python-list at python.org>
Subject: Re: dangerous class neighborhood

On Fri, Dec 28, 2018 at 8:47 AM Avi Gross <avigross at verizon.net> wrote:
> Question 2: Do you want the variables available at the class level or 
> at the instance level?

For constants, definitely put them on the class. They'll be available on
instances as well ("for free", if you like). For mutables, obviously you
need to decide on a case-by-case basis.

> Question 3: Which python variations on syntactic sugar, such as list 
> comprehensions, get expanded invisibly in ways that make the problem 
> happen by asking for variables to be found when no longer in the visible
range?

The oddities with comprehensions were tackled partly during the discussion
of PEP 572. If you want to know exactly why this isn't changing, go read a
few hundred emails on the subject. A lot of the main points are summarized
in the PEP itself:

https://www.python.org/dev/peps/pep-0572/

> There may be matters of efficiency some would consider but some of the 
> examples seen recently seemed almost silly and easy to compute. The 
> people asking about this issue wanted to define a bunch of CONSTANTS, 
> or things that might as well be constants, like this:
>
>
>
> def Foo():
>
>                 A = ("male", "female", "other")
>
>                 B = [ kind[0] for kind in A ]            # First letters
> only
>
>                 # And so on making more constants like a dictionary 
> mapping each string to a number or vice versa.
>
>
>
> All the above can be evaluated at the time the class is defined but 
> unintuitive scope rules make some operations fail as variables defined 
> in the scope become unavailable to other things that SEEM to be 
> embedded in the same scope.

If you write simple and Pythonic code, these will almost always work
perfectly. The recent thread citing an oddity worked just fine until it was
written to iterate over range(len(x)) instead of iterating directly.

> If they are ONLY to be used within an instance of Foo or invoked from 
> within there, there may be a fairly simple suggestion. If you already 
> have a __init__ method, then instantiate the variables there carefully 
> using the self object to reference those needed.

But why? __init__ should initialize an instance, not class-level constants.
A Python class is not restricted to just methods, and there's no reason to
avoid class attributes.

> Create a function either outside the class or defined within. Have it 
> do any internal calculations you need in which all internal variables 
> can play nicely with each other. Then let it return all the variables 
> in a tuple like
> this:
>
> def make_sexual_constants():
>
>                 A = .
>
>                 B = .
>
>                 C = f(A,B)
>
>                 D = .
>
> def Foo():
>
>                 (A, B, C, D) = make_sexual_constants():

Lovely. Now you have to define your variables once inside the function, then
name them a second time in that function's return statement, and finally
name them all a *third* time in the class statement (at least, I presume
"def Foo():" is meant to be "class Foo:"). A mismatch will create bizarre
and hard-to-debug problems.
What do you actually gain? Can you show me real-world code that would truly
benefit from this?

> Can we agree that the class Foo now has those 4 variables defined and 
> available at either the class level or sub-class or instance levels? 
> But the values are created, again, in a unified safe environment?

Unified? No more so than the class statement itself. Safe? Definitely not,
because of the mandatory duplication of names.

> As noted in section 3, it would be good to know what python features 
> may be unsafe in this kind of context. I had an unrelated recent 
> discussion where it was mentioned that some proposed feature changes 
> might not be thread safe. Valid consideration when that may lead to
hard-to-explain anomalies.

Uhh..... nope, that's nothing but FUD. There is no reason to believe that
some language features would be "unsafe".

> We now hear that because a list comprehension can be unwound 
> internally into a "while" loop and an "if" statement and that some 
> parts may expand to calls to a "range" statement, perhaps some 
> variables are now in more deeply embedded contexts that have no access to
any class variables.

No idea what you're looking at. A comprehension can be unwound in a fairly
straight-forward way, although there are some subtleties to them.

B = [ kind[0] for kind in A ]
# equivalent to, approximately:
def listcomp(iter):
    result = []
    for kind in iter:
        result.append(kind[0])
    return result
B = listcomp(A)

For casual usage, you can describe a list comp very simply and neatly:

B = [ kind[0] for kind in A ]
# equivalent to, more approximately:
B = []
for kind in A:
    B.append(kind[0])

Nothing here expands to a call to range(), nothing has a while loop.
The only way you'll get an "if" is if you had one in the comprehension
itself.

> I think that
> is quite reasonable; hence my suggestion we need to know which ones to 
> avoid, or use a workaround like expanding it out ourselves and perhaps 
> carefully import variables into other contexts such as by passing the 
> variable into the function that otherwise cannot access it from a 
> point it can still be seen.

Sure. If the comprehension doesn't work for you, just put a for loop inside
your class statement. This is not a problem.

> Least, but at least last, I ask if the need really exists for these 
> variables as constants versus functions. If creating C this way runs 
> into problems, but A and B are fine, consider making a method with 
> some name like
> Foo.get_C() that can see A and B and do the calculation and yet return 
> the value of C needed. Less efficient but .

Definitely not. That would imply that the value of C might change, or might
have significant cost, or in some other way actually merits a getter
function. Python isn't built to encourage that.

Class scope has edge cases, to be sure, but they're much more notable in
carefully-crafted exploratory code than in actual real-world code.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list


From steve at pearwood.info  Thu Dec 27 21:55:49 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 28 Dec 2018 13:55:49 +1100
Subject: [Tutor] dangerous class neighborhood
In-Reply-To: <00c501d49e57$c047c7a0$40d756e0$@verizon.net>
References: <005f01d49e2d$8ed49380$ac7dba80$@verizon.net>
 <CAPTjJmpyxNQeUuzMh3qo9f6VQj2dnvsWz0TTyNk38S75Ji4mWg@mail.gmail.com>
 <00c501d49e57$c047c7a0$40d756e0$@verizon.net>
Message-ID: <20181228025549.GC13616@ando.pearwood.info>

On Thu, Dec 27, 2018 at 09:48:02PM -0500, Avi Gross wrote:
> Sometimes when I post something I get back comments and evaluate them and
> learn quite a bit. I then reply and debate every little point and it can
> continue for a few rounds.

I think you sent this to the wrong mailing list. The original post and 
discussion was on Python-List, but you replied to Tutor.



-- 
Steve

From avigross at verizon.net  Thu Dec 27 22:29:31 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 27 Dec 2018 22:29:31 -0500
Subject: [Tutor] dangerous class neighborhood
In-Reply-To: <00c501d49e57$c047c7a0$40d756e0$@verizon.net>
References: <005f01d49e2d$8ed49380$ac7dba80$@verizon.net>
 <CAPTjJmpyxNQeUuzMh3qo9f6VQj2dnvsWz0TTyNk38S75Ji4mWg@mail.gmail.com>
 <00c501d49e57$c047c7a0$40d756e0$@verizon.net>
Message-ID: <00cf01d49e5d$8c0ebbf0$a42c33d0$@verizon.net>

My apologies. This reply went to the wrong forum.

Hopefully it contains little to be debated. An even shorter version would
be:

If at first you don't succeed ...

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of Avi
Gross
Sent: Thursday, December 27, 2018 9:48 PM
To: tutor at python.org
Subject: Re: [Tutor] dangerous class neighborhood

Sometimes when I post something I get back comments and evaluate them and
learn quite a bit. I then reply and debate every little point and it can
continue for a few rounds.

I don't seem to be in that mood today so let me simply restate my entire
post in a few sentences with no examples, no lectures, no advice on what
anyone else can do and very little for anyone to bother replying to. Here
goes:

Sometimes when I run up against a wall and find that a solution to a problem
does not work because things may not work as I expected, I pause. I
reconsider what I actually need to get done. Then I look to see if I can
come up with other ways to do it that will work while still getting the
important parts done. Failing that, I ask if perhaps there is another tool,
such as another programming language that is a better fit for the task. And,
if the work needed seems excessive, I ask if perhaps the problem does not
really need to be solved by me and I move on.


From avigross at verizon.net  Thu Dec 27 23:07:10 2018
From: avigross at verizon.net (Avi Gross)
Date: Thu, 27 Dec 2018 23:07:10 -0500
Subject: [Tutor] Re Module
In-Reply-To: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
References: <CAG3LsKEh-VG=4CEO6Js7VVxutYNK6VpwnLzpxLCXQ_909MUt0w@mail.gmail.com>
Message-ID: <00d701d49e62$ce7eb120$6b7c1360$@verizon.net>

Asad,

After reading replies to you by Alan and Steven I want to ask you if you can
first tell us in normal words what the exact outline of the program does. If
you only want help on one small part, tell us  about that.

I was first fooled into thinking you wanted to show us how you solve the
majority of the entire problem, whatever it was so I wanted to hear things
like I show next.

An example would be to search two files for error matches of various kinds
and report if they contain any matches. Just report True versus False or
something.

Another goal might be to show the first match in some way then quit.

Another might be to do the same search in two files and report ALL the
matches in some format.

After being clear on the goal, you might specify the overall algorithm you
want to use. For example, do you process one file to completion and save
some results then process the other the same way then compare and produce
output? Or do you process both nearly simultaneously in one pass, or perhaps
multiple passes. Do you search for one error type at a time or all at once?
Can there be multiple errors on the same line of the same kind or different
ones? What does error even mean? Is it something like "Fail: 666" versus
"Warn: 42" or something where multiple errors share a part or ...

Once we have some idea of the goal, we could help you see if the approach
seems reasonable even before reading the code. And, when reading the code,
we might see if your implementation  seems to match the plan so perhaps we
can see where you diverge from it perhaps with a mistake.

If I just look at what you provided, you do some of what I asked. You are
not clear on what the two files contain other than they may have an error
that you can identify with a set of patterns. Can you tell us if you are
looking at one line at a time, assuming it is a text file? Your code shows
no evidence of a file at all. Your focus in what you share with us is mainly
on creating a list of compiled search patterns and applying it to one
uninitialized "st" and trying to figure out which one matched. 

You do not show any examples of the pattern but suggest something is
failing. For all we know one of your patterns just matched the presence of a
single common character or even was not formatted properly and failed to be
compiled.

My impression is you are not actually asking about the overall problem. Your
real question may be how to use a regular expression on a string and find
out what matched. If so, that would be the headline, not about two files.
And it may even be your entire approach could change. An example would be to
store your patterns as a text keyword in a dictionary with the value being
the compiled version so when you evaluate a line using the pattern, you know
which one you matched with. I am NOT saying this is a good solution or a
better one. I am asking you to think what you will need and what techniques
might make life easier in doing it.

So besides trying to alter some code based of the feedback, from others,
could you resubmit the question with a focus on what you are doing and what
exactly is not working that you want looked at. Specifics would be useful
including at least one pattern and a line of sample text that should be
matched by the pattern as an example and perhaps one that should not. And
any error messages are vital.

When you do, I am sure Steven and Alan and others might be able to zoom
right in and help you diagnose, if you don't figure it out by yourself first
by being able to see what your goal is and perhaps doing a little debugging.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Asad
Sent: Thursday, December 27, 2018 10:10 AM
To: tutor at python.org
Subject: [Tutor] Re Module

Hi All ,

          I trying find a solution for my script , I have two files :

file1 - I need a search a error say x if the error matches

Look for the same error x in other file 2

Here is the code :
I have 10 different patterns therefore I used list comprehension and
compiling the pattern so I loop over and find the exact pattern matching

re_comp1 = [re.compile(pattern) for pattern in str1]

for pat in re_comp1:
    if pat.search(st,re.IGNORECASE):
        x = pat.pattern
print x                ===> here it gives the expected output it correct
match
print type(x)

<type 'unicode'>

if re.search('x', line, re.IGNORECASE) is not None:  ===> Gives a wrong
match
      print line

Instead if I use :

if re.search(x, line, re.IGNORECASE) is not None: then no match occurs
      print line

Please advice where I going wrong or what can be done to make it better .

Thanks,


--
Asad Hasan
+91 9582111698
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Fri Dec 28 00:58:00 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 28 Dec 2018 00:58:00 -0500
Subject: [Tutor] Interpreter pasting  Question
Message-ID: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>

This is a serious question. I have tried things and searched and remain
stumped. It is about python and perhaps just the interpreter.

 

Copying and pasting multiple lines into the interpreter fails in mysterious
ways, unless they are a logical single entity.

 

Is there a way to change this behavior, or perhaps an editor/environment
that feeds multiple lines more carefully to the interpreter?

 

I am used to using R Studio and other tools with R. This is not about R but
please permit me an example.

 

I like to be able to tell it in the EDIT window that I want a single line or
a group of lines to be run on the console. That is especially useful when I
am debugging and want to run some code and go to the console and examine
things then continue. I may want to run one line or a region. Clearly
indentation is not an issue there.

 

The python interpreter I use (cpython 3.7.0) on windows lets me copy/paste a
simple single line of code. If it is a complex construct like an if/else
that is syntactically seen as a single statement, it allows it to be copied
in as a unit.

 

But something as simple as this:

 

X=5

Y=6

 

When copied in looks like this:

 

>>> X=5

Y=6

 

SyntaxError: multiple statements found while compiling a single statement

 

Yet I have seen programs that build up a string and give that to eval. Can
those be multiple statements? My tests using eval confuse me. But things I
can type in to the interpreter could be the same after a paste but perhaps
there are reasons they can't be?

 

It may be something as silly as python being too fast and reading all the
input. When I type and hit CARRIAGE RETURN/ENTER it may finish evaluating
before I even start typing the next line. 

 

The current behavior makes it hard to copy code from something like an email
message. Yes, I can (and do) insert the code in a file using IDLE or other
tools then ask to run the file but that is far from the same. Having to copy
one line at a time is beyond frustrating for me.

 

Feel free to tell me I am doing it wrong or of a way that works for others. 

 

Python has an eval() and an exec() and I would assume the latter would be a
way to see what works.

 

Here are three lines using \n:

 

>>> exec("x=6\ny=7\nprint(x+y)")

                

13

 

That seems to work. Again, if I copied and pasted the same as three lines,
it fails.

 

This works too:

 

>>> dothis = """x=6

y=7

print(x+y)

"""

                

>>> exec(dothis)

                

13

 

I did some HW before writing this and some suggest ipython has a better
repr.

 

Some solutions are arguably weird even if they work. One suggestion is to
wrap it all in an if that is always true and indent all subsequent lines:

 

if 1:

                x = 6

                y = 7

                print(x+y)

 

When I do a copy and paste of that including the otherwise meaningless if
statement, it works:

 

>>> if 1:

                x = 6

                y = 7

                print(x+y)

 

13

 

So, sure, I can stick nonsense in but wonder if there is a reasonable trick
like setting options to the interpreter or using an editor that feeds lines
slowly or .

 

But, yes, I can live with this or perhaps learn how to use a debugger than
might let me run things with breakpoints or .

 

 

 


From steve at pearwood.info  Fri Dec 28 01:57:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 28 Dec 2018 17:57:42 +1100
Subject: [Tutor] Interpreter pasting  Question
In-Reply-To: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
References: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
Message-ID: <20181228065742.GD13616@ando.pearwood.info>

On Fri, Dec 28, 2018 at 12:58:00AM -0500, Avi Gross wrote:

[...]
> Copying and pasting multiple lines into the interpreter fails in mysterious
> ways, unless they are a logical single entity.
> 
> Is there a way to change this behavior, or perhaps an editor/environment
> that feeds multiple lines more carefully to the interpreter?

Which interpreter are you using?

If it is the regular Python REPL (Read Eval Print Loop), then pasting 
multiple lines should work, with some limitations. For example, I can 
paste:

x = 5
y = 6

as two lines, and it works fine under Linux. I see no reason why it 
should be different under Windows.

If you just run "python" in the Windows shell (cmd.exe or whatever its 
called), you should get an interactive interpreter. What happens when 
you paste multiple lines in that?



[...]
> When copied in looks like this:
> 
> >>> X=5
> 
> Y=6
> 
> SyntaxError: multiple statements found while compiling a single statement

That looks like a bug. Are you using IDLE? Perhaps it has been fixed 
in newer versions of Python, and if not, you should report it as a 
bug on the bugtracker

https://bugs.python.org/

but:

(1) try searching for similar bug reports first;
(2) read this first: 

    http://www.sscce.org/

(3) and do try to keep your bug report short and to the point.


It looks like others have this problem with IDLE too:

https://duckduckgo.com/?q=idle+paste+multiple+lines


IPython/Jupyter allows pasting of multiple lines; I expect that bpython 
will too.

https://ipython.org/

https://bpython-interpreter.org/


But as I said, the vanilla Python REPL ought to work. How are you 
starting the interpreter? My guess is that you're using IDLE.



[...]
> Python has an eval() and an exec() and I would assume the latter would be a
> way to see what works.

No, exec() executes Python code, it doesn't try to simulate a REPL.


> Here are three lines using \n:
> 
> >>> exec("x=6\ny=7\nprint(x+y)")
> 13
>
> 
> That seems to work. Again, if I copied and pasted the same as three lines,
> it fails.

Without knowing the system you are using, and how you copy and paste, it 
is hard to comment except to say "Works for me". Here are three lines:

x = 6
y = 7
print(x + y)


If I select those three lines with the mouse, copy, then paste into a 
standard Python interactive interpreter, I get this:


py> x = 6
py> y = 7
py> print(x + y)
13


exactly as expected. But if I do it in IDLE, I get this:

>>> x = 6
y = 7
print(x + y)

SyntaxError: multiple statements found while compiling a single 
statement
>>> 

That *really* sounds like a bug to me. But perhaps I just don't 
understand IDLE.




-- 
Steve

From alan.gauld at yahoo.co.uk  Fri Dec 28 07:52:06 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 28 Dec 2018 12:52:06 +0000
Subject: [Tutor] Interpreter pasting Question
In-Reply-To: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
References: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
Message-ID: <q0565j$9om$1@blaine.gmane.org>

On 28/12/2018 05:58, Avi Gross wrote:
> This is a serious question. I have tried things and searched and remain
> stumped. It is about python and perhaps just the interpreter.
> 
> Copying and pasting multiple lines into the interpreter fails in mysterious
> ways, unless they are a logical single entity.

That will depend on the interpreter. Which one are you using?

> Is there a way to change this behavior, or perhaps an editor/environment
> that feeds multiple lines more carefully to the interpreter?

IDLE does what you say but IDLEX accepts multiple lines.

The vanilla command line >>> accepts multiple lines (on Linux at least)

PyCharm has multiple options for how it processes pasted lines/blocks.

There are even IDEs that will evaluate/execute selected text
without the need to paste.

It all depends on the tool.



-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From huseyin at piramit.com.tr  Fri Dec 28 04:18:04 2018
From: huseyin at piramit.com.tr (=?utf-8?B?SMO8c2V5aW4gRXJ0dcSfcnVs?=)
Date: Fri, 28 Dec 2018 09:18:04 +0000
Subject: [Tutor] dangerous class neighborhood
In-Reply-To: <20181228025549.GC13616@ando.pearwood.info>
References: <005f01d49e2d$8ed49380$ac7dba80$@verizon.net>
 <CAPTjJmpyxNQeUuzMh3qo9f6VQj2dnvsWz0TTyNk38S75Ji4mWg@mail.gmail.com>
 <00c501d49e57$c047c7a0$40d756e0$@verizon.net>
 <20181228025549.GC13616@ando.pearwood.info>
Message-ID: <DB7PR06MB5515289CEAEB7875117BEF6992B70@DB7PR06MB5515.eurprd06.prod.outlook.com>

That was nice ? I did not know the python-list.

-----Original Message-----
From: Tutor <tutor-bounces+huseyin=piramit.com.tr at python.org> On Behalf Of Steven D'Aprano
Sent: Friday, December 28, 2018 5:56 AM
To: tutor at python.org
Subject: Re: [Tutor] dangerous class neighborhood

On Thu, Dec 27, 2018 at 09:48:02PM -0500, Avi Gross wrote:
> Sometimes when I post something I get back comments and evaluate them 
> and learn quite a bit. I then reply and debate every little point and 
> it can continue for a few rounds.

I think you sent this to the wrong mailing list. The original post and discussion was on Python-List, but you replied to Tutor.



-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

From mike at pathtomathclarity.com  Thu Dec 27 19:54:48 2018
From: mike at pathtomathclarity.com (Mike Mossey)
Date: Thu, 27 Dec 2018 16:54:48 -0800
Subject: [Tutor] decomposing a problem
In-Reply-To: <008f01d49e3c$7c9e9120$75dbb360$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008f01d49e3c$7c9e9120$75dbb360$@verizon.net>
Message-ID: <74D52B49-B1DC-4531-A9D3-5142A7BBD926@pathtomathclarity.com>


> On Dec 27, 2018, at 3:32 PM, Avi Gross <avigross at verizon.net> wrote:
> 
> [Mark Lawrence please press DELETE now in case the rest of this message is
> all about you.]
> [[If that is not working, if on Windows, try Control-ALT-DELETE as that will
> really get rid of my message.]]
> 


Hi Avi,

Mark doesn?t have a basis for complaining, of course, as he can simply not read your posts.


> Back to replying to Steven,
> 
> Of course I want to be corrected when wrong.
> 
> I think everyone here knows I tend to be quite expansive in my thoughts and
> sometimes to the point where they suggest I am free-associating. I am trying
> to get to the point faster and stay there.

Since you are expressing interest, I'll give some thoughts. 

I think it?s important not only for writing, but for economy of thinking to use fewer words and simpler concepts, and it can make us better programmers and teachers. 

Previously, when I worked alone as a programmer, I was stuck in overcomplicated ways of thinking. It?s ?getting out there? and interacting with people that rejuvenated my thinking, and I?ll be forever grateful.

One form of practice at this is to edit my posts for brevity. Here?s a link about brevity in writing:

http://copymatter.com/embracing-brevity/

It helps me as well that I tutor students in math and computer science regularly, because it forces me to get more simple and concrete. A student is a ?feedback device? ? when I?m doing better, I can read the results in their expression and their understanding.

I think it?s important both to have something you are aiming for (a sense of what level of brevity you?d like to achieve) and a feedback mechanism that helps you to know if you are succeeding.

Take or leave these thoughts as you see fit.
Mike



From avigross at verizon.net  Fri Dec 28 12:13:28 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 28 Dec 2018 12:13:28 -0500
Subject: [Tutor] Interpreter pasting  Question
In-Reply-To: <20181228065742.GD13616@ando.pearwood.info>
References: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
 <20181228065742.GD13616@ando.pearwood.info>
Message-ID: <005901d49ed0$a6fc3640$f4f4a2c0$@verizon.net>

Steven,

Thanks. You are right. I am no longer going to talk IDLE worship!

When I open up a command window and run what I assumed was the same version
of python, no problems. Should have tried that but frankly that means I am
stuck with having to then do other things differently like reloading an
edited file. Oh well.

It sounds like the problem is that IDLE is (invisibly) intercepting what I
type or paste and then passing it on to a slaved interpreter. There likely
are other side effects and another editor/environment may not have this
issue. I will try some.

As one of my programming styles includes lots of hands-on incremental
analysis of bits and pieces to get them working before combining them,  I
have no time for an idyllic experience.

OK, just kidding. Nice to have choices and I particularly want to move on to
using notebooks in which I can have batches of code I can run inline and
even interleave bits of multiple languages. 

Yes, I am aware that my experiments with exec are not testing the same
thing. They do indicate ways to get around the issue even using IDLE and I
am now thinking of ways to do more dynamic things by building code and then
executing it. So, not really time wasted. The fact that python relies on
indentation means you can have problems if things are not aligned just
right.

Here is my work-around if I happen to be using IDLE and want to copy in
something huge from say a web page.

Type this:

Cmd = """

Do the paste.

Type on a line by itself:

"""

Now type

exec(Cmd)

May not work all the time, but perhaps a reasonable work around.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 28, 2018 1:58 AM
To: tutor at python.org
Subject: Re: [Tutor] Interpreter pasting Question

On Fri, Dec 28, 2018 at 12:58:00AM -0500, Avi Gross wrote:

[...]
> Copying and pasting multiple lines into the interpreter fails in 
> mysterious ways, unless they are a logical single entity.
> 
> Is there a way to change this behavior, or perhaps an 
> editor/environment that feeds multiple lines more carefully to the
interpreter?

Which interpreter are you using?

If it is the regular Python REPL (Read Eval Print Loop), then pasting
multiple lines should work, with some limitations. For example, I can
paste:

x = 5
y = 6

as two lines, and it works fine under Linux. I see no reason why it should
be different under Windows.

If you just run "python" in the Windows shell (cmd.exe or whatever its
called), you should get an interactive interpreter. What happens when you
paste multiple lines in that?



[...]
> When copied in looks like this:
> 
> >>> X=5
> 
> Y=6
> 
> SyntaxError: multiple statements found while compiling a single 
> statement

That looks like a bug. Are you using IDLE? Perhaps it has been fixed in
newer versions of Python, and if not, you should report it as a bug on the
bugtracker

https://bugs.python.org/

but:

(1) try searching for similar bug reports first;
(2) read this first: 

    http://www.sscce.org/

(3) and do try to keep your bug report short and to the point.


It looks like others have this problem with IDLE too:

https://duckduckgo.com/?q=idle+paste+multiple+lines


IPython/Jupyter allows pasting of multiple lines; I expect that bpython will
too.

https://ipython.org/

https://bpython-interpreter.org/


But as I said, the vanilla Python REPL ought to work. How are you starting
the interpreter? My guess is that you're using IDLE.



[...]
> Python has an eval() and an exec() and I would assume the latter would be
a
> way to see what works.

No, exec() executes Python code, it doesn't try to simulate a REPL.


> Here are three lines using \n:
> 
> >>> exec("x=6\ny=7\nprint(x+y)")
> 13
>
> 
> That seems to work. Again, if I copied and pasted the same as three lines,
> it fails.

Without knowing the system you are using, and how you copy and paste, it 
is hard to comment except to say "Works for me". Here are three lines:

x = 6
y = 7
print(x + y)


If I select those three lines with the mouse, copy, then paste into a 
standard Python interactive interpreter, I get this:


py> x = 6
py> y = 7
py> print(x + y)
13


exactly as expected. But if I do it in IDLE, I get this:

>>> x = 6
y = 7
print(x + y)

SyntaxError: multiple statements found while compiling a single 
statement
>>> 

That *really* sounds like a bug to me. But perhaps I just don't 
understand IDLE.




-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From mats at wichmann.us  Fri Dec 28 13:41:53 2018
From: mats at wichmann.us (Mats Wichmann)
Date: Fri, 28 Dec 2018 11:41:53 -0700
Subject: [Tutor] Interpreter pasting Question
In-Reply-To: <005901d49ed0$a6fc3640$f4f4a2c0$@verizon.net>
References: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
 <20181228065742.GD13616@ando.pearwood.info>
 <005901d49ed0$a6fc3640$f4f4a2c0$@verizon.net>
Message-ID: <7454d0eb-65dc-f6ea-bb62-cb66cefbae7d@wichmann.us>

On 12/28/18 10:13 AM, Avi Gross wrote:

> As one of my programming styles includes lots of hands-on incremental
> analysis of bits and pieces to get them working before combining them,  I
> have no time for an idyllic experience.

don't presume to know what environment will actually work for you, but
that working model sounds somewhat like Jupyter, which has a "notebook"
for recording live code snippets, commentary, etc. - and a way to clip
those bits and pieces into your combination.





From alan.gauld at yahoo.co.uk  Fri Dec 28 14:43:08 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Fri, 28 Dec 2018 19:43:08 +0000
Subject: [Tutor] Interpreter pasting Question
In-Reply-To: <005901d49ed0$a6fc3640$f4f4a2c0$@verizon.net>
References: <00dd01d49e72$4a67bd40$df3737c0$@verizon.net>
 <20181228065742.GD13616@ando.pearwood.info>
 <005901d49ed0$a6fc3640$f4f4a2c0$@verizon.net>
Message-ID: <q05u89$pev$1@blaine.gmane.org>

On 28/12/2018 17:13, Avi Gross wrote:

> It sounds like the problem is that IDLE is (invisibly) intercepting what I
> type or paste and then passing it on to a slaved interpreter. There likely
> are other side effects and another editor/environment may not have this
> issue. I will try some.

Do not confuse the interactive prompt with the interpreter.
The prompt is unique to the tool and the interpreter is, ultimately,
used by the tool in the background. But each tool handles the
input differently before invoking the interpreter.

> As one of my programming styles includes lots of hands-on incremental
> analysis of bits and pieces to get them working before combining them,  I
> have no time for an idyllic experience.

Remember tat IDLE is not intended as an industrial strength
IDE  like Eclipse, Netbeans, VS or Blackadder etc it was
intended to be an easy to learn tool for beginners.

There are several tools that do what you want better than
IDLE - including IDLEX. (And if things go to plan many IDLEX
features will appear in the 3.7/3.8 versions of IDLE.)

But even IDLEX is not the last word in toolsets. There are
many alternatives including IPython and Jupyter etc.

> OK, just kidding. Nice to have choices and I particularly want to move on to
> using notebooks in which I can have batches of code I can run inline and
> even interleave bits of multiple languages. 

Sounds like you should take a look at IPython/Jupyter


> Now type
> 
> exec(Cmd)
> 
> May not work all the time, but perhaps a reasonable work around.

The biggest issue is that it requires the environment to
pre-exist - variables initialised sensibly, etc. But thats true of any
tool that offers that kind of eval/exec function.
But mostly you can avoid all of that scaffolding with a better toolset.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From avigross at verizon.net  Fri Dec 28 15:34:19 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 28 Dec 2018 15:34:19 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181227223813.GD13061@ando.pearwood.info>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
Message-ID: <008801d49eec$b5ecc310$21c64930$@verizon.net>

Steve,

I am going to just respond to one part of your message and will snip the
rest. I am not is disagreement with most of what you say and may simply
stress different aspects. I will say that unless I have reason to, I don't
feel a need to test speeds for an academic discussion. Had this been a real
project, sure. Even then, if it will need to run on multiple machines using
multiple incarnations of python, the results will vary, especially if the
data varies too. You suggest that discussions backed by real data are
better. Sure. But when a discussion is abstract enough, then I think it
perfectly reasonable to say "may be faster" to mean that until you try it,
there are few guarantees. Many times a method seems superior until you reach
a pathological case. One sorting algorithm is fast except when the data is
already almost fully sorted already.

So why do I bother saying things like MAY? It seems to be impossible to
please everybody. There are many things with nuance and exceptions. When I
state things one way, some people (often legitimately) snipe. When I don't
insist on certainty, other have problem with that. When I make it short, I
am clearly leaving many things out. When I go into as much detail as I am
aware of, I get feedback that it is too long or boring or it wanders too
much. None of this is a problem as much as a reality about tradeoffs.

So before I respond, here is a general statement. I am NOT particularly
interested in much of what we discuss here from a specific point of view.
Someone raises a question and I think about it. They want to know of a
better way to get a random key from a dictionary. My thought is that if I
needed that random key, maybe I would not have stored it in a dictionary in
the first place. But, given that the data is in a dictionary, I wonder what
could be done. It is an ACADEMIC discussion with a certain amount of hand
waving. Sometimes I do experiment and show what I did. Other times I say I
am speculating and if someone disagrees, fine. If they show solid arguments
or point out errors on my part or create evidence, they can change my mind. 

You (Steve) are an easy person to discuss things with but there are some who
are less. People who have some idea of my style and understand the kind of
discussion I am having at that point and who let me understand where they
are coming from, can have a reasonable discussion. The ones who act like TV
lawyers who hear that some piece of evidence has less than one in a
quadrillion chance of happening then say BUT THERE IS A CHANCE so reasonable
doubt ... are hardly worth debating.

You replied to one of my points with this about a way to partition data:

---
The obvious solution:

keys = list(mydict.keys())
random.shuffle(keys)
index = len(keys)*3//4
training_data = keys[:index]
reserved = keys[index:]
---

(In the above, "---" is not python but a separator!)

That is indeed a very reasonable way to segment the data. But it sort of
makes my point. If the data is stored in a dictionary, the way to access it
ended up being to make a list and play with that. I would still need to get
the values one at a time from the dictionary such as in the ways you also
show and I omit.

For me, it seems more natural in this case to simply have the data in a data
frame where I have lots of tools and methods available. Yes, underneath it
all providing an array of indices or True/False Booleans to index the data
frame can be slow but it feels more natural. Yes, python has additional
paradigms I may not have used in R such as list comprehensions and
dictionary comprehensions that are conceptually simple. But I did use the
R-onic (to coin a phrase nobody would ironically use) equivalents that can
also be powerful and I need not discuss here in a python list. Part of
adjusting to python includes unlearning some old habits and attitudes and
living off this new land. [[Just for amusement, the original R language was
called S so you might call its way of doing things Sonic.]]

I see a balance between various ways the data is used. Clearly it is
possible to convert it between forms and for reasonable amounts of data it
can be fast enough. But as you note, at some point you can just toss one
representation away so maybe you can not bother using that in the first
place. Keep it simple.

In many real life situations, you are storing many units of data and often
have multiple ways of indexing the data. There are representations that do
much of the work for you. Creating a dictionary where each item is a list or
other data structure can emulate such functionality and even have advantages
but if your coding style is more comfortable with another way, why bother
unless you are trying to learn other ways and be flexible.

As I have mentioned too many times, my most recent work was in R and I
sometimes delight and other times groan at the very different ways some
things are done when using specific modules or libraries. But even within a
language and environment there are radical differences and approaches. The
naked (albeit evolving) languages often offer a reasonable way to do things
but developers often add new layers and paradigms that can become the more
standard way of doing things. Base python can do just about anything with
just lists. All you have to remember is that you stored the zip code in the
23rd element. But programmers created things like named tuples or
specialized objects like dataframes that may be easier or more powerful or
less prone to some kinds of errors or  whatever. 




From alan.gauld at yahoo.co.uk  Fri Dec 28 19:33:06 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 29 Dec 2018 00:33:06 +0000
Subject: [Tutor] decomposing a problem
In-Reply-To: <008801d49eec$b5ecc310$21c64930$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008801d49eec$b5ecc310$21c64930$@verizon.net>
Message-ID: <q06f7v$nhm$1@blaine.gmane.org>

On 28/12/2018 20:34, Avi Gross wrote:

> So before I respond, here is a general statement. I am NOT particularly
> interested in much of what we discuss here from a specific point of view.
> Someone raises a question and I think about it. They want to know of a
> better way to get a random key from a dictionary. My thought is that if I
> needed that random key, maybe I would not have stored it in a dictionary in
> the first place. But, given that the data is in a dictionary, I wonder what
> could be done. It is an ACADEMIC discussion with a certain amount of hand
> waving.

But you need to apply real world constraints.
The choice of data type is intrinsic to the language in use.
(The same is true of control structures - loops,
decision points etc - but that is less pertinent here.)

If you program in C pretty much all you get is the array.
Everything else (including struct/union/typedef) is hand
crafted by the programmer.

If its Lisp then you get the list and anything else
is coded (or simulated in code) by hand.

In Python we have lists, tuples, dictionaries and sets.
Anything else, including subclassing the basic types, is
down to the programmer (or finding a module already created
by somebody else).

In Smalltalk we have over a hundred basic collection
types to choose from. And the choice to subclass any
of them.

So when you address a problem in any given language the
available solutions must be constrained by whatever the
language in question offers. Wishing for non-existent
data structures that may exist elsewhere is simply to
request a new feature to be designed and programmed
in the language at hand. That may be the best solution
depending on the nature of the problem but we need to
recognise the nature of the request. It is still a new
feature.

Of course we can learn a great deal by comparing features
on one language against another but in terms of solving
a specific request we need specific answers too.


> ---
> The obvious solution:
> 
> keys = list(mydict.keys())
> random.shuffle(keys)
> index = len(keys)*3//4
> training_data = keys[:index]
> reserved = keys[index:]
> ---

> For me, it seems more natural in this case to simply have the data in a data
> frame where I have lots of tools and methods available. 

But only if such a data frame exists. In Python it does not
(at least, not in the standard library). So any reference
to such a non existent structure is in effect a work request
for someone to build one. To do so requires a specification
or design that the OP can follow or better still a prototypical
template. It also assumes a much higher level of skill than the
original request and the "obvious solution".

> but if your coding style is more comfortable with another way, why bother
> unless you are trying to learn other ways and be flexible.

If your current language does not support the structure
you desire you have three choices:
1) change your programming language or
2) build the missing feature or
3) find a workaround using the structures available.

Most people opt for #3. (Although all may be valid options
depending on the circumstance)


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Fri Dec 28 20:04:23 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 29 Dec 2018 12:04:23 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <008801d49eec$b5ecc310$21c64930$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008801d49eec$b5ecc310$21c64930$@verizon.net>
Message-ID: <20181229010422.GF13616@ando.pearwood.info>

On Fri, Dec 28, 2018 at 03:34:19PM -0500, Avi Gross wrote:

[...]
> You replied to one of my points with this about a way to partition data:
> 
> ---
> The obvious solution:
> 
> keys = list(mydict.keys())
> random.shuffle(keys)
> index = len(keys)*3//4
> training_data = keys[:index]
> reserved = keys[index:]
> ---
> 
> (In the above, "---" is not python but a separator!)
> 
> That is indeed a very reasonable way to segment the data. But it sort of
> makes my point. If the data is stored in a dictionary, the way to access it
> ended up being to make a list and play with that. I would still need to get
> the values one at a time from the dictionary such as in the ways you also
> show and I omit.

Yes? How else do you expect to get the value given a key except by 
looking it up?


> For me, it seems more natural in this case to simply have the data in 
> a data frame where I have lots of tools and methods available.


I'm not sure if your understanding of a data frame is the same as my
understanding. Are you talking about this?

http://www.r-tutor.com/r-introduction/data-frame

In other words, a two-dimensional array of some sort?

Okay, you have your data frame. Now what? How do you solve the problem
being asked? I'm not interested in vague handwaving that doesn't solve
anything. You specified data in a key:value store, let's say like this:


mydict = {'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14,
          'aardvark': 3, 'argument': 11, 'parrot': 16}

Here it is as a data frame:

df = [['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument', 'parrot'],
      [25, 2, 7, 1, 14, 3, 11, 16]]

Now what? How do you randomly split that into randomly selected set of
training data and reserved data?

Feel free to give an answer in terms of R, provided you also give an
answer in terms of Python. Remember that unlike R, Python doesn't have a
standard data frame type, so you are responsible for building whatever
methods you need.




-- 
Steve

From avigross at verizon.net  Fri Dec 28 22:39:53 2018
From: avigross at verizon.net (Avi Gross)
Date: Fri, 28 Dec 2018 22:39:53 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181229010422.GF13616@ando.pearwood.info>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008801d49eec$b5ecc310$21c64930$@verizon.net>
 <20181229010422.GF13616@ando.pearwood.info>
Message-ID: <00fe01d49f28$29081210$7b183630$@verizon.net>

I will answer this question then head off on vacation.

I said that "I" have lots of experience dealing with data in dataframes.
Most of it is not in Python, obviously. It works well for some kinds of
data, not others. Let me not diverge from the purpose of the group by saying
a version of it is commonly used in the python module pandas. Python has
been extended in many ways including arrays in many dimensions in numpy.
There are also matrix types that are 2-D or more but all the above can only
hold one kind of data at a time. In real life, you have tabular data where
each column can be of any type.

There are quite a few implementations out there. Yes, your pointer to R
shows a basic version. But I note python has its own versions too.
Implementations may have some nice tools for taking subsets of rows or
columns, modifying existing structures and adding additional rows/columns
and iterating over them and generating all kinds of logical subsets. The
implementation may be as sort of a list of vectors of the same length. Lots
of flexibility if your data fits, not so much otherwise. It is one serious
reason R was used by so many for purposes like statistics and graphics but
python has been extended and now more and more people are using it for such
things as well as so many other things that R is not designed for. 

Just one point since it may be relevant. You can index a data.frame in ways
not that different than objects in python and one is to generate a condition
like "variable > 5" (or much more complex) that generates a Boolean vector
that can be used to take a subset of all rows into a copy for one purpose,
such as training in machine learning and simply negating the vector gets the
remaining ones. You can do similar things in python. My recent expertise and
experience relied heavily on the data structures R uses, where everything is
a vector and most operations are vectorized. In python, I am learning to do
things in whatever way works and that has been interesting too. 

You asked about the dictionary. R has things sort of similar but not quite.
It has environments that most people would not use as a dictionary. So in R,
I might have made a data.frame to hold key/value columns albeit keys would
not be forced to be unique. Thus selecting random keys would just mean
selecting a random number in the range of the number of rows and taking that
item in a key column. 

What you call a data.frame is not a data.frame. It is a list of two lists.
If you will pardon my putting python code here, it might look like this and
note numpy and pandas and other mathematical/scientific modules have plenty
of methods available. No real need for me to show you in R, unless offline.

import numpy as np
import pandas as pd

Your two forms of data will now be instantiated and then put into the
alternate formats to show some things.

>>> mydict = {'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14,
          'aardvark': 3, 'argument': 11, 'parrot': 16}

>>> df = [['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'],
      [25, 2, 7, 1, 14, 3, 11, 16]]

>>> mydict
		     
{'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14, 'aardvark': 3,
'argument': 11, 'parrot': 16}
>>> df
		     
[['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'], [25, 2, 7, 1, 14, 3, 11, 16]]

There are many ways to create a data.frame from lists, vectors,
dictionaries, ...

Your data is a bit skimpy for showing much but it can easily be imported:

>>> df_from_list = pd.DataFrame(data=df)
		     
>>> df_from_list
		     
      0    1     2        3     4         5         6       7
0  spam  ham  eggs  cheddar  brie  aardvark  argument  parrot
1    25    2     7        1    14         3        11      16

I would prefer to import this as columns:

I will first convert your list of 2 lists to a vertical format more suitable
in such tables:

>>> newdf = list(zip(df[0], df[1]))
>>> newdf
[('spam', 25), ('ham', 2), ('eggs', 7), ('cheddar', 1), ('brie', 14),
('aardvark', 3), ('argument', 11), ('parrot', 16)]

>>> vert_df = pd.DataFrame(newdf, columns=["FOOD", "AMOUNT"])
>>> vert_df
       FOOD  AMOUNT
0      spam      25
1       ham       2
2      eggs       7
3   cheddar       1
4      brie      14
5  aardvark       3
6  argument      11
7    parrot      16

It does not format well here but you see the two columns as well as a sort
of index.

You can take subsets of rows:

>>> vert_df[2:4]
      FOOD  AMOUNT
2     eggs       7
3  cheddar       1

You can select by conditions:

>> vert_df[vert_df['AMOUNT'] >= 7]
		    
       FOOD  AMOUNT
0      spam      25
2      eggs       7
4      brie      14
6  argument      11
7    parrot      16

And so on. The point is you can often read in data and manipulate it. R has
multiple sets of tools including one in what they call the tidyverse. In
English, given such a data structure with any number of rows and columns,
you have names for the columns and optionally the rows. The tools allow you
to select any combination of rows and columns based on all kinds of search
and matching criteria. You can re-arrange them, add new ones or create new
ones using the data in existing ones, generate all kinds of statistical info
such as the standard deviation of each column or apply your own functions.
All this can be done in a pipelined fashion.

What you often do is read in a data.frame from a Comma Separated Values file
(CSV) or all kinds of data from other programs including EXCEL spreadsheets,
Stata and so on, including the Feather format python can make, and massage
the data such as removing rows with any NA (not available) values, or
interpolate new values, split it into multiple dataframes as discussed and
so on. You can do many statistical analyses by feeding entire dataframes  or
selected subsets to functions to do many things like linear and other forms
of regression and it really shines when you feed these data structures to
graphics engines like ggplot2 letting you make amazing graphs. Like I said,
R is designed with vectors and data.frames as principal components.

But once python is augmented, it can do much of the same. Not quite sure how
much is ported or invented. Some data types like "formulas" seem to be done
differently. It will take me a while to study it all.

I can point to resources if anyone is interested but again, this is a python
forum. So it is of interest to me that it is possible to combine bits and
pieces of R and python in the same programming environment. I mean you can
use one to do what it does best, have the data structures silently be
translated into something the other one understands and do some more
processing where you have software that shines, then switch back and forth
as needed. This kind of duality may mean it is not necessary to keep
changing one language to be able to do what the other does, in some cases.
And, amusingly, much of the underlying functionality accessed is in C or C++
with some data structures being translated to/from the compiled C/C++
equivalents as you enter a function, them translated back at exit. 

This is not very deep so just making a point since Alan asked. You can find
strengths and weaknesses in any language. I love how python consistently
enough has everything being object-oriented. R started off without and has
grafted on at least a dozen variations which can be a tad annoying.



-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 28, 2018 8:04 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Fri, Dec 28, 2018 at 03:34:19PM -0500, Avi Gross wrote:

[...]
> You replied to one of my points with this about a way to partition data:
> 
> ---
> The obvious solution:
> 
> keys = list(mydict.keys())
> random.shuffle(keys)
> index = len(keys)*3//4
> training_data = keys[:index]
> reserved = keys[index:]
> ---
> 
> (In the above, "---" is not python but a separator!)
> 
> That is indeed a very reasonable way to segment the data. But it sort 
> of makes my point. If the data is stored in a dictionary, the way to 
> access it ended up being to make a list and play with that. I would 
> still need to get the values one at a time from the dictionary such as 
> in the ways you also show and I omit.

Yes? How else do you expect to get the value given a key except by looking
it up?


> For me, it seems more natural in this case to simply have the data in 
> a data frame where I have lots of tools and methods available.


I'm not sure if your understanding of a data frame is the same as my
understanding. Are you talking about this?

http://www.r-tutor.com/r-introduction/data-frame

In other words, a two-dimensional array of some sort?

Okay, you have your data frame. Now what? How do you solve the problem
being asked? I'm not interested in vague handwaving that doesn't solve
anything. You specified data in a key:value store, let's say like this:


mydict = {'spam': 25, 'ham': 2, 'eggs': 7, 'cheddar': 1, 'brie': 14,
          'aardvark': 3, 'argument': 11, 'parrot': 16}

Here it is as a data frame:

df = [['spam', 'ham', 'eggs', 'cheddar', 'brie', 'aardvark', 'argument',
'parrot'],
      [25, 2, 7, 1, 14, 3, 11, 16]]

Now what? How do you randomly split that into randomly selected set of
training data and reserved data?

Feel free to give an answer in terms of R, provided you also give an
answer in terms of Python. Remember that unlike R, Python doesn't have a
standard data frame type, so you are responsible for building whatever
methods you need.




-- 
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From steve at pearwood.info  Fri Dec 28 23:12:19 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 29 Dec 2018 15:12:19 +1100
Subject: [Tutor] decomposing a problem
In-Reply-To: <00fe01d49f28$29081210$7b183630$@verizon.net>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008801d49eec$b5ecc310$21c64930$@verizon.net>
 <20181229010422.GF13616@ando.pearwood.info>
 <00fe01d49f28$29081210$7b183630$@verizon.net>
Message-ID: <20181229041217.GG13616@ando.pearwood.info>

On Fri, Dec 28, 2018 at 10:39:53PM -0500, Avi Gross wrote:
> I will answer this question then head off on vacation.

You wrote about 140 or more lines, but didn't come close to answering 
the question: how to randomly split data from a dictionary into training 
data and reserved data.



-- 
Steve

From bhatkarthik1996 at gmail.com  Sat Dec 29 01:12:16 2018
From: bhatkarthik1996 at gmail.com (Karthik Bhat)
Date: Sat, 29 Dec 2018 11:42:16 +0530
Subject: [Tutor] Defining variable arguments in a function in python
Message-ID: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>

Hello,

        I have the following piece of code. In this, I wanted to make use
of the optional parameter given to 'a', i.e- '5', and not '1'

def fun_varargs(a=5, *numbers, **dict):
    print("Value of a is",a)

    for i in numbers:
        print("Value of i is",i)

    for i, j in dict.items():
        print("The value of i and j are:",i,j)

fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)

How do I make the tuple 'number'  contain the first element to be 1 and not
2?


-- 
Regards,
Karthik A Bhat

From alan.gauld at yahoo.co.uk  Sat Dec 29 04:41:17 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Sat, 29 Dec 2018 09:41:17 +0000
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
Message-ID: <q07fbq$jg1$1@blaine.gmane.org>

On 29/12/2018 06:12, Karthik Bhat wrote:

> def fun_varargs(a=5, *numbers, **dict):
>     print("Value of a is",a)
> 
>     for i in numbers:
>         print("Value of i is",i)
> 
>     for i, j in dict.items():
>         print("The value of i and j are:",i,j)
> 
> fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> 
> How do I make the tuple 'number'  contain the first element to be 1 and not
> 2?

You need to provide a value for a.

The default 5 will only be used if the function is called
without *any* arguments. Otherwise it will always take
the first argument value. So, if you want a to be 5 and
then provide a tuple etc you must explicitly pass a 5 in:

fun_varargs(5, 1,2,3,4,5,6,7,8,9,10, Jack=111,John=222,Jimmy=333)


HTH
-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Sat Dec 29 06:01:52 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 29 Dec 2018 22:01:52 +1100
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
Message-ID: <20181229110152.GI13616@ando.pearwood.info>

On Sat, Dec 29, 2018 at 11:42:16AM +0530, Karthik Bhat wrote:
> Hello,
> 
>         I have the following piece of code. In this, I wanted to make use
> of the optional parameter given to 'a', i.e- '5', and not '1'
> 
> def fun_varargs(a=5, *numbers, **dict):
[...]
> 
> fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> 
> How do I make the tuple 'number' contain the first element to be 1 and not 2?


You can't. Python allocates positional arguments like "a" first, and 
only then collects whatever is left over in *numbers. How else would you 
expect it to work? Suppose you called:

fun_varargs(1, 2, 3)

wanting a to get the value 1, and numbers to get the values (2, 3). And 
then immediately after that you call 

fun_varargs(1, 2, 3)

wanting a to get the default value 5 and numbers to get the values 
(1, 2, 3). How is the interpreter supposed to guess which one you 
wanted?

If you can think of a way to resolve the question of when to give "a" 
the default value, then we can help you program it yourself:


def func(*args, **kwargs):
    if condition:
        # When?
        a = args[0]
        numbers = args[1:]
    else:
        a = 5  # Default.
        numbers = args
    ...

But writing that test "condition" is the hard part.




-- 
Steve

From avigross at verizon.net  Sat Dec 29 07:39:24 2018
From: avigross at verizon.net (Avi Gross)
Date: Sat, 29 Dec 2018 07:39:24 -0500
Subject: [Tutor] decomposing a problem
In-Reply-To: <20181229041217.GG13616@ando.pearwood.info>
References: <002d01d49cae$0a179ff0$1e46dfd0$@verizon.net>
 <97386C41-E167-4462-9434-CBD24DA10000@pathtomathclarity.com>
 <005301d49cd7$585c1b40$091451c0$@verizon.net>
 <20181226054513.GW13061@ando.pearwood.info>
 <004d01d49d34$5a36d470$0ea47d50$@verizon.net>
 <20181227223813.GD13061@ando.pearwood.info>
 <008801d49eec$b5ecc310$21c64930$@verizon.net>
 <20181229010422.GF13616@ando.pearwood.info>
 <00fe01d49f28$29081210$7b183630$@verizon.net>
 <20181229041217.GG13616@ando.pearwood.info>
Message-ID: <000001d49f73$87d89300$9789b900$@verizon.net>

Steven,

As I head out the door, I will sketch it.

Given a data.frame populated with N rows and columns you want to break it
into training and test data sets.

In a data.frame, you can refer to a row by using an index like 5 or 2019.
You can ask for the number of rows currently in existence. You can also
create an array/vector of length N consisting of instructions that can tell
which random rows of the N you want and which you don't. For the purposes of
this task, you choose random numbers in the range of N and either keep the
numbers as indices or as a way to mark True/False in the vector. You then
ask for a new data.frame made by indexing the existing one using the vector.
You can then negate the vector and ask for a second new data.frame indexing
it.

Something close to that.

Or, you can simply add the vector as a new column in the data.frame in some
form. It would then mark which rows are to be used for which purpose. Later,
when using the data, you include a CONDITION that row X is true, or
whatever.



-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 28, 2018 11:12 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Fri, Dec 28, 2018 at 10:39:53PM -0500, Avi Gross wrote:
> I will answer this question then head off on vacation.

You wrote about 140 or more lines, but didn't come close to answering the
question: how to randomly split data from a dictionary into training data
and reserved data.



--
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From __peter__ at web.de  Sat Dec 29 08:31:42 2018
From: __peter__ at web.de (Peter Otten)
Date: Sat, 29 Dec 2018 14:31:42 +0100
Subject: [Tutor] Defining variable arguments in a function in python
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
Message-ID: <q07srr$qhn$1@blaine.gmane.org>

Karthik Bhat wrote:

> Hello,
> 
>         I have the following piece of code. In this, I wanted to make use
> of the optional parameter given to 'a', i.e- '5', and not '1'
> 
> def fun_varargs(a=5, *numbers, **dict):
>     print("Value of a is",a)
> 
>     for i in numbers:
>         print("Value of i is",i)
> 
>     for i, j in dict.items():
>         print("The value of i and j are:",i,j)
> 
> fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> 
> How do I make the tuple 'number'  contain the first element to be 1 and
> not 2?

One option is to change the function signature to

def fun_varargs(*numbers, a=5, **dict):
    ...

which turns `a` into a keyword-only argument.

>>> fun_varargs(1, 2, 3, foo="bar"):
Value of a is 5
Value of i is 1
Value of i is 2
Value of i is 3
The value of i and j are: foo bar

To override the default you have to specify a value like this:

>>> fun_varargs(1, 2, 3, foo="bar", a=42)
Value of a is 42
Value of i is 1
Value of i is 2
Value of i is 3
The value of i and j are: foo bar



From avigross at verizon.net  Sat Dec 29 10:35:31 2018
From: avigross at verizon.net (Avi Gross)
Date: Sat, 29 Dec 2018 15:35:31 +0000 (UTC)
Subject: [Tutor] decomposing a problem
References: <23441516.10574689.1546097731107.ref@mail.yahoo.com>
Message-ID: <23441516.10574689.1546097731107@mail.yahoo.com>


Steven,

A more practical answer about splitting a data frame is to import modules such as for machine learning. 

Import sklearn.model_selection

Then use train_test_split() to return 4 parts. Not sure what answers you need and why here. Plenty of ways and tools exist to specify choosing percentages to partition by or other ways.


Sent from AOL Mobile Mail
On Saturday, December 29, 2018 Avi Gross <avigross at verizon.net> wrote:
Steven,

As I head out the door, I will sketch it.

Given a data.frame populated with N rows and columns you want to break it
into training and test data sets.

In a data.frame, you can refer to a row by using an index like 5 or 2019.
You can ask for the number of rows currently in existence. You can also
create an array/vector of length N consisting of instructions that can tell
which random rows of the N you want and which you don't. For the purposes of
this task, you choose random numbers in the range of N and either keep the
numbers as indices or as a way to mark True/False in the vector. You then
ask for a new data.frame made by indexing the existing one using the vector.
You can then negate the vector and ask for a second new data.frame indexing
it.

Something close to that.

Or, you can simply add the vector as a new column in the data.frame in some
form. It would then mark which rows are to be used for which purpose. Later,
when using the data, you include a CONDITION that row X is true, or
whatever.



-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Friday, December 28, 2018 11:12 PM
To: tutor at python.org
Subject: Re: [Tutor] decomposing a problem

On Fri, Dec 28, 2018 at 10:39:53PM -0500, Avi Gross wrote:
> I will answer this question then head off on vacation.

You wrote about 140 or more lines, but didn't come close to answering the
question: how to randomly split data from a dictionary into training data
and reserved data.



--
Steve
_______________________________________________
Tutor maillist? -? Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

_______________________________________________
Tutor maillist? -? Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

From avigross at verizon.net  Sun Dec 30 00:07:20 2018
From: avigross at verizon.net (Avi Gross)
Date: Sun, 30 Dec 2018 00:07:20 -0500
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <20181229110152.GI13616@ando.pearwood.info>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
Message-ID: <000001d49ffd$8b560640$a20212c0$@verizon.net>

I have my usual off the wall answer.

OK, seriously. Not exactly an answer but perhaps an experiment.

The question was how to have a non-named first argument to a function with
some form of default. 

As was pointed out, this does not fit well with being able to have python
gather all positional arguments after it as well as all keyword arguments.

But bear with me. Say I want to have a way to signal that I want a default
for the first argument?

An empty comma fails but try this:

def hello(a, *n, **m) :
	if a == None: a=5
	print(a)
	print(*n)
	print(**m)

The above says "a" is required. It can be followed by any number of
positional args gathered into "n" and any number of keyword args gathered
into "m"

But what if you define a sentinel to watch for such as None, in the above?

If the first and only arg is None, it switches to the default of 5.

>>> hello(None)
5

Add a few more args and it properly takes it.

>>> hello(1,2,3)
1
2 3

Switch the first to None:

>>> hello(None,2,3)
5
2 3

The keywords don't work for print but no biggie.

But is this only for None? What I say any negative arg is replaced by 5?

def hello(a, *n, **m) :
	if a < 0: a=5
	print(a)
	print(*n)

Seems to work fine:

>>> hello(-666, 2, 3, 4)
5
2 3 4

And I wonder if we can use the darn ellipsis for something useful?

def hello(a, *n, **m) :
	if a == ... : a=5
	print(a)
	print(*n)

>>> hello(1,2,3)
1
2 3
>>> hello(...,2,3)
5
2 3
>>> hello(...,2,...)
5
2 Ellipsis

OK, all kidding aside, is this helpful? I mean if you want a function where
you MUST give at least one arg and specify the first arg can be some odd
choice (as above) and then be replaced by  a default perhaps it would be
tolerable to use None or an Ellipsis.

Or on a more practical level, say a function wants an input from 1 to 10.
The if statement above can be something like:

>>> def hello(a, *n, **m) :
	if not (1 <= a <= 10) : a=5
	print(a)
	print(*n)

	
>>> hello(1,2,3)
1
2 3
>>> hello(21,2,3)
5
2 3
>>> hello(-5,2,3)
5
2 3
>>> hello("infinity and beyond",2,3)
Traceback (most recent call last):
  File "<pyshell#32>", line 1, in <module>
    hello("infinity and beyond",2,3)
  File "<pyshell#28>", line 2, in hello
    if not (1 <= a <= 10) : a=5
TypeError: '<=' not supported between instances of 'int' and 'str'

As expected, it may take a bit more code such as checking if you got an int
but the idea may be solid enough. It is NOT the same as having a default
from the command line but it may satisfy some need.

Other than that, I fully agree that the current python spec cannot support
anything like this in the function definition.

Side note: To spare others, I sent Steven alone a deeper reply about ways to
select random rows from a pandas DataFrame. I am still learning how pandas
works and doubt many others here have any immediate needs.









-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Saturday, December 29, 2018 6:02 AM
To: tutor at python.org
Subject: Re: [Tutor] Defining variable arguments in a function in python

On Sat, Dec 29, 2018 at 11:42:16AM +0530, Karthik Bhat wrote:
> Hello,
> 
>         I have the following piece of code. In this, I wanted to make 
> use of the optional parameter given to 'a', i.e- '5', and not '1'
> 
> def fun_varargs(a=5, *numbers, **dict):
[...]
> 
> fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> 
> How do I make the tuple 'number' contain the first element to be 1 and not
2?


You can't. Python allocates positional arguments like "a" first, and only
then collects whatever is left over in *numbers. How else would you expect
it to work? Suppose you called:

fun_varargs(1, 2, 3)

wanting a to get the value 1, and numbers to get the values (2, 3). And then
immediately after that you call 

fun_varargs(1, 2, 3)

wanting a to get the default value 5 and numbers to get the values (1, 2,
3). How is the interpreter supposed to guess which one you wanted?

If you can think of a way to resolve the question of when to give "a" 
the default value, then we can help you program it yourself:


def func(*args, **kwargs):
    if condition:
        # When?
        a = args[0]
        numbers = args[1:]
    else:
        a = 5  # Default.
        numbers = args
    ...

But writing that test "condition" is the hard part.




--
Steve
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From steve at pearwood.info  Sun Dec 30 05:39:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 30 Dec 2018 21:39:06 +1100
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <000001d49ffd$8b560640$a20212c0$@verizon.net>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
Message-ID: <20181230103905.GK13616@ando.pearwood.info>

On Sun, Dec 30, 2018 at 12:07:20AM -0500, Avi Gross wrote:

[...]
> Or on a more practical level, say a function wants an input from 1 to 10.
> The if statement above can be something like:
> 
> >>> def hello(a, *n, **m) :
> 	if not (1 <= a <= 10) : a=5
> 	print(a)
> 	print(*n)
> 
> 	
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(21,2,3)
> 5
> 2 3
> >>> hello(-5,2,3)
> 5
> 2 3


This design is an example of "an attractive nuisance", possibly even a 
"bug magnet". At first glance, when used for mickey-mouse toy examples 
like this, it seems quite reasonable:

    hello(999, 1, 2)  # I want the default value instead of 999

but thinking about it a bit more deeply, and you will recognise some 
problems with it.

First problem: 

How do you know what value to pass if you want the default? Is 999 out 
of range? How about 11? 10? Who knows? If you have to look up the docs 
to know what counts as out of range, you might as well read the docs to 
find out what the default it, and just pass that:

    hello(5, 1, 2)  # I want the default value 5

but that kind of defeats the purpose of a default. The whole point of a 
default is that you shouldn't need to pass *anything at all*, not even a 
placeholder.

(If you need a placeholder, then you probably need to change your 
function parameters.)

But at least with sentinels like None, or Ellipsis, it is *obvious* 
that the value is probably a placeholder. With a placeholder like 11 or 
999, it isn't. They look like ordinary values.


Second problem:

Most of the time, we don't pass literal values to toy functions. We do 
something like this example:

    for number, widget_ID, delivery_date in active_orders:
        submit_order(number, widget_ID, delivery_date)

Can you see the bug? Of course you can't. There's no obvious bug. But 
little do you know, one of the orders was accidentally entered with an 
out-of-range value, let's say -1, and instead of getting an nice error 
message telling you that there's a problem that you need to fix, the 
submit_order() function silently replaces the erroneous value with the 
default.

The bug here is that submit_order() function exceeds its authority.

The name tells us that it submits orders, but it also silently decides 
to change invalid orders to valid orders using some default value. But 
this fact isn't obvious from either the name or the code. You only learn 
this fact by digging into the source code, or reading the documentation, 
and let's be honest, nobody wants to do either of those unless you 
really have to.

So when faced with an invalid order, instead of getting an error that 
you can fix, or even silently skipping the bad order, the submit_order() 
function silently changes it to a valid-looking but WRONG order that you 
probably didn't want. And that costs real money.

The risk of this sort of bug comes directly from the design of the 
function. While I suppose I must acknowledge that (hypothetically) 
there could be use-cases for this sort of design, I maintain that in 
general this design is a bug magnet: responsibility for changing 
out-of-range values to in-range values belongs with the caller, not 
the called function.

The caller may delegate that responsibility to another:

    for number, widget_ID, delivery_date in active_orders:
        number = validate_or_replace(number)
        submit_order(number, widget_ID, delivery_date)

which is fine because it is explicit and right there in plain sight.

This then allows us to make the submit_order() far more resiliant: if it 
is passed an invalid order, it can either fail fast, giving an obvious 
error, or at least skip the invalid order and notify the responsible 
people.


-- 
Steven

From __peter__ at web.de  Sun Dec 30 06:15:55 2018
From: __peter__ at web.de (Peter Otten)
Date: Sun, 30 Dec 2018 12:15:55 +0100
Subject: [Tutor] Defining variable arguments in a function in python
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
Message-ID: <q0a999$ppa$1@blaine.gmane.org>

Avi Gross wrote:

> To spare others, 

Thank you for that.

> I sent Steven alone

'Tis well deserved ;)



From bhatkarthik1996 at gmail.com  Sun Dec 30 04:54:36 2018
From: bhatkarthik1996 at gmail.com (Karthik Bhat)
Date: Sun, 30 Dec 2018 15:24:36 +0530
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <000001d49ffd$8b560640$a20212c0$@verizon.net>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
Message-ID: <CAPi8shaGC_naFYKcZn3dgaWpDM+hkXynEVdAkpMWHnp=_N+Egg@mail.gmail.com>

Thank you all for the quick response!

On Sun, Dec 30, 2018 at 10:39 AM Avi Gross <avigross at verizon.net> wrote:

> I have my usual off the wall answer.
>
> OK, seriously. Not exactly an answer but perhaps an experiment.
>
> The question was how to have a non-named first argument to a function with
> some form of default.
>
> As was pointed out, this does not fit well with being able to have python
> gather all positional arguments after it as well as all keyword arguments.
>
> But bear with me. Say I want to have a way to signal that I want a default
> for the first argument?
>
> An empty comma fails but try this:
>
> def hello(a, *n, **m) :
>         if a == None: a=5
>         print(a)
>         print(*n)
>         print(**m)
>
> The above says "a" is required. It can be followed by any number of
> positional args gathered into "n" and any number of keyword args gathered
> into "m"
>
> But what if you define a sentinel to watch for such as None, in the above?
>
> If the first and only arg is None, it switches to the default of 5.
>
> >>> hello(None)
> 5
>
> Add a few more args and it properly takes it.
>
> >>> hello(1,2,3)
> 1
> 2 3
>
> Switch the first to None:
>
> >>> hello(None,2,3)
> 5
> 2 3
>
> The keywords don't work for print but no biggie.
>
> But is this only for None? What I say any negative arg is replaced by 5?
>
> def hello(a, *n, **m) :
>         if a < 0: a=5
>         print(a)
>         print(*n)
>
> Seems to work fine:
>
> >>> hello(-666, 2, 3, 4)
> 5
> 2 3 4
>
> And I wonder if we can use the darn ellipsis for something useful?
>
> def hello(a, *n, **m) :
>         if a == ... : a=5
>         print(a)
>         print(*n)
>
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(...,2,3)
> 5
> 2 3
> >>> hello(...,2,...)
> 5
> 2 Ellipsis
>
> OK, all kidding aside, is this helpful? I mean if you want a function where
> you MUST give at least one arg and specify the first arg can be some odd
> choice (as above) and then be replaced by  a default perhaps it would be
> tolerable to use None or an Ellipsis.
>
> Or on a more practical level, say a function wants an input from 1 to 10.
> The if statement above can be something like:
>
> >>> def hello(a, *n, **m) :
>         if not (1 <= a <= 10) : a=5
>         print(a)
>         print(*n)
>
>
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(21,2,3)
> 5
> 2 3
> >>> hello(-5,2,3)
> 5
> 2 3
> >>> hello("infinity and beyond",2,3)
> Traceback (most recent call last):
>   File "<pyshell#32>", line 1, in <module>
>     hello("infinity and beyond",2,3)
>   File "<pyshell#28>", line 2, in hello
>     if not (1 <= a <= 10) : a=5
> TypeError: '<=' not supported between instances of 'int' and 'str'
>
> As expected, it may take a bit more code such as checking if you got an int
> but the idea may be solid enough. It is NOT the same as having a default
> from the command line but it may satisfy some need.
>
> Other than that, I fully agree that the current python spec cannot support
> anything like this in the function definition.
>
> Side note: To spare others, I sent Steven alone a deeper reply about ways
> to
> select random rows from a pandas DataFrame. I am still learning how pandas
> works and doubt many others here have any immediate needs.
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
> Steven D'Aprano
> Sent: Saturday, December 29, 2018 6:02 AM
> To: tutor at python.org
> Subject: Re: [Tutor] Defining variable arguments in a function in python
>
> On Sat, Dec 29, 2018 at 11:42:16AM +0530, Karthik Bhat wrote:
> > Hello,
> >
> >         I have the following piece of code. In this, I wanted to make
> > use of the optional parameter given to 'a', i.e- '5', and not '1'
> >
> > def fun_varargs(a=5, *numbers, **dict):
> [...]
> >
> > fun_varargs(1,2,3,4,5,6,7,8,9,10,Jack=111,John=222,Jimmy=333)
> >
> > How do I make the tuple 'number' contain the first element to be 1 and
> not
> 2?
>
>
> You can't. Python allocates positional arguments like "a" first, and only
> then collects whatever is left over in *numbers. How else would you expect
> it to work? Suppose you called:
>
> fun_varargs(1, 2, 3)
>
> wanting a to get the value 1, and numbers to get the values (2, 3). And
> then
> immediately after that you call
>
> fun_varargs(1, 2, 3)
>
> wanting a to get the default value 5 and numbers to get the values (1, 2,
> 3). How is the interpreter supposed to guess which one you wanted?
>
> If you can think of a way to resolve the question of when to give "a"
> the default value, then we can help you program it yourself:
>
>
> def func(*args, **kwargs):
>     if condition:
>         # When?
>         a = args[0]
>         numbers = args[1:]
>     else:
>         a = 5  # Default.
>         numbers = args
>     ...
>
> But writing that test "condition" is the hard part.
>
>
>
>
> --
> Steve
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>


-- 
Regards,
Karthik A Bhat

From avigross at verizon.net  Sun Dec 30 11:07:19 2018
From: avigross at verizon.net (Avi Gross)
Date: Sun, 30 Dec 2018 11:07:19 -0500
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <20181230103905.GK13616@ando.pearwood.info>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
Message-ID: <005d01d4a059$be289090$3a79b1b0$@verizon.net>

Steve,

I had the same thoughts and many more when I played with these ideas last night. I thought I stated clearly that this was an EXPLORATION and not a serious solution. So if that is accepted, we can certainly discuss merits without giving demerits as well as the many flaws and dangers lurking in any such design.


To begin with, anyone trying any variants of a design like this needs to be sure it is well documented. I have seen functions that do something vaguely similar and often with unexpected results.

Consider a NORMAL function with no external gimmicks. Say it accepts an argument (positional or by name does not matter) and processes it by checking they argument type, the specific values against a range expected, and so on. It then unilaterally makes changes. If it expects a single value of type integer, ang gets a floating point number, it may round or truncate it back to an integer. If it gets a complex number, it drops the imaginary part. If it gets a list or tuple or iterator of any sort, it just takes the first entry and coerces it into an integer. If it sees a matrix or other challenging object, it flattens it then ... If it sees a character string, it tries to convert something like "one" to an integer. If your choice is in some sense wrong, or just not convenient, it replaces it with another choice. For example, it may move your integer to the nearest prime integer. It may reject your suggestion to make a window on your screen a trillion pixels wide and drop it to 1024. 

Sometimes it prints a warning about changes it has made. Sometimes it won't run but print out a message telling you what value it might have accepted.

Again, discussing a different scenario. Would you agree that kind of design does happen and sometimes is seen as a smart thing but also as full of pitfalls even sometimes for the wary?

In that spirit, any design like the ones I played with is equally scary. Even worse, I have a QUESTION. Let me remind readers of the initial idea behind this. You have a function in which you want to communicate with the python ruler that it have a positional argument that will be switched to a default. But the current rules are that the only way to make something optional is by making it a keyword parameter and declare the default right there. If the function template is:

def hello(a=5)

then you can do any of the following with reasonable results:

hello()
hello(7)
hello(a=7)

The default is only applied in the first version.

The problem is if you want a non-key word to be used as a default and then also want to gather up additional positional arguments and so on. There is no way designed to do that and possibly there should not be.

So what I am trying to understand is this. When someone types in a function invocation and it is evaluated, when does the programmer get a chance to intervene? Something in what I will call the interpreter seems to look at what the programmer typed just now and sees:
hello()
hello(1)
hello(1,2)

and so on. It looks up the object that encapsulates the function hello is pointing to. It examines the many parts hidden within and determines what the designer asked for. If it sees that the actual call does not match, it geerally produces an error message like this:

>>> def hello(n): print(n)

>>> hello()
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    hello()
TypeError: hello() missing 1 required positional argument: 'n'

That is a reasonable thing to do. But in my idiotic design, I would like the user to be told a tad more. I want them told that if they want the default, use an n consisting of an ellipsis as in
hello(...)

to get the default.

Is there a way to create a function and set it up so you have some control over the error message? Without that, this kind of function is even more dangerous.

This message is long enough. I will reply to Steven's specific points in a bit. Still on vacation ?


-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of Steven D'Aprano
Sent: Sunday, December 30, 2018 5:39 AM
To: tutor at python.org
Subject: Re: [Tutor] Defining variable arguments in a function in python

On Sun, Dec 30, 2018 at 12:07:20AM -0500, Avi Gross wrote:

[...]
> Or on a more practical level, say a function wants an input from 1 to 10.
> The if statement above can be something like:
> 
> >>> def hello(a, *n, **m) :
> 	if not (1 <= a <= 10) : a=5
> 	print(a)
> 	print(*n)
> 
> 	
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(21,2,3)
> 5
> 2 3
> >>> hello(-5,2,3)
> 5
> 2 3


This design is an example of "an attractive nuisance", possibly even a "bug magnet". At first glance, when used for mickey-mouse toy examples like this, it seems quite reasonable:

    hello(999, 1, 2)  # I want the default value instead of 999

but thinking about it a bit more deeply, and you will recognise some problems with it.

First problem: 

How do you know what value to pass if you want the default? Is 999 out of range? How about 11? 10? Who knows? If you have to look up the docs to know what counts as out of range, you might as well read the docs to find out what the default it, and just pass that:

    hello(5, 1, 2)  # I want the default value 5

but that kind of defeats the purpose of a default. The whole point of a default is that you shouldn't need to pass *anything at all*, not even a placeholder.

(If you need a placeholder, then you probably need to change your function parameters.)

But at least with sentinels like None, or Ellipsis, it is *obvious* that the value is probably a placeholder. With a placeholder like 11 or 999, it isn't. They look like ordinary values.


Second problem:

Most of the time, we don't pass literal values to toy functions. We do something like this example:

    for number, widget_ID, delivery_date in active_orders:
        submit_order(number, widget_ID, delivery_date)

Can you see the bug? Of course you can't. There's no obvious bug. But little do you know, one of the orders was accidentally entered with an out-of-range value, let's say -1, and instead of getting an nice error message telling you that there's a problem that you need to fix, the
submit_order() function silently replaces the erroneous value with the default.

The bug here is that submit_order() function exceeds its authority.

The name tells us that it submits orders, but it also silently decides to change invalid orders to valid orders using some default value. But this fact isn't obvious from either the name or the code. You only learn this fact by digging into the source code, or reading the documentation, and let's be honest, nobody wants to do either of those unless you really have to.

So when faced with an invalid order, instead of getting an error that you can fix, or even silently skipping the bad order, the submit_order() function silently changes it to a valid-looking but WRONG order that you probably didn't want. And that costs real money.

The risk of this sort of bug comes directly from the design of the function. While I suppose I must acknowledge that (hypothetically) there could be use-cases for this sort of design, I maintain that in general this design is a bug magnet: responsibility for changing out-of-range values to in-range values belongs with the caller, not the called function.

The caller may delegate that responsibility to another:

    for number, widget_ID, delivery_date in active_orders:
        number = validate_or_replace(number)
        submit_order(number, widget_ID, delivery_date)

which is fine because it is explicit and right there in plain sight.

This then allows us to make the submit_order() far more resiliant: if it is passed an invalid order, it can either fail fast, giving an obvious error, or at least skip the invalid order and notify the responsible people.


--
Steven
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From avigross at verizon.net  Sun Dec 30 12:26:39 2018
From: avigross at verizon.net (Avi Gross)
Date: Sun, 30 Dec 2018 12:26:39 -0500
Subject: [Tutor] Defining variable arguments in a function in python
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info> 
Message-ID: <006401d4a064$d2e19580$78a4c080$@verizon.net>

Replying to Steve's points. Again, it was not a serious design and said so
but was an ACADEMIC exploration of what could be done. I fully agree with
Steve that it is probably not a great idea to do this but note the original
request might not have been a great one in the first place.

There are people who make a family of related functions and tell users to
pick the right one.

def add_two(x, y): return x+y
def add_three(x, y, z): return add_two(x,y)+z

You get the idea, I hope. So you can create one function in which an
argument is required and a sister function in which it is not allowed and a
default is built in. The user picks which function to call.

Again, NOT suggesting that is a great solution, just exploring
possibilities. There are often many ideas you can brainstorm before
rejecting most or all of them.

-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Steven D'Aprano
Sent: Sunday, December 30, 2018 5:39 AM
To: tutor at python.org
Subject: Re: [Tutor] Defining variable arguments in a function in python.

So everything Steve says as commentary and even criticism is indeed true. It
is a bug magnet and so on. So is just about any code I have seen, albeit not
as bad. A function with 40 positional arguments very often gets called with
arguments out of order for example. Some languages allow a row of empty
commas so you can specify the third and then the eight and then the
sixteenth and skip the others. 

Languages that use named arguments may be better but some get cute and will
match any argument that starts right. I mean you can call them with
layout="portrait" but also lay="landscape" or even l="..." and who knows
what they do if your allowed arguments include layers=True".

That is not a defense. It is an acknowledgement that many things we do have
drawbacks as well as advantages. On the whole, I doubt I would use the
design I offered except as an academic display of what not to do even if the
language allows it.

Having said that, I do defend the placeholder construct in places where it
is needed as a serious way to make a point.

Consider a request to do something frequent where the default should be
something like ALL, or ALL UP TO HERE, or ALL FROM HERE. Think of how you
take slices out of a list or other data structure including
multi-dimensional structures.

Here is an assortment of examples with many more possible. This is not a
function call exactly but the sub-language used allows many variations with
the use of what seems to be a series of up to three arguments with a colon
between them. But sometimes the absence of an argument makes it look like
the colon defaulted to something.

>>> a = list(range(10))
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> a[ : 3]
[0, 1, 2]
>>> a[ 3: ]
[3, 4, 5, 6, 7, 8, 9]
>>> a[3:9]
[3, 4, 5, 6, 7, 8]
>>> a[3:9:2]
[3, 5, 7]
>>> a[9:3:-1]
[9, 8, 7, 6, 5, 4]
>>> a[-1]
9
>>> a[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

So the DEFAULT for anything before the colon is a loose concept. It is
anything before whatever the next argument states. This shows that the
default works in the extreme case to be nothing:

>>> a[:0]
[]

The third optional argument is defaulted to be "1".

But underneath it all this is what is termed syntactic sugar. Each and every
such pattern is translated into a slice call:

>>> slice(1,9,4)
slice(1, 9, 4)
>>> a[slice(1,9,4)]
[1, 5]
>>> a[slice(None,9,4)]
[0, 4, 8]

Did you see the value None is used to mean use the darn DEFAULT?

It can be used for any of the parts or even all.

>>> a[slice(None,None,None)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

So here is an example of a dangerous function we all use regularly, albeit
usually indirectly, that uses my odd design principle. 

I can drown you with more examples, but since I don't hate anyone here, just
one more. Without details, the Ellipsis seems to have been partially created
to be used in the numpy module as a way to deal with indexing arrays with
multiple indices. It is not easy to say I want the first and the last an
skip the middle ones. Enough said. 

Steven is quite reasonably more concerned when the placeholder is invisibly
changed than when it is obviously just a placeholder. As he says you can
just look up the default in documentation and use it. I could argue that a
pythonic way to do things is to make it so if the default later changes, you
are not stuck. Then again, maybe you don't want it to change invisibly.  A
compromise would be to have the user explicitly ask for whatever is the
default as in soup="Du Jour" gets you whatever the soup of the day is, and
may even get you a salad if they ran out.

Explicit is often better. But you can have too much of a good thing. People
often create a wrapper to a function with too many required or optional
arguments. An example is a function used to read data in from a file.  There
are general functions where you must tell it in excruciating detail what to
do. What character (or sequence) separates the fields? Is it a comma as in a
CSV file? Is it a tab or some other character? Or are the fields of
specified constant widths with no separator? Are all rows the same or is
there a header row? Should the program make an educated guess about the
number of columns of data or will you tell it how many or which ones you
want? Ditto for the types of data in each. If the file might have characters
embedded such as in quotes containing a comma, how is that recognized or
dealt with? If a column read in has repeated values, should it be read in as
a more compressed categorical form such as a factor in R. The list of
possibilities is HUGE.

So instead of using a function like read.table() people make available lots
of others like read.csv() that is a wrapper that merges your supplied
arguments with a set of defaults that make sense when reading a Comma
Separated value file. 

But in the above example, to be clear, the defaults are all done in a normal
way. You do not use silly placeholders.

Steve writes:

" But at least with sentinels like None, or Ellipsis, it is *obvious* that
the value is probably a placeholder. With a placeholder like 11 or 999, it
isn't. They look like ordinary values."

I partially agree. I can think of cases where None really means None as in I
want None for Dessert, not the gentle substitution of sugar-free Jell-O.

Steve then provides an example of a more subtly bug. To be succinct, it is a
class of arguments that boils down to localized knowledge fails when
evaluated elsewhere or worse when a substitution is evaluated elsewhere.
That is also true in more normal cases but is indeed a problem.

Back to my CSV example, if the wrapper function ASSUMES that there will
normally be a header line in a CSV file and calls the wrapped function
appropriately with that added request, then mistakes can happen. If it makes
the opposite assumption, mistakes can happen. If it makes no assumption and
specifies nothing, mistakes can happen!

So I ask the question of how to deal with real life scenarios? There are
many no-win situations out there. Do you try to give warnings? Or maybe a
customized warning that says you should call the function again and this
time specify one of several alternatives as an explicit argument? I have
seen that done.

Steve is right that the scenario where a function changes things quietly can
be nefarious especially downstream. A good example is when dealing with
missing data your program may use a placeholder like Nan or NA or 999 or a
blank string. But if you call a function to take the mean of a sample, it
may not see the marker as a suggestion to skip it. It may treat nonsense as
a 0 and get a wrong mean. It may skip it but count it when dividing by the
number of items. Or, it may need an argument like na.rm=TRUE and perhaps
even a second argument specifying what an 'na' looks like. Worse, I often
save the results of a calculation in a file using some format like a CSV. I
then have it re-read into perhaps a program in another programming language.
Some expect strange things like a single period to mean not available. So
you may want to be careful how you write a file containing NOT AVAILABLE
items. Even worse, you often cannot export some things reliably as the
receiver cannot handle any form. Can you save a value of Inf that is
meaningful even to read back in? I don't mean from using something like
pickle, but from some storage formal like CSV or EXCEL.

I see Steve wrote a bit more and will just say I agree. God programming
style tries to avoid surprises when it can.

On Sun, Dec 30, 2018 at 12:07:20AM -0500, Avi Gross wrote:

[...]
> Or on a more practical level, say a function wants an input from 1 to 10.
> The if statement above can be something like:
> 
> >>> def hello(a, *n, **m) :
> 	if not (1 <= a <= 10) : a=5
> 	print(a)
> 	print(*n)
> 
> 	
> >>> hello(1,2,3)
> 1
> 2 3
> >>> hello(21,2,3)
> 5
> 2 3
> >>> hello(-5,2,3)
> 5
> 2 3


This design is an example of "an attractive nuisance", possibly even a "bug
magnet". At first glance, when used for mickey-mouse toy examples like this,
it seems quite reasonable:

    hello(999, 1, 2)  # I want the default value instead of 999

but thinking about it a bit more deeply, and you will recognise some
problems with it.

First problem: 

How do you know what value to pass if you want the default? Is 999 out of
range? How about 11? 10? Who knows? If you have to look up the docs to know
what counts as out of range, you might as well read the docs to find out
what the default it, and just pass that:

    hello(5, 1, 2)  # I want the default value 5

but that kind of defeats the purpose of a default. The whole point of a
default is that you shouldn't need to pass *anything at all*, not even a
placeholder.

(If you need a placeholder, then you probably need to change your function
parameters.)

But at least with sentinels like None, or Ellipsis, it is *obvious* that the
value is probably a placeholder. With a placeholder like 11 or 999, it
isn't. They look like ordinary values.


Second problem:

Most of the time, we don't pass literal values to toy functions. We do
something like this example:

    for number, widget_ID, delivery_date in active_orders:
        submit_order(number, widget_ID, delivery_date)

Can you see the bug? Of course you can't. There's no obvious bug. But little
do you know, one of the orders was accidentally entered with an out-of-range
value, let's say -1, and instead of getting an nice error message telling
you that there's a problem that you need to fix, the
submit_order() function silently replaces the erroneous value with the
default.

The bug here is that submit_order() function exceeds its authority.

The name tells us that it submits orders, but it also silently decides to
change invalid orders to valid orders using some default value. But this
fact isn't obvious from either the name or the code. You only learn this
fact by digging into the source code, or reading the documentation, and
let's be honest, nobody wants to do either of those unless you really have
to.

So when faced with an invalid order, instead of getting an error that you
can fix, or even silently skipping the bad order, the submit_order()
function silently changes it to a valid-looking but WRONG order that you
probably didn't want. And that costs real money.

The risk of this sort of bug comes directly from the design of the function.
While I suppose I must acknowledge that (hypothetically) there could be
use-cases for this sort of design, I maintain that in general this design is
a bug magnet: responsibility for changing out-of-range values to in-range
values belongs with the caller, not the called function.

The caller may delegate that responsibility to another:

    for number, widget_ID, delivery_date in active_orders:
        number = validate_or_replace(number)
        submit_order(number, widget_ID, delivery_date)

which is fine because it is explicit and right there in plain sight.

This then allows us to make the submit_order() far more resiliant: if it is
passed an invalid order, it can either fail fast, giving an obvious error,
or at least skip the invalid order and notify the responsible people.


--
Steven
_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From breamoreboy at gmail.com  Sun Dec 30 16:35:15 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Sun, 30 Dec 2018 21:35:15 +0000
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <006401d4a064$d2e19580$78a4c080$@verizon.net>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <006401d4a064$d2e19580$78a4c080$@verizon.net>
Message-ID: <q0bdig$mps$1@blaine.gmane.org>

On 30/12/2018 17:26, Avi Gross wrote:
> Replying to Steve's points. Again, it was not a serious design and said so
> but was an ACADEMIC exploration of what could be done. I fully agree with
> Steve that it is probably not a great idea to do this but note the original
> request might not have been a great one in the first place.
> 

The usual massive snip, but what has an academic exploration got to do 
with learning Python on the tutor mailing list?  From 
https://mail.python.org/mailman/listinfo/tutor "This list is for folks 
who want to ask questions regarding how to learn computer programming 
with the Python language and its standard library. "  Also note the 
latter, nothing in there about pandas dataframes.

Now will you please go away.


From avigross at verizon.net  Sun Dec 30 17:25:13 2018
From: avigross at verizon.net (Avi Gross)
Date: Sun, 30 Dec 2018 17:25:13 -0500
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <q0bdig$mps$1@blaine.gmane.org>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <006401d4a064$d2e19580$78a4c080$@verizon.net> <q0bdig$mps$1@blaine.gmane.org>
Message-ID: <008001d4a08e$8888bd90$999a38b0$@verizon.net>

Mark,

I will be happy to make a somewhat less brief reply to a reasonable enough question.

I admit I have not studied the charter for the group. The standard library is not something I consider as many distributions automatically add libraries like np and pandas. I do accept your point. I still think it reasonable to MENTION the existence of some functionality that the person might use as some people just want to solve a problem, but clearly detailed examples and tutorials are way beyond that scope. I have unfortunately had people who got a private reply tell me to put it on the forum where clearly many are not interested.

On to the main point about recent posts that you replied to. Someone asked a question on how to do something using python.

Some responses boiled down to you can't do that.

One suggested something similar  in another way by asking what could be done in a context to make it happen, implying there was no answer.

I took the bait and in a very limited way suggested a context. Some may think the answer is not much of an answer or not useful and they probably have a point.

What I call an ACADEMIC exercise will vary. It may simply mean that I do not perceive any personal need to do things that way BUT since you asked, here is something ...

The language at any particular time and in a particular implementation allows you to do some things that are not necessarily a good idea. It is fair to suggest people avoid doing that but unfair to say it is WRONG as in it should not work. A dumb example is features that are deprecated but still exist. The goal is to switch, not keep using it till the last minute. But for a program you will run today then toss, it works!

I have to approach my role here carefully and appreciate that some choices will not be met well by some, including of course you. I can ignore some requests, and mainly do. There are others who can reply, often better. But when something catches my interest, it more often is not about an aspect of very basic python. I can reply and suggest the person not bother doing that, or perhaps not that way, or perhaps not use python. That might indeed teach them some things. Or, I can give a measured reply helping them see what can legally be done and maybe point out some tradeoffs including why it may not be the suggested way.

Last point. I promise! We often debate efficiency here. In retrospect, some people just want to solve a problem so all they may want is to find their error and let them do it as planned. Others ask if there is a more efficient way. Their question invites another approach. Unfortunately, that is when I often enter into an academic discussion. Maybe I should go back into teaching and bore people elsewhere. ? I am sure that would make you happy for as long as 5 minutes.


-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of Mark Lawrence
Sent: Sunday, December 30, 2018 4:35 PM
To: tutor at python.org
Subject: Re: [Tutor] Defining variable arguments in a function in python

On 30/12/2018 17:26, Avi Gross wrote:
> Replying to Steve's points. Again, it was not a serious design and 
> said so but was an ACADEMIC exploration of what could be done. I fully 
> agree with Steve that it is probably not a great idea to do this but 
> note the original request might not have been a great one in the first place.
> 

The usual massive snip, but what has an academic exploration got to do with learning Python on the tutor mailing list?  From https://mail.python.org/mailman/listinfo/tutor "This list is for folks who want to ask questions regarding how to learn computer programming with the Python language and its standard library. "  Also note the latter, nothing in there about pandas dataframes.

Now will you please go away.

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


From alan.gauld at yahoo.co.uk  Sun Dec 30 19:39:07 2018
From: alan.gauld at yahoo.co.uk (Alan Gauld)
Date: Mon, 31 Dec 2018 00:39:07 +0000
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <008001d4a08e$8888bd90$999a38b0$@verizon.net>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <006401d4a064$d2e19580$78a4c080$@verizon.net> <q0bdig$mps$1@blaine.gmane.org>
 <008001d4a08e$8888bd90$999a38b0$@verizon.net>
Message-ID: <q0bob7$k0e$1@blaine.gmane.org>

On 30/12/2018 22:25, Avi Gross wrote:

> I admit I have not studied the charter for the group.

As moderator I feel the need to step in here because the
charter is extremely apropos to that function and some
clarification may be helpful.

Mark is correct in that the group is focused on the
core language and standard library as documented on
python.org.

That is, it does not cover third party modules
like SciPy or Pillow or frameworks like wxPython or
Django etc, regardless of how popular these may be.

That having been said, we do normally allow questions
on these modules to go through because there is a fair
amount of in-house expertise and so a fair chance
of providing an answer without protracted discussion.

The group also has a secondary mission included in
the statement Mark quoted, which is to help beginners
"learn computer programming" and that aspect of our
charter encompasses the occasional non-python led
topic on the nature of programming and the wider areas
of software engineering - although we prefer topics to
be illustrated using Python. Occasionally, language
comparisons are valid in highlighting areas where
Python is stronger or weaker relative to other tools.

The final valid area for discussion is the meta topic
of what (and how) the group should discuss. But that
subject raises its head very rarely these days.

> The standard library is not something I consider as 
> many distributions automatically add libraries ...

While that is true, the term 'standard library' is
generally understood to be the one distributed from
and documented with the official language. In our
case that means the python.org web site.

Distributions like Anaconda, Entropy, Maya, and
RaspberryPi etc all exist for specific audiences and
add modules suited to those tasks. But they also
have their own support communities better qualified
to deal with their issues.

Sometimes the best answer is a simple referral to
one of those community fora.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos



From steve at pearwood.info  Mon Dec 31 00:53:21 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 31 Dec 2018 16:53:21 +1100
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <005d01d4a059$be289090$3a79b1b0$@verizon.net>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <005d01d4a059$be289090$3a79b1b0$@verizon.net>
Message-ID: <20181231055320.GL13616@ando.pearwood.info>

On Sun, Dec 30, 2018 at 11:07:19AM -0500, Avi Gross wrote:
> Steve,
> 
> I had the same thoughts and many more when I played with these ideas 
> last night.

Pity that one of those thoughts wasn't "I shouldn't suggest a bad 
solution on a mailing list populated by beginners who won't recognise 
how bad it is". That would have saved both you and me a lot of time.

> I thought I stated clearly that this was an EXPLORATION 

And I explored the problems with the design.

[...]
> Consider a NORMAL function with no external gimmicks. Say it 
> accepts an argument (positional or by name does not matter) and 
> processes it by checking they argument type, the specific values 
> against a range expected, and so on. It then unilaterally makes 
> changes. If it expects a single value of type integer, ang gets a 
> floating point number, it may round or truncate it back to an integer.
[...]

Sure. But in general, such a design applies a known transformation of 
input argument to value actually used:

- any number -> ignore any fraction part;
- any string -> ignore leading and trailing whitespace;
- any string -> treat as case-insensitive;
- list of values -> sorted list of values;

which is not the same replacing out-of-band values with some default 
value. In each of the above examples, there is a known, direct 
connection between the caller's argument, and the value used:

- the argument, ignoring any fraction part;
- the argument, ignoring leading and trailing whitespace;
- the argument, ignoring case differences;
- the argument, ignoring the original order of items;

rather than:

- throw away the argument, and use an unrelated default value.

The closest analogue would be clamping values to a given size, so 
that out-of-band values are truncated to the outermost value:

    clamp(999, min=1, max=10)
    => returns 10

but even that should be used with care. And as I said, it is acceptable 
to use an dedicated sentinel value like None or Ellipsis.

But using otherwise valid-looking, but out of range, values as a trigger 
for the default should be used with care, if at all.

Some designs are better (more likely to be useful, less likely to lead 
to bugs and surprises) than others.


> If your choice is in some sense wrong, or just not convenient, it 
> replaces it with another choice. 

That would very likely be a poor or dangerous design. If your argument 
is "wrong", that's an error, and errors should not in general pass 
silently.


[...]
> Again, discussing a different scenario. Would you agree that kind of 
> design does happen and sometimes is seen as a smart thing but also as 
> full of pitfalls even sometimes for the wary?

Sure, it does happen, often badly. 


> In that spirit, any design like the ones I played with is equally 
> scary. Even worse, I have a QUESTION. Let me remind readers of the 
> initial idea behind this. You have a function in which you want to 
> communicate with the python ruler that it have a positional argument 
> that will be switched to a default. But the current rules are that the 
> only way to make something optional is by making it a keyword 
> parameter and declare the default right there.

No, that's not the current rules. Positional values can have default 
values too. You just can't skip positional values: there's no way of 
saying "I'm not supplying argument 1, but I am supplying argument 2" 
using only positional arguments:

    function(a, b)  # provide argument 1 and 2

    function(a)     # only provide argument 1

    function(b)     still argument 1, not argument 2

[...]
> The problem is if you want a non-key word to be used as a default and 
> then also want to gather up additional positional arguments and so on. 
> There is no way designed to do that and possibly there should not be.

I'm open to suggestions for good designs that allow more flexible 
calling conventions, but as Python exists now, you can't skip over 
positional arguments except at the end.

> So what I am trying to understand is this. When someone types in a 
> function invocation and it is evaluated, when does the programmer get 
> a chance to intervene?

Aside from compile-time errors, any exception can be caught and 
processed *by the caller*:

    try:
        function(a, b)
    except TypeError as e:
        ...

Python also allows you to customize the way exceptions are printed:

https://docs.python.org/3/library/sys.html#sys.excepthook

But the callee -- in this case, "function()" -- doesn't get a chance to 
customize the TypeError generated by the interpreter when you pass the 
wrong number of arguments, or an invalid keyword argument.

(Aside: it is unfortunate that TypeError gets used for both actual type 
errors, and mismatched arguments/parameter errors.)

[...]
> Is there a way to create a function and set it up so you have some 
> control over the error message?

Not easily.

You could, I suppose, take over parameter parsing yourself. Write your 
function like this:

def function(*args, **kwargs):
    ...

and then do all the parameter parsing yourself. Then you can provide any 
error messages you like.

But that's a lot of trouble for not much benefit. I have, however, done 
it myself in order to support keyword-only arguments in Python 2. Its a 
real PITA.


-- 
Steve

From david at graniteweb.com  Mon Dec 31 11:35:45 2018
From: david at graniteweb.com (David Rock)
Date: Mon, 31 Dec 2018 10:35:45 -0600
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <q0bob7$k0e$1@blaine.gmane.org>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <006401d4a064$d2e19580$78a4c080$@verizon.net> <q0bdig$mps$1@blaine.gmane.org>
 <008001d4a08e$8888bd90$999a38b0$@verizon.net> <q0bob7$k0e$1@blaine.gmane.org>
Message-ID: <46B8CB01-E534-4CE3-BDBD-1CD2162B2FB9@graniteweb.com>


> On Dec 30, 2018, at 18:39, Alan Gauld via Tutor <tutor at python.org> wrote:
> 
> On 30/12/2018 22:25, Avi Gross wrote:
> 
>> I admit I have not studied the charter for the group.
> 
> As moderator I feel the need to step in here because the
> charter is extremely apropos to that function and some
> clarification may be helpful.

I would like to add an observation?

The core complaint appears to be the length of the postings.  To my eye, Avi?s style of posting is extremely verbose, which is not necessarily a bad thing; but perhaps taking some time to distill the thoughts to make a concise, on topic, point would be helpful in this case.  When discussions appear to ramble at length in odd tangents, the helpfulness to the beginner is diluted and the original point of the discussion is lost.


? 
David Rock
david at graniteweb.com





From robertvstepp at gmail.com  Mon Dec 31 13:50:45 2018
From: robertvstepp at gmail.com (boB Stepp)
Date: Mon, 31 Dec 2018 12:50:45 -0600
Subject: [Tutor] Defining variable arguments in a function in python
In-Reply-To: <46B8CB01-E534-4CE3-BDBD-1CD2162B2FB9@graniteweb.com>
References: <CAPi8shatT7Wpf4ju55NwgAtLTtSNmJGoKDPMg+FKXeXdi7Gd5w@mail.gmail.com>
 <20181229110152.GI13616@ando.pearwood.info>
 <000001d49ffd$8b560640$a20212c0$@verizon.net>
 <20181230103905.GK13616@ando.pearwood.info>
 <006401d4a064$d2e19580$78a4c080$@verizon.net>
 <q0bdig$mps$1@blaine.gmane.org> <008001d4a08e$8888bd90$999a38b0$@verizon.net>
 <q0bob7$k0e$1@blaine.gmane.org>
 <46B8CB01-E534-4CE3-BDBD-1CD2162B2FB9@graniteweb.com>
Message-ID: <CANDiX9+w9Q2kLR4KdTiBNbW=fvWhWZgwASYgpVcemSoUkLZ==A@mail.gmail.com>

On Mon, Dec 31, 2018 at 10:36 AM David Rock <david at graniteweb.com> wrote:
>
> > On Dec 30, 2018, at 18:39, Alan Gauld via Tutor <tutor at python.org> wrote:
> >
> > On 30/12/2018 22:25, Avi Gross wrote:
> >
> >> I admit I have not studied the charter for the group.
> >
> > As moderator I feel the need to step in here because the
> > charter is extremely apropos to that function and some
> > clarification may be helpful.
>
> I would like to add an observation?
>
> The core complaint appears to be the length of the postings.  To my eye, Avi?s style of posting is extremely verbose, which is not necessarily a bad thing; but perhaps taking some time to distill the thoughts to make a concise, on topic, point would be helpful in this case.  When discussions appear to ramble at length in odd tangents, the helpfulness to the beginner is diluted and the original point of the discussion is lost.

While I guess I am not considered here a rank beginner anymore, I
still know I have a LOT to learn.  In general I *try* to read all
posts on both the Tutor list and the main Python list.  However, I
must echo David's point -- I find Avi's posts way too long and
rambling, and while I am sure there are many useful nuggets to
explore, I find myself more and more often just doing a quick scan for
anything that just *pops out* that interests me, and, if not, hit
delete.

To Avi:  I know I would enjoy your posts much more if you would:  (1)
Get to the point quickly and stay on topic while addressing that
point.  Or, (2)  When you have a variety of disparate topics you would
like to discuss, break your one long post into a series of separate
emails, each with an appropriate subject line, and stick to each
post's topic implied by the subject line you choose.

I know I am often quite guilty on rambling on, so I know how hard it
is to do this.

I say all of this, Avi, not to be critical, but to hopefully enhance
everyone's opportunity to process and learn from your thoughts.

In any event, I hope all you Pythonistas have both a HAPPY AND BLESSED
NEW YEAR!!!

-- 
boB